From fm316383@gmail.com Fri Jan 1 09:44:59 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.3 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM,HTML_MESSAGE autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 71BB57F5D for ; Fri, 1 Jan 2016 09:44:59 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 6134B304059 for ; Fri, 1 Jan 2016 07:44:56 -0800 (PST) X-ASG-Debug-ID: 1451663093-04cbb0101a1b0a30001-NocioJ Received: from nh604-vm9.bullet.mail.ssk.yahoo.co.jp (nh604-vm9.bullet.mail.ssk.yahoo.co.jp [182.22.90.66]) by cuda.sgi.com with ESMTP id KWW3eGgt0yF7VmSk for ; Fri, 01 Jan 2016 07:44:54 -0800 (PST) X-Barracuda-Envelope-From: fm316383@gmail.com X-Barracuda-Apparent-Source-IP: 182.22.90.66 Received: from [182.22.66.104] by nh604.bullet.mail.ssk.yahoo.co.jp with NNFMP; 01 Jan 2016 15:44:52 -0000 Received: from [182.22.91.132] by t602.bullet.mail.ssk.yahoo.co.jp with NNFMP; 01 Jan 2016 15:44:52 -0000 Received: from [127.0.0.1] by omp605.mail.ssk.yahoo.co.jp with NNFMP; 01 Jan 2016 15:44:52 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 522202.35882.bm@omp605.mail.ssk.yahoo.co.jp Received: (qmail 83860 invoked by uid 60001); 1 Jan 2016 15:44:51 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=yj20050223; d=yahoo.co.jp; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:X-YMail-JAS:Sender:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=L4eajkGMPwDRomRElhrBG35hI4T+i34zp/M2tkefJtSDMQ8UZzKSlmmxDYC2BpF66SgOMTxzLsHT6uCP2mSOO122SQ7AKCiKM9a8jf3glcLJ6GdnNxSR7Tk+D7WDNvDb ; Message-ID: <640672.76249.qm@web101604.mail.kks.yahoo.co.jp> X-YMail-OSG: gBRL.KMVM1mwxlasxO_TDXkTSoDvVGKNcf2xxbADtgRzn3FOiWu1qJtA9JWZZSL63mI9_y4voOTOW1JpAIHmx4I9ptsD5BoXoTkApV6MpDGhQuVHt7s6yuqslMNJ8MoZtntFR8h1fA760GZ5vznhWlJMTU2FV228suq8guC91aS48cZ8dPz.fCvhfFeXjR8FQTRfrAHAtw41NTm.OA5Dyf5uxAKjECKGhyj3d6BHSEzCNIXPpbwWL_XyqCsUTOYwODN5nfycVAVmW3RzFZd3cSEpq8z2NQQ6qcCvin3aQCTri8QCfRSxp37YBsRARtTkrkQPQoUwy8BbaScBCWRo2j.uE.Z1crG6SUPtqPXvJKMurDc4n0Y4dsxtmf05V0nMLKV79z1DeFF5zi4dJYWg959rxpko1VEm0s.ugGUABuG7GlNb8EMXwAGPVTycJaFHFjCCKWijZIz8Y3NHNJFQ1bGQ6n6Tmr.rkF2vJ_27_EUT6oTyc6eHohzPPiUQi3EaWfHK_1uu2gCFwH8pIEEla7LobfRqgx_xB..vkj6bwb3YzVzsr.v9vkl7doVwrwf27OsqsgNLAC5Amni0f2mFzo4OHVhZk2o.XoY5LAqDz9Xm6qoa_RoxXAUc9p3QmnZIBmit.rxNAsSKkmhv3DRBgMrOUONODYnCMcd1iDy73.DdWaynxML7tTMPV2.VsrL2c1_fTqc79PWZrsb1L6FADjPoxCQBCeg7miNZ7O2vf17268q6BkD1SLclhzNCvf0De8.nsd3dqOGDCaZu.C8Eztn1lpnMMvMo Received: from [197.148.123.121] by web101604.mail.kks.yahoo.co.jp via HTTP; Sat, 02 Jan 2016 00:44:51 JST X-RocketYMMF: mrglenfrank X-Mailer: YahooMailWebService/0.8.111_67 X-YMail-JAS: nvTofeEVM1mxKUXuA.0naHAuYMqg8imDHQ9yXAVastJ4NQPM8C2Pi70HoxHReGFUn1mHpgwxTNUFjqcbvfg6qnej55VE3d0tipldPTA9CF3QQf4eIUHXM.f7cTO5t4LX8Czi Sender: mrglenfrank@yahoo.co.jp Date: Sat, 2 Jan 2016 00:44:51 +0900 (JST) From: Fred Martins Reply-To: Fred Martins Subject: May Almighty God Bless You!!! To: undisclosed recipients: ; X-ASG-Orig-Subj: May Almighty God Bless You!!! MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="2024049926-1842152024-1451663091=:76249" X-Barracuda-Connect: nh604-vm9.bullet.mail.ssk.yahoo.co.jp[182.22.90.66] X-Barracuda-Start-Time: 1451663093 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.46 X-Barracuda-Spam-Status: No, SCORE=0.46 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE, PLING_PLING X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25749 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 0.00 HTML_MESSAGE BODY: HTML included in message 0.46 PLING_PLING Subject has lots of exclamation marks --2024049926-1842152024-1451663091=:76249 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable =A0May Almighty God Bless You!!!=0A=0AI have a Mutual/Beneficial Business P= roject that would be benificial to you. I only have two questions to ask of= you,=0Aif you are interested.=0A=0A1. Can you handle this project?=0A2. Ca= n I give you this trust ?=0A=0APlease note that the deal requires high leve= l of maturity, honesty and secrecy. This will involve moving=A0 some money = from my office, on trust to your hands or bank account.=0AAlso note that i = will do everything to make sure that the money is moved as a purely legitim= ate fund, so you will not be exposed to any risk.=0AI request for your full= co-operation. I will give you details and procedure when I recieve your re= ply.=0ATo commence this transaction, I require you to immediately indicate= =A0 your interest by a return reply.=0AI will be waiting for your response = in a timely manner.=0A=0ABest Regard,=0AMr. Fred Martins=0A --2024049926-1842152024-1451663091=:76249 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
 May Almighty God Bless You!!!
I have a Mutual/Beneficial Business Project that would be benificial to y= ou. I only have two questions to ask of you,
if you are interested.
<= br>1. Can you handle this project?
2. Can I give you this trust ?
Please note that the deal requires high level of maturity, honesty and sec= recy. This will involve moving  some money from my office, on trust to= your hands or bank account.
Also note that i will do everything to make= sure that the money is moved as a purely legitimate fund, so you will not = be exposed to any risk.
I request for your full co-operation. I will giv= e you details and procedure when I recieve your reply.
To commence this = transaction, I require you to immediately indicate  your interest by a= return reply.
I will be waiting for your response in a timely manner.

Best Regard,
Mr. Fred Martins
--2024049926-1842152024-1451663091=:76249-- From sandeen@sandeen.net Fri Jan 1 17:40:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D842529DF6 for ; Fri, 1 Jan 2016 17:40:17 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 695D2AC002 for ; Fri, 1 Jan 2016 15:40:13 -0800 (PST) X-ASG-Debug-ID: 1451691607-04cbb0101d1b61c0001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id JRn8duPdcVVeiUxD for ; Fri, 01 Jan 2016 15:40:07 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from [30.34.82.18] (unknown [172.56.16.88]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id EC8C463C5FF0; Fri, 1 Jan 2016 17:40:06 -0600 (CST) From: Eric Sandeen Content-Type: multipart/alternative; boundary=Apple-Mail-92929011-3B27-4518-AAA8-7F1F5F943123 Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) Subject: Re: xfs_growfs can expand capacity,then how to reduce xfs filesystem capacity? Message-Id: X-ASG-Orig-Subj: Re: xfs_growfs can expand capacity,then how to reduce xfs filesystem capacity? Date: Fri, 1 Jan 2016 15:40:06 -0800 References: <1CB94550540AE44E9CAAE8CD5AF40BF846D18C5E@H3CMLB12-EX.srv.huawei-3com.com> In-Reply-To: <1CB94550540AE44E9CAAE8CD5AF40BF846D18C5E@H3CMLB12-EX.srv.huawei-3com.com> To: "zhao.mingyue@h3c.com" , xfs@oss.sgi.com X-Mailer: iPhone Mail (13C75) X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1451691607 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.92 X-Barracuda-Spam-Status: No, SCORE=0.92 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA085, HTML_MESSAGE, MIME_QP_LONG_LINE, MIME_QP_LONG_LINE_2 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25757 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 HTML_MESSAGE BODY: HTML included in message 0.00 MIME_QP_LONG_LINE RAW: Quoted-printable line longer than 76 chars 0.82 MIME_QP_LONG_LINE_2 RAW: Quoted-printable line longer than 76 chars 0.10 BSF_SC0_SA085 Custom Rule SA085 --Apple-Mail-92929011-3B27-4518-AAA8-7F1F5F943123 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable http://xfs.org/index.php/XFS_FAQ#Q:_Is_there_a_way_to_make_a_XFS_filesystem_= larger_or_smaller.3F > On Dec 2, 2015, at 6:11 PM, "zhao.mingyue@h3c.com" w= rote: >=20 > =20 > --------------------------------------------------------------------------= ----------------------------------------------------------- > =E6=9C=AC=E9=82=AE=E4=BB=B6=E5=8F=8A=E5=85=B6=E9=99=84=E4=BB=B6=E5=90=AB=E6= =9C=89=E6=9D=AD=E5=B7=9E=E5=8D=8E=E4=B8=89=E9=80=9A=E4=BF=A1=E6=8A=80=E6=9C=AF= =E6=9C=89=E9=99=90=E5=85=AC=E5=8F=B8=E7=9A=84=E4=BF=9D=E5=AF=86=E4=BF=A1=E6=81= =AF=EF=BC=8C=E4=BB=85=E9=99=90=E4=BA=8E=E5=8F=91=E9=80=81=E7=BB=99=E4=B8=8A=E9= =9D=A2=E5=9C=B0=E5=9D=80=E4=B8=AD=E5=88=97=E5=87=BA > =E7=9A=84=E4=B8=AA=E4=BA=BA=E6=88=96=E7=BE=A4=E7=BB=84=E3=80=82=E7=A6=81=E6= =AD=A2=E4=BB=BB=E4=BD=95=E5=85=B6=E4=BB=96=E4=BA=BA=E4=BB=A5=E4=BB=BB=E4=BD=95= =E5=BD=A2=E5=BC=8F=E4=BD=BF=E7=94=A8=EF=BC=88=E5=8C=85=E6=8B=AC=E4=BD=86=E4=B8= =8D=E9=99=90=E4=BA=8E=E5=85=A8=E9=83=A8=E6=88=96=E9=83=A8=E5=88=86=E5=9C=B0=E6= =B3=84=E9=9C=B2=E3=80=81=E5=A4=8D=E5=88=B6=E3=80=81 > =E6=88=96=E6=95=A3=E5=8F=91=EF=BC=89=E6=9C=AC=E9=82=AE=E4=BB=B6=E4=B8=AD=E7= =9A=84=E4=BF=A1=E6=81=AF=E3=80=82=E5=A6=82=E6=9E=9C=E6=82=A8=E9=94=99=E6=94=B6= =E4=BA=86=E6=9C=AC=E9=82=AE=E4=BB=B6=EF=BC=8C=E8=AF=B7=E6=82=A8=E7=AB=8B=E5=8D= =B3=E7=94=B5=E8=AF=9D=E6=88=96=E9=82=AE=E4=BB=B6=E9=80=9A=E7=9F=A5=E5=8F=91=E4= =BB=B6=E4=BA=BA=E5=B9=B6=E5=88=A0=E9=99=A4=E6=9C=AC > =E9=82=AE=E4=BB=B6=EF=BC=81 > This e-mail and its attachments contain confidential information from H3C,= which is=20 > intended only for the person or entity whose address is listed above. Any u= se of the=20 > information contained herein in any way (including, but not limited to, to= tal or partial=20 > disclosure, reproduction, or dissemination) by persons other than the inte= nded=20 > recipient(s) is prohibited. If you receive this e-mail in error, please no= tify the sender=20 > by phone or email immediately and delete it! > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs --Apple-Mail-92929011-3B27-4518-AAA8-7F1F5F943123 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

On Dec 2, 2015, at 6:11 PM, "zhao.mingyue@h3c.com" <zhao.mingyue@h3c.com> wrote:

=

 

--------------------------------------= ----------------------------------------------------------------------------= -------------------
=E6=9C=AC=E9=82=AE=E4=BB=B6=E5=8F=8A=E5=85=B6=E9=99=84=E4=BB=B6=E5=90= =AB=E6=9C=89=E6=9D=AD=E5=B7=9E=E5=8D=8E=E4=B8=89=E9=80=9A=E4=BF=A1=E6=8A=80=E6= =9C=AF=E6=9C=89=E9=99=90=E5=85=AC=E5=8F=B8=E7=9A=84=E4=BF=9D=E5=AF=86=E4=BF=A1= =E6=81=AF=EF=BC=8C=E4=BB=85=E9=99=90=E4=BA=8E=E5=8F=91=E9=80=81=E7=BB=99=E4=B8= =8A=E9=9D=A2=E5=9C=B0=E5=9D=80=E4=B8=AD=E5=88=97=E5=87=BA
=E7=9A=84=E4=B8=AA=E4=BA=BA=E6=88=96=E7=BE=A4=E7=BB=84=E3=80=82=E7=A6= =81=E6=AD=A2=E4=BB=BB=E4=BD=95=E5=85=B6=E4=BB=96=E4=BA=BA=E4=BB=A5=E4=BB=BB=E4= =BD=95=E5=BD=A2=E5=BC=8F=E4=BD=BF=E7=94=A8=EF=BC=88=E5=8C=85=E6=8B=AC=E4=BD=86= =E4=B8=8D=E9=99=90=E4=BA=8E=E5=85=A8=E9=83=A8=E6=88=96=E9=83=A8=E5=88=86=E5=9C= =B0=E6=B3=84=E9=9C=B2=E3=80=81=E5=A4=8D=E5=88=B6=E3=80=81
=E6=88=96=E6=95=A3=E5=8F=91=EF=BC=89=E6=9C=AC=E9=82=AE=E4=BB=B6=E4=B8= =AD=E7=9A=84=E4=BF=A1=E6=81=AF=E3=80=82=E5=A6=82=E6=9E=9C=E6=82=A8=E9=94=99=E6= =94=B6=E4=BA=86=E6=9C=AC=E9=82=AE=E4=BB=B6=EF=BC=8C=E8=AF=B7=E6=82=A8=E7=AB=8B= =E5=8D=B3=E7=94=B5=E8=AF=9D=E6=88=96=E9=82=AE=E4=BB=B6=E9=80=9A=E7=9F=A5=E5=8F= =91=E4=BB=B6=E4=BA=BA=E5=B9=B6=E5=88=A0=E9=99=A4=E6=9C=AC
=E9=82=AE=E4=BB=B6=EF=BC=81
This e-mail and its attac= hments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any us= e of the
information contained herein in any way (including, but not limited to, tota= l or partial
disclosure, reproduction, or dissemination) by persons other than the intend= ed
recipient(s) is prohibited. If you receive this e-mail in error, please noti= fy the sender
by phone or email immediately and delete it!
____________________= ___________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/l= istinfo/xfs
= --Apple-Mail-92929011-3B27-4518-AAA8-7F1F5F943123-- From s_w9gc9x@student.uni-kassel.de Sat Jan 2 18:26:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=LOTS_OF_MONEY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 774237F5D for ; Sat, 2 Jan 2016 18:26:18 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 187B6AC005 for ; Sat, 2 Jan 2016 16:26:15 -0800 (PST) X-ASG-Debug-ID: 1451780767-04cbb0101b1ca9b0001-NocioJ Received: from mx4.its.uni-kassel.de (mx4.its.uni-kassel.de [141.51.12.63]) by cuda.sgi.com with ESMTP id VtOYFZXVplPKhw3S for ; Sat, 02 Jan 2016 16:26:08 -0800 (PST) X-Barracuda-Envelope-From: s_w9gc9x@student.uni-kassel.de X-Barracuda-Apparent-Source-IP: 141.51.12.63 X-IPAS-Result: A2Ao5wAYaohW/5GqM40bQ4MkCDAwHEIBAQ6DN3SELaopi1EOgwiCXgEWgUwBAQEBAQECgQkYAQGCGIF+LkMoIAImAkkWLYUvB4F3bAqfBYddiBKEDIJJikKBAY56EQGDAQwuE4E2BYJwhnKDV3eIVgGBNYNPiRSBFIcLPAQjhTETijSDc2SBSgEKAQEBAYJPIIQCgUIBAQE X-IronPort-AV: E=Sophos;i="5.20,514,1444687200"; d="scan'208";a="25896876" X-UKS-Mail: fzgewiuhfgadjshfgeijrgzaiwefghieuwfgehjfgiw54567456757fguhklghfrg57gtftjhjtd X-CGP-ClamAV-Result: CLEAN X-VirusScanner: Niversoft's CGPClamav Helper v1.18.7a (ClamAV engine v0.98.7) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body Subject: Charity Donation To: Me X-ASG-Orig-Subj: Charity Donation From: "Jeff Skoll" Date: Sat, 02 Jan 2016 18:49:39 -0500 Reply-To: jeff.skoll@skollcharity.org X-Priority: 1 (High) Sensitivity: Company-Confidential Message-ID: X-Barracuda-Connect: mx4.its.uni-kassel.de[141.51.12.63] X-Barracuda-Start-Time: 1451780767 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=ADVANCE_FEE_1, BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25782 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 ADVANCE_FEE_1 Appears to be advance fee fraud (Nigerian 419) Hi, My name is Jeffrey Skoll, a philanthropist and the founder of one of the la= rgest private foundations in the world. I believe strongly in =E2=80=98givi= ng while living.=E2=80=99 I had one idea that never changed in my mind =E2= =80=94 that you should use your wealth to help people and I have decided to= secretly give USD2.498 Million to a randomly selected individual. On recei= pt of this email, you should count yourself as the individual. Kindly get b= ack to me at your earliest convenience, so I know your email address is val= id. Visit the web page to know more about me: http://www.theglobeandmail.com/ne= ws/national/meet-the-canadian-billionaire-whos-giving-it-all-away/article42= 09888/ or you can read an article of me on Wikipedia. Regards, Jeffrey Skoll. From s_w9gc9x@student.uni-kassel.de Sat Jan 2 18:59:45 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=LOTS_OF_MONEY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 892707F5D for ; Sat, 2 Jan 2016 18:59:45 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 5A7C38F8033 for ; Sat, 2 Jan 2016 16:59:45 -0800 (PST) X-ASG-Debug-ID: 1451782779-04cbb0101a1cb020001-NocioJ Received: from mx4.its.uni-kassel.de (mx4.its.uni-kassel.de [141.51.12.63]) by cuda.sgi.com with ESMTP id 1AI2p5QEgmgauh2b for ; Sat, 02 Jan 2016 16:59:40 -0800 (PST) X-Barracuda-Envelope-From: s_w9gc9x@student.uni-kassel.de X-Barracuda-Apparent-Source-IP: 141.51.12.63 X-IPAS-Result: A2Ao5wAycYhW/5CqM40bQ4MkCDAwHEIBAQ6DN3SELaopi1EOgwiCXgEWgUwBAQEBAQECgQkYAQGCGIF+LkMoIAImAkkWLYUvB4F3bAqfAYddiBKEDIJJikSBAY56EQGDAQwuE4E2BYJwhnKDV3eIVgGBNYNPiRSBFIcLPAQjhTETijSDc2SBSgEKAQEBAYJPIIQCgUIBAQE X-IronPort-AV: E=Sophos;i="5.20,514,1444687200"; d="scan'208";a="25969998" X-UKS-Mail: fzgewiuhfgadjshfgeijrgzaiwefghieuwfgehjfgiw54567456757fguhklghfrg57gtftjhjtd X-CGP-ClamAV-Result: CLEAN X-VirusScanner: Niversoft's CGPClamav Helper v1.18.7a (ClamAV engine v0.98.7) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body Subject: Charity Donation To: Me X-ASG-Orig-Subj: Charity Donation From: "Jeff Skoll" Date: Sat, 02 Jan 2016 19:36:28 -0500 Reply-To: jeff.skoll@skollcharity.org X-Priority: 1 (High) Sensitivity: Company-Confidential Message-ID: X-Barracuda-Connect: mx4.its.uni-kassel.de[141.51.12.63] X-Barracuda-Start-Time: 1451782779 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=ADVANCE_FEE_1, BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25782 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 ADVANCE_FEE_1 Appears to be advance fee fraud (Nigerian 419) Hi, My name is Jeffrey Skoll, a philanthropist and the founder of one of the la= rgest private foundations in the world. I believe strongly in =E2=80=98givi= ng while living.=E2=80=99 I had one idea that never changed in my mind =E2= =80=94 that you should use your wealth to help people and I have decided to= secretly give USD2.498 Million to a randomly selected individual. On recei= pt of this email, you should count yourself as the individual. Kindly get b= ack to me at your earliest convenience, so I know your email address is val= id. Visit the web page to know more about me: http://www.theglobeandmail.com/ne= ws/national/meet-the-canadian-billionaire-whos-giving-it-all-away/article42= 09888/ or you can read an article of me on Wikipedia. Regards, Jeffrey Skoll. From s_w9gc9x@student.uni-kassel.de Sun Jan 3 02:00:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=LOTS_OF_MONEY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 20DB47F5D for ; Sun, 3 Jan 2016 02:00:03 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 02206304032 for ; Sun, 3 Jan 2016 00:00:03 -0800 (PST) X-ASG-Debug-ID: 1451807990-04cb6c3032407b40002-NocioJ Received: from mx4.its.uni-kassel.de (mx4.its.uni-kassel.de [141.51.12.63]) by cuda.sgi.com with ESMTP id 4UDV6KH7mAXsW3Ik for ; Sat, 02 Jan 2016 23:59:59 -0800 (PST) X-Barracuda-Envelope-From: s_w9gc9x@student.uni-kassel.de X-Barracuda-Apparent-Source-IP: 141.51.12.63 X-IPAS-Result: A2BhqQBK1IhW/5GqM40bQ4MkCDAwHAFBAQEOgzd0hC2qKYtRDoMIgl4BFoFMAQEBAQEBAoEJGAEBghiBfi5DKCACJgJJFi2FLweBd2wKnnCHXYgShAyCSYpEgQGOehEBgwEMLhOBNgWCcIZyg1d3iFYBgTWDT4kUgRSHCzwEI4UxE4o0g3NkgUoBCgEBAQGCTyCEAoFCAQEB X-IronPort-AV: E=Sophos;i="5.20,515,1444687200"; d="scan'208";a="26079350" X-UKS-Mail: fzgewiuhfgadjshfgeijrgzaiwefghieuwfgehjfgiw54567456757fguhklghfrg57gtftjhjtd X-CGP-ClamAV-Result: CLEAN X-VirusScanner: Niversoft's CGPClamav Helper v1.18.7a (ClamAV engine v0.98.7) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body Subject: Charity Donation To: Me X-ASG-Orig-Subj: Charity Donation From: "Jeff Skoll" Date: Sun, 03 Jan 2016 02:59:54 -0500 Reply-To: jeff.skoll@skollcharity.org X-Priority: 1 (High) Sensitivity: Company-Confidential Message-ID: X-Barracuda-Connect: mx4.its.uni-kassel.de[141.51.12.63] X-Barracuda-Start-Time: 1451807994 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=ADVANCE_FEE_1, BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25789 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 ADVANCE_FEE_1 Appears to be advance fee fraud (Nigerian 419) Hi, My name is Jeffrey Skoll, a philanthropist and the founder of one of the la= rgest private foundations in the world. I believe strongly in =E2=80=98givi= ng while living.=E2=80=99 I had one idea that never changed in my mind =E2= =80=94 that you should use your wealth to help people and I have decided to= secretly give USD2.498 Million to a randomly selected individual. On recei= pt of this email, you should count yourself as the individual. Kindly get b= ack to me at your earliest convenience, so I know your email address is val= id. Visit the web page to know more about me: http://www.theglobeandmail.com/ne= ws/national/meet-the-canadian-billionaire-whos-giving-it-all-away/article42= 09888/ or you can read an article of me on Wikipedia. Regards, Jeffrey Skoll. From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:09:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 7DE1E7F37 for ; Sun, 3 Jan 2016 06:09:41 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 2D4FEAC004 for ; Sun, 3 Jan 2016 04:09:41 -0800 (PST) X-ASG-Debug-ID: 1451822978-04bdf0264e42fb40001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id mdBisgMlHur88DXF (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:09:38 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from [83.175.99.196] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhTd-0007fZ-Ru; Sun, 03 Jan 2016 12:09:38 +0000 From: Christoph Hellwig To: darrick.wong@oracle.com Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: [PATCH 1/3] xfs: pass inode instead of file to xfs_reflink_dirty_range Date: Sun, 3 Jan 2016 13:07:51 +0100 X-ASG-Orig-Subj: [PATCH 1/3] xfs: pass inode instead of file to xfs_reflink_dirty_range Message-Id: <1451822873-12969-2-git-send-email-hch@lst.de> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1451822873-12969-1-git-send-email-hch@lst.de> References: <1451822873-12969-1-git-send-email-hch@lst.de> X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451822978 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 We don't actually need a file for write_begin/end, those can be passed as NULL for disk based file systems. This is important as we won't even have a file pointer during a truncate operation, which gives a guaranteed NULL pointer dererference with the current code. Signed-off-by: Christoph Hellwig --- fs/xfs/xfs_reflink.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c index da4a715..4a3f0ee 100644 --- a/fs/xfs/xfs_reflink.c +++ b/fs/xfs/xfs_reflink.c @@ -1316,7 +1316,7 @@ out_error: */ STATIC int xfs_reflink_dirty_range( - struct file *filp, + struct inode *inode, xfs_off_t pos, xfs_off_t len) { @@ -1330,14 +1330,14 @@ xfs_reflink_dirty_range( unsigned long bytes; /* Bytes to write to page */ void *fsdata; - mapping = filp->f_mapping; + mapping = inode->i_mapping; a_ops = mapping->a_ops; flags = AOP_FLAG_UNINTERRUPTIBLE; do { offset = (pos & (PAGE_CACHE_SIZE - 1)); bytes = min_t(unsigned long, len, PAGE_CACHE_SIZE) - offset; - rpage = xfs_get_page(file_inode(filp), pos); + rpage = xfs_get_page(inode, pos); if (IS_ERR(rpage)) { error = PTR_ERR(rpage); break; @@ -1346,24 +1346,24 @@ xfs_reflink_dirty_range( break; } - error = a_ops->write_begin(filp, mapping, pos, bytes, flags, + error = a_ops->write_begin(NULL, mapping, pos, bytes, flags, &page, &fsdata); page_cache_release(rpage); if (error < 0) break; - trace_xfs_reflink_unshare_page(file_inode(filp), page, + trace_xfs_reflink_unshare_page(inode, page, pos, bytes); if (!PageUptodate(page)) { - pr_err("%s: STALE? ino=%lu pos=%llu\n", - __func__, filp->f_inode->i_ino, pos); + pr_err("%s: STALE? ino=%llu pos=%llu\n", + __func__, XFS_I(inode)->i_ino, pos); WARN_ON(1); } if (mapping_writably_mapped(mapping)) flush_dcache_page(page); - error = a_ops->write_end(filp, mapping, pos, bytes, bytes, + error = a_ops->write_end(NULL, mapping, pos, bytes, bytes, page, fsdata); if (error < 0) break; @@ -1454,7 +1454,7 @@ xfs_reflink_dirty_extents( flen = XFS_FSB_TO_B(mp, rlen); if (fpos + flen > isize) flen = isize - fpos; - error = xfs_reflink_dirty_range(filp, fpos, flen); + error = xfs_reflink_dirty_range(VFS_I(ip), fpos, flen); xfs_ilock(ip, XFS_ILOCK_EXCL); if (error) goto out; -- 1.9.1 From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:09:43 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id ADEE37F47 for ; Sun, 3 Jan 2016 06:09:43 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 3EA59AC005 for ; Sun, 3 Jan 2016 04:09:39 -0800 (PST) X-ASG-Debug-ID: 1451822976-04cbb0101a1d42b0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id e4t8MBBEmU0DfEIx (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:09:36 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from [83.175.99.196] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhTa-0007fK-Vq; Sun, 03 Jan 2016 12:09:35 +0000 From: Christoph Hellwig To: darrick.wong@oracle.com Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: reflink fixes Date: Sun, 3 Jan 2016 13:07:50 +0100 X-ASG-Orig-Subj: reflink fixes Message-Id: <1451822873-12969-1-git-send-email-hch@lst.de> X-Mailer: git-send-email 1.9.1 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451822976 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 Hi Darrick, below are three patches I need to xfstests to pass reliably for the reflink code. I actually still see a lot of issues when testing NFS clones on XFS due to extents hanging on the COW for too long, but that's a separate issue I still need to investigate. From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:09:44 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 14A247F51 for ; Sun, 3 Jan 2016 06:09:44 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 9A826AC001 for ; Sun, 3 Jan 2016 04:09:43 -0800 (PST) X-ASG-Debug-ID: 1451822981-04cbb0101b1d42d0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id eyFl5Gq20cYFZpMR (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:09:42 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from [83.175.99.196] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhTg-0007hN-Ma; Sun, 03 Jan 2016 12:09:41 +0000 From: Christoph Hellwig To: darrick.wong@oracle.com Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: [PATCH 2/3] xfs: only end a COW operation in xfs_zero_remaining_bytes if we started one Date: Sun, 3 Jan 2016 13:07:52 +0100 X-ASG-Orig-Subj: [PATCH 2/3] xfs: only end a COW operation in xfs_zero_remaining_bytes if we started one Message-Id: <1451822873-12969-3-git-send-email-hch@lst.de> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1451822873-12969-1-git-send-email-hch@lst.de> References: <1451822873-12969-1-git-send-email-hch@lst.de> X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451822981 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 Without these we can see NULL pointer dereferences due to a non-existing COW fork during xfstests runs. Signed-off-by: Christoph Hellwig --- fs/xfs/xfs_bmap_util.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c index 7bee3c7..e777095 100644 --- a/fs/xfs/xfs_bmap_util.c +++ b/fs/xfs/xfs_bmap_util.c @@ -1226,13 +1226,15 @@ xfs_zero_remaining_bytes( error = xfs_bwrite(bp); xfs_buf_relse(bp); - if (error) { - err2 = xfs_reflink_end_cow_failed(ip, offset, + if (should_fork) { + if (error) { + err2 = xfs_reflink_end_cow_failed(ip, offset, + lastoffset - offset + 1); + return error; + } + error = xfs_reflink_end_cow(ip, offset, lastoffset - offset + 1); - return error; } - error = xfs_reflink_end_cow(ip, offset, - lastoffset - offset + 1); if (error) return error; } -- 1.9.1 From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:09:50 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 974DE7F47 for ; Sun, 3 Jan 2016 06:09:50 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 86FBE8F8037 for ; Sun, 3 Jan 2016 04:09:50 -0800 (PST) X-ASG-Debug-ID: 1451822984-04cb6c303140acf0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id uHuhLrmfm9B97UfG (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:09:44 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from [83.175.99.196] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhTj-0007hV-HS; Sun, 03 Jan 2016 12:09:43 +0000 From: Christoph Hellwig To: darrick.wong@oracle.com Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Date: Sun, 3 Jan 2016 13:07:53 +0100 X-ASG-Orig-Subj: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-Id: <1451822873-12969-4-git-send-email-hch@lst.de> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1451822873-12969-1-git-send-email-hch@lst.de> References: <1451822873-12969-1-git-send-email-hch@lst.de> X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451822984 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 Otherwise we leak COW allocations done earlier in writepage. This can be reproduced fairly easily when we hit the non-blocking writeback EAGAIN case. Signed-off-by: Christoph Hellwig --- fs/xfs/xfs_aops.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 185415a..9c69dc3 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -588,6 +588,7 @@ xfs_cancel_ioend( { xfs_ioend_t *next; struct buffer_head *bh, *next_bh; + int error; do { next = ioend->io_list; @@ -605,6 +606,12 @@ xfs_cancel_ioend( unlock_buffer(bh); } while ((bh = next_bh) != NULL); + if (ioend->io_flags & XFS_IOEND_COW) { + error = xfs_reflink_end_cow_failed( + XFS_I(ioend->io_inode), + ioend->io_offset, ioend->io_size); + WARN_ON_ONCE(error); + } mempool_free(ioend, xfs_ioend_pool); } while ((ioend = next) != NULL); } -- 1.9.1 From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:13:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6527D7F4E for ; Sun, 3 Jan 2016 06:13:56 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 5354B8F8035 for ; Sun, 3 Jan 2016 04:13:56 -0800 (PST) X-ASG-Debug-ID: 1451823234-04cbb0101c1d44b0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id gRCgcEdkOPJ91XKQ (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:13:55 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhXl-00011s-So; Sun, 03 Jan 2016 12:13:53 +0000 Date: Sun, 3 Jan 2016 04:13:53 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 02/76] xfs: fix log ticket type printing Message-ID: <20160103121353.GA28429@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 02/76] xfs: fix log ticket type printing References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219085636.12713.59155.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219085636.12713.59155.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823234 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Sat, Dec 19, 2015 at 12:56:36AM -0800, Darrick J. Wong wrote: > Update the log ticket reservation type printing code to reflect > all the types of log tickets, to avoid incorrect debug output and > avoid running off the end of the array. This looks okay, but I think we should use name array initializers to make sure it doesn't got out of sync ever. Also I don't think we'd ever run past the array as the users does a size check. From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:15:31 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 649737F4E for ; Sun, 3 Jan 2016 06:15:31 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 471B4304032 for ; Sun, 3 Jan 2016 04:15:28 -0800 (PST) X-ASG-Debug-ID: 1451823325-04bdf0265042fca0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id HP63iPaVZujyrl1p (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:15:25 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhZF-0002Jo-AV; Sun, 03 Jan 2016 12:15:25 +0000 Date: Sun, 3 Jan 2016 04:15:25 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 05/76] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct Message-ID: <20160103121525.GB28429@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 05/76] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219085655.12713.60317.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219085655.12713.60317.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823325 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Sat, Dec 19, 2015 at 12:56:55AM -0800, Darrick J. Wong wrote: > Because struct xfs_agfl is 36 bytes long and has a 64-bit integer > inside it, gcc will quietly round the structure size up to the nearest > 64 bits -- in this case, 40 bytes. This results in the XFS_AGFL_SIZE > macro returning incorrect results for v5 filesystems on 64-bit > machines (118 items instead of 119). As a result, a 32-bit xfs_repair > will see garbage in AGFL item 119 and complain. > > Therefore, tell gcc not to pad the structure so that the AGFL size > calculation is correct. Do you have a testcase for this? From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:17:31 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id CE1CA7F37 for ; Sun, 3 Jan 2016 06:17:31 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 544E2AC003 for ; Sun, 3 Jan 2016 04:17:31 -0800 (PST) X-ASG-Debug-ID: 1451823449-04cb6c303240aea0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id PMltMAXhjqORWQKd (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:17:29 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhbE-0002P3-Rp; Sun, 03 Jan 2016 12:17:28 +0000 Date: Sun, 3 Jan 2016 04:17:28 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 34/76] xfs: implement XFS_IOC_SWAPEXT when rmap btree is enabled Message-ID: <20160103121728.GC28429@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 34/76] xfs: implement XFS_IOC_SWAPEXT when rmap btree is enabled References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090008.12713.25755.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219090008.12713.25755.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823449 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Sat, Dec 19, 2015 at 01:00:08AM -0800, Darrick J. Wong wrote: > Implement extent swapping when reverse-mapping is enabled. Can you just fold this into the previous patch? There seem to be a lot of patches in the series that would benefit from merging, but this is the one that really screams for it :) From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:18:59 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id DAFF37F37 for ; Sun, 3 Jan 2016 06:18:59 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 7A883AC002 for ; Sun, 3 Jan 2016 04:18:59 -0800 (PST) X-ASG-Debug-ID: 1451823512-04cbb0101d1d45c0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id f5X7FpIVXZfUUBd6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:18:33 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhcG-0002Sf-30; Sun, 03 Jan 2016 12:18:32 +0000 Date: Sun, 3 Jan 2016 04:18:32 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 35/76] libxfs: refactor short btree block verification Message-ID: <20160103121832.GD28429@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 35/76] libxfs: refactor short btree block verification References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090014.12713.75657.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219090014.12713.75657.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823512 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS > +/** > + * xfs_btree_sblock_v5hdr_verify() -- verify the v5 fields of a short-format > + * btree block > + * > + * @bp: buffer containing the btree block > + * @max_recs: pointer to the m_*_mxr max records field in the xfs mount > + * @pag_max_level: pointer to the per-ag max level field > + */ We're so far avoided using kerneldoc comments and they silly super verbosity in XFS. It might be good to tone this down to the normal XFS level (same for some other patches). Otherwise looks good: Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:21:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 29BA07F58 for ; Sun, 3 Jan 2016 06:21:01 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id F19E5304053 for ; Sun, 3 Jan 2016 04:21:00 -0800 (PST) X-ASG-Debug-ID: 1451823658-04bdf0264f42fe30001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id gCDpCC5IoRGQQe3a (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:20:59 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhec-0003i2-9i; Sun, 03 Jan 2016 12:20:58 +0000 Date: Sun, 3 Jan 2016 04:20:58 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 57/76] xfs: allocate delayed extents in CoW fork Message-ID: <20160103122058.GE28429@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 57/76] xfs: allocate delayed extents in CoW fork References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090248.12713.32309.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219090248.12713.32309.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823659 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS I really don't like the separate XFS_IOEND_COW flag. This should be an ioend type just like XFS_IO_DELALLOC, XFS_IO_UNWRITTEN and XFS_IO_OVERWRITE. We might need some work in the direct I/O completions for that to really work, though. From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:22:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 14D9D7F5A for ; Sun, 3 Jan 2016 06:22:13 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id A7A7CAC003 for ; Sun, 3 Jan 2016 04:22:12 -0800 (PST) X-ASG-Debug-ID: 1451823730-04cbb0101b1d4710001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id gXQiAyWowKdBHDCb (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:22:11 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhfm-0003kQ-1l; Sun, 03 Jan 2016 12:22:10 +0000 Date: Sun, 3 Jan 2016 04:22:10 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 01/22] xfsdocs: fix asciidoc confusion w.r.t. include file newline at EOF Message-ID: <20160103122210.GA14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 01/22] xfsdocs: fix asciidoc confusion w.r.t. include file newline at EOF References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091215.15558.79820.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091215.15558.79820.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823730 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Sat, Dec 19, 2015 at 01:12:15AM -0800, Darrick J. Wong wrote: > Space each include one line apart so that asciidoc knows that each > section has ended at included file's EOF. This prevents it from > spewing horribly confused XML that blows up xmllint if each included > file doesn't itself end with a blank line. Looks fine, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:22:46 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 6FDAA7F51 for ; Sun, 3 Jan 2016 06:22:46 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 0C45BAC001 for ; Sun, 3 Jan 2016 04:22:45 -0800 (PST) X-ASG-Debug-ID: 1451823762-04bdf02650430000001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id LJnRI2HtCmuWBrEv (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:22:42 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhgH-0003nj-OO; Sun, 03 Jan 2016 12:22:41 +0000 Date: Sun, 3 Jan 2016 04:22:41 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 02/22] xfsdocs: update book subtitles and headings Message-ID: <20160103122241.GB14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 02/22] xfsdocs: update book subtitles and headings References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091221.15558.28333.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091221.15558.28333.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823762 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks fine, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:23:07 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D6F287F51 for ; Sun, 3 Jan 2016 06:23:07 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 6BD37AC001 for ; Sun, 3 Jan 2016 04:23:07 -0800 (PST) X-ASG-Debug-ID: 1451823785-04cb6c302f40afe0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id EQ4ZHmdL17Ybqihi (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:23:06 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhgf-0003od-1X; Sun, 03 Jan 2016 12:23:05 +0000 Date: Sun, 3 Jan 2016 04:23:05 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 03/22] xfsdocs: restore the old rendering of structure field definitions Message-ID: <20160103122305.GC14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 03/22] xfsdocs: restore the old rendering of structure field definitions References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091227.15558.73975.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091227.15558.73975.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823786 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks fine, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:23:37 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id A26607F51 for ; Sun, 3 Jan 2016 06:23:37 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 324BAAC002 for ; Sun, 3 Jan 2016 04:23:37 -0800 (PST) X-ASG-Debug-ID: 1451823815-04cbb0101a1d4760001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id AySEF8f8E1qyq3Qo (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:23:35 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhh8-0003qI-Gd; Sun, 03 Jan 2016 12:23:34 +0000 Date: Sun, 3 Jan 2016 04:23:34 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 04/22] xfsdocs: convert html entities to regular characters Message-ID: <20160103122334.GD14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 04/22] xfsdocs: convert html entities to regular characters References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091234.15558.35406.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091234.15558.35406.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823815 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks fine, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:23:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 26BC27F51 for ; Sun, 3 Jan 2016 06:23:56 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id AD074AC001 for ; Sun, 3 Jan 2016 04:23:55 -0800 (PST) X-ASG-Debug-ID: 1451823833-04cbb0101d1d4780001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id JOcxjRPcr8OytxVE (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:23:54 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhhR-0003sq-23; Sun, 03 Jan 2016 12:23:53 +0000 Date: Sun, 3 Jan 2016 04:23:53 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 05/22] xfsdocs: convert images to text Message-ID: <20160103122353.GE14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 05/22] xfsdocs: convert images to text References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091242.15558.50391.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091242.15558.50391.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823833 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks fine, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:24:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 893177F51 for ; Sun, 3 Jan 2016 06:24:41 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 2CBA18F8035 for ; Sun, 3 Jan 2016 04:24:41 -0800 (PST) X-ASG-Debug-ID: 1451823879-04cb6c302f40b040001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id X765gceaKNfb2pr6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:24:40 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhiA-000469-RQ; Sun, 03 Jan 2016 12:24:38 +0000 Date: Sun, 3 Jan 2016 04:24:38 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 07/22] xfsdocs: update types Message-ID: <20160103122438.GF14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 07/22] xfsdocs: update types References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091259.15558.84471.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091259.15558.84471.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823880 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks fine, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:25:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 89B7A7F51 for ; Sun, 3 Jan 2016 06:25:08 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 390F1AC001 for ; Sun, 3 Jan 2016 04:25:08 -0800 (PST) X-ASG-Debug-ID: 1451823906-04bdf02651430330001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 9no1YcSSIQsQ1Fi2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:25:06 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhic-00059j-5b; Sun, 03 Jan 2016 12:25:06 +0000 Date: Sun, 3 Jan 2016 04:25:06 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 08/22] xfsdocs: make example section titles consistent Message-ID: <20160103122506.GG14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 08/22] xfsdocs: make example section titles consistent References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091305.15558.97231.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091305.15558.97231.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823906 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:25:42 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 821717F51 for ; Sun, 3 Jan 2016 06:25:42 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 631BF304032 for ; Sun, 3 Jan 2016 04:25:42 -0800 (PST) X-ASG-Debug-ID: 1451823940-04bdf02650430360001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id Vj003xTeCOm8PNUb (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:25:41 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhjA-0005SW-I1; Sun, 03 Jan 2016 12:25:40 +0000 Date: Sun, 3 Jan 2016 04:25:40 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 09/22] xfsdocs: fix various errors and missing bits in the text Message-ID: <20160103122540.GH14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 09/22] xfsdocs: fix various errors and missing bits in the text References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091311.15558.80832.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091311.15558.80832.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823941 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:26:12 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 2FD317F51 for ; Sun, 3 Jan 2016 06:26:11 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 7B3E6AC001 for ; Sun, 3 Jan 2016 04:26:11 -0800 (PST) X-ASG-Debug-ID: 1451823970-04bdf0264e430370001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id hQ1F0KLA8Wq8PXBc (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:26:10 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhje-0005YZ-0A; Sun, 03 Jan 2016 12:26:10 +0000 Date: Sun, 3 Jan 2016 04:26:09 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 10/22] xfsdocs: add missing structure field definitions Message-ID: <20160103122609.GI14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 10/22] xfsdocs: add missing structure field definitions References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091318.15558.52742.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091318.15558.52742.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823970 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:26:35 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id B64CB7F55 for ; Sun, 3 Jan 2016 06:26:35 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 57BA0AC004 for ; Sun, 3 Jan 2016 04:26:35 -0800 (PST) X-ASG-Debug-ID: 1451823993-04cbb0101a1d4810001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id Zj6SaqljADYYi5lh (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:26:34 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhk1-0005c1-34; Sun, 03 Jan 2016 12:26:33 +0000 Date: Sun, 3 Jan 2016 04:26:33 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 11/22] xfsdocs: add missing xfs_db examples Message-ID: <20160103122633.GJ14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 11/22] xfsdocs: add missing xfs_db examples References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091325.15558.37906.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091325.15558.37906.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451823994 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:26:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 234867F7B for ; Sun, 3 Jan 2016 06:26:56 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id B67D2AC004 for ; Sun, 3 Jan 2016 04:26:55 -0800 (PST) X-ASG-Debug-ID: 1451824014-04bdf0264e430380001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 8gHwUk0M7TkYAKp2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:26:55 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhkM-0005gg-FL; Sun, 03 Jan 2016 12:26:54 +0000 Date: Sun, 3 Jan 2016 04:26:54 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 12/22] xfsdocs: fix quoted text markings Message-ID: <20160103122654.GK14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 12/22] xfsdocs: fix quoted text markings References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091331.15558.64509.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091331.15558.64509.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451824014 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:28:02 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 46DD67F7C for ; Sun, 3 Jan 2016 06:28:02 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 15F228F8033 for ; Sun, 3 Jan 2016 04:28:02 -0800 (PST) X-ASG-Debug-ID: 1451824080-04bdf026504303d0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 9C9JmrWdF8ylvuPA (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:28:01 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhlQ-0005n8-FZ; Sun, 03 Jan 2016 12:28:00 +0000 Date: Sun, 3 Jan 2016 04:28:00 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 13/22] xfsdocs: introduce XFS at a high level Message-ID: <20160103122800.GL14361@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 13/22] xfsdocs: introduce XFS at a high level References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091338.15558.24376.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091338.15558.24376.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451824081 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25794 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks great! Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 06:31:39 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 4B55D7F6B for ; Sun, 3 Jan 2016 06:31:39 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 392788F8033 for ; Sun, 3 Jan 2016 04:31:39 -0800 (PST) X-ASG-Debug-ID: 1451824297-04cb6c303240b1d0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id XhVHNYsPRYzuAROV (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 04:31:38 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFhou-0007L1-Vl; Sun, 03 Jan 2016 12:31:36 +0000 Date: Sun, 3 Jan 2016 04:31:36 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 06/22] xfsdocs: delete images of hex dumps Message-ID: <20160103123136.GA27913@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 06/22] xfsdocs: delete images of hex dumps References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091249.15558.18497.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091249.15558.18497.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451824298 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25793 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 07:40:52 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id CFC6E7F81 for ; Sun, 3 Jan 2016 07:40:52 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id BED7B8F8033 for ; Sun, 3 Jan 2016 05:40:49 -0800 (PST) X-ASG-Debug-ID: 1451828443-04cbb0101a1d5be0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id fXoR2VmqVBKMSvpo (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 05:40:44 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFitm-0004ZX-K7; Sun, 03 Jan 2016 13:40:42 +0000 Date: Sun, 3 Jan 2016 05:40:42 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 14/22] xfsdocs: document the xfs log format Message-ID: <20160103134042.GA9144@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 14/22] xfsdocs: document the xfs log format References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091344.15558.83896.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091344.15558.83896.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451828444 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25795 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 07:41:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id D03DF7F7B for ; Sun, 3 Jan 2016 07:41:56 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id B1B4B304032 for ; Sun, 3 Jan 2016 05:41:53 -0800 (PST) X-ASG-Debug-ID: 1451828512-04bdf0264e431490001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 4USlGAfHahtiruWm (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 05:41:52 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFiut-0004yF-JA; Sun, 03 Jan 2016 13:41:51 +0000 Date: Sun, 3 Jan 2016 05:41:51 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 15/22] xfsdocs: document the operation of the realtime device and inodes Message-ID: <20160103134151.GB9144@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 15/22] xfsdocs: document the operation of the realtime device and inodes References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091351.15558.51417.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091351.15558.51417.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451828512 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25795 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks generally good, although I wonder the take on the realtime device is a bit too positive given that we generally advise people not to use it. Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 07:46:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9432C7F7B for ; Sun, 3 Jan 2016 07:46:25 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 25BA9AC002 for ; Sun, 3 Jan 2016 05:46:25 -0800 (PST) X-ASG-Debug-ID: 1451828782-04cbb0101d1d5d90001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 6iaGTqVT0KcjIbam (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 05:46:23 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFizG-0006PK-1B; Sun, 03 Jan 2016 13:46:22 +0000 Date: Sun, 3 Jan 2016 05:46:22 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 16/22] xfsdocs: add documentation of v5 fields Message-ID: <20160103134621.GA19752@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 16/22] xfsdocs: add documentation of v5 fields References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091358.15558.23715.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091358.15558.23715.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451828783 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25795 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 07:47:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 734667F7B for ; Sun, 3 Jan 2016 07:47:01 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 43F02304032 for ; Sun, 3 Jan 2016 05:47:01 -0800 (PST) X-ASG-Debug-ID: 1451828819-04bdf0264e4315e0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id NKW2EkepUgiVtElU (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 05:46:59 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFizr-0006QI-BO; Sun, 03 Jan 2016 13:46:59 +0000 Date: Sun, 3 Jan 2016 05:46:59 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 22/22] xfsdocs: move directory chapter before xattr chapter Message-ID: <20160103134659.GB19752@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 22/22] xfsdocs: move directory chapter before xattr chapter References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091437.15558.91782.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091437.15558.91782.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451828819 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25795 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 07:54:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E4A3E7F7B for ; Sun, 3 Jan 2016 07:54:03 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 73C6FAC003 for ; Sun, 3 Jan 2016 05:54:03 -0800 (PST) X-ASG-Debug-ID: 1451829241-04cb6c303140cd80001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 4eIXdxzrT3UFz4GG (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 05:54:02 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFj6e-0008Bx-W7; Sun, 03 Jan 2016 13:54:01 +0000 Date: Sun, 3 Jan 2016 05:54:00 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 17/22] xfsdocs: add a chapter discussing v5 disk format Message-ID: <20160103135400.GA30378@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 17/22] xfsdocs: add a chapter discussing v5 disk format References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091405.15558.33731.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091405.15558.33731.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451829241 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25795 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 07:54:34 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 0D3187F7B for ; Sun, 3 Jan 2016 07:54:34 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id F07CE304039 for ; Sun, 3 Jan 2016 05:54:33 -0800 (PST) X-ASG-Debug-ID: 1451829269-04cb6c303040cd90001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id B9CJAikAAJ8lRjEO (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 05:54:30 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFj76-0008P2-SS; Sun, 03 Jan 2016 13:54:28 +0000 Date: Sun, 3 Jan 2016 05:54:28 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: hch@infradead.org, david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 18/22] xfsdocs: document magic numbers and theoretical limits Message-ID: <20160103135428.GB30378@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 18/22] xfsdocs: document magic numbers and theoretical limits References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091411.15558.9135.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091411.15558.9135.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451829269 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25795 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, : Christoph Hellwig From BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 3 07:55:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 1D4657F7B for ; Sun, 3 Jan 2016 07:55:18 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id EE3658F8035 for ; Sun, 3 Jan 2016 05:55:17 -0800 (PST) X-ASG-Debug-ID: 1451829316-04bdf026504317b0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id qcr3chfB929PFHu8 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 05:55:16 -0800 (PST) X-Barracuda-Envelope-From: BATV+79bc6ce57c55c66c7247+4515+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aFj7r-0001Mb-K0; Sun, 03 Jan 2016 13:55:15 +0000 Date: Sun, 3 Jan 2016 05:55:15 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: david@fromorbit.com, xfs@oss.sgi.com Subject: Re: [PATCH 19/22] xfsdocs: document the sparse inodes feature Message-ID: <20160103135515.GC30378@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 19/22] xfsdocs: document the sparse inodes feature References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091418.15558.27198.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219091418.15558.27198.stgit@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451829316 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25795 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks good, Reviewed-by: Christoph Hellwig From dan.j.williams@intel.com Sun Jan 3 12:13:15 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6702D7F37 for ; Sun, 3 Jan 2016 12:13:15 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 588428F8033 for ; Sun, 3 Jan 2016 10:13:12 -0800 (PST) X-ASG-Debug-ID: 1451844786-04cbb0101c1da030001-NocioJ Received: from mail-yk0-f179.google.com (mail-yk0-f179.google.com [209.85.160.179]) by cuda.sgi.com with ESMTP id UCdaEiGMFC4qW2kO (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 10:13:07 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.160.179 Received: by mail-yk0-f179.google.com with SMTP id v14so136130211ykd.3 for ; Sun, 03 Jan 2016 10:13:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=THPfaZpE7+X0uM6y+NRIvVhHaCeXdS1lxSOT2ZjsP6I=; b=wSBA0euLFETAt/lRVPUj8s8y3wIIQ1/VTZRsK/MzWmLrg1Qwfi5KS0x2hyiqpe5vCk hfJJOLnO8oc6cOwuBJVZYlhFGVb8O5aqbnv3mVZJUJRnbO2El2Wn4P6yubS66KqgXM04 7Klkn9QM59Rs47DWxt1Wr7AAQizQK6wLxZnbCCvxRqeABSMQTL8guCk6Bn7XH4k7VAU+ qBwJtz/1VMVMAcYUpY99CGe+T2w9c8oTSHQZ1A579yVWW4PnqD/IrBN0SN/nCiSqWfcJ YFxiWx6NNs8dtbgggAV+jQ9GfwAxLdV4V9sRVxkQFBjxvlwbs1rrOhWnkkI0ft0DSB6k dHhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=THPfaZpE7+X0uM6y+NRIvVhHaCeXdS1lxSOT2ZjsP6I=; b=B47BLl08VZrIx1hnCzbpgVRtZ46EcMZhKFmbnupb/9vaubJ0Kk1WlCZSActtctGwCa ZequP2Sjwt07OjldBQFgIyvsQ9U4iDTBi6VKC8apq2LEjwX4lR/l5MX9kCiV1nhn9+Dj qquMBWmLk17uwRr8TOVT+secCs9HDbbQIXRozXNrUv0cliRKbRGB7cgEXt4k+lB9HsHW EwUBMwF55jK232vhCXldWmGowdTf4JmXjQ4he1q6eO0cqBkeYMAKXzH9OPT3yp/13bKm BzFybn09vs1BdhGQmU333YLchYtnVjtHRhTpisPhYpWb4aFYHCsuybNAIThPBlq1STwY pu2Q== X-Gm-Message-State: ALoCoQnjDgECfC1c5DgXMn6C8p81ckxI62SIo/ESso9dupRyD1D1OE/ZvO0d2LQ/WxY9QK/E3PukRnxzJmbwOmhrF8U6+F+KMnVeI8AOvUhFavgDhnsFbkY= MIME-Version: 1.0 X-Received: by 10.129.85.83 with SMTP id j80mr60067671ywb.28.1451844786282; Sun, 03 Jan 2016 10:13:06 -0800 (PST) Received: by 10.37.202.20 with HTTP; Sun, 3 Jan 2016 10:13:06 -0800 (PST) In-Reply-To: <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> Date: Sun, 3 Jan 2016 10:13:06 -0800 Message-ID: Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync From: Dan Williams X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync To: Ross Zwisler Cc: "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-yk0-f179.google.com[209.85.160.179] X-Barracuda-Start-Time: 1451844786 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25799 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Wed, Dec 23, 2015 at 11:39 AM, Ross Zwisler wrote: > To properly handle fsync/msync in an efficient way DAX needs to track dirty > pages so it is able to flush them durably to media on demand. > > The tracking of dirty pages is done via the radix tree in struct > address_space. This radix tree is already used by the page writeback > infrastructure for tracking dirty pages associated with an open file, and > it already has support for exceptional (non struct page*) entries. We > build upon these features to add exceptional entries to the radix tree for > DAX dirty PMD or PTE pages at fault time. > > Signed-off-by: Ross Zwisler I'm hitting the following report with the ndctl dax test [1] on next-20151231. I bisected it to commit 3cb108f941de "dax-add-support-for-fsync-sync-v6". I'll take a closer look tomorrow, but in case someone can beat me to it, here's the back-trace: ------------[ cut here ]------------ kernel BUG at fs/inode.c:497! [..] CPU: 1 PID: 3001 Comm: umount Tainted: G O 4.4.0-rc7+ #2412 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 task: ffff8800da2a5a00 ti: ffff880307794000 task.ti: ffff880307794000 RIP: 0010:[] [] clear_inode+0x71/0x80 RSP: 0018:ffff880307797d50 EFLAGS: 00010002 RAX: ffff8800da2a5a00 RBX: ffff8800ca2e7328 RCX: ffff8800da2a5a28 RDX: 0000000000000001 RSI: 0000000000000005 RDI: ffff8800ca2e7530 RBP: ffff880307797d60 R08: ffffffff82900ae0 R09: 0000000000000000 R10: ffff8800ca2e7548 R11: 0000000000000000 R12: ffff8800ca2e7530 R13: ffff8800ca2e7328 R14: ffff8800da2e88d0 R15: ffff8800da2e88d0 FS: 00007f2b22f4a880(0000) GS:ffff88031fc40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00005648abd933e8 CR3: 000000007f3fc000 CR4: 00000000000006e0 Stack: ffff8800ca2e7328 ffff8800ca2e7000 ffff880307797d88 ffffffffa01c18af ffff8800ca2e7328 ffff8800ca2e74d0 ffffffffa01ec740 ffff880307797db0 ffffffff81281038 ffff8800ca2e74c0 ffff880307797e00 ffff8800ca2e7328 Call Trace: [] xfs_fs_evict_inode+0x5f/0x110 [xfs] [] evict+0xb8/0x180 [] dispose_list+0x3b/0x50 [] evict_inodes+0x144/0x170 [] generic_shutdown_super+0x3f/0xf0 [] kill_block_super+0x27/0x70 [] deactivate_locked_super+0x43/0x70 [] deactivate_super+0x5c/0x60 [] cleanup_mnt+0x3f/0x90 [] __cleanup_mnt+0x12/0x20 [] task_work_run+0x76/0x90 [] syscall_return_slowpath+0x20a/0x280 [] int_ret_from_sys_call+0x25/0x9f Code: 48 8d 93 30 03 00 00 48 39 c2 75 23 48 8b 83 d0 00 00 00 a8 20 74 1a a8 40 75 18 48 c7 8 3 d0 00 00 00 60 00 00 00 5b 41 5c 5d c3 <0f> 0b 0f 0b 0f 0b 0f 0b 0f 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 RIP [] clear_inode+0x71/0x80 RSP ---[ end trace 3b1d8898a94a4fc1 ]--- [1]: git://git@github.com:pmem/ndctl.git pending make TESTS="test/dax.sh" check From david@fromorbit.com Sun Jan 3 14:48:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 687967F37 for ; Sun, 3 Jan 2016 14:48:08 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 01C60AC002 for ; Sun, 3 Jan 2016 12:48:04 -0800 (PST) X-ASG-Debug-ID: 1451854079-04bdf0264e4372e0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id Gl29kEOnwgNIhorr for ; Sun, 03 Jan 2016 12:48:00 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CCCwDjh4lWPGu7LXleKAECgw9SbYJihXegOBIGigaBT4k+HoVrBAICgRJNAQEBAQEBBwEBAQFBP4Q1AQEEOhwjEAgDGAklDwUlAwcaE4guv1wBCyEZhXWFR4JcgVsBAYUDAQSXBoVAiAiBZUqMVUSFFIEKD4dJhQIqNINOgUIBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 07:17:59 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aFpZG-00008J-Ar; Mon, 04 Jan 2016 07:47:58 +1100 Date: Mon, 4 Jan 2016 07:47:58 +1100 From: Dave Chinner To: Andrea Gelmini Cc: linux-kernel@vger.kernel.org, xfs@oss.sgi.com Subject: Re: BUG: KASAN: use-after-free in xfs_iflush_cluster+0x9d7/0xaf0 Message-ID: <20160103204758.GW19802@dastard> X-ASG-Orig-Subj: Re: BUG: KASAN: use-after-free in xfs_iflush_cluster+0x9d7/0xaf0 References: <20151214180048.GA15690@glen> <20151214195422.GM26718@dastard> <20151214201526.GA25152@glen> <20151214212220.GO26718@dastard> <20151215091145.GA19282@glen> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151215091145.GA19282@glen> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451854079 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-BRTS-Evidence: gelma.net X-Barracuda-Spam-Score: 2.60 X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0249, MARKETING_SUBJECT X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25802 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.60 MARKETING_SUBJECT Subject contains popular marketing words 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 2.00 BSF_SC0_MV0249 Custom rule MV0249 On Tue, Dec 15, 2015 at 10:11:45AM +0100, Andrea Gelmini wrote: > On Tue, Dec 15, 2015 at 08:22:20AM +1100, Dave Chinner wrote: > > $ gdb vmlinux > > .... > > (gdb) l *(xfs_iflush_cluster+0x9d7) > > It's not working. Shame on me, I forgot to set "Compile the kernel with debug info". > > I'm recompiling, to try it again. > Maybe, in the meanwhile, you can do something with my files. You can find 'em here: > http://mail.gelma.net/xfs_kasan Any update on this problem, Andrea? Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Sun Jan 3 15:06:36 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 92CB97F37 for ; Sun, 3 Jan 2016 15:06:36 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 18190AC002 for ; Sun, 3 Jan 2016 13:06:33 -0800 (PST) X-ASG-Debug-ID: 1451855185-04cbb0101a1dc490001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id pFYdkyxhgBUJmYPq for ; Sun, 03 Jan 2016 13:06:25 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2C9EACBjIlWPGu7LXleKAECgw+BP4Jig36BeaBKBoFliXCJPoYJAgIBAQKBEk0BAQEBAQEHAQEBAUE/hDUBAQQnExwYCQIQCAMYCSUPBSUDBxoTiC6/VwEBAQcCASAZhXWFR4QkhRgBBJMKg3yNSIFlhEaIWY46gnUcgXEqNAGDRIFLAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 07:35:45 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aFpqS-0000A0-Q7; Mon, 04 Jan 2016 08:05:44 +1100 Date: Mon, 4 Jan 2016 08:05:44 +1100 From: Dave Chinner To: Dan Williams Cc: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, xfs@oss.sgi.com Subject: Re: [PATCH 2/3] xfs: handle shutdown notifications Message-ID: <20160103210544.GX19802@dastard> X-ASG-Orig-Subj: Re: [PATCH 2/3] xfs: handle shutdown notifications References: <20151201235836.36836.90416.stgit@dwillia2-desk3.jf.intel.com> <20151201235846.36836.1476.stgit@dwillia2-desk3.jf.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151201235846.36836.1476.stgit@dwillia2-desk3.jf.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451855185 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25802 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Dec 01, 2015 at 03:58:46PM -0800, Dan Williams wrote: > Force a filesystem shutdown when the backing device is known to be dead. > I.e. blk_queue_enter() permanently returns -ENODEV. > > Cc: xfs@oss.sgi.com > Suggested-by: Dave Chinner > Signed-off-by: Dan Williams > --- > fs/block_dev.c | 3 ++- > fs/xfs/xfs_super.c | 9 +++++++++ > include/linux/fs.h | 2 ++ > 3 files changed, 13 insertions(+), 1 deletion(-) > > diff --git a/fs/block_dev.c b/fs/block_dev.c > index 39989e990df9..dfe9a53a7c53 100644 > --- a/fs/block_dev.c > +++ b/fs/block_dev.c > @@ -1802,7 +1802,7 @@ int __invalidate_device(struct block_device *bdev, bool kill_dirty) > } > EXPORT_SYMBOL(__invalidate_device); > > -static void generic_bdi_gone(struct super_block *sb) > +void generic_bdi_gone(struct super_block *sb) > { > struct inode *inode, *_inode = NULL; > > @@ -1828,6 +1828,7 @@ static void generic_bdi_gone(struct super_block *sb) > spin_unlock(&sb->s_inode_list_lock); > iput(_inode); > } > +EXPORT_SYMBOL(generic_bdi_gone); > > void shutdown_partition(struct gendisk *disk, int partno) > { > diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c > index 36bd8825bfb0..63c36508e9db 100644 > --- a/fs/xfs/xfs_super.c > +++ b/fs/xfs/xfs_super.c > @@ -1618,6 +1618,14 @@ xfs_fs_free_cached_objects( > return xfs_reclaim_inodes_nr(XFS_M(sb), sc->nr_to_scan); > } > > +static void > +xfs_fs_bdi_gone( > + struct super_block *sb) > +{ > + xfs_force_shutdown(XFS_M(sb), SHUTDOWN_DEVICE_REQ); > + generic_bdi_gone(sb); > +} You haven't cc'd me or the XFS list on the other patches in this series, so I cannot comment as to whether this will work or not. As I've asked before, please do not do single patch CC's like this as it does not give the reviewer the context to be able to review the change, nor is it possible to *apply and test* the patch they are being asked to comment on. Please resend the series with complete CC's to the relevant lists. Sure, go ahead an CC individual people on certain patches, but you need to make sure the whole patch series hit each mailing list.... Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Sun Jan 3 15:29:37 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 6EA377F37 for ; Sun, 3 Jan 2016 15:29:37 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 0A6C9AC002 for ; Sun, 3 Jan 2016 13:29:36 -0800 (PST) X-ASG-Debug-ID: 1451856574-04bdf02651437e00001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id ppdbFuMyZlC81sgz for ; Sun, 03 Jan 2016 13:29:34 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AWEAB3kolWPGu7LXleKAECgw+BP4Jig36BeaBQi1WJPoYJBAICgRJNAQEBAQEBBwEBAQFBP4Q1AQEEOhwjEAgDDgoJJQ8FJQMHGhOILr9OAQEIAiEZhXWFR4E8AYMHhHgFjTmJTY1IgWWNH4VYiGKDAoIAKjSFEAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 07:59:06 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aFqD3-0000Bg-Hp; Mon, 04 Jan 2016 08:29:05 +1100 Date: Mon, 4 Jan 2016 08:29:05 +1100 From: Dave Chinner To: Christoph Hellwig Cc: "Darrick J. Wong" , xfs@oss.sgi.com Subject: Re: [PATCH 02/76] xfs: fix log ticket type printing Message-ID: <20160103212905.GY19802@dastard> X-ASG-Orig-Subj: Re: [PATCH 02/76] xfs: fix log ticket type printing References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219085636.12713.59155.stgit@birch.djwong.org> <20160103121353.GA28429@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160103121353.GA28429@infradead.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451856574 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25803 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Sun, Jan 03, 2016 at 04:13:53AM -0800, Christoph Hellwig wrote: > On Sat, Dec 19, 2015 at 12:56:36AM -0800, Darrick J. Wong wrote: > > Update the log ticket reservation type printing code to reflect > > all the types of log tickets, to avoid incorrect debug output and > > avoid running off the end of the array. > > This looks okay, but I think we should use name array initializers > to make sure it doesn't got out of sync ever. Good idea, separate patch is fine. Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Sun Jan 3 15:31:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 49AAB7F37 for ; Sun, 3 Jan 2016 15:31:08 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3A417304032 for ; Sun, 3 Jan 2016 13:31:05 -0800 (PST) X-ASG-Debug-ID: 1451856662-04cb6c30314139c0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id Q4QSwc3BbGlokUrC for ; Sun, 03 Jan 2016 13:31:02 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AZEAB3kolWPGu7LXleKAECgw+BP4Jig36BeaBKBotViT6GCQQCAoESTQEBAQEBAQcBAQEBQT+ENQEBBDocIxAIAw4KCSUPBSUDBxoTiC6/TgEBCAIhGYV1hUeJPAWXBo1IjwSOOoUCKjSFEAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 08:00:38 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aFqEY-0000Br-CB; Mon, 04 Jan 2016 08:30:38 +1100 Date: Mon, 4 Jan 2016 08:30:38 +1100 From: Dave Chinner To: Christoph Hellwig Cc: "Darrick J. Wong" , xfs@oss.sgi.com Subject: Re: [PATCH 35/76] libxfs: refactor short btree block verification Message-ID: <20160103213038.GZ19802@dastard> X-ASG-Orig-Subj: Re: [PATCH 35/76] libxfs: refactor short btree block verification References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090014.12713.75657.stgit@birch.djwong.org> <20160103121832.GD28429@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160103121832.GD28429@infradead.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451856662 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25802 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Sun, Jan 03, 2016 at 04:18:32AM -0800, Christoph Hellwig wrote: > > +/** > > + * xfs_btree_sblock_v5hdr_verify() -- verify the v5 fields of a short-format > > + * btree block > > + * > > + * @bp: buffer containing the btree block > > + * @max_recs: pointer to the m_*_mxr max records field in the xfs mount > > + * @pag_max_level: pointer to the per-ag max level field > > + */ > > We're so far avoided using kerneldoc comments and they silly super > verbosity in XFS. It might be good to tone this down to the normal > XFS level (same for some other patches). I'll do that before I commit it. Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Sun Jan 3 21:58:44 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9D7F529DF5 for ; Sun, 3 Jan 2016 21:58:44 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 7228D304032 for ; Sun, 3 Jan 2016 19:58:44 -0800 (PST) X-ASG-Debug-ID: 1451879921-04bdf0265143df00001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id MIKYD4waopokCJJQ for ; Sun, 03 Jan 2016 19:58:41 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CtCwAc7YlWPGu7LXlWCCgBAoMPUm2CYoV3oEYBAQEBAQEGi1WJPhgKhWcEAgKBF00BAQEBAQEHAQEBAUE/hDQBAQEDAQEBASQTHBgLBQsIAxgJJQ8FExIDBxoTiCcHD79ZAQEBAQEBBAEBAQEfGYV1hUeCbYFFg2+BGwWXBoVAiAiPBI46gnUcgXEqNIUQAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 14:28:41 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aFwI4-0000hQ-AP; Mon, 04 Jan 2016 14:58:40 +1100 Date: Mon, 4 Jan 2016 14:58:40 +1100 From: Dave Chinner To: Eric Sandeen Cc: xfs@oss.sgi.com Subject: Re: [PATCH V2] xfs: create helper for bmap finish & trans join in attr code Message-ID: <20160104035840.GA19802@dastard> X-ASG-Orig-Subj: Re: [PATCH V2] xfs: create helper for bmap finish & trans join in attr code References: <56441B8E.6070603@redhat.com> <5644BEF8.6070201@sandeen.net> <20151112165801.GA14854@infradead.org> <20151112201231.GS19199@dastard> <20151113090840.GA16423@infradead.org> <56465643.5070504@sandeen.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56465643.5070504@sandeen.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451879921 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25809 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Fri, Nov 13, 2015 at 03:29:39PM -0600, Eric Sandeen wrote: > On 11/13/15 3:08 AM, Christoph Hellwig wrote: > > On Fri, Nov 13, 2015 at 07:12:31AM +1100, Dave Chinner wrote: > >> On Thu, Nov 12, 2015 at 08:58:01AM -0800, Christoph Hellwig wrote: > >>> I think the problem here is simply that our interfaces suck. > >>> xfs_trans_roll really needs to rejoin any inode to the new transaction > >>> to that was joined to the previous one. Once we've fixed that we can > >>> get rid of the silly committed arguments and everyone will be happy. > >> > >> xfs_trans_roll is not specifically for rolling transactions with > >> locked inodes in them. We could use it for any object that needs > >> multiple transactions to modify. e.g. we could roll transactions > >> across an AGF (using hold+join) so that it remains locked across > >> multiple allocation/free transactions. > > > > xfs_trans_roll already logs the inode core, which requires the > > inode to be attached to the transaction. While I could see the > > point of moving this out of the core __xfs_trans_roll into an > > xfs_trans_roll_inode helper we might as well follow the current > > interface for now. > > Trying to follow you guys ;) > > Something like this? yes, something like that ;) (better late than never!) -Dave. > diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c > index dbae649..d23bce8 100644 > --- a/fs/xfs/xfs_bmap_util.c > +++ b/fs/xfs/xfs_bmap_util.c > @@ -91,32 +91,32 @@ xfs_zero_extent( > * last due to locking considerations. We never free any extents in > * the first transaction. > * > - * Return 1 if the given transaction was committed and a new one > - * started, and 0 otherwise in the committed parameter. > + * If an inode *ip is provided, rejoin it to the transaction if > + * the transaction was committed. > */ > int /* error */ > xfs_bmap_finish( > struct xfs_trans **tp, /* transaction pointer addr */ > struct xfs_bmap_free *flist, /* i/o: list extents to free */ > - int *committed)/* xact committed or not */ > + xfs_inode_t *ip) > { > struct xfs_efd_log_item *efd; /* extent free data */ > struct xfs_efi_log_item *efi; /* extent free intention */ > int error; /* error return value */ > + int committed;/* xact committed or not */ > struct xfs_bmap_free_item *free; /* free extent item */ > struct xfs_bmap_free_item *next; /* next item on free list */ > > ASSERT((*tp)->t_flags & XFS_TRANS_PERM_LOG_RES); > - if (flist->xbf_count == 0) { > - *committed = 0; > + if (flist->xbf_count == 0) > return 0; > - } > + > efi = xfs_trans_get_efi(*tp, flist->xbf_count); > for (free = flist->xbf_first; free; free = free->xbfi_next) > xfs_trans_log_efi_extent(*tp, efi, free->xbfi_startblock, > free->xbfi_blockcount); > > - error = __xfs_trans_roll(tp, NULL, committed); > + error = __xfs_trans_roll(tp, ip, &committed); > if (error) { > /* > * If the transaction was committed, drop the EFD reference > @@ -128,16 +128,13 @@ xfs_bmap_finish( > * transaction so we should return committed=1 even though we're > * returning an error. > */ > - if (*committed) { > + if (committed) { > xfs_efi_release(efi); > xfs_force_shutdown((*tp)->t_mountp, > (error == -EFSCORRUPTED) ? > SHUTDOWN_CORRUPT_INCORE : > SHUTDOWN_META_IO_ERROR); > - } else { > - *committed = 1; > } > - > return error; > } > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > -- Dave Chinner david@fromorbit.com From debbugs@buxtehude.debian.org Sun Jan 3 22:57:28 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E4D0429DF5 for ; Sun, 3 Jan 2016 22:57:28 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id C38AA304032 for ; Sun, 3 Jan 2016 20:57:25 -0800 (PST) X-ASG-Debug-ID: 1451883437-04cb6c302f41a750001-NocioJ Received: from buxtehude.debian.org (buxtehude.debian.org [206.12.19.147]) by cuda.sgi.com with ESMTP id bUKQLa6FEMKTmsaV (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 03 Jan 2016 20:57:18 -0800 (PST) X-Barracuda-Envelope-From: debbugs@buxtehude.debian.org X-Barracuda-Apparent-Source-IP: 206.12.19.147 Received: from debbugs by buxtehude.debian.org with local (Exim 4.84) (envelope-from ) id 1aFxCk-0002mj-2M; Mon, 04 Jan 2016 04:57:14 +0000 X-Loop: owner@bugs.debian.org Subject: Bug#766811: spurious library links Reply-To: Marco d'Itri , 766811@bugs.debian.org X-ASG-Orig-Subj: Bug#766811: spurious library links Resent-From: Marco d'Itri Resent-To: debian-bugs-dist@lists.debian.org Resent-CC: XFS Development Team X-Loop: owner@bugs.debian.org Resent-Date: Mon, 04 Jan 2016 04:57:11 +0000 Resent-Message-ID: X-Debian-PR-Message: followup 766811 X-Debian-PR-Package: xfslibs-dev X-Debian-PR-Keywords: X-Debian-PR-Source: xfsprogs Received: via spool by 766811-submit@bugs.debian.org id=B766811.14518831228992 (code B ref 766811); Mon, 04 Jan 2016 04:57:11 +0000 Received: (at 766811) by bugs.debian.org; 4 Jan 2016 04:52:02 +0000 Received: from attila.bofh.it ([85.94.204.146] ident=postfix) by buxtehude.debian.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84) (envelope-from ) id 1aFx7h-0002K8-Rz; Mon, 04 Jan 2016 04:52:02 +0000 Received: by attila.bofh.it (Postfix, from userid 10) id E8D542E4DA; Mon, 4 Jan 2016 05:51:59 +0100 (CET) Received: by bongo.bofh.it (Postfix, from userid 1000) id A0B1E2C0039; Mon, 4 Jan 2016 05:41:36 +0100 (CET) Date: Mon, 4 Jan 2016 05:41:36 +0100 From: Marco d'Itri To: 766811@bugs.debian.org Cc: 766811-submitter@bugs.debian.org Message-ID: <20160104044136.GD11738@bongo.bofh.it> References: <20141026004825.GA27863@bongo.bofh.it> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="a2FkP9tdjPU2nyhF" Content-Disposition: inline In-Reply-To: <20141026004825.GA27863@bongo.bofh.it> User-Agent: Mutt/1.5.24 (2015-08-30) X-CrossAssassin-Score: 2 X-Barracuda-Connect: buxtehude.debian.org[206.12.19.147] X-Barracuda-Start-Time: 1451883438 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25810 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header --a2FkP9tdjPU2nyhF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Oct 26, Marco d'Itri wrote: > These links do not appear to have any purpose and should be removed: >=20 > /lib/libhandle.a -> /usr/lib/libhandle.a > /lib/libhandle.la -> /usr/lib/libhandle.la > /usr/lib/libhandle.so -> /lib/libhandle.so >=20 > Also, policy forbids to thip the .la files at all for normal libraries. This is an obvious bug and I have not heard back from you in over one year, what are your plans about fixing this? Do you need help? --=20 ciao, Marco --a2FkP9tdjPU2nyhF Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGcBAABCAAGBQJWifgAAAoJECv1pcEixPek0JAL/ivJ8g5KklMSzitaMhhBqkw2 I2BJ5hb3PIyBxqRWqzwrdHyGUJX8rkyJz+bFxYcUi9a0+/CYx7KYkqjkHGPiXiA7 WWOQdCp8IPMdmR4QpY3KkamoGODvXJplSJvYeZd7wKWNfvznU+XsoREaXMm48vu4 Xvyi76vYXxN9Gt0fRJ12K1Tsw5wSV2bxZollaHSyShIDEW8aR955O77gYQmnc6xW RTeM79ZkRBEEPhOgFw3wu1pjORUVAIPnNTIwXnMGxOgLLnxuoN6wfkcVMuYhCBK+ 8vUEgo+YezaMFxVEU/5+ew0C55RobZoSNTVTcNwrTSuSc974Y+W+Nxvb8/U6buDi JEVLea/aIzugUT9NmQ/UwtcdDClX2QY5G5XaFP8oniLI4NCQBPM3WRoi4kpjtkbc FVtdn/kjOIKVwIe5xYzwhczfp6YP1mm6Cr8a8ZJdK+o/LhUi41H/O4PyR6e+spoI 2TV8F5j4guNpVCdTJzpzij1K7kzC3HcZQVDreCGwcA== =p1lD -----END PGP SIGNATURE----- --a2FkP9tdjPU2nyhF-- From dave@fromorbit.com Sun Jan 3 23:55:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E98057F37 for ; Sun, 3 Jan 2016 23:55:02 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id B843E304032 for ; Sun, 3 Jan 2016 21:55:02 -0800 (PST) X-ASG-Debug-ID: 1451886898-04cb6c303241b3e0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id GTeZtjwCCR1bXnBZ for ; Sun, 03 Jan 2016 21:54:58 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2C/DQAYCIpWPGu7LXleKAECgw9SbYZggXmgLwIKEgaBDJQhhw9NAQEBAQEBBwEBAQFBP4URIxiBAgMHiFvAEoYOigJohBkFlwaFQIltjR9EhRSIYoIuAQEIAgE6HIFxKjSDRoFKAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 16:24:57 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aFy6Z-0000tY-SW; Mon, 04 Jan 2016 16:54:55 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aFy6Z-000473-RU; Mon, 04 Jan 2016 16:54:55 +1100 From: Dave Chinner To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, ext4@vger.kernel.org Subject: [PATCH 0/3] fs: FS_IOC_FS[GS]ETXATTR promotion Date: Mon, 4 Jan 2016 16:54:49 +1100 X-ASG-Orig-Subj: [PATCH 0/3] fs: FS_IOC_FS[GS]ETXATTR promotion Message-Id: <1451886892-15548-1-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451886898 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25811 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Hi folks, As I mentioned on -fsdevel this morning, the XFS_IOC_FS[SG]SETXATTR API promotion patch for ext4 project quota really needs to be split up into two patches: one for the API promotion and the other for the ext4 implementation of the API. This short series has the isolated API promotion patch, as well as a followup cleanup patch in XFS to get rid of the XFS definitions from the kernel code completely, followed by the first conflicting patch I had against the ext4-all-in-one patch for a per-inode DAX enablement flag. I've pushed these patches out to the following branch: git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git xfs-setxattr-promotion And I'll update them and merge them into the for-next branch according to feedback and review. Cheers, Dave. From dave@fromorbit.com Sun Jan 3 23:55:04 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 29D0E7F37 for ; Sun, 3 Jan 2016 23:55:04 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 19C88304059 for ; Sun, 3 Jan 2016 21:55:03 -0800 (PST) X-ASG-Debug-ID: 1451886901-04cb6c303141b3e0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id vLKH841x3GCNaIFr for ; Sun, 03 Jan 2016 21:55:01 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2B/DQAYCIpWPGu7LXleKAECgw+BP4ZggXmgTQacPE0BAQEBAQEHAQEBAUE/hDUBBVYjEAgYMTkDBxQZiC7AEoYOjwMFh16PKI8th26FMUSFFIhigi4BglMqNIUQAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 16:24:57 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aFy6Z-0000tb-T2; Mon, 04 Jan 2016 16:54:55 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aFy6Z-000476-SI; Mon, 04 Jan 2016 16:54:55 +1100 From: Dave Chinner To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, ext4@vger.kernel.org Subject: [PATCH 1/3] fs: XFS_IOC_FS[SG]SETXATTR to FS_IOC_FS[SG]ETXATTR promotion Date: Mon, 4 Jan 2016 16:54:50 +1100 X-ASG-Orig-Subj: [PATCH 1/3] fs: XFS_IOC_FS[SG]SETXATTR to FS_IOC_FS[SG]ETXATTR promotion Message-Id: <1451886892-15548-2-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1451886892-15548-1-git-send-email-david@fromorbit.com> References: <1451886892-15548-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451886901 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25811 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner Hoist the ioctl definitions for the XFS_IOC_FS[SG]SETXATTR API from fs/xfs/libxfs/xfs_fs.h to include/uapi/linux/fs.h so that the ioctls can be used by all filesystems, not just XFS. This enables (initially) ext4 to use the ioctl to set project IDs on inodes. Based-on-patch-from: Li Xi Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_fs.h | 51 +++++++++++++++++-------------------------------- include/uapi/linux/fs.h | 32 +++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 33 deletions(-) diff --git a/fs/xfs/libxfs/xfs_fs.h b/fs/xfs/libxfs/xfs_fs.h index b2b73a9..dd29d0a 100644 --- a/fs/xfs/libxfs/xfs_fs.h +++ b/fs/xfs/libxfs/xfs_fs.h @@ -36,38 +36,23 @@ struct dioattr { #endif /* - * Structure for XFS_IOC_FSGETXATTR[A] and XFS_IOC_FSSETXATTR. + * Flags for the bs_xflags/fsx_xflags field in FS_IOC_FS[GS]ETXATTR[A] */ -#ifndef HAVE_FSXATTR -struct fsxattr { - __u32 fsx_xflags; /* xflags field value (get/set) */ - __u32 fsx_extsize; /* extsize field value (get/set)*/ - __u32 fsx_nextents; /* nextents field value (get) */ - __u32 fsx_projid; /* project identifier (get/set) */ - unsigned char fsx_pad[12]; -}; -#endif - -/* - * Flags for the bs_xflags/fsx_xflags field - * There should be a one-to-one correspondence between these flags and the - * XFS_DIFLAG_s. - */ -#define XFS_XFLAG_REALTIME 0x00000001 /* data in realtime volume */ -#define XFS_XFLAG_PREALLOC 0x00000002 /* preallocated file extents */ -#define XFS_XFLAG_IMMUTABLE 0x00000008 /* file cannot be modified */ -#define XFS_XFLAG_APPEND 0x00000010 /* all writes append */ -#define XFS_XFLAG_SYNC 0x00000020 /* all writes synchronous */ -#define XFS_XFLAG_NOATIME 0x00000040 /* do not update access time */ -#define XFS_XFLAG_NODUMP 0x00000080 /* do not include in backups */ -#define XFS_XFLAG_RTINHERIT 0x00000100 /* create with rt bit set */ -#define XFS_XFLAG_PROJINHERIT 0x00000200 /* create with parents projid */ -#define XFS_XFLAG_NOSYMLINKS 0x00000400 /* disallow symlink creation */ -#define XFS_XFLAG_EXTSIZE 0x00000800 /* extent size allocator hint */ -#define XFS_XFLAG_EXTSZINHERIT 0x00001000 /* inherit inode extent size */ -#define XFS_XFLAG_NODEFRAG 0x00002000 /* do not defragment */ -#define XFS_XFLAG_FILESTREAM 0x00004000 /* use filestream allocator */ -#define XFS_XFLAG_HASATTR 0x80000000 /* no DIFLAG for this */ +#define XFS_XFLAG_REALTIME FS_XFLAG_REALTIME +#define XFS_XFLAG_PREALLOC FS_XFLAG_PREALLOC +#define XFS_XFLAG_IMMUTABLE FS_XFLAG_IMMUTABLE +#define XFS_XFLAG_APPEND FS_XFLAG_APPEND +#define XFS_XFLAG_SYNC FS_XFLAG_SYNC +#define XFS_XFLAG_NOATIME FS_XFLAG_NOATIME +#define XFS_XFLAG_NODUMP FS_XFLAG_NODUMP +#define XFS_XFLAG_RTINHERIT FS_XFLAG_RTINHERIT +#define XFS_XFLAG_PROJINHERIT FS_XFLAG_PROJINHERIT +#define XFS_XFLAG_NOSYMLINKS FS_XFLAG_NOSYMLINKS +#define XFS_XFLAG_EXTSIZE FS_XFLAG_EXTSIZE +#define XFS_XFLAG_EXTSZINHERIT FS_XFLAG_EXTSZINHERIT +#define XFS_XFLAG_NODEFRAG FS_XFLAG_NODEFRAG +#define XFS_XFLAG_FILESTREAM FS_XFLAG_FILESTREAM +#define XFS_XFLAG_HASATTR FS_XFLAG_HASATTR /* * Structure for XFS_IOC_GETBMAP. @@ -514,8 +499,8 @@ typedef struct xfs_swapext #define XFS_IOC_ALLOCSP _IOW ('X', 10, struct xfs_flock64) #define XFS_IOC_FREESP _IOW ('X', 11, struct xfs_flock64) #define XFS_IOC_DIOINFO _IOR ('X', 30, struct dioattr) -#define XFS_IOC_FSGETXATTR _IOR ('X', 31, struct fsxattr) -#define XFS_IOC_FSSETXATTR _IOW ('X', 32, struct fsxattr) +#define XFS_IOC_FSGETXATTR FS_IOC_FSGETXATTR +#define XFS_IOC_FSSETXATTR FS_IOC_FSSETXATTR #define XFS_IOC_ALLOCSP64 _IOW ('X', 36, struct xfs_flock64) #define XFS_IOC_FREESP64 _IOW ('X', 37, struct xfs_flock64) #define XFS_IOC_GETBMAP _IOWR('X', 38, struct getbmap) diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index f15d980..df175dd 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -110,6 +110,36 @@ struct inodes_stat_t { #define MS_MGC_VAL 0xC0ED0000 #define MS_MGC_MSK 0xffff0000 +/* + * Structure for FS_IOC_FSGETXATTR[A] and FS_IOC_FSSETXATTR. + */ +struct fsxattr { + __u32 fsx_xflags; /* xflags field value (get/set) */ + __u32 fsx_extsize; /* extsize field value (get/set)*/ + __u32 fsx_nextents; /* nextents field value (get) */ + __u32 fsx_projid; /* project identifier (get/set) */ + unsigned char fsx_pad[12]; +}; + +/* + * Flags for the fsx_xflags field + */ +#define FS_XFLAG_REALTIME 0x00000001 /* data in realtime volume */ +#define FS_XFLAG_PREALLOC 0x00000002 /* preallocated file extents */ +#define FS_XFLAG_IMMUTABLE 0x00000008 /* file cannot be modified */ +#define FS_XFLAG_APPEND 0x00000010 /* all writes append */ +#define FS_XFLAG_SYNC 0x00000020 /* all writes synchronous */ +#define FS_XFLAG_NOATIME 0x00000040 /* do not update access time */ +#define FS_XFLAG_NODUMP 0x00000080 /* do not include in backups */ +#define FS_XFLAG_RTINHERIT 0x00000100 /* create with rt bit set */ +#define FS_XFLAG_PROJINHERIT 0x00000200 /* create with parents projid */ +#define FS_XFLAG_NOSYMLINKS 0x00000400 /* disallow symlink creation */ +#define FS_XFLAG_EXTSIZE 0x00000800 /* extent size allocator hint */ +#define FS_XFLAG_EXTSZINHERIT 0x00001000 /* inherit inode extent size */ +#define FS_XFLAG_NODEFRAG 0x00002000 /* do not defragment */ +#define FS_XFLAG_FILESTREAM 0x00004000 /* use filestream allocator */ +#define FS_XFLAG_HASATTR 0x80000000 /* no DIFLAG for this */ + /* the read-only stuff doesn't really belong here, but any other place is probably as bad and I don't want to create yet another include file. */ @@ -169,6 +199,8 @@ struct inodes_stat_t { #define FS_IOC32_SETFLAGS _IOW('f', 2, int) #define FS_IOC32_GETVERSION _IOR('v', 1, int) #define FS_IOC32_SETVERSION _IOW('v', 2, int) +#define FS_IOC_FSGETXATTR _IOR ('X', 31, struct fsxattr) +#define FS_IOC_FSSETXATTR _IOW ('X', 32, struct fsxattr) /* * Inode flags (FS_IOC_GETFLAGS / FS_IOC_SETFLAGS) -- 2.5.0 From dave@fromorbit.com Sun Jan 3 23:55:06 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id B0C877F54 for ; Sun, 3 Jan 2016 23:55:06 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id A1BF38F8035 for ; Sun, 3 Jan 2016 21:55:03 -0800 (PST) X-ASG-Debug-ID: 1451886898-04cb6c303241b3e0002-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id RAhxPNMP4VwXHou2 for ; Sun, 03 Jan 2016 21:55:01 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CCDQAYCIpWPGu7LXleKAECgw+BP4ZggXmgTQabIoEaTQEBAQEBAQcBAQEBQT+ENQEFJy8jEAgYMTkDBxQZiC7AEoYOikSEPwWXBo8th26FMY46gi4BQQUcgXEqNIUQAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 16:24:57 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aFy6Z-0000td-U2; Mon, 04 Jan 2016 16:54:55 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aFy6Z-00047G-TH; Mon, 04 Jan 2016 16:54:55 +1100 From: Dave Chinner To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, ext4@vger.kernel.org Subject: [PATCH 3/3] xfs: introduce per-inode DAX enablement Date: Mon, 4 Jan 2016 16:54:52 +1100 X-ASG-Orig-Subj: [PATCH 3/3] xfs: introduce per-inode DAX enablement Message-Id: <1451886892-15548-4-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1451886892-15548-1-git-send-email-david@fromorbit.com> References: <1451886892-15548-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451886900 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25811 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner Rather than just being able to turn DAX on and off via a mount option, some applications may only want to enable DAX for certain performance critical files in a filesystem. This patch introduces a new inode flag to enable DAX in the v3 inode di_flags2 field. It adds support for setting and clearing flags in the di_flags2 field via the XFS_IOC_FSSETXATTR ioctl, and sets the S_DAX inode flag appropriately when it is seen. When this flag is set on a directory, it acts as an "inherit flag". That is, inodes created in the directory will automatically inherit the on-disk inode DAX flag, enabling administrators to set up directory heirarchies that automatically use DAX. Setting this flag on an empty root directory will make the entire filesystem use DAX by default. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_format.h | 9 +++++++++ fs/xfs/xfs_inode.c | 32 +++++++++++++++++++++++--------- fs/xfs/xfs_ioctl.c | 18 +++++++++++++++++- fs/xfs/xfs_iops.c | 4 ++-- include/uapi/linux/fs.h | 1 + 5 files changed, 52 insertions(+), 12 deletions(-) diff --git a/fs/xfs/libxfs/xfs_format.h b/fs/xfs/libxfs/xfs_format.h index f28eeab..b4ae7ce 100644 --- a/fs/xfs/libxfs/xfs_format.h +++ b/fs/xfs/libxfs/xfs_format.h @@ -1024,6 +1024,15 @@ static inline void xfs_dinode_put_rdev(struct xfs_dinode *dip, xfs_dev_t rdev) XFS_DIFLAG_EXTSZINHERIT | XFS_DIFLAG_NODEFRAG | XFS_DIFLAG_FILESTREAM) /* + * Values for di_flags2 These start by being exposed to userspace in the upper + * 16 bits of the XFS_XFLAG_s range. + */ +#define XFS_DIFLAG2_DAX_BIT 0 /* use DAX for this inode */ +#define XFS_DIFLAG2_DAX (1 << XFS_DIFLAG2_DAX_BIT) + +#define XFS_DIFLAG2_ANY (XFS_DIFLAG2_DAX) + +/* * Inode number format: * low inopblog bits - offset in block * next agblklog bits - block number in ag diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index ca9ca5a..8929908 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -610,7 +610,9 @@ __xfs_iflock( STATIC uint _xfs_dic2xflags( - __uint16_t di_flags) + __uint16_t di_flags, + uint64_t di_flags2, + bool has_attr) { uint flags = 0; @@ -645,25 +647,32 @@ _xfs_dic2xflags( flags |= FS_XFLAG_FILESTREAM; } + if (di_flags2 & XFS_DIFLAG2_ANY) { + if (di_flags2 & XFS_DIFLAG2_DAX) + flags |= FS_XFLAG_DAX; + } + + if (has_attr) + flags |= FS_XFLAG_HASATTR; + return flags; } uint xfs_ip2xflags( - xfs_inode_t *ip) + struct xfs_inode *ip) { - xfs_icdinode_t *dic = &ip->i_d; + struct xfs_icdinode *dic = &ip->i_d; - return _xfs_dic2xflags(dic->di_flags) | - (XFS_IFORK_Q(ip) ? FS_XFLAG_HASATTR : 0); + return _xfs_dic2xflags(dic->di_flags, dic->di_flags2, XFS_IFORK_Q(ip)); } uint xfs_dic2xflags( - xfs_dinode_t *dip) + struct xfs_dinode *dip) { - return _xfs_dic2xflags(be16_to_cpu(dip->di_flags)) | - (XFS_DFORK_Q(dip) ? FS_XFLAG_HASATTR : 0); + return _xfs_dic2xflags(be16_to_cpu(dip->di_flags), + be64_to_cpu(dip->di_flags2), XFS_DFORK_Q(dip)); } /* @@ -862,7 +871,8 @@ xfs_ialloc( case S_IFREG: case S_IFDIR: if (pip && (pip->i_d.di_flags & XFS_DIFLAG_ANY)) { - uint di_flags = 0; + uint64_t di_flags2 = 0; + uint di_flags = 0; if (S_ISDIR(mode)) { if (pip->i_d.di_flags & XFS_DIFLAG_RTINHERIT) @@ -898,7 +908,11 @@ xfs_ialloc( di_flags |= XFS_DIFLAG_NODEFRAG; if (pip->i_d.di_flags & XFS_DIFLAG_FILESTREAM) di_flags |= XFS_DIFLAG_FILESTREAM; + if (pip->i_d.di_flags2 & XFS_DIFLAG2_DAX) + di_flags2 |= XFS_DIFLAG2_DAX; + ip->i_d.di_flags |= di_flags; + ip->i_d.di_flags2 |= di_flags2; } /* FALLTHROUGH */ case S_IFLNK: diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index 94b35eb3..478d04e 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -945,6 +945,7 @@ xfs_set_diflags( unsigned int xflags) { unsigned int di_flags; + uint64_t di_flags2; /* can't set PREALLOC this way, just preserve it */ di_flags = (ip->i_d.di_flags & XFS_DIFLAG_PREALLOC); @@ -977,8 +978,18 @@ xfs_set_diflags( if (xflags & FS_XFLAG_EXTSIZE) di_flags |= XFS_DIFLAG_EXTSIZE; } - ip->i_d.di_flags = di_flags; + + /* diflags2 only valid for v3 inodes. */ + if (ip->i_d.di_version < 3) + return; + + di_flags2 = 0; + if (xflags & FS_XFLAG_DAX) + di_flags2 |= XFS_DIFLAG2_DAX; + + ip->i_d.di_flags2 = di_flags2; + } STATIC void @@ -1004,6 +1015,11 @@ xfs_diflags_to_linux( inode->i_flags |= S_NOATIME; else inode->i_flags &= ~S_NOATIME; + if (xflags & FS_XFLAG_DAX) + inode->i_flags |= S_DAX; + else + inode->i_flags &= ~S_DAX; + } static int diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c index 245268a..a1b8af1 100644 --- a/fs/xfs/xfs_iops.c +++ b/fs/xfs/xfs_iops.c @@ -1201,8 +1201,8 @@ xfs_diflags_to_iflags( inode->i_flags |= S_SYNC; if (flags & XFS_DIFLAG_NOATIME) inode->i_flags |= S_NOATIME; - /* XXX: Also needs an on-disk per inode flag! */ - if (ip->i_mount->m_flags & XFS_MOUNT_DAX) + if (ip->i_mount->m_flags & XFS_MOUNT_DAX || + ip->i_d.di_flags2 & XFS_DIFLAG2_DAX) inode->i_flags |= S_DAX; } diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index df175dd..4cad4c8 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -138,6 +138,7 @@ struct fsxattr { #define FS_XFLAG_EXTSZINHERIT 0x00001000 /* inherit inode extent size */ #define FS_XFLAG_NODEFRAG 0x00002000 /* do not defragment */ #define FS_XFLAG_FILESTREAM 0x00004000 /* use filestream allocator */ +#define FS_XFLAG_DAX 0x00008000 /* use DAX for IO */ #define FS_XFLAG_HASATTR 0x80000000 /* no DIFLAG for this */ /* the read-only stuff doesn't really belong here, but any other place is -- 2.5.0 From dave@fromorbit.com Sun Jan 3 23:55:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D7D177F3F for ; Sun, 3 Jan 2016 23:55:09 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 4E1D4AC002 for ; Sun, 3 Jan 2016 21:55:06 -0800 (PST) X-ASG-Debug-ID: 1451886898-04cb6c303241b3e0003-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id tUBGl2Gxqcps4pVJ for ; Sun, 03 Jan 2016 21:55:03 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CCDQAYCIpWPGu7LXleKAECgw+BP4ZggXmgTQabIoEaTQEBAQEBAQcBAQEBQT+ENQEFJy8jEAgYMTkDBxQZiC7AEoYOikSEPwWHXo8onExEjXaCLgFGHIFxKjSDRyWBJAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 16:24:57 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aFy6Z-0000tc-TY; Mon, 04 Jan 2016 16:54:55 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aFy6Z-00047B-Sl; Mon, 04 Jan 2016 16:54:55 +1100 From: Dave Chinner To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, ext4@vger.kernel.org Subject: [PATCH 2/3] xfs: use FS_XFLAG definitions directly Date: Mon, 4 Jan 2016 16:54:51 +1100 X-ASG-Orig-Subj: [PATCH 2/3] xfs: use FS_XFLAG definitions directly Message-Id: <1451886892-15548-3-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1451886892-15548-1-git-send-email-david@fromorbit.com> References: <1451886892-15548-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451886902 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25811 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner Now that the ioctls have been hoisted up to the VFS level, use the VFs definitions directly and remove the XFS specific definitions completely. Userspace is going to have to handle the change of this interface separately, so removing the definitions from xfs_fs.h is not an issue here at all. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_format.h | 2 -- fs/xfs/libxfs/xfs_fs.h | 19 ------------ fs/xfs/xfs_inode.c | 32 ++++++++++---------- fs/xfs/xfs_ioctl.c | 74 +++++++++++++++++++++++----------------------- 4 files changed, 53 insertions(+), 74 deletions(-) diff --git a/fs/xfs/libxfs/xfs_format.h b/fs/xfs/libxfs/xfs_format.h index 8774498..f28eeab 100644 --- a/fs/xfs/libxfs/xfs_format.h +++ b/fs/xfs/libxfs/xfs_format.h @@ -984,8 +984,6 @@ static inline void xfs_dinode_put_rdev(struct xfs_dinode *dip, xfs_dev_t rdev) /* * Values for di_flags - * There should be a one-to-one correspondence between these flags and the - * XFS_XFLAG_s. */ #define XFS_DIFLAG_REALTIME_BIT 0 /* file's blocks come from rt area */ #define XFS_DIFLAG_PREALLOC_BIT 1 /* file space has been preallocated */ diff --git a/fs/xfs/libxfs/xfs_fs.h b/fs/xfs/libxfs/xfs_fs.h index dd29d0a..fffe3d0 100644 --- a/fs/xfs/libxfs/xfs_fs.h +++ b/fs/xfs/libxfs/xfs_fs.h @@ -36,25 +36,6 @@ struct dioattr { #endif /* - * Flags for the bs_xflags/fsx_xflags field in FS_IOC_FS[GS]ETXATTR[A] - */ -#define XFS_XFLAG_REALTIME FS_XFLAG_REALTIME -#define XFS_XFLAG_PREALLOC FS_XFLAG_PREALLOC -#define XFS_XFLAG_IMMUTABLE FS_XFLAG_IMMUTABLE -#define XFS_XFLAG_APPEND FS_XFLAG_APPEND -#define XFS_XFLAG_SYNC FS_XFLAG_SYNC -#define XFS_XFLAG_NOATIME FS_XFLAG_NOATIME -#define XFS_XFLAG_NODUMP FS_XFLAG_NODUMP -#define XFS_XFLAG_RTINHERIT FS_XFLAG_RTINHERIT -#define XFS_XFLAG_PROJINHERIT FS_XFLAG_PROJINHERIT -#define XFS_XFLAG_NOSYMLINKS FS_XFLAG_NOSYMLINKS -#define XFS_XFLAG_EXTSIZE FS_XFLAG_EXTSIZE -#define XFS_XFLAG_EXTSZINHERIT FS_XFLAG_EXTSZINHERIT -#define XFS_XFLAG_NODEFRAG FS_XFLAG_NODEFRAG -#define XFS_XFLAG_FILESTREAM FS_XFLAG_FILESTREAM -#define XFS_XFLAG_HASATTR FS_XFLAG_HASATTR - -/* * Structure for XFS_IOC_GETBMAP. * On input, fill in bmv_offset and bmv_length of the first structure * to indicate the area of interest in the file, and bmv_entries with diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index 8ee3939..ca9ca5a 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -616,33 +616,33 @@ _xfs_dic2xflags( if (di_flags & XFS_DIFLAG_ANY) { if (di_flags & XFS_DIFLAG_REALTIME) - flags |= XFS_XFLAG_REALTIME; + flags |= FS_XFLAG_REALTIME; if (di_flags & XFS_DIFLAG_PREALLOC) - flags |= XFS_XFLAG_PREALLOC; + flags |= FS_XFLAG_PREALLOC; if (di_flags & XFS_DIFLAG_IMMUTABLE) - flags |= XFS_XFLAG_IMMUTABLE; + flags |= FS_XFLAG_IMMUTABLE; if (di_flags & XFS_DIFLAG_APPEND) - flags |= XFS_XFLAG_APPEND; + flags |= FS_XFLAG_APPEND; if (di_flags & XFS_DIFLAG_SYNC) - flags |= XFS_XFLAG_SYNC; + flags |= FS_XFLAG_SYNC; if (di_flags & XFS_DIFLAG_NOATIME) - flags |= XFS_XFLAG_NOATIME; + flags |= FS_XFLAG_NOATIME; if (di_flags & XFS_DIFLAG_NODUMP) - flags |= XFS_XFLAG_NODUMP; + flags |= FS_XFLAG_NODUMP; if (di_flags & XFS_DIFLAG_RTINHERIT) - flags |= XFS_XFLAG_RTINHERIT; + flags |= FS_XFLAG_RTINHERIT; if (di_flags & XFS_DIFLAG_PROJINHERIT) - flags |= XFS_XFLAG_PROJINHERIT; + flags |= FS_XFLAG_PROJINHERIT; if (di_flags & XFS_DIFLAG_NOSYMLINKS) - flags |= XFS_XFLAG_NOSYMLINKS; + flags |= FS_XFLAG_NOSYMLINKS; if (di_flags & XFS_DIFLAG_EXTSIZE) - flags |= XFS_XFLAG_EXTSIZE; + flags |= FS_XFLAG_EXTSIZE; if (di_flags & XFS_DIFLAG_EXTSZINHERIT) - flags |= XFS_XFLAG_EXTSZINHERIT; + flags |= FS_XFLAG_EXTSZINHERIT; if (di_flags & XFS_DIFLAG_NODEFRAG) - flags |= XFS_XFLAG_NODEFRAG; + flags |= FS_XFLAG_NODEFRAG; if (di_flags & XFS_DIFLAG_FILESTREAM) - flags |= XFS_XFLAG_FILESTREAM; + flags |= FS_XFLAG_FILESTREAM; } return flags; @@ -655,7 +655,7 @@ xfs_ip2xflags( xfs_icdinode_t *dic = &ip->i_d; return _xfs_dic2xflags(dic->di_flags) | - (XFS_IFORK_Q(ip) ? XFS_XFLAG_HASATTR : 0); + (XFS_IFORK_Q(ip) ? FS_XFLAG_HASATTR : 0); } uint @@ -663,7 +663,7 @@ xfs_dic2xflags( xfs_dinode_t *dip) { return _xfs_dic2xflags(be16_to_cpu(dip->di_flags)) | - (XFS_DFORK_Q(dip) ? XFS_XFLAG_HASATTR : 0); + (XFS_DFORK_Q(dip) ? FS_XFLAG_HASATTR : 0); } /* diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index d42738d..94b35eb3 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -859,25 +859,25 @@ xfs_merge_ioc_xflags( unsigned int xflags = start; if (flags & FS_IMMUTABLE_FL) - xflags |= XFS_XFLAG_IMMUTABLE; + xflags |= FS_XFLAG_IMMUTABLE; else - xflags &= ~XFS_XFLAG_IMMUTABLE; + xflags &= ~FS_XFLAG_IMMUTABLE; if (flags & FS_APPEND_FL) - xflags |= XFS_XFLAG_APPEND; + xflags |= FS_XFLAG_APPEND; else - xflags &= ~XFS_XFLAG_APPEND; + xflags &= ~FS_XFLAG_APPEND; if (flags & FS_SYNC_FL) - xflags |= XFS_XFLAG_SYNC; + xflags |= FS_XFLAG_SYNC; else - xflags &= ~XFS_XFLAG_SYNC; + xflags &= ~FS_XFLAG_SYNC; if (flags & FS_NOATIME_FL) - xflags |= XFS_XFLAG_NOATIME; + xflags |= FS_XFLAG_NOATIME; else - xflags &= ~XFS_XFLAG_NOATIME; + xflags &= ~FS_XFLAG_NOATIME; if (flags & FS_NODUMP_FL) - xflags |= XFS_XFLAG_NODUMP; + xflags |= FS_XFLAG_NODUMP; else - xflags &= ~XFS_XFLAG_NODUMP; + xflags &= ~FS_XFLAG_NODUMP; return xflags; } @@ -948,33 +948,33 @@ xfs_set_diflags( /* can't set PREALLOC this way, just preserve it */ di_flags = (ip->i_d.di_flags & XFS_DIFLAG_PREALLOC); - if (xflags & XFS_XFLAG_IMMUTABLE) + if (xflags & FS_XFLAG_IMMUTABLE) di_flags |= XFS_DIFLAG_IMMUTABLE; - if (xflags & XFS_XFLAG_APPEND) + if (xflags & FS_XFLAG_APPEND) di_flags |= XFS_DIFLAG_APPEND; - if (xflags & XFS_XFLAG_SYNC) + if (xflags & FS_XFLAG_SYNC) di_flags |= XFS_DIFLAG_SYNC; - if (xflags & XFS_XFLAG_NOATIME) + if (xflags & FS_XFLAG_NOATIME) di_flags |= XFS_DIFLAG_NOATIME; - if (xflags & XFS_XFLAG_NODUMP) + if (xflags & FS_XFLAG_NODUMP) di_flags |= XFS_DIFLAG_NODUMP; - if (xflags & XFS_XFLAG_NODEFRAG) + if (xflags & FS_XFLAG_NODEFRAG) di_flags |= XFS_DIFLAG_NODEFRAG; - if (xflags & XFS_XFLAG_FILESTREAM) + if (xflags & FS_XFLAG_FILESTREAM) di_flags |= XFS_DIFLAG_FILESTREAM; if (S_ISDIR(ip->i_d.di_mode)) { - if (xflags & XFS_XFLAG_RTINHERIT) + if (xflags & FS_XFLAG_RTINHERIT) di_flags |= XFS_DIFLAG_RTINHERIT; - if (xflags & XFS_XFLAG_NOSYMLINKS) + if (xflags & FS_XFLAG_NOSYMLINKS) di_flags |= XFS_DIFLAG_NOSYMLINKS; - if (xflags & XFS_XFLAG_EXTSZINHERIT) + if (xflags & FS_XFLAG_EXTSZINHERIT) di_flags |= XFS_DIFLAG_EXTSZINHERIT; - if (xflags & XFS_XFLAG_PROJINHERIT) + if (xflags & FS_XFLAG_PROJINHERIT) di_flags |= XFS_DIFLAG_PROJINHERIT; } else if (S_ISREG(ip->i_d.di_mode)) { - if (xflags & XFS_XFLAG_REALTIME) + if (xflags & FS_XFLAG_REALTIME) di_flags |= XFS_DIFLAG_REALTIME; - if (xflags & XFS_XFLAG_EXTSIZE) + if (xflags & FS_XFLAG_EXTSIZE) di_flags |= XFS_DIFLAG_EXTSIZE; } @@ -988,19 +988,19 @@ xfs_diflags_to_linux( struct inode *inode = VFS_I(ip); unsigned int xflags = xfs_ip2xflags(ip); - if (xflags & XFS_XFLAG_IMMUTABLE) + if (xflags & FS_XFLAG_IMMUTABLE) inode->i_flags |= S_IMMUTABLE; else inode->i_flags &= ~S_IMMUTABLE; - if (xflags & XFS_XFLAG_APPEND) + if (xflags & FS_XFLAG_APPEND) inode->i_flags |= S_APPEND; else inode->i_flags &= ~S_APPEND; - if (xflags & XFS_XFLAG_SYNC) + if (xflags & FS_XFLAG_SYNC) inode->i_flags |= S_SYNC; else inode->i_flags &= ~S_SYNC; - if (xflags & XFS_XFLAG_NOATIME) + if (xflags & FS_XFLAG_NOATIME) inode->i_flags |= S_NOATIME; else inode->i_flags &= ~S_NOATIME; @@ -1016,11 +1016,11 @@ xfs_ioctl_setattr_xflags( /* Can't change realtime flag if any extents are allocated. */ if ((ip->i_d.di_nextents || ip->i_delayed_blks) && - XFS_IS_REALTIME_INODE(ip) != (fa->fsx_xflags & XFS_XFLAG_REALTIME)) + XFS_IS_REALTIME_INODE(ip) != (fa->fsx_xflags & FS_XFLAG_REALTIME)) return -EINVAL; /* If realtime flag is set then must have realtime device */ - if (fa->fsx_xflags & XFS_XFLAG_REALTIME) { + if (fa->fsx_xflags & FS_XFLAG_REALTIME) { if (mp->m_sb.sb_rblocks == 0 || mp->m_sb.sb_rextsize == 0 || (ip->i_d.di_extsize % mp->m_sb.sb_rextsize)) return -EINVAL; @@ -1031,7 +1031,7 @@ xfs_ioctl_setattr_xflags( * we have appropriate permission. */ if (((ip->i_d.di_flags & (XFS_DIFLAG_IMMUTABLE | XFS_DIFLAG_APPEND)) || - (fa->fsx_xflags & (XFS_XFLAG_IMMUTABLE | XFS_XFLAG_APPEND))) && + (fa->fsx_xflags & (FS_XFLAG_IMMUTABLE | FS_XFLAG_APPEND))) && !capable(CAP_LINUX_IMMUTABLE)) return -EPERM; @@ -1095,8 +1095,8 @@ out_cancel: * extent size hint validation is somewhat cumbersome. Rules are: * * 1. extent size hint is only valid for directories and regular files - * 2. XFS_XFLAG_EXTSIZE is only valid for regular files - * 3. XFS_XFLAG_EXTSZINHERIT is only valid for directories. + * 2. FS_XFLAG_EXTSIZE is only valid for regular files + * 3. FS_XFLAG_EXTSZINHERIT is only valid for directories. * 4. can only be changed on regular files if no extents are allocated * 5. can be changed on directories at any time * 6. extsize hint of 0 turns off hints, clears inode flags. @@ -1112,10 +1112,10 @@ xfs_ioctl_setattr_check_extsize( { struct xfs_mount *mp = ip->i_mount; - if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) && !S_ISREG(ip->i_d.di_mode)) + if ((fa->fsx_xflags & FS_XFLAG_EXTSIZE) && !S_ISREG(ip->i_d.di_mode)) return -EINVAL; - if ((fa->fsx_xflags & XFS_XFLAG_EXTSZINHERIT) && + if ((fa->fsx_xflags & FS_XFLAG_EXTSZINHERIT) && !S_ISDIR(ip->i_d.di_mode)) return -EINVAL; @@ -1132,7 +1132,7 @@ xfs_ioctl_setattr_check_extsize( return -EINVAL; if (XFS_IS_REALTIME_INODE(ip) || - (fa->fsx_xflags & XFS_XFLAG_REALTIME)) { + (fa->fsx_xflags & FS_XFLAG_REALTIME)) { size = mp->m_sb.sb_rextsize << mp->m_sb.sb_blocklog; } else { size = mp->m_sb.sb_blocksize; @@ -1143,7 +1143,7 @@ xfs_ioctl_setattr_check_extsize( if (fa->fsx_extsize % size) return -EINVAL; } else - fa->fsx_xflags &= ~(XFS_XFLAG_EXTSIZE | XFS_XFLAG_EXTSZINHERIT); + fa->fsx_xflags &= ~(FS_XFLAG_EXTSIZE | FS_XFLAG_EXTSZINHERIT); return 0; } @@ -1168,7 +1168,7 @@ xfs_ioctl_setattr_check_projid( if (xfs_get_projid(ip) != fa->fsx_projid) return -EINVAL; - if ((fa->fsx_xflags & XFS_XFLAG_PROJINHERIT) != + if ((fa->fsx_xflags & FS_XFLAG_PROJINHERIT) != (ip->i_d.di_flags & XFS_DIFLAG_PROJINHERIT)) return -EINVAL; -- 2.5.0 From david@fromorbit.com Mon Jan 4 00:22:37 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id D699C7F37 for ; Mon, 4 Jan 2016 00:22:37 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id C6347304032 for ; Sun, 3 Jan 2016 22:22:34 -0800 (PST) X-ASG-Debug-ID: 1451888552-04cb6c303241ba00001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id RAZnCcMAw4YDrPAG for ; Sun, 03 Jan 2016 22:22:32 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CMDQBKD4pWPGu7LXleKAECgw+BP4ZggXmgTQaLVYk+hgkEAoEZTQEBAQEBAQcBAQEBQT+EdRwjGCQ0BSUDBw4fiC7AChmFdYpqgn6BGwWTCoN8jUiBZY0fhViIYoUCKjSFEAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 16:52:20 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aFyX5-0000xF-Ep; Mon, 04 Jan 2016 17:22:19 +1100 Date: Mon, 4 Jan 2016 17:22:19 +1100 From: Dave Chinner To: xfs@oss.sgi.com Cc: ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org Subject: lazytime implementation questions Message-ID: <20160104062219.GB19802@dastard> X-ASG-Orig-Subj: lazytime implementation questions MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451888552 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25811 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Hi folks, I've been looking at implementing the lazytime mount option for XFS, and I'm struggling to work out what it is supposed to mean. AFAICT, on ext4, lazytime means that pure timestamp updates are not journalled and they are only ever written back when the inode is otherwise dirtied and written, or they are timestamp dirty for 24 hours which triggers writeback. This poses a couple of problems for XFS: 1. we log every timestamp change, so there is no mechanism for delayed/deferred update. 2. we track dirty metadata in the journal, not via the VFS dirty inode lists, so all the infrastructure written for ext4 to do periodic flushing is useless to us. These are solvable problems, but what I'm not sure about is exactly what the intended semantics of lazytime durability are. That is, exactly what guaranteed are we giving userspace about timestamp updates when lazytime is used? The guarantees we have to give will greatly influence the XFS implementation, so I really need to nail down what we are expected to provide userspace. Can we: a) just ignore all durability concerns? b) if not, do we only need to care about the 24 hour writeback and unmount? c) if not, are fsync/sync/syncfs/freeze/unmount supposed to provide durability of all metadata changes? d) do we have to care about ordering - if we fsync one inode with 1 hour old timestamps, do we also need to guarantee that all the inodes with older dirty timestamps also get made durable? I really want to completely ignore all ordering and forced durability requirements for lazytime (i.e. implement only periodic, optimistic and freeze/unmount writeback), but I haven't found any documentation of what durability lazytime is supposed to provide and can really only guess what was intended from the ext4 implementation.... Clarity would be appreciated. Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Mon Jan 4 00:29:14 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 349297F37 for ; Mon, 4 Jan 2016 00:29:14 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 15149304032 for ; Sun, 3 Jan 2016 22:29:13 -0800 (PST) X-ASG-Debug-ID: 1451888948-04cb6c302f41bb40001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id tqEDSj6IPAk95EjE for ; Sun, 03 Jan 2016 22:29:09 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DQDQBNEIpWPGu7LXleKAECgw8iMG2GYIF5oC8CChIGgQyKSYk+GAKFbwSBG00BAQEBAQEHAQEBAUE/hGIvOyQ0BSUDBy2ILp5noRoJGYV1igIJgnUMQRuBGwWNOYVRg3yFQIgIgWWHboUvAo46gjqCSCo0g0aBSgEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Jan 2016 16:59:07 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aFydf-0000z8-DB for xfs@oss.sgi.com; Mon, 04 Jan 2016 17:29:07 +1100 Date: Mon, 4 Jan 2016 17:29:07 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [ANNOUNCE] xfs: for-next branch updated to 887a9f2 Message-ID: <20160104062907.GC19802@dastard> X-ASG-Orig-Subj: [ANNOUNCE] xfs: for-next branch updated to 887a9f2 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="s2ZSL+KKDSLx8OML" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451888948 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25811 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- --s2ZSL+KKDSLx8OML Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi folks, The for-next branch of the xfs kernel repository at git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git has just been updated. The new head of the for-next branch is commit: 887a9f2 Merge branch 'xfs-dax-fixes-for-4.5' into for-next New Commits: Alexander Kuleshov (1): [211fe1a] xfs: make xfs_buf_ioend_async() static Brian Foster (8): [a70f9fe] xfs: detect and handle invalid iclog size set by mkfs [9d94901] xfs: refactor log record unpack and data processing [b94fb2d] xfs: refactor and open code log record crc check [d7f3769] xfs: return start block of first bad log record during reco= very [6528250] xfs: support a crc verification only log record pass [eed6b46] xfs: refactor log record start detection into a new helper [841af8e] xfs: detect and trim torn writes during log recovery [35de070] xfs: debug mode log record crc error injection Darrick J. Wong (5): [2e9101d] libxfs: make xfs_alloc_fix_freelist non-static [9b434a3] xfs: fix log ticket type printing [6d3eb1e] libxfs: use a convenience variable instead of open-coding t= he fork [96f859d] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is = correct [c5ab131] libxfs: refactor short btree block verification Dave Chinner (4): [289542c] Merge branch 'xfs-misc-fixes-for-4.5' into for-next [3b0fe47] xfs: Don't use reserved blocks for data blocks with DAX [a6d7636] xfs: fix recursive splice read locking with DAX [887a9f2] Merge branch 'xfs-dax-fixes-for-4.5' into for-next Eric Sandeen (2): [233135b] xfs: print name of verifier if it fails [f1f96c4] xfs: get mp from bma->ip in xfs_bmap code Jia He (1): [1d4292b] libxfs: Optimize the loop for xfs_bitmap_empty Markus Elfring (1): [a841b64] XFS: Use a signed return type for suffix_kstrtoint() Masatake YAMATO (1): [ffc671f] xfs: send warning of project quota to userspace via netlink Code Diffstat: fs/xfs/libxfs/xfs_alloc.c | 4 +- fs/xfs/libxfs/xfs_alloc.h | 1 + fs/xfs/libxfs/xfs_alloc_btree.c | 35 +-- fs/xfs/libxfs/xfs_attr_leaf.c | 1 + fs/xfs/libxfs/xfs_attr_remote.c | 1 + fs/xfs/libxfs/xfs_bit.c | 6 +- fs/xfs/libxfs/xfs_bmap.c | 27 +- fs/xfs/libxfs/xfs_bmap_btree.c | 1 + fs/xfs/libxfs/xfs_btree.c | 58 ++++ fs/xfs/libxfs/xfs_btree.h | 3 + fs/xfs/libxfs/xfs_da_btree.c | 1 + fs/xfs/libxfs/xfs_dir2_block.c | 1 + fs/xfs/libxfs/xfs_dir2_data.c | 2 + fs/xfs/libxfs/xfs_dir2_leaf.c | 2 + fs/xfs/libxfs/xfs_dir2_node.c | 1 + fs/xfs/libxfs/xfs_dquot_buf.c | 1 + fs/xfs/libxfs/xfs_format.h | 2 +- fs/xfs/libxfs/xfs_ialloc.c | 1 + fs/xfs/libxfs/xfs_ialloc_btree.c | 27 +- fs/xfs/libxfs/xfs_inode_buf.c | 2 + fs/xfs/libxfs/xfs_log_recover.h | 1 + fs/xfs/libxfs/xfs_sb.c | 2 + fs/xfs/libxfs/xfs_symlink_remote.c | 1 + fs/xfs/xfs_buf.c | 2 +- fs/xfs/xfs_buf.h | 1 + fs/xfs/xfs_error.c | 4 +- fs/xfs/xfs_file.c | 25 +- fs/xfs/xfs_iomap.c | 11 +- fs/xfs/xfs_log.c | 51 +++- fs/xfs/xfs_log_priv.h | 3 + fs/xfs/xfs_log_recover.c | 569 ++++++++++++++++++++++++++++++---= ---- fs/xfs/xfs_super.c | 2 +- fs/xfs/xfs_sysfs.c | 36 +++ fs/xfs/xfs_trans_dquot.c | 14 +- 34 files changed, 691 insertions(+), 208 deletions(-) --=20 Dave Chinner david@fromorbit.com --s2ZSL+KKDSLx8OML Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBAgAGBQJWihEyAAoJEK3oKUf0dfodH6sP/0MHR5pfBQZM+ug4YiZDWr4F PNs1VrgWyq/D4vLYXq9Qn8lLnsLf0x+hk+wcbXLFDJzhg4vR7ypUtjkMC8QblMIj H/3/djztjSUrm36cBBWhHEL/awqGAFus3ll/RDC+F0qvbdeOmWsOYmXlVzXhOtmB e42HzZBCU9Rb5cjP2AAViCj4qDBj9mD73XXznJPVFPNKsCknerOvz5YZejOqTfjJ 5iz0eeZ+u+W2ydVoqTB9LVv+IyJ200w8LK7yHyT5asbEp5wjwt0f7Fd7ODjZamLg 6j4BuKJpofDlOHW6EQ+COfNwz0TtBeEG0T7AhgXMh/YpJ0k9lAu5TGmYwmNYxFzw vGETWkPZxdwOld355bskQrtXUriUfLr0GBJpjc+GjipjzGAA474B6M8qQssT7p1e ehmE3HT3TTH93ULjCNjHriru4uZiTk5hDituWxZR6M1j2U0rd+DRobkuMgr1P+Za vW9s/VhLIlup02U4MruTMSSHbxqdxMka3EFRUWGA2m/EGQGSo0/5pFG+zf4Kh75B v5GxzfKVTUOncLrrv3c4Yatrk1QHNzwT5j2gYpbH8nvEISnUgDWt2ji+g3T3YRpM CmIC6t/yBsl8GlftPthxgoc7CK9bkK2ge/DARZtCsPkmF8nCFQvf/VAzqpymcrW4 Je+7A3NhQUnVCpfi9Tzx =yEBf -----END PGP SIGNATURE----- --s2ZSL+KKDSLx8OML-- From fengguang.wu@intel.com Mon Jan 4 02:00:27 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 220627F37 for ; Mon, 4 Jan 2016 02:00:27 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 130A7304032 for ; Mon, 4 Jan 2016 00:00:24 -0800 (PST) X-ASG-Debug-ID: 1451894419-04cb6c303241ce50001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id cRFKgPuu4o99dekP for ; Mon, 04 Jan 2016 00:00:19 -0800 (PST) X-Barracuda-Envelope-From: fengguang.wu@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 04 Jan 2016 00:00:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,519,1444719600"; d="gz'50?scan'50,208,50";a="23149670" Received: from bee.sh.intel.com (HELO bee) ([10.239.97.14]) by fmsmga004.fm.intel.com with ESMTP; 04 Jan 2016 00:00:16 -0800 Received: from kbuild by bee with local (Exim 4.83) (envelope-from ) id 1aG03p-0007lp-2Y; Mon, 04 Jan 2016 16:00:13 +0800 Date: Mon, 4 Jan 2016 15:59:28 +0800 From: kbuild test robot To: Brian Foster Cc: kbuild-all@01.org, xfs@oss.sgi.com, Dave Chinner Subject: [xfs:for-next 7/10] xfs_log_recover.c:undefined reference to `__moddi3' Message-ID: <201601041527.YSzP8A8J%fengguang.wu@intel.com> X-ASG-Orig-Subj: [xfs:for-next 7/10] xfs_log_recover.c:undefined reference to `__moddi3' MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="ew6BAiZeqk4r7MaW" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: fengguang.wu@intel.com X-SA-Exim-Scanned: No (on bee); SAEximRunCond expanded to false X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1451894419 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 --ew6BAiZeqk4r7MaW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline tree: https://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git for-next head: 887a9f293305476865fdd1d5258c0dce3d96104f commit: 841af8ebef885af1db95eb538dde51c439e174e1 [7/10] xfs: detect and trim torn writes during log recovery config: i386-randconfig-r0-01040711 (attached as .config) reproduce: git checkout 841af8ebef885af1db95eb538dde51c439e174e1 # save the attached .config to linux build tree make ARCH=i386 All errors (new ones prefixed by >>): fs/built-in.o: In function `xlog_find_tail': >> xfs_log_recover.c:(.text+0x1c5db5): undefined reference to `__moddi3' xfs_log_recover.c:(.text+0x1c5e9d): undefined reference to `__moddi3' --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/kbuild-all Intel Corporation --ew6BAiZeqk4r7MaW Content-Type: application/octet-stream Content-Disposition: attachment; filename=".config.gz" Content-Transfer-Encoding: base64 H4sICFMlilYAAy5jb25maWcAjDzbdtu2su/7K7TS87D3Qxrf4qTrLD9AICihIgkaAGXLL1yq raRedewcS+5O//7MALwA5FBtX1LNDG6DuWPon/7104y9HV6+bQ+P99unp79mX3fPu9ftYfcw +/L4tPvfWaJmhbIzkUj7MxBnj89vPz48nn++nF38fPHzyfvX+8+z1e71efc04y/PXx6/vsHo x5fnf/0E1FwVqVzUlxdzaWeP+9nzy2G23x3+1cBvP1/W52dXfwW/+x+yMFZX3EpV1IngKhG6 R5ZCp7VYi8IaILQiq6uCKy16ClXZsrJ1qnTO7NW73dOX87P3uO93LQXTfAkzp/7n1bvt6/3v H358vvxw786xd6esH3Zf/O9uXKb4KhFlbaqyVNr2SxrL+MpqxsUYt2RrUWfMioJvrCIG53nV /yiESOokZ3XOSpzWigHOLBw6E8XCLnvcQhRCS15LwxA/RsyrxRi4vBFysQz24jiTs43fdcnr NOE9Vt8Ykde3fLlgSVKzbKG0tMt8PC9nmZxr2DxwOWObwfxLZmpeVrUG3C2FY3wJLJMFcFPe iQErjbBViWLg5mBasAGHWpTI5/ArldrYmi+rYjVBV7KFoMn8juRc6II5aSyVMXKeiQGJqUwp imQKfcMKWy8rWKXM4QKXTJMUjnksc5Q2m4/WcGJjalVamQNbEhB/4JEsFlOUiYBLd8djGYju 8Fz+7mueZmxhrt69/4Jm4P1+++fu4f3u/scsBjz8eEevUpVazYXpZ0/lbS2Yzjbwu85FIB7l wjJgDwjvWmTm6ryFd6oIl25AZT88Pf724dvLw9vTbv/hf6qC5QKFRTAjPvw80Empr+sbpYNb m1cyS4BHoha3fj3j9c0ZpoWzck9ojN6+A6QdpNVKFDXs2ORlaIqkrUWxhjPj5nJpr87PWiTX cN81V3kp4c7fvevNXgOrrTCU9YPLYNlaaAMyheMIcM0qqwaSvwI5BHu3uJMljZkD5oxGZXeh UQgxt3dTIybWz+4uekS8p44B4YZCBgwJcFvH8Ld3x0er4+gLgvkgbKzKQCGVsShZV+/+/fzy vPtPdw3mhgX8NRuzliUfAfBfbrPw0KD+IPr5dSUqQe7LCwyohNKbmllwGktig+mSFUloRCoj wJyGK7EqId2quyanno4Ctwiq34o+qMps//bb/q/9YfetF/3OdYAmOV0mvAqgzFLd0Bi+DAUS IYnKmSzCHfdQb5aIzSMJOG0Ols0uwawnkWkzJdNGIFEP4+iQjapgDJhQy5eJGhrDkCRhltGD 1+CvEnRXGUMvsOEZwQRnUtY9T4c+D+fzgclRZJ0DD1jya2UsQZcrtKiJ9/vu1uzjt93rnro4 K/kKLJaAmwmmKlS9vEMLlKvoBgAIHk+qRHKC+X6U9HLXjXFQSs7Ac4BFNo4n2rRbBW/6wW73 f8wOsOfZ9vlhtj9sD/vZ9v7+5e358Pj8NYwN3fadD+dcVYWF656QCsf5nirc4dwkKLRcgGIB BaUUlpkVRlLBtSDIhyZu0ABxS8CkijfgDqJ5NTPUvRSbGnBB4MQhHLkF9ocxo6fo2YHbRErS dOAMcIgsa26WJLJaCEfpQlGCFYhbeeUCrkl1dRKvgHsEuyHquVIUK51rreeyOAsMolw1cXSo 7qvWJsLdkB4QJ0vBqMjUXp1+igxYBfG/99sQByZedwK9XWhVlSZczYPGhmVIkAJ/7oQ+RlLK xBzDJ2It+YR1L8FQ26OjgWQopB0JuAV+dPdNXHh095AcHcM7hpIE82zVENHjl4KvSgUJF6q9 hXyLsiHgTcFM8zAYrMDgFcFv9JxFdHnANA0gYj64iwFtIeyAtEN5WcGoafoUYLtTDItLLThY 2ITSkDhfQbbAlbs4UCdxXKhZDrN5DxJEcToZRGgAGARmAInjMQCEYZjDq8Hvi0jieZcHoM90 eRJxmi5GaV1iAZGlLCCjDq7E65xMToMU3A8Ee8BF6TIfZ1EGY0puypWuS0huMdkOuFam4Wa9 5SO2N1g0h7BMojgE+wCRz9H+jtyuv8seHF4ybr3BEKuuAGw2ecCBFlJHK4CBLGyUVwTBhcjS Oi47THMD0pY6rcKp08qKIPMVpYqOJhcFy9JA3JybDQEulAgBwHKCR8so/WIykCmWrCXsqxkz 0kgXI6eUhpRc1teV1KuAgbDMnGkt3dX1944JeEKqmZefvpQzqO/ADup13matztc2laZy9/rl 5fXb9vl+NxN/7p4humAQZ3CMLyBM6p0wOXmTEY+XaPDr3A+pnaf3kU3Ly6ya+/GBK4I0j1nI HVeRAGZsTukiTDCQUytyF5TWkPTJVHJXZCAtF7jRVGZ0hORU09nmoVIrPyysjTWQusill6oo a/OpPLHGr1VeQnA8F6F0QUwFIetKbEAPQR8w2Y3EaDxbH4zi9lxxEHQOxB1NN8cgbkpWRAr8 kXg1VRGPGAT8eK8YpkAQBsFhlMo5T6OFrXQBka0Fjoemxi0jQaWx8gZbH5bwVsM6h4fCfCQC zDM9wEOx0pBSRtUdwiGWSq0GSKz8Qd6oh5MiHH5buahURaQfBm4PY/smsSKqT+AWN+C/Mc1x 5tYVYAeraLEAE1kkvtzZsL9mpaR2Wcqhtjjc8gbURTAfSwxwubyFW+3Rxq04dFR/f4lhiRjk nMISE7dWQTfHS6p8WGZx3KJUoSldrr1eGZYKyORKLJQOZ2hE1/PX1eYGFM04XwyawCWqmqgy YnjlE+W2vEScwAiOlqsGQ2BHzIPIwp0fNUBwiPQCc0egwogkRsLtFFRMMiaEy6gy9jezAccU af683pJZIqVXBRYSRFOQJfjvrxKLteAdAgHIVVJloLdoX9D5a+L+jceAiqh8XLjmqtw0Cljb LBA/SHAKMFpw3BumkwChINmCYKApL5+PEMy9kbQecsHV+v1v2/3uYfaHd5bfX1++PD4N0m4k aypkBDe7kziy1ltEUYU7TGtZvOVZCmRp6CwZJIlpGMJaCO8g6AntoQuMDPpgyENjNoey4EG+ NARqwah4oqGpCsRPDvZoOvtSSaMxtMNq5oGkvSvQxsHliFLSyVyDRgXUtL9rJc1l8xk4gSoQ w3mcDGfzhKXjdGVuFiTQlxFHuY0VCy3tIO3heeKeYJw50a2QldvXwyO+8s3sX993+1CwgNBK lyxAbMkKTgZ+uUmU6UmDiDaVEdhXLNXM3P++w4eAMLaTymd7hVJh4bGBJqBguPMxhqfXV9/6 WNDXgQPy7igtDhc4UkJu5r1697DbPoCe7bokEO5W5KXtPFQYpZjiNIhJCveKA/JSQsCCAsqH D1DdSwuzCh20zm8GFGiTXNU5cdO4wuY0ib5pCbrzEoURf9uvL/e7/f7ldXaA23YVvS+77eHt dRfcxh1aL//m2Ae3OcU3fPVNBQPfLXyxoN8kosStBTOHL3N9DtXNiAQLsHupNEtSsZDAPwhn paGVGElY3s9PVHN6iUnrfC57cWkhXVQTzaoTfn52ejv50g0OSkYc9wIJ92mB+fi+5SKe+Apa e7uB0ASSNXCCi0qEVWPQF7aWOoq6W9hkmX0FSU47Tx/5r/MmOUpp1mVuiB9IZyjtugM3TGWQ LemgslcoV3f0OWtvMC8+X9KW9OMRhDV8EpfntzTucmpCcL9WVrmUf4M+js+PYi9o7GpiS6tP E/DPNJzryii6cJm75EpMpJ75jSzwhYdPbKRBn5PWXmQseoTIF0IlYnF7Ss/lsHU2cT18o+Xt JJPXkvHzmn5tdMgJhmEVZ2IUGtxJQ9JEQBMq75QZC21NN4MvdF+EJNnpNM7lCTmGqWGBDDFN DnB5EYNzWci8yt17Sgp5XLa5+hjinXJzm+UmDOiBGLyLN5tjMJjKMZDDplhFTOJi61xYFjX2 LEthu0JD6z7DnMzcSBV1O0iV51W9FFkZjilcr4a5Og1rIq2PJfOMFr1WGVgipjeRlfTII8Oc /RqaeWRKOXnneZjMu1ssfMlihAAAMP1XEabYTiQExPQbyDzCdqj4F5KdXs7lYKQwZSpvQ85b BXI1j7yy/LyaFGct0PbCHFVJBqWSa4WtWEFG1II6+ek1p0OBDB2bzWUyTldSXwqJ9gTCOrlf kAZJh/KFwudJ8CJ05cnjLshU0uMuLxahj4VUUKWpEfbq5Ac/8f8NRsR3UbJhbl4uNxC+JYmu ra96DfCuYkKiXaeSsyOwjOsJixJLNzqFcAawtSgY0Zvk4vdptMhACtueBEg9QkMgMxTIrI1N 8Jm8En2uRo/tg8pmWzkrKkblmv3WPElgFFrMsELkl8LMKc43u5l8UXI8bJA3ReCGv2G2729F Gg4JeTg8LmA20UqNqa6bZOphDze9VLbMyIjMlBmEgaV1e3FW7iLah7+4lgzjVzt4EsU6NI/z qVwuNItBR6SwjeHx0hahkZ2DGSQjZB/rQcQW1hvRco+LcisT3GObOzlh9q0Qib66OPnlsjNc R4s0FBbc5Q3bRPE1SZb7Rw/q6TJsCFwF2+WZACuObjrOl1RhsbhMP5ZO9DTdlUpRenA3rxJI NHo6498lCNK27OL669pq9VS+CKwWWsfVRvdyGWg4FoUdHEvOq0FzhTNmKODeL03a09LSku/s NL4c13NIn/C9TFfl8I0j8kEGMh0setxcXXY6AGEY6EBeZQNxzq3W8a/aMDgdGMhJeKtLjRQG daeYzIkOFiYx8GiJT2MjPzTikGKbusSU3snLsMruy4KxpzDR7fUZLMR44T2IlI55m0IufS93 9enJyRTq7OMJ5f/u6vOTk8jMuVlo2qvz3g/6dHapsacnyivFraDkmGtmloMSO5onycHdgA3X 6GxPG1/bt6gIrIq73hAqcmvHu+AZxp9Frrp5rFsnRkWV5qbMNad1E7yaTDd1ltjx82ponGMj 31XLXv67e5192z5vv+6+7Z4ProLCeClnL9+xiBbVz5oy74SWdU3IdPadE3vHhYLdwq82wHQs NKPiou/+dl3BvhaOQ8qwA9xBmpfLUt34xxILU/Vd7X2TIm/fgBYTPTd+fogEU+NnmzgEXPy6 VmswZjIRYZd1PJPgfrmUCmAdBRseZc4sBDabIbSyNm6ec+A1rK6mpk7ZeEACXnqK3iVqWlzX ZfSE2XJEGKz3yKgPNEaOVuuHscUCTBdkr5MMtUuhc5aN5uCVgZy3TkxCV3kckW+G8ULUXcs0 +XRblt82B4nKyHYz73/G79R+q+CCGWjuEdEyc7qY5ZBTnVAhHyCdXaojZOBcK+xXXUKYeIMe QxUZ7SYdOfzfdNeuk89SDJ8rO3jzlBnPiIgJgyCxRQjEgHa3JpVXfdfmLH3d/d/b7vn+r9n+ ftu8GLUGshHUOON3ortQa6LRtEOjaZjqlPMUbTCIE+H7K36yU0y31xGD0GoY4NM/H4JPvK5L 7Z8PUUUCcVZBZbIkPeAwkAFzvRYTbAtOO0XRHm0C350jDB0jin+ybWK7nVB8GQrF7OH18c/o JaYPWcr2Y6U4euQc18KlpivyjTkcErXBT8aKFh1kfPzj2ckY7NZExhXqpl5d9uX7GPFpasSn 1mnGpb5b55zzCVPgIrkSMghwi75ApmVBf5kQk0o+/ZTRU5mcVnG38wtfRB9sLYy/PYvA9uAT aPzJHQQAxUJXRShBLXgJ0jy5rOilc/xotP99+7p7oCKc+FyZnI/GotjJh6ddbH6GbektzElv BgkkGTlEVLkoom5rF+pg1md6Oq6qMpvwCl7IkWy05/nbvj3s7N/gzWa7w/3P/wk60XgkUOjv FgoTItpwO3Se+59HSBKpBdnv7tGsCKIaBOGKMcTPEMPahQeU7ssNMziGwIgP8v+JPQBmWG2I jpCbCdfFj8UMiNX+g7Y29sam30laYyu6ZRqRUq0ncaWe3l7JjJzqjWxbg3wCAOLw+8v+MLt/ eT68vjw9QTowMqLNN5tNJ1UvcoZRjptjXhM/JSJkqX2cREV8mYzSyULYjx9PTgnKhQhlxBog Ownizzypi3koGVge621syXMuWWhMPMQ1bdRcUoKCM/gSUsOv9/fb14fZb6+PD1/j9oINPjzQ N5Jcfjr7hVbbz2cnv9CvO02qhndG4jVcRyKPmPGNSccGTPzY3b8dtr897dxX1DPXrHrYzz7M xLe3p21rEpt5sEkmt9g1FBbzII7EzLgLKbCpaClYErURNEMN17IcNtIxVUXa1tAimDxOg8+l IXN1hWHuuBnOdyNIFdUwypw7TFjB7r69LHaH/768/oGRRO8d2oGMr4AyFB4HASvFqKJpVcig gxp/OcpwAvxaYCXoeBwSB5oXAMfPSbG2kjNNpW84bWnB0WcMEtF0M1AtN7pcbpw6QlCUl4NX 8JDYt+BROmvz8CjwEwwL+c5lbNilpsNeIcjLFpHj9JB6jSGVX5p+oXcEn0/OTq/D0T20Xqw1 1eARUOTrcCuJ4F4Ousk8pNYgkyQDsizI1eHHWdCKEVfH4Gfz3EpnXpZl1DXenn0MOZyxkura LpeqcGLZLiaEwAN+DN5fe1hdZM3/uG8eJL7ghu8aAaVRjby3N8f4cF7kj2kqiU5/rt92bzvQ ng9NU9Sgs6+hr/mciqNb7NLOBxLrwSmp/C16II8tGD8iOzLKfdFxTQ3UZHdYiwXLSu3RpHQW 0eKtuKZqeB16nlKzLvREzNcSJGZoCUYk8O/Ee2M3iaakvGPUtfscb3j1NV+qlaD4d/03rMDH 1WO8SK89SS+APWYMWy5T8vYl/ZbSYLEQ+q2J65+2+/3jl8f79i+EBLQ87IVtANgbKnm8EQRb LiGpvR1uBlHpzcReEFmdB/ajAbgcPEorGvgRu48k2qzL8YYRejkGp/iWQex2/K3dkAHliOXt fKS9RALI9i1fxpvwMN9NHfztggDF83KoFg2mmG/s1A03JFXYchDAsRGERFhxayeW46yYetJv Ts8mvp1s8ZIsdnSSLcOO5IQHoWxS4McGRuGfX+gFZQ5Wj7l21sCpdrC64JFv7RE5Z+TffQhI Bt9GrL07GH7smMli5V5Kp07tCCYipLwcKhZC6kX8/uFg6IXpSGBpdFRf8t+RuhBITwTHAY0P kabuRN9ijrip46/f5tdd9tTEi7PDbn+IapJLlmuWOIPZdCXf/7E7zPT24fEFu94PL/cvT1H2 wMDdk+lHEdgFuBvNInVF0JxTjyuIWdx09o0Vs2T35+P9bpZ06V00yxpJ6GlMNtpGItbDbXCW cYjULX71ORFQIlkmJj5gLr2GTe3hV4bPbfEuGiBmojSia9Qd7FXkpsn9JvfpSejUC/GlYKu/ m6OhmZ5ktWb4Qc/RdbLbIT5keiMh8U00A/3ZJ2duyI6snQ8vJB7PP32KXl47SasMhKr4veOX 7f1uJGnIWUdBnkiYBLFn8YUu3JAY1jBvBHdcH0E/46cEI6hRqfsWhwLW3Aw561uQfVc2VS2Q OmGtysnXhEW63tFkoTZJjS8U4e+E1RCWatYaDzfRKB91dM1fosI/CpIZFrlJh08RQwZ2Dt0a eb/M85dXLIy+//7yepg9eGPR14IcjZF6jOlmtHZTA0XLgOTl+evTbrZ/+44z9oSJKhbhs6Ew soV1XMBuIbMxPbw7lxUrzfIGQcqmVTI/P4M46QhNJq3wvmlE0+U7l2BAxusvpJ7L7Mg4Xp6d np2MToSNhHPx/4xdWZPjtq7+K366lVSdOWPJm/yQB5mSbY21RZSXnheX0+1kutJbdfeczPz7 C5CURFKgOg+zGIC4LyAIfEx3CDDTr6w/HqdEZujwwvf5UE12PAq/fkWXEVeJdnw5W3aJi75Z D3QkTJJmIHc2vGQDOlCcIg4ZtVtyhpyuSsckXxV5pIha1wifTzuVLpsMEXsYc2QTpomd4iHl iUP6kIRmoTLGTcKq1v+PkZ2RcY8LtGqN05O2SsEXeUyrPsDbJpGbR2+BwCF7EOhaAHNHVN4E zQxePXy/vj8/v39zdi18I31hjWqzzPhd1SZ/y5JVbaycGlF8/ZNiVDqwg2Ts0W/msU87b6eU KKTNS6s/GlZYbyfUIUgTMWEXNMbkmJD4IJpIz2PYKJVzPDQivzPKNK5ah2X+eHLqtWYJKotx cFT0Nb1XSm5Upx7xTT2ho0sUO93HaNd2F7GmeuoAf6y8supAHeJXqn+MFI6ID7DTE8BuSOkY w2OShSd9OYGfavcV4Hy/BZohfL1LHLGOkqUCeGl3E1T0l9Z5eVkK91HQ360T7nIQxCdMaHCb fE1CDB3rfZ53VzE99byDA7y/VeRR0b+03EtECBmQQOQD2kqdlWvj6NbQzhlGEZBG2zCPQoz7 1UznlcwJltBMOJMItKWOvz6KixTdubEVhV1PhvRq2/4JdJxWQgOoadOR0e52pAXJBk0nTTFC WbsNStPiKAJItQsCrQWEJlclB4eDTqvqVQ6ICFRPutA3UqRFTSv3A0qjLoVXhRbGHQxe4wZD /j4nOsSVovEy0yICFTHLdMNd87UOU4d3ewL1M0LMq7XR2HHO4hZcp71OlvuLqdkXMJgZPcey Whsm8EPgThrKNRIhf3RYF1G2ZIgHyGhBvyIBjRVWC0VWBd2/wYTJJEqnAGGpXy9Pb/KOa5Re fho7I6awSnfQS9wsqvSst4oq3fIryrS81re9vPfrXGnBsYngdx22js4GgXOEt+1+Zip1oyxF QQfEAMuEF0FKG8EMfZ3B4aALnAal+nNVZJ/XD5e3b6Pbb/cvffVB9MA6Mdv9SxzFTI5Zgw7D +tyQjQJDCsLEVAhXa1fZcZSuQtDQj7DJbc+eWQ+L6w9yp3aTWXxHRCJRCEeIYV9y4rsGMFQ+ sSojaL5dSEF1BF027GAoF1Rh0Jj5aHPCLOJ11KfDoh/2qfs6SU0qDBWzAlVhEcKViH9Rh43s 8vKieWfhnbMcYJdbjN62xleRlVBsbFC8mOR2s6AXtcvgKMY8Ol4xh+qNArCPCxmngHQrcE2p FTtvTid7TEsfB3SoXcNpnEJNFc2SRYv5CVvLaM+EbRXRSDPmK79yBHiKltoF4+lpSIKzlX/u Fchujffrg5OdTqfjjSPcFTvDYTySPKdtTCwPJUJWRhG9c4pmES5NBwRqcWwqwsaFo9FozlQY 60XYjlrb+PXhz0/o4XK5f7rejUCof0Iyy56x2cwRA4ztmkKm7opvLa6+TNSRPXvQmlEXNcYV YOicHmmkuHElgEiQ6/mBxYSphh7nOlJmu0P5WFPbQBfdv/39qXj6xHAGum3CmAgM583EUZMc gZZixsyJ31DxJN/nELIr/T7ISEFyzHbPhtBF26+jGCG4HFbTVgoHoD3lBKMQazkMIaHfDiWR RL3lSdBBmytca4AsYcJ3hQhLN4euxZT7ZwtXQjQUIRshbIs9HGhhjI36d8U8r1b1sUpMB9pO DsaJe6cSIixcO6IPWwlYdicUQkUrgX/xJCOajLLzt8xtwpPZ2F0+UIZyywtZTIS0xNXp/+S/ /gjWs9Hj9fH59adr1ZAfOPYNdK83gyI0soA/mQqnHtCgHQg/IL5f0ettsSaytaNlJPSW+bZD Q3i0CCBsujpJKhy4kpA+ancfintMylWlk+B7AfXcK4coY99LTzE3pAdIww1PQbBYzqkPYdWk kNYbdl6I+naFyUvjhzoLZjCvwk3c+QL2bfsgbAY1KQwjw5KkYI3yfZriD9pQo4TW9H0zlNwV 0tJ8if6vnOOOk5QT/0Tv4QJHqfwdfR/5OaI36ybBKGTLOa01NSJ7V1B7I8DgRD6wgjdiqQVr JPetagXb9v0b+i7ejf643l6+v11HGFSEmDCgUgqXOfnJw/X2/Xqnz8+21VdufGHkF3yYz0/0 aaHhuxQDFoGidi53NYsOjoCdOhTBSue4ptU1aZL6cMhshytQ8YGxIFrokPV9yLP7t1vt1N8c SOOcw3KFr2JM0sPY184UYTTzZ6dzVOo4lRrRtF9E+yy7EUYJbfYmK9BuON2c5TbMXeAnfIOO 04xe8OtknQmDEHUnxvhy4vPp2DCpxjlLC46QUBiEYttwWrFteU5SMvqujPgyGPthqhksEp76 y/F4YvpTI82nTh5NQ9cgAoeTruEaxmrrLQIHfUHQRaGWY81DdZux+WRmnEIj7s0D2jl5lZXj YIad6LzLL7d7ym6NVwP8mKCKvubhchqMjXsmWnVmvtqluvkkKDBw4IOwOvverH8VHcclno56 F5CSDhPO164dFFEGoPbIWXiaB4tZ14GKvpywk7HlsNXCG/cGmHwo4frj8jZKnt7eX78/CmRg FYnyjnYpceuKGG24xt3ev+B/dch+OJ1rliBt4omZpE7a4cP79fUyWpebcPTn/evjP5D+6O75 n6eH58vdSL5Y0yUaooNtiMf+MjXu1KTN9pCZCrS6JcbjYpYwYQeU54b2dpgla4LcfbLFSAMX k6FfPZGgU/75pUV/4++X9+so6+KYf2EFz36lzjUx2zrcgk6pQHx0MtWrLWHpcJgAkTim9GmJ AKpD08gfUot4uF5gH3u7wins+VYMDGGh/Hx/d8U//33/8S4sJt+uDy+f75/+fB49P41w9xda qH4Bj3A2dUioH4LFjddFkLIxUB0kBVMgq9exS+pYpeVkuk206oO4+/7gS20DMciIFrsqENm0 qoqKU+oUyEHRhnUikBHB7a76qZc0WE1daonQbqlttC4T0AtoyAKpZpX5/Mf3v/68/2H3i7Lx 93uGghJstaUsmk+HNS5IG3TO3izFo69WuDdtDewlQeCE9bWisk7mPm0QaRWLrzYSQ08kjNl8 SBUVMmnizU6TYZksWkw/SqdOkpPjAl5v3+FU6ipZp/GwDB5cHeZEXWTyL0RmH4vQJuhW+Svr yXxY5IsAtxpWwDnz/A/6soTmHR6adeAtHNpDJ+J7w10tRD5QWHmwmHrDTVdGzB/D0EM3nH8n mMfH4SY6HHcOL45GIkkyOC9+IAN9+kET8JQtx/EHvVpXGSiPgyKHJAx8dvpg3tQsmLPxeHiu w9oSmSBCar/mSWNZ7WldyES4En2lq8IkkoDxlNILH2iaK35uvBwpKMrHWE9UZtRGvDtSttdy UXZVaIkm+wsoYn//Z/R+ebn+Z8SiT6Dv/UotoJwMYN9WkqkdfxpawQ0/niaZqr/58Qp2vjwy QM6bhHWnxYamG1NFJeH/eJWvv7wg6Gmx2ZjPlyGVo49jyG9yZrRK3einb1ZvovVKwv0/GvQ1 I8mJ+Jvi8JDb7wa09DRZwT/EB/hAnvkaomRVJZlWWhyl60inAwm6uHMWr7X0BtEqP/lSij5o NjInqLEDu3QV++4Emm6eHM8wLU9iOrhz2pacVs0EF9JYuuZ2IwCt5uaHtkuQxQ7ZcPHChC0G C4ACyw8Elq5tXU7qw2ANssPeAUUgF48SrQ608i/zRzs7jIQBiYq5MB8FP4by+Y4bLThYiuUO NhZQuYZlBvDMWpnhpgA94CMB3xLQ2fs13zLjeKCRndd+hsyQ+7kaz3VSOB7CEjNrz2GZcmim 6oxaHuzJpfiwaqw11wnxszBcEZxTFhnnde7IWK4xg9woO028pTcwV2LXKUsun3uBQC5BDdxi m8hhKGwW24FOSsqBsYH4SY7YmYYfeuQNudxYy9Be+HUHd0n5mpTnuCwFerCVOrI4+m6xemCq 8dqhlkvuTTabsADWVFoDFUK/i+GFtwO04qSEwvN6oLOR/8EWkZZDCURsspz9GFhzsCbLBW3O lDoQLycD1TxGC29JXazJ7G24eNmH2QerfZkFLiVRTt71cLNJY7abz7ZxypPCPUuNPX3I2UDW krLLCE7BIzngQ+MtmZa3110cW2okHl8Tdri4w13s2HajWvO9Wcsj6tifOU4oIthjUI+IKNtp HVYbDHWTj8F1hnCWnRPhJUDbyYGNT6uQ4dPILIV6/mh+gR6ZlOcTXuigd6YqjP0Z6R8rtTP1 gY5tuucWZJi0xcRxPPImy+nol/X96/UIf36lzB7rpIpt/+8e85wX3LH9hgw6vkBwSOE7Si+j cCpx+3sehOJqwQ1IomMzk8zKUNgFzfqpHnCGE0MiIu1MU3Z+yIwfdownkkAzQTHd7RmoVVWy foM/vXx/d572krw0MT4EoRf9ZzDXa4SpTI030yQHIzDlNapBlrDKO3SPfTQ5WVhXyWknHWdb b9AHhLc0AtPMjwoYv9ZtrcmBUR/uqaXUEuOsiuP8fPrNG/vTYZmb3xbzwBT5UtwQlY0PJHHV BbHLHum59Bgf7OKbVRHqT4U2FFB/y9ksCLqmtDhL6pt6t6LS+r32xgvjZkdj+d6cUh5aiXRH J6o8diiyGCFxRGZYs3A+NV03CZFg6lFVlwNJHxFdMbNg4tOmG0NmQrlQaRmcFpMZ1bYZ41SB ysrzPbKicLKoSTDFVqIo4xw3Ak5kx+viGB7DGzJpvs93K8rO0TXhMZ2OJ2OixCcxRh7JOeWc SjAHuALrVfSGcg7zMNXfUe8YE2M/7egRtai2bFasKk1jbembtb8j09tUCYWMYPDPul2i4+wR sj7T78RbnoCICVlNlIQnEWxLeaTH/rfMOtMBcLvkBJ4eIZ+ot8b4iqycYvsOjbKVO+J7qqRj ZiuC1s80NeOTuzohNGRRUVfEpszKwDTueBiiSzfIMYngB1m7r9s43+4pXawViVZLajCEWQw0 sipwWlsVmypc0weSbhzy2dijdeZWBtf+vQMyuxU6lY5X4rTeSXcwmmANpvOTk0wg5NKIFUqg 2LOt3KgGpGxcLrEZbS+vd+L6OflcjGzjoen8RTiqWRLi5zkJxlPfJsLfpkubJLM68NnCM50L BAd0hp3DCUgJMDgdU1qsZKfJCth2fhKXwSCp63UUtjhAQuShftmgJc503nvZIm1COB5tQOuG BgdC2K6JRFqBdGqENDfkONt74x09YlqhNZz8vF53s2+X18vtO8IH2l5Ctf5y30G3hxc5L9JY voUg0fO5LtkIaE4qxz4N5DoyQsVFhp6MyGvL4FzWN8Y7y4ey5hL+qUwxPA2t5Mx811FhkeOX ztESpvi8j4yWc1hP8+JrkVEbEKIaGKq2CLNyI45JNreCKKEqltddx9jJF0qUD/zr/eWhH9Oj aiEebGD68wWKEfimv1FL1N6pb0J5aDnDnVJn9LrS+Ioljq+4OZ0ael6d9yL8a0pxK3ztLYtb EbsXhVDzwN9QdwvBNaedYI3mofCejCLVfhAYqqWRBeoD9NlcF8uSvuNy/vz0CZlAEX0urrT6 93EyFWyQ1HLstlhNR7nrY740rxG1PrZT/+Jw7VNszljuuL9vJbx5whcn0rQlRaDLV3EVhWQB 1Pr8pQ43WNOhrJToR2Lo9mbLWAlVrDd6cdGHwSveRNGftlHsqnTY9iQbhuI5LZ0lE3Yqcjkp Swni0yytB6ZMGtrSKj1Omz7UVfgyS2CzzqOUfhLy2Dwn8tgjSRjYpMC1ieBa+AQdA197I8ji ST69aB3r4LAI6hK2FbJf2lIrZn4woneqyXJu7KRhWaYJc/mlFvlN2feoU6bLW/cGiheyAqxE X/rwtg7BuqYWKlJDnRqqT3a0MOibpGXcmnmyLlmwmMx/SGpnHuLMooDO0xsyGKcv6BhI68/m WquXDlUThtFGvsLjAgiu2Ub1g2Y2RKdIypakOOgWiaiQ+mDSWQlQ8rjIaW6+PxS1qe4jO3e8 2ok8kZeT22TnFGAVDQPdlIrXk8nX0nd4Nccps025mqpj6YknOITeWLjY0oDkM8KS59uPkmAD NU83aAYpoIoDv/mwNpLl0zYWDd+kMAxaQMz2p0Zfyb4/vN+/PFx/wIzAcolwNapw+JE1MBtq ycLlbOppxkGD8aP/BVTKWOaArAAGbBBvTYJniBD92LVh+PDX8+v9+7fHN7OkYYrA6lZDILFk aztbSe772GD67dkKnW27NlHLyQjKA3Q3rLeVT+K5HM9a/pw2c7V8h7+e4GfRYkb7Lil24DkO xWKaBq4TLDK544ECycwcR1tgovcaPZXE7JdvETj5wmtr6W4z4M8d7n6KvZzTtgJku7YtxStN 2AHp941erY4O5iwj3Lhxpv98e78+jv5APAcVmf3LIwyah5+j6+Mf17u7693os5L6BColOpP+ ag5ohhj6diAlMqKYJ5tcXtE77qFRLN74Y3cnYcKOKbeLszKNzIldSMumVRKYQ8O+CELoFDrC RWV/ZXVsYGki9YR4Nqdey8Y/YA9/Aq0bZD7LmXi5u7y8u2dglBQIS733yVswFEhz36xqF7Zn VkMFtKVoS3EkVhWrol7vv349FzxZm8nWYcHPsX5NJKhJfqOQ86yxWKJXtnXuFbUr3r/JZVu1 gDbG9Nor7SNklBVQtHy9X5lF5Knx4kxLUgEZdifJyD07HIMQweX2AxFrx+xqQTrEKyyWTsvh hOpX8v6WBkQjhrHkzqfS87oU4j+75G4f7mVoCZHsGbRIBPDZyQerrEwUM40Sh5KjCamBRp4q WiG1LrRF+wuhjC7vz6/9PasuoeDPt38TxYYqerMgUG+xq+TiJ/GCQbm9wUda8FbPCWz//jzC 2AoYkDAP7+4xtAImp8jt7b/6YDRzwrMcUT1hqjyGUDtp8oADGFYTStGNSZKgkIQsohksqj7E ODgRGq6Zq3D8mTqO+L5xW9RpXaiB1KNkyPLj5eUF1nOxpBIzUZYmi8gnp2X5j2Fp3BjomQ2v sELS9ZaQYKY3cNbHR3zdIodTMKN3XMH+2l+JSxhIn1S90Qg9WHdvPD3jcxoujBchtF54QUCZ G2QN62BhmFVFD5HDqGFNPO/Ua9M+/ki7ZYsqXH+8wIgnO1De4LorIC4bHY79nYDvrKLQm3Xs OkVdB7OFTeUnbzbu1Pl19FHpWXXDa2E3Io+qchSGy/FsZuWk9kczNakhuNJJy8lyOrHrEaZZ aE+nis3qWTCxqHXJ57NgTnQeMpYedb+t8307PXV7qj3L81FfDyjOQmBVBw5fWjla5UOuDh8v ORjSc1IMzNpyaEpXEZtYkRvSDo2Pi77+m9GMSKoTPg56SSAgYu9bY61oNRMNQtBrNg/v0z/3 6uyUXUC5NbM+egoYS1z9F3QLdkIR96dLqrdNkcDXS9JxvGNGMfStUxWXP1z+d7VLKlUf8Vgm XQQpwA1bW0vGgo0DJwOfsYoUpl4/U5TxKM8GM5W5I3l/4mBMPBdjos9xiwWLB6U961KL+ZhO eRE4GZ4ryyAeU4gSrcjqd39hGObUY7T7skz1d880qu2OVUah5BtrjNpvw4g1b8NSJjBQf9pv FU1J91pCpwcuuuegG8tuw7FeNbW42DKn06mfoGIoe1Mv1YZNaRp20WCT0D1RwlPpg1633sOC twn3m7ifOex73gJtpkTOikfdyjYiavnGB9xZP/HqNNMaEA1fG+j1hJeYrBE8rliQYwBVIDJs JHreNg0jLYOFv+jTlfJIZJWHG/p5wq4w3nS2WDgLulwMfw0SQb9A0J9Tb3aiyiRY5KKqS/gz oprIWExmJGMGbdpnwAF1Ml1QxRBb9XiwHELE9xb9Phfj7JzWzF9ONbi/5sOqXk5Bj/mpD9Hx Sc1Za9HAV+dre80QRPxbvEVuMUMdxHJ7zHQzrfiJT0XrVZZEdVjeJv2A4vzyDnozdXus8CRW Sb3f7Ku96dNgMWkrYSsWLaYetaoaAoF5k9dwMm/siE82ZahHNkwJbccyGUsHY+KRlY6ypT8d AvMIo3px8sZUqjVU1MnQhpPBmPsOxmJMtxmy6CNVK8PZwhX33cjsgjp2OCy1It74Q5l1mHmz rfOZ9Q64pExjBJfrNw5feWMK6qQ+lWQPRXzuCNvuJDyr9rZAnKawgmT90qiDCtHuyWwHyjtl 8WqbAo6Z49m6XxVx/vTXm35268VssphxKr+MeZNFMEGVYShPOI1mUT/LTTrzAk5UEBj+mGfE F6BehFRBgEF7VSm2OGGHeT/FbbKdexNiPiTwRbO8Ee08c5x0Gwk0C9rj0k5Enukt6hc29ftU GLqV5/vEEETI4nATEwyxP8yo0Qks2B+HBh9K+N6MTHXq+74rVX86tAgKifnY+fF8eDVAbWH+ /5RdW3PbuJL+K3ramq09UxGpG7Vb8wCBlIQxbyFISc6LSmNrZlzr2Nk42T3594sGSAoAu+mc h8R2f43Ghbg0gEb3dDk+rWimAI+I6vAsMSs2m2ONfB3w1bOcrdGxANCcsqeweBZjE7fmWK+w JlLQLFihOkPfm+smRKZwtdudTUN0osqSfBsGm4ybzj728bLlDOkP2QqnLoaNp6hIkypqhFEj ZFCCyTuWW4TmFq2w75Rmo02oYGStU9QZIWwRzoj3czbPfHS0aQ6kDsZaAmkIAOYhWr+85mbX LiTuMb1n5LUaBGi1AFq9s34rHrWpG5t1gWPt7kJu5d9GizU+3MsMj6vdpZX7OkDaSpGxvq/I PMDqOHbR2q+0WRKsZqtRnkQtgnN0Q2VxhEqVGxZZActjOMWLl0k+X2Vj3aZjWaPTsUE3s/V4 8dXavFiqve/QdeJAXqbmLlwd5UEYxVEwNqMypfBMAyK9VNv90cSqoaIQbSeRs3A6Pt0DC2HJ 17q+46v5sOfU+4xj/vDqrFQaO0FHpidF99z+2cioDujt/QcCIAwRL5t3FWDFt4yWaEj1jqMO wgBt4EMdhbOxQh4jpQMG8bB7A7AOEMVPAyEFIE2o6ciYN3TQ7HhdpVjpFUe6ihZo0AeXZ+m4 LLlBy3C136JZKyTR0KidRN+HwbxpsAUZstV30wA949eTuh1RuSUM1dQOAK/K8AwDXE2hgRw6 xi7a+q4ArzJJeT4KmWASbcYtE5WJ7YFWCEuio6rolzc/naQ9dUjTgvtRdgbp6FIhjKP1BIYN y3f6v3fz/Mlq/avVSbLGvFzAjQf0JYuWx1NGTACGSRb8HNeyE0xxzubT03s8ibFEHuM6go/8 uMCMCqTcgEN5KTZp76RQvr48PbxN5NPz08Pry2RzefjvL88X20GltOODgQhZOlEhtFQu9Ctl S/oQdSYIiC82nwHUhoanyqttaH3hjpwbCyHDPMAuuJ+QMr7QmLZ+hfJpY3cqb5cNn1pubITZ 04ZnbPBhNl9fL48Pr58nb1+uDxC/esKyDXO8VHHEUZc2rvzz+8uDjoVDRYvLtvHAbYCmDVzV WSCTs5W7SJWZvrAofS95diJWh9Fq6r0jAEQVf7Genk4u1RyXDspVgfkS6iMciqCvBQaJ9EFN 6Huy9RkWbv5As4/betoMER+gm0kNOoZdQIHDGud2xCK2jl5vh7Y1WHxJwTG9FkDFX6bOOW9a ctIQAzDcWuE2QUE7+jU0aFpKs3yisj0+MmyIYvud5Z/OPCtidBAAR2/756SLIu2ShEhj0IVf dk1eTtGoA9v4dvXhJQMridVyjYbG6OBoPnP7h7nUWfnl1mT0WLpH1ytE0joalKpeztC7GA12 ZwiuKOymHOhVUje+fLUfXKgejh/i60TDK38Xr+UglrHHsJiir+X71E4YSk3tzTNcSfmiXqJ7 HUBlwpGpRor5annCgGzh7g16IjVXa4a7+0j1Hm+Aw47PuvbZnBbT6WCSZZtZMB16kbbF3Evu 6pNArcFb0Gy2UBqC5Ph5L7C1xi9eYrg5jKgW801j4OIrmC6cOEjmvgxXizW08mbx7oINoZo7 Nbdyhk6V0LLI8anrAMvCMcKxqe6LhxZRU8vMOrnoNn3Y8thhrIkJD0eKYzmdj37gYxqEqxkq P81mi5FReHvzRrNkRMH0lEAa2umVtxKfipyNrJfHLJpPp36hFXUWnEiH8R3LYvoey3qNXRH2 ET7tfHsiqcPdOLbilKi6F2nNbPuAGwM8FWu0dVoum8y+pb3xwB5DbzFuXGhxGK+jiDgjt7ji xWyNG/NZTLn6ge8rLCajXo02QKu3fUbTa3VrNHmvpyHN0ipeBGKb87nIMiSRGfGd2TpEJyCP JcAEb1mu1Fq8ON7Dt54uZLqeTRcEtAxXAcMwmIFXaCE0EuK107Yd458BWPAapDWfOT58bhCm 5riomlhHswWNZDlf471Hg6jHH5fHKEc4tAiJgkdr+27Dg9a0QK0+YZBW7FCs1cL1lIyJNfes FBSt0d4MShveHf0l10K2zacksK3LLOyY4Z/xIy8y/Shh9DsMrF1vkAyzkrnn4C4oA+wE0uJZ ZNFquUJlD3SlG6YW30WwnFFYp4agWAgXM0h+RvMIZ3iH7bSV0dr0yguRta/CeGi0/Anx64AW 76gvDuapKhbmG43doH7R7hbbJBZMGyaat6O3U4PP18eny+Th9esV865n0nGW6cAiJjm+mmtG 48rpXB8wXoczFjsBxuE3Vr+wFQPz/RvoZSXj6icKVPGfYFJ/1BX4G8Nu7g4iToqz87LVkA7z 1JnXDRUC4dIhxQ2P0U0yAYHiKpbvUK+C8WEz9PFYw8lZGwYHTwLvtVnMylppLnZQdcDi+5zB wYnOefhaONM9AYuFp1tJB/p5ty2hgPS3h1J0Bp+dw8X+TNL0PojnmfEPEnTt9qGddYRlOkVf wR8uvU7YYuXMHqYPqY2gHRpJ18TQ3LdoHe02yPv0hE9/EJVVEWGYAmgsN/iJs5GtvpfQv2Gz h8l9z6o7pFBAxj0Lbc53SZJbixqQKlYlWZE7Lzt16ZX6hc7ztxZdOgGXHeB8qokT6baUjK1W 0yV2CtXJ2S4jRzHUZLNd6rrGMNIS4NE/J9us7a+TX2Q9+ePydn10XPrz+xKiZ/ZR6Yk+f3l5 eHp+vtwiQU5++fb9Rf38h+J8eXuFX57Ch39M/vz6+vLt+vL4Zr2b7abWTVwd9KtymaQJH0xn rK6ZdudvLq6+Pz69Th6vD6+POps+9NGbfv32+emfpttr5iqWPWtHOzw9Xl8JKki4OBm4+PXF pfLLZ4gyZapquavQoIkf7hGNnKfPqtj/awI06cg0Haxr98EwPbwqLlU1OJh2mCAE3fUZbh1e 4XX79fnL9avLIU3bT76rzwpBkt5eH84PpqyPXRApp43rJnd8V9yI8Oa3tG8obKyOWRSuB6ul BTonHS4YKDQg0XUUrQhQDyAqpQaJlFkdugfoFnaCaCcRhS2mQ52gw+YklvH5XCkhM0dtePum ehlE+Prl7fJNfcanb9d/vw2P/hu7rA/6led/TNScrrrEN3BmhCRSi8SvclwusNRqkFJynGWP yWAanrfDwQ9SeFsqJBNWS4Xmak36e8JUr3t6uLx8uFOK0uVFzUh9zh+4rlVcHxAZQsbjJb1x uVX+t59MGj/99fTt8mw3qRoqzz8mZgL7UKZpP5wS3vmP6MazDkSm27tjql9fn9/gka3iuD6/ fpm8XP+PbloT5xFp2t3Xy5e/4YoR0SnZDrPQPOzUHFlZr2xbgtZodmUjfwss5zMAmqiDSVVg a2dcWTau6o/zXSZb1x9D+nbTQT9saLsBv779jbALpgWLz2qUxP3i0o0POPBqp9hJG+fNmtlu 7aeEaI8Z8WG1IB7+dTx8v5pOsS17xyBFGriuizokP5V6IlpH+EE+8NUx4eoSwCoI8UMrDbKY coIDMMti9elIOC+aQ8JoXKyJMFAAZjvcrQZglMsNwCQ7MPTiWSfcJZnfiofsuBtpn13GKMNk gJsY149080hclW6rt6NidgHORVU18vxRdU+S5+OJzntT8D3u/8A0oPYU5n08O7m+vu+6fPz0 9uX58kNHc9S+uCebr0+Pf1laxFe1bk/++P7nn2ooxL5DrK3ji6IbT3p0Idmr0cqzGGywbbVU UeMYP2lWkA4KdEgkuom15Kp/W5GmVWL77W0BXpT3qlhsAOiwYJtUOP47WqzSodVOSQomcefN fY31PMUn7yWeMwBozgBQOav9VSJ2ees5fiTHopSO0DjZJlWldqfu6zJgV1MuOGcgGjhjcMdL xNKFD8D4HeXGBJKrtO0cbO3qFFCLVFewNk5Ah53p7853ErJxhW+ghwlVqjLDbcch4f0mqcIp egOtYOP4z06g2ofwNaDAfE7M8LD0EBOZgnr/3sQnDOLOvMDJ7CBiYgKEHikOJCZWRHxKhaVJ NF2s8KVAdwDydTVkSq8T0Jj1PbXIGJSCJL4rB2QwyzuoILsEtXRAuyaFGm+CnGTu7iv8Gk5h M2qJhSyLIi4Ksn8carVBJitaQ+T4HF9HdD/FfY7qzk8K5azK1PxKwdpHI9El/bt1TZO8oavv rZBWb9yo1fVUzxfTqT/WzOUdMZckqifmRZZ4ibJN5EcrteanSulzcp8k7uzLmuJ8F6ztkyKL OkWpw+qvAvyxcTslnlMeD09BgchTJuFV2kHwxEX6OLOfh+K8VH1ZbhxIjNgBj7l5GJD7yz9E rH7MNy40i9bz4Hw0sR0GsGR7pn33Y8LjMorQayePx41TYYHkran1JbLZcjZlWNk0tMaaJC2j hfvy2aoweB2t8CnFqnd7X/IOW3cPMd4Gg4tf67uXhKGAVZnDIpyuUiLSbs+2iZfBFH/loBZC WTNU0dnHmehWcf768vb6rMOja92xPcgbeIyCrR4feh1WZPXbWRZbCLMDx/dQOExb1XtU7vum dsjqZ9pkufwtmuJ4VRzB/Wk/2iuWJZtmq5Qly5suDarBWCvFDrz3ZqxyQhlg3FVRD2yQu7Ff 7BzNDP6G94jN6Qynu+gHsXhoJcVi4mlThyH6crtocvvRAfwJ4U8HNxUucgbH4ikT2KWFdATm 8blzZmaRSp65hP0xTkqXVLFjptQelyiTj43SoFznaC1gvjBeIig57Psti+YcbmxO6iMpyJem ygdkQhagplI/BoUwpfMTO1z7inKjC6h7p+OVl520A2H52yz0cjaLDURPPuPx33XB+4i6FrGL 3A6gF2LVQUVeY6NRl3kQoq4jdulHWvJUNbnvWkWnz9TebqcGkktuuwW0tf/VijKdQfQHwMjW V0zzd5nkhh2TUY42EoPPY7fcCbLxy4gEJXPaQ/jNyOIgioinWQCnkvIX1sJzKui7wcViviBe DAMuxZ4wGdYwHcP9Buu9IOFxG5iaKApGSqhgygtACxOeXDV8JN4SA/apns2IXQrgmzpa4Tou oJxNgyk+82o4E2QQVuiFp3sqxqxOLedhRH8VBS+pMLx5PBbMvocXtPGj5qlPW7r0MatSNvJR dvo9IQmn7H40uRFPvAfuxNOwEU/jalklAlHqJYHGEr4vZrgdAMAQ2ITwUXqDR9rcMMS/vyuB /vKdCJpjLIKMhY8IyGUwW9Efz+AjGchgPaMHHcBLGh7EtnFX1ljSkxGA9CykdlaBt6sb4iOd Sj+eiE50u3QMdBHuimoXhCNlSIuU7pzpaTlfzonzOqM8JFLtowkXP0YbIsNnKDjPQsJNuFm5 TnviTSYs2qKsRUw8rAI8S6gAuQZd0zlrlNhpGaWMcOWtQSFXU8rRBeBFLvhBbEbadey8RusB gkXhyGzd4u+skvqApJD07HE4hSHdCPfZ1luO9JZtH/+qr/adp7Z6rDDTYQk9BXC1BdAvMFUb fkp+W85t3MSucwnG3N5V54HcsMB72N0B8hRS6jzgnAk20AF7wIQ+G0neyCAMUyz9citQs/8O 34stsyP3aWWBx6HjT7BjhmuNJZZLWWB3IRa6j4fS6iJPWsfiHnJglWCnwTaLC+ap+6ey4Hdu iF/NG+vvw7cjYwH1T6Q3hrAPdo7n+v3isM+JeHgmsPfcnYn45q2wrpJ8V2MWR4oNwrpZCZs9 esME8jy3yxLeY16edXGQmwZIweZqj0/ke2a8aiybjZ503m49alm6Bx09UaCReQBtYGwNGiRJ 74hw8waui1LlTjLA1TrqjdKAQv3lxBbVZLUHZWQ51ZYxFnfJvXXlpBOVYRCEHs0YbrlNo77e rsgreD5uXeF3tEFLJnCzv/WbBYyziIA+BkbDBwHySRXdzWGXZBtRxX4r7LaE42wA90VKxQHT aetlNKMaUBVAR3B02+ruPnEJDU+LneB+sY4s9Z6W2PneV52Rg5NIcIYe9QNWH0W+tx18mTLm UqgBaEfiAXrKB85PNTnJiwO+TGlYVWRkVOnrGB141s0sE/B2uNjWHrmA2FnJoNvquIG6ZYl8 crVq71xZalZN7lxSyXJ4o58WbpewyN54s9MmuapH7hW4TGoGvtTdJi7V2Et57LEaItylY8y3 o1scBnk4kMTeMCxTBvbSStOR/tfU55u47gJwVXDOsEUWQDVxmBZ1kkiWySbH3npp1MxB/SKS 3yMjXpZJAuYC2HGUxuskSSH6WjKoj8q6TFHHSLo6mfAT7CC0KpNUoDIQmSmt+ffifkRuLQ6F +3HVuJWqEn5u9V6NNOxQ1YBVI2vjgNoxIrfoY/P/kXHUylxjQmRFnbj94iRUF3ZJn5KqgIre atNRBrP1p/tYLYiFN5sYDy3nfbMZfBqDcFUXeAij/xooD+BZHFUgjF42GEJOC7c8cXIYyNVB mzC5OsCUsIYSCCn2XJzBhiFNWosMzxbdv/rTWqn2DeXacOtoFXsmz3vuZuGx5bmaSngCAcTb a8Deyt61uoX2ef0CTiPe3LbpvMGAPYZwnXhomHxJYFe73p2PezWoUyPBgpq0FOeN3S+0Kbob oA1IR90UG7bFPy2ExOK3kFix7/FCp16uTtOpbjJP+Am+i6KjQwAYEoTBruCpCYPpvsRkgzfo YHkaFQ88s2U4ksNWtaHKos3A71IkVR8MewXqMSkxX6Vu8tvlsiOjGW8PmUZBMCxVT1YVLvye ZECO75f1u4WILZeL9Wok37ZWbq5A1H7TYaPc9X7oNMb6bMKfL29vQx8pevzY1016d2WCbrrE Yzxonzobxs3O1Tz5nxNd1bqowCbl8foFbGnBol1yKSZ/fP822aR3MFLPMp58vvzorOYuz2+v kz+uk5fr9fH6+F8TCHxjS9pfn79oE97P8Hzr6eXPV7ciLZ9fzJY88k7J5oK9BaWxOtJYzbaM 6lsd11YtkLzw2rcDhYRdMY6p31mNQzKOq+maxuwHtDb2e5OVcl8QUlmqdriMaju1saYURpvt jlUZKaN7naIaDo2YZfOq3ei52SwheLNT2ob1kQqhc4vPl7+eXv6y3ms4OWcxx32qaBCUZkep VVRReg9kDe1wm3/sDBRCeGQyoHdvp3PVgzV2zeluQEGYyPYcOxbvEiJHwxHDM/+quDlZKp8v 39So+TzZPX+/TtLLD/2SxCyPeobImBpRj1fnDZyeB0ShPnyKbYp1Rkc+82sBNL3ckbXQHKP1 1Byj9dQc79TTLI7dMztvuYf0ZtFwqSFSn3BQWmPyf3n86/rtQ/z98vzrVzDsgCacfL3+z/en r1ejahiWTmuCVwZqcrvq2F+PgwKFoHyIUu2c3LONHu7rO9ZwIemg5iaHcCbRM0Cg9js1PKRM YMvh3nprFWMvlO6YYL4eu3XeidFhEQeq3A0A/2WVZ/ViM5geMd4CHS/aVv2sob8GuhQ2Uq5C f0bWgaoHk5oJX01blllM7WMsROqZiYqD+0QcrO5mSqtCMXPchUJ8P7ODxlqIVk/3yWBhMSi8 UVbKAE/SxI/OaEsvlYaGWRXaPO1Mn0WEkCQrE6oPdoHBazBjsI/ELfCgVKuKkC1Khhkd2BwV KjRR/Wuk4h2s9qvvlDwKQvu9v91dtKEpConySFWowV4lWAxwxliyHA6pUdEtPoads7Ia6Kou RyMZYQ5AMeMnIyQ3fohAspMqxIA5WHsb0CFH5L+XRoQc32+gYP3x+C/UQrG/05cs5rkuwHss KceZUinw719shJqhOD4lZLw+N1Rn1obHOFLIlZlHsQbQaLDAonBS7BEaicRmOjXuxY+F5eyQ scFWuwXLNJyhnrUtnqIWy2gRocI/ctaccEStQXCqgYKy5GV08hX1FmNbfD0AQLVaHCcx+pGl SKqKHUWlZnApcRH32aZIUajGO4h+GPI743coelLr12B70y42R2LKMaFHcSjLRZ7gnxGScSLd CQ7Ezhk1dR+F3G+KHDvYt9tGNsFgO9Z+y3qgGXaLpq9N9VqGe/SEXKKBlCQTaFSRFgu9xZ/F Td2chkU5SHI5rUSx8GuVJruibmM1OZJS8tDBsWDXSmC7xvP7FV/OfEw77/U0wLi7uXCy1Ot8 kqLPtnSV4S4wVopdyu69byOk+nHY+YteRwb7Yn/KRkNqAqCU3pwnB7GptBs2t+TFkVWqHQfN BQcu1MfbQ4ArfSKzFae68faUSlOFy4ntYM2/V5yUdpV80g128qbjvRQcfpktpoP9GNwDgMmX fmRPFpbvWfH/lD1Zc+M4zn/FtU8zVds7PhP7YR5kibbV1hVRcpx+UWXSnrSrO3YqR+1kf/0H 8JBICnT39zCTNgBSPEEQxMG3zBnfoHI3Nr4sOC9yovge33adizoL1gmTVdhayBo1Gym5ZYpv H6/Hh/sf8pJKi+jFxniVzPJCVhqyeGc3QGbSRKVr77oxGTrisbxaUDDqCq8wynbXXWFmOXRx JJ3Z+oSc/gYmqBev+GMCq5UlWZ020qSeA103moeX4/O3wwuMZ6c0tgdTK1xrM++f+ELZh2nV pDujxT4Yk9GmEJnu+hUhbNJTIWPqhwVto4PoZRRiTV58kEaz2eSqJsOGIgGcK+PxtbN3FBDN qd2JFKi5P9LNOt/Srshiq67HwwsaDuFs4dfwJvESTtUi53Bds9u76mtr6+6GakEZ8mC3NEW6 avIl27uwzP3OqmEEiPVAvF5ydzOtmjIDHu4CU3Tc6rS2Fm7Vo+7U0rY2Qvxz5Vep4xucf9m4 hr7mJFYOowOA6siHC8aBsHdwfwTltu11rM5CFIhW3O1Zh8FPeLtgkInG+R5XOrLO2MfegcZ0 XGZaFaaBdBmmR1Mi/BEuqvPW9ItDGGGY424b2LsPeP829qmg1rhNmpS7LRSWDySQmleNgob0 uHxDm1JJXLRcF/0GI9TvKWLQqHHsffKWLcOANrWpb2lX9TT1RKNlKQdxmmoIvqLaVhD4S3pZ UrBmBf9vIz0BnJJ1BbmI/Etd5DT2ajp2vgBMeDq3Pb4F/LYMKNMegZNZ3t2aFLSXSVYgPXGw Zbsw3PS0WzQKOJu1GY2cLyHOTBPVAScE0AwGpoDz2bBfHJ1BHcowYTsMqRYnVF9newJ6ZQay FVA3uq0kNZ1hBcSMUGzClxGckePe/KiY+3w6Jt9fZJ+qyWwxcb7cC0MqoFUYYEjL3rxVSThb jEgnZ1mbG0a1XYezf5yO5NXYjAglixtR553VLd4h//pxPH3/bfS7ELfK9VLgoSnvmNWdsqEc /NbZsRgx1eRIoqye6tctrKl6OT4+UhsJzbPWtJ8U6m8xx0acxJXleBmMRncN3G7iJGHa85Io z6IgBMk/R3MHDlK9YV4lUESASoQTNZVVKFy8zFh6ABI8g+RJIIMp6w0SHdR7dQ2kuGdsR/oD 2T+MaWMfxBUYvA5OwbgkVcVAEQGHVBRuxQHzSKEYAJCVYe4J0yA+HMbUm4FFAzyPlKaxeFnb 7pgITFdXpPcqNL1Z3hXIolWa7G4foMG8EZHSgIqBVFHvXt4wqF6fm6twOb6nJoVeogE86dWr CIQNutkbBU+d+O7Kkufh5fx6/vttsPl4Prx82g0e3w+vb32bJF4Faxk9pa2XJ3FIO6wCA2em F6387d76Wqi0EoK7ljDrb7bLP8fD6fwCGTAgk3LYtUkRpzEPKbdHly7mwa+Q4fr6JTI0fb/g bqnoxGVFrZKnXi1RtZh7HHIURSaqwLQKF78RmSpUC7wKeOVB8Xht2xoo7C7dzocejxJFMh/P Zg2n3xsUyVb+dUIAKZoiyJhxiRI/tTr7z6EDLnPUAvw56z4hEeEGY+HCGcO5L3RLWfHZeDjv 7YUY9sfrmzJ9aLenjOL38HCAa/756fBmQk/3P86PIs6dCqD3cD5BMTvabBBdXw0NjaP83Qh/ Duwe7GfRbbNKXd9fx09fjy8HmZGJrry6noyurCNJgtyUCSpI5/P9A9R8ejj8QrNHZhJF8Xts deN6eqXbHYlWtoEE+cfp7dvh9diLMPj4Aezm4fx8GKiYqJogO7z99/zyXQzAx/8OL/8exE/P h6+ipSHZPJBy2miSyfHx25tRpeZQOo8kT8aL4YjIVisxY+tFtQLYbE69XiDmn+t/dKfT+8fT 4U0uCv/HMRmllffHQajMImqdvcG6GxxOh5fHj4GoGFdjHJodZ9fz2dScGAEwqykPr+cfKE75 5lhGaVSC0ODTQEbX/HE+WeYqsQhbjK/s/TAHLZGMTTPzRnzar/suYCDD3X9/f8Y2iegdr8+H w8M3yyGsYMG2pi4jQlMKMhRjWcPvsgDfL7gwmS3T2IoxrI4sGYa014bg9PXlfDQsRm5jESQy Sq/HZsYKC9zc5Jbdho3cCjNBos02mfhlXKRsbG26sSUVk4ipdVmL1hklG0broDBbt4ZbZLEO MGwdfV2lraz28ysjvnZr+dsxmJCVm4gWAwNew00AJoX0EYlYkiis1R0E8zSf+0Jgr+rPccVr ouIeiUibSXd3U0g7DB9SP/GReDhdAuGYeakReAXZFkHUy3nm7vwoKKwhlRc7uBglOf3SLXKQ 3qa0ZIiG/VVQXmxZzjfxMmiWVVOutnBbuUi1CQrPbQGbEaYFLWbLXsSeshiIq6xI707pnkGs DI258QRrEM9YzTqtacFENqj0WMWp2zS6UgAkYyFNVuxg7j0XC5VXMw2h1zQFr0t50Jf5BGTX qvIEsAs3JYhG7b4jXBh1iEL+fDz9OD98d2WUUAD5+f2FStsYJlteQjNBSDOOIoCyXeVCxc8m ycOtRblMopaym4EqxRGIPb7IGxm0BtbMTwjSqvZ4EmuKyhOjlKnAOMDvqT2HipxlbgXWKkJ6 rtBEuQyadJlTN8U4T9PazX6xxqP6+DAQyEFx/3h4E4GcOwNJdSA/nd8OGJ+9PzMYUb/CbAzG ew4Hvi98s5tSIWQ1z0+vPdkUXWR/4x+vb4enQX4ahN+Oz793mT8jIq9xne3jhpcepSemdCRH shCnwapkN7r36udgfYZvnCzpR6Fkwl4pQ+QZdCjILPcYk6xgJS59fDEmzw6DEl/ZebBj9inS EbQ5t3rbSDeZGJeufw3b+bzr2R6ZhU/7m5O+rrGZJgqFKvmsSMGacGmDt6t4JZA2WOmqWETW pbA2MFTZxzgOdEsy9lxwVMFlGozmxi1gmYZwK5DnaLdcTahKSKqP1c7JSmLtF0ohYVYaFezJ Z5btPvy8HQ1HdnLCNJg7se46zGI2G/Wz5Em4t4SZmVNEsrdT3lfb+YSMwIiYZSBM6v9/t8Lx YmRdrxYLQ7eM2dXm88ZJH49ptyZXnjR/YTGZjqkWZkF9LdXJCtCmfWpi5wMdZkenisRsklE4 nI8MizhfsnuR0RDg64Kuq8szqvopx+/p+QewLvNk+3Z4EgYM3L1j7b7MxaDJE/L4VVGIq3x4 fno6nyx3f70U5XawXeMctN4fT+YWSnl3cexuW5wX+rvuN9X6tgvROLVc7Xj8mIZDrCZ6Cc2G dlYVTA7oeVsH1HRKBX9Pr8YT0xQRlthsZOj2YVlNr8czPTsrtMU/nB4+2jv+//AKGUXczBIg ZRFxON6/nV/+iI6YVeCvd9RjaJri2/3r4VMChIevg+R8fh78BjVgdgL9hVfjC7+iSGi50Xpk vurI316+tL4rc5otSYyHK8XVeiL9jOTie386fj2+fRjN6U6mTeUJ3LyJwtFw6PF1BomI4lY8 vpa8yfg9HuqlE8OQv+H7yNPh/vX9ReZVeT8d34wB2qZ7M2dInO2atKivhphnUxxVpE7FvgQE ieeGEH2GkZ2QWd+CBFba0NIuBkXEFxPyNUugFmaqtuVmdG26LoXpZDyaGz1BwMTKpwWQiSfs MaCursjzYF2MgwLGNBgOzWMXVUWjsXUqmMebZ0AMkqIkBcvPPMAQTN2XQOYb4ltn975WlU5A 5LyoYNSoxhdQ2XiISFOc3E4mI8tyuAr5ZDqiHjgE5tpojuZQQi9mJ9kE0HQ2oZpR89loPjas G3dhlqj0ML+gPAu288W1sciD7XCxMFMiKpEjDdZ2wuNgDWuPZoHGTGAxVsGFC93XJ7RzbZqG k9mYNJFW/AFrkfyhx9UFCpVSGq0258OP46nXZ6qFcRYmcXa5hQa5VNOQMV8v6vqMVgtbkbIu Klp85Hd8xY1j0eLGz3C7OQFzb8VHUypxV4jk/pjO6f2FZN8g+Qw9u0xYx1KnWJGMRsZ6gd8T BHQiHZ9dmetH/rYPBYRNrntz6QSRMaH2iV7NpkNbJX1C7a2j6S5ezv8cn+yjQu/zOEI1Tgy3 v525AfeLWbdxqsPTMx6r9uB1A57sF8MremenxXB4Ze/fO05yEYEYW1e1rKKNYXYpQy98ihWZ 9g7wQ64h6yYOwLCkb+OIk0nJ6aoByXuVIcz7gtoRKC2Lp2Jh0jG3GD2CQZz1Vgu4RpkpuLyi QL8/aTHbFpFCLyZi9GVnaf2X8rAiY9TD8mOVkXWyG+lVameVSMNmFWyZE4LCwMKm3ln2iwi8 LXEZyjSRNibcQOPZTvOAYnM34O9/vQoNRLeW1Xu8sizuBmtzh6atzXiepcLGmpoDk6bmS8sQ ZhmmzRaTjSMCST0VaNWhWx5xbH+X5XwqbHQBTc9qR7cfjX+Fbjae/aS+CrCjsed4EkqNMKD1 t6nt2y0H/vCCjrniiQxuHUcQtPvv9mVgLbxqU2cRXrqSvu9t9yLSiWdZVOYxffok8TLbRXFK a5EyYAtkpO7K4AnwQ74s2SCe12XYmtMYVzChbRXmo44Gttq4m95Fr02r0xbKPZXB0rmg78Wv VTSLaQloDxZ8jsHnzL+Pj+/AwvFVt+dKjTTGhoNfTboumy932Y3GybqOL09CIdwLWMIiQ+6C H01uxslpMyPBBKXiwUhfU/AZplzWBiSMloFloxClMWmhCXDF3Z8sUIhOkEG4iTPWZHnWsFUM zEiG2bceQDCIRRMvV+jYkVFfWOf5OmG9PGnr8/nxx+HCYKhy0Nmwjd6vxg9fHgXXMvWWIbSW NbcYZ0tafRmmWvtqbFlQK0CzD6qq7NE1aGO7h1qSPoqzsC4dWzLATZoVdZACZmo5MyiA5wtT 5wsmhmVheee4iukiBs5u1tTr3v55GRmCCv5yrXyg3nQpBtW44LAYzjfAmIPZAoHUfHlo4SKS fJytLENToyo5DVQj9ZfaUp/NsfOUoKYI4d6hwDIofqMJsDFXe/l1oxaE3NQgqlOPr/SsIti2 D0QIbARaFb2/0M71iturGEPwj50B0rAmH5NRRVo89tjqnMRI/5s04Nskp6PDmHTkol9W7grR EGuAOsFAY8X6EZx4jfNHfr0lxkQAPMiATrhy0Xd4Se2PdCPxAYeVSL8aZHEix4uSwMa9BSJA OLIXS7Rsxy7nW9o9Kr3AfV+Q42iyHV2SZhQCK1SnAfmCg0MY7C3OQPIvfF+zvhoD/8bXSGl/ 2JbO8ipeGfwtcgGxBIjXIKuhgUQQbRTb0qQVADQeFT5/4o68orsnfBAUPW5Mx1pSInybUmKr kllPWjertGp21P1MYkyVPlYQVtaOwDzSKz71LKIao50awxw6bqT5jpVJcOeUVmZkD9+s1Itc Mnjz1iFBkj+Qi0HiN8Au83UZpNYEKaRvtDQ+X37GhDh2aDiBEi6ClhljC/XWapCYrZI9jj7B PeiPaBcJuaEnNsQ8X1xdDZ19/DlPYtI/50vciwATraiRjnL+xyqo/sgq+rsrwSNMNQeUsJjm ziXB39qYNswjVqCb2HRyTeHjHC976Hj2r+PreT6fLT6N/mWqBAQ/6N9MXg/vX8+Dv6kW9/LB CMDWfa0T0F2KYHKvYQzwOy4XvAnE3mAMwFg6Vdj1gRyaRCWjjKe3rMysLDVKVaHvTWlhz5cA XJQhJIUjGm7qNbCSpVm1AomWG6tY/NEng55dNGsWK/SOVyw1MEHUO0QUqCk9FkWr3txpDiyY uy3oapAyrLU48cZpJfyWwTntSKgt9OKoLZl76DN99BoPAT2Bjvk683mlhJsPF6IqHfbgQvPR Pqsbsp/Gowk6MEf6BJFkHG61MkeXW9pZEC2cOApbHC2KIhI9+VARCUexigDhH4Qvjt+IhCZf KB8tiRNJo/pFynrpiQkOwp7HZTQEZkpOEL+pA74xZ1xD5MGvrw6dZsxCR3HpM9xqCSMMFlg0 GFGa9M5xCaU/I/VJkwDNKEJPXuq2gO9K0hK4k9Iiki+exB8dATVx3Ze/GMxMA6ciQCTGiRQJ FPoELF0yOxJKN9xlsE4ZiDjqYMcKJu0ps3e2bRpnsJydS0Xa26Udeyh8O/gm2097+x2AV/7K yktfKtAX15MW5I7vfMVqL7vc5312JGA+QUOGQuxOUVbd5uXWZOyUEUdi8DD4oc9o62Q20Ppo b+Bot75m4q4ndPZFm+iazp9uEc1n1EOZQ2I9zTq42c+L+/tBp/R0SIzXHwcz9mImXszUi5nZ 82Rgri50YPGzDiwm/uKLn4/+YuIf/cX0p1+fXzsdBgkWV10z9/R2NDa9R1zUyEYFPIxjuv6R 22qNIC2dDPzEV5DmqyaFby1q/JXdeg2+pnuw8HRsQtcy8gy0+byJ8G0ez5vSXRICSh9KiE6D ELkjGRdI40MGp1voViwxcP+tS+rcaUnKPKhiOzBYi7sr4yQhn3w0yTpgSRzaHRVwuBlv7fFC cBxijKOoTx9ndVy5C6DtvJPOvkdU1eU25lRYXqSoq9Vc3wu3h5fT4cfg2/3D9+PpsbvlCB0I +oqukmDNXavh55fj6e374P70dfD16fD62A+fLgMaNEr+0meqdGpDKSthO5a0R8DUfFnMK106 Yo5bb6fGV6HX6VeK8Pz0DFe3T2/Hp8MAbvoP319FWx8k/MVoblejUOShdpY6IjP0yhBaESDE PK5BZSciUBRpzSupd6Ju/ZjkVlTy52g4nrYCSlXGBXAQfF5OrStHyYJIVAtIykQlq0UsVhG8 zbyQitDMt5n5ptqPJ7SBytGY1tFWS0IQ2FEWxwtbGlThxpLwHJwcFk9EYBGs/zYAqUv2vsiF Csq8zpvwXoNzfEi7ZcFWGP2CuOoYy8TiGk36a8sa8F7MktZo5/B0fvkYRIe/3h8frQUvBo3t K0zcYqsFZT2IF6m7vF+SyhzeL6oQMHDJyhNuyybE8OD+aoSzACVh2WQokvkrKcNarAB6d1mk 8poAp1+NM/TT76qdoLf2qF3mSb3UpNb4CoRPzbYJdkzPZMrSBFZCv1Ma420b1B5um5pLDYVT ekc/ACskppAj7RfUC4GwmQe+YXNridzE6w39imx0S7QNdZWrJL/t7UMaKYqLrYVdp3fwBmMU aHtcXO8DNFV9f5ascHN/ejQNd/JwWxcqQbcZzQNNJfpIi1ujC3BqEhauu9FPiZtdkNTszyFV MSYw+uWKXeJ+xSoH+QZNK6qA0xH9ZVlUSeQFaRhk4ttvWEjcpXldmZ/mMFGR91Ilsep50ITp FyqLTq5olkUt/3bWHn5/y1gRZ/STj3Y5cradNP1C4+eWRw5+e1UeW6//Hjy9vx3+OcA/Dm8P //nPf37vH6FlBadgxfZkhDu1NqFVtneW2kyyXL8zt7cSB7wivy0CMq+dpMRqG8GnLb3qjniA QQAmULfyyENpHJOL9ctCFliHRkmYXWFXCB15gyJuDwFqdMTnYYdhXEjxmmfoby15zFgNuA4E kmCLkiV7+wL/ddEK7QY72W7UERgLxAVuyem1JpHizSr2haCQNGHJMC9o7BgjS+exsCYPbjG7 gHS05RIIMlTBUFgjpRNe4OOAoOsJIM48dEIZEJs4WoGDRJ5nRMSxm04n7AzBUqwCkERQ00VN nh7HhpWlMLT9LAUyw+ZGvn20COvpME54QiYoQZQUg/T+sUqBtLXFPCg3ta/TggrfTyXr89Os cC+TLbDaTQitLkW3/FH75OT4SmDes/COduVus0jpg7aMMao1RykzL+4kWyVYkYeQEs+5ZLC6 gb0EVximVKBK52TXMfl+gl2XQbH5JZpV4bATKX+oS9RK8w8/srmNMcghM1mP/JBEp0I6BIIQ 0/7ZJPjMBStKNlRsM7eSUBWUtThMD43d943TRPnV0D5ESuSxrjuh8IQU9Bbzhz8V7jEOHQv7 g9ij19agHsL+5LqD6p1T33R2pmXlDYg1K4XxSg69etWqk6PPe6PKs0Dm/TEfBW2UvnPBKJFx 9ZYYHXmjY9mgfZz7BCrgmImtQh6sCngiebXksFYoQvPE6/VWp47S5hvW0zTUvGQXfGINPCm0 ePbJhS3S8Ts9u6p31Px59lBXh5rJKoBTp/AfOhggy3fodJuvWQJT3KRBSe8cA21owEwCXzus rckw+nAaFOJRwF1iWJkc7V6IRXnSv5+E6qY6vL7Js77j6duoou9rIqg2yidwCfBYMS07XgxS mK8D5RLNWCTDdGxryx1GxNdYei3J27KnbiknXk1bMdDYltj+DdtHdVp0R7mA4pmTocYmKWTI LRO5BWxlxwcQcKFDo4OcCHwJh99GRBvx7YldHDGRDm80WUwxDp+45lJ8oI4TuNjkIS8tsQGL oMzru/LIKdumTo/as9WBL4uVAzGMaa0KhOLQ8jtmqW+5ygEOKtjIKitzd2EL0O/fq5aQF/B1 ZL1/4m/6bU4KDsCkaJUARq1R4ri40NbGJYMFZXKntKKW1tGAiyCx5JdFPJwK11XjmqMQNBek 8zLH9Hb+ax0TeQbrBP24zHS9Sv7f6z2lwXkNq0+qiRxaNDFJahE/Vm8/Gb2ichJi4zy0fK9/ CmPIRlwLTXVXsGa4nw//r7Cr7Ykch8F/Bd0vYAaGm/uwH9I2ncnSN/oCM/OlmmW5BemA1QC6 5d9fnKSpU7t7EhLCj9s0wXVtx7FHP32KyWQMVoWYk6clj5rP3gVSFAMKw7GriTj4IzcD7gb+ ZC4t+MYVY+ITekT95FMnw0TZIWQyk+1QMal+4z54pU18qF+o3XhVzEUY7Ehay9d8DB9Ezke5 4PUDvamV0v8wpxXX96jqtGoy6tX5Ve6QQfNw/3GCs2pkh8K87GOMwLZ4BVtRA6BuERiN7Og7 1jVg0gCd3TEwGUeOYRI+N7kwUEu0McegjGLgLB2SAT9QgkSv4X5uG55BIGLCPoJernLPV0H3 PKLScpKXrOE58OyFKZSPvpSbabqPJ/aN2hQC3Gf+3cg5BaOVZqD3bvMeElO0b9h1/HkO9x6M qy1irGRC9MsfPvVgp11q44Kj76xo9kU86QBpaTvcFMSSqht0wBtkwB9Ij0+fP99fz+6hTenr 6ezx4Z+fuPOCZdbm70Z/M5GngslLSpciYYmUVdvIsWklOI/Qi7YCa2FEpKx14GB5Gsvo9wem WAUJSsw00QOinCx7u4YTGQcONW6nN3R0OpJLtma5+0Q1ZnfDRPAI1yZdLNd5lxGg6DKeSIeH V/imk50kiPmVMPPPLTK/BqJrt1ohMZfOGEXDdeBVWauTzjXrpMNAOfviJx/vj3CS+v74/vD9 TL7cg8zDMbF/n94fz8Tb2+v9k4GS4/uRyH4c53QghhZvhf5Znldltl9cnK8IQyNv1C2hSn2R /l75k5+RqfIBHTrf6KNEMR22pesQM3IgcREkR8vqO2b9Kz3M/PrvmHvrzwlUufdlO45vj3Mz yEU8VUX9Nhd0XjuY7DN5uNtc0AqsydMP7ZXRwer4YsmsmCHbU4XM9A08P38D6zXKuHdKg+3i PFEplRhWZ83KSp5ckmXKkxWlKS0+UOVQcW9SnSeLsBEixU0tEObC5YorbDPiF7j56CDhW7Gg Yq9fitUVR14tOM3ZburFTHOdQR1Vq7BolP2SmQ5CVOSEpAKraf1qTR8K6IXyojEBiy5SzL3q mP6v9Nf6LlXNdhYYSyUR+RO5zDK2X4jngISO+eublsu1QvAVc1nC+pQOTM1v+t5vxUEkVAxE 1ghOPCydXflBezJaU1JbQn8OK1nQB3L0vmnk0gwzfYRWCmbu2kZN+dSkkIE00ZjAdmI+3Qeq cdjaTHSxU9gI+J2QHyb1vKc4n6nswPUl92ZNsp8JuB1rHR9fvr8+nxUfz98eTkPJqSdchMu/ E43SPhJnZSV1BDGOouMRVulbxGpKsmaAxWziN+Igt/yq2lbW4PFB8IYzrHrOtB0A+zRUZDze OCNw/rE8K6zSVCA9yNrMMDg5nTtgd8yQ2tzPodmpDRxZR/uTAasuyhxP00UhG8K0gW9M2RHc rc7/6mNZw3YfJJz1ZrcSmSDVddz86bPqeNTGCGUQDQMPTHumlbRHEMxREBiBK7wTQzmuv409 92a6m7w9/XixxVtMZl2wF2uzv7FvXAe7KBRvkOtlgjnXt8jcGyiwBwLtxXkkne44OXpfl10b nvQZUBN9xtcBUYtGbIJ6ClJEOhxWBBjCTJMLrMOXMgPkjWKoEOGoZSZ2NpocS6zogeE2nY4x bDwlqm73WWmTAiE6C884M2/b3mMEXRKTOgi3HTzef3xJzMXmY4wkxcwy57PE7Jp0plcaHwvP NtO9YgdEqhC1C1ymX3zBoW+n4+nz7PT68f70gg3aWqjkKnCsI9XWEvpPBHsLY1R2xLkAv1kG gayOYZGbti7iat+ndZlPHEHMksliBi1k22uZw9ttAwRlISBgDYFs1VIcenSoMihoMUCzZKRv hu1saI1hz/dUmQr9tlj7V1pDB6TFVchBTWo9Ttv1uN0GWOUBhzbS6X6Go2v1JqP9OlSqCOGT 2R2LqO/mP93AEbH7ERpDVbCgbSLxTuKgY4IRYbuGEH4QLdeNx0tPkZQ5O2VtCvgDbSHVnrsK 6WB1wAcHrJMJ1dksaA6HkrkzULk7G/uC5b9k+XcHIE//Nm79lGaK/FSUV4mrS0IUdc7R2m2X RwSAXBx63yj+ikXHUWezaIe59ZuDCpK0PBBpYMki2QEHMRGAT6IF/OUMHa3E8K6alJqwUVot IY+vzMqgjCumwl3xCxrFyK2F9IVGgsxytP46r1h6lLPktEF0/fEoY6W1pFGntUC2HKgYrY5k PiXBvksfqCmTXIOX1NZj8OFftHg3SBcXWXjA1Gs2v3dq5C01B07hEdEA2QH6JyBCWSfhSZAk 4azHvFL2GKP7u1QJ7OQrrezR3NMS3B23xYRLBZR83oDhX/9aT+6w/hX2u2kgDy1jFZmfewOr J1TBLAuYJf2wWfYfLpsmYGCQAQA= --ew6BAiZeqk4r7MaW-- From eflorac@intellique.com Mon Jan 4 05:26:58 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 334C37F37 for ; Mon, 4 Jan 2016 05:26:58 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1FA138F8033 for ; Mon, 4 Jan 2016 03:26:57 -0800 (PST) X-ASG-Debug-ID: 1451906815-04cb6c3032420500001-NocioJ Received: from mail1.g1.pair.com (mail1.g1.pair.com [66.39.3.162]) by cuda.sgi.com with ESMTP id GrB2B7GLSvga7p6J (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 03:26:55 -0800 (PST) X-Barracuda-Envelope-From: eflorac@intellique.com X-Barracuda-Apparent-Source-IP: 66.39.3.162 Received: from localhost (localhost [127.0.0.1]) by mail1.g1.pair.com (Postfix) with SMTP id C7BA02CA96; Mon, 4 Jan 2016 06:26:54 -0500 (EST) Received: from harpe.intellique.com (labo.djinux.com [82.225.196.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail1.g1.pair.com (Postfix) with ESMTPSA id DBE1C2CAB6; Mon, 4 Jan 2016 06:26:53 -0500 (EST) Date: Mon, 4 Jan 2016 12:26:56 +0100 From: Emmanuel Florac To: "zhao.mingyue@h3c.com" Cc: "'xfs@oss.sgi.com'" , Chenyehua Subject: Re: Facing problem when attempting to recover the file in the tgt shared directory without the remount operation Message-ID: <20160104122656.4091b5a2@harpe.intellique.com> X-ASG-Orig-Subj: Re: Facing problem when attempting to recover the file in the tgt shared directory without the remount operation In-Reply-To: <1CB94550540AE44E9CAAE8CD5AF40BF846D24DC6@H3CMLB14-EX.srv.huawei-3com.com> References: <1CB94550540AE44E9CAAE8CD5AF40BF846D24DC6@H3CMLB14-EX.srv.huawei-3com.com> Organization: Intellique X-Mailer: Claws Mail 3.13.1 (GTK+ 2.24.20; i486-slackware-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Barracuda-Connect: mail1.g1.pair.com[66.39.3.162] X-Barracuda-Start-Time: 1451906815 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25816 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Le Sun, 6 Dec 2015 06:08:44 +0000 "zhao.mingyue@h3c.com" =C3=A9crivait: > My question: > Are there any methods to show the updated file information in the > shared directory without using the remount operation, thanks! Using XFS, probably not. Maybe using some form of cluster filesystem (like GFS2, OCFS2, etc) and some SCSI magic but I doubt it. I'm pretty sure you must use CEPH as a filesystem for these sort of things to work as you intend.=20 --=20 ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | | +33 1 78 94 84 02 ------------------------------------------------------------------------ From andrea.gelmini@gmail.com Mon Jan 4 08:12:55 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 1B9D37F37 for ; Mon, 4 Jan 2016 08:12:55 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id B46AF304053 for ; Mon, 4 Jan 2016 06:12:51 -0800 (PST) X-ASG-Debug-ID: 1451916764-04bdf0264e44b2b0001-NocioJ Received: from mail-wm0-f44.google.com (mail-wm0-f44.google.com [74.125.82.44]) by cuda.sgi.com with ESMTP id tFzlMwXhZE5kiiZT (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 04 Jan 2016 06:12:44 -0800 (PST) X-Barracuda-Envelope-From: andrea.gelmini@gmail.com X-Barracuda-Apparent-Source-IP: 74.125.82.44 Received: by mail-wm0-f44.google.com with SMTP id f206so141748268wmf.0 for ; Mon, 04 Jan 2016 06:12:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=bxKk3BpTPTHsZV9oogycKlpVPNS3r7D1mcwgB2C7pbw=; b=AU02oHNKray8FZg2ZTHa06hDR9tuA9GwnuIMR1n3C7oxIAQfH9cFFXwHuW8i7LEPSd J+FuFESDG7rEKEXX7dKuY/Ge7U2xSQnJ0DugRa8306K9+l+PBy53cf7O3jIVVSehuhzQ hRNwbQxcVVTHd8yXFJjm3LEl+SJcQhVxUcOyIKCS7NOtr/E8SgynbDcaRj3tE6hxTnS7 jTu5SsI9glsPweek7EWQYIZRpeGRdCtp96wl8CLTRdT6I5txoHF8S3cJkoaoe3gFnO1I l4H5QATN2ALFJbs6vTWdy08vJz7Wjs1rKXVP725hiybT4/jdC/QU+Ktj75LtrqnJ6UsM N0sg== X-Received: by 10.28.187.67 with SMTP id l64mr77822037wmf.39.1451916763677; Mon, 04 Jan 2016 06:12:43 -0800 (PST) Received: from glen ([195.206.31.238]) by smtp.gmail.com with ESMTPSA id o132sm73180728wmb.7.2016.01.04.06.12.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Jan 2016 06:12:41 -0800 (PST) Sender: Andrea Gelmini Date: Mon, 4 Jan 2016 15:12:39 +0100 From: Andrea Gelmini To: Dave Chinner Cc: linux-kernel@vger.kernel.org, xfs@oss.sgi.com Subject: Re: BUG: KASAN: use-after-free in xfs_iflush_cluster+0x9d7/0xaf0 Message-ID: <20160104141239.GA7054@glen> X-ASG-Orig-Subj: Re: BUG: KASAN: use-after-free in xfs_iflush_cluster+0x9d7/0xaf0 References: <20151214180048.GA15690@glen> <20151214195422.GM26718@dastard> <20151214201526.GA25152@glen> <20151214212220.GO26718@dastard> <20151215091145.GA19282@glen> <20160103204758.GW19802@dastard> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="jho1yZJdad60DJr+" Content-Disposition: inline In-Reply-To: <20160103204758.GW19802@dastard> User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mail-wm0-f44.google.com[74.125.82.44] X-Barracuda-Start-Time: 1451916764 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Barracuda-BRTS-Status: 1 X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-Spam-Score: 2.60 X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0249, DKIM_SIGNED, DKIM_VERIFIED, MARKETING_SUBJECT X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25819 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.60 MARKETING_SUBJECT Subject contains popular marketing words 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 2.00 BSF_SC0_MV0249 Custom rule MV0249 --jho1yZJdad60DJr+ Content-Type: multipart/mixed; boundary="OgqxwSJOaUobr8KG" Content-Disposition: inline --OgqxwSJOaUobr8KG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 04, 2016 at 07:47:58AM +1100, Dave Chinner wrote: > > Maybe, in the meanwhile, you can do something with my files. You can fi= nd 'em here: > > http://mail.gelma.net/xfs_kasan >=20 > Any update on this problem, Andrea? Hi Dave, and thanks a lot for your interest. So, to make long story short. Recompiled kernel with debug info and all the rest. Run it. Then started a flood of this kind: Dec 15 12:12:24 glen kernel: [ 5326.351571] BUG: KASAN: use-after-free in _= _check_element+0x1e0/0x200 at addr ffff88004a201ff5 Dec 15 12:12:24 glen kernel: [ 5326.351574] Read of size 1 by task kworker/= u8:2/10221 Dec 15 12:12:24 glen kernel: [ 5326.351578] page:ffffea0001288040 count:1 m= apcount:0 mapping: (null) index:0x0 Dec 15 12:12:24 glen kernel: [ 5326.351580] flags: 0x4000000000000000() Dec 15 12:12:24 glen kernel: [ 5326.351583] page dumped because: kasan: bad= access detected Dec 15 12:12:24 glen kernel: [ 5326.351587] CPU: 1 PID: 10221 Comm: kworker= /u8:2 Tainted: G B 4.4.0-rc5KASan #1 Dec 15 12:12:24 glen kernel: [ 5326.351590] Hardware name: LENOVO 2356LRG/2= 356LRG, BIOS G7ETA4WW (2.64 ) 10/08/2015 Dec 15 12:12:24 glen kernel: [ 5326.351594] Workqueue: kcryptd kcryptd_crypt Dec 15 12:12:24 glen kernel: [ 5326.351596] ffff88004a201ff5 ffff8801086bf= a10 ffffffff819d2e3a 00000000ffffff6b Dec 15 12:12:24 glen kernel: [ 5326.351601] ffff8801086bfa98 ffffffff813f4= b61 0000000000000010 dffffc0000000000 Dec 15 12:12:24 glen kernel: [ 5326.351606] 0000000000000046 ffffed0009440= 3fe 00000000813f42cd 0000000000000000 Dec 15 12:12:24 glen kernel: [ 5326.351610] Call Trace: Dec 15 12:12:24 glen kernel: [ 5326.351614] [] dump_stac= k+0x4e/0x84 Dec 15 12:12:24 glen kernel: [ 5326.351619] [] kasan_rep= ort_error+0x511/0x540 Dec 15 12:12:24 glen kernel: [ 5326.351623] [] __asan_re= port_load1_noabort+0x3e/0x40 Dec 15 12:12:24 glen kernel: [ 5326.351628] [] ? __check= _element+0x1e0/0x200 Dec 15 12:12:24 glen kernel: [ 5326.351632] [] __check_e= lement+0x1e0/0x200 Dec 15 12:12:24 glen kernel: [ 5326.351636] [] remove_el= ement+0x206/0x430 Dec 15 12:12:24 glen kernel: [ 5326.351640] [] mempool_a= lloc+0x155/0x2a0 Dec 15 12:12:24 glen kernel: [ 5326.351644] [] ? memset+= 0x28/0x30 Dec 15 12:12:24 glen kernel: [ 5326.351648] [] ? remove_= element+0x430/0x430 Dec 15 12:12:24 glen kernel: [ 5326.351652] [] ? bvec_al= loc+0x250/0x250 Dec 15 12:12:24 glen kernel: [ 5326.351656] [] ? set_tsc= _mode+0x60/0x60 Dec 15 12:12:24 glen kernel: [ 5326.351661] [] kcryptd_c= rypt+0x5dd/0xea0 Dec 15 12:12:24 glen kernel: [ 5326.351667] [] process_o= ne_work+0x48a/0x1160 Dec 15 12:12:24 glen kernel: [ 5326.351671] [] worker_th= read+0xd4/0x1170 Dec 15 12:12:24 glen kernel: [ 5326.351676] [] ? process= _one_work+0x1160/0x1160 Dec 15 12:12:24 glen kernel: [ 5326.351681] [] kthread+0= x1c0/0x260 Dec 15 12:12:24 glen kernel: [ 5326.351686] [] ? kthread= _worker_fn+0x560/0x560 Dec 15 12:12:24 glen kernel: [ 5326.351691] [] ? kthread= _worker_fn+0x560/0x560 Dec 15 12:12:24 glen kernel: [ 5326.351696] [] ret_from_= fork+0x3f/0x70 Dec 15 12:12:24 glen kernel: [ 5326.351700] [] ? kthread= _worker_fn+0x560/0x560 Dec 15 12:12:24 glen kernel: [ 5326.351703] Memory state around the buggy a= ddress: Dec 15 12:12:24 glen kernel: [ 5326.351707] ffff88004a201e80: ff ff ff ff = ff ff ff ff ff ff ff ff ff ff ff ff Dec 15 12:12:24 glen kernel: [ 5326.351711] ffff88004a201f00: ff ff ff ff = ff ff ff ff ff ff ff ff ff ff ff ff Dec 15 12:12:24 glen kernel: [ 5326.351715] >ffff88004a201f80: ff ff ff ff = ff ff ff ff ff ff ff ff ff ff ff ff Dec 15 12:12:24 glen kernel: [ 5326.351717] = ^ Dec 15 12:12:24 glen kernel: [ 5326.351721] ffff88004a202000: 00 00 00 00 = 00 00 00 00 00 00 00 00 00 00 00 00 Dec 15 12:12:24 glen kernel: [ 5326.351725] ffff88004a202080: 00 00 00 00 = 00 00 00 00 00 00 00 00 00 00 00 00 Dec 15 12:12:24 glen kernel: [ 5326.351727] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D Dec 15 12:12:24 glen kernel: [ 5326.351730] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D Everytime it happened (usually when writing) I had a little stall of the= system. After a few hours it was impossible to work this way, so I got back to an Ubuntu vanilla kernel. = (I guess it's related to my luks partition). Anyway, now I compile rc8 and try it again. In attachment you can find my .config. If you please can give it a look and tell me if it's good for you, about= info you could need after. Thanks again, Andrea --OgqxwSJOaUobr8KG Content-Type: application/gzip Content-Disposition: attachment; filename="config.gz" Content-Transfer-Encoding: base64 H4sICGZ9ilYCA2NvbmZpZwCMPEtz5DbO9/yKLmcPu4fJjD0er7/a8oGiqG6mJVERqXa3L6qO 3ZO44kfWbWcz//4DQD1IinKyh800AFIkCIB40d9/9/2Cvb0+P+5f72/3Dw/fFr8cng4v+9fD 3eLr/cPhP4tULUplFiKV5gcgzu+f3v78+OflxeL8h/MfPn14ub1crA8vT4eHBX9++nr/yxsM vn9++u7777gqM7lsL84Taa6+9T+3lxcA8n6PP2SpTd1wI1XZpoKrVNQjshJ11oqNKI0GQiPy tim5qsVIoRpTNabNVF0wc3VyePh6cf4B1vrh4vykp2E1X8Hcmf15dbJ/uf0V9/PxlpZ/xH/D Atu7w1cLGUbmiq9TUbW6qSpVO1vShvG1qRkXU9yKbUSbMyNKvjMqMrgomvFHKUTapgVrC1bh tEYEOL0kdC7KpVmNuKUoRS15KzVD/BSRNMsosK0FLE7CGiuFPK31lGx1LeRy5SyZWFiwnd1c xdss5SO2vtaiaLd8tWRp2rJ8qWppVsV0Xs5ymdSwRziOnO2C+VdMt7xqaIHbGI7xFXBWlsB0 eSMCjmthmgolhuZgtWABI3uUKBL4lclam5avmnI9Q1expYiT2RXJRNQlI8GtlNYyyUVAohtd iTKdQ1+z0rSrBr5SFXDOK1ZHKYh5LCdKkycjyY0CTsDZfz5zhjWgtTR4shaSQt2qysgC2JeC RgEvZbmco0wFiguygeWgCSPZmmlW4oJTdd2qLAPWX3368+4r/O/20/A/73Ss9prtxCy0uqhC zlrpa3mWs6W+OvnwFe3Uh+P+j8Pdh5e7+4UPOIaAuz8DwG0IuAx+/1/w+/RTCDg9ifOoqWqV CEeFMrltBavzHfxuC+EoQbU0DIQANHkjcn113sMHywSircGGfXy4//nj4/Pd28Ph+PEfTckK gSohmBYffwgMFPzHmk/lqrGsf2qvVe1IbNLIPIVzF63Y2lVoa5LAaH+/WNIF8LA4Hl7ffh/N OAiHaUW5gS3j2gqw6Z/Phi/XINTw/aKSINgnJzDNsCaCtUZos7g/Lp6eX3Fmx6ayfANmBxQH zhZkaf90EkOCLBsVKPkaVA7kaHkjqzgmAcxZHJXfuGbSxWxv5kaoOYRznflrGtjgLshlQ0iA y3oPv715f7R6H30eOQKQONbkYHuUNiheVyf/fHp+OvxrOAZ9zRz+6p3eyIpPAPhfbnJHwpUG 6S9+akQj4tDJECtGoCeq3rXMwLXqGK5sxcrUNZuNFnCBBNYuOCLST0Lgt8ByBeRxKJha49lM AppaiF5JQKkWx7efj9+Or4fHUUmGexh0jmxB5IoGlF6p6ykGbwcwwEgRH8ZXrqAjJFUFk2UM BjcS3BOw+910rkLL+Ec6xDjtIEHOxHQRROQIScDz4nCXmBVcuKl3meiK1Vr4n+XoUWnVwBjL 8VSF149LkjLD4oM34Emk6EjkDO/nHc8jjCeDt5kc+OCN4HzWu3wX2Sa1YimHD71PVgCrWPpj E6UrFF4WqfXvSKDM/ePh5RiTKSP5uoXLHYTGmapU7eoGrWuhvIMCILgsUqWSR87IjpJWjYYx Fpo1eT43xNEquIzhAtLETrpmaPngIn00++Nvi1fYx2L/dLc4vu5fj4v97e3z29Pr/dMvwYbI LeNcNaWxcjKsZiNrE6CRcVHDhjJH5zrSRraQ6BS1kQswLkDocDHEtJvPI9IwvUYfXPsg660G ExFiG4FJ5W+TuFXzZqGnJ12BgSkq0wLaZQj8hIsaTjV2geqAmBaNQyK0OBFsKM8jcoPGjQgo jokyu18H2ELRJkrFlkOuRZvI8sy5IeS6C7YeQwgxfyTMFc6QgX2Umbk6vXThKBEF27r4M8/U N+D8WGcGAoPU6uKcM1s2EEQlLGcln7q85GcnaI9gmqbEUAw87TbLGz3rR4NHe3p26VinZa2a SrsctqBZ89mhMziHGzfq7eCVTPUEmIqN5I6pAx6B9+3QIXtxaIeZTABgX2YpzCFxddcANzJf Bj97t2DY3wgFNxEPIo3s0hKtPVe4W8skoul3DmH/BEhn7PgGTNZtFMMzsNjgOFzLlALmUZqN OyAmyvm6+5o7zIZBI27+JK8h7BUJIyEchWAl+JpCbTSi4KiLmMkFLwzuS+5GEg1KrevVg8fl /objrT0Anrr7uxTG+221BL3qyR7hjsww4AN7xOGKijGn9iN25AcIIwUJdermc+A3K2A2e1Nj aNDPkAaOOwACfx0gvpsOANc7J7wKfp977OZDhIu+CWUKYkY08ENZCQGHLFXqnoA1MjI9vfB8 XxgIVpGLimJ/sp7BmIrral23Vc4MZqYcrlWZu9hZGx98tABvXeJ5O+sA1Snwupm4N/YsR7B7 yLj0DhP56hrAeld4NqyHtfEhVQ2C7QWajtUQedb66bp5xkB0S/6Io+GNEU4aSFTK26VclizP HMkj98QFkFvmAoD7EXatPNPEpCNeLN1ILfoxHl/wNCiKymLKUnHZ/tTIeq09G5qwupZ0iqMI YDYqjWqcFaUxBdr7El3ytTq8fH1+edw/3R4W4o/DE/heDLwwjt4XOJaOk+FNEdg1QsJm2k1B WZ7IOjaFHd3fEq5JyZvETuTYX4j/GVyntWcGdc6SmB7CBMF0NvdWG8l8oTaioICg3YC3m0lO qTfv4stk7t3tysIEeCHDOnpYtyXSrCoX2zn+D3NMZm3LQlqhc3IBYTrox6aoIEZJRO4lj8Eh hhB9B2oKOoKJEk+07CRRj4zWRPl1UEhQATToHJ3ZufWLDFglcatN6Y8IHBuUAfTlwGsFJ9lL AKxrYcKN0e0D8KYuIWAwcCAuH2zOD1QfXSkYGgbpEz5ZaOQ73SHE4d00ECu1WcwO08YIsVJq HSAxq86MqXUEDr+NXDaqiUSGGk4U46ku5o24knCT7uBGxwiULDRl6IKv1GIJNrVMbSmhO5KW VTKg43l03ZUM1Y5wq2vQO8GsvxHgCrmFsx/RmtYQ3nZ/faxuUQZUIIaNTNzbmbrbcNoUVey0 YgrTVQA2VuU0y4AtRYX1hnCGTsAtx8m1Ddlpx9lE4wwuVc1Msr4zbeiJ2eRGn8eM0CqIikb6 2Fa14EjQgjHxfO85uF0ktwxEpRKY93UMbwTl+kU+Eo63jHlGU0I4zSZnfzEbsFxFQ3G7atAn sTWki2vPRhN6JgEQ2pNp6D+j7yXmnkRXXYlIgRUorLzArRcVQ60y06awLMflLVTa5GBt0FKi a4MecWSJYgvGGb1JTPsZNglJsZRFw0H7VTEtZHFV7TrbAlGoo0coUeD2dFWlz5Mbt8Mz3l2M NrfP1ebDz/vj4W7xm/Ubfn95/nr/YPMzw4EiWZc0jpzhsG4i6++/wL2kxfdG0hrRlcBTiF77 LJFl5nryeBeDjLg2npxCjX7J1afgELwwlEA2EwmKzaJRqKVpSsSHR9oNjSA7/daTEbrmQ5nC 9SZ7tFxGVqjRq8VPzS8QSQK+Ohi9YqdRt8CnOTs7/ztUXy7+BtXny/P3Vws0X07PYhxAiVld nRx/3QPByeQDKLV13HPpNY1SVDlc3U3lRp5emqcPRBO9jAK9gsEYtRqxhGg9EtCC5iljfEeS gv8ipdIzGf66V69q//J6j90PC/Pt94PrdKMHSwEiBBGs5G6GgkFQVI4Us4iWNwUr2TxeCK22 82jJ9TySpdk72EpdQ2Qq+DxFLTWX7sflNrYlpbPoTgswVFGEYbWMIQrGo2CdKh1DYKI3lXod +BKFLGGhukkiQ7SCa05qqkxH0A2MvIZ7JTZtnhaxIQgO/DS9jG4P4pA6zkHdRGVlzeoiykGR RT+ARbuLyxjGkewJE1HXuvtlKISphb799YD1aTfKlMrml0ql3JJVB03hNsOPOAnhDsMztz6U /dTl6Dr0iOqLlpbeiWItGD/7Tqmzn+7k9ut/T8Lse+/ourk05teemC5PPUkobR9KBdES3hvz iWdmFEYCdXHt5nPgprODQZPUdem6ebabZgZJ5zGDG8I4KrWmREYls5FkHhMOrq/jQyfwMX1t DeLL8+3heHx+WbyCQaS60NfD/vXtxTWOfeeKI51uNID6lwkGQYiwmVH3wBG5PYMAiEdvL0QX FRnpWTx4ouBAYatPl4eKCA7S2aaVvNKTBbBiHNzl5GP1UTR+RSI9oSdIaBRwzkFiump4xmTe uMkzqy8gTQaYjl0hXQOX4z/uIMzaSA3++LIR2svy1wy9bHcjPWy2QDEQRORl67rn8ANrxZ98 SLVZbQofBI7AMvFB2kbDlBn32Cy6wnObxVyE7gOOPdwU4aYR1E8R3fhfFxIH0qC4VCqqh9nU 4ejZrC/jLlWl49JaYHo03h5SoOGIrGioKFeNLz8kC5jB7hrlbMnswiXJT+dxRnN/vi4cDjo3 sZK98SF4pRZNQQFcBhd7vru6OHcJ6Ay4yQtde5UrFHmrZFMwaNgUyCGaYI0r8JUwYQ6OYKJo sAUTwgpnVyllK8aYB+470MiiaKIHwFkOFLspxdAkI5VXvSLCdiXyyk/xFmwL5i1W9aFGQ6cV zGq4Lkyo9AX35He4t8q4kesJNioHyYZNRDP3ROMmUO0g0gb/hClbgkmnQESw1D0B1gIucGOr L0mt1qCoqCoYD+tAcNyEXQcI5aEHe/LQAzFK1iuIe2PT/Ci4GS0viTjEoxBNtptJdmRTXF7M XAKnF5NmZqGrTG5DxegbTTrZk361XV6uY4GO5LXCpmfXQe1AISdGhMeLEYzxPxmBjE0YC5rn sQLkTqYjiG7/arUDdyVN69aELdy2hRoTj1E0Kbmsga3tMsF8i2dusfQ+l1SwTUJgRrsKcuhF DejOkQvxIsePdjcmhI2TtCHVw9doZlvMQDnamudiCcLQXaLYUdQIbGM97O8+fZq2sb77qXGd ELA1LIYJs7F2HoyAhatuDkO2EP4WIobawP8VQ9U9RkG1nNYuqGqNWgoU/nfmmi4viLM9cEv3 jDfMiggEhaxOI8O7/UqMsULdoKm767TFtBBNH78V7TQrZao86rPoKgf/qDI2cPGsK63Qsqwn QzfO+AulYIcHQaZc1kG5yZ2szzLG6N5Rqt4LbrHP+ep0iLnA/rr6az0V8DfcYgTeNNP8/FoX 04CJJMX2pqX11cWXL58v4pHKxO8Mud5hIlx/P+caw4KEXrOd9huSImSFrcHGEtW5YCW5M24w okrjF684hRnjrQ78nvi7U2zU50QsPizQV/8eztD/2E2llKP8N0njWOobXQSN9H1LOJxQFfTE 9cRULXzHDaRe877UNt/Jn4m69gsj1Knh3VBY0yJMn6WPfNYGHpug8kAJE1ReNKl8N+lSCvBh NEUtMm0CwRG2C9QNNVbMXMdoL2t8YKOuHS+zMLXbvAS/IK6A/UmvscqH99rb3yynM2QkklgE QX9tQmxzRWYaSGs4XXyuxPw0BqFt6j8I/wq/uWoMQMEPjzBDZNLP+WjjltIRQhW/SV0p6B09 /fQpXmm+ac++zKI++6O86ZwwcHVzdereoeTMrmrsB/V75beCz/SxAwYrYfHrgNdMr6iWGPNw wfRKdFxBhWp8oXLq3+i1QL/W+FfoUJWh6sMcnPoSh5cv54GrgEdHT4noC9q/DGhFVKicrqi7 1rrYbtglXo0YiBYuQfxobLj7l2RdxWiT6vjzgU6PB5e2pBdjsdccAWEXd7mrn8w1F7X0eUjY bh4vCshs1+apmTbx2Dd3sMQqaON3bmnfdRjyVc//O7wsHvdP+18Oj4enV8pYMV7JxfPvmNd3 slZd4c3NzNr3XZMOzv5tGIbieY4dgd5F5zwdi4lt0epcCMeY95AuSzaGeQV18xEuPtE1W4sg ceNCu1c9p6MEetile1UV3hRB/gpX0pU5BpS7TGxx6dn1zpanY1NakH0oEB8Y9H30ED/yB6jX SHH9k61wOO0x0wCDu80Y+KsXZTIZelKYso8k6emaLSTjkCrlwSRdK5P9Pr341M4b0WHnREv8 WIo6Fj7RXJ2g+KMw5sm0/cLcyFpsWrUBr0CmIvZAEWnA8nbOkBMpIoKFe0qYgRhqF0IbY/zy FgA38EEVzJexkCr1M+oIovRSLeDkvN6lfsNCY9pTppNdDMgALqtCBuvw7W/8C2y5BF+Aof/j D+6yC+GqG20U6IsGs5WFT+VCiqhZtIulNlsrecO5zZPPZXTtXjgKjjKhfxYm0uzqwKFmspzA e1ZJ5WeArFgmocRYHyi68QJCU5XOLTZZRtQCXNkGTc0KYk2qyKky383NAP8ybgfgAJzl0qi9 rBJhg9MA95ufIuQj5XIldAwOJyHYhLWEErL8cbJqi8EXye8esNhCTLt0R1dYglIVCG78ztHk TPbvehbZy+G/b4en22+L4+3+wXvK02uhn5wlvVyqzfjMJESioRtlwgPDZkDE3E6XAd3HsDg1 dpDhM38bHI9eQ5QWTR9WFuKORmwI9qZRC/7fH6LKFKIu/0Z7fwS+14XwZc6b8ljp7zdK0e8y wlhvSzP4fv0zaHexQDJIx9dQOhZ3L/d/eMXgMXipgr9pQKaGU/mCxMjLSvZ2/n0M/DcJJkRG lOq6XV8Gw4q0kzNRaojnNtjy4VGASyJSuJBtcaGWpfLx1bkt/YD32fPg+Ov+5XA39Q/96bDv xGGavHs4+FoUvsnrYcT7nKVp9PL2qApReoECeSAYeeiRjqumir/MsQfULYMWWhwen1++LX4n n/i4/wPO9uh2icl/Q9xHI9zpiCJ5O/YcWfwT7Nri8Hr7w7+cfhgu3WcAsssX+7Ci6J6NeVCv 1EhD6Vmq9oAC/ScvRTYa19EMAgi8mjoea3YDutJBvKtfMO15wh1k4g+P8N6r9D+DuPeN1Eg2 2oGZNWFuxecQMGIm6Uhs1nICmHkFTOyeKw9zvIltlqqLsfwn++RoYFbi0avacol9VpSqE67x WRn/sTAO995iIkC61UcEVHWwmYppmfqgvlnHRnwgnL8+H18Xt89Pry/PDw8g6xMD1v1ZFb8h G61JmfjswXR3PEnSOUj4uejbSpg6lWq2SQGf8SQTNRN/Hm7fXvc/Pxzoj/Us6AXI63HxcSEe 3x72gUXCHsvCYLfp9HlGFIXNTdiA3d9Z2Ke6Eiz1emO6oZrXsjKTPD5TTfSZqB1USM2d60ah J+e2fkj2+cwroLlwnNq3zlv3b58MbUr+TrEO2Vyc25xL4Rdbuj9kEI6kpxkh0FayNyQWqvIi ak5dPt7Lu8kyAJbLcg02W+uuQua805PlsrZvMuiYy8Pr/55ffsO7dXLNwIW+Fl5rB/4Ge8qc 57bYq+Z3rgUE28x9foW/6A/3+ATkQbh+JAF1k4DtyiXfxYUXaWwlJGaw7BT4BEYbr0GREHAa mKJ8dHkDp+GuoQO98wnp8V9W9pWP/4AfoEPGgurVtYfLZNKCNyDa4Dl4P1mF7z8oDeDhbOXb UjD3zzUNOAjdEuUmFgcMz5n2LBdgqrIKf7fpinshfwemJGH8ybwlqFkdbUJA6atkwHNZLdEW gCZuQ0RrmrIEczqlj00R+dsJyEPacrCLwuXewN/Y+eLj6EIX7ebU544FnrkKXoL+qbWc6Fu1 MdJfb5PGt5apZgIY2eCePyLZyun2QoDQVQAJZZyAJP3h5wkTBVolwyKsrZph8miW4v0JEiHC sXmtgm35xsOui1cxMLLRByMh/HMZ6YQeUInkkQG8SdzcwgC/FtpcK5VGhqzgXzGwnoHvkpxF 4BuxZDoCx7wzFQymqLyKzuMGFAN4J1w5GcAyhytCydiHUx7fAE8dTg89sh3fJl2yNSznnS7Z ftark9u3n+9vT9yvFekX7XrmoEEX/q/OjmJPQxbDtN2LD0fnAWVfa+Nd0KYsnbHmFxO9upgq 1v9T9mU9juPIun/FmIeLGeA02paXtC/QD9Rim2VtKcq2sl6E7Oqc6cLUhqrsM9P//jJISorg ouzbQFdaX4RIimswGIzYuSMLMih4vbMgjttcvxocf7sA+uYI3L0xBHezYxBTVe2ZW+1aLqKf Q6YyhQjeuki/Ixf1AS3BYENZW7RPdWYRnUIDSKZ5Xb/h9RDyvcZwi8OGxwXABd9I0J3kQbKi 1vgSAf9ncOJZsOZCp/66rc1aeyQn0cNL9flJaULlyl/U/lNwyTpezsPvazC4YZo43Jkwbnh6 ylDKg+7g6/cXEAeltP8qNyoBj5tTypMg6ZCMBEoWM0rSLm5m6Nq11wyD1jsOZLj+X5bKnICg yruL1hR6mXur1TDJbVNMBZsaanyGqfpgxNcymMt2CUWI0DNgGxKmqn4ToKt+aiXdqgvSlZzh 8QSPKVS0QgSRtIFX5Cqc8zYLVC8DrR8LEI92miPlvI7WARJvkgBlEgD9dNlzlOVHKQIMoixC BarrYFkFK0NfL3jopfbo1qc7ajA89ocA2VgDz4yYU36Vsj/tUCWjCZZq/5ileMIwcKDvTCRf T5ioTg8Ckqd7AHz0sNrtDphdv4C1vpfl5l8rAT3VIyV3WcLuibxkFgQX0ls+Dy5huWfHlBaO R85pQ7EiaxlFSLHkc6PWMYqpC470LeNIiYDWpNiaY3daACYerQyhdihk9YvWmY/Va9TwecKc ShoutPvqvxvrWi1DndI5/Vh8+Pr5149fXn5bGAelviWoa/X87U1VjbYZslBFJHm+Pn//18tr KKuWNSfYdylXkJ/nWJSZmbgWb3ANosE81/xXIK5hvZpnfKPoqUjqeY5z/gb97UKAXl8fAc+y 5Vn6BgPp7h6GmaKU7I2aKMHt0Rt1UR7fLEJ5HIaOR35DbJVaGGZluYkb1FGZeOMD5ia9iavN sjcY7NnRx9OQsz8fy1/qnXL7VwjxJo/coYi2UZM/Gb+fn18//D4zVbTgsDVNG7UF+TzDBN6z 5ujGd9ssS34VbbCHGx4pmYKaeJ6nLOOnNhNvcumtyZtc1org55ppqonJFrc9XPV1lm5JEx6G 7PZ2Vc/MWZohS8p5uph/H1bft+stLIFNLPPt49FJuywNK0/ZGzy3+d6SR+18LiYGwCzLm/VR sOQN+ht9TG/PibrDw1UeQ3vJkaUS88NZX2+e49CnEPMs5ydBJUAPz6V9c+55vFZEQnQ55md/ w5OxvHiDI3lr7rFkdg9DpU6IZlmUVcZbHEot9wZXAxcX5lhmVw/DIqWOWYbrOsJaJyMlkmcw yf4l2u4sNOYgL/S8dvhHChkRlGhp9TQN5h1fgganA4jS5tIDWjhVoJZZ4GXfFyiCfGP2xTnC HC38HZLIj0TsMFQIbuC0G54R1eOgVsbnQjcR9kCuqHKrAg0mwD+wdlEhp9rF6/fnLz++ff3+ Ch6PXr9++Ppp8enr82+LX58/PX/5AEerP/74BnR06q+S0zvjNqFHbyNBbqj9BKaXLC8tSGBn P64G+Z/oc34MPjfs4jaNXYd3F8oTh8mFjpWNVLejk1LsvgiYk2V6thHhIngvoaHycZAf1WeL c/jLZXcbm36P3nn+9u3Txw9KRbr4/eXTN/dNoo0w+R6T1mmKzCgzTNr/9y9oYo9wmtIwpa3e hLRlNmnQYlg47FnBAb85PnGow77eIaRwsT6QCWuoJsbLCxpamxEwhzFQBK0cCnyOj6ZAUIJc s4alvo8ForcO5H7KnxxoDuG6Dk8C2i9bsaootUdTRTWfsnNInNe2OkrjZkNz9uNE6MWEph4V /x5q2+Y2wc8+7jKpFogQXd2aJpPNN3ljapgAg70ttwpjb3mHTytPeShFs1PjoUQ9FTlsRd26 atjdhuTO99oQc1CNy17vb1cWaiFJmD7FzBT/u/v/nSt24bliF5grdrNzxS40V+y8c8XONyns AmN955sYdr4JYDczAezemAB2/glgF5gAyCnvLjREd6ExigjZle82ARq0V4AECpAA6ZwHCFBu bWEXYChChfR1R0xuHYJHVWgogZSCk8nujdlk5x/eO89Y3IUG484zJe3enJMwR1mPuuQ0S768 vP6FMSkZld/8o1wcWAx2iFXjG376vJb2RHOG654lGIKrmtfxL6ykhqPgY5/Fdv81NEmAg7Vr m3lJrdOghEgqFVH2y6hfeymsqPAuD1OwkIBwHoJ3XtzSWyAK3U4hgrNrRzTR+rO/5awMfUaT 1fmTl5iGKgzK1vtJ7pqHixdKkOitEW5ptOW6Q3V02kYrmSyydKeXwCJJePrD6e14f6XeA7Zo xjMC5ls7ZskqJ+OX/vz84d+Wd9nh5XDyQ/mVQ2//Rs9ReyhEx9xCanwA+zQ+9VX8Lin9V+o0 z2AspewD4YAkASMn3+2oEDv4YyVO7kOMAV/Lit/KH7t+sqgmuzH2BZoX5EOvDcQmA/M0vG9u wSX1Z/wkZy3ZdejmlrVIVyUfpEzGSRYDBu7QeOK9hA8sOTmLB6SoK0aRuIl2+42duEZl6+v5 zHcBnShN4cm9s6pQHOdJAdx+L8O6VTKDnMgsV7hTnjNo+UluPQQ4paSOZDUVpiEzRbtOrNVQ EMg3ouKWc/MKHRBPWH+6NbWPuS8IQS9suILNUqet9nxVizf68oGo4Dqcknw07ue8PYDlF2yj eOtZXecZhXmdprX12Gdlgp2adNEWFYjV6GJJfa6IvmqXV/caz/UG6Mtz4gVluUTrp4DERw+E MPVc1X4ClUgxpahinhNpB1OhPYhOFROvpJKyLIOm3m58WF/m5ocKM8PhE/D1ZMRpK4wRyalI ORHZeeq5WF/xVdP+4x8vf7zINeBn4yiW3CA13H0SPzpJ9Oc29oBHkbgomZ0GsG545aLqyMKT W2MdZStQHGMf6Hm9zR5zDxofXfDkzSoVzmmLwuXfzPNxadN4vu3R/83JubpkLvzo+5BEeTNz 4ONjmOJppbPnu2vuKcNgo+hyg2OOwfDx0/OPHx//afR4tPskuWXOLwFH0WPgNuFlmnUuQc17 Gxc/3l1Mnz9MVzU1pK7meT1TabJre6ryFbfaUxqJ7jyFAT9HDuo56NZVYB2Qj0lY52h9VlA/ KRNmQhZMgX4RKbFv5hhcnYR7KeTcBuHWFmkiKB99PkLCSp56KbwW1mGX+mxGbOrAQgiMI+HA 0Coq4BA0AS/32o4ydhMoeOOMZKZ0Iq0L2sYsugiZbaikYMHtylXoJfazJ7Ydk0Lp9mZAnV6h EpjMCYgQDlXHvU5axrHPcdyGNEG1lJYQVEZUEMIZh3FoC6b84CMxacT6GEfRQHiKL98hvEy8 cGEuGU1ek1BSATG4qrPyJu4cxsJnD0gVxJhw60hVk3eyMsNX8m96dUXT3a1QTituRcI91PGi IHHoNqD0ak1R2zMhIP1JVJTHFSoUKnutZeN/Fo3dI/RnSbEt0CfyNahitN07upvb4CuSzVFF WyWOXWs3uqUyz29wBDZEcO7HKeEVwoOKp54Gh4sfcxr17R23JgOYFI3SgV65XLy+/Hh15JX6 0kI8GjKK2sTdfJ5Z0TDrVu/oThi9Lx+oHgyAOCkocLqPiyErF+nL/3788LJI7avKwHlzUhe5 AxEzCAASlidwegYXS/A2BGh5RqKPSuQdK9/LXR4r11ZO75hy+uYD4R61nzCGACTUrBDKUwAB 64xdwiiOUQH45cYgypTLn3cumLjNkhhuX/EMzU6l8KSSPDwsPZBbIRpG+Y0NLmq++Aix/P75 /OHFavAiqaPtqsPsVxEH2aFeJd2qbJECGFmdzsNp6tTBVRs46B62eA4KEZX0DIQ8bA8u3rXD SO9dz0ZZoutjle8pGw/z8WX9hkzGvKGmCw0Y9OHnlKlwG2w8X4d0nXvWik+7S1dBxnPBSCx2 ST0C3jQ0bHrqcTCqs/nyz+/gOeQnZZTwmx7Ok+cBxSN441LGpNv2SQoK492g9OuXf316cc0Y 0kop3idfkIIPGPLK1HLxJBy8zS7gaNCBK16sIynT2gS4W6AXJotQsJ0c5jZ64o3c+7rMsjuv IpcdXFrHWX6BIPTuB0TLpZsUOC6FkCkOLlL2/j34+3QIh+1hQlXNHmeaQfbsoVdOsUilKJrl UjTCcrZIKHDnZVyBu0AMiiKBHmqxspxT4JYLG+GMAkUiKBBjHTOcF2Qplspkzz3SkTFCfUsi I8l3y6ymiUlA5ujEqBtI+nzdQz3zlKZ0FoSO+618nNQPk1zXjlEzfX4m2lH/Z0Z3/OmPl9ev X19/D7YonEsoX+vkCxOrZlpKf0wY/Y6Exy2Z9RCoUvvTR2hwFOKBIFK8o9folTWtD+vPGzsB BccJNjlCBNae1xcvJc+98PrOm8xLsTzUk9wLLw61Zn9uUkTLdefUWy0FBRc9eqr4dsYrMpzL NLfcAXqnVnVNYOTO6V0vdpQSZqN05ZNe2mDBXUXTXRjaIoIrieZKLrJChebkluaA9MRH/T1T F2Zw7SsI7jFakKifHCZ+w5GVTqC2WxHpXikIV8o7ZxHyuD68CGt0lsuNTtPL3UsJs+w8P3hM DlURSlQflpDNwEQcBrKbulY3sxyySeO5LBJYkScHmzb5TmqcwKA4JS/lPB4q0UJkLk+17D94 mrRoCVGfWMT2Qk9WRnKoBo0aduUoZlcqUGOTkMhppkUS8AMv2oaElfBQ+3NLA5h4WG7ndqZY wDo6oJ/Nc/CQ9bfPH7/8eP3+8qn//fVvDmORibPnfbpLGeGp43jSEYOLc8vjOn1b+Vab+0DR ssH4tlNR5qcYlXcOtsifyaNJNYdB9Mt+2hdfOJ519bP1WQbkZY3vHRtUB2clRgqGcqpttfCh tp9VcAHFRrb8hzo8chk/UhmeH2eZnSuYCqTbg6w+G395U7oGA28SUhoJ5jCwQegyv/qpPBLz P9nj+InDSQgBS7yEGADiXrkgXYgBPdvvinOaJ5Ni4fn74vjx5RPEZv/8+Y8vg2nq3yXrP4xQ gm9HyQTqcrte0zTtpQmwtjk+HB6WzMqdFxSASWyFd+gAHtPaAXoeWXUlC7LZeCCXU8WhUQGx /bDnDbJKDwjtGhPq1LqCvYm67SbaaCX/Mj/qpiK3dU6H0FiI19NXutrTqzToSWV9vDfl1gtS 7vxulHb02HhSc+m9oa0t0tGAX768fP/4wcCLyt7zXpXvgymGlQ/ulWerv/0NHei2RY1n2wHp CxNKajKQaOFmee63hpAjXWUjNzFaQQrh6JG+73hX0X1xwUZWuUHU0YvRrreTk/zIgQo8pqPj kbsBu7wMcrOvXbl7Sq4CfoPWw3VoB+4R7wGahVpO/FMpv9FzeZshuzWZXwiDnf0UDNDLguLI zahfMBc4d9Qaz2kRyk5ES62faX81mMCeFEcMu+A2YFFgBfCQInYMCR74xJlBQJP4ejxajZeV iV7f/fKsikxNA1KNDkyduRhOSaxYJyqglO0ipGhTvHjJRyVUe2MbtxAXIlUh0SDiKtL8Y5K2 HoOABTqQ0k8rmjxJor+Wyg0nazO/M0r3DZiSA16zgRnHhLVKyJqHEVY1d/0hp5JC+wNYsC+/ LVq4eaOdUS7y5z/JXhtSiPOL7HFWsvozXahvkBRzbMmKYT/1zR37LSP05pjS14U4pglWxFAy FKACP48EUeF+rJYe4+ZCoD51EuJ0roYVPzdV8fPx0/OP3xcffv/4zdVCqGo/cprfuyzNkuGY AeEncKRqnz6YFNSRV6WC6IQ6IAypmJUXKVWk7blfkZ5sU6NZ6obI0C59H+yQdiF2f5VzHYV6 rfx4bn2MwiK7kArdBDNU5P1cLqD5IGfVY+UXqWhTF5dLH3PRa8utPie7igVUFsBioa2RtM/m 52/fkA9u8AirO9jzBwjEa/WvqqhlsYeAU3QGU3dlC0//1rBxFhKoFBEn/anrrJIW6cOucz6A J2cXzEQcOWBy2S83Lq9I4giipuELZYDLrcbryyeK5ZvN8mSVi5zgqOFUQyjLFHuFUQVVHrJv TV9WFgXOa5yWykdXBkPjiJdP//wJ/Aw/K6cpkimohFSpFsl2u3LqX6E9bGl5F+yyhiuoL5As ENJ6qDXy7kjo7w3XTkr58emtVPqqra2pIdrW+6XVVlLK3lp9XORO3dVnB5L/2xgcTbRVCzGd YMe9WR52FjVrIDyToq6ivbOaRHql1iLyxx///qn68lMCoyV0uqq+uEpO2C5eO1qQcn/xy2rj ou0vG9IppcjbZ0liV/qAwxFAoKpL4px/fClOzh4UkoqxUY+q6cJxtDa+kGZSauBBgjtKMDFt wzSRNOYu+0mPhuV/j8fVcr9c7Z1XjJbCrZlKTfrg9QN2ATMV1PNUeMoipdbKV00pF5eqTM68 9mY7kfX6OrrqmyuB56VUWbgt53OI41YNuNm0ZefbeL4jYcfMA8M/RAEwUtwj7ZF05oJvl1Yu Up5x+58BzQyhpwva4wYOs3OwK3kgW86KPRxRB5V5ginA7GnzWlbr4v/ov9GiTorFZx2xwDub KjZa/EcVXdQjTwmICWNP8gZUuqWN8jwoBXJrzbzG1iCRQH/PUWhia5JSDHEWG+uTaGnT4FCZ bKoGAniK8+U2RFod67k6hsLtouhUdQKCqx11ykC+3XlJY4+V9ago1Q7NHaG3dk/q5Vs0rJbc Chizoel8T0N9ec1zePCueAPT0TcuByJYhwgBCwmv1xEWTt6ThQWe+vz9Rkk4xnsmySmpH/uE CyHXvtnSpCw57JYzJboWyq7XeTGp7maennk5r7CpOUZVeFcdt3tv09X5QuV/N21ism+F514r 7XUQDD5boDJO3TTFxQNWwsfZ7V2QNAwCzeetdj6aWu5JZFtMTRlyhZ+kUpoEe64kveHANxg2 WgVwKjDpownDXdmF+C87qPBc9DIJhL3Qm15P2Autgge6r2ec5zq41XwjLLpurtluxWjoUHz8 8cFVe0C0HDnTgc+OdX5bRjhSeLqNtl2f1lXrBanuBxOIgjS9FsWT0upMdjlx0TNR4DsdrGyx 6C9OEP8j2eCbS8dCH5BS6KHr0MkYT8RhHYnNEmFZmeSVuDZSVgQtWYIvWZ7rnuc4Rl2disN+ GbEcX+8VeXRYLtc2EiH5d6jHVlK2Ww8hPq8e9gH8wYOrkhzwsfi5SHbrLbLhSsVqt49wfcC0 9LBdRY4BagxKIywaxkW93KMbaPqZNqnBSGuCIYOxfj0KdtjgT5JSRyurV4qn9bo34X2m79Kr +zDGIupXRj/LviK5WNNHK1WFOg5KJhfxwjV90rgchhHqJRO4dUATkdiGC9bt9g8u+2GddDsP 2nUbBCfxw2o59MtpClFo8CRrosphIK6FVuAMsk/78t/nHwsOR6R/QJzSH0MYqsnNzSe5x1z8 Jgf0x2/wc6qUFrQPbmeC0W3aVmXB4F708+JYn9jinx+/f/6PTH/x29f/fFFuc7TDzylRBqY9 DNQade41bR6t9r7IPfmi4IlSseoNFjJq0yGOeaLimw/b5oQfvdxAwIxT2meI6BNO/FyJ1n0p ef7+28xLENuHFMlXHE+qX799/wq6l6/fF+L1+fVlUUzhZf+eVKL4h2efCflVePqTAvT9MbOf RzFe7nQbKZs2WQLrzdN0+pwlZ7KjSrpcxSz2W2tIoj44gVBHQZYsO4fiJPKUxvFI3dhgIhF8 0Hs4IxaI4OYeh1rmqRysbYNmW+DCN2PBorFgFjK4aiY3ciH1MQif9w6u0OE9+8lkTBXYlHTx +ue3l8Xf5bj69/8sXp+/vfzPIkl/kqMeBTcbJRosUZwbjbUuVgmMjm83vsVcNBBEIq28weCG PE6efLGSQH3kuORZuNJaMHKeofC8Op2IXbhCBVjtMvFUJqS22mEa+mE1LWyiPI3ZHxMvzNW/ PopgIojnPBbM/4LdSQCF2cAEUyakpvbmkFd3bbcxnS0onETk0JA6YYHQXXYaSXeK15rJQ9l4 KXHZRUFCJ2uwwhePs8hiHXrQ+t538j81rqyEzjU2R1eQ5D50eK80oG4FMzNBEowlnnwYTx5I ogaAcyZws9WYI8hp+zIwyO0Y3IfOcvbUF+KXLURLn0zxDJNeNXUMOp9QTtgKJi7TbDnlczKm JmA4R4KEDR9wsD/g4PmA6eLcwPHmFxz+2hccZr7g8Je+4LDpyLVtAwVFEj153nTDW1MqoDP+ HBBTK0ubZ204/Wth9++0hn1EZXci0MTJsWXDTVKIxp7rZM4RVudIMU+tK2V2P+F4kiMBG0JP ION5XHUeii03jgR3mEhJbu1FI6geZT52Itpq/NYcPXJTvR7FOUm9INXnEoKjGTSTg5RAa3vq uQq5LuCLtno2hwOBuiLdzsiC9Y1OS7AZ1u84W2E5jWNjLfWI5zj3qT+WTllESdWOI2iGjzcC iBEpuvXqsLLrL2Mt80CyZ59OWWqc8P/p0kEYydShGwRUED4WaF2ZjECKDV031xb2piYgJ837 lLZnd9W025bXztJa8pZXLsiIbZiurTazVwDxVGzXyV5OF1GQoiJmatW3FBxU8KJfViHeIeaU 5+snrrF+dpsQBzEXMZ/eOJXRGG/oDmdPbXIU/Ki6eC+HnF0xjznrjyRUWQFY5K6YwOlbu/P6 6PZOAN/uncn6sP2vPftBPRweNhZ8Tx9WB7tIOkgtxerCt17XxX6JtSV66B/ptytwNCAlIs05 ywWvrPGqy3C2xfhz36QscdhACyPuLpwVHl6WX+0RWolU93hqHDvSrnnqQVO1dKotMnRd2kqK IRR4XQmayHkNA6c/2t6tSQOaQhMMUm/lsN8d1lM1uQDofV2lqYXVxeglNhnD2f5Y/Ofj6+8y wy8/ieNx8eX5VW42p7uBSEBXORHz1RHyrQpAk3WarHZRZ78CsqAvLcFzrItR0PE47h1k+T7Y Bf/wx4/Xr58XKVhpuYWuU7lzgM0fzedR0HZWGXVWznGhd406b4n4C6DYkKoDKprjgKoq9eJm ZVfaACiHuMjcGnEQYSO3u4Vcc7tmb5w5SCun3lG5W//VD6xVC+IMNFKkNtK0WCLQWCurxgXr /e6hs1ApLO82DvhUU28gCpWrA9kOK1BKJOud30xopD908/QuKt9gWPtOv1TXb/fRam0VVIH2 N71TZs+lhRaskbvI3ELLrE08KC/fMew9Q6Ni/7BZbS20ylPVPS1Uim9kmChUjp5oGTktA4Oq yu3mhrvsRODWKDabUwjRNWgkk5/aQDw9+23Z+3d4Wa2HAfAnQdpKnHlsl75t+DHP7MKTgaAQ c9VyHAi8+unrl09/2oPBGgGqey6pnKwbzlO9uimWnkq3q+u9uR5PLLD/+fzp06/PH/69+Hnx 6eVfzx88Z9j1uA6QOWw4WKf15exHUueQtcfDuUhBPM1YQyAQBZYOsnIRl2mz3RFsCoKMUSXz kWK6IVlidbhGdGH6GnZ4u2kYjH4r7PduPPgslFlJy0u3ltKC2GIXAfUh5ghlp7I5YvEIEA4W CFzg6UHCddbIUdCCAXyq9xtODsYgVnnoGW4w+XNVh77YYBpcTtbijA/tJNieuTIjvXEprZXE PwgkYprBQmR9eEDhRZM8YySIR6ostMgzuNSpKAt4RQXTe1GTSAKSQmVVCbzPGlq9nn6H0R67 7CIEQetGKWwIom9BkFqV214dnXyCwDKmpZDtYcZ8oTKewbF+h6Bh5DhTbjC4ZRsN2JHnGa8o VtPtBkBQi2gRATOAWPUelZeVZGJrM20uEdcTNlncX4VlJ6APRLIsW6zWh83i78eP31/u8v9/ uOr/I28ydb/4s430FREmR1gWIvLAxOfDhFaCBI8vYNDA2mKuVwi/UxtzIQKpibgVLZzeyoXF hg4fON9G54SPVylevbfdcB3RLM5tt3NtxgoXMfGCPUE1CUNTXcu0qWJeBjnkzqQKZgDuIW4Z dBXbz9jEA3dkYpaDdRy5hUq9UAHQUtfwlMFypWS7TwLxSG6kqtyL9elTyQq8GqroJHluxfyV CJxztI38QS4HtrFzQbG9lngFkI/9TTV5UwnRe/WzN2LuYSw6aLTFnEa7lendGnKTkjXgjtO7 xIArVE9/Na5f+BEdAv/mugfiR+XPgV7AhGMZkbNb5lzmVJSnMgnd5zwLbqVl6wBu6nyFfL6G 6PykMRo/6GYOasciaSZd9ZkcZMo1EYg6rg/ijz9ev3/89Y/Xl98WQu5yPvy+YN8//P7x9eXD 6x/fscU19TosYCe332e7ruvCvokxl4rF2te1736w7AiZ8j8zTR2pfQ9Xn+f166RCxg+3qiFK t/apPldOl9FvspTVLXZ8ZAAQDZojmU3xW6cMz1lZu1qvOj9n3mYVEo3kKmzpVDXSy+2tnBH4 qSp77tNxaBuBVgQKhIVU+bBfrVbUcKqGIWH7EBzebuioH3FogUpgdUoeYRE6X9GnjD5i86U8 UD9XKbGgg0X93Jfxfk+u2cKkwVK4kEauosfeRPWcjrtEjK/eygfYmahbeCLLM+ze0NBgTZqj IyApYJ7Fh9Blhx1BEa26auA15e2sx140vEL+JjSo53YM0ixb+zVaH1B5+G3mrbiE3fgVVVt7 lmsfRFiWHRS7PsT4LYDH+KpIzh+vPDR6jY4TWS8ZpWe7wofxA9avTh7WtYd148Oon0WEKxWr h3A7+kstBTtU5oy0c9L1WYJ9VqaWx2iUUJolAX+AA4MxxRiSyiMkwQpZ40ZumFSsBlMa/fm0 Mym1Z0gdF2cR7czquT/fCzyLGdTqkzjZ9+pagJfUMSQ1iIgsd92JfAg8mw2bOnOAO8Dz33NG mZ7rlT2HGC7L205G+DLqg0894sAYp5g82HUDUIp980iAxP7qSALyKbMe7XrVoJ2NAmlGBoot iOS+Id8in5zcAKOJAEJHB0A42WOxWl6sx5nuwffRtkO97l3hX9YGNd805d+oDCAutMPAc/gQ HogwWRMdmbg8RTSJpyiYBC6bLBgrK/QVRd5teuzb0gC0NgfQqh0F0w2rgvRJE2bbumwa6jOa nri7nBIjSjn8QTwhrmUuYr/fRNi7236/XdnPMtEcY+/lS91sHhWdGuSaHu3f7Za4DQZMb+yD d+gkWxdtJJ9/lBdPDVqo4Wm1xAEGBoSOomPG8tIvrJRMyl8FSnMAkKe6/Xof+YuzXx+WeDRE l2A1lTe5WOIyVU2SpZkVsXbgri4cy6LnyvLdXTPwiytr68TLDPu6kALUGRXgKQMPC0ful5T1 Se70+mPO1uSs9jGnso5+tsUSg5IhYTBrRBjUmvcecys4MBhB0HyxV2n54J9aQI2gfPb55icI LttmaM3dr9YHbKIGz21VOUBfKzlm7MgDLHe+Wd/eufDHhxjY9qvoYL+uPDc2xvjJ5yNtv9od vJ/YwFTHhP8LU9SUzW658ffZBlwqN94UBCuknIFtGtXyE+qkIsse/elw0uoiOUTL9cqfBpm3 uTgQawsuVgf8TA7YwZ9Ni498AUhSsJctKWr115HRseHEBSsEqs6s5gk1BJHkw2pFLMYGTN8m k/vTi9daAbg2gQlFtGo6RWVtCwhuY0VfUZh7BJ3eAZ/uFlr8ambyiauy0ecXRvFUVjWcdflq qs3O1xZNV+bZ93mEjfdJLdcuRjypOj7azZs3PHeCd8/mzHHcnBGyTF4AB898CVE7o4Tv/D1R SOjn/r4lzT2i6yVZ1QweX4XxSeLVkSAuXrp8Lhcrkb/CY5riu0HZkdopKiC0LRCXI7nJK5fp mofc18dUSq7PT+B2zNgDFJwvJBK8oc/k7Fu2srYlE1YZ7JfrzsKKlAJGJKRgKvetCVwqw+Aj LM0UysGZIwbkTp+lVjHM0T8FQfsCAZkTQXEYoBSJk0KvurpGUDgjrZRwuJXRrA3uH2yQJ3V+ tXI36xQFpVDCK/DnSAvayo0MNi4AN7hZu1quVtYHaDnQquBayjWbvQfcPbhvV/r+NoaPvMt0 SxJtHvhMjxnxpg8odbinoCpRijQKQl1fS06CdpviS+n0cNiSQ/u6Jg99LKA9LFCOkJwEdAfQ 9r4LWFHXFpc6AKN7dQlXJLoOAOS1luZf0ShckKy+IUAgQKgmWuQ4CJfIcWwmoI0XcjO8CAIB Aue0FqYOJeDXDnkuE7HW32rtOb2ylrA2ociF3ck6BFidnZi4Wq82bb5f4Rt9ExhRUE73D3ss cQIo/ycKi6GYrNvvVw9diHDoVw975lKTNFF6Zy+lz3AsIUwoEw/hfJV1wMN0IBQx91DS4rDD RoMDLprDw3LpxfdeXM5OD9uu81MOXsop30VLT82UMFnsPZnAtBS7cJGIh/3aw9/IVU0M18g8 VSKusbBbFHxhFNvdOqL9kZXRQ7SkmHZybvE1hRyX146iWS2qMtrv9xS+JNHqsHQ7/nt2bezO q8rc7aP1atk73R2IF5YX3FObj3Jput+x/AKUM442MrDyst2uOqs3QEXZkdqUO/X67JRD8Kxp WO/w3vKdr9MkZymBY3GGyOdyTcuaFi4jSAEKXAJavnYpMahIcTlJJ2Ic2yLIGRS7FFe6ewqx 9mGXbJedSWUyRWu4KLYbTwHga9+nq8g3zHQPvRPxkqXFnjgJVlMUHg8aeHCAkGvd8516F7nn e+wvuz07/mYUxnS0SnSieZ4xpT1vLxbz9jLPTo83JHS4yKKSQkjEUzSJxm1SZZ3rIlpRbWZP Puwc25DjgFjDolVSjv4rQC6jNsPq1afS72RSk+/V3U7VdiJsvvXMwEuoOvUmLq6H76iywqkx vB6OkPs1sjnzw4oGONHI0G3QkbIhmES8p8qG5V4nbh6yGRu8gZXF2l3IIIJny7m7Aa1xZdBg R2q222iNd0RyUVktHaDnogEdJSkT0STrZ3s7atCZfjyScWXfk3K9oxsiA83UabzB53WbNQid 2PO+RISIKSC3ZRDYOoFLHGVq6JOJHeHwW+GNLELE/kLNHGGu/UeYJHXhj600fKBRZOEkHeD8 1J9cqHShvHaxs1ViOkkCMvRWUuygOeJmbdttjtBcTU0cM0e+A5dTRoP7SmpIb5aXXr5GJbKq e+JWXQq87KlLQf5cJVcwDAHJw2Ebd6tJQT0qAiLITguQoxcxQUviBGsZLWIhTvH16CGr+ceF r2SQjWklPKOwzyE94GnsawcQQ7FJlX6efB3/GSD05Y043DDkGps5DBhW8vP6HhFlkQFAD8m1 d1wUC1iTQn0I6JGdVkTSsggqolpLgi8Zir5fkVyrq3CJj5WwyxXNlEvKdpLFkgslEv6Qe37n 2F+rAawBJ9H0VhCuwnpWb1W12r/Kf645ozVqOGIw+JvCRvvDtBveK6tF6ppIvfz6x7/+Bd5C J8/fzqtvfK+7QEmCNYlJZHPYbQmwPmy2gxH8x/98gsfFz/ALON8s2fy8RBlw2QaJnVwq1TI8 Ed1HlMoRE1xz75bhzZ0CDVwh9+hFRgRhjWiR3zfSNVlbUx/vPRhalTiYIYS1G1Md1H5F6mAl mJflDlzJnlQlFa2MertxRGXAHCZ6UwICSdNqMtB419N47fFcr5GMVtCLNuqWZJ8SbZZLkqGE tg60W9k8e/c1Dclf6zU+0CeUbYiyDb8T4R24Lh6ps6Z9WFsAvO2HAsUzlG2Y8rD2U7bB1LaB 1K7lpazupU3qiYp4wvTBxGfahPMEu2UG3K6SzpPrwOsue4io3d95SfZWBZE8grXNFhr591yO +ZQzchJbyJ3+cnUlu+xxU4w33ocIG1MbSLhQakEP0Zq5UGy/uN9nblo2tI9WzOVKaOnp0mcA uznNWkR3YQMcDJ9l8nOmIPNRPlxv4TlWzgF313VXF5HdGpzzY50yOOzk+HEVYaMR/Ux9ng0Y WS8AxOpJ+bynz9S2RT/bCWuMJqxU9JPzp5Q4tgGV0GrVEOllwMIeoaEVsMWxAaycB1TJrA5K 4/jcV0Sy089KJWUlSij44xvB+wM9c24En9noApWmDYgulpIh7h8L1i3g9smnlx8/FvH3r8+/ /fr85TfX66GO08RhnSns6E0apZ9LKN7wTneig2SlmhiQqA1RgsgTDQo8IFRJq1Btx0mxY2MB 5OxIIR32gCdbVDaCeEJDSRa4IzrL9XJJLEaOrKEHO6lIsENG9QgpGwdRNixbB9+gyvF8AU/g 8GGqv5zVsXWwIb8AzpbwnaSSyBzwPB5q+TaJcC8JdJlS5HBOhRDtyC5ZHntJrN3vmmOEjwl8 VI+kPHEVkmXzbuNPIkmibRRKnfRBZUek7q/4PIsWHZjPo9a7vuOtuPZUBOUi9RpH3LCZ0a2w jfoAOmVliTczgDU6XqJxivftj9eg6zcropl61KvyZ4odj3JNLXJyS1xT4GIPubyjYVHLfppd CnLvSFEK1ja8M5QxfMknmBB8wZPNS3KDmZHo5hTva8HwOY1FFUmTyXbqflkto808z9MvD7s9 ZXlXPXm+MLt5ypPdtIMPVPchv/b6hUv2FFfgxGuynzRIz9J6u90jD70W5eCjtJfYl9Zju1pi bTEiRKudj5BfLti58IjTM3YCq76Q+V5qE7bbrHZ+yn6z8n2k7ie+khX7NdYSE8J67U2qe1hv ffVVJMKH1o1cuj2EMru3WPAZCVWdlSA0+FIbLN48lLa6szu+5DqRpOTvrf+u9eNwQavPEme0 QfdGYjA8ysGCjshHqGc5DgI64WDBKf/WtY8olxlWm4MUhzh4lvAlyo9ZXFUXHw1UcRfLl9VE zeQ63mYkzsJUmgxM9LG9KUq1uibnC/emeawSMInxJ3orvLUosobjIIYaZXWdZyormxInxZb4 CdJw8sRqZoPwjdTrL8UV7c8ATRTx1anzm5CSN3Myskxm9IcNTeorwUQky+A4U8KhGmrUAelZ yWQ/ml6YCOvUh2J7tBFNqhjfXR/x0xHfbZngBlu3ELgvvJQrlzNYgS/pjzQV75YlPpLgaXbn cIXJQ2wL7K1jSk4ZZwcJVG9tEyNs0TAS5Ra84ZWvDAU7qWsHvrLDDf+qiUMkcFHto7W8PPm/ 985T+eChvD9n5fnqaz8mtkt8SD4SYOm9epuqq1kagPvj0VM7imJphlWnbcHlI5oV9LM2AEmy BGeDSbwGBYOPdGrxzgARzqy8E6s1RLvE8sGh6DlGtqzccm/s0aZmGS22oNJPIGjn66xpOb5H j+ksFQ977DabEh/2Dw8ztMMcjU4dHjrZdBI6mGj0Bb6nSMhXsJbvEt746fE1Wi2xjyBMTJ72 SVucVvhMmdLbVtS2AwiXIfhxhh78OE3fvJnD5q0sNuE8UnZYYhs4QoPZG9vIYOKZFbU481DJ sqwN5Hhsd9E60I3MjsdPPFVVimU8TJO7x4jEnycvXsv3oWKSeY5SAh+uhld/px73XIZgk0j5 crXah16WMuaWGD0TYiFWq02AZokRpHbKrOOBzykuD6tA+0sxVYeE9VdQKnd67bZbBtpS/W4g gs0M/c4D1d+Cr8T1etv1rQhU5DWJV5tQVc2N+nvaKtPnYBPd5d5g1YVoh4duhrbc+qdPoK2i GdraT1OGeFVRV4K3WajPrNYP+8AspowQTyzHUVVses3Kd1jUtenrIkzj7Qwxa69NXIXpemQG yWmRQPOvljPZN7rfhxlS+56VUwi4niLXzTcSOlXgTC9IfgehRJOZqshn6iGLeJj4/qltqpLP pd3K9T3ZbIkYaDPpOSCcBhNPMzWgfvM2Ci2WspmU/FEFydFy2c0sZZpjM0fczhEDwkedYKUS poh2Fa0DM5+4NpvAxCK6/W4bKmYtdtvlQ2B2eG8J8ZjWVOdCSyNYWWF25Rxf/NKYlItW2DUj Rum8RihEDNCUuGDEMt6optbdUpanJQoMo8NLRH1pnELqOaiv743/taJg+42bUVFf10sXlrvc EtveavRUR8zF4I5GltVZ4yW1PG8dXZPJpM2Z6OO2FC6F9w1s7bLIJl2yJ7nVKQ3ZoXbtuwPR 1k6wKUXIGmrQkN6zpmBuyk9yGoNrHxacFKvlwQab7HTNwZGulNJg++3S22u4pVQvjlb7MMfV qwquk+N2uVvLPlBc3U5z3G8fnE1JfS/m2q6pWtY8wRXsKnVZtOjaVyXx6aZpegXvfb3XVTKz tMvXvtGkYP9w0iTPeOKFkJlc3XZiayLXEdiXB5wVXOLUf1Zg8pJrG2xURS5/xcypIVElZhD3 crfPnFpKm1u0k7Py2SjlfOTddp78gMhKkX1+/v6bikXEf64WdogRtUL8ORPk0OJQjz3fLzeR Dcp/afwpDSftPkoeVksbrxNONJkazXkMqJU0OdbTkPGN5ElCQnB+5bzQJJT7ai2PJ1Zk9AMG pC/Fdrv34PnGA2bFdbW8rDyUY6G3Gdo46/fn788fXl++uyepcMlrup+KxvXgta1tWClyNkS4 GjkHBh8mO6Uc2MiW6u7lnuA+5tor33QppuTdQU5DLb7nK8dD3QrjCFK+xZXXbuL9b7A8Je9N oIlNuMTtJcU/5P8b2Z/A3fT/x9iVNMeNI+u/UsfuiO4x92Id5sAiWVW0uJlglShdKmRJ3a0Y LQ5Znmf9+4cEuCCBhDwHy9L3YSPWBJDI7DU/Z1dpmWT4Ui69uoYzP+rdfdUMiTQGU+LTUw6L d3HI1shVneJpfkLUc60JO+9Vk3XNdaOaiShU4221dnvNN8Wq8ypp+KZrjsiyq0QZKg6v/irH Jlfz0wWHTDdS968PN4/mu9yxuvOkK69S9DBeErEXOsgn1ALzvNoObB7lmTDdy1uMemKsRJBu UAnC6IsoG+SrSk1OfU6iEnV3PvKmZP8OYorueIcrqnwMszidUcPkA0zx6OmkwlZJzTtv0yG3 TwrPDqAwDB4arVXHtyc9hPhFlXUssaWxY+UvImfaxDkn2ntxPNBF38HguKDjVcVsFbp+ef4T MJ616FjCfqLpn0xGrpLBx+YRVHwwcGiYEm2wRwIbDlVAaw/6rA6sEWNpWqvvomfYjQoGRxF4 X6TTH0REVzsGy1qzv44r2ec+2cNXG2mP/K84qErZH4MPAm2TY9aBNOu6Id+kaSGL3RANkdlK YCuQwGCUySxdjexaz4jAsWVYGkOOd+Vz2ZIfWcFezfWV3ebhlI7KJ8p6xjE53BRgUA/XR2CR 3t41i6OGodSirQo498/KHFnTFzjfdBTpWRg1JvX0q2J0aiUvn3bIFrOgWWEAowP0UeMXqeUD z4qdFuUSXMVn6t2cLBxsW5qdEpov7LrN2hk6w2TIhaEqJ1lpOY0gkIeDBUYG5VUYr61K9i1S gkFOjDt/E6luKNq2BLMPqi5dohq8BYWvsW8o9iMGiecn9m8vjBZjcerVEfwF2+KWgKYWUS0p 1fv0kMOdGVSeagt6jz9IAKqLihEQ173am1aVgjeyNbJ/oLL18dT0Olmjs9l0TyVPJzvkGpCq t4pTxqz3/etWdUiiM9ohrs7ie4+8TLFpbhAm0SPSoSjLq636AntCJo1KqazjpYR+FNrD8SoT uhbgwl4ZVlJRtk16DePLN1Yb4mB1HGbHzz8e3x6+Pd7/5PI7ZJ7+8/CNLAFE0u7pJ5RvfTdh 4NqInwRR1GnflSbR5XsMHvISvOrB0zb8YVK9AIVNyn2zLXoT5OVQK3jeSIJr2eVbpYuIdMVT 5vg/4Eh28RFhipsy8cIN1Ql9BiOfAAcdrLK16i5hwc4siFXvzSMDlnExWKALH4EgLxwSqbRK AT8WAYZqcYjokSAvzSbWvpIVfC+5McEIqUNKbBMNGEMT6wjIC0HpgThtC7rCWVoVaKy8f3+7 f1p95Q02hl/99sRb7vF9df/09f7u7v5u9WkM9SeX82559/4dJ5nCGDQ7Nt+HFftavJbDMpRG mrap9ADoBR3n8io/afVsZn+RV63qhwWwRtPtEi2ZJmoRlpUdOL4R5LlTi7pol6rPtYllAGXZ eW7If/Jd/TOXiDn1SY6Km7ubb2+20ZAVDSj/HPXpKitr7XvbpJPnDvgWDyguNjT97nh9fW6w kMC5PgHtspNW133BdzBYDUj0qBa8aimum5u3f+QkN36L0mm0HiGV2MDuHT4rhlrrj1u9noX9 dPoZnje5c4Om+0UQmKl+EYR3VWqfJITxZWPV2h9VAFclTNoDkrtpPtSqm+/QqIvnMFNPVLjv FfK3ItZyzDCBBOBk7BqBuBcAgns9II1sTQzybuxJp4LKlmPGybDYxSUYsk/dmM9NjlaCQRhR xNA0BBTs+qr+UrXn/Re06st6nsa5hl+aZeAY33Wa3wG4fBc1N0j7+vL2cvvyOLaM1g78H1rS AevLPPIGdcfTqmc2yFj/QTgoXkQIebrKCs0l0gI/PoCn+aUQB+HbTxwLy0WzZabM0LboIIv/ afbKOfaYhZIKipiWBTjEuRASKvVucAlTZgVDBqdmZuxuc55/gwumm7eXV1MI6Fteopfb/xBf 1bdnN4zj8yTwyZny+ebr4/1KWk1bgRJ5nffg9UpYC4BSsz6pWnit+/bCC36/4nMRn0zvHuCR K59hRW7f/6XkM0pIY+u0D88Wh1VFXakq5xCP/7YAo41EhVAO2GBOGXOialUy2D7/BG6Tq75L itJk+Gai665ORa4c2UyctgObE+uaAe0BJoId665gudTCfULH+5cJb1N5WMf31tC40obf9N0U AHJ5p57oChD7WBwjwh58tPWrVRgEIGpLJCVdfuPkDSuVAhXa4M4ih98/vby+r55uvn3jIovI wlibZGGrrO21tLLLpN1q2Nzuhnwi6EIVEgVSXtWDpu0s8NMQh6EW9nqYOyYfHX+OpYZ7mQ9K 7jp8UwWG71QVfMHs1i4c4mkl7FVzRLJ2jUJzxJe2SWeRUJTg/uc3PgqJ2tNfMyht4VCopxdL 7Gl8E4VryKU1d9kvytEV143qx0C2o6YzJ0G0bgroc1Jfn/u+1GBdZpTN2vob1QiMvLuGm9iN q6drKD4JVFdamsDNJpinUy5lfvy9+t5FNjHMBuIoFne5ik8Ujd7UXZb63tLUsEx+mKVsVD3L KvX9OHbUVF5ef91vqrT1fObE82Mhtv04AhI/p1d16lNS9yzvKERy7p//9zDuR40Vn4eUEpt4 i6I6IV+YjHmBaj4YM+pGUmXcy4oi1HVyLBh7vPnvPS6TEHrP8CgPJyJxhk7gZhhK48RWAkyC ZuBsSnmCq4ZQdYRw1MhCeLYYvmsjfGRhCVN8a0C5u1BDrSOHTnkdWwlLWeJcVVuame0Xb43N 18IZqbBbVl6py5WKW3cELViQhYDY9RYsHUmW8mW+531CkchHBQjdJdgIaynB+2odG1M06krF YxvuWnDPxNmWmSDUHbL7rhH42kMn57XUCKHP3BMOOrlrZJ9cY1RnOwdwqdfN6i4mU7AW4pgE TyzeOESMso3X3trE8dZrSaYG51CqHxlJ8AoI3HAwYwhCnXlUwgvXNLFWj8wUIoyppPhu3Q+I lEZ1orVZt/vkuM/PZZ96m8A1I3b9JlBFGjlOhIl6RZpdQPjZIzuzkkxOTFVDQMbYxZ98FciQ kRcBjgcCXLY0tkK1dNFNXG3XrOnYOeHb7eP+2B3VuyWN8gkuW/uqereCB1Y8pvDKddQniZgI bURkIzYWwqfz2HiBQxH9enAthG8jAjvhWojIsxBrW1JrqkpYuo6oSryI+xwpY0y469DELqnc 8KDPr3M+8ASQVSlVgq12hT3i/dAS5cpY5DkU7JKfkYGBc4aOVSZG6tUlGVGmIrzgQuKW+Ei+ N3DCHU3E3m5PMaG/DplJTLqkZAl2fC+hXgNO+L4M3ZhVJOE5JMHXtYSEiQ4kN0DqY5GJORSH yPWJmi+2VZJXJN7mA4HzHOT0RFR7SPUEOK2k+xzelU3o5zQgPo13zM71qL4jXE7vc4IQc3ZI EnwFcmnCcy0xPM+zELY8vMixEUTm4smNayEiJwotjLuxEFFME5s1iUeRv7EQVIMIInQshCUP 311viCiH/kiNfr5R8sklosrrneduq9TWFfmgHIjOW1aRT6HUnMtROmxIomsSjSk0JnOLydxi MreYzG1DprvxSJTMbRN6fmAhAtdGEEVs03jtU70fiMAjil/3qdwAFqxviBWoTnveoX2aWFON wgku+3s0sXECqnC7ONwo39nie+E5HA2DtOHR3cPjQnhkmaHITiKJRUdeVZBCs0JAjjPPWYf0 TOIHQUAP2jiKiZJwwTjgOwuiGo9ptnGoCR8IjyKuy4iUFdihpyZdDlODn8MpBet3yLOkUOXu 2ie6XM6X8MDxScJzLUR0iQylz7lXLA3W1QcMNQ4lt/XFbKkoM45seggjoXYHtslIncYpmSqK SIE5db04i2nxm7mOS0ZifAscW4g1Jc/yWomplirqxHM2NE5Nz326Jrpzf6hSap3pq9alhrfA fRIPqKYDnCo9+EBK2yMtunAyiiNCMDv1rkct4qcefCOY+GXMpUg3o4mNlfBshG/BQ6qHSQYk RsutjRKwXMdhz8jUORXVe5KKvPVhZ2NykppOiD/Q3ph7XtoW1i1Lf+HgV/GwuiSl6tNXAPpu e4LB4yKYLDj3XaFakZn4yQDpvjmBT6H2fFkwZIWaCrhLik6qQZO381QUeJIhDVf8z1HGI4Wy bFJYPIiGnWLhMpkfqX8cQcOVvfhB00vxaV4rqxkIHPOKVx3qUy446xcx0zJRxydfxc7tBRwW Vu3c4k84Hjw4yno+DTVsp+vjoABUfD9wBpMQHXEqcJcbWbbpQYm0nAiPeqrUyIMXzw1jxVZo b8sLhpfnh9vvK/bw+HD78rza3tz+59vjzfO9MjBUzRVIgoFxXEV4ALcwcNmMni8xYTMVjHSr WZqsls5oeX/bFdleiwDWE/X0ltlHCWD5chGXN4MezaqKApzQb50N2NNfggMZyUvWcikrvQFo bSJsP96+PK2+f7u/ffjr4XaVVNtkaRHhI+EJJWE0gEBldaUFWWUoBG3Lfg7Bq80eYvn+X4bZ g1X5tKp/HVCrLi0QqachVEX/+vF8CzoLVu941S7T3jYAIqxOOaoAIcLp1sIUULP6pBBICaet wB+rPH1HOY6HTeihgsDRdSogcCo06EUbQZyZSqDigXeUNmFF6uPAcib5cky6C0LPr2xTfA8P ALrjXtIoW9XoIcalAoONRHpIwInb47RqsHlXTuj3x+Jz+8hXDwYENm3lFT/j14N8AYvqmroE BRyeCWPEvPOYH/2iw7oZxTU5XlzrHqMhq/nWWAV7Npj1IlCkRyjQOuwjVRoHkBXBOhqI7FgV qvLqDMnyosphF1exG6iXRsl2CKePwEGvGHrQBhiyY4LqCFj91h/uSlxHvcGRJgWQsGUYGRD5 GLcsChoT6MbVRuKEmmMRzCSvfaIay8oP8UWsSIj2/Cl6GlZSAWhSsXgnQHNymQhjvF9WIWxZ 3nVMtR0isXjDB4qJxQYGsjKBEfUzKVnMHjOXo4XFLvL8Jt9qSXsOIR1ZnpqyR6evSwB42nIU D9FqdkSvQZYwIB0K4fDDUEnax7G6w1WoLPQ3McnUCdhCIRhzAVG+W7t8RQzy3KQxLsXwnVXo hyFZcjzrLHjByo3vhBaKb5vchOJglK5dK+PRTLz2BhtDF7vsUx9Zal0oOJIK48hCxVGwsVKR Y6M2dGsISj2hVahxVcVTAeaRJSJMxRs6Q7540Y2sL2sLg+04qLi+Qinc7nido2MyhVOVbBb4 C5jywjq6C2msOQtlrhoLxyfa0I18G6ctAJjzfLo55aLg+XZubU0TLxs6F9jjIc20hdMV0RCD JkphYH325Kc+dXq6v3u4Wd2+vBJ2qGWsNKnglanhBlCy0oDouT/ZAmTFvujB6JM1hPBjZCNZ 1lnjpTYGfPt1YAOmU5+CZrnQd1V9QwN0CkoP3JSC/Y9EXXEWWl14JZpkJ3N50cLIxaUqaujB Sb3PqbcKMmh/rNVVQxSpyiuP/9OKDMyuTNgBjGaeU/4b0z5oe9yBxhKBgiMxtieIUyXOLqgo p62Jetq0tOC8vE3LKAbyhmYp9hRrLYBnLbSHy8b/0EoFCPLv3vewNZW+dnAweEaaZEnbg9H7 WGXAMiPsqEQrstkjuhg0xlavS/X5mkdEsye8WxEmdpArJvURdtEJ4AyhMFznc2yEd2lowSMS /3yi02FNfUUTSX3V0Mwh6VqSqbg0dLHNSG6oiDiiauBRterrKFXsQ6Ek8hr/fSiG8JB5CCvQ raosE34VxcP0XGgrcPF0KwlQ9fqTXKjeHN7b+7g++i5Pqmtk+IhPxkW9berMyKjYN11bHvdG ofbHRFWF5lAP3g0KXF1l07RCN1QNKLQnUEZyRhSeg+fJUjoAuf96e/NkGqkRzoTFrKTNLhpB G9wXBvyYfGCqQFWI3hqJ4vQnJ9J9Vu/LONK8GIvUztu8/kLhHMgHkmiLxKWIrE+Zo8otC5X3 TcUoAh5WtwWZz+ccHkh9JqkSbOlt04wiL3iSaU8yYIMwoZgq6cjiVd0GFLvIOPVl7JAFb06h qnaBCPUGXSPOZBy+/fFUy4CIWft62yuUSzYSywOHJuoNz0m929M58mP55FYMWytDNh/8CB3H Trl2KrRTkZ2KrVRkzcsNLZXxZWMpBRCphfEt1QcXUYGFcZEJD5XiAzym6+9Yg6N5iuKCPjk2 +T64a2jiiG1+K9QpDn2y651SBz1lURg+9iqKGIpOmIJKC3LUXqe+Ppm1l6kqOY6Q9Vhi4sl5 dZx4+aSmfc9150eBnjNvlct8a3wI8zx1QyzT5ER/mu8qn28eX/5e9SfxosNYG2SM9tRx1tMT GmH9zRcmQWh71yXlmYSaKXapVU4+ZDwo8QGnghWNLvLKvhk5hmM0zCapekyFOD3Kvlkj+6cq ik9KEVM2CZJ19GiiXZwzepwtG+LT3cPfD283j79okOToIHUHFZUbmHeS6lKjgw0e31IOelIj bI9wTkqW2GKZm4hzX0VI70ZFybRGSiYlnbb+ompAfNdc7I2QdQROfBKrZZtjFVshytBJTuRZ XGFffZD6FDQls3DWVN7Hqj+jw+mJSAfU7ya42qCFcEmfb8ZPJn5q146qA6fiHpHOvo1bdmHi dXPis+4Zzw4TKYRYAs/6nstJR5MANwmqDDc3z26DbBRj3NgvTHSb9qcg9AgmA9fMRMlSLqN1 +6tzT5aay09UU+26Qj0UnQt3zSXgNVEreXqoC5bYau3k0R/qWirAp/D6iuXEdyfHKKI6FZTV Icqa5hHy7jbheeqqirpzL+HCPNF8ZZV7IZVtNZSu67KdyXR96cXIaeXcDKctu7jCuOho5+0x 2+c9xaCdNauYTKjTxsXWS+GoJR/SpqUmEp3/cFZJmOxgys7rD5i5frtB8/zvH83yeeXF5tQs 0emYiqKo6XSkiJl5ZLrZVx17+etNWAe6u//r4fn+bvV6c/fwQhdUumDpWHuFBY8D36d2O21L yAoPiddypwpnNNpOVR633N58e/tBHVPKclf5VW4s8U3ZROgpzLiYXEbGannddElNgucs9R2a AeHLiSzk9nhtS8+1RCmrUt2ZGlRni5icWMQrgC3P55Uq+3QzS3WWyitOvXEuChjZd3ZbMvwh H4ojuD6sirqwkE1HiGnVYHTCrPfdMPzoYz798/719eHug29KBzc2JRI3tooqsaqoPp5uSwuL aUGED5G+KIItWcQxhVkDn7clHzbbQtUWUFhi7Ao8r4XG36n1nTAwxTUeYqSoyFWb62et520f B9rsziFzJmJJsnb9wAKTnzlxplw5McRXThQtjQs2Mr+u2SZlj3uUIlyDMlEibTlpImRyWruu cy6Uq4QFxrUyBm1YhsPKlYg4nqaWqClwQcLJiYRbUML7YPmSuh325Qsm3Y+WL75Z7xtNEMkq /rGasNH2rg6oGkBJDTYDzXqQBMYOTYus1Yvj/D06+BWlyEbdPYSyqsB+mcfLgGMLtq5xnwpK xX306PhWHzXJLj+nqWqzBPQr5c0YhRFWRiCf+UqHzma58RGGRUtkWHTs3YfzKT/iVMWjckuS p0I+uTVBcekh7GZGgU7zYmjNaFaQlA3k1MuFgqpKP4FC5GQRTdV05hIWUFjEkrd+863LO8b7 PAnX6HZWXhIWwdoZ8GnziGk29jC2xNYPl3Rs/lKdmJLVE6i6WD8jzNi20/PmtV2I34xCHZLu ggS1g56LHHV+IRYnsNeptWOxKtmoR5NKharT4pgRny3XTnQwg++iWH2eKWGpoDTNov39z5vv q+L5+9vrjydhrgr4+OdqV42XY6vfWL/6evP9/k4xMrgkFS/WS+aOtHt4vQev5KvfwLvzyvU3 we+WqXlXdHmmb2dHUHeiPF0Mw0nQZNJ86sW3L09PoMEpi/zyDfQ5v5vihOcHrrHo9Sf9QnH0 rgoFqbAhNH3aJZcu1fEfgs+nE961sCKpecdCNbDg6pK6oIvrbWUdvHm+fXh8vHl9X2xGvv14 5v//wdeB5+8v8MuDd/vH6q/Xl+e3++e777/rSgJwf96dhFlSlpdwsaHrCfR9oro8GmXMbtQ8 k2deP2BjcXd/+3InMv/2+sJ3F5C/MCH29PATdYGpAZIjGlYjnCXrwPcIeBMH5oYgB0/IYUri nhG8Yv9P2ZU1uY0D578yT6mkUlsrkjqopPYB4iHBw8sEKVF+Yc3a2l1XjT3OjJ1k/n26AVIC GsDYeZrR94E4G427u4mW9vZVIlbRcuVCiyi05j59ymAWY+XwVMbGU8Abqj9tnTR1E25E2dgz MTzK3XX5qDhZuW0qrlVL6xAkbK2sQMmgx8+fLk/ewCw9btYLa4KFcGwXB2aKwdYBrtYOcG2B 92IRhO45nr0UUrCjizYrw/PSvAgMY7sY3WlrGA7R0LVd/UOknnNr1YaS+mAIsqO2N8FmcK4d liS2y9c34vBUTLxytU2wWbhhZ+hVcPNn8vDl8vwwaQLf3kQNMmF3K0TtZMtue1zc4s8fH17+ 0eLVSv/5C6iA/77g2HKHlmqtZPsmhWSjwO5ckogjQ7X8rmIFdf/tGfQK3tp3xordYbMKD7fV 9OeXj5dHfKbyhJaQL4/fYJxwflquQvU6fPLMIbXn3Q8YAO8gtZenj+NHVZ1K087F1Yi5nq33 YrfJJS+HhfHK8EZB3ZaGv0zCmY/zDa4zrYyYXKDfQDO54yJ0c/UxNGTCoFbmg3ydIk/ydWpj XPU0qK0/re3GQ7XvVsvKXWjs8YG1OUXuaWkgmg9udBcDOgejThxuF2+Qxi1zkwyADbzsNtYf 0huknPEFb5GeL0vBDRkyuC40n60Qbr14g4u8XKirfsIFkScv77tg4WmjcSC3BEzO9Pxqcksv Vw4FfKgbUrHZTedhk+VSxAtfDbAhDNart2Qg8BQmTxaGk26LC9/gojdT9HyZ+WsoT2AM89Ve HLcCD/E8NdT1sG7xiZ3gYbDyiCvvtkHkEck2Dn3pQXtFi0DflTZkqwzSAKpoGRrXZV++w/j+ 8Pzp7l9fHr7DkPD5++XfbtNic3kiut0i3m7JzkS3W1vHRnhTYrv4XwtcwySHoFCLqYjU43BX tj5Kw7v/fgerJRjjvqMDH28G03YgZ3izuknCNCW54ZOAqrPX4+438St1AJOapbVPJkH9CrUs WBcFZIfpQwE1pXuwvoG0VleHwJinz7UaxrFd/wtX/Yd2S8n6d7XUwqq1eBFHdlUuFvHaDhrS Q7FjJoJhS7+fZDsNrOwqSlVt5Ip/oOGZLXPq87UL3Liaa2HLw0DTEaBzSTgQViv/5S5es2Dt qq9NoItYBwvRX5Bj0cAgSPOH2GAVJNw46gFAurfZDkT+i/XSsLZ4y/KSpFINnS1hIN0rh3RH K9J+Kd9hfZU7N5xYMFq+LJ1o48ws6Q7yJJnkIUucCipaW3KRhqBeWwe6DOjWrTzBpWfHCgyd IN78d6ggmn88Vx3zTJeaZNKCXnnB/haHC1f9hM4mprpK6YvNdXHRCUizenr+/s8dg0n8548P X3+/f3q+PHy9627y+3sidXPaHb05A9lB781manW7Mu1xzGBAq26XlJF1sl7s0y6KaKQTunKi ulEQBYfGbairyl0Qncn6eBWGLmy0Nuom/LgsHBHflodcpL+uCLa0/aAvxG79Ey6EkYQ5nP3L /yvdLsEHdyG9maR9Cqu/x9c7tav2e1MU5vcAuDQ83gZabBZeSltoZslsDn9eSd/9BatIOU5b g360Hc7vSAtXu0O4srCG1qfESAPji74llSQJ0q8VSDoTLo1o/2pCKoAi3tPhhnU7mA1Fdj+G BSSZNfEhXC1WR2sq1cKIRaVDXqKJ6GFU24uIUd2T1F14VT3d09PjC7ozgAa4PD59u/t6+R9D YIzjt7QvyzMoLsuqwP754ds/aCLDOtZme02lww80Im54UAOI+BNASOgnawiYzobkw/h9p02S j3s2slbzFDAB8sRo3/Tij2CtU+LEO/RxUGsPulP96At+jCVHzx6CG0HG+1JMHq3M0Ijnu5ky PsnlQy2HQRUk8ablCLP49Lb7bvBdV+qm8afdNbR47t7UwW+kq6nrlhghkgMMtWsbF7wwDqFn vBoauWrfyqOPm0Rg3tJ8cL4mQ7INwthPsjSrKy8NYgKt5jrZBbKq+2PGet27rQTUScEfKyd8 dZoXOaKSprCVIyOj7vk2WNEyIzY2bVbwklfogfxwmi//eItT7pmXA1n3ciD2/u/Yke0z/6d7 0+2uTpWnfT6YDa0wkFRlpMCIal/imwJPZH1amDEx0dEqg+LvQ28MCW9BU43vM90zPBLvh4LG tKuTg/AVSznOBLkxo2kYeoB6nce5l2+PD693zcPXyyPpNVeLOkZIDsrw+a+Hj5e73fPnT3/f 9nvz54cvl7s/f/z1FzpbonvMuaaM5o4tu7kGw8yzTNGOqoGluiEI+L2r6w7nf473rBhDjueH RdEaB1gTkdTNGdJlFsFLkJ1dwTu9rSeuBV3V8CEr8LnUuDt3maPCIZw4i1vKXwhxTZkSt5S/ GCnndZvxfTVmVcpZ9UaKxmNNrK4sz9o2S0dpnEOPUoCmR18+ri6S49CABlScL1yx1m2NgN/A B5OKFwbR8UKWqlNOym0B+Wd2WHjbJTcqXnYCX1abMvRRyXmXtTgj8AVgbeKlQOVDZXc+npei 69zVA1Wrr4YB6VFEDUQCeiVlOSdNVMGs2Zf6waM0c3nLupKO9HwBRJBKYwQ+vjrylHujb/mR uYvNjW15AIosXqx0U6LYJKyFblHjA2p5w8kUOvT/4U3YPypic3Vn35CqWG9tRF7GP4ggy70V TEYmrWazGno4N1XY/Vm+g9IjiHxTB4y8rtO69grGsYvXobdIHWjxzC/TrL3397LIXSZo0JLr F1luGAwcrByzozRweI3MIJNedHXp0WimrSGJiKTXR2fsWvoYi1K4g0F56JbqKrLZLNL2iyMx 6WxazkOvLqf1KMsM5LKqS7OQuOo0TIbeMHn/bp8mpnqcOMMDByrTFia54pBl5kDB+nq8D7aL wYkunCipLXWC9WpU4CYwvHhMenwsktQeQRFUT4PVU27NkBcwxTJfLMJl2OmHIJIoRRhH+1xf Qkq8O0arxfujiYKW3Yb6KeAMRvqODoJdWofL0sSO+324jEK2NGH7up4s4DpbRyWJtUi3hv8S xFgpovU23+srgalkIJL3OS3xYYij1cZVr0b1vdq85TtNa5LZRpQdKdGhVgDDJssNtp24zIz0 peCMqYy3y2A8FVnqogU7MP3G5I2htky0tNImjk3HOAa1cVK2eRqjptbRgnmprZNp4tXKmUHb BM2Nc/lkubalYTxJS+m4ChebonFxu3QdGHee97BEYPprp0Na8nnSlDx9fXl6hGnSNP2ebrjZ F+b38iG9qE2LjQDDf6OocyhkggZVMCOuFYPczLjF4ILhb9GXlfgjXrj5tj6hD/er9gF1D2vN PMejDhqzg4RO08HkGVaTMCtuz2+Hbetu3j24lrao9y5raqLudfUuf461EKNlfNNgcFULnZm7 xiphRFil1NE8Qk1SmsDhlGaNCbXsVMLUywTfGe7CEBHZ+x7dz5IUAFYtYMJQANxcMaMoYQnT ImVYYFbZRNhdSMmqsr2an12z9MbHh9ZRLZjpibhuQhg5tQy26KVgA+rCVPwRhUakaiAb6yI1 LQDJMrR1MuYkpiOacRSZJCVnlO7G8qpzz5BkVn2GYDGKq1Ni/QNpCgdE2YQnMcC6IU3ZFNGI rjsVYyQO3HLmvDkUO3bKaAiNB0EJFveBnXLZ9MtFMPas7dxZMtHjYGP4LJtawZGVQC+6K0ET pG84JJuhxRSSMG/tHlR2jf7MQoldy1kx9sF6ZdxNuhbVkffJ2xw7ZmZeCXkV44WR4M56ta7K SaSTpUGsG7JT5RSG884JM29tKJCvliuad8EPDakR0JV8aFyYXMQTTcX6OA4WNhY6sIhip5AA H7ooMkz5A7jr1NmuIawSHOsj2tOuncMUhkrYItCnaRKTr0WIoA5nmGvZYqlw8r1YhnFgYYZx mxsGq7rTmIrGbMikG3KShZS1BaOVtpcOCUysYGc7oPp66fh66fqagKXhjFUNAQTIkkMd7U2M Vynf1y6MO9H0nTvs4A5M4En9OEEatBJBtFm4QPq9CLZRbGNrJ0afGWiMeu1hMHkZU1Uhofm5 C25PkpH3kFLNhgjpcbBYCIyF2hWk7SqPe+Jh4UZJtPd1uw9CGm9RF0QSimG9XC8zQQSGZQLW wZEbdVUczDKsAaMqw9WaKsHhQKYFLW86nmYELLMotKDt2gGtSDg0CpQc+Y6WadoPoYMIi0Pa 1yfQpR/lvkItajoGms65ADqXudJLyvN8+pu8xaxdCZbSwKh4MHoKOMNqCvlKYZioSsBmlCWa Xeb66sbJMprDFwaQbxJn2yfW53JkR/+2hWG7x6SVcUkfK/i+ZM6CKv7ImY+SiyQPp3aPvSza G2NUBDSemR47bJbKJGXtsUELIa8o+ivEfNc7s9amwbWJfjILUVG3mf0l5NHbtPKgz0IbbGsY mCGXHzLtzaHsbnTSjeYWKDCSdz4z3LOAKlUJiyE823DCOHvvgV06SUUVhGFhf7TGt2Y2fOC5 8XJTTk6SNLTmX9IYBq+ytQ03deoEDw64A5Gc7LoS5shg2mpNlDDXJ+7x3qK0n/uUQ61wcYHt OsWS81rlVEzpK57aOw0H/TUq/Lh5J+7arNp32gsxYGFlo731tL69ybW6BIPeKx4eZcLW8wkM z5ZobMWMgyVtPzigMc+NrNAtryuk21qUYI+CTgqZFfe8olhXN5iKgeI9Ct27tMI4/KIgrG4Z TRuWoym/z86ChG1C4/2GxNQrRROECt/XVYtucrRT3xmzqiTDWxqkAPj4T38NqbCaAB8gk7Qt S9PSgQTzlkR1qM3xQv22crbv1nFEKgeS7Oqetv/9mTRqD6sH48gFwRMMU/raUaZxbtUekoHy hKUkxu7EqwOraG4qwUHg6fdFQnxBSTCr6iOpQ8ylLc4zOuoza4OAH41WkiuuVyGCbV/uiqxh aWhR++1yYYGnQ4an3LQl5PlVWfeCVErJ0X1JnXcExuOUlgpHCSMTdzReBfOxvQmBHjTkA3sF DNfQrYpaFy8NtPLcZBXkuCJZa7KOFeeKKIsG+maRpE7QuLeg447zcJ3G+NxElgo3k+j2KSRR MLSBDdNYQXUELxkpRFsnCSPFBe1i1aRgpegrUuPC0E3y6SmtUNFkGd7NoNF1KDKgwjOSR8s8 pMykPmWTHbDNsooJfT50hewslLC0eFefzXh11Pqk47TPgQ4QWUYauztAPy4pBlPIbtrD07y3 3VArtROz9OaJc9OSGYIDB+E0oQ9ZW5vlmhErlQ/nFIY3qnSUv7nx0O+cuDp0nX7Noy3af3IO 8WoKZPWI1LTaAKCyb3m9jOeMDO/MGZFJu3CHhI94P6TIpisuxCIEPZjsb9tpBsZaVIhMjIfE zB4JVlWgDpJM7dxcrVQ7noZipVj2A5RtMOXZDu+6cEGyZu1ZGyZD685tZH7ixtMBOmrBnX7I 5Oy1aPi4o8ZWja1FBE6GHcEZGZMdy6kR0yvhcYwlRePp5TueP+GV5Ee800UnZDKO9WZYLGTt GykP2MC0TRRq7IDeUOv8Vpqlc0Yj0RbvgYG8j13nYLsOm1rAZMz1rZWFOZ1bNgy6HvowWBwa u5xcNEGwHtxEtA5tIocWh8hsQrrFDQObqJ2VUF+zTAtTv12Y3hldj+tWK2lRxEHwBgzFrM14 2hgvb8M6wvroNKdr2pU7MQeYpMRl3YwK04rfDEsDOriJ4ZRkdRvxLnl8eHmx1xVSPyRE8OTB j675ZQlSEqorr0uXCtT8f9wpy711i7eIPl2+4bVxfDUuEsHv/vzx/W5X3KP6GUV69+Xhdb46 +fD48nT35+Xu6+Xy6fLpPyHzFyOmw+Xxm3wM8AWdfHz++teTmfspHK2YCX7TWvEcxtrGmQBp D6QhEnSNmHUsZztquFiROQzkxniok1ykIbW5M3PwP+vclEjTdrH1c6uVm3vXl4041J5YWcH6 lLk5WJiTWavO3rO29Hw425CBKko8NQRr37HfrcMVqYieCX2I5l8e/v789W+3qcYyTSzjRXJi Tu1p84bsAyns6OqUgEtHm6Y88eZNSSplX0xby2C2IoiXTTvEnqEdtLciT9GNV1vfvF42jw/f oVt8uds//rjcFQ+vN3MMpez3JYMu80nzTCrjQQcLdVWcyRB+SohdKETk2EvHc0m8WSIZ4s0S yRA/KZEacWfLSmQ6gt8bZ14TGlrFCsfJbap6nfLw6e/L99/THw+Pvz3j1RKsobvny3/9+Px8 UfMfFWSeyuGDGFBOl6/49u2TlYtwGoBo+cM3vOtcg3QtrKNAYIXIcBmSC09tJQcO88+M9LUZ tevhyvT6lTyDmerEyNOVfEvQcWTfUBcXE2iPq4oIpowYqV2/QX+pKAXeuppDKomywjpCWpKF 7Spb0zn+qdNxoqbUifm8Rffq4KjFDI1ivE3QiqWbbO8j4+WyxtGdNo1KDtEycDJyBn3IrHFD sWgzG4b4JCvILqsedwOTssFNTaq8jJ10Zlrc1Ji8wzsdvHaSR66WYjbDG/beTbjDZyAU3nLN JKyF3XmMg5A6WJhbXl7b9WTx5Mb73onjjmbDqrGxxlmDf/PbsmmdQjjzvWBh/PMQwy8EYb8Q ZvezMMH2pyF+nplge/p5kPe/Eob/LMzy50lBkMKtCe4L4Zav+3rHQVEkbuksk27sffInL2G7 mVpsNuHCyxn24XRu6L0dpWLH0iOBTRFGi8hJ1R1fxyu3YnifsN6tUd6DdsbdDCcpmqSJh5Wb Y7k9zb9RY8PSNEt9c/1ZaWdty/D4qMioc6g5yLnc1e7hwKNG5Lubd4YvK40dYDCo3aU9nTyV riw8uqmy4lXWeT9LPN8NuP81lu4PT1wcdnXlHrCE6IOFW6bed27pVa8vtXm8uc/kHIazkq9J bACFZKBkad/ZgnUUdBRqeW05Pyqyfd2ZxxMSpitw4/a3nBpNg2By3iTriHK4EU/al6fkzEBu fuCImBW0yeUxXApzl4KdqXFXgc5h9syS+pnA83yPwBekDB1en8+OfNdKj8NmdusTa6HGCIwb C3RLSsAUTG445HzoerKmghkYHgvkZHw8QzjSZNkHWSMDafCD4An+E62oypG+i6CGpPEvmq3k wGphnMnJiu1or8N9fMeKNhnwzJSsQzO2LzIrikEu0EtdtJt/Xl8+f3x4VGswt2w3B23BVU0O F4Yk49T1HC7PjsZuZ8cOx9r0znOF1Ox5d7bvEl934SwvX2yy2W0I1DS3fnvRogfC156eJ3d2 UN/CZgqF5R3lYXnoYOcNg6ovR3UNXkC4W+1fnj9/++fyDPV/2641K3/edOxT6jastbF5C4/s iA3MsHEnl+VH+2vEotTaBMAYQ08l7NJkisdcBDsXvhjYWu+xMl2torWVGRglwnATOkG8jG1K iyQsd2f1PekX2V5ZJDOXs/IlBCz/fLqI72AIbGrBO6oT7d3HHAacsSDbRr1zldaPGSpf63tH 0Hysd1QJ5WNlJ57ZUHOorREXAmZ2xvudsAO2VcoFBUt8KHXbuzS43ArdsyRwYKGFHRMrIeOC uMKsI6ncveer/qXZmdG5nl+dJLarm5EN4aYq70fZW8xc8e4Aqv49H2e+aKdGd5NG67mD5CDD IMleNvdTsrXfIEMvKRvbRx7oIaYe6zHxcrNoGDwe0RI1ZN5vm9S3XRboskSldAdXGyFsNc/e 7rIqIavP9FWCc2g/LjPy6uEc+dFY57aQv0dPVdGxdk+1w96prOR7mOtIbQR399MUb3Y6FSxO N+45da2KvXMshTUNUFdVfMO1rTf2Y7qTZn1INIg63g/ZYVwqZD+6nD+iv3B5X+YW9qQPEid5 nGYCeOpmIjxYxrqvtrI0tuzhp+dQWDmIUD4iEjRVaZ0I47c7fLphxK6g+Ww8tpmdPJvXbm2i d4Hp5ZGRL5z2vpmtn55ZYywiNerkCpnTCoRBwOuDrKBXOzRLGncsRZeXJnHaifT/KLuW5sZt Zf1XVGeVVN3U4VvU4i74ksQRQXIISpazYU08Go8qY8sla+rG59dfNMBHNwkqOQvLxa8BEADx RvfXFKnTtRiyRyAfJxiFS8KOy6RKtIjHMI+ghPchsRgCbM+30RiJt6lXFdkoZHfZN62AVkDW 3jL/Bd+mYTCNwWr88RPGxXZ8R75ji801stPL5frBb+enP3XcIH3sfS4PPMSedM/0B+eMl1Wh WtiMfCqcZGG+QU2zJL8p0+8I+kCf5O1c3tj+8X7AarRe1oQYvto/CwibhelQBCoxVC0NnlrX 4RqsWYvfbbdYF/h0vycDS2t2bGbfg/YUJFziEiyjYEVc/2JUWV7TzFFjbJVwaa8cB7e+FnZd S8+nMcjt+3LPuif3XcPUVHUnJcbwbc0mB3Axk2a6AmOr9R717DGqLPAbMCfH22cpGzMBtGBk Wg43sL8BlT6mE5BIlWyArw2fj0g8jMWOaZyuoqTj3CGX/Kr0te2uxg0gTrIMdl9hUezGGWeR aS/9cYw6CjwXm9ErNIvcFSG17Rsd5hNU6Sb52jLDQYNjaMlS1+KPH+fXP38xlT/HahNKufik P1+Bb06jqb74ZdAl/HXUF0I4u2H4TfX1/Pw87TQwvW+IcSyGx2bmRCbW6lTHgUi3iZhPQ3JD RuSDQqteHpX7mTxp+l0n6nTm5HmsLPr57QbXyO+Lmyr/UKP56fbt/OMGDH6X12/n58UvUE23 L9fn0+1XfS3JAz2eEnMumunOvVffP+EWkPM0TLN0hiAoFb+5mNpy3U4+ET2mEb0AdP14VO3R JZAUDXqLfXqAa1Kq6kiapn5gYDTmArSNxFT7qAc7lpF/XW9Pxr9wACGsi21EY7XgKFafUQii n5OFZHHueN9Qi4UYoquvx/7CexyM6zWwUhslL+7wZp8mkhpS+21kFquDfh0IGqiQ08ls1MUK wtD9PeFoKBkkR584G2vxmJs2HmQwjt13ULx5iGta6lbm4ZOoAbfJBViHs+DoEdafTlBxN7J1 KaU8My3DnxNYs1FcTRGPgE/hMhKDprXUCnzXsmcE3pzAmJX4GgFzzNo35nB9vYefbWs3jcLF OmSFGWQ6wZrZpq2r9qPIk6nFDUtTUwmzDV1FVQeBryJLJ/H9wSuA2Azcb85Q6NVMJa1mGqfu kwLu2DP4TONfGXMNVleu1dIwZ9qx48/0CGeu7dm6T+BoPw2LyuXKnRwr363ViBVc23kt7AgA 4YRUFuOuvk49323WAUuzxznxeGzsJau7g6IIsrR892/DOP8gjH8vjCqDZFURq0L+dwHl5DAJ qcuYdnS0HEPXmEfL2b5b1ztzWQe+rlv4te4TAm67etxdaXDOPEuX1fCz42ubf+lGuvYP44Cm G41ptFDr7+iwZHO+vP4m1mT3G7PYJpS6biHJIDWC2rMHTnxY2ir3V/dfgsxMYHmK7A5ZMJha DLy1PTqz2gCtzQl/LBDnJPmGkMMC1jL8wfVnnicZJ29XRzQflCg7FQLM69yiRVDHbMKqDeu8 o2kYDZHFD/Acjd3gKnQajJyRbPm+TawvbPTjDK43EU04f8yjpj62N1S46sYrn0n1NFWAz7jj COU72B+7S/bhBJGop6VFE6VrCpTwkTdJnlafqSAGT7c6QYAZfwDgSRUVeOEl0wXGxLFdOwjE RvA4ClrtiaYKeGNde5gZ47AWWFowtm/qxzIxRxLRTD6vYwqOgpCv1CFAoIZY1DuUsaDUwKLN oFsd8comfCzhxIQFebCRG7eBmU003I4JSfNBFXvz4EnwegN3hNPDp5blee4IqBWHYLc/w6Ta BpGG8/cCMJYW06Oy89P18n75dltsP95O198Oi+efp/cbsgLrk9mKz1IdJikcT69TUrs+EtB0 3ss8yCWZ/KGOtvNBYHOb0A0dlq/5bEzYZamcg7rmDJGo+IP7+55J9IOmsclrkcfZV2zEJraW pZDsCjrWo4e0qLMQQg/NC6LWDF/AAlKKFhUxcv0OMAyEcsYWm19aDBKMRQlYAs8UcwssE+VB 9DGaC8VjjN+2r4vmmAGpIcXJcNwneShxirwONoo1ugOyNCpw9xdl5MyiZ9JRAbwauNgKmdXk 7sXqhCLcryWtRbML/9cyHP9OMLHOxSGNUVCW8mhKcdYKwwITB7YgVeZpwTKoqKpii6vbIsvA K41WlPJg9sVllC3xQhXBeBjFsKeF8dZogH3T0sOeHvY1MLN1WQlYmYn6TAtRZCjhTIAysmzv vtyztXIxuBELGgxPCxUHkRYVS11m6nDD175VxtChurxA4Bncc3TZqS3CUoVgcwZ29LCrh5da GFP6djBjthVMW/E6czUtJoC1Q1qYVuNrZWlaFY2m2lJ5dWcZu2giirwjaP8XEwErI0/X3OLP pjUZV5pcSOomsIgTYior9AKWzgtML9bJsiAsI22rEZ0kiHVoHGg7INO9XcB7XYXAtfdnezrg uNqRIO2HmrHMt1yXrqb6uhU/D8ApFWNKKCwNIGHTsK17YlfTFbBY00Kw2HPuiQnl30Rs3c+a Zd3Nmk188E3FrqbTIvFRm7UM6toj535Utjzas/F8U1sbUrYyTfOOTPc+2HGmJrmzHsss+47M mZM1sabFkqlB2+DQ1HBX7tl35ak1OzGBUDMlRrASjGZzruYF3Svjmjpb7ODHXN61m4amDWzE gmNbxtPExB7pOM14GpWqs2uy9TksgmrEc9UKP1X6StoBL/SeKuR0tRBCDDlLzcvmJHEwI2Hz kZguFht5u+1hKK9u/PVca6nHNZUPuGfo8aUeV+O7ri5zObLG+ikDyqeRVHXsaoYV7mmGbUZ0 o4akxf5ObRqoROq1zozycb3SLfpyGcvTjWQCj/fHGXgdaNbASiRJAieyA9v5us4gZh9HOyXp 5ynNJLtT/wlvrGbEuTfa6Du8bsEea4rWNZG7a4OZiDVuIVUtltora08QUi713ETVYym2bFHE yjlZvUtnZQ9JOXkpameVvzRJJsT6308QAE9ijlMslB8ommVjv2bqeRqwxcNa1ENyJNwsVS2W L7jmD7Xn4bYgn73eo1xaLN5vrbl9f9iiPIQ+PZ1+nK6Xl9ONnIIGcSpW8hZu7x1kTyFnCq0m EO7NLUTcVaTczgwrxhyIUdBOGyqvr19+XJ6lI8PWj+XT5VUUZpzzpYd5lNVzI6kPYfsZZBlx RIjFRFNNSJY+yfOSbPzEs2gQ5Flp2ePMdjn94/zb1/P19AQMPDPZrpc2TV4C4zwpEFyvtC+K vrx9eRLveH06/YOqISt9+UxLsHS8wVko5Lf3Gco/Xm/fT+9nkt6KOMmVz87Y2ejzx/Xy/nR5 Oy3e5bH7pJUZXl9r+en2f5frn7L2Pv5zuv7PIn15O32VhYu0JXJXw91Bdn7+fpu+RZ3ic7gV tlYGXuFRCSbzrgVC7moB+Gv5V+/c/Mvz6+mmus38G7cscn18CTkStN9xRqi+fNtNb6LbLk6v p+vzx0K+FTpzGuGqSJZLl3ROAJwx4I+BFQX8cRQB0Hx2IMpfdXq//IDDzr9tgRZfkRZocZPM Ogox+xbRKRQtflsoX/I/Lq+I+SKFU+KkSos4jeTVTasjg2484KBeGvT31PuDUsxI1iSHJK/1 x6aTsMpkac7j09Klno2OmxTRgX758+cbVJB0gvL+djo9fUetp0yC3R6Tayug4Y95vW2CSGQx mJWWRYZZLUfSfVzW1Zw0zPmcKE6iOtvdkSbHek6a3YlJiRZHsnJX7Gel9bGsZoUjJwvSMpBH wMEHAQIwpuWSwq1iKeYlVoe0jSIz/sBcb2HErImTcnApbqxGrqZZ7vvO0tKBWN0vraLpobBE w9q3VqOQKaFFU5EDjo/JFTYyOUCg0ioTS2Vi8KECYAYGifyeKnrrdir7er2cEUXKQyod7sZs SdjFCdx8xgoHVLQbrXSoVD5Zc1LFgdf3SiqUyvLTDjlJocHUT1mdKJmDPZS1VMSTGl0/1PWj dNhSFzUY0RdiMYZYowc5OHRpxYNXF1ZLUuk8kO48rBW6lMSiIo/TJIkIt0q2Bypivc1lvMnx De0G349veLMuNwE4GEVLa7XE5buEUGHkKX/kXKyS0E2qxBSvByXrQ4KcXHYigSKu1oq2IXUk Wa/r8XMTbJhpec6uWWcTWRh7nu1gfbVWAE7MHCPM9YJlrMVdewbXhAenbibW8EI4cfZGcFeP OzPhHVOLO/4c7k3wMorFPD2toCrw/eU0O9yLDSswdbhpWhqcx6aFfbwg3DY8PW7r07FdDV4v l7ZbaXF/dZjgdZo/Eov/Ds+4T/YoLb6PTM80dfDS0MBlLIIvNek8SKbwoqbNd50lx0nQdQi/ YyeED2kWmQaoY8hTCTz0KYEmaMdaH4jRK6gl5T3MXH0KaITsEldkniLWY9NHZQmrikftmNlG 4zF1uTsISv3lfB9g+9AURQjqA7pVEivw0Qg8Ub2TIGVNBCq1BMmT+qGodhQceUID6OBkFnE9 18QpGyFkUQsAuXo9+l7PNIm4VXuF66RqHjDxLyDbGI3nQZYmuXQITcPxPW8ysQQpiHWdtBVQ sG58BylJpkPgBmmEclb45F5NolVY4xFx/ymt+V6TESqZvWjugtXAhoU+JKgAFk213qUZ5pov FVEVQaZcLQDiQjKeDjkc+LED6QVlIgE7ySCbwJL4WQeWqYqCDlaAJa0M4klwMH/YgYBaxxEY XFlpvHfTMLI610EEivopblCaYLjXUXFrJQZGAZpPQ8MeRJmKufdsi3qXPDawZ0CaCd0+NA6w F+5WdSzJs+IBN3uRzOSjEQDIqeugmlSqjNpa3OHQygQvrCetqBNtSb46dNQb4esyfNrYasSR IoUMDvZQs1A84pOssiOj6avEimAnVtWpJoHP+LJI8sA0G4aPiVUCFR4DW/MiYP4WSE4crpcH 0e7TSWlKJrYGuJB8X6mDLvBkFO5rQuMfbauCJf2oRvQXlazgmjFoGqYEY2i9eSKQ5zfgEGYr lr8z6kFdmKzUqQV1UlGEGjXcKNuBMlFWFLA3HrjDQdVGyGBCFOvWhKiIiE0TyAavoy8vl9dF 9OPy9Kfy2w6HTcO+ZogxUVdFIp66ZLWCRMq+DOvq9JJjoNfW6QJEcZQsjf7wbds5kudv51eZ 39HBrSoEv/y8Pp2mSqwixeQgFoa+5dpodIPHRhoy42oNs7gPOVgU1QzaV6pXnuNbZbMlutjf BGD13rofomZ7bYCkddEp1nszhqCi54XUzXpXUKkxSTxmKmgYnRVpKByonZ8WUrgovzyfpBnX lKFFxU6LAzoVCFiscA0kjRW5TnCwiC5vpSbQiTYvTReBDT8w/J2waLBL09UJCrjOirJ8FAtX qtZZJUoBtD3Se7ncTm/Xy5NGRzoB+n9pBoXOThLlU6ypWoFK5u3lfXLrAH58fuEf77fTy6IQ HfL7+e1XOAZ7On8TX0NjjCzG1GPa8CpgM/5/wPRcU+hSLt3WVfK5K1b7uNhcxDteyaltK2o2 xaE76RP7cFEnOdY8RoHKpIJxNCBW/iQAUJJRR5tYvOdCWgZRMs7cxL5/KIc6okT2eUeYLLoE kr9uT2KEa5nBJ8mowE0QRw0ltWsF1OaxBavaXy3tYIJz5rpYS6uFO7IrYi5fYL/HKRbCsao6 SSUBms4vckiD7tbpWgop3J76JnGXlk5KXwAel/dieOHwGfsglj5VVekv9+/MQhaY+MIoZJHp GuOFL0bpFoRIyLUPskVQUnxYIQ+m604QHDEjEZHBMeY9uXjlWL478niFH6NPO9MwbUyOEPjE xbgAVq5rjqwIWnQM4ISOkWNgQy5e78QcbFEgDNz/+j5Q+eODXVAd0Os6j17nWStz9EwuS5bO koZfjsIvVza9LvSX5HllUflqhVcYctNsQufEH953sBUPC44mMb8DwMZHUSwqbQsraADgYCNJ sYJvfjd9n74pD/ZLohn6OQLPKqXYYaYk4IAfCF6DVlNk+OYEMy2fE6smgJltu0eahUNagn9q ODYNBtq04OXth5gWRt/Ut71+rRR9P71Imj4+voc7/O7LKlZLqvPXzvAHLsTVYpD6YWv7mBoe qJPjkVh7zcT4cKM4XNByXnbvHb9Tdj9e9rHUS8f9sw9A3OC0XZe+UC8jY8lI1nZTcmMretQX 1bf0Hco1PHJH6NqeQZ/pvbnrWCZ9drzRM7mEFPMKTd+znGp8Ee56Pk1kiRUg4HmUyXFvJdy3 zLNsfJUt+pWL9YFFL3KW+AgXgJXsVrLq1sA0f3p9+ujvyf8Dl5ZxzP9dZhldtMtV55fb5frv +Px+u57/+AlaAeRaXZkcK/vO71/eT79lIuLp6yK7XN4Wv4gUf11869/4jt6IU1mLocP472/j /YmeBjEc7iBvos1BG8Gx4o5LZsON6U2exzOgxObmvs1jVeimPoVrZzYpmp/4pFgz76X1xrYG zZPtz5fz1/PtY1pn8bbGR8jbGEZx7MRMbH+wD9B0Sec48Wz1r0lFu7gB/cbL6cv7z+vp5fR6 W/x8Pd8mH8kxJl/Ewd9tx46eiQ8LDuAO3jPE8mKy+oLo1J04RofF2T0Vi+78C2fqk/huNq6c IBNdDNumB2XMV4TfSSIr0ky2JrlKj5htmfgOBACivSxmFqKNy8RUgVcem9IKSlHrgWHgVSSo gJi4h+NFV8a1uNjooG/9iQfUKbXYCIl5z5xqeEzoguqKqBYWJejLmvjQ07Qsg2JiOWTb+Jip jrjtYJMHCeCb6O79UrvFo9otjmsTIkrX9C1sWhjlmYOUse4rwAQ7sWfAw/HOWK1wW2jXuizY 5HjQ3djERzGqagiZ1AVLwEWdTRnGbJdokrV9G2LMdHspmh8VpBiPCm3vfPpxfp0rMV4f5FGW 5pqsojDqQqGpirpz1flPtV0kSWC1L+sZVRdplj6IyOD/Jnb0r2KyGe9cYDJTn19NOdfTO4w/ 01KGrCRMsqQvkKN8sX02sZ2NeLYpwF1y+6ae6XzQYpRATmD2cvLNRq/HqHZtpCQk5dp1DKpL 9gpKVtPGze2VPfAvXC9/nV+0k0OWxnDsndYJOfbhx5U7dKT69PIGywFtfbPsuDI80qdZaeC7 1Vp8cDwqyGfccfM6JA9jH+wAlWm+KQvsPxPQuihG4aQKCbXfO7BE+vBr5wfxuAiv56/PmtMH CBoFKzM6Yp4BQGsO7I29WTCkcdEyMR5YCuHFTsXFoefOOyAsHLLQt5Vpgd4PZ7Yf6GHM+gOQ JFyzSRyBEePNFqH67wM6cQMIIkl3hknK0hL8BhEtC7UfqqUhGPbS3TksKaIaq6eIVp3U0k6j KrIMn3msMbeieGjWwS4h96sAioHjQPVdgOexgiacwIEfo5L2lL/vC9vHBf/5x7s82Rs+Q2sO T7nSxQOcijeWnzPJDo/qBov2PERfK4xYsyvyQMIyQRKru35pI/UHhCBLjo95wR3JFy7E2oNE FO5oWv8knGu50/RQqFrIWn3GDoVDwQhb2rd3QUGZ4f1FSB7ahqWq+XQFxhepFSv2k2exlZg6 KK0Ccru4z2PYXGf1VKtrOPPP46pIdQRhYiGYH+KUoa7S+fwribE2eJTH2n7iOcqCFMWDEFhV IsSOLvMDSY3X9GGs1wcQL/ZVlMjjvAL7ZVKVihmJO4T20R7daMNyLSpamS7dmphq97ieEASU skCp+dv5+acY/0ExfHLvAGFQjxNPDdtUze+P+edOptI6X1/kddH0yDeOMZlb3BTY8+46rZhU URDVTsgfWpUBVMo4isOAEJGkhJGDpe3I+UKgCFwcBdEWODzyQlrVi5Eny4C7gNxjgDPJJg1B SSad4TTYFMUmS/pMT2pUpL34JflLrHHez3CN09dK2rG8/YqqeLhoWsNZlpavAkQJHzHfyRLs ujqbiVXtc7i9EmNnUJbEdH0P4qgc9lTrMygByzET7ywjUWsiPvgBV/R+/1/ZszW3jfP6VzJ9 OmfmbDd2nDR56AMlUTYb3SJKiZ0XTTf1tplukk4u87X//iNAUeIFdPfMtNMaACleARAEAacR 2245RAI8KNwJ7SaoMCsnPDkClIQclHDCOj0UREuvpdiqFhQhSvK0V6Jh52F4hd6FTpobU8TC ub1ZRT1NPiXZ0iaG33G3FDmUCY6c9VqGCyUuIeC5ExZ7AiviSCzdiQSjQokqp7xLreqHLeu6 lvwyNY42OhzLT6bF1m+ikk9u4XmYchkfUigDJw8IpGsth63+pPP7qldnFBdkt2IOtAHRsOjL YECpDVtFkbF2rnO5dMZgBOAdNrq7FZZkqVOf3ECGemmL0wk8XcApIdW7KdEnGu8mV8N1+pOS yUvwk7ZGwUaTWzDpWm9eDYQe1gmLqxRlyrr1gnz6pIr3KE2oUlQYhEOG9cXTwWg8k2pkqIhM lSj8Qc6XXocQAMPmrKWRzN8iBkysbIOiVjfi9JCQw2zKOvxm5hEZ2zq/SSYHBx2nB6LgZuXZ pau6E7m1cTMfIDRAH0XmgsynM5CR38PFJOQQFXAlMBF5+xF/glskplFC40Cub5RnZRWi9Y+E sA1V88mp1xSxvaixXcstHesqL7vheuED7FskKJV2hddgBQl87yC8Ti5X7lJC2WTHfXFyb9XX vC3YThcZX8PdfbMDrOVSC4MHDzBt63lFjYiNYor1OuZpYKjiG8hQ1MknnnbwrJHaR0iDmZYc h6EJeuADFhHZVj0O2R/qsPZndp2hehFoF0LWF2dnx66AqQth50W5FZC61lJcMi+BjfpdFdPR P6vlnznr/qw6+pO5ZgiWIUeVcCDXPgn8Nk4taZ3xBhJwnJ8eU3hRw2EUcsW8u395Oj8/vfhj YYUBrjpkCMFANS/7ty9PR39TLUap79gCAHCJV2YuTGm+ziJHILR2KGvFh+vWQyl9uMhaO+7V JW8r+1OeFaIrm+AnxbU0wuOxm36t+ENiVzCCsI22y5xJDWNNglprSjt2xgGDQeES3CmpWdqO TlmgZ42gob0h9gHL/a8hv3aVVANSGreU3oupjVde/W6KPgajhizhQZMRFGOESTA+vp7mS0gD Gd16jwM4GlimixZLhTN4CNCleJ2SDSRL0IRSHY5ZuyP1vbEib2FMcFKvNAoQoZkCCvg3GFGV eBqzWkqf5NYJBa5hxW3tg1oMsBl0vO0TUR3obwruZnCo5L8laiAFYiw0uk0IIdCiA6hJcnZd 963TDdXQYA0ZGMR6AQ+xTA8YlTTHUJJ1mkH0KmZ4uhz9GemcDfWaVI1SJS/s1SmveiY3buMN TOs8KHGoAH4OVSZacFn+RdSScei7GtxqHUkH7pPGshWRdKAoOYH8JypvvU9wf0wnRHG7OvRV Z4bmr9ySda3QQJagy/Dtb7rNy4RHctzOo9yydcmVKjdqMKrSjyeT7Nx6bKgUldrSzmmo9Hll 4wGuqu0qBJ0Fa3sExjhkG3xJQzBeZQZZLZPRCXk+aXsEZUebgIKK6m4TbYFiTInr7awd1x3l GCHGfEgbezVJU0qqsyO2tbNaKbF47apKIXvQmxs5P51s88AA820dVKhhsRKOM974iIqW4JU3 dfDb1ujx94n/25UdCFu5NPLGti9qimERQCwLVFMZ1qJ0fOclOGK8jFwIU0o3SQuP8ciaTDsG 9PeF3YX3sYPIhqwumag+vvu+f37c//P+6fnru6BUKZT27Z4uR5yRmhDahhf+8BoL1ayeamsE JDJm6U4dISnObYhAV+QFELn1mldDMyhzepqpiQwmKoPZ9AEU1coDNI6+OtKE/ba7mA2jrl7B 2UiQfupqQOH9CsZ0sDwk1Fz7P/0mQaMnD39nhkdnvplz91XrvpqB38PavtEbYcBqxijcfnlv SSuI6jxUMly2yWlQkzc7KW82zkYbAZ7xcoRSGloqnOLCMlbNh8UJSl1QIfaGM3i5Mmwg/ZVT +dA38FzUA3oyFWHYOg8WtDV81TZDo63LYs2QZbIMQeBw5AKDEa3SxmVwKZ6lQKR04CpcOecL jVWHy64IbT8aKbu2DqGw7pwNitBaqZGVM0NjHaXqoToPU7teV1cEdfFt51zyqRMyc3rGwrPY ARHBqMG6cMcKf1Ik1PrUiPD44HalkFOO63dvr3+fv7Mx5tA/rE4+uGUmzAfbGcTF2B5IDubc 9ijzMMsoJl5brAVOQjQPs4hioi2wPbY8zCqKibbadn73MBcRzMVJrMxFdEQvTmL9uVjFvnP+ weuPkDVYdOwAuk6BxTL6fYXyhprJVAi6/gUNXtLgExocafspDT6jwR9o8EWk3ZGmLCJtWXiN uazF+dASsN6FQd4HpVuzKgSnXJ3EUgpedby3UwpPmLZWuhNZ164VRUHVtmachrecX4ZgoVrF 7NgAE6LqRRfpG9mkrm8vhdy4iL7L7YhAhfP6U/2M3PxfokZ59O3z3ff7x6+zvbFD1QdyOhRs Lf3Hij+e7x9fvx99fvxy9OVh//L16OkHOA049kpIyzuM1pTpuQfYzDAAfsGvQW0cue3KsnfV dWdKZ2rEKBvSGD5iEuLmRe2P+3/2f7zeP+yP7r7t776/YAPvNPw5bKO+GoOrVCcFyAQdWp71 KadPfxaZbIrI21SLKLthbb6ifQmyBFLdiSYS7otX8C4T70tUjU3LIVxGdoi07GUX3iUbS7U6 H+raPi6Ol9bQK+VBNIolgd9ZJMlqy1mGX1BUlPmoUmeCDIondeGoFsj/6puKjFJm7ijtAhv1 JXgRF+uFLiO17g5W4BLCXtuRA1yMHr66Kizpz1qEV904Jk2Nl1b2PZsND1dJXoO/j9ZadQZH KroIWws0y7dX1laYgdO1gZ68j8c/F24n9RnLrHOdO/co2//19vWr3rPuKCtdjFdSRPJ56CqB MMiF4dHoWyN6HciiTwxZFadAbZ8YEnwpP3au5GWhBjAcXIM50ERVPyQqBqZygOq6PITU6Y2i K0y/GFV7WHRhE8dJV7PZ/Kab2FK4I8whZMUvsiMGHasJlyuMCu4Ju5IJeLgok8y+5hl/zrUA YKh7uL+lbKwaL6rCyREF1/pjXQ9BtzZKgoTpp2DpHsEzorcfmkdvPj9+dR5Zw/Gmh4gvnVqC kXQqGjlswMmyY5JeJxiqAqxvdU1OkYMfrlnR8/lSRCNBhoGh5niOBbtRTM47Q2ugL0wQGuwC Fz2uYg7+iQeZHTTkkvNGn/vmeCxjZAJ6q40LVekkZTPJbxj7mYMc/c/LGNnh5f+OHt5e9z/3 6j/717v379//rz0ruq62U/Kl41suD+1M1cpoXvBxZ/22kpsbTaSYSX3TMNe86tFimvs4S2ta tbvMqTlyR6EqgFE88BGTl7ZQs/CbtoBLEGsgm0yRo+dL/KNqgSuVjg+RzGO4PFAfI3ikZtLR eVd/r8HXVhJlfXcAn7eJ31HI9QEkuo8IL/WWR5MqBUup5IIV4WV4m/YRGYczCWhyRzVw8Q5o I7VJstiQz4qOquA38wIkwPzV/BTFxCXOjr1KYOKin+BXh9wrxjnERaKkOlzh0K01gz3wtsVX P4RNc/YKids953WpNMwq3dERwFCYbJicZFcrFB8GjyJ11G12motJSsiRhJQeKTVHG9d9GMkK hVreV1rHQ6I2hl23rNn8K5q80ZmiXCJz2sjNHowjhxvRbTARp/8hjS4xj4MiSOs280jAbwMX ElDi2g0qUXvFDl6h4w2NtemqLbHcal9C586pBa7kh7iwgMhAbzxjMtQEJETm7TxY3Xrvvj3i Kazbv7x6u7e4zDpaHUM9AjjOIGPXyJdq7ydc2g6FtOPgvHAU9z2wyRPwIYvjkX9cY7zaQ2T6 yjfGJrTMOFtNksA6f2AWSkgneealKMPB2PBt1tv5AhAKu6iC81HROFkjtB6msJ39EBSheJzO PWALRnYd8efB5a4iU0e1TSoWJxcrzHsJyi/NoCADaCMOsDA9q5cHpnxiB3GSpMnjSMoF3/tC YEeY4zbxMjqt42izTm3vS76LaHCQk4LLw3q3OuA7erb6Tcp5ZI+KycPDk1928MdRhqNxxI44 xllb7EZriePEbsGHLFnTyopDBQ8WSJ9oiD3ZwTrE+Ay/oaG/k4uhWXeDT+BqlXXGbGdSG2q4 sq8ebg/YBTqIANUX8K612oZlq21sv2Z1r/aEtksFxcB1reglrYiacFvgiRRbEJAvzJdpzjiC M0kG0m/kFmRIMW0Zwzysw/H2/Hg+m/g4nn1c0Di9K6woQw4WXJk+nth9G7HwuUNtwk/+Igoe 2IUTje9ANY298au0mqj65StKaDpkLYsYr9KGHeBU4DdWQtZJPODGTgn6SyiIDym/pSAn0Jvp 8eQPDAYEiGLIVF7QiTRHv6KZVfeKLaN4idh25f7u7RleAweWT2BnlnODkiNK7oI7j0KAdLGQ SUDetfA+IdPQ2fVQe7cbuH0GNtfwWcklPgtFRkdxwOB+zkAcd1RTnwnAG8UM27wtCTQcJf0n j922IxutplFU/ZZm/hNRySJH3YlETVO9qw/TsEat3jJi7piodqxkUXUl9iDDbKF5MlhqM1sX +/HddPW5VecJPIHYl7mot3gxkhEGdqhm50NVHT6oufIhWg0CZfbajw0/mfiff/14fTq6e3re Hz09H33b//Nj/2xrl2Mo+Vw0tLchYlmxdsIxOuBlCOdOQsgZGJImxWUqmo2tXPuYsJDn6zAD Q9LWvtqfYSThdLfiN70BZ0UaSnQ+2mwW62orWQAz2bkj8LD28aW8P7Ej/ZAJibcQgWHFJV/n i+V52RdBb6u+oIFhSxr8NyAG9nLV854HBfCfcMWUETjruw23MyaYbGBuyuKRGDw4teYf4NZF z0ccyIzpbfXb6zcItHH3+XX/5Yg/3sEegge+/7l//XbEXl6e7u4RlX1+/WxFbZ3ylYUfImDp hqk/y+OmLnaLEzuskMnuwK9EsK8HrgopWTs92k8wKNXD0xf73Yz5RJKGY92F45B2kvhOEpQt 2htiFyThTGyJCpWMg9e0c4jcl2+xZivREBTflCzszJb6+LUubkKnqAN1+IU2PVkSY4PgMcQG iQyHDqCQPpraMwrZLY4zkVOVaUys6Bp5XDDYsbViEKj6OIktx62UrUJOkp2GMKGWF8QJF+G4 tiUkjCDBtqPMDF6enlFgJ8eGWesbtggaA8BBSslPKHpVexx5uljGkYuhTCI1Um2ApNA+uFu3 TvY4w+IaN4O0NZcDzjNkQzQRXLQUvv/xzY39bKSlJMSqjM0voKyqPWTVJyLckOqgHVak9Jab XBBLzyCC0Io+PtLClJW8KASLIn5XEPoIOaqvt/+echknhdt+uieAO6Whh78uuzMaeqhYxiUF Oxl4xmNlclrAXm7YLaF4SchsQO05DY/2Z5RPUUSsoOScaARvGx2El4SrzcqX8Qo1zYFRtEii 1XQ8XHzqzEOu9hEeWyIGHfuSgx5ObtguSuN0avLKgRBeTlTJaWXkcM8Q1HbrJoYdxbX93GWE na+WBN2Kgm3mKOCfH788PRxVbw9/7Z9NAEyqeaySEByjtWNRmZa3CQav7WkMKd41hlL2EZN2 LYkIvvAJ06aBBaJuwplAayqzQ4l5iIGUxRNWGrU+SkGNx4QkD0soMcabcQ9zE/YZ4viwzH1Q EeJQphzCK1lHHCCAIk0jKSVmkit4zLs5vzj9mab/H9rrnLoAkLuy5GDzALVR265+EcimT4qR RvaJS2bh1DnVO2RtT48vhpS3cAeImZTwDtR+53uZyg+T9x2N1dcH3I5oJNYVz4aG64cv+OYT 6hdzjPcUgoT+jWeLl6O/IfTT/ddHHT8OPfAcl8LRHiu1QenjuztV+OVPKKHIhu/7X+9/7B/e We+WOki7MBubWsffPcRLy2qBttbL69K7LFMQ8GlMN6KhMXlGw4e27jv3sa/BYjx2uxwA1V5I 0fgvwEeltx9pAxqswF4BbRXJiQ+UUhBQMBi2vGBbfUuV8qZza7zO/W+Y2GeZaLtdUWtvPsxt oh9LWqR+QARnLJJdw6Sl0Y0uVOLWe34E4/9g1+ppdNhvOyk6AMZolOoDavmVOr/AfLEnKtaO VxZ5YO8s7v96/vz86+j56e31/tE+i2nbkm1zSkTXcki77jhGzHc2M566+ceOMutYZYZWdm2V Nrshb+vSi7FhkxS8imAr3g1qpdnPEgwKYlHBbRdciIkuxDepgCsI++rWoKJgzxcSXnSlZbNN N9o1peW5RwHX/pBJXr92bQrhWnVSxV1F52hG6eLMpQhPkaolXT+4pU6cQEv6ZHrgTmQkUEyS J7tzoqjGrCLcHElYe8M6foAiEZFPf7BzlibhgTw9d16+9BkY7mE49T4zE0JfUrMqq8vDvVeK 0PQG3srmvlqOb5NdOOhXII9R+frlQI1KNmckva3nmh2oVbMFXxHtQPWLhpO1bG8B7P9Go5YP wwCGTUgr2NkqADL7MmCGdZu+TAIEOBKF9SbppwDmWurmDg3rW+GEsZsQiUIsSUxxWzISge+6 Kfo6Al+F233MtFg7UcHA/7AuaudgYEPhYuecLgAfPICy932SbuzAmRt85Gwu62xf0m0nOewM CjZcus4QEzwpSXAuSye0FfdyG02uHbY6JutUKPaOcqBljrcNvL1zruY1CG5xB4e/om+UPZE6 OhRINOYqAyjF3WiHV7ZYKWrHcwB+H+IDVeE+iJxY9uSEgpsjxydy0Elr67b94MUnSotbSAll v+prM9uWlmXWt8pGOCE2apGBJ5SQjs9Sn0J0kM7V5fIaDs2Ee1FdkeZ9pD//ee7VcP4Tl9x8 Y7YOn2b4AyNhcuBtdThmEE3UTdw1oUB/G9ARQCH/CxXSPlsVgQIA --OgqxwSJOaUobr8KG-- --jho1yZJdad60DJr+ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQJ8BAEBCgBmBQJWin3XXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFQTQyRTkzNjBCNzVERDZFODMyNTQ3RjdB NDMwOTA3NUYwNTk2MEY2AAoJEKQwkHXwWWD2QYQP/06hzbYsHnUpmBqgoejoplLu WRG9MqfDxM1aIX5fs30Xa5gjQNmyxG+8h+NyXLP7gTB0TGe2xOMFet17jyq2BF4r 1atLKCDY7gCUFR/fLrD380dUVZH0/pNiaNyETjPHNUxHO5eESLHa18WTYcTNGct+ ZazqBob/mldtrN8oJEEC3YiWcbcAc0Pb6730JwXzluaIlrZ72i32lkeXr4oqvYC0 snKbBCleHuUsz8OYjTpnV1Tef/iSxmEZV1W2xQAi93DeChdXB8WsHr/63xCLf3oJ AuWJny3ZLwofY8DfPhpuGZfD4PDLVPlszPBPZzw/mwDoh8lYhEDQNgUOdXxNOgor ePv59R7A4iJSSiQ2YRAFaqDGsDux65x+IZg7TFbaggbPXXTSCW3IV78V6cXty+Uz s3vJAUXsiesORleiqAJTOqTVXtFkx1rkU6S/7cucMje6+nV0KhgyGwuOhK/73aRz 7LTl3pMzavG3Gh9tsTimRpK4SxhaAnmzrw9WaKCEm3+C9gGv+KJn1TuV6eLsF1A1 LT8bNxuUnczNoM4gYlAIhq5VvcsQD78rZHgaxTxEKf40H0XA+3tmfvrlf/aJddqX byBBBhF3NfUYHbAoWwiz6N487fSIa7dw3KHGJkJmsg+FaaXFV6v7s6LC5leec0ko UX/slcHDJZuOynZIrFoA =gygm -----END PGP SIGNATURE----- --jho1yZJdad60DJr+-- From bfoster@redhat.com Mon Jan 4 09:53:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 3CBBA7F37 for ; Mon, 4 Jan 2016 09:53:01 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1DB318F804B for ; Mon, 4 Jan 2016 07:52:58 -0800 (PST) X-ASG-Debug-ID: 1451922776-04bdf0265044d910001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 7DiGfEVrGI5J9RrV (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 07:52:57 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id 9B641C1090AE for ; Mon, 4 Jan 2016 15:52:56 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u04Fquca019228 for ; Mon, 4 Jan 2016 10:52:56 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id A3B5C120068; Mon, 4 Jan 2016 10:52:55 -0500 (EST) From: Brian Foster To: xfs@oss.sgi.com Subject: [PATCH] xfs: fix kbuild regression due to log recovery crc verification Date: Mon, 4 Jan 2016 10:52:55 -0500 X-ASG-Orig-Subj: [PATCH] xfs: fix kbuild regression due to log recovery crc verification Message-Id: <1451922775-15502-1-git-send-email-bfoster@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1451922777 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 The auto kbuild infrastructure reports several failures on non-standard arches (i386, sh, arm) due to a regression in the recent log recovery crc verification (short write detection) patches: fs/built-in.o: In function `xlog_find_tail': >> xfs_log_recover.c:(.text+0x1c5db5): undefined reference to `__moddi3' xfs_log_recover.c:(.text+0x1c5e9d): undefined reference to `__moddi3' This is due to the introduction of 64-bit modulus operations that were 32-bit operations prior to the change. xlog_find_tail() used an int to store the record header start block for the head of the log. This was converted to an xfs_daddr_t as this is how the value is represented in the callee functions that handle record detection/verification, etc. This was included as a minor cleanup as there wasn't an obvious reason for using an int for this value in xlog_find_tail(). The previously implicit cast was likely safe as this value represents a log relative block number and the log can only be so big (2G). That said, since all of the surrounding functionality uses xfs_daddr_t, including the log buffer I/O helpers, update these calculations to use do_mod() and handle the 64-bit modulus correctly regardless of architecture. Signed-off-by: Brian Foster --- Compile tested with ARCH=i386 and spot tested with xfstests on x86-64. Brian fs/xfs/xfs_log_recover.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 5bf8076..26e67b4 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -1330,12 +1330,13 @@ xlog_find_tail( } else { hblks = 1; } - after_umount_blk = (rhead_blk + hblks + (int) - BTOBB(be32_to_cpu(rhead->h_len))) % log->l_logBBsize; + after_umount_blk = rhead_blk + hblks + BTOBB(be32_to_cpu(rhead->h_len)); + after_umount_blk = do_mod(after_umount_blk, log->l_logBBsize); tail_lsn = atomic64_read(&log->l_tail_lsn); if (*head_blk == after_umount_blk && be32_to_cpu(rhead->h_num_logops) == 1) { - umount_data_blk = (rhead_blk + hblks) % log->l_logBBsize; + umount_data_blk = rhead_blk + hblks; + umount_data_blk = do_mod(umount_data_blk, log->l_logBBsize); error = xlog_bread(log, umount_data_blk, 1, bp, &offset); if (error) goto done; -- 2.4.3 From bfoster@redhat.com Mon Jan 4 10:03:47 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C16677F37 for ; Mon, 4 Jan 2016 10:03:47 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 5CFFFAC002 for ; Mon, 4 Jan 2016 08:03:44 -0800 (PST) X-ASG-Debug-ID: 1451923422-04bdf0264f44dcc0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id xtIyNEWjI8aqhPJx (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 08:03:43 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id A453BA3B6E; Mon, 4 Jan 2016 16:03:42 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u04G3gd1021329; Mon, 4 Jan 2016 11:03:42 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id AE092120068; Mon, 4 Jan 2016 11:03:41 -0500 (EST) From: Brian Foster To: fstests@vger.kernel.org Cc: xfs@oss.sgi.com Subject: [PATCH v2] xfs: test XFS torn log write detection Date: Mon, 4 Jan 2016 11:03:41 -0500 X-ASG-Orig-Subj: [PATCH v2] xfs: test XFS torn log write detection Message-Id: <1451923421-16647-1-git-send-email-bfoster@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1451923423 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 XFS torn log write detection includes a mechanism to inject CRC errors into log records at runtime and shutdown the fs accordingly. This ensures that the CRC verification pass on the subsequent mount discovers an invalid record near the head of the log and considers it a torn write. This test runs a workload with error injection enabled and verifies that the subsequent mount is successful. The test repeats for several iterations using a random frequency factor for the error event each time. Signed-off-by: Brian Foster --- Here's a v2 of the XFS log crc verification test case now that the associated kernel code has been merged. The only change is the bugfix noted in the changelog below. Brian v2: - Update random failure frequency calculation to ensure factor is never 0. v1: http://article.gmane.org/gmane.comp.file-systems.fstests/1406 tests/xfs/350 | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/350.out | 2 ++ tests/xfs/group | 1 + 3 files changed, 88 insertions(+) create mode 100755 tests/xfs/350 create mode 100644 tests/xfs/350.out diff --git a/tests/xfs/350 b/tests/xfs/350 new file mode 100755 index 0000000..1a1de6f --- /dev/null +++ b/tests/xfs/350 @@ -0,0 +1,85 @@ +#! /bin/bash +# FS QA Test No. 350 +# +# Use the XFS log record CRC error injection mechanism to test torn writes to +# the log. The error injection mechanism writes an invalid CRC and shuts down +# the filesystem. The test verifies that a subsequent remount recovers the log +# and that the filesystem is consistent. +# +# Note that this test requires a DEBUG mode kernel. +# +#----------------------------------------------------------------------- +# Copyright (c) 2015 Red Hat, Inc. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + killall -9 fsstress > /dev/null 2>&1 + wait > /dev/null 2>&1 +} + +rm -f $seqres.full + +# get standard environment, filters and checks +. ./common/rc + +# real QA test starts here + +# Modify as appropriate. +_supported_fs xfs +_supported_os Linux +_require_xfs_sysfs $(_short_dev $SCRATCH_DEV)/log/log_badcrc_factor +_require_scratch + +echo "Silence is golden." + +_scratch_mkfs >> $seqres.full 2>&1 || _fail "mkfs failed" +_scratch_mount + +sdev=$(_short_dev $SCRATCH_DEV) + +for i in $(seq 1 5); do + # Enable error injection. Use a random bad crc factor up to 100 + # (increase this value to run fsstress longer). + factor=$((RANDOM % 100 + 1)) + echo iteration $i log_badcrc_factor: $factor >> $seqres.full 2>&1 + echo $factor > /sys/fs/xfs/$sdev/log/log_badcrc_factor + + # Run fsstress until the filesystem shuts down. It will shut down + # automatically when error injection triggers. + $FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 999999 >> $seqres.full 2>&1 + + # Verify that we can remount the fs. Log recovery should handle the torn + # write. + _scratch_unmount + _scratch_mount || _fail "failed to mount" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/350.out b/tests/xfs/350.out new file mode 100644 index 0000000..c725135 --- /dev/null +++ b/tests/xfs/350.out @@ -0,0 +1,2 @@ +QA output created by 350 +Silence is golden. diff --git a/tests/xfs/group b/tests/xfs/group index 634f7e7..5fbace9 100644 --- a/tests/xfs/group +++ b/tests/xfs/group @@ -232,3 +232,4 @@ 303 auto quick quota 304 auto quick quota 305 auto quota +350 auto log metadata -- 2.4.3 From dan.j.williams@intel.com Mon Jan 4 12:20:31 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id A51387F37 for ; Mon, 4 Jan 2016 12:20:31 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 87F4F304032 for ; Mon, 4 Jan 2016 10:20:28 -0800 (PST) X-ASG-Debug-ID: 1451931626-04cbb0101c1f6b20001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id UrH0UeOiO7FIipHx for ; Mon, 04 Jan 2016 10:20:27 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP; 04 Jan 2016 10:20:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,521,1444719600"; d="scan'208";a="883602987" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by orsmga002.jf.intel.com with ESMTP; 04 Jan 2016 10:20:26 -0800 Subject: [resend PATCH 0/3] fs, bdev: handle end of life From: Dan Williams X-ASG-Orig-Subj: [resend PATCH 0/3] fs, bdev: handle end of life To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , Jens Axboe , linux-fsdevel@vger.kernel.org, Jan Kara , Tejun Heo , Matthew Wilcox , Ross Zwisler Date: Mon, 04 Jan 2016 10:20:00 -0800 Message-ID: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1451931626 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Per Dave, resend to include the xfs list on the full set. These are against v4.4-rc5. --- As mentioned in [PATCH 1/3] "block, fs: reliably communicate bdev end-of-life", historically we have waited for filesystem specific heuristics to attempt to guess when a block device is gone. Sometimes this works, but in other cases the system can hang waiting for the fs to trigger its shutdown protocol. Now with DAX we need new actions, like unmapping all inodes, to be taken upon a shutdown event. Those actions need to be taken whether the shutdown event comes from the block device being torn down, or some other file system specific event. For now, the approach taken in the following patches only affects xfs and block drivers that are converted to use del_gendisk_queue(). We can add more filesystems and driver support over time. Note that 'bdi_gone' was chosen over 'shutdown' so as not to be confused with generic_shutdown_super() --- Dan Williams (3): block, fs: reliably communicate bdev end-of-life xfs: handle shutdown notifications writeback: fix false positive WARN in __mark_inode_dirty block/genhd.c | 87 +++++++++++++++++++++++++++++++++++------- drivers/block/brd.c | 3 - drivers/nvdimm/pmem.c | 3 - drivers/s390/block/dcssblk.c | 6 +-- fs/block_dev.c | 79 +++++++++++++++++++++++++++++++++----- fs/xfs/xfs_super.c | 9 ++++ include/linux/fs.h | 4 ++ include/linux/genhd.h | 1 mm/backing-dev.c | 7 +++ 9 files changed, 166 insertions(+), 33 deletions(-) From dan.j.williams@intel.com Mon Jan 4 12:20:40 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 18E4F7F50 for ; Mon, 4 Jan 2016 12:20:40 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 099DA304032 for ; Mon, 4 Jan 2016 10:20:39 -0800 (PST) X-ASG-Debug-ID: 1451931637-04cbb0101c1f6b30001-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id ShtXkN0OTyPUFxc9 for ; Mon, 04 Jan 2016 10:20:38 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP; 04 Jan 2016 10:20:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,521,1444719600"; d="scan'208";a="874409453" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by fmsmga001.fm.intel.com with ESMTP; 04 Jan 2016 10:20:37 -0800 Subject: [resend PATCH 2/3] xfs: handle shutdown notifications From: Dan Williams X-ASG-Orig-Subj: [resend PATCH 2/3] xfs: handle shutdown notifications To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, Dave Chinner , linux-nvdimm@lists.01.org Date: Mon, 04 Jan 2016 10:20:11 -0800 Message-ID: <20160104182011.24118.30446.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1451931637 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25825 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Force a filesystem shutdown when the backing device is known to be dead. I.e. blk_queue_enter() permanently returns -ENODEV. Cc: xfs@oss.sgi.com Suggested-by: Dave Chinner Signed-off-by: Dan Williams --- fs/block_dev.c | 3 ++- fs/xfs/xfs_super.c | 9 +++++++++ include/linux/fs.h | 2 ++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 739e43a37e64..7d6c66148948 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1806,7 +1806,7 @@ int __invalidate_device(struct block_device *bdev, bool kill_dirty) } EXPORT_SYMBOL(__invalidate_device); -static void generic_bdi_gone(struct super_block *sb) +void generic_bdi_gone(struct super_block *sb) { struct inode *inode, *_inode = NULL; @@ -1832,6 +1832,7 @@ static void generic_bdi_gone(struct super_block *sb) spin_unlock(&sb->s_inode_list_lock); iput(_inode); } +EXPORT_SYMBOL(generic_bdi_gone); void shutdown_partition(struct gendisk *disk, int partno) { diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index 36bd8825bfb0..63c36508e9db 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -1618,6 +1618,14 @@ xfs_fs_free_cached_objects( return xfs_reclaim_inodes_nr(XFS_M(sb), sc->nr_to_scan); } +static void +xfs_fs_bdi_gone( + struct super_block *sb) +{ + xfs_force_shutdown(XFS_M(sb), SHUTDOWN_DEVICE_REQ); + generic_bdi_gone(sb); +} + static const struct super_operations xfs_super_operations = { .alloc_inode = xfs_fs_alloc_inode, .destroy_inode = xfs_fs_destroy_inode, @@ -1632,6 +1640,7 @@ static const struct super_operations xfs_super_operations = { .show_options = xfs_fs_show_options, .nr_cached_objects = xfs_fs_nr_cached_objects, .free_cached_objects = xfs_fs_free_cached_objects, + .bdi_gone = xfs_fs_bdi_gone, }; static struct file_system_type xfs_fs_type = { diff --git a/include/linux/fs.h b/include/linux/fs.h index 0e201ed38045..b1e8e049e4b8 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2265,6 +2265,7 @@ extern struct super_block *freeze_bdev(struct block_device *); extern void emergency_thaw_all(void); extern int thaw_bdev(struct block_device *bdev, struct super_block *sb); extern int fsync_bdev(struct block_device *); +extern void generic_bdi_gone(struct super_block *sb); extern struct super_block *blockdev_superblock; @@ -2277,6 +2278,7 @@ static inline void bd_forget(struct inode *inode) {} static inline int sync_blockdev(struct block_device *bdev) { return 0; } static inline void kill_bdev(struct block_device *bdev) {} static inline void invalidate_bdev(struct block_device *bdev) {} +static inline void generic_bdi_gone(struct super_block *sb) {} static inline struct super_block *freeze_bdev(struct block_device *sb) { From dan.j.williams@intel.com Mon Jan 4 12:20:40 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 303487F51 for ; Mon, 4 Jan 2016 12:20:40 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 22D598F804B for ; Mon, 4 Jan 2016 10:20:37 -0800 (PST) X-ASG-Debug-ID: 1451931632-04cb6c3032429b60001-NocioJ Received: from mga04.intel.com ([192.55.52.120]) by cuda.sgi.com with ESMTP id gtgyJ4izzRnOUXDj for ; Mon, 04 Jan 2016 10:20:32 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.120 Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga104.fm.intel.com with ESMTP; 04 Jan 2016 10:20:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,521,1444719600"; d="scan'208";a="627873704" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by FMSMGA003.fm.intel.com with ESMTP; 04 Jan 2016 10:20:31 -0800 Subject: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life From: Dan Williams X-ASG-Orig-Subj: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , Jens Axboe , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler Date: Mon, 04 Jan 2016 10:20:05 -0800 Message-ID: <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: UNKNOWN[192.55.52.120] X-Barracuda-Start-Time: 1451931632 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25825 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 Historically we have waited for filesystem specific heuristics to attempt to guess when a block device is gone. Sometimes this works, but in other cases the system can hang waiting for the fs to trigger its shutdown protocol. The initial motivation for this investigation was to prevent DAX mappings (direct mmap access to persistent memory) from leaking past the lifetime of the hosting block device. However, Dave points out that these shutdown operations are needed in other scenarios. Quoting Dave: For example, if we detect a free space corruption during allocation, it is not safe to trust *any active mapping* because we can't trust that we having handed out the same block to multiple owners. Hence on such a filesystem shutdown, we have to prevent any new DAX mapping from occurring and invalidate all existing mappings as we cannot allow userspace to modify any data or metadata until we've resolved the corruption situation. The current block device shutdown sequence of del_gendisk + blk_cleanup_queue is problematic. We want to tell the fs after blk_cleanup_queue that there is no possibility of recovery, but by that time we have deleted partitions and lost the ability to find all the super-blocks on a block device. Introduce del_gendisk_queue to trigger ->quiesce() and ->bdi_gone() notifications to all the filesystems hosted on the disk. Where ->quiesce() are 'shutdown' operations while the bdev may still be alive, and ->bdi_gone() is a set of actions to take after the backing device is known to be permanently dead. generic_quiesce_super and generic_bdi_gone, are the default operations when a filesystem does not implement ->quiesce(), ->bdi_gone(). They invalidate inodes and unmap DAX-inodes respectively. For now only ->bdi_gone() has an associated super operation as xfs will implement ->bdi_gone() in a later patch. Cc: Jan Kara Cc: Jens Axboe Cc: Matthew Wilcox Cc: Ross Zwisler Suggested-by: Dave Chinner Signed-off-by: Dan Williams --- block/genhd.c | 87 +++++++++++++++++++++++++++++++++++------- drivers/block/brd.c | 3 - drivers/nvdimm/pmem.c | 3 - drivers/s390/block/dcssblk.c | 6 +-- fs/block_dev.c | 78 ++++++++++++++++++++++++++++++++------ include/linux/fs.h | 2 + include/linux/genhd.h | 1 7 files changed, 147 insertions(+), 33 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index e5cafa51567c..967fcfd63c98 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -634,24 +634,14 @@ void add_disk(struct gendisk *disk) } EXPORT_SYMBOL(add_disk); -void del_gendisk(struct gendisk *disk) +static void del_gendisk_start(struct gendisk *disk) { - struct disk_part_iter piter; - struct hd_struct *part; - blk_integrity_del(disk); disk_del_events(disk); +} - /* invalidate stuff */ - disk_part_iter_init(&piter, disk, - DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); - while ((part = disk_part_iter_next(&piter))) { - invalidate_partition(disk, part->partno); - delete_partition(disk, part->partno); - } - disk_part_iter_exit(&piter); - - invalidate_partition(disk, 0); +static void del_gendisk_end(struct gendisk *disk) +{ set_capacity(disk, 0); disk->flags &= ~GENHD_FL_UP; @@ -670,9 +660,78 @@ void del_gendisk(struct gendisk *disk) pm_runtime_set_memalloc_noio(disk_to_dev(disk), false); device_del(disk_to_dev(disk)); } + +#define for_each_part(part, piter) \ + for (part = disk_part_iter_next(piter); part; \ + part = disk_part_iter_next(piter)) +void del_gendisk(struct gendisk *disk) +{ + struct disk_part_iter piter; + struct hd_struct *part; + + del_gendisk_start(disk); + + /* invalidate stuff */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) { + invalidate_partition(disk, part->partno); + delete_partition(disk, part->partno); + } + disk_part_iter_exit(&piter); + invalidate_partition(disk, 0); + + del_gendisk_end(disk); +} EXPORT_SYMBOL(del_gendisk); /** + * del_gendisk_queue - combined del_gendisk + blk_cleanup_queue + * @disk: disk to delete, invalidate, unmap, and end i/o + * + * This alternative for open coded calls to: + * del_gendisk() + * blk_cleanup_queue() + * ...is for notifying the filesystem that the block device has gone + * away. This distinction is important for triggering a filesystem to + * abort its error recovery and for (DAX) capable devices. DAX bypasses + * page cache and mappings go directly to storage media. When such a + * disk is removed the pfn backing a mapping may be invalid or removed + * from the system. Upon return accessing DAX mappings of this disk + * will trigger SIGBUS. + */ +void del_gendisk_queue(struct gendisk *disk) +{ + struct disk_part_iter piter; + struct hd_struct *part; + + del_gendisk_start(disk); + + /* pass1 sync fs + evict idle inodes */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) + invalidate_partition(disk, part->partno); + disk_part_iter_exit(&piter); + invalidate_partition(disk, 0); + + blk_cleanup_queue(disk->queue); + + /* pass2 the queue is dead, trigger fs shutdown via ->bdi_gone() */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) { + shutdown_partition(disk, part->partno); + delete_partition(disk, part->partno); + } + disk_part_iter_exit(&piter); + shutdown_partition(disk, 0); + + del_gendisk_end(disk); +} +EXPORT_SYMBOL(del_gendisk_queue); + +/** * get_gendisk - get partitioning information for a given device * @devt: device to get partitioning information for * @partno: returned partition index diff --git a/drivers/block/brd.c b/drivers/block/brd.c index a5880f4ab40e..f149dd57bba3 100644 --- a/drivers/block/brd.c +++ b/drivers/block/brd.c @@ -532,7 +532,6 @@ out: static void brd_free(struct brd_device *brd) { put_disk(brd->brd_disk); - blk_cleanup_queue(brd->brd_queue); brd_free_pages(brd); kfree(brd); } @@ -560,7 +559,7 @@ out: static void brd_del_one(struct brd_device *brd) { list_del(&brd->brd_list); - del_gendisk(brd->brd_disk); + del_gendisk_queue(brd->brd_disk); brd_free(brd); } diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 8ee79893d2f5..6dd06e9d34b0 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -158,9 +158,8 @@ static void pmem_detach_disk(struct pmem_device *pmem) if (!pmem->pmem_disk) return; - del_gendisk(pmem->pmem_disk); + del_gendisk_queue(pmem->pmem_disk); put_disk(pmem->pmem_disk); - blk_cleanup_queue(pmem->pmem_queue); } static int pmem_attach_disk(struct device *dev, diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c index 94a8f4ab57bc..0c3c968b57d9 100644 --- a/drivers/s390/block/dcssblk.c +++ b/drivers/s390/block/dcssblk.c @@ -388,8 +388,7 @@ removeseg: } list_del(&dev_info->lh); - del_gendisk(dev_info->gd); - blk_cleanup_queue(dev_info->dcssblk_queue); + del_gendisk_queue(dev_info->gd); dev_info->gd->queue = NULL; put_disk(dev_info->gd); up_write(&dcssblk_devices_sem); @@ -751,8 +750,7 @@ dcssblk_remove_store(struct device *dev, struct device_attribute *attr, const ch } list_del(&dev_info->lh); - del_gendisk(dev_info->gd); - blk_cleanup_queue(dev_info->dcssblk_queue); + del_gendisk_queue(dev_info->gd); dev_info->gd->queue = NULL; put_disk(dev_info->gd); device_unregister(&dev_info->dev); diff --git a/fs/block_dev.c b/fs/block_dev.c index 44d4a1e9244e..739e43a37e64 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1778,27 +1778,83 @@ fail: } EXPORT_SYMBOL(lookup_bdev); +static int generic_quiesce_super(struct super_block *sb, bool kill_dirty) +{ + /* + * no need to lock the super, get_super holds the read mutex so + * the filesystem cannot go away under us (->put_super runs with + * the write lock held). + */ + shrink_dcache_sb(sb); + return invalidate_inodes(sb, kill_dirty); +} + int __invalidate_device(struct block_device *bdev, bool kill_dirty) { struct super_block *sb = get_super(bdev); int res = 0; - if (sb) { - /* - * no need to lock the super, get_super holds the - * read mutex so the filesystem cannot go away - * under us (->put_super runs with the write lock - * hold). - */ - shrink_dcache_sb(sb); - res = invalidate_inodes(sb, kill_dirty); - drop_super(sb); - } + if (!sb) + goto out; + + res = generic_quiesce_super(sb, kill_dirty); + drop_super(sb); + out: invalidate_bdev(bdev); + return res; } EXPORT_SYMBOL(__invalidate_device); +static void generic_bdi_gone(struct super_block *sb) +{ + struct inode *inode, *_inode = NULL; + + spin_lock(&sb->s_inode_list_lock); + list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { + spin_lock(&inode->i_lock); + if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) + || !IS_DAX(inode)) { + spin_unlock(&inode->i_lock); + continue; + } + __iget(inode); + spin_unlock(&inode->i_lock); + spin_unlock(&sb->s_inode_list_lock); + + unmap_mapping_range(inode->i_mapping, 0, 0, 1); + iput(_inode); + _inode = inode; + cond_resched(); + + spin_lock(&sb->s_inode_list_lock); + } + spin_unlock(&sb->s_inode_list_lock); + iput(_inode); +} + +void shutdown_partition(struct gendisk *disk, int partno) +{ + struct block_device *bdev = bdget_disk(disk, partno); + struct super_block *sb = get_super(bdev); + + if (!bdev) + return; + + if (!sb) { + bdput(bdev); + return; + } + + if (sb->s_op->bdi_gone) + sb->s_op->bdi_gone(sb); + else + generic_bdi_gone(sb); + + drop_super(sb); + bdput(bdev); +} + void iterate_bdevs(void (*func)(struct block_device *, void *), void *arg) { struct inode *inode, *old_inode = NULL; diff --git a/include/linux/fs.h b/include/linux/fs.h index 3aa514254161..0e201ed38045 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1713,6 +1713,7 @@ struct super_operations { struct shrink_control *); long (*free_cached_objects)(struct super_block *, struct shrink_control *); + void (*bdi_gone)(struct super_block *); }; /* @@ -2390,6 +2391,7 @@ extern int revalidate_disk(struct gendisk *); extern int check_disk_change(struct block_device *); extern int __invalidate_device(struct block_device *, bool); extern int invalidate_partition(struct gendisk *, int); +extern void shutdown_partition(struct gendisk *, int); #endif unsigned long invalidate_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t end); diff --git a/include/linux/genhd.h b/include/linux/genhd.h index 847cc1d91634..028cf15a8a57 100644 --- a/include/linux/genhd.h +++ b/include/linux/genhd.h @@ -431,6 +431,7 @@ extern void part_round_stats(int cpu, struct hd_struct *part); /* block/genhd.c */ extern void add_disk(struct gendisk *disk); extern void del_gendisk(struct gendisk *gp); +extern void del_gendisk_queue(struct gendisk *disk); extern struct gendisk *get_gendisk(dev_t dev, int *partno); extern struct block_device *bdget_disk(struct gendisk *disk, int partno); From dan.j.williams@intel.com Mon Jan 4 12:20:44 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id CAC257F51 for ; Mon, 4 Jan 2016 12:20:44 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id BC0CB8F8035 for ; Mon, 4 Jan 2016 10:20:44 -0800 (PST) X-ASG-Debug-ID: 1451931637-04cbb0101c1f6b30002-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id sGs31rhbZLgpBR15 for ; Mon, 04 Jan 2016 10:20:42 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP; 04 Jan 2016 10:20:42 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,521,1444719600"; d="scan'208";a="720045429" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by orsmga003.jf.intel.com with ESMTP; 04 Jan 2016 10:20:42 -0800 Subject: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty From: Dan Williams X-ASG-Orig-Subj: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , Jens Axboe , linux-fsdevel@vger.kernel.org, Jan Kara , Tejun Heo Date: Mon, 04 Jan 2016 10:20:16 -0800 Message-ID: <20160104182016.24118.33718.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1451931639 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25825 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- This warning was added as a debugging aid way back in commit 500b067c5e6c "writeback: check for registered bdi in flusher add and inode dirty" when we were switching over to per-bdi writeback. Once the block device has been torn down it's no longer useful to complain about unregistered bdi's. Clear the writeback capability under the the wb->list_lock(), so that __mark_inode_dirty has no opportunity to race bdi_unregister() to this WARN() condition. Alternatively we could just delete the warning... Found this while testing block device remove from underneath an active fs triggering traces like: WARNING: CPU: 6 PID: 2129 at fs/fs-writeback.c:2065 __mark_inode_dirty+0x261/0x350() bdi-block not registered [..] Call Trace: [] dump_stack+0x44/0x62 [] warn_slowpath_common+0x82/0xc0 [] warn_slowpath_fmt+0x5c/0x80 [] __mark_inode_dirty+0x261/0x350 [] generic_update_time+0x79/0xd0 [] file_update_time+0xbd/0x110 [] ext4_dax_fault+0x68/0x110 [] __do_fault+0x4e/0xf0 [] handle_mm_fault+0x5e7/0x1b50 [] ? handle_mm_fault+0x51/0x1b50 [] __do_page_fault+0x191/0x3f0 [] trace_do_page_fault+0x4f/0x120 [] do_async_page_fault+0x1a/0xa0 [] async_page_fault+0x28/0x30 Cc: Jan Kara Cc: Tejun Heo Cc: Jens Axboe Cc: Dave Chinner Signed-off-by: Dan Williams --- mm/backing-dev.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/backing-dev.c b/mm/backing-dev.c index 7340353f8aea..9ea843bca709 100644 --- a/mm/backing-dev.c +++ b/mm/backing-dev.c @@ -343,10 +343,17 @@ static void wb_shutdown(struct bdi_writeback *wb) { /* Make sure nobody queues further work */ spin_lock_bh(&wb->work_lock); + if (!test_and_clear_bit(WB_registered, &wb->state)) { spin_unlock_bh(&wb->work_lock); return; } + + /* tell __mark_inode_dirty that writeback is no longer possible */ + spin_lock(&wb->list_lock); + wb->bdi->capabilities |= BDI_CAP_NO_WRITEBACK; + spin_unlock(&wb->list_lock); + spin_unlock_bh(&wb->work_lock); /* From bfoster@redhat.com Mon Jan 4 13:11:55 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0ADA27F37 for ; Mon, 4 Jan 2016 13:11:55 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id F03DB8F8040 for ; Mon, 4 Jan 2016 11:11:51 -0800 (PST) X-ASG-Debug-ID: 1451934710-04cbb0101d1f8550001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id JpreCJ1UNyyFmqzF (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:11:50 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id F15508E900; Mon, 4 Jan 2016 19:11:49 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u04JBnus004581; Mon, 4 Jan 2016 14:11:49 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id C77E2120068; Mon, 4 Jan 2016 14:11:48 -0500 (EST) Date: Mon, 4 Jan 2016 14:11:48 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 1/9] metadump: clean up btree block region zeroing Message-ID: <20160104191147.GA19852@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 1/9] metadump: clean up btree block region zeroing References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-2-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-2-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1451934710 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:01AM +1100, Dave Chinner wrote: > From: Dave Chinner > > Abstract out all the common operations in the btree block zeroing > so that we only have one copy of offset/len calculations and don't > require lots of casts for the pointer arithmetic. > > Signed-off-by: Dave Chinner > --- Reviewed-by: Brian Foster > db/metadump.c | 61 +++++++++++++++++++++++------------------------------------ > 1 file changed, 24 insertions(+), 37 deletions(-) > > diff --git a/db/metadump.c b/db/metadump.c > index 8455fdd..a185da5 100644 > --- a/db/metadump.c > +++ b/db/metadump.c > @@ -258,8 +258,8 @@ zero_btree_node( > xfs_inobt_key_t *ikp; > xfs_alloc_ptr_t *app; > xfs_alloc_key_t *akp; > - void *zp1, *zp2; > - int zlen1, zlen2; > + char *zp1, *zp2; > + char *key_end; > > nrecs = be16_to_cpu(block->bb_numrecs); > > @@ -268,43 +268,36 @@ zero_btree_node( > case TYP_BMAPBTD: > bkp = XFS_BMBT_KEY_ADDR(mp, block, 1); > bpp = XFS_BMBT_PTR_ADDR(mp, block, 1, mp->m_bmap_dmxr[1]); > - zp1 = &bkp[nrecs]; > - zlen1 = (char *)&bpp[0] - (char *)&bkp[nrecs]; > - zp2 = &bpp[nrecs]; > - zlen2 = (char *)block + mp->m_sb.sb_blocksize - > - (char *)&bpp[nrecs]; > + zp1 = (char *)&bkp[nrecs]; > + zp2 = (char *)&bpp[nrecs]; > + key_end = (char *)bpp; > break; > case TYP_INOBT: > case TYP_FINOBT: > ikp = XFS_INOBT_KEY_ADDR(mp, block, 1); > ipp = XFS_INOBT_PTR_ADDR(mp, block, 1, mp->m_inobt_mxr[1]); > - zp1 = &ikp[nrecs]; > - zlen1 = (char *)&ipp[0] - (char *)&ikp[nrecs]; > - zp2 = &ipp[nrecs]; > - zlen2 = (char *)block + mp->m_sb.sb_blocksize - > - (char *)&ipp[nrecs]; > + zp1 = (char *)&ikp[nrecs]; > + zp2 = (char *)&ipp[nrecs]; > + key_end = (char *)ipp; > break; > case TYP_BNOBT: > case TYP_CNTBT: > akp = XFS_ALLOC_KEY_ADDR(mp, block, 1); > app = XFS_ALLOC_PTR_ADDR(mp, block, 1, mp->m_alloc_mxr[1]); > - zp1 = &akp[nrecs]; > - zlen1 = (char *)&app[0] - (char *)&akp[nrecs]; > - zp2 = &app[nrecs]; > - zlen2 = (char *)block + mp->m_sb.sb_blocksize - > - (char *)&app[nrecs]; > + zp1 = (char *)&akp[nrecs]; > + zp2 = (char *)&app[nrecs]; > + key_end = (char *)app; > break; > default: > - zp1 = NULL; > - break; > + return; > } > > - if (zp1 && zp2) { > - /* Zero from end of keys to beginning of pointers */ > - memset(zp1, 0, zlen1); > - /* Zero from end of pointers to end of block */ > - memset(zp2, 0, zlen2); > - } > + > + /* Zero from end of keys to beginning of pointers */ > + memset(zp1, 0, key_end - zp1); > + > + /* Zero from end of pointers to end of block */ > + memset(zp2, 0, (char *)block + mp->m_sb.sb_blocksize - zp2); > } > > static void > @@ -316,8 +309,7 @@ zero_btree_leaf( > struct xfs_bmbt_rec *brp; > struct xfs_inobt_rec *irp; > struct xfs_alloc_rec *arp; > - void *zp; > - int zlen; > + char *zp; > > nrecs = be16_to_cpu(block->bb_numrecs); > > @@ -325,29 +317,24 @@ zero_btree_leaf( > case TYP_BMAPBTA: > case TYP_BMAPBTD: > brp = XFS_BMBT_REC_ADDR(mp, block, 1); > - zp = &brp[nrecs]; > - zlen = (char *)block + mp->m_sb.sb_blocksize - (char *)&brp[nrecs]; > + zp = (char *)&brp[nrecs]; > break; > case TYP_INOBT: > case TYP_FINOBT: > irp = XFS_INOBT_REC_ADDR(mp, block, 1); > - zp = &irp[nrecs]; > - zlen = (char *)block + mp->m_sb.sb_blocksize - (char *)&irp[nrecs]; > + zp = (char *)&irp[nrecs]; > break; > case TYP_BNOBT: > case TYP_CNTBT: > arp = XFS_ALLOC_REC_ADDR(mp, block, 1); > - zp = &arp[nrecs]; > - zlen = (char *)block + mp->m_sb.sb_blocksize - (char *)&arp[nrecs]; > + zp = (char *)&arp[nrecs]; > break; > default: > - zp = NULL; > - break; > + return; > } > > /* Zero from end of records to end of block */ > - if (zp && zlen < mp->m_sb.sb_blocksize) > - memset(zp, 0, zlen); > + memset(zp, 0, (char *)block + mp->m_sb.sb_blocksize - zp); > } > > static void > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Mon Jan 4 13:12:02 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id C936F7F37 for ; Mon, 4 Jan 2016 13:12:02 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id B7528304062 for ; Mon, 4 Jan 2016 11:11:59 -0800 (PST) X-ASG-Debug-ID: 1451934718-04cb6c303142adb0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 6ZF7xEFmkMmpe24q (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:11:58 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id E3CEF1EB24; Mon, 4 Jan 2016 19:11:57 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u04JBve1021358; Mon, 4 Jan 2016 14:11:57 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id BDA83120068; Mon, 4 Jan 2016 14:11:56 -0500 (EST) Date: Mon, 4 Jan 2016 14:11:56 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 2/9] metadump: bounds check btree block regions being zeroed Message-ID: <20160104191156.GB19852@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 2/9] metadump: bounds check btree block regions being zeroed References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-3-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-3-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1451934718 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:02AM +1100, Dave Chinner wrote: > From: Dave Chinner > > Arkadiusz Miskiewicz reported that metadump was crashing on one of > his corrupted filesystems, and the trace indicated that it was > zeroing unused regions in inode btree blocks when it failed. The > btree block had a corrupt nrecs field, which was resulting in an out > of bounds memset() occurring. Ensure that the region being > generated for zeroing is within bounds before executing the zeroing. > > Reported-by: Arkadiusz Miskiewicz > Signed-off-by: Dave Chinner > --- > db/metadump.c | 32 ++++++++++++++++++++++++++++++++ > 1 file changed, 32 insertions(+) > > diff --git a/db/metadump.c b/db/metadump.c > index a185da5..1769fdf 100644 > --- a/db/metadump.c > +++ b/db/metadump.c ... > @@ -300,6 +316,11 @@ zero_btree_node( > memset(zp2, 0, (char *)block + mp->m_sb.sb_blocksize - zp2); > } > > +/* > + * We could be processing a corrupt block, so we can't trust any of > + * the offsets or lengths to be within the buffer range. Hence check > + * carefully! > + */ > static void > zero_btree_leaf( > struct xfs_btree_block *block, > @@ -312,20 +333,31 @@ zero_btree_leaf( > char *zp; > > nrecs = be16_to_cpu(block->bb_numrecs); > + if (nrecs < 0) > + return; > > switch (btype) { > case TYP_BMAPBTA: > case TYP_BMAPBTD: > + if (nrecs > mp->m_bmap_dmxr[1]) > + return; > + Shouldn't we use the 0 index max recs value (for leaf blocks) throughout this function? (e.g, mp->m_bmap_dmxr[0]) Brian > brp = XFS_BMBT_REC_ADDR(mp, block, 1); > zp = (char *)&brp[nrecs]; > break; > case TYP_INOBT: > case TYP_FINOBT: > + if (nrecs > mp->m_inobt_mxr[1]) > + return; > + > irp = XFS_INOBT_REC_ADDR(mp, block, 1); > zp = (char *)&irp[nrecs]; > break; > case TYP_BNOBT: > case TYP_CNTBT: > + if (nrecs > mp->m_alloc_mxr[1]) > + return; > + > arp = XFS_ALLOC_REC_ADDR(mp, block, 1); > zp = (char *)&arp[nrecs]; > break; > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Mon Jan 4 13:12:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 16CA77F37 for ; Mon, 4 Jan 2016 13:12:10 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id B7F13AC002 for ; Mon, 4 Jan 2016 11:12:09 -0800 (PST) X-ASG-Debug-ID: 1451934725-04bdf0264f451fd0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id nFGBa0XyZb03WZax (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:12:05 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 79A6AA8D; Mon, 4 Jan 2016 19:12:05 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u04JC5BW003128; Mon, 4 Jan 2016 14:12:05 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 7305B120068; Mon, 4 Jan 2016 14:12:04 -0500 (EST) Date: Mon, 4 Jan 2016 14:12:04 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 3/9] xfs_mdrestore: correctly account bytes read Message-ID: <20160104191204.GC19852@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 3/9] xfs_mdrestore: correctly account bytes read References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-4-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-4-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1451934725 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:03AM +1100, Dave Chinner wrote: > From: Dave Chinner > > Progess indication comes in the form of a "X MB read" output. This > doesn't match up with the actual number of bytes read from the > metadump file because it only accounts header blocks in the file, > not actual metadata blocks that are restored, Hence the number > reported is usually much lower than the size of the metadump file, > hence it's impossible to use to guage progress of the restore. gauge Otherwise looks good: Reviewed-by: Brian Foster > > While there, fix the progress output so that it overwrites the > previous progress output line correctly. > > Signed-off-by: Dave Chinner > --- > mdrestore/xfs_mdrestore.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/mdrestore/xfs_mdrestore.c b/mdrestore/xfs_mdrestore.c > index ebc5e54..70a160c 100644 > --- a/mdrestore/xfs_mdrestore.c > +++ b/mdrestore/xfs_mdrestore.c > @@ -133,7 +133,7 @@ perform_restore( > > for (;;) { > if (show_progress && (bytes_read & ((1 << 20) - 1)) == 0) > - print_progress("%lld MB read\n", bytes_read >> 20); > + print_progress("%lld MB read", bytes_read >> 20); > > for (cur_index = 0; cur_index < mb_count; cur_index++) { > if (pwrite64(dst_fd, &block_buffer[cur_index << > @@ -160,7 +160,7 @@ perform_restore( > 1, src_f) != 1) > fatal("error reading from file: %s\n", strerror(errno)); > > - bytes_read += block_size; > + bytes_read += block_size + (mb_count << tmb.mb_blocklog); > } > > if (progress_since_warning) > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Mon Jan 4 13:12:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 564387F58 for ; Mon, 4 Jan 2016 13:12:13 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id DAA29AC002 for ; Mon, 4 Jan 2016 11:12:12 -0800 (PST) X-ASG-Debug-ID: 1451934730-04cbb0101d1f8580001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id TRWRGYW30vWrBOAA (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:12:11 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id C21AFC0B7E03; Mon, 4 Jan 2016 19:12:10 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u04JCA8w004730; Mon, 4 Jan 2016 14:12:10 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id AA443120068; Mon, 4 Jan 2016 14:12:09 -0500 (EST) Date: Mon, 4 Jan 2016 14:12:09 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 4/9] repair: parallelise phase 7 Message-ID: <20160104191209.GD19852@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 4/9] repair: parallelise phase 7 References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-5-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-5-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1451934731 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:04AM +1100, Dave Chinner wrote: > From: Dave Chinner > > It operates on a single AG at a time, sequentially, doing inode updates. All the > data structures accessed and modified are per AG, as are the modification to > on-disk structures. Hence we can run this phase concurrently across multiple > AGs. > > This is important for large, broken filesystem repairs, where there can be > millions of inodes that need link counts updated. Once such repair image takes > more than 45 minutes to run phase 7 as a single threaded operation. > > Signed-off-by: Dave Chinner > --- Reviewed-by: Brian Foster > repair/phase7.c | 77 ++++++++++++++++++++++++++++++++++------------------- > repair/progress.c | 4 +-- > repair/protos.h | 2 +- > repair/xfs_repair.c | 2 +- > 4 files changed, 54 insertions(+), 31 deletions(-) > > diff --git a/repair/phase7.c b/repair/phase7.c > index b1e3a55..91dad02 100644 > --- a/repair/phase7.c > +++ b/repair/phase7.c > @@ -26,6 +26,7 @@ > #include "dinode.h" > #include "versions.h" > #include "progress.h" > +#include "threads.h" > > /* dinoc is a pointer to the IN-CORE dinode core */ > static void > @@ -108,45 +109,67 @@ update_inode_nlinks( > IRELE(ip); > } > > -void > -phase7(xfs_mount_t *mp) > +/* > + * for each ag, look at each inode 1 at a time. If the number of > + * links is bad, reset it, log the inode core, commit the transaction > + */ > +static void > +do_link_updates( > + struct work_queue *wq, > + xfs_agnumber_t agno, > + void *arg) > { > ino_tree_node_t *irec; > - int i; > int j; > __uint32_t nrefs; > > + irec = findfirst_inode_rec(agno); > + > + while (irec != NULL) { > + for (j = 0; j < XFS_INODES_PER_CHUNK; j++) { > + ASSERT(is_inode_confirmed(irec, j)); > + > + if (is_inode_free(irec, j)) > + continue; > + > + ASSERT(no_modify || is_inode_reached(irec, j)); > + > + nrefs = num_inode_references(irec, j); > + ASSERT(no_modify || nrefs > 0); > + > + if (get_inode_disk_nlinks(irec, j) != nrefs) > + update_inode_nlinks(wq->mp, > + XFS_AGINO_TO_INO(wq->mp, agno, > + irec->ino_startnum + j), > + nrefs); > + } > + irec = next_ino_rec(irec); > + } > + > + PROG_RPT_INC(prog_rpt_done[agno], 1); > +} > + > +void > +phase7( > + struct xfs_mount *mp, > + int scan_threads) > +{ > + struct work_queue wq; > + int agno; > + > if (!no_modify) > do_log(_("Phase 7 - verify and correct link counts...\n")); > else > do_log(_("Phase 7 - verify link counts...\n")); > > - /* > - * for each ag, look at each inode 1 at a time. If the number of > - * links is bad, reset it, log the inode core, commit the transaction > - */ > - for (i = 0; i < glob_agcount; i++) { > - irec = findfirst_inode_rec(i); > - > - while (irec != NULL) { > - for (j = 0; j < XFS_INODES_PER_CHUNK; j++) { > - ASSERT(is_inode_confirmed(irec, j)); > + set_progress_msg(PROGRESS_FMT_CORR_LINK, (__uint64_t) glob_agcount); > > - if (is_inode_free(irec, j)) > - continue; > + create_work_queue(&wq, mp, scan_threads); > > - ASSERT(no_modify || is_inode_reached(irec, j)); > + for (agno = 0; agno < mp->m_sb.sb_agcount; agno++) > + queue_work(&wq, do_link_updates, agno, NULL); > > - nrefs = num_inode_references(irec, j); > - ASSERT(no_modify || nrefs > 0); > + destroy_work_queue(&wq); > > - if (get_inode_disk_nlinks(irec, j) != nrefs) > - update_inode_nlinks(mp, > - XFS_AGINO_TO_INO(mp, i, > - irec->ino_startnum + j), > - nrefs); > - } > - irec = next_ino_rec(irec); > - } > - } > + print_final_rpt(); > } > diff --git a/repair/progress.c b/repair/progress.c > index 418b803..2a09b23 100644 > --- a/repair/progress.c > +++ b/repair/progress.c > @@ -75,9 +75,9 @@ progress_rpt_t progress_rpt_reports[] = { > {FMT2, N_("moving disconnected inodes to lost+found"), /* 12 */ > &rpt_fmts[FMT2], &rpt_types[TYPE_INODE]}, > {FMT1, N_("verify and correct link counts"), /* 13 */ > - &rpt_fmts[FMT1], &rpt_types[TYPE_INODE]}, > + &rpt_fmts[FMT1], &rpt_types[TYPE_AG]}, > {FMT1, N_("verify link counts"), /* 14 */ > - &rpt_fmts[FMT1], &rpt_types[TYPE_INODE]} > + &rpt_fmts[FMT1], &rpt_types[TYPE_AG]} > }; > > pthread_t report_thread; > diff --git a/repair/protos.h b/repair/protos.h > index 9d5a2a6..b113aca 100644 > --- a/repair/protos.h > +++ b/repair/protos.h > @@ -50,7 +50,7 @@ void phase3(struct xfs_mount *); > void phase4(struct xfs_mount *); > void phase5(struct xfs_mount *); > void phase6(struct xfs_mount *); > -void phase7(struct xfs_mount *); > +void phase7(struct xfs_mount *, int); > > int verify_set_agheader(struct xfs_mount *, struct xfs_buf *, > struct xfs_sb *, struct xfs_agf *, struct xfs_agi *, > diff --git a/repair/xfs_repair.c b/repair/xfs_repair.c > index 3eaced4..fcdb212 100644 > --- a/repair/xfs_repair.c > +++ b/repair/xfs_repair.c > @@ -893,7 +893,7 @@ main(int argc, char **argv) > phase6(mp); > timestamp(PHASE_END, 6, NULL); > > - phase7(mp); > + phase7(mp, phase2_threads); > timestamp(PHASE_END, 7, NULL); > } else { > do_warn( > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Mon Jan 4 13:12:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E7F947F37 for ; Mon, 4 Jan 2016 13:12:17 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 78810AC005 for ; Mon, 4 Jan 2016 11:12:17 -0800 (PST) X-ASG-Debug-ID: 1451934735-04cb6c303042ade0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id wTMLzirxCiYXbBCo (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:12:16 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 86FAFA8D; Mon, 4 Jan 2016 19:12:15 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u04JCF6K003173; Mon, 4 Jan 2016 14:12:15 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 6D88B120068; Mon, 4 Jan 2016 14:12:14 -0500 (EST) Date: Mon, 4 Jan 2016 14:12:14 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 5/9] repair: parallelise uncertin inode processing in phase 3 Message-ID: <20160104191213.GE19852@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 5/9] repair: parallelise uncertin inode processing in phase 3 References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-6-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-6-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1451934735 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:05AM +1100, Dave Chinner wrote: > From: Dave Chinner > > This can take a long time when there are millions of uncertain inodes in badly > broken filesystems. THe processing is per-ag, the data structures are all > per-ag, and the load is mostly CPU time spent checking CRCs on each > uncertaini inode. Parallelising reduced the runtime of this phase on a badly > broken filesytem from ~30 minutes to under 5 miniutes. > > Signed-off-by: Dave Chinner > --- This one seems a bit more scary simply because the amount of work involved in phase 3, such as inode processing and whatnot. I don't think that everything in there is necessarily AG local as the commit log description above implies. On a skim through, I do see that we have ag_locks[agno].lock for cases that can cross AG boundaries, such as checking block allocation state (get_bmap()/set_bmap()) of bmapbt blocks (iiuc?), for example. So I can't really spot any actual problems, but there's a lot of code down in there. I'm fine with it as long as testing bears out and this gets a reasonable amount of soak time such that we can hopefully catch any serious issues or areas currently lacking sufficient locking: Reviewed-by: Brian Foster > repair/phase3.c | 59 ++++++++++++++++++++++++++++++++++++++++++++--------- > repair/protos.h | 2 +- > repair/xfs_repair.c | 2 +- > 3 files changed, 51 insertions(+), 12 deletions(-) > > diff --git a/repair/phase3.c b/repair/phase3.c > index 76c9440..0890a27 100644 > --- a/repair/phase3.c > +++ b/repair/phase3.c > @@ -28,6 +28,7 @@ > #include "dinode.h" > #include "progress.h" > #include "bmap.h" > +#include "threads.h" > > static void > process_agi_unlinked( > @@ -87,10 +88,33 @@ process_ags( > do_inode_prefetch(mp, ag_stride, process_ag_func, false, false); > } > > +static void > +do_uncertain_aginodes( > + work_queue_t *wq, > + xfs_agnumber_t agno, > + void *arg) > +{ > + int *count = arg; > + > + *count = process_uncertain_aginodes(wq->mp, agno); > + > +#ifdef XR_INODE_TRACE > + fprintf(stderr, > + "\t\t phase 3 - ag %d process_uncertain_inodes returns %d\n", > + *count, j); > +#endif > + > + PROG_RPT_INC(prog_rpt_done[agno], 1); > +} > + > void > -phase3(xfs_mount_t *mp) > +phase3( > + struct xfs_mount *mp, > + int scan_threads) > { > - int i, j; > + int i, j; > + int *counts; > + work_queue_t wq; > > do_log(_("Phase 3 - for each AG...\n")); > if (!no_modify) > @@ -129,20 +153,35 @@ phase3(xfs_mount_t *mp) > */ > do_log(_(" - process newly discovered inodes...\n")); > set_progress_msg(PROG_FMT_NEW_INODES, (__uint64_t) glob_agcount); > + > + counts = calloc(sizeof(*counts), mp->m_sb.sb_agcount); > + if (!counts) { > + do_abort(_("no memory for uncertain inode counts\n")); > + return; > + } > + > do { > /* > * have to loop until no ag has any uncertain > * inodes > */ > j = 0; > - for (i = 0; i < mp->m_sb.sb_agcount; i++) { > - j += process_uncertain_aginodes(mp, i); > -#ifdef XR_INODE_TRACE > - fprintf(stderr, > - "\t\t phase 3 - process_uncertain_inodes returns %d\n", j); > -#endif > - PROG_RPT_INC(prog_rpt_done[i], 1); > - } > + memset(counts, 0, mp->m_sb.sb_agcount * sizeof(*counts)); > + > + create_work_queue(&wq, mp, scan_threads); > + > + for (i = 0; i < mp->m_sb.sb_agcount; i++) > + queue_work(&wq, do_uncertain_aginodes, i, &counts[i]); > + > + destroy_work_queue(&wq); > + > + /* tally up the counts */ > + for (i = 0; i < mp->m_sb.sb_agcount; i++) > + j += counts[i]; > + > } while (j != 0); > + > + free(counts); > + > print_final_rpt(); > } > diff --git a/repair/protos.h b/repair/protos.h > index b113aca..0290420 100644 > --- a/repair/protos.h > +++ b/repair/protos.h > @@ -46,7 +46,7 @@ void thread_init(void); > > void phase1(struct xfs_mount *); > void phase2(struct xfs_mount *, int); > -void phase3(struct xfs_mount *); > +void phase3(struct xfs_mount *, int); > void phase4(struct xfs_mount *); > void phase5(struct xfs_mount *); > void phase6(struct xfs_mount *); > diff --git a/repair/xfs_repair.c b/repair/xfs_repair.c > index fcdb212..5d5f3aa 100644 > --- a/repair/xfs_repair.c > +++ b/repair/xfs_repair.c > @@ -871,7 +871,7 @@ main(int argc, char **argv) > if (do_prefetch) > init_prefetch(mp); > > - phase3(mp); > + phase3(mp, phase2_threads); > timestamp(PHASE_END, 3, NULL); > > phase4(mp); > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From jwollrath@web.de Mon Jan 4 13:26:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 157F67F37 for ; Mon, 4 Jan 2016 13:26:18 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id CBA4F304032 for ; Mon, 4 Jan 2016 11:26:17 -0800 (PST) X-ASG-Debug-ID: 1451935572-04bdf02651452470001-NocioJ Received: from mout.web.de (mout.web.de [212.227.17.12]) by cuda.sgi.com with ESMTP id KQIzDIggIjQ4HT1y (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:26:13 -0800 (PST) X-Barracuda-Envelope-From: jwollrath@web.de X-Barracuda-Apparent-Source-IP: 212.227.17.12 Received: from saldaea ([134.76.62.130]) by smtp.web.de (mrweb102) with ESMTPSA (Nemesis) id 0MDg9K-1aRm9q0Peg-00H4Xp for ; Mon, 04 Jan 2016 20:26:12 +0100 Date: Mon, 4 Jan 2016 20:26:11 +0100 From: Julian Wollrath To: xfs@oss.sgi.com Subject: Suspend does not work with v4.4-rc8 Message-ID: <20160104202611.5b100540@saldaea> X-ASG-Orig-Subj: Suspend does not work with v4.4-rc8 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:2kSjhqlK0Kw1JMzJGByIyNvPQbv7+/diV8ECM802X9rFF48eeJD qvXVDg8+aNHLfPSzVljYQaGg4qhVOGvTqd/koRgKP/RXUREnCH3UXdhZr8QAPClvRG58IeH k5lBhdRLMpC17hRlRrYDsaEycdvRoHbX+W6erCjUXW5Jdmd0UMn0VWkJWVxeBoLSjrN36RU BATXtPg0Ywmt9tAsQWQAg== X-UI-Out-Filterresults: notjunk:1;V01:K0:6tYB1zneba8=:sNWbiR7WxIiPTDirPhpK8H cE9UD2e65eP+gXOZprZ7u7xocDkc8T8tZwY5U9W6gyx0LE3dgt4w6d97JErDf0g9YUwth1emH L9uft5vUW1gRfsjfujXo6iSfHSuAFZBZtMwm5E07i8nI7YcBXTgsGDp6pPXCXrsI+Acv1Ov7Y tL1+JK1T5v38NAwQHzm0S5pLmabGAYSZY24PYHwK9yVcdYTznA/74/+bzWZwpFnB58W6P2pq+ gPCQcU+Z5UQMkxXSwXPgLA2SZGFpq4YX1j+p5LoZDfoixMb17dXMe6KSaXHoWhh9jAtaFK8kO D1zB/0dWndUOWslYWxOReKFyLDj4b0Dqk7KplEmR4sMKsq06jOhZrYI5wPgYq5SvH+SM2pXFa bNEzIDkL/i+Zx923ViyFrCJiUGAj4azzyp4Ssw4rJToKSWsrG2ldtxnn+MpeOBZMhBtHt5Jya JCkN0jsAoZY6YK5P3uG9twv64Gw5IMxDvFuSasM30VUkK93IfcW/p8pimzJPr3kOJ4f0kBVrB fUhrX286dy1dBNMjfJEI77uSnwLtmjIbXX8tpwkK4i7Z0D9XtFKobuUX5duYca5uZlRXivLbY FB/XnvpwFHH3EEWDYzjLVHFQ8SMR/owpaEV/i5eOFV+LFxrtLpwlosEEkzZG+7aBxNoeh7549 4g0WypLFYa90aWK1RE+kvykoqhC61Adx4X85SpPJkviXorpObMselbfb7MLI995rjkrJAhuuS UZ0uXn7FPpdsc5iHpvjxcgPpOnLQqyoh85yDpHwgtRm22mUCDCf06IbcbDYKy1k2pRcHXhsGN 9RP+npnuAAnKiGGO35+NvZ/je/BtQ== X-Barracuda-Connect: mout.web.de[212.227.17.12] X-Barracuda-Start-Time: 1451935573 X-Barracuda-Encrypted: DHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25828 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Hi, suspending (either to RAM or disk) fails with 4.4-rc8 with the following message while v4.3.3 worked fine: kernel: Freezing of tasks failed after 20.006 seconds (2 tasks refusing to freeze, wq_bu_busy=0): kernel: xfsaild/dm-1 S 0000000000014100 0 283 2 0x00000000 kernel: ffff880213f53e10 ffffffff8180e4c0 ffff880213f05040 0000000000000000 kernel: 0000000000000000 ffff880213f54000 0000000000000000 0000000000000000 kernel: ffff8800ca389e40 ffff8800ca392000 ffff880213f53ed0 ffffffff814cd8ac kernel: Call Trace: kernel: [] ? schedule+0x2c/0x70 kernel: [] ? xfsaild+0x4fd/0x5b0 kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 kernel: [] ? kthread+0xb8/0xd0 kernel: [] ? kthread_worker_fn+0x150/0x150 kernel: [] ? ret_from_fork+0x3f/0x70 kernel: [] ? kthread_worker_fn+0x150/0x150 kernel: xfsaild/sda1 S 0000000000014100 0 591 2 0x00000000 kernel: ffff88021193be10 ffff8802159c4dc0 ffff880213ab9340 0000000000000000 kernel: 0000000000000000 ffff88021193c000 0000000000000000 0000000000000000 kernel: ffff8800ca2a4240 ffff880214eea000 ffff88021193bed0 ffffffff814cd8ac kernel: Call Trace: kernel: [] ? schedule+0x2c/0x70 kernel: [] ? xfsaild+0x4fd/0x5b0 kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 kernel: [] ? kthread+0xb8/0xd0 kernel: [] ? kthread_worker_fn+0x150/0x150 kernel: [] ? ret_from_fork+0x3f/0x70 kernel: [] ? kthread_worker_fn+0x150/0x150 Please tell me what more information you need to be able to fix this issue. With best regards, Julian Wollrath From MAILER-DAEMON Mon Jan 4 13:36:34 2016 Return-Path: <> X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: * X-Spam-Status: No, score=1.9 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLYTO, TO_MALFORMED,T_FREEMAIL_DOC_PDF autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9CEE77F37 for ; Mon, 4 Jan 2016 13:36:34 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 5692F8F8035 for ; Mon, 4 Jan 2016 11:36:34 -0800 (PST) X-ASG-Debug-ID: 1451936189-04bdf026514527e0001-NocioJ Received: from mail.gaf.ni.ac.rs (mail.gaf.ni.ac.rs [160.99.31.9]) by cuda.sgi.com with ESMTP id khG8mOhaeD5cGtcN (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:36:31 -0800 (PST) X-Barracuda-Envelope-From: X-Barracuda-Apparent-Source-IP: 160.99.31.9 Received: from localhost (localhost [127.0.0.1]) by mail.gaf.ni.ac.rs (Postfix) with ESMTP id 0E0E68F315729; Mon, 4 Jan 2016 20:29:19 +0100 (CET) Received: from mail.gaf.ni.ac.rs ([127.0.0.1]) by localhost (mail.gaf.ni.ac.rs [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id JILdZ19_CQXK; Mon, 4 Jan 2016 20:29:18 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mail.gaf.ni.ac.rs (Postfix) with ESMTP id 9B72D8F3A824F; Mon, 4 Jan 2016 20:27:38 +0100 (CET) X-Virus-Scanned: amavisd-new at gaf.ni.ac.rs Received: from mail.gaf.ni.ac.rs ([127.0.0.1]) by localhost (mail.gaf.ni.ac.rs [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ODRxRiRrw92O; Mon, 4 Jan 2016 20:27:38 +0100 (CET) Received: from [100.67.150.48] (unknown [116.202.38.88]) by mail.gaf.ni.ac.rs (Postfix) with ESMTPSA id 4A0998F2F1C3E; Mon, 4 Jan 2016 20:25:22 +0100 (CET) Content-Type: multipart/mixed; boundary="===============1096080894==" MIME-Version: 1.0 Subject: Re To: Recipients <> X-ASG-Orig-Subj: Re From: "<"@LIVE.COM Date: Tue, 05 Jan 2016 00:55:08 +0530 Reply-To: cruzcelia2011@gmail.com Message-Id: <20160104192523.4A0998F2F1C3E@mail.gaf.ni.ac.rs> X-Barracuda-Connect: mail.gaf.ni.ac.rs[160.99.31.9] X-Barracuda-Start-Time: 1451936190 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.00 X-Barracuda-Spam-Status: No, SCORE=2.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0646, BSF_SC0_MV0646_4, EMPTY_ENV_FROM, NO_REAL_NAME, TO_MALFORMED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25828 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 TO_MALFORMED To: has a malformed address 0.00 EMPTY_ENV_FROM Empty Envelope From Address 0.00 NO_REAL_NAME From: does not include a real name 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.50 BSF_SC0_MV0646 Custom rule MV0646 1.50 BSF_SC0_MV0646_4 BSF_SC0_MV0646_4 You will not see this in a MIME-aware mail reader. --===============1096080894== Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body please kindly open the fill --===============1096080894== Content-Type: application/octet-stream MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="BUSINESS PROPOSAL.docx" UEsDBBQABgAIAAAAIQDd/JU3ZgEAACAFAAATAAgCW0NvbnRlbnRfVHlwZXNdLnhtbCCiBAIooAAC AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC0 VMtuwjAQvFfqP0S+Vomhh6qqCBz6OLZIpR9g7A1Y9Uv28vr7bgJEVQtBKuUSKVnvzOzsxIPR2pps CTFp70rWL3osAye90m5Wso/JS37PsoTCKWG8g5JtILHR8PpqMNkESBl1u1SyOWJ44DzJOViRCh/A UaXy0Qqk1zjjQchPMQN+2+vdcekdgsMcaww2HDxBJRYGs+c1fd4qiWASyx63B2uukokQjJYCSSlf OvWDJd8xFNTZnElzHdINyWD8IENdOU6w63sja6JWkI1FxFdhSQZf+ai48nJhaYaiG+aATl9VWkLb X6OF6CWkRJ5bU7QVK7Tb6z+qI+HGQPp/FVvcLnrSOY4+JE57OZsf6s0rUDlZESCihnZ1x0cHRLLs EsPvkLvGb1KAlHfgzbN/tgcNzEnKin6JiZgaOJvvV/Ja6JMiVjB9v5j738C7hLT5kz7+wYz9dVF3 H0gdb+634RcAAAD//wMAUEsDBBQABgAIAAAAIQAekRq38wAAAE4CAAALAAgCX3JlbHMvLnJlbHMg ogQCKKAAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAjJLbSgNBDIbvBd9hyH032woi0tneSKF3IusDhJnsAXcOzKTavr2jILpQ217m9OfLT9ab g5vUO6c8Bq9hWdWg2JtgR99reG23iwdQWchbmoJnDUfOsGlub9YvPJGUoTyMMaui4rOGQSQ+ImYz sKNchci+VLqQHEkJU4+RzBv1jKu6vsf0VwOamabaWQ1pZ+9AtcdYNl/WDl03Gn4KZu/Yy4kVyAdh b9kuYipsScZyjWop9SwabDDPJZ2RYqwKNuBpotX1RP9fi46FLAmhCYnP83x1nANaXg902aJ5x687 HyFZLBZ9e/tDg7MvaD4BAAD//wMAUEsDBBQABgAIAAAAIQDWZLNR+gAAADEDAAAcAAgBd29yZC9f cmVscy9kb2N1bWVudC54bWwucmVscyCiBAEooAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AKySzWrDMBCE74W+g9h7LTv9oYTIuZRArq37AIq9/qGyJLSbtn77CkNShwb34otgRmjmk7Sb7Xdv xCcG6pxVkCUpCLSlqzrbKHgvdnfPIIi1rbRxFhUMSLDNb282r2g0x0PUdp5ETLGkoGX2aympbLHX lDiPNu7ULvSaowyN9Lr80A3KVZo+yTDNgPwiU+wrBWFf3YMoBh+b/892dd2V+OLKY4+Wr1TILzy8 IXO8HMVYHRpkBRMzibQgr4OslgShPxQnZw4hWxSBBxM/8/wMNOq5+scl6zmOCP62j1KOazbH8LAk Q+0sF/pgJhxn6wQhLwY9/wEAAP//AwBQSwMEFAAGAAgAAAAhAKsR9BCDCgAAyx0AABEAAAB3b3Jk L2RvY3VtZW50LnhtbNRYzW7bRhC+F+g7DIQcWkCWKAcuEjVW4FpO40NTI7ZzX5FLcWFyl91dSuGt r9FbniWP0ifpN0tSliXFFgSjRU+0l9z5n++b0Zu3n4ucFtI6ZfRpbzSIeiR1bBKl56e925t3R696 5LzQiciNlqe9Wrre28n3371ZjhMTV4XUniBCu/ECbzPvy/Fw6OJMFsINTCk1XqbGFsLjXzsfFsLe VeVRbIpSeDVTufL18DiKfuq1Ysxpr7J63Io4KlRsjTOp5ytjk6Yqlu2ju2H30dvcnLYmB41DK3PY YLTLVOk6acWh0uBi1glZPObEosi775blPtoSK5bIR5E3Zi+NTUprYukcTqfNy5XEUfSY7jaALGJ1 Yx8THursLCmE0isxXB0b+V8lb4DkDRvdQxZ17whiMUEtzUxS87Ok5Ri1mHw87UXR+Un07mLa646m MhVV7vnNL+fHr6Yvw83yyvJF2zxmwyDt3OE5bE/x7D564svl2E9+tVJ6xNUN6CqXwkkqpS2Up0KS N6S0tyapYklF7WSe8lltKjKaj4yVA7okUZDA94mKRU4xCgxmCzTKrCYkLuXEGU1oKkIw7thQ35gb 7Ov8ZUePR6PXoyg4+oiHuM/dM3aliJGF0kon7UL2Jo6EY5VxZWVo1UJoMZeWlspn9Jjily9P3p1M 91A8uS6FVi6jq0ygzWNZeXZ7U/bTkd/tAZ0zUuiaw7qEM0rTpU6U4D+Oo9GIAC4INnssbJxRWdnS IGlJZZFEWmYKh5eUKBcbwJxMyGfC0w+b9h0U9MkNlAj6JK34MVgnFkLlYpbLlZ2D51A0s205H1wf QQKX2c2mPYcmJni+KeygKO5OPAd1q0KfNnZyLZFj5ahyeHJxhD5EmvKuNlx/w2w0iEkvLLpi7OsS DTS3orj2wvpegJOnAIbj+vULUr6P3Aud7C11d2TI+IxbGDiKfvDsrNDQL2KvFpLwbsbecwmi3asU 500zmJTOtFf0/vIT3eAzUdZ9eg+PrZc6QNIV2FlpB3C6mJlc0FShrZx0dOHjAe5AlWvj60oZK7Cx yPMaca4AZlAIyyhTfCSsFASN16YC1FwI5+nMKdEPsCfiGNZzgwI+0TDGCm9sTV7is1UvszAMHZRa U7RdnwECZlJqBrUFHrg+Ay7jwxmuMkhkAjFYKKc88t9ABdzxGWCdTBwL9tPxPSBFDHF8u3OLEZll JZhe8CcrA3qDDSrGDUMW4SA4jFBjKEFUKosRJ3h0q4NCVA18YL/PCmkRnYBbCEgwWCWBQ5gvWA3L RnKgzgM2+BKf5oZZI5EiR5JF6jnVEkERADIxhxsByGENcwesmkvcsls482SrPASV/T7nSr9EcVUh 6zsimG60APfUis/3JrMJ2vYjCPhl1KcoioZHzMEEcruTnsM02tTypPXf4MdoXjTVtRn4Ps0qJKYK +VLekRUWk2pINX+bqXnWlkkfRYdJIN8Cq8Ncv69GpPi3uuO/AGUvthxf17E/ZffpJAqhfa440u01 TU2eC+s2RR5m4dcvaMBNUYdmme6kLENv8fQFxAy0EMa3tbECYGDnwIj0HhkYBFxVloAzPg6DH2qh AiTk7TyCw8Iw1CBbgrRcoipKMEdoUgDiVMaymKF6c0bAGkMK46hcpZUxRpslibI0mC0hqJECudtD W4amZywJ8HY/By2lvHN///kXedUMqduN+e/BAzunnEPnWPSID84BX8PUhTnM4MBjVsI0zH3EAdvl KqIJPMckxZNloqwETGKHKCoNTOWdrZlgWcQ6XqJtZSyY/B6qylTBCUoVEgo+CqqZFnkoAEqHrLMs JrcmP82gHLKI1Bf3fQhfPvx+Q2IJgusge1vUOqOEfQBgbRWXCU+fnefNRgSM5KTKz2FwlZhSa4AA uFGlIHOEDITywHJwQWt3y8nQxlEFj8I3pojk524wCEfe1i3XItaeYwmHuNrmDGor1rxDITJVsjjk pdsVuFtSniy4ennWAK/yivNHBcpjElsoLD+ta6CqWGi2JmY8CMweRpEgdlZhZcX687AceUHj3v6v l79uSN49dgGL0aQcelQSUzjCxKHBn+BprHT8MwlKCW/X+q+pqd0k07GJBrsfBQoBNuE6C2zXnm82 yDLD9IPqECirBQoEUBOGnaZ613uiqa4gvivxzoNNgF3H6v3JmoPQsvUzkfUEvNwwMFu8Cx/QMc9j fNgKmjmQiYEnTc4tfMJE/WLUP9lU8zQJTcCt0ea1w0Ib0e1gxawwqFlnUXhNq7/eUrOHdX0kiS4/ fAyVAgwL4NHCoJZz41UDsMjqDIwolGbsAAhiCydsB0k/4Nc3ssK/d+Q5Q0ybOvxWshZXBo9S1CHG mMWz/yMQnIWZ/B8AAAD//6xWXW7bOBC+ysBA31xXTtLYzTYG6iQt+rCA0e4FKGkkEZZIlaSiuk+9 Rt/2LD3KnmS/oawkNookBQokkvgzw5lvvm/oUDG1zmad44ZNoEYZVbKjSnlqO5fhzTnJLs/4KJxt aGe7qTyo13VNOeddFuKWe4PM+kDK5LLN0dtX/UVYydOt3uJB+Pc6/3Q5SZLzk/nN4noS5zdxOX0l g/TK4y0mw2xYBaduua61KY/9PW1MX5v6wrcq48tJ69izu+XJivhryybXAdlPY7hI1AypOFvoMGSY IvlKOaSfcugZOzpPKtBp8oJe0iJ5MaW+0llFDSvj75EJasuyLA7Nz3/p4+AuTsN2Fqccq7reUW7J WJynUINgqbaeyXaBrISjPaWd14Y9PjhTHRYRHKaVIdu21oXO6LCjwjoEIQ4aObuxhjEpNZMyZLZp ldnN6J1HLF7pnFi5WrObxpxzRvVzssVwJGgRK4tj5kmSENDbcvB/jUenEl9le+HMblyN2WcIa+TC cLwQqLaZErqoejzQdE0KruHAve9ofWe5p1dg1yD3GKJqbAeExGJfIB5ALTnMDmgB5rTCi/Yh2c7W i+Wbm8k4dc2F6uogNFxfnSyvTwcaPkmn1EWKhl+z6iOpBlibQueiqFCBKr0O1aCFCjXxKJUgjZJm 7KRwklDnWql6v88oVrC0FqLTRplMA7xSaUP4O6DEbCRW2eHECCFYU3QOoLsobigXqgFbPHmLNbxF vjFGF/XsCDzEJmEY2HOA5Kjax9RZKx8+QUoMlWzQQNZwt30MpN667YBKwzNag0kqtbdMEMMU+Wy1 yaEKx186oBWjRVhb5nbIHVRuQPlMBY187sAGRjPa1IymJYryiEj0EJsQNgWFRrUnneB/nObIi9/r SyrP0VE86eIxfyfz+Zt58ow+92tS/aFYgcR/3384tA8DWQFazo+DfpL9YTWjfyplttJ+cvoAhqY1 ADjU33NYM+potT4O4mElno3cSqjiuFQu99M/4fEuvncQ6TdFV7Vy6tjxc/A6sBk7k+csbA7uwsP2 9BnrwsTXN4vk9fvInLb8/A3Q9JcTsCk5l0ZW4ft8ebqcRLW15d9KXAbbYv7sLIm9TpcVPI3D1IZg m/txzcWD1Qp9gN3lZJEsxbawFkS5G5ZdiMNkOC6ztcem/cUqJjGK3GYfHG6X/gK3NW90yBDl6Xk0 QvZD4vFyT22+ix8w6eQHyOp/AAAA//8DAFBLAwQUAAYACAAAACEAlrWt4pYGAABQGwAAFQAAAHdv cmQvdGhlbWUvdGhlbWUxLnhtbOxZT2/bNhS/D9h3IHRvYyd2Ggd1itixmy1NG8Ruhx5piZbYUKJA 0kl9G9rjgAHDumGHFdhth2FbgRbYpfs02TpsHdCvsEdSksVYXpI22IqtPiQS+eP7/x4fqavX7scM HRIhKU/aXv1yzUMk8XlAk7Dt3R72L615SCqcBJjxhLS9KZHetY3337uK11VEYoJgfSLXcduLlErX l5akD8NYXuYpSWBuzEWMFbyKcCkQ+AjoxmxpuVZbXYoxTTyU4BjI3hqPqU/QUJP0NnLiPQaviZJ6 wGdioEkTZ4XBBgd1jZBT2WUCHWLW9oBPwI+G5L7yEMNSwUTbq5mft7RxdQmvZ4uYWrC2tK5vftm6 bEFwsGx4inBUMK33G60rWwV9A2BqHtfr9bq9ekHPALDvg6ZWljLNRn+t3slplkD2cZ52t9asNVx8 if7KnMytTqfTbGWyWKIGZB8bc/i12mpjc9nBG5DFN+fwjc5mt7vq4A3I4lfn8P0rrdWGizegiNHk YA6tHdrvZ9QLyJiz7Ur4GsDXahl8hoJoKKJLsxjzRC2KtRjf46IPAA1kWNEEqWlKxtiHKO7ieCQo 1gzwOsGlGTvky7khzQtJX9BUtb0PUwwZMaP36vn3r54/RccPnh0/+On44cPjBz9aQs6qbZyE5VUv v/3sz8cfoz+efvPy0RfVeFnG//rDJ7/8/Hk1ENJnJs6LL5/89uzJi68+/f27RxXwTYFHZfiQxkSi m+QI7fMYFDNWcSUnI3G+FcMI0/KKzSSUOMGaSwX9nooc9M0pZpl3HDk6xLXgHQHlowp4fXLPEXgQ iYmiFZx3otgB7nLOOlxUWmFH8yqZeThJwmrmYlLG7WN8WMW7ixPHv71JCnUzD0tH8W5EHDH3GE4U DklCFNJz/ICQCu3uUurYdZf6gks+VuguRR1MK00ypCMnmmaLtmkMfplW6Qz+dmyzewd1OKvSeosc ukjICswqhB8S5pjxOp4oHFeRHOKYlQ1+A6uoSsjBVPhlXE8q8HRIGEe9gEhZteaWAH1LTt/BULEq 3b7LprGLFIoeVNG8gTkvI7f4QTfCcVqFHdAkKmM/kAcQohjtcVUF3+Vuhuh38ANOFrr7DiWOu0+v Brdp6Ig0CxA9MxHal1CqnQoc0+TvyjGjUI9tDFxcOYYC+OLrxxWR9bYW4k3Yk6oyYftE+V2EO1l0 u1wE9O2vuVt4kuwRCPP5jeddyX1Xcr3/fMldlM9nLbSz2gplV/cNtik2LXK8sEMeU8YGasrIDWma ZAn7RNCHQb3OnA5JcWJKI3jM6rqDCwU2a5Dg6iOqokGEU2iw654mEsqMdChRyiUc7MxwJW2NhyZd 2WNhUx8YbD2QWO3ywA6v6OH8XFCQMbtNaA6fOaMVTeCszFauZERB7ddhVtdCnZlb3YhmSp3DrVAZ fDivGgwW1oQGBEHbAlZehfO5Zg0HE8xIoO1u997cLcYLF+kiGeGAZD7Ses/7qG6clMeKuQmA2Knw kT7knWK1EreWJvsG3M7ipDK7xgJ2uffexEt5BM+8pPP2RDqypJycLEFHba/VXG56yMdp2xvDmRYe 4xS8LnXPh1kIF0O+EjbsT01mk+Uzb7ZyxdwkqMM1hbX7nMJOHUiFVFtYRjY0zFQWAizRnKz8y00w 60UpYCP9NaRYWYNg+NekADu6riXjMfFV2dmlEW07+5qVUj5RRAyi4AiN2ETsY3C/DlXQJ6ASriZM RdAvcI+mrW2m3OKcJV359srg7DhmaYSzcqtTNM9kCzd5XMhg3krigW6Vshvlzq+KSfkLUqUcxv8z VfR+AjcFK4H2gA/XuAIjna9tjwsVcahCaUT9voDGwdQOiBa4i4VpCCq4TDb/BTnU/23OWRomreHA p/ZpiASF/UhFgpA9KEsm+k4hVs/2LkuSZYRMRJXElakVe0QOCRvqGriq93YPRRDqpppkZcDgTsaf +55l0CjUTU4535waUuy9Ngf+6c7HJjMo5dZh09Dk9i9ErNhV7XqzPN97y4roiVmb1cizApiVtoJW lvavKcI5t1pbseY0Xm7mwoEX5zWGwaIhSuG+B+k/sP9R4TP7ZUJvqEO+D7UVwYcGTQzCBqL6km08 kC6QdnAEjZMdtMGkSVnTZq2Ttlq+WV9wp1vwPWFsLdlZ/H1OYxfNmcvOycWLNHZmYcfWdmyhqcGz J1MUhsb5QcY4xnzSKn914qN74OgtuN+fMCVNMME3JYGh9RyYPIDktxzN0o2/AAAA//8DAFBLAwQU AAYACAAAACEAoQmDWCUDAAAfBwAAEQAAAHdvcmQvc2V0dGluZ3MueG1snFXbbts4EH1fYP9B0PM6 lmQ7ToQ4RWNb3RZJW6zSD6CkkUyEN5CUFffrO5TEKul6g2KfTJ0zczicm2/ePXMWHEEbKsUmjC+i MABRyoqKZhN+e8xmV2FgLBEVYVLAJjyBCd/d/vnHTZcasBbNTIASwqRyE7ZapKY8ACdmxmmppZG1 nZWSp7KuaQnjTzh66E14sFal8/nodCEVCFSrpebEmgupm/nguZNly0HYeRJFl3MNjFgM2ByoMl6N /181vOrgRY5vPeLImbfr4ugty/G5ndTVT4/fCc85KC1LMAYzy9nwXE6o8DKG/Y7OkM97WmiiTy9E brFs36XkQZcq0CUmFGseReHcEZX8LO2OGsXI6Stp4E62WHZNwfQ0xiXr3BIL6G0UMNb3SMmAYHRd 2mjCOcGaDsggCTVpmX0kRW6lQqMjwfjXyXhjeSCalBZ0rkiJalsprJbM2/UBbSVXGvMxxIi9pIh1 0bYGsv09OcnW4lXzLp0obObKOBt3+EdK6wWj6G6bXO0Wg5ZjJyaJ4+t4jOs1s1isstXunM/ybn11 vT/HrPbraJWdYy6TeL8+q/Y+W27jsz7/HfV2FWX7s2rZ1fI6XrsIMDdjRnjqev2rvr0ZThnmO+BD UbaEF5qS4MFNA3rxtNBPd1R4vgCcSnjJ5G3hydlsIAwnjGVYU0/gIAxMhX21g7oXZg9EN5Nyn3Se 6rNoBfWnn2quYUF/0LJVg2qnifooKoT9hfFyOepRYe8p97hpi9x7CRyKFxR2+ZejdoLzKUFdanGP gcvQPRGN7xMQs2+5M8V+Yzp3uw4eiFLYvGhSNPEmZLQ52NhNhMUvHKCn/qNokpFLeg6/HNd/kNK9 DK3HgzMYjmg1HiZs4bHFhC09tpywlcdWE3bpsUuHHU64BXCMn3Cn+KPDa8mY7KD624Ob8F/QkARz IAqwrm7KceRk2gNYtB4Ijik844qBilr8G1G04uR5E67iJHHuozWuGxziV7aOc8bqFRpUxBJcWH2p Xjn3Tf5LLLjQoKTYkPmJF9NSuRgCZ9TYHBTuHys1PrlfTH/1ytM/2+0PAAAA//8DAFBLAwQUAAYA CAAAACEAb2NQm4sBAAAHBAAAEgAAAHdvcmQvZm9udFRhYmxlLnhtbKSS326CMBTG75fsHUjvJwX/ TI1onNPLXSzuAY5YpAltSU+V+fY7UORiZolkkDThO+3H6e98i9W3KoKLsCiNTlg04CwQOjVHqU8J +9rvXqYsQAf6CIXRImFXgWy1fH5aVPPMaIcBndc4twnLnSvnYYhpLhTgwJRCUy0zVoGjT3sKTZbJ VLyb9KyEdmHM+SS0ogBH/8Zclshat+oRt8rYY2lNKhCpWVV4PwVSs2XbXVDNNSjqegOFPFjZFErQ BkVEtQsUCeMx3/ExrfU74sN6ZWHtkOZgUbhuI/dyBkoW15uKlUT0hVK6NL/pF7ASDoXwJZQnKpzx wBO2jTjn8W7HvBIlbETCetMpMTXln1m7Z9gpNB5qrPFptkSzxocU8mlPNX2Gfj53JPZSCQw+RBV8 GgUe1T2RmE+IxJh41GSGvYjYxrch+CgRajxed/enm2xIeZ2Oovb+vYh4n8eJbEBRNOCPbNQEPIma SL9s9CexrkcYb39lg/PR2x2JJgmUqP9kow0JLn8AAAD//wMAUEsDBBQABgAIAAAAIQAoh3GlzwAA AB8BAAAUAAAAd29yZC93ZWJTZXR0aW5ncy54bWyMj8tOAzEMRfdI/MMoe5qBRYVGnamEUNlQqMRj n2Y8nUiJHdmB0H495rFhx/LaV8fHq/VHis07sATC3lwuWtMAehoDHnrz8ry5uDaNFIeji4TQmyOI WQ/nZ6vaVdg/QSnalEYpKB33Zi4ld9aKnyE5WVAG1N1EnFzRyAdL0xQ83JJ/S4DFXrXt0jJEV9RA 5pDF/NLqf2iVeMxMHkRUJMUfXnIBzaCOlEtI4QQb4humKsD2a6z3jo/4ur3/Ti5GqruHOw32z1vD JwAAAP//AwBQSwMEFAAGAAgAAAAhAPeMfebgAQAA2wMAABAACAFkb2NQcm9wcy9hcHAueG1sIKIE ASigAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnFPBbtswDL0P2D8Yvjd2nCUoAkXFkGLo YVsDxG3PnEzHwmRJkNSg2dePshtX2XqqT+Qj/fT0SLGbl15lR3ReGr3J57Myz1AL00h92OQP9ber 6zzzAXQDymjc5Cf0+Q3//IntnLHogkSfEYX2m7wLwa6LwosOe/AzKmuqtMb1ECh1h8K0rRR4a8Rz jzoUVVmuCnwJqBtsruxEmI+M62P4KGljRNTnH+uTJcGc1dhbBQH5zyhHsWICWG0CqFr2yBcl4VPG dnBAz+esGAP2ZFzj+bIkZAzZtgMHIpB7vLpe0d8JwL5aq6SAQMbyH1I4400bsvvBgiwSsCJtYWTL HsWzk+HEiSpN2XepSUq1YMUYkTYHBwe283wVBU4Z2wtQuKXL8xaUR1a8AewOIQ52B5IUs2NYH1EE 4zIv/9Boqzz7BR6jZZv8CE6CDmRdbBuTIVbWB8drGRRxU23MhzBtS2P5JbpIvRRcNkZw1ECFS3XD Cf6+pbuFd8TOU7GDhlFqIicJpzP+Yd2a3oI+0eFTRAb/9g+2NrdxXV49vASTuT/J0O0tCJrOYrFc phuQlNieFgUbGumZ8A1gd+S3U/FU2h59wObc838h7tTj+Fb5vJqV9A1LdMZoE6ZHxP8CAAD//wMA UEsDBBQABgAIAAAAIQBTfUAmVgEAAHkCAAARAAgBZG9jUHJvcHMvY29yZS54bWwgogQBKKAAAQAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACMkstuwjAQRfeV+g+R94mTAC2ykiD1wQraSk0f6s6y B4gaP2S7BP6+TgIpqF10Ob7XZ+6Mnc12og62YGylZI6SKEYBSKZ4Jdc5einn4RQF1lHJaa0k5GgP Fs2Ky4uMacKUgSejNBhXgQ08SVrCdI42zmmCsWUbENRG3iG9uFJGUOdLs8aask+6BpzG8RUW4Cin juIWGOqBiA5Izgak/jJ1B+AMQw0CpLM4iRL843VghP3zQqecOEXl9trPdIh7yuasFwf3zlaDsWma qBl1MXz+BL8vF8/dqGEl210xQEXGGWEGqFOmqEGqrcrwyVG7vppat/SbXlXAb/bF4v7h8fUxw7+V 1mxgW7VvVEw6x1D6Pt1YfTPggQ9K+rGOytvo9q6coyKNk0mYpGEyLuNrMhmROP5oQ53db4P3B+IQ 7R/EdFqmMUnG58QjoOgSn3+W4hsAAP//AwBQSwMEFAAGAAgAAAAhAOYmwWoaBwAA+zkAAA8AAAB3 b3JkL3N0eWxlcy54bWy0m1FT2zgQx99v5r6Dx+80ISnJlWnaoRSuzLQcbWDuWbEVoqlj5SylQD/9 SStbGBvbu9g8EcvW/lba1X+dIL3/eL9Ngl88U0Kmi/DwzTgMeBrJWKS3i/Dm+vzgrzBQmqUxS2TK F+EDV+HHD3/+8f7uWOmHhKvAGEjVcbYIN1rvjkcjFW34lqk3csdTc28tsy3T5jK7Hcn1WkT8s4z2 W57q0WQ8no0ynjBt4GojdirMrd1hrN3JLN5lMuJKGW+3ibO3ZSINPxj3Yhl95mu2T7Syl9lVll/m V/DnXKZaBXfHTEVCXBvHzRC3IpXZl5NUidDc4UzpEyVY+eZZ3mbvb+yD5Zu+Z6R0yeAnEYtwZKHq t+n2iyWLcDIpWk6tE0/aEpbeFm08Pbi4LDuzCH3TythdhCw7WJ5YYyMYafG3NOLdk/GbK3BlxyIz d4bD1pqbGJqQWE4ibKwn81lx8WOfmAa21zKHgAEDK5s1l5VJN6E1gV66RDF3+fqrjH7yeKnNjUUI LNN4c3GVCZkJ/bAI372zTNO45FvxRcQxt3mZt92kGxHzfzc8vVE8fmz/fg5ZlluM5D7Vxv3ZHBIh UfHZfcR3NsuM6ZTZIF/aDok1q0occGgvHr1xDRUqNP5XIA9dDJ+lbDizKykA/1tBMOp9b9DEjqg8 ALBL8nXa38Tb/iaO+puA5O03F/P+Xhj97BsRlxulrMQHVcvIJV95HqbvWlLW9qhlUWePWtJ09qjl SGePWkp09qhlQGePWsA7e9Ti29mjFs7WHhED4apm0RRmA7Wwr4VOuO3fKkCHPaUuLzXBFcvYbcZ2 m8DW1qrbbWK53K80zlWQ05eL5VJnMr3tnBFTne3SfbEmn213G6aEeanpmPpJz6m/ZquEB39nIu5E Hbnkq40JXkyeLWFXCYv4RiYxz4Jrfu8iSuh/KYOle8vodK5nWL+K240OlhsouZ2wWcOkN8+Es/9V KJiD1sU0axhKl3FUDGcNedls/BuPxX5bTA3ibWTm9JwQ5goCXGyforc2RPXV1TkKGwDMEFy5oA8B 7CP8d8WFbt/GGOO/K0UvtI/w3xWuF9qH/GiPL1lpPrPsZ4BaXnPy2j2ViczW+6RYA53yMCevYI/A DYG8iL19lEjMySv4iXwGJ1Fkvrlh8pQci0cdJVDI4XAUWGz4sZCDUpG9Q8KIyAGqsCYEVj+tJYDI ovuD/xL2tydqMQCV9u+anct52jADpgSh3qG/76XufoeeNGgelnKRmp9LFA9wtGnDysPS8nxy9Y4Q 436FjwDqVwEJoH6lkABqyI/mdx5fE/GQ/sWRwCLLsq9ikHZoZZ6TldmDaCVgoLqJeP9qWL3NuVCv mwgKOUD1uomgkKNTqWW+biJYg9VNBKuhajTHqKyplEGR62YZ5N8EECMaRrwRoGHEGwEaRrwRoP7i 3Q0ZTrwRLLI2eE0tizcCBI9Qvup7UFm8ESCyNji1y38zKuoeWGn/cjuAeCMo5ADVxRtBIUenSbwR LHiEkgkVlpc6BGsY8UaAhhFvBGgY8UaAhhFvBGgY8UaA+ot3N2Q48UawyNrgNbUs3ggQWR48qCze CBA8QtGGZ8UbVv2rizeCQg5QXbwRFHJ0KoLqX1IRLHKAKiwv3ggWPEJJhpwFyU0Z1DDijRjRMOKN AA0j3gjQMOKNAPUX727IcOKNYJG1wWtqWbwRILI8eFBZvBEgsjY8K96wGF9dvBEUcoDq4o2gkKNT EVSvcwgWOUAVlhdvBAvypbd4I0DwyEtBlBENI96IEQ0j3gjQMOKNAPUX727IcOKNYJG1wWtqWbwR ILI8eFBZvBEgsjY8K96wRl5dvBEUcoDq4o2gkKNTEVQv3ggWOUAVlpc6BGsY8UaAIDF7izcCBI+8 AASriBKmYcQbMaJhxBsB6i/e3ZDhxBvBImuD19SyeCNAZHnwoLJ4I0BkbbD7bM1+UfT21MOGJMDu Myh2NaCBk4YgYYH5AH/wNc/MYSbevTukJ7AYIYHYkB7YIX6S8meA29g9bUgQNEqsEiFhS/cD7NIp HUSYzltOElz/cxp8cQdgav0gpZ7uvDGnh8rHheB4kj04ZPzUDztzZGdX7Cy31swBIXu0Kz8CBEfR LsyBoPxYj+1sz/mYB+FQVd4M/7fNqfDZHHuLi2fG46Oz+fjoPD/gBCbrTkQb40Vkzkq1OJFvhfe7 k2AjfNWlhv3y4NbjYY3CuXzf/OPblXvuye5N02TmsMFvbfeIt/gMe8hbZy+AR1y86w6aY1vgUpeH fr8VPK1XiTuIZj5cpDYU5uQf/G/NhTy+Z86suX/Kk+Qbg2NrWu6aH034Wru7h2OokxVTK6m13Db3 z2AbOXjynAEzxWVn3KUdRPPcp/vtimfmHFjL/F9KW1/gvNrTxHU7Yl24/coz3kNeY2f90bfik/rw PwAAAP//AwBQSwECLQAUAAYACAAAACEA3fyVN2YBAAAgBQAAEwAAAAAAAAAAAAAAAAAAAAAAW0Nv bnRlbnRfVHlwZXNdLnhtbFBLAQItABQABgAIAAAAIQAekRq38wAAAE4CAAALAAAAAAAAAAAAAAAA AJ8DAABfcmVscy8ucmVsc1BLAQItABQABgAIAAAAIQDWZLNR+gAAADEDAAAcAAAAAAAAAAAAAAAA AMMGAAB3b3JkL19yZWxzL2RvY3VtZW50LnhtbC5yZWxzUEsBAi0AFAAGAAgAAAAhAKsR9BCDCgAA yx0AABEAAAAAAAAAAAAAAAAA/wgAAHdvcmQvZG9jdW1lbnQueG1sUEsBAi0AFAAGAAgAAAAhAJa1 reKWBgAAUBsAABUAAAAAAAAAAAAAAAAAsRMAAHdvcmQvdGhlbWUvdGhlbWUxLnhtbFBLAQItABQA BgAIAAAAIQChCYNYJQMAAB8HAAARAAAAAAAAAAAAAAAAAHoaAAB3b3JkL3NldHRpbmdzLnhtbFBL AQItABQABgAIAAAAIQBvY1CbiwEAAAcEAAASAAAAAAAAAAAAAAAAAM4dAAB3b3JkL2ZvbnRUYWJs ZS54bWxQSwECLQAUAAYACAAAACEAKIdxpc8AAAAfAQAAFAAAAAAAAAAAAAAAAACJHwAAd29yZC93 ZWJTZXR0aW5ncy54bWxQSwECLQAUAAYACAAAACEA94x95uABAADbAwAAEAAAAAAAAAAAAAAAAACK IAAAZG9jUHJvcHMvYXBwLnhtbFBLAQItABQABgAIAAAAIQBTfUAmVgEAAHkCAAARAAAAAAAAAAAA AAAAAKAjAABkb2NQcm9wcy9jb3JlLnhtbFBLAQItABQABgAIAAAAIQDmJsFqGgcAAPs5AAAPAAAA AAAAAAAAAAAAAC0mAAB3b3JkL3N0eWxlcy54bWxQSwUGAAAAAAsACwDBAgAAdC0AAAAA --===============1096080894==-- From david@fromorbit.com Mon Jan 4 13:40:48 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 42D327F37 for ; Mon, 4 Jan 2016 13:40:48 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id DEC44AC004 for ; Mon, 4 Jan 2016 11:40:47 -0800 (PST) X-ASG-Debug-ID: 1451936445-04bdf026514529d0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id QAvbNy9BbmyDhxrE for ; Mon, 04 Jan 2016 11:40:45 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DkCABpyopWPGu7LXleKAECgw+BP4ZggXmgSQEBAQEBAQaLVYk+hgkCAgEBAoEcTQEBAQEBAQcBAQEBQT+ENQEBBDocIxAIAw4KCSUPBSUDBxoTiC7BTgEBAQEGAQEBAQEeGYV1hUeIIYEbBYdbhxKIGY1IgWWERohZjjqFAio0g0UjgSgBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 06:10:15 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGAzG-0002Rc-UX; Tue, 05 Jan 2016 06:40:14 +1100 Date: Tue, 5 Jan 2016 06:40:14 +1100 From: Dave Chinner To: Julian Wollrath Cc: xfs@oss.sgi.com, Jiri Kosina Subject: Re: Suspend does not work with v4.4-rc8 Message-ID: <20160104194014.GD19802@dastard> X-ASG-Orig-Subj: Re: Suspend does not work with v4.4-rc8 References: <20160104202611.5b100540@saldaea> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104202611.5b100540@saldaea> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451936445 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25828 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 08:26:11PM +0100, Julian Wollrath wrote: > Hi, > > suspending (either to RAM or disk) fails with 4.4-rc8 with the > following message while v4.3.3 worked fine: > > kernel: Freezing of tasks failed after 20.006 seconds (2 tasks refusing to freeze, wq_bu_busy=0): > kernel: xfsaild/dm-1 S 0000000000014100 0 283 2 0x00000000 > kernel: ffff880213f53e10 ffffffff8180e4c0 ffff880213f05040 0000000000000000 > kernel: 0000000000000000 ffff880213f54000 0000000000000000 0000000000000000 > kernel: ffff8800ca389e40 ffff8800ca392000 ffff880213f53ed0 ffffffff814cd8ac > kernel: Call Trace: > kernel: [] ? schedule+0x2c/0x70 > kernel: [] ? xfsaild+0x4fd/0x5b0 > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > kernel: [] ? kthread+0xb8/0xd0 > kernel: [] ? kthread_worker_fn+0x150/0x150 > kernel: [] ? ret_from_fork+0x3f/0x70 > kernel: [] ? kthread_worker_fn+0x150/0x150 > kernel: xfsaild/sda1 S 0000000000014100 0 591 2 0x00000000 > kernel: ffff88021193be10 ffff8802159c4dc0 ffff880213ab9340 0000000000000000 > kernel: 0000000000000000 ffff88021193c000 0000000000000000 0000000000000000 > kernel: ffff8800ca2a4240 ffff880214eea000 ffff88021193bed0 ffffffff814cd8ac > kernel: Call Trace: > kernel: [] ? schedule+0x2c/0x70 > kernel: [] ? xfsaild+0x4fd/0x5b0 > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > kernel: [] ? kthread+0xb8/0xd0 > kernel: [] ? kthread_worker_fn+0x150/0x150 > kernel: [] ? ret_from_fork+0x3f/0x70 > kernel: [] ? kthread_worker_fn+0x150/0x150 > > Please tell me what more information you need to be able to fix this > issue. The freezer detection is broken. The thread is sleeping in schedule until a wakeup occurs some time in the future, which means it cannot "enter then freezer" because it's not a running thread. This is a problem introduced by commit 24ba16b ("xfs: clear PF_NOFREEZE for xfsaild kthread"). Jiri, I'm tempted just to revert this change - if the freezer doesn't detect processes that are not in TASK_RUNNABLE state as frozeni or can't mark them as frozen, then this change will never work reliably for XFS.... Cheers, Dave. -- Dave Chinner david@fromorbit.com From MAILER-DAEMON Mon Jan 4 13:43:04 2016 Return-Path: <> X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: * X-Spam-Status: No, score=1.9 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLYTO, TO_MALFORMED,T_FREEMAIL_DOC_PDF autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 5AF2C7F37 for ; Mon, 4 Jan 2016 13:43:04 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4921A8F8035 for ; Mon, 4 Jan 2016 11:43:04 -0800 (PST) X-ASG-Debug-ID: 1451936581-04cbb0101c1f8fd0001-NocioJ Received: from mail.gaf.ni.ac.rs (mail.gaf.ni.ac.rs [160.99.31.9]) by cuda.sgi.com with ESMTP id pN6oe33DT2FR706E (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:43:02 -0800 (PST) X-Barracuda-Envelope-From: X-Barracuda-Apparent-Source-IP: 160.99.31.9 Received: from localhost (localhost [127.0.0.1]) by mail.gaf.ni.ac.rs (Postfix) with ESMTP id 63AEC8F2F1C32; Mon, 4 Jan 2016 20:32:15 +0100 (CET) Received: from mail.gaf.ni.ac.rs ([127.0.0.1]) by localhost (mail.gaf.ni.ac.rs [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id wZkcjzMM0HJY; Mon, 4 Jan 2016 20:32:15 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mail.gaf.ni.ac.rs (Postfix) with ESMTP id B33D48F377A96; Mon, 4 Jan 2016 20:32:12 +0100 (CET) X-Virus-Scanned: amavisd-new at gaf.ni.ac.rs Received: from mail.gaf.ni.ac.rs ([127.0.0.1]) by localhost (mail.gaf.ni.ac.rs [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id tSadQxvBtZRd; Mon, 4 Jan 2016 20:32:12 +0100 (CET) Received: from [100.67.150.48] (unknown [116.202.38.88]) by mail.gaf.ni.ac.rs (Postfix) with ESMTPSA id A38108F3C3D6E; Mon, 4 Jan 2016 20:31:52 +0100 (CET) Content-Type: multipart/mixed; boundary="===============2055748759==" MIME-Version: 1.0 Subject: Re To: Recipients <> X-ASG-Orig-Subj: Re From: "<"@LIVE.COM Date: Tue, 05 Jan 2016 01:01:51 +0530 Reply-To: cruzcelia2011@gmail.com Message-Id: <20160104193152.A38108F3C3D6E@mail.gaf.ni.ac.rs> X-Barracuda-Connect: mail.gaf.ni.ac.rs[160.99.31.9] X-Barracuda-Start-Time: 1451936581 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.00 X-Barracuda-Spam-Status: No, SCORE=2.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0646, BSF_SC0_MV0646_4, EMPTY_ENV_FROM, NO_REAL_NAME, TO_MALFORMED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25827 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 TO_MALFORMED To: has a malformed address 0.00 EMPTY_ENV_FROM Empty Envelope From Address 0.00 NO_REAL_NAME From: does not include a real name 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.50 BSF_SC0_MV0646 Custom rule MV0646 1.50 BSF_SC0_MV0646_4 BSF_SC0_MV0646_4 You will not see this in a MIME-aware mail reader. --===============2055748759== Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body please kindly open the fill --===============2055748759== Content-Type: application/octet-stream MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="BUSINESS PROPOSAL.docx" UEsDBBQABgAIAAAAIQDd/JU3ZgEAACAFAAATAAgCW0NvbnRlbnRfVHlwZXNdLnhtbCCiBAIooAAC AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC0 VMtuwjAQvFfqP0S+Vomhh6qqCBz6OLZIpR9g7A1Y9Uv28vr7bgJEVQtBKuUSKVnvzOzsxIPR2pps CTFp70rWL3osAye90m5Wso/JS37PsoTCKWG8g5JtILHR8PpqMNkESBl1u1SyOWJ44DzJOViRCh/A UaXy0Qqk1zjjQchPMQN+2+vdcekdgsMcaww2HDxBJRYGs+c1fd4qiWASyx63B2uukokQjJYCSSlf OvWDJd8xFNTZnElzHdINyWD8IENdOU6w63sja6JWkI1FxFdhSQZf+ai48nJhaYaiG+aATl9VWkLb X6OF6CWkRJ5bU7QVK7Tb6z+qI+HGQPp/FVvcLnrSOY4+JE57OZsf6s0rUDlZESCihnZ1x0cHRLLs EsPvkLvGb1KAlHfgzbN/tgcNzEnKin6JiZgaOJvvV/Ja6JMiVjB9v5j738C7hLT5kz7+wYz9dVF3 H0gdb+634RcAAAD//wMAUEsDBBQABgAIAAAAIQAekRq38wAAAE4CAAALAAgCX3JlbHMvLnJlbHMg ogQCKKAAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAjJLbSgNBDIbvBd9hyH032woi0tneSKF3IusDhJnsAXcOzKTavr2jILpQ217m9OfLT9ab g5vUO6c8Bq9hWdWg2JtgR99reG23iwdQWchbmoJnDUfOsGlub9YvPJGUoTyMMaui4rOGQSQ+ImYz sKNchci+VLqQHEkJU4+RzBv1jKu6vsf0VwOamabaWQ1pZ+9AtcdYNl/WDl03Gn4KZu/Yy4kVyAdh b9kuYipsScZyjWop9SwabDDPJZ2RYqwKNuBpotX1RP9fi46FLAmhCYnP83x1nANaXg902aJ5x687 HyFZLBZ9e/tDg7MvaD4BAAD//wMAUEsDBBQABgAIAAAAIQDWZLNR+gAAADEDAAAcAAgBd29yZC9f cmVscy9kb2N1bWVudC54bWwucmVscyCiBAEooAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AKySzWrDMBCE74W+g9h7LTv9oYTIuZRArq37AIq9/qGyJLSbtn77CkNShwb34otgRmjmk7Sb7Xdv xCcG6pxVkCUpCLSlqzrbKHgvdnfPIIi1rbRxFhUMSLDNb282r2g0x0PUdp5ETLGkoGX2aympbLHX lDiPNu7ULvSaowyN9Lr80A3KVZo+yTDNgPwiU+wrBWFf3YMoBh+b/892dd2V+OLKY4+Wr1TILzy8 IXO8HMVYHRpkBRMzibQgr4OslgShPxQnZw4hWxSBBxM/8/wMNOq5+scl6zmOCP62j1KOazbH8LAk Q+0sF/pgJhxn6wQhLwY9/wEAAP//AwBQSwMEFAAGAAgAAAAhAKsR9BCDCgAAyx0AABEAAAB3b3Jk L2RvY3VtZW50LnhtbNRYzW7bRhC+F+g7DIQcWkCWKAcuEjVW4FpO40NTI7ZzX5FLcWFyl91dSuGt r9FbniWP0ifpN0tSliXFFgSjRU+0l9z5n++b0Zu3n4ucFtI6ZfRpbzSIeiR1bBKl56e925t3R696 5LzQiciNlqe9Wrre28n3371ZjhMTV4XUniBCu/ECbzPvy/Fw6OJMFsINTCk1XqbGFsLjXzsfFsLe VeVRbIpSeDVTufL18DiKfuq1Ysxpr7J63Io4KlRsjTOp5ytjk6Yqlu2ju2H30dvcnLYmB41DK3PY YLTLVOk6acWh0uBi1glZPObEosi775blPtoSK5bIR5E3Zi+NTUprYukcTqfNy5XEUfSY7jaALGJ1 Yx8THursLCmE0isxXB0b+V8lb4DkDRvdQxZ17whiMUEtzUxS87Ok5Ri1mHw87UXR+Un07mLa646m MhVV7vnNL+fHr6Yvw83yyvJF2zxmwyDt3OE5bE/x7D564svl2E9+tVJ6xNUN6CqXwkkqpS2Up0KS N6S0tyapYklF7WSe8lltKjKaj4yVA7okUZDA94mKRU4xCgxmCzTKrCYkLuXEGU1oKkIw7thQ35gb 7Ov8ZUePR6PXoyg4+oiHuM/dM3aliJGF0kon7UL2Jo6EY5VxZWVo1UJoMZeWlspn9Jjily9P3p1M 91A8uS6FVi6jq0ygzWNZeXZ7U/bTkd/tAZ0zUuiaw7qEM0rTpU6U4D+Oo9GIAC4INnssbJxRWdnS IGlJZZFEWmYKh5eUKBcbwJxMyGfC0w+b9h0U9MkNlAj6JK34MVgnFkLlYpbLlZ2D51A0s205H1wf QQKX2c2mPYcmJni+KeygKO5OPAd1q0KfNnZyLZFj5ahyeHJxhD5EmvKuNlx/w2w0iEkvLLpi7OsS DTS3orj2wvpegJOnAIbj+vULUr6P3Aud7C11d2TI+IxbGDiKfvDsrNDQL2KvFpLwbsbecwmi3asU 500zmJTOtFf0/vIT3eAzUdZ9eg+PrZc6QNIV2FlpB3C6mJlc0FShrZx0dOHjAe5AlWvj60oZK7Cx yPMaca4AZlAIyyhTfCSsFASN16YC1FwI5+nMKdEPsCfiGNZzgwI+0TDGCm9sTV7is1UvszAMHZRa U7RdnwECZlJqBrUFHrg+Ay7jwxmuMkhkAjFYKKc88t9ABdzxGWCdTBwL9tPxPSBFDHF8u3OLEZll JZhe8CcrA3qDDSrGDUMW4SA4jFBjKEFUKosRJ3h0q4NCVA18YL/PCmkRnYBbCEgwWCWBQ5gvWA3L RnKgzgM2+BKf5oZZI5EiR5JF6jnVEkERADIxhxsByGENcwesmkvcsls482SrPASV/T7nSr9EcVUh 6zsimG60APfUis/3JrMJ2vYjCPhl1KcoioZHzMEEcruTnsM02tTypPXf4MdoXjTVtRn4Ps0qJKYK +VLekRUWk2pINX+bqXnWlkkfRYdJIN8Cq8Ncv69GpPi3uuO/AGUvthxf17E/ZffpJAqhfa440u01 TU2eC+s2RR5m4dcvaMBNUYdmme6kLENv8fQFxAy0EMa3tbECYGDnwIj0HhkYBFxVloAzPg6DH2qh AiTk7TyCw8Iw1CBbgrRcoipKMEdoUgDiVMaymKF6c0bAGkMK46hcpZUxRpslibI0mC0hqJECudtD W4amZywJ8HY/By2lvHN///kXedUMqduN+e/BAzunnEPnWPSID84BX8PUhTnM4MBjVsI0zH3EAdvl KqIJPMckxZNloqwETGKHKCoNTOWdrZlgWcQ6XqJtZSyY/B6qylTBCUoVEgo+CqqZFnkoAEqHrLMs JrcmP82gHLKI1Bf3fQhfPvx+Q2IJgusge1vUOqOEfQBgbRWXCU+fnefNRgSM5KTKz2FwlZhSa4AA uFGlIHOEDITywHJwQWt3y8nQxlEFj8I3pojk524wCEfe1i3XItaeYwmHuNrmDGor1rxDITJVsjjk pdsVuFtSniy4ennWAK/yivNHBcpjElsoLD+ta6CqWGi2JmY8CMweRpEgdlZhZcX687AceUHj3v6v l79uSN49dgGL0aQcelQSUzjCxKHBn+BprHT8MwlKCW/X+q+pqd0k07GJBrsfBQoBNuE6C2zXnm82 yDLD9IPqECirBQoEUBOGnaZ613uiqa4gvivxzoNNgF3H6v3JmoPQsvUzkfUEvNwwMFu8Cx/QMc9j fNgKmjmQiYEnTc4tfMJE/WLUP9lU8zQJTcCt0ea1w0Ib0e1gxawwqFlnUXhNq7/eUrOHdX0kiS4/ fAyVAgwL4NHCoJZz41UDsMjqDIwolGbsAAhiCydsB0k/4Nc3ssK/d+Q5Q0ybOvxWshZXBo9S1CHG mMWz/yMQnIWZ/B8AAAD//6xWXW7bOBC+ysBA31xXTtLYzTYG6iQt+rCA0e4FKGkkEZZIlaSiuk+9 Rt/2LD3KnmS/oawkNookBQokkvgzw5lvvm/oUDG1zmad44ZNoEYZVbKjSnlqO5fhzTnJLs/4KJxt aGe7qTyo13VNOeddFuKWe4PM+kDK5LLN0dtX/UVYydOt3uJB+Pc6/3Q5SZLzk/nN4noS5zdxOX0l g/TK4y0mw2xYBaduua61KY/9PW1MX5v6wrcq48tJ69izu+XJivhryybXAdlPY7hI1AypOFvoMGSY IvlKOaSfcugZOzpPKtBp8oJe0iJ5MaW+0llFDSvj75EJasuyLA7Nz3/p4+AuTsN2Fqccq7reUW7J WJynUINgqbaeyXaBrISjPaWd14Y9PjhTHRYRHKaVIdu21oXO6LCjwjoEIQ4aObuxhjEpNZMyZLZp ldnN6J1HLF7pnFi5WrObxpxzRvVzssVwJGgRK4tj5kmSENDbcvB/jUenEl9le+HMblyN2WcIa+TC cLwQqLaZErqoejzQdE0KruHAve9ofWe5p1dg1yD3GKJqbAeExGJfIB5ALTnMDmgB5rTCi/Yh2c7W i+Wbm8k4dc2F6uogNFxfnSyvTwcaPkmn1EWKhl+z6iOpBlibQueiqFCBKr0O1aCFCjXxKJUgjZJm 7KRwklDnWql6v88oVrC0FqLTRplMA7xSaUP4O6DEbCRW2eHECCFYU3QOoLsobigXqgFbPHmLNbxF vjFGF/XsCDzEJmEY2HOA5Kjax9RZKx8+QUoMlWzQQNZwt30MpN667YBKwzNag0kqtbdMEMMU+Wy1 yaEKx186oBWjRVhb5nbIHVRuQPlMBY187sAGRjPa1IymJYryiEj0EJsQNgWFRrUnneB/nObIi9/r SyrP0VE86eIxfyfz+Zt58ow+92tS/aFYgcR/3384tA8DWQFazo+DfpL9YTWjfyplttJ+cvoAhqY1 ADjU33NYM+potT4O4mElno3cSqjiuFQu99M/4fEuvncQ6TdFV7Vy6tjxc/A6sBk7k+csbA7uwsP2 9BnrwsTXN4vk9fvInLb8/A3Q9JcTsCk5l0ZW4ft8ebqcRLW15d9KXAbbYv7sLIm9TpcVPI3D1IZg m/txzcWD1Qp9gN3lZJEsxbawFkS5G5ZdiMNkOC6ztcem/cUqJjGK3GYfHG6X/gK3NW90yBDl6Xk0 QvZD4vFyT22+ix8w6eQHyOp/AAAA//8DAFBLAwQUAAYACAAAACEAlrWt4pYGAABQGwAAFQAAAHdv cmQvdGhlbWUvdGhlbWUxLnhtbOxZT2/bNhS/D9h3IHRvYyd2Ggd1itixmy1NG8Ruhx5piZbYUKJA 0kl9G9rjgAHDumGHFdhth2FbgRbYpfs02TpsHdCvsEdSksVYXpI22IqtPiQS+eP7/x4fqavX7scM HRIhKU/aXv1yzUMk8XlAk7Dt3R72L615SCqcBJjxhLS9KZHetY3337uK11VEYoJgfSLXcduLlErX l5akD8NYXuYpSWBuzEWMFbyKcCkQ+AjoxmxpuVZbXYoxTTyU4BjI3hqPqU/QUJP0NnLiPQaviZJ6 wGdioEkTZ4XBBgd1jZBT2WUCHWLW9oBPwI+G5L7yEMNSwUTbq5mft7RxdQmvZ4uYWrC2tK5vftm6 bEFwsGx4inBUMK33G60rWwV9A2BqHtfr9bq9ekHPALDvg6ZWljLNRn+t3slplkD2cZ52t9asNVx8 if7KnMytTqfTbGWyWKIGZB8bc/i12mpjc9nBG5DFN+fwjc5mt7vq4A3I4lfn8P0rrdWGizegiNHk YA6tHdrvZ9QLyJiz7Ur4GsDXahl8hoJoKKJLsxjzRC2KtRjf46IPAA1kWNEEqWlKxtiHKO7ieCQo 1gzwOsGlGTvky7khzQtJX9BUtb0PUwwZMaP36vn3r54/RccPnh0/+On44cPjBz9aQs6qbZyE5VUv v/3sz8cfoz+efvPy0RfVeFnG//rDJ7/8/Hk1ENJnJs6LL5/89uzJi68+/f27RxXwTYFHZfiQxkSi m+QI7fMYFDNWcSUnI3G+FcMI0/KKzSSUOMGaSwX9nooc9M0pZpl3HDk6xLXgHQHlowp4fXLPEXgQ iYmiFZx3otgB7nLOOlxUWmFH8yqZeThJwmrmYlLG7WN8WMW7ixPHv71JCnUzD0tH8W5EHDH3GE4U DklCFNJz/ICQCu3uUurYdZf6gks+VuguRR1MK00ypCMnmmaLtmkMfplW6Qz+dmyzewd1OKvSeosc ukjICswqhB8S5pjxOp4oHFeRHOKYlQ1+A6uoSsjBVPhlXE8q8HRIGEe9gEhZteaWAH1LTt/BULEq 3b7LprGLFIoeVNG8gTkvI7f4QTfCcVqFHdAkKmM/kAcQohjtcVUF3+Vuhuh38ANOFrr7DiWOu0+v Brdp6Ig0CxA9MxHal1CqnQoc0+TvyjGjUI9tDFxcOYYC+OLrxxWR9bYW4k3Yk6oyYftE+V2EO1l0 u1wE9O2vuVt4kuwRCPP5jeddyX1Xcr3/fMldlM9nLbSz2gplV/cNtik2LXK8sEMeU8YGasrIDWma ZAn7RNCHQb3OnA5JcWJKI3jM6rqDCwU2a5Dg6iOqokGEU2iw654mEsqMdChRyiUc7MxwJW2NhyZd 2WNhUx8YbD2QWO3ywA6v6OH8XFCQMbtNaA6fOaMVTeCszFauZERB7ddhVtdCnZlb3YhmSp3DrVAZ fDivGgwW1oQGBEHbAlZehfO5Zg0HE8xIoO1u997cLcYLF+kiGeGAZD7Ses/7qG6clMeKuQmA2Knw kT7knWK1EreWJvsG3M7ipDK7xgJ2uffexEt5BM+8pPP2RDqypJycLEFHba/VXG56yMdp2xvDmRYe 4xS8LnXPh1kIF0O+EjbsT01mk+Uzb7ZyxdwkqMM1hbX7nMJOHUiFVFtYRjY0zFQWAizRnKz8y00w 60UpYCP9NaRYWYNg+NekADu6riXjMfFV2dmlEW07+5qVUj5RRAyi4AiN2ETsY3C/DlXQJ6ASriZM RdAvcI+mrW2m3OKcJV359srg7DhmaYSzcqtTNM9kCzd5XMhg3krigW6Vshvlzq+KSfkLUqUcxv8z VfR+AjcFK4H2gA/XuAIjna9tjwsVcahCaUT9voDGwdQOiBa4i4VpCCq4TDb/BTnU/23OWRomreHA p/ZpiASF/UhFgpA9KEsm+k4hVs/2LkuSZYRMRJXElakVe0QOCRvqGriq93YPRRDqpppkZcDgTsaf +55l0CjUTU4535waUuy9Ngf+6c7HJjMo5dZh09Dk9i9ErNhV7XqzPN97y4roiVmb1cizApiVtoJW lvavKcI5t1pbseY0Xm7mwoEX5zWGwaIhSuG+B+k/sP9R4TP7ZUJvqEO+D7UVwYcGTQzCBqL6km08 kC6QdnAEjZMdtMGkSVnTZq2Ttlq+WV9wp1vwPWFsLdlZ/H1OYxfNmcvOycWLNHZmYcfWdmyhqcGz J1MUhsb5QcY4xnzSKn914qN74OgtuN+fMCVNMME3JYGh9RyYPIDktxzN0o2/AAAA//8DAFBLAwQU AAYACAAAACEAoQmDWCUDAAAfBwAAEQAAAHdvcmQvc2V0dGluZ3MueG1snFXbbts4EH1fYP9B0PM6 lmQ7ToQ4RWNb3RZJW6zSD6CkkUyEN5CUFffrO5TEKul6g2KfTJ0zczicm2/ePXMWHEEbKsUmjC+i MABRyoqKZhN+e8xmV2FgLBEVYVLAJjyBCd/d/vnHTZcasBbNTIASwqRyE7ZapKY8ACdmxmmppZG1 nZWSp7KuaQnjTzh66E14sFal8/nodCEVCFSrpebEmgupm/nguZNly0HYeRJFl3MNjFgM2ByoMl6N /181vOrgRY5vPeLImbfr4ugty/G5ndTVT4/fCc85KC1LMAYzy9nwXE6o8DKG/Y7OkM97WmiiTy9E brFs36XkQZcq0CUmFGseReHcEZX8LO2OGsXI6Stp4E62WHZNwfQ0xiXr3BIL6G0UMNb3SMmAYHRd 2mjCOcGaDsggCTVpmX0kRW6lQqMjwfjXyXhjeSCalBZ0rkiJalsprJbM2/UBbSVXGvMxxIi9pIh1 0bYGsv09OcnW4lXzLp0obObKOBt3+EdK6wWj6G6bXO0Wg5ZjJyaJ4+t4jOs1s1isstXunM/ybn11 vT/HrPbraJWdYy6TeL8+q/Y+W27jsz7/HfV2FWX7s2rZ1fI6XrsIMDdjRnjqev2rvr0ZThnmO+BD UbaEF5qS4MFNA3rxtNBPd1R4vgCcSnjJ5G3hydlsIAwnjGVYU0/gIAxMhX21g7oXZg9EN5Nyn3Se 6rNoBfWnn2quYUF/0LJVg2qnifooKoT9hfFyOepRYe8p97hpi9x7CRyKFxR2+ZejdoLzKUFdanGP gcvQPRGN7xMQs2+5M8V+Yzp3uw4eiFLYvGhSNPEmZLQ52NhNhMUvHKCn/qNokpFLeg6/HNd/kNK9 DK3HgzMYjmg1HiZs4bHFhC09tpywlcdWE3bpsUuHHU64BXCMn3Cn+KPDa8mY7KD624Ob8F/QkARz IAqwrm7KceRk2gNYtB4Ijik844qBilr8G1G04uR5E67iJHHuozWuGxziV7aOc8bqFRpUxBJcWH2p Xjn3Tf5LLLjQoKTYkPmJF9NSuRgCZ9TYHBTuHys1PrlfTH/1ytM/2+0PAAAA//8DAFBLAwQUAAYA CAAAACEAb2NQm4sBAAAHBAAAEgAAAHdvcmQvZm9udFRhYmxlLnhtbKSS326CMBTG75fsHUjvJwX/ TI1onNPLXSzuAY5YpAltSU+V+fY7UORiZolkkDThO+3H6e98i9W3KoKLsCiNTlg04CwQOjVHqU8J +9rvXqYsQAf6CIXRImFXgWy1fH5aVPPMaIcBndc4twnLnSvnYYhpLhTgwJRCUy0zVoGjT3sKTZbJ VLyb9KyEdmHM+SS0ogBH/8Zclshat+oRt8rYY2lNKhCpWVV4PwVSs2XbXVDNNSjqegOFPFjZFErQ BkVEtQsUCeMx3/ExrfU74sN6ZWHtkOZgUbhuI/dyBkoW15uKlUT0hVK6NL/pF7ASDoXwJZQnKpzx wBO2jTjn8W7HvBIlbETCetMpMTXln1m7Z9gpNB5qrPFptkSzxocU8mlPNX2Gfj53JPZSCQw+RBV8 GgUe1T2RmE+IxJh41GSGvYjYxrch+CgRajxed/enm2xIeZ2Oovb+vYh4n8eJbEBRNOCPbNQEPIma SL9s9CexrkcYb39lg/PR2x2JJgmUqP9kow0JLn8AAAD//wMAUEsDBBQABgAIAAAAIQAoh3GlzwAA AB8BAAAUAAAAd29yZC93ZWJTZXR0aW5ncy54bWyMj8tOAzEMRfdI/MMoe5qBRYVGnamEUNlQqMRj n2Y8nUiJHdmB0H495rFhx/LaV8fHq/VHis07sATC3lwuWtMAehoDHnrz8ry5uDaNFIeji4TQmyOI WQ/nZ6vaVdg/QSnalEYpKB33Zi4ld9aKnyE5WVAG1N1EnFzRyAdL0xQ83JJ/S4DFXrXt0jJEV9RA 5pDF/NLqf2iVeMxMHkRUJMUfXnIBzaCOlEtI4QQb4humKsD2a6z3jo/4ur3/Ti5GqruHOw32z1vD JwAAAP//AwBQSwMEFAAGAAgAAAAhAPeMfebgAQAA2wMAABAACAFkb2NQcm9wcy9hcHAueG1sIKIE ASigAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnFPBbtswDL0P2D8Yvjd2nCUoAkXFkGLo YVsDxG3PnEzHwmRJkNSg2dePshtX2XqqT+Qj/fT0SLGbl15lR3ReGr3J57Myz1AL00h92OQP9ber 6zzzAXQDymjc5Cf0+Q3//IntnLHogkSfEYX2m7wLwa6LwosOe/AzKmuqtMb1ECh1h8K0rRR4a8Rz jzoUVVmuCnwJqBtsruxEmI+M62P4KGljRNTnH+uTJcGc1dhbBQH5zyhHsWICWG0CqFr2yBcl4VPG dnBAz+esGAP2ZFzj+bIkZAzZtgMHIpB7vLpe0d8JwL5aq6SAQMbyH1I4400bsvvBgiwSsCJtYWTL HsWzk+HEiSpN2XepSUq1YMUYkTYHBwe283wVBU4Z2wtQuKXL8xaUR1a8AewOIQ52B5IUs2NYH1EE 4zIv/9Boqzz7BR6jZZv8CE6CDmRdbBuTIVbWB8drGRRxU23MhzBtS2P5JbpIvRRcNkZw1ECFS3XD Cf6+pbuFd8TOU7GDhlFqIicJpzP+Yd2a3oI+0eFTRAb/9g+2NrdxXV49vASTuT/J0O0tCJrOYrFc phuQlNieFgUbGumZ8A1gd+S3U/FU2h59wObc838h7tTj+Fb5vJqV9A1LdMZoE6ZHxP8CAAD//wMA UEsDBBQABgAIAAAAIQBTfUAmVgEAAHkCAAARAAgBZG9jUHJvcHMvY29yZS54bWwgogQBKKAAAQAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACMkstuwjAQRfeV+g+R94mTAC2ykiD1wQraSk0f6s6y B4gaP2S7BP6+TgIpqF10Ob7XZ+6Mnc12og62YGylZI6SKEYBSKZ4Jdc5einn4RQF1lHJaa0k5GgP Fs2Ky4uMacKUgSejNBhXgQ08SVrCdI42zmmCsWUbENRG3iG9uFJGUOdLs8aask+6BpzG8RUW4Cin juIWGOqBiA5Izgak/jJ1B+AMQw0CpLM4iRL843VghP3zQqecOEXl9trPdIh7yuasFwf3zlaDsWma qBl1MXz+BL8vF8/dqGEl210xQEXGGWEGqFOmqEGqrcrwyVG7vppat/SbXlXAb/bF4v7h8fUxw7+V 1mxgW7VvVEw6x1D6Pt1YfTPggQ9K+rGOytvo9q6coyKNk0mYpGEyLuNrMhmROP5oQ53db4P3B+IQ 7R/EdFqmMUnG58QjoOgSn3+W4hsAAP//AwBQSwMEFAAGAAgAAAAhAOYmwWoaBwAA+zkAAA8AAAB3 b3JkL3N0eWxlcy54bWy0m1FT2zgQx99v5r6Dx+80ISnJlWnaoRSuzLQcbWDuWbEVoqlj5SylQD/9 SStbGBvbu9g8EcvW/lba1X+dIL3/eL9Ngl88U0Kmi/DwzTgMeBrJWKS3i/Dm+vzgrzBQmqUxS2TK F+EDV+HHD3/+8f7uWOmHhKvAGEjVcbYIN1rvjkcjFW34lqk3csdTc28tsy3T5jK7Hcn1WkT8s4z2 W57q0WQ8no0ynjBt4GojdirMrd1hrN3JLN5lMuJKGW+3ibO3ZSINPxj3Yhl95mu2T7Syl9lVll/m V/DnXKZaBXfHTEVCXBvHzRC3IpXZl5NUidDc4UzpEyVY+eZZ3mbvb+yD5Zu+Z6R0yeAnEYtwZKHq t+n2iyWLcDIpWk6tE0/aEpbeFm08Pbi4LDuzCH3TythdhCw7WJ5YYyMYafG3NOLdk/GbK3BlxyIz d4bD1pqbGJqQWE4ibKwn81lx8WOfmAa21zKHgAEDK5s1l5VJN6E1gV66RDF3+fqrjH7yeKnNjUUI LNN4c3GVCZkJ/bAI372zTNO45FvxRcQxt3mZt92kGxHzfzc8vVE8fmz/fg5ZlluM5D7Vxv3ZHBIh UfHZfcR3NsuM6ZTZIF/aDok1q0occGgvHr1xDRUqNP5XIA9dDJ+lbDizKykA/1tBMOp9b9DEjqg8 ALBL8nXa38Tb/iaO+puA5O03F/P+Xhj97BsRlxulrMQHVcvIJV95HqbvWlLW9qhlUWePWtJ09qjl SGePWkp09qhlQGePWsA7e9Ti29mjFs7WHhED4apm0RRmA7Wwr4VOuO3fKkCHPaUuLzXBFcvYbcZ2 m8DW1qrbbWK53K80zlWQ05eL5VJnMr3tnBFTne3SfbEmn213G6aEeanpmPpJz6m/ZquEB39nIu5E Hbnkq40JXkyeLWFXCYv4RiYxz4Jrfu8iSuh/KYOle8vodK5nWL+K240OlhsouZ2wWcOkN8+Es/9V KJiD1sU0axhKl3FUDGcNedls/BuPxX5bTA3ibWTm9JwQ5goCXGyforc2RPXV1TkKGwDMEFy5oA8B 7CP8d8WFbt/GGOO/K0UvtI/w3xWuF9qH/GiPL1lpPrPsZ4BaXnPy2j2ViczW+6RYA53yMCevYI/A DYG8iL19lEjMySv4iXwGJ1Fkvrlh8pQci0cdJVDI4XAUWGz4sZCDUpG9Q8KIyAGqsCYEVj+tJYDI ovuD/xL2tydqMQCV9u+anct52jADpgSh3qG/76XufoeeNGgelnKRmp9LFA9wtGnDysPS8nxy9Y4Q 436FjwDqVwEJoH6lkABqyI/mdx5fE/GQ/sWRwCLLsq9ikHZoZZ6TldmDaCVgoLqJeP9qWL3NuVCv mwgKOUD1uomgkKNTqWW+biJYg9VNBKuhajTHqKyplEGR62YZ5N8EECMaRrwRoGHEGwEaRrwRoP7i 3Q0ZTrwRLLI2eE0tizcCBI9Qvup7UFm8ESCyNji1y38zKuoeWGn/cjuAeCMo5ADVxRtBIUenSbwR LHiEkgkVlpc6BGsY8UaAhhFvBGgY8UaAhhFvBGgY8UaA+ot3N2Q48UawyNrgNbUs3ggQWR48qCze CBA8QtGGZ8UbVv2rizeCQg5QXbwRFHJ0KoLqX1IRLHKAKiwv3ggWPEJJhpwFyU0Z1DDijRjRMOKN AA0j3gjQMOKNAPUX727IcOKNYJG1wWtqWbwRILI8eFBZvBEgsjY8K96wGF9dvBEUcoDq4o2gkKNT EVSvcwgWOUAVlhdvBAvypbd4I0DwyEtBlBENI96IEQ0j3gjQMOKNAPUX727IcOKNYJG1wWtqWbwR ILI8eFBZvBEgsjY8K96wRl5dvBEUcoDq4o2gkKNTEVQv3ggWOUAVlpc6BGsY8UaAIDF7izcCBI+8 AASriBKmYcQbMaJhxBsB6i/e3ZDhxBvBImuD19SyeCNAZHnwoLJ4I0BkbbD7bM1+UfT21MOGJMDu Myh2NaCBk4YgYYH5AH/wNc/MYSbevTukJ7AYIYHYkB7YIX6S8meA29g9bUgQNEqsEiFhS/cD7NIp HUSYzltOElz/cxp8cQdgav0gpZ7uvDGnh8rHheB4kj04ZPzUDztzZGdX7Cy31swBIXu0Kz8CBEfR LsyBoPxYj+1sz/mYB+FQVd4M/7fNqfDZHHuLi2fG46Oz+fjoPD/gBCbrTkQb40Vkzkq1OJFvhfe7 k2AjfNWlhv3y4NbjYY3CuXzf/OPblXvuye5N02TmsMFvbfeIt/gMe8hbZy+AR1y86w6aY1vgUpeH fr8VPK1XiTuIZj5cpDYU5uQf/G/NhTy+Z86suX/Kk+Qbg2NrWu6aH034Wru7h2OokxVTK6m13Db3 z2AbOXjynAEzxWVn3KUdRPPcp/vtimfmHFjL/F9KW1/gvNrTxHU7Yl24/coz3kNeY2f90bfik/rw PwAAAP//AwBQSwECLQAUAAYACAAAACEA3fyVN2YBAAAgBQAAEwAAAAAAAAAAAAAAAAAAAAAAW0Nv bnRlbnRfVHlwZXNdLnhtbFBLAQItABQABgAIAAAAIQAekRq38wAAAE4CAAALAAAAAAAAAAAAAAAA AJ8DAABfcmVscy8ucmVsc1BLAQItABQABgAIAAAAIQDWZLNR+gAAADEDAAAcAAAAAAAAAAAAAAAA AMMGAAB3b3JkL19yZWxzL2RvY3VtZW50LnhtbC5yZWxzUEsBAi0AFAAGAAgAAAAhAKsR9BCDCgAA yx0AABEAAAAAAAAAAAAAAAAA/wgAAHdvcmQvZG9jdW1lbnQueG1sUEsBAi0AFAAGAAgAAAAhAJa1 reKWBgAAUBsAABUAAAAAAAAAAAAAAAAAsRMAAHdvcmQvdGhlbWUvdGhlbWUxLnhtbFBLAQItABQA BgAIAAAAIQChCYNYJQMAAB8HAAARAAAAAAAAAAAAAAAAAHoaAAB3b3JkL3NldHRpbmdzLnhtbFBL AQItABQABgAIAAAAIQBvY1CbiwEAAAcEAAASAAAAAAAAAAAAAAAAAM4dAAB3b3JkL2ZvbnRUYWJs ZS54bWxQSwECLQAUAAYACAAAACEAKIdxpc8AAAAfAQAAFAAAAAAAAAAAAAAAAACJHwAAd29yZC93 ZWJTZXR0aW5ncy54bWxQSwECLQAUAAYACAAAACEA94x95uABAADbAwAAEAAAAAAAAAAAAAAAAACK IAAAZG9jUHJvcHMvYXBwLnhtbFBLAQItABQABgAIAAAAIQBTfUAmVgEAAHkCAAARAAAAAAAAAAAA AAAAAKAjAABkb2NQcm9wcy9jb3JlLnhtbFBLAQItABQABgAIAAAAIQDmJsFqGgcAAPs5AAAPAAAA AAAAAAAAAAAAAC0mAAB3b3JkL3N0eWxlcy54bWxQSwUGAAAAAAsACwDBAgAAdC0AAAAA --===============2055748759==-- From darrick.wong@oracle.com Mon Jan 4 13:47:23 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 738967F37 for ; Mon, 4 Jan 2016 13:47:23 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4370A8F8033 for ; Mon, 4 Jan 2016 11:47:23 -0800 (PST) X-ASG-Debug-ID: 1451936841-04bdf02650452c40001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id tAFFWHZItvdqmJ34 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:47:21 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u04JlG2c010922 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 19:47:17 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u04JlGm1031101 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 19:47:16 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u04JlFh7015076; Mon, 4 Jan 2016 19:47:15 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 11:47:14 -0800 Date: Mon, 4 Jan 2016 11:47:08 -0800 From: "Darrick J. Wong" To: Dave Chinner Cc: linux-ext4@vger.kernel.org, linux-btrfs@vger.kernel.org, fstests@vger.kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH 06/11] reflink: more tests Message-ID: <20160104194708.GA28357@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 06/11] reflink: more tests References: <20151219091052.15320.94973.stgit@birch.djwong.org> <20151219091131.15320.12391.stgit@birch.djwong.org> <20151221063111.GO19802@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151221063111.GO19802@dastard> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0021.oracle.com [156.151.31.71] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1451936841 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25828 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Mon, Dec 21, 2015 at 05:31:11PM +1100, Dave Chinner wrote: > On Sat, Dec 19, 2015 at 01:11:31AM -0800, Darrick J. Wong wrote: > > Add more tests for unaligned copy-on-write things, and explicitly > > test the ability to pass "len == 0" to mean reflink/dedupe all > > the way to the end of the file". > > > > Signed-off-by: Darrick J. Wong > ..... > > +_cleanup() > > +{ > > + cd / > > + rm -rf "$tmp".* "$TESTDIR" > > +} > ..... > > + > > +TESTDIR="$TEST_DIR/test-$seq" > > +rm -rf "$TESTDIR" > > +mkdir "$TESTDIR" > > This use of TESTDIR is highly confusing. When I see TESTDIR I think > that it's the mount point of the test device, not potentially some > directory on the scratch device. > > I see that this occurs through lots of tests, not just these new > ones. Can you do a pass across the tests with, say, sed and rename > all these to something less confusing? e.g. "testdir", in lower > case, makes it clear that it's a local variable, not the global test > device mount point.... (i.e. test local variables are lower case, > exported test harnes variables are upper case ;) > > Separate patch/pull req is fine. No problem. Will give it a spin on the test harness before sending. (Yikes, it's a 272K patch...) --D > > Cheers, > > dave. > -- > Dave Chinner > david@fromorbit.com From darrick.wong@oracle.com Mon Jan 4 13:58:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 633057F37 for ; Mon, 4 Jan 2016 13:58:18 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id F3C57AC006 for ; Mon, 4 Jan 2016 11:58:17 -0800 (PST) X-ASG-Debug-ID: 1451937495-04cb6c303042bdf0001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id iJ7lxcKU0tFkAtRt (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 11:58:16 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u04Jvq5H023823 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 19:57:52 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u04JvpVl005911 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 19:57:52 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u04JvpqS020052; Mon, 4 Jan 2016 19:57:51 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 11:57:51 -0800 Date: Mon, 4 Jan 2016 11:57:48 -0800 From: "Darrick J. Wong" To: Dave Chinner Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [PATCH 02/76] xfs: fix log ticket type printing Message-ID: <20160104195748.GC28357@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 02/76] xfs: fix log ticket type printing References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219085636.12713.59155.stgit@birch.djwong.org> <20160103121353.GA28429@infradead.org> <20160103212905.GY19802@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160103212905.GY19802@dastard> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0022.oracle.com [156.151.31.74] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1451937496 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25828 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Mon, Jan 04, 2016 at 08:29:05AM +1100, Dave Chinner wrote: > On Sun, Jan 03, 2016 at 04:13:53AM -0800, Christoph Hellwig wrote: > > On Sat, Dec 19, 2015 at 12:56:36AM -0800, Darrick J. Wong wrote: > > > Update the log ticket reservation type printing code to reflect > > > all the types of log tickets, to avoid incorrect debug output and > > > avoid running off the end of the array. > > > > This looks okay, but I think we should use name array initializers > > to make sure it doesn't got out of sync ever. > > Good idea, separate patch is fine. Will do. --D > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com From MAILER-DAEMON Mon Jan 4 14:07:07 2016 Return-Path: <> X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: * X-Spam-Status: No, score=1.9 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLYTO, TO_MALFORMED,T_FREEMAIL_DOC_PDF autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3FE0C7F37 for ; Mon, 4 Jan 2016 14:07:07 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 218EE304039 for ; Mon, 4 Jan 2016 12:07:04 -0800 (PST) X-ASG-Debug-ID: 1451938019-04cb6c303242c120001-NocioJ Received: from mail.gaf.ni.ac.rs (mail.gaf.ni.ac.rs [160.99.31.9]) by cuda.sgi.com with ESMTP id gC71a6wp0QOSoQcH (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 12:07:00 -0800 (PST) X-Barracuda-Envelope-From: X-Barracuda-Apparent-Source-IP: 160.99.31.9 Received: from localhost (localhost [127.0.0.1]) by mail.gaf.ni.ac.rs (Postfix) with ESMTP id A56A2811C449B; Mon, 4 Jan 2016 21:05:56 +0100 (CET) Received: from mail.gaf.ni.ac.rs ([127.0.0.1]) by localhost (mail.gaf.ni.ac.rs [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id FII5rwOfGjGL; Mon, 4 Jan 2016 21:05:56 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mail.gaf.ni.ac.rs (Postfix) with ESMTP id 2FAC180C673F4; Mon, 4 Jan 2016 21:05:56 +0100 (CET) X-Virus-Scanned: amavisd-new at gaf.ni.ac.rs Received: from mail.gaf.ni.ac.rs ([127.0.0.1]) by localhost (mail.gaf.ni.ac.rs [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id rjmkrAXBPKu8; Mon, 4 Jan 2016 21:05:56 +0100 (CET) Received: from [100.67.150.48] (unknown [116.202.38.88]) by mail.gaf.ni.ac.rs (Postfix) with ESMTPSA id 29C4182457A74; Mon, 4 Jan 2016 21:05:34 +0100 (CET) Content-Type: multipart/mixed; boundary="===============0921340515==" MIME-Version: 1.0 Subject: Re To: Recipients <> X-ASG-Orig-Subj: Re From: "<"@LIVE.COM Date: Tue, 05 Jan 2016 01:35:14 +0530 Reply-To: cruzcelia2011@gmail.com Message-Id: <20160104200535.29C4182457A74@mail.gaf.ni.ac.rs> X-Barracuda-Connect: mail.gaf.ni.ac.rs[160.99.31.9] X-Barracuda-Start-Time: 1451938020 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.00 X-Barracuda-Spam-Status: No, SCORE=2.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0646, BSF_SC0_MV0646_4, EMPTY_ENV_FROM, NO_REAL_NAME, TO_MALFORMED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25828 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 TO_MALFORMED To: has a malformed address 0.00 EMPTY_ENV_FROM Empty Envelope From Address 0.00 NO_REAL_NAME From: does not include a real name 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.50 BSF_SC0_MV0646 Custom rule MV0646 1.50 BSF_SC0_MV0646_4 BSF_SC0_MV0646_4 You will not see this in a MIME-aware mail reader. --===============0921340515== Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body please kindly open the fill --===============0921340515== Content-Type: application/octet-stream MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="BUSINESS PROPOSAL.docx" UEsDBBQABgAIAAAAIQDd/JU3ZgEAACAFAAATAAgCW0NvbnRlbnRfVHlwZXNdLnhtbCCiBAIooAAC AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC0 VMtuwjAQvFfqP0S+Vomhh6qqCBz6OLZIpR9g7A1Y9Uv28vr7bgJEVQtBKuUSKVnvzOzsxIPR2pps CTFp70rWL3osAye90m5Wso/JS37PsoTCKWG8g5JtILHR8PpqMNkESBl1u1SyOWJ44DzJOViRCh/A UaXy0Qqk1zjjQchPMQN+2+vdcekdgsMcaww2HDxBJRYGs+c1fd4qiWASyx63B2uukokQjJYCSSlf OvWDJd8xFNTZnElzHdINyWD8IENdOU6w63sja6JWkI1FxFdhSQZf+ai48nJhaYaiG+aATl9VWkLb X6OF6CWkRJ5bU7QVK7Tb6z+qI+HGQPp/FVvcLnrSOY4+JE57OZsf6s0rUDlZESCihnZ1x0cHRLLs EsPvkLvGb1KAlHfgzbN/tgcNzEnKin6JiZgaOJvvV/Ja6JMiVjB9v5j738C7hLT5kz7+wYz9dVF3 H0gdb+634RcAAAD//wMAUEsDBBQABgAIAAAAIQAekRq38wAAAE4CAAALAAgCX3JlbHMvLnJlbHMg ogQCKKAAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAjJLbSgNBDIbvBd9hyH032woi0tneSKF3IusDhJnsAXcOzKTavr2jILpQ217m9OfLT9ab g5vUO6c8Bq9hWdWg2JtgR99reG23iwdQWchbmoJnDUfOsGlub9YvPJGUoTyMMaui4rOGQSQ+ImYz sKNchci+VLqQHEkJU4+RzBv1jKu6vsf0VwOamabaWQ1pZ+9AtcdYNl/WDl03Gn4KZu/Yy4kVyAdh b9kuYipsScZyjWop9SwabDDPJZ2RYqwKNuBpotX1RP9fi46FLAmhCYnP83x1nANaXg902aJ5x687 HyFZLBZ9e/tDg7MvaD4BAAD//wMAUEsDBBQABgAIAAAAIQDWZLNR+gAAADEDAAAcAAgBd29yZC9f cmVscy9kb2N1bWVudC54bWwucmVscyCiBAEooAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AKySzWrDMBCE74W+g9h7LTv9oYTIuZRArq37AIq9/qGyJLSbtn77CkNShwb34otgRmjmk7Sb7Xdv xCcG6pxVkCUpCLSlqzrbKHgvdnfPIIi1rbRxFhUMSLDNb282r2g0x0PUdp5ETLGkoGX2aympbLHX lDiPNu7ULvSaowyN9Lr80A3KVZo+yTDNgPwiU+wrBWFf3YMoBh+b/892dd2V+OLKY4+Wr1TILzy8 IXO8HMVYHRpkBRMzibQgr4OslgShPxQnZw4hWxSBBxM/8/wMNOq5+scl6zmOCP62j1KOazbH8LAk Q+0sF/pgJhxn6wQhLwY9/wEAAP//AwBQSwMEFAAGAAgAAAAhAKsR9BCDCgAAyx0AABEAAAB3b3Jk L2RvY3VtZW50LnhtbNRYzW7bRhC+F+g7DIQcWkCWKAcuEjVW4FpO40NTI7ZzX5FLcWFyl91dSuGt r9FbniWP0ifpN0tSliXFFgSjRU+0l9z5n++b0Zu3n4ucFtI6ZfRpbzSIeiR1bBKl56e925t3R696 5LzQiciNlqe9Wrre28n3371ZjhMTV4XUniBCu/ECbzPvy/Fw6OJMFsINTCk1XqbGFsLjXzsfFsLe VeVRbIpSeDVTufL18DiKfuq1Ysxpr7J63Io4KlRsjTOp5ytjk6Yqlu2ju2H30dvcnLYmB41DK3PY YLTLVOk6acWh0uBi1glZPObEosi775blPtoSK5bIR5E3Zi+NTUprYukcTqfNy5XEUfSY7jaALGJ1 Yx8THursLCmE0isxXB0b+V8lb4DkDRvdQxZ17whiMUEtzUxS87Ok5Ri1mHw87UXR+Un07mLa646m MhVV7vnNL+fHr6Yvw83yyvJF2zxmwyDt3OE5bE/x7D564svl2E9+tVJ6xNUN6CqXwkkqpS2Up0KS N6S0tyapYklF7WSe8lltKjKaj4yVA7okUZDA94mKRU4xCgxmCzTKrCYkLuXEGU1oKkIw7thQ35gb 7Ov8ZUePR6PXoyg4+oiHuM/dM3aliJGF0kon7UL2Jo6EY5VxZWVo1UJoMZeWlspn9Jjily9P3p1M 91A8uS6FVi6jq0ygzWNZeXZ7U/bTkd/tAZ0zUuiaw7qEM0rTpU6U4D+Oo9GIAC4INnssbJxRWdnS IGlJZZFEWmYKh5eUKBcbwJxMyGfC0w+b9h0U9MkNlAj6JK34MVgnFkLlYpbLlZ2D51A0s205H1wf QQKX2c2mPYcmJni+KeygKO5OPAd1q0KfNnZyLZFj5ahyeHJxhD5EmvKuNlx/w2w0iEkvLLpi7OsS DTS3orj2wvpegJOnAIbj+vULUr6P3Aud7C11d2TI+IxbGDiKfvDsrNDQL2KvFpLwbsbecwmi3asU 500zmJTOtFf0/vIT3eAzUdZ9eg+PrZc6QNIV2FlpB3C6mJlc0FShrZx0dOHjAe5AlWvj60oZK7Cx yPMaca4AZlAIyyhTfCSsFASN16YC1FwI5+nMKdEPsCfiGNZzgwI+0TDGCm9sTV7is1UvszAMHZRa U7RdnwECZlJqBrUFHrg+Ay7jwxmuMkhkAjFYKKc88t9ABdzxGWCdTBwL9tPxPSBFDHF8u3OLEZll JZhe8CcrA3qDDSrGDUMW4SA4jFBjKEFUKosRJ3h0q4NCVA18YL/PCmkRnYBbCEgwWCWBQ5gvWA3L RnKgzgM2+BKf5oZZI5EiR5JF6jnVEkERADIxhxsByGENcwesmkvcsls482SrPASV/T7nSr9EcVUh 6zsimG60APfUis/3JrMJ2vYjCPhl1KcoioZHzMEEcruTnsM02tTypPXf4MdoXjTVtRn4Ps0qJKYK +VLekRUWk2pINX+bqXnWlkkfRYdJIN8Cq8Ncv69GpPi3uuO/AGUvthxf17E/ZffpJAqhfa440u01 TU2eC+s2RR5m4dcvaMBNUYdmme6kLENv8fQFxAy0EMa3tbECYGDnwIj0HhkYBFxVloAzPg6DH2qh AiTk7TyCw8Iw1CBbgrRcoipKMEdoUgDiVMaymKF6c0bAGkMK46hcpZUxRpslibI0mC0hqJECudtD W4amZywJ8HY/By2lvHN///kXedUMqduN+e/BAzunnEPnWPSID84BX8PUhTnM4MBjVsI0zH3EAdvl KqIJPMckxZNloqwETGKHKCoNTOWdrZlgWcQ6XqJtZSyY/B6qylTBCUoVEgo+CqqZFnkoAEqHrLMs JrcmP82gHLKI1Bf3fQhfPvx+Q2IJgusge1vUOqOEfQBgbRWXCU+fnefNRgSM5KTKz2FwlZhSa4AA uFGlIHOEDITywHJwQWt3y8nQxlEFj8I3pojk524wCEfe1i3XItaeYwmHuNrmDGor1rxDITJVsjjk pdsVuFtSniy4ennWAK/yivNHBcpjElsoLD+ta6CqWGi2JmY8CMweRpEgdlZhZcX687AceUHj3v6v l79uSN49dgGL0aQcelQSUzjCxKHBn+BprHT8MwlKCW/X+q+pqd0k07GJBrsfBQoBNuE6C2zXnm82 yDLD9IPqECirBQoEUBOGnaZ613uiqa4gvivxzoNNgF3H6v3JmoPQsvUzkfUEvNwwMFu8Cx/QMc9j fNgKmjmQiYEnTc4tfMJE/WLUP9lU8zQJTcCt0ea1w0Ib0e1gxawwqFlnUXhNq7/eUrOHdX0kiS4/ fAyVAgwL4NHCoJZz41UDsMjqDIwolGbsAAhiCydsB0k/4Nc3ssK/d+Q5Q0ybOvxWshZXBo9S1CHG mMWz/yMQnIWZ/B8AAAD//6xWXW7bOBC+ysBA31xXTtLYzTYG6iQt+rCA0e4FKGkkEZZIlaSiuk+9 Rt/2LD3KnmS/oawkNookBQokkvgzw5lvvm/oUDG1zmad44ZNoEYZVbKjSnlqO5fhzTnJLs/4KJxt aGe7qTyo13VNOeddFuKWe4PM+kDK5LLN0dtX/UVYydOt3uJB+Pc6/3Q5SZLzk/nN4noS5zdxOX0l g/TK4y0mw2xYBaduua61KY/9PW1MX5v6wrcq48tJ69izu+XJivhryybXAdlPY7hI1AypOFvoMGSY IvlKOaSfcugZOzpPKtBp8oJe0iJ5MaW+0llFDSvj75EJasuyLA7Nz3/p4+AuTsN2Fqccq7reUW7J WJynUINgqbaeyXaBrISjPaWd14Y9PjhTHRYRHKaVIdu21oXO6LCjwjoEIQ4aObuxhjEpNZMyZLZp ldnN6J1HLF7pnFi5WrObxpxzRvVzssVwJGgRK4tj5kmSENDbcvB/jUenEl9le+HMblyN2WcIa+TC cLwQqLaZErqoejzQdE0KruHAve9ofWe5p1dg1yD3GKJqbAeExGJfIB5ALTnMDmgB5rTCi/Yh2c7W i+Wbm8k4dc2F6uogNFxfnSyvTwcaPkmn1EWKhl+z6iOpBlibQueiqFCBKr0O1aCFCjXxKJUgjZJm 7KRwklDnWql6v88oVrC0FqLTRplMA7xSaUP4O6DEbCRW2eHECCFYU3QOoLsobigXqgFbPHmLNbxF vjFGF/XsCDzEJmEY2HOA5Kjax9RZKx8+QUoMlWzQQNZwt30MpN667YBKwzNag0kqtbdMEMMU+Wy1 yaEKx186oBWjRVhb5nbIHVRuQPlMBY187sAGRjPa1IymJYryiEj0EJsQNgWFRrUnneB/nObIi9/r SyrP0VE86eIxfyfz+Zt58ow+92tS/aFYgcR/3384tA8DWQFazo+DfpL9YTWjfyplttJ+cvoAhqY1 ADjU33NYM+potT4O4mElno3cSqjiuFQu99M/4fEuvncQ6TdFV7Vy6tjxc/A6sBk7k+csbA7uwsP2 9BnrwsTXN4vk9fvInLb8/A3Q9JcTsCk5l0ZW4ft8ebqcRLW15d9KXAbbYv7sLIm9TpcVPI3D1IZg m/txzcWD1Qp9gN3lZJEsxbawFkS5G5ZdiMNkOC6ztcem/cUqJjGK3GYfHG6X/gK3NW90yBDl6Xk0 QvZD4vFyT22+ix8w6eQHyOp/AAAA//8DAFBLAwQUAAYACAAAACEAlrWt4pYGAABQGwAAFQAAAHdv cmQvdGhlbWUvdGhlbWUxLnhtbOxZT2/bNhS/D9h3IHRvYyd2Ggd1itixmy1NG8Ruhx5piZbYUKJA 0kl9G9rjgAHDumGHFdhth2FbgRbYpfs02TpsHdCvsEdSksVYXpI22IqtPiQS+eP7/x4fqavX7scM HRIhKU/aXv1yzUMk8XlAk7Dt3R72L615SCqcBJjxhLS9KZHetY3337uK11VEYoJgfSLXcduLlErX l5akD8NYXuYpSWBuzEWMFbyKcCkQ+AjoxmxpuVZbXYoxTTyU4BjI3hqPqU/QUJP0NnLiPQaviZJ6 wGdioEkTZ4XBBgd1jZBT2WUCHWLW9oBPwI+G5L7yEMNSwUTbq5mft7RxdQmvZ4uYWrC2tK5vftm6 bEFwsGx4inBUMK33G60rWwV9A2BqHtfr9bq9ekHPALDvg6ZWljLNRn+t3slplkD2cZ52t9asNVx8 if7KnMytTqfTbGWyWKIGZB8bc/i12mpjc9nBG5DFN+fwjc5mt7vq4A3I4lfn8P0rrdWGizegiNHk YA6tHdrvZ9QLyJiz7Ur4GsDXahl8hoJoKKJLsxjzRC2KtRjf46IPAA1kWNEEqWlKxtiHKO7ieCQo 1gzwOsGlGTvky7khzQtJX9BUtb0PUwwZMaP36vn3r54/RccPnh0/+On44cPjBz9aQs6qbZyE5VUv v/3sz8cfoz+efvPy0RfVeFnG//rDJ7/8/Hk1ENJnJs6LL5/89uzJi68+/f27RxXwTYFHZfiQxkSi m+QI7fMYFDNWcSUnI3G+FcMI0/KKzSSUOMGaSwX9nooc9M0pZpl3HDk6xLXgHQHlowp4fXLPEXgQ iYmiFZx3otgB7nLOOlxUWmFH8yqZeThJwmrmYlLG7WN8WMW7ixPHv71JCnUzD0tH8W5EHDH3GE4U DklCFNJz/ICQCu3uUurYdZf6gks+VuguRR1MK00ypCMnmmaLtmkMfplW6Qz+dmyzewd1OKvSeosc ukjICswqhB8S5pjxOp4oHFeRHOKYlQ1+A6uoSsjBVPhlXE8q8HRIGEe9gEhZteaWAH1LTt/BULEq 3b7LprGLFIoeVNG8gTkvI7f4QTfCcVqFHdAkKmM/kAcQohjtcVUF3+Vuhuh38ANOFrr7DiWOu0+v Brdp6Ig0CxA9MxHal1CqnQoc0+TvyjGjUI9tDFxcOYYC+OLrxxWR9bYW4k3Yk6oyYftE+V2EO1l0 u1wE9O2vuVt4kuwRCPP5jeddyX1Xcr3/fMldlM9nLbSz2gplV/cNtik2LXK8sEMeU8YGasrIDWma ZAn7RNCHQb3OnA5JcWJKI3jM6rqDCwU2a5Dg6iOqokGEU2iw654mEsqMdChRyiUc7MxwJW2NhyZd 2WNhUx8YbD2QWO3ywA6v6OH8XFCQMbtNaA6fOaMVTeCszFauZERB7ddhVtdCnZlb3YhmSp3DrVAZ fDivGgwW1oQGBEHbAlZehfO5Zg0HE8xIoO1u997cLcYLF+kiGeGAZD7Ses/7qG6clMeKuQmA2Knw kT7knWK1EreWJvsG3M7ipDK7xgJ2uffexEt5BM+8pPP2RDqypJycLEFHba/VXG56yMdp2xvDmRYe 4xS8LnXPh1kIF0O+EjbsT01mk+Uzb7ZyxdwkqMM1hbX7nMJOHUiFVFtYRjY0zFQWAizRnKz8y00w 60UpYCP9NaRYWYNg+NekADu6riXjMfFV2dmlEW07+5qVUj5RRAyi4AiN2ETsY3C/DlXQJ6ASriZM RdAvcI+mrW2m3OKcJV359srg7DhmaYSzcqtTNM9kCzd5XMhg3krigW6Vshvlzq+KSfkLUqUcxv8z VfR+AjcFK4H2gA/XuAIjna9tjwsVcahCaUT9voDGwdQOiBa4i4VpCCq4TDb/BTnU/23OWRomreHA p/ZpiASF/UhFgpA9KEsm+k4hVs/2LkuSZYRMRJXElakVe0QOCRvqGriq93YPRRDqpppkZcDgTsaf +55l0CjUTU4535waUuy9Ngf+6c7HJjMo5dZh09Dk9i9ErNhV7XqzPN97y4roiVmb1cizApiVtoJW lvavKcI5t1pbseY0Xm7mwoEX5zWGwaIhSuG+B+k/sP9R4TP7ZUJvqEO+D7UVwYcGTQzCBqL6km08 kC6QdnAEjZMdtMGkSVnTZq2Ttlq+WV9wp1vwPWFsLdlZ/H1OYxfNmcvOycWLNHZmYcfWdmyhqcGz J1MUhsb5QcY4xnzSKn914qN74OgtuN+fMCVNMME3JYGh9RyYPIDktxzN0o2/AAAA//8DAFBLAwQU AAYACAAAACEAoQmDWCUDAAAfBwAAEQAAAHdvcmQvc2V0dGluZ3MueG1snFXbbts4EH1fYP9B0PM6 lmQ7ToQ4RWNb3RZJW6zSD6CkkUyEN5CUFffrO5TEKul6g2KfTJ0zczicm2/ePXMWHEEbKsUmjC+i MABRyoqKZhN+e8xmV2FgLBEVYVLAJjyBCd/d/vnHTZcasBbNTIASwqRyE7ZapKY8ACdmxmmppZG1 nZWSp7KuaQnjTzh66E14sFal8/nodCEVCFSrpebEmgupm/nguZNly0HYeRJFl3MNjFgM2ByoMl6N /181vOrgRY5vPeLImbfr4ugty/G5ndTVT4/fCc85KC1LMAYzy9nwXE6o8DKG/Y7OkM97WmiiTy9E brFs36XkQZcq0CUmFGseReHcEZX8LO2OGsXI6Stp4E62WHZNwfQ0xiXr3BIL6G0UMNb3SMmAYHRd 2mjCOcGaDsggCTVpmX0kRW6lQqMjwfjXyXhjeSCalBZ0rkiJalsprJbM2/UBbSVXGvMxxIi9pIh1 0bYGsv09OcnW4lXzLp0obObKOBt3+EdK6wWj6G6bXO0Wg5ZjJyaJ4+t4jOs1s1isstXunM/ybn11 vT/HrPbraJWdYy6TeL8+q/Y+W27jsz7/HfV2FWX7s2rZ1fI6XrsIMDdjRnjqev2rvr0ZThnmO+BD UbaEF5qS4MFNA3rxtNBPd1R4vgCcSnjJ5G3hydlsIAwnjGVYU0/gIAxMhX21g7oXZg9EN5Nyn3Se 6rNoBfWnn2quYUF/0LJVg2qnifooKoT9hfFyOepRYe8p97hpi9x7CRyKFxR2+ZejdoLzKUFdanGP gcvQPRGN7xMQs2+5M8V+Yzp3uw4eiFLYvGhSNPEmZLQ52NhNhMUvHKCn/qNokpFLeg6/HNd/kNK9 DK3HgzMYjmg1HiZs4bHFhC09tpywlcdWE3bpsUuHHU64BXCMn3Cn+KPDa8mY7KD624Ob8F/QkARz IAqwrm7KceRk2gNYtB4Ijik844qBilr8G1G04uR5E67iJHHuozWuGxziV7aOc8bqFRpUxBJcWH2p Xjn3Tf5LLLjQoKTYkPmJF9NSuRgCZ9TYHBTuHys1PrlfTH/1ytM/2+0PAAAA//8DAFBLAwQUAAYA CAAAACEAb2NQm4sBAAAHBAAAEgAAAHdvcmQvZm9udFRhYmxlLnhtbKSS326CMBTG75fsHUjvJwX/ TI1onNPLXSzuAY5YpAltSU+V+fY7UORiZolkkDThO+3H6e98i9W3KoKLsCiNTlg04CwQOjVHqU8J +9rvXqYsQAf6CIXRImFXgWy1fH5aVPPMaIcBndc4twnLnSvnYYhpLhTgwJRCUy0zVoGjT3sKTZbJ VLyb9KyEdmHM+SS0ogBH/8Zclshat+oRt8rYY2lNKhCpWVV4PwVSs2XbXVDNNSjqegOFPFjZFErQ BkVEtQsUCeMx3/ExrfU74sN6ZWHtkOZgUbhuI/dyBkoW15uKlUT0hVK6NL/pF7ASDoXwJZQnKpzx wBO2jTjn8W7HvBIlbETCetMpMTXln1m7Z9gpNB5qrPFptkSzxocU8mlPNX2Gfj53JPZSCQw+RBV8 GgUe1T2RmE+IxJh41GSGvYjYxrch+CgRajxed/enm2xIeZ2Oovb+vYh4n8eJbEBRNOCPbNQEPIma SL9s9CexrkcYb39lg/PR2x2JJgmUqP9kow0JLn8AAAD//wMAUEsDBBQABgAIAAAAIQAoh3GlzwAA AB8BAAAUAAAAd29yZC93ZWJTZXR0aW5ncy54bWyMj8tOAzEMRfdI/MMoe5qBRYVGnamEUNlQqMRj n2Y8nUiJHdmB0H495rFhx/LaV8fHq/VHis07sATC3lwuWtMAehoDHnrz8ry5uDaNFIeji4TQmyOI WQ/nZ6vaVdg/QSnalEYpKB33Zi4ld9aKnyE5WVAG1N1EnFzRyAdL0xQ83JJ/S4DFXrXt0jJEV9RA 5pDF/NLqf2iVeMxMHkRUJMUfXnIBzaCOlEtI4QQb4humKsD2a6z3jo/4ur3/Ti5GqruHOw32z1vD JwAAAP//AwBQSwMEFAAGAAgAAAAhAPeMfebgAQAA2wMAABAACAFkb2NQcm9wcy9hcHAueG1sIKIE ASigAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnFPBbtswDL0P2D8Yvjd2nCUoAkXFkGLo YVsDxG3PnEzHwmRJkNSg2dePshtX2XqqT+Qj/fT0SLGbl15lR3ReGr3J57Myz1AL00h92OQP9ber 6zzzAXQDymjc5Cf0+Q3//IntnLHogkSfEYX2m7wLwa6LwosOe/AzKmuqtMb1ECh1h8K0rRR4a8Rz jzoUVVmuCnwJqBtsruxEmI+M62P4KGljRNTnH+uTJcGc1dhbBQH5zyhHsWICWG0CqFr2yBcl4VPG dnBAz+esGAP2ZFzj+bIkZAzZtgMHIpB7vLpe0d8JwL5aq6SAQMbyH1I4400bsvvBgiwSsCJtYWTL HsWzk+HEiSpN2XepSUq1YMUYkTYHBwe283wVBU4Z2wtQuKXL8xaUR1a8AewOIQ52B5IUs2NYH1EE 4zIv/9Boqzz7BR6jZZv8CE6CDmRdbBuTIVbWB8drGRRxU23MhzBtS2P5JbpIvRRcNkZw1ECFS3XD Cf6+pbuFd8TOU7GDhlFqIicJpzP+Yd2a3oI+0eFTRAb/9g+2NrdxXV49vASTuT/J0O0tCJrOYrFc phuQlNieFgUbGumZ8A1gd+S3U/FU2h59wObc838h7tTj+Fb5vJqV9A1LdMZoE6ZHxP8CAAD//wMA UEsDBBQABgAIAAAAIQBTfUAmVgEAAHkCAAARAAgBZG9jUHJvcHMvY29yZS54bWwgogQBKKAAAQAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACMkstuwjAQRfeV+g+R94mTAC2ykiD1wQraSk0f6s6y B4gaP2S7BP6+TgIpqF10Ob7XZ+6Mnc12og62YGylZI6SKEYBSKZ4Jdc5einn4RQF1lHJaa0k5GgP Fs2Ky4uMacKUgSejNBhXgQ08SVrCdI42zmmCsWUbENRG3iG9uFJGUOdLs8aask+6BpzG8RUW4Cin juIWGOqBiA5Izgak/jJ1B+AMQw0CpLM4iRL843VghP3zQqecOEXl9trPdIh7yuasFwf3zlaDsWma qBl1MXz+BL8vF8/dqGEl210xQEXGGWEGqFOmqEGqrcrwyVG7vppat/SbXlXAb/bF4v7h8fUxw7+V 1mxgW7VvVEw6x1D6Pt1YfTPggQ9K+rGOytvo9q6coyKNk0mYpGEyLuNrMhmROP5oQ53db4P3B+IQ 7R/EdFqmMUnG58QjoOgSn3+W4hsAAP//AwBQSwMEFAAGAAgAAAAhAOYmwWoaBwAA+zkAAA8AAAB3 b3JkL3N0eWxlcy54bWy0m1FT2zgQx99v5r6Dx+80ISnJlWnaoRSuzLQcbWDuWbEVoqlj5SylQD/9 SStbGBvbu9g8EcvW/lba1X+dIL3/eL9Ngl88U0Kmi/DwzTgMeBrJWKS3i/Dm+vzgrzBQmqUxS2TK F+EDV+HHD3/+8f7uWOmHhKvAGEjVcbYIN1rvjkcjFW34lqk3csdTc28tsy3T5jK7Hcn1WkT8s4z2 W57q0WQ8no0ynjBt4GojdirMrd1hrN3JLN5lMuJKGW+3ibO3ZSINPxj3Yhl95mu2T7Syl9lVll/m V/DnXKZaBXfHTEVCXBvHzRC3IpXZl5NUidDc4UzpEyVY+eZZ3mbvb+yD5Zu+Z6R0yeAnEYtwZKHq t+n2iyWLcDIpWk6tE0/aEpbeFm08Pbi4LDuzCH3TythdhCw7WJ5YYyMYafG3NOLdk/GbK3BlxyIz d4bD1pqbGJqQWE4ibKwn81lx8WOfmAa21zKHgAEDK5s1l5VJN6E1gV66RDF3+fqrjH7yeKnNjUUI LNN4c3GVCZkJ/bAI372zTNO45FvxRcQxt3mZt92kGxHzfzc8vVE8fmz/fg5ZlluM5D7Vxv3ZHBIh UfHZfcR3NsuM6ZTZIF/aDok1q0occGgvHr1xDRUqNP5XIA9dDJ+lbDizKykA/1tBMOp9b9DEjqg8 ALBL8nXa38Tb/iaO+puA5O03F/P+Xhj97BsRlxulrMQHVcvIJV95HqbvWlLW9qhlUWePWtJ09qjl SGePWkp09qhlQGePWsA7e9Ti29mjFs7WHhED4apm0RRmA7Wwr4VOuO3fKkCHPaUuLzXBFcvYbcZ2 m8DW1qrbbWK53K80zlWQ05eL5VJnMr3tnBFTne3SfbEmn213G6aEeanpmPpJz6m/ZquEB39nIu5E Hbnkq40JXkyeLWFXCYv4RiYxz4Jrfu8iSuh/KYOle8vodK5nWL+K240OlhsouZ2wWcOkN8+Es/9V KJiD1sU0axhKl3FUDGcNedls/BuPxX5bTA3ibWTm9JwQ5goCXGyforc2RPXV1TkKGwDMEFy5oA8B 7CP8d8WFbt/GGOO/K0UvtI/w3xWuF9qH/GiPL1lpPrPsZ4BaXnPy2j2ViczW+6RYA53yMCevYI/A DYG8iL19lEjMySv4iXwGJ1Fkvrlh8pQci0cdJVDI4XAUWGz4sZCDUpG9Q8KIyAGqsCYEVj+tJYDI ovuD/xL2tydqMQCV9u+anct52jADpgSh3qG/76XufoeeNGgelnKRmp9LFA9wtGnDysPS8nxy9Y4Q 436FjwDqVwEJoH6lkABqyI/mdx5fE/GQ/sWRwCLLsq9ikHZoZZ6TldmDaCVgoLqJeP9qWL3NuVCv mwgKOUD1uomgkKNTqWW+biJYg9VNBKuhajTHqKyplEGR62YZ5N8EECMaRrwRoGHEGwEaRrwRoP7i 3Q0ZTrwRLLI2eE0tizcCBI9Qvup7UFm8ESCyNji1y38zKuoeWGn/cjuAeCMo5ADVxRtBIUenSbwR LHiEkgkVlpc6BGsY8UaAhhFvBGgY8UaAhhFvBGgY8UaA+ot3N2Q48UawyNrgNbUs3ggQWR48qCze CBA8QtGGZ8UbVv2rizeCQg5QXbwRFHJ0KoLqX1IRLHKAKiwv3ggWPEJJhpwFyU0Z1DDijRjRMOKN AA0j3gjQMOKNAPUX727IcOKNYJG1wWtqWbwRILI8eFBZvBEgsjY8K96wGF9dvBEUcoDq4o2gkKNT EVSvcwgWOUAVlhdvBAvypbd4I0DwyEtBlBENI96IEQ0j3gjQMOKNAPUX727IcOKNYJG1wWtqWbwR ILI8eFBZvBEgsjY8K96wRl5dvBEUcoDq4o2gkKNTEVQv3ggWOUAVlpc6BGsY8UaAIDF7izcCBI+8 AASriBKmYcQbMaJhxBsB6i/e3ZDhxBvBImuD19SyeCNAZHnwoLJ4I0BkbbD7bM1+UfT21MOGJMDu Myh2NaCBk4YgYYH5AH/wNc/MYSbevTukJ7AYIYHYkB7YIX6S8meA29g9bUgQNEqsEiFhS/cD7NIp HUSYzltOElz/cxp8cQdgav0gpZ7uvDGnh8rHheB4kj04ZPzUDztzZGdX7Cy31swBIXu0Kz8CBEfR LsyBoPxYj+1sz/mYB+FQVd4M/7fNqfDZHHuLi2fG46Oz+fjoPD/gBCbrTkQb40Vkzkq1OJFvhfe7 k2AjfNWlhv3y4NbjYY3CuXzf/OPblXvuye5N02TmsMFvbfeIt/gMe8hbZy+AR1y86w6aY1vgUpeH fr8VPK1XiTuIZj5cpDYU5uQf/G/NhTy+Z86suX/Kk+Qbg2NrWu6aH034Wru7h2OokxVTK6m13Db3 z2AbOXjynAEzxWVn3KUdRPPcp/vtimfmHFjL/F9KW1/gvNrTxHU7Yl24/coz3kNeY2f90bfik/rw PwAAAP//AwBQSwECLQAUAAYACAAAACEA3fyVN2YBAAAgBQAAEwAAAAAAAAAAAAAAAAAAAAAAW0Nv bnRlbnRfVHlwZXNdLnhtbFBLAQItABQABgAIAAAAIQAekRq38wAAAE4CAAALAAAAAAAAAAAAAAAA AJ8DAABfcmVscy8ucmVsc1BLAQItABQABgAIAAAAIQDWZLNR+gAAADEDAAAcAAAAAAAAAAAAAAAA AMMGAAB3b3JkL19yZWxzL2RvY3VtZW50LnhtbC5yZWxzUEsBAi0AFAAGAAgAAAAhAKsR9BCDCgAA yx0AABEAAAAAAAAAAAAAAAAA/wgAAHdvcmQvZG9jdW1lbnQueG1sUEsBAi0AFAAGAAgAAAAhAJa1 reKWBgAAUBsAABUAAAAAAAAAAAAAAAAAsRMAAHdvcmQvdGhlbWUvdGhlbWUxLnhtbFBLAQItABQA BgAIAAAAIQChCYNYJQMAAB8HAAARAAAAAAAAAAAAAAAAAHoaAAB3b3JkL3NldHRpbmdzLnhtbFBL AQItABQABgAIAAAAIQBvY1CbiwEAAAcEAAASAAAAAAAAAAAAAAAAAM4dAAB3b3JkL2ZvbnRUYWJs ZS54bWxQSwECLQAUAAYACAAAACEAKIdxpc8AAAAfAQAAFAAAAAAAAAAAAAAAAACJHwAAd29yZC93 ZWJTZXR0aW5ncy54bWxQSwECLQAUAAYACAAAACEA94x95uABAADbAwAAEAAAAAAAAAAAAAAAAACK IAAAZG9jUHJvcHMvYXBwLnhtbFBLAQItABQABgAIAAAAIQBTfUAmVgEAAHkCAAARAAAAAAAAAAAA AAAAAKAjAABkb2NQcm9wcy9jb3JlLnhtbFBLAQItABQABgAIAAAAIQDmJsFqGgcAAPs5AAAPAAAA AAAAAAAAAAAAAC0mAAB3b3JkL3N0eWxlcy54bWxQSwUGAAAAAAsACwDBAgAAdC0AAAAA --===============0921340515==-- From MAILER-DAEMON Mon Jan 4 14:59:22 2016 Return-Path: <> X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: * X-Spam-Status: No, score=1.9 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLYTO, TO_MALFORMED,T_FREEMAIL_DOC_PDF autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6ED7F7F37 for ; Mon, 4 Jan 2016 14:59:22 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 241338F8033 for ; Mon, 4 Jan 2016 12:59:19 -0800 (PST) X-ASG-Debug-ID: 1451941155-04bdf0264e454270001-NocioJ Received: from mail.gaf.ni.ac.rs (mail.gaf.ni.ac.rs [160.99.31.9]) by cuda.sgi.com with ESMTP id vXOw5gynBeZoB4qv (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 12:59:16 -0800 (PST) X-Barracuda-Envelope-From: X-Barracuda-Apparent-Source-IP: 160.99.31.9 Received: from localhost (localhost [127.0.0.1]) by mail.gaf.ni.ac.rs (Postfix) with ESMTP id 837CB810E48C4; Mon, 4 Jan 2016 21:59:14 +0100 (CET) Received: from mail.gaf.ni.ac.rs ([127.0.0.1]) by localhost (mail.gaf.ni.ac.rs [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id M8fTBcaomELQ; Mon, 4 Jan 2016 21:59:14 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mail.gaf.ni.ac.rs (Postfix) with ESMTP id 50726810E48C0; Mon, 4 Jan 2016 21:59:14 +0100 (CET) X-Virus-Scanned: amavisd-new at gaf.ni.ac.rs Received: from mail.gaf.ni.ac.rs ([127.0.0.1]) by localhost (mail.gaf.ni.ac.rs [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 4fDr0toLp2E1; Mon, 4 Jan 2016 21:59:14 +0100 (CET) Received: from [100.67.150.48] (unknown [116.202.38.88]) by mail.gaf.ni.ac.rs (Postfix) with ESMTPSA id DF9AF810E48C9; Mon, 4 Jan 2016 21:59:05 +0100 (CET) Content-Type: multipart/mixed; boundary="===============0781748591==" MIME-Version: 1.0 Subject: Re To: Recipients <> X-ASG-Orig-Subj: Re From: "<"@LIVE.COM Date: Tue, 05 Jan 2016 02:29:04 +0530 Reply-To: cruzcelia2011@gmail.com Message-Id: <20160104205905.DF9AF810E48C9@mail.gaf.ni.ac.rs> X-Barracuda-Connect: mail.gaf.ni.ac.rs[160.99.31.9] X-Barracuda-Start-Time: 1451941156 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.00 X-Barracuda-Spam-Status: No, SCORE=2.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0646, BSF_SC0_MV0646_4, EMPTY_ENV_FROM, NO_REAL_NAME, TO_MALFORMED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25830 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 TO_MALFORMED To: has a malformed address 0.00 EMPTY_ENV_FROM Empty Envelope From Address 0.00 NO_REAL_NAME From: does not include a real name 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.50 BSF_SC0_MV0646 Custom rule MV0646 1.50 BSF_SC0_MV0646_4 BSF_SC0_MV0646_4 You will not see this in a MIME-aware mail reader. --===============0781748591== Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body please kindly open the fill --===============0781748591== Content-Type: application/octet-stream MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="BUSINESS PROPOSAL.docx" UEsDBBQABgAIAAAAIQDd/JU3ZgEAACAFAAATAAgCW0NvbnRlbnRfVHlwZXNdLnhtbCCiBAIooAAC AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC0 VMtuwjAQvFfqP0S+Vomhh6qqCBz6OLZIpR9g7A1Y9Uv28vr7bgJEVQtBKuUSKVnvzOzsxIPR2pps CTFp70rWL3osAye90m5Wso/JS37PsoTCKWG8g5JtILHR8PpqMNkESBl1u1SyOWJ44DzJOViRCh/A UaXy0Qqk1zjjQchPMQN+2+vdcekdgsMcaww2HDxBJRYGs+c1fd4qiWASyx63B2uukokQjJYCSSlf OvWDJd8xFNTZnElzHdINyWD8IENdOU6w63sja6JWkI1FxFdhSQZf+ai48nJhaYaiG+aATl9VWkLb X6OF6CWkRJ5bU7QVK7Tb6z+qI+HGQPp/FVvcLnrSOY4+JE57OZsf6s0rUDlZESCihnZ1x0cHRLLs EsPvkLvGb1KAlHfgzbN/tgcNzEnKin6JiZgaOJvvV/Ja6JMiVjB9v5j738C7hLT5kz7+wYz9dVF3 H0gdb+634RcAAAD//wMAUEsDBBQABgAIAAAAIQAekRq38wAAAE4CAAALAAgCX3JlbHMvLnJlbHMg ogQCKKAAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAjJLbSgNBDIbvBd9hyH032woi0tneSKF3IusDhJnsAXcOzKTavr2jILpQ217m9OfLT9ab g5vUO6c8Bq9hWdWg2JtgR99reG23iwdQWchbmoJnDUfOsGlub9YvPJGUoTyMMaui4rOGQSQ+ImYz sKNchci+VLqQHEkJU4+RzBv1jKu6vsf0VwOamabaWQ1pZ+9AtcdYNl/WDl03Gn4KZu/Yy4kVyAdh b9kuYipsScZyjWop9SwabDDPJZ2RYqwKNuBpotX1RP9fi46FLAmhCYnP83x1nANaXg902aJ5x687 HyFZLBZ9e/tDg7MvaD4BAAD//wMAUEsDBBQABgAIAAAAIQDWZLNR+gAAADEDAAAcAAgBd29yZC9f cmVscy9kb2N1bWVudC54bWwucmVscyCiBAEooAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AKySzWrDMBCE74W+g9h7LTv9oYTIuZRArq37AIq9/qGyJLSbtn77CkNShwb34otgRmjmk7Sb7Xdv xCcG6pxVkCUpCLSlqzrbKHgvdnfPIIi1rbRxFhUMSLDNb282r2g0x0PUdp5ETLGkoGX2aympbLHX lDiPNu7ULvSaowyN9Lr80A3KVZo+yTDNgPwiU+wrBWFf3YMoBh+b/892dd2V+OLKY4+Wr1TILzy8 IXO8HMVYHRpkBRMzibQgr4OslgShPxQnZw4hWxSBBxM/8/wMNOq5+scl6zmOCP62j1KOazbH8LAk Q+0sF/pgJhxn6wQhLwY9/wEAAP//AwBQSwMEFAAGAAgAAAAhAKsR9BCDCgAAyx0AABEAAAB3b3Jk L2RvY3VtZW50LnhtbNRYzW7bRhC+F+g7DIQcWkCWKAcuEjVW4FpO40NTI7ZzX5FLcWFyl91dSuGt r9FbniWP0ifpN0tSliXFFgSjRU+0l9z5n++b0Zu3n4ucFtI6ZfRpbzSIeiR1bBKl56e925t3R696 5LzQiciNlqe9Wrre28n3371ZjhMTV4XUniBCu/ECbzPvy/Fw6OJMFsINTCk1XqbGFsLjXzsfFsLe VeVRbIpSeDVTufL18DiKfuq1Ysxpr7J63Io4KlRsjTOp5ytjk6Yqlu2ju2H30dvcnLYmB41DK3PY YLTLVOk6acWh0uBi1glZPObEosi775blPtoSK5bIR5E3Zi+NTUprYukcTqfNy5XEUfSY7jaALGJ1 Yx8THursLCmE0isxXB0b+V8lb4DkDRvdQxZ17whiMUEtzUxS87Ok5Ri1mHw87UXR+Un07mLa646m MhVV7vnNL+fHr6Yvw83yyvJF2zxmwyDt3OE5bE/x7D564svl2E9+tVJ6xNUN6CqXwkkqpS2Up0KS N6S0tyapYklF7WSe8lltKjKaj4yVA7okUZDA94mKRU4xCgxmCzTKrCYkLuXEGU1oKkIw7thQ35gb 7Ov8ZUePR6PXoyg4+oiHuM/dM3aliJGF0kon7UL2Jo6EY5VxZWVo1UJoMZeWlspn9Jjily9P3p1M 91A8uS6FVi6jq0ygzWNZeXZ7U/bTkd/tAZ0zUuiaw7qEM0rTpU6U4D+Oo9GIAC4INnssbJxRWdnS IGlJZZFEWmYKh5eUKBcbwJxMyGfC0w+b9h0U9MkNlAj6JK34MVgnFkLlYpbLlZ2D51A0s205H1wf QQKX2c2mPYcmJni+KeygKO5OPAd1q0KfNnZyLZFj5ahyeHJxhD5EmvKuNlx/w2w0iEkvLLpi7OsS DTS3orj2wvpegJOnAIbj+vULUr6P3Aud7C11d2TI+IxbGDiKfvDsrNDQL2KvFpLwbsbecwmi3asU 500zmJTOtFf0/vIT3eAzUdZ9eg+PrZc6QNIV2FlpB3C6mJlc0FShrZx0dOHjAe5AlWvj60oZK7Cx yPMaca4AZlAIyyhTfCSsFASN16YC1FwI5+nMKdEPsCfiGNZzgwI+0TDGCm9sTV7is1UvszAMHZRa U7RdnwECZlJqBrUFHrg+Ay7jwxmuMkhkAjFYKKc88t9ABdzxGWCdTBwL9tPxPSBFDHF8u3OLEZll JZhe8CcrA3qDDSrGDUMW4SA4jFBjKEFUKosRJ3h0q4NCVA18YL/PCmkRnYBbCEgwWCWBQ5gvWA3L RnKgzgM2+BKf5oZZI5EiR5JF6jnVEkERADIxhxsByGENcwesmkvcsls482SrPASV/T7nSr9EcVUh 6zsimG60APfUis/3JrMJ2vYjCPhl1KcoioZHzMEEcruTnsM02tTypPXf4MdoXjTVtRn4Ps0qJKYK +VLekRUWk2pINX+bqXnWlkkfRYdJIN8Cq8Ncv69GpPi3uuO/AGUvthxf17E/ZffpJAqhfa440u01 TU2eC+s2RR5m4dcvaMBNUYdmme6kLENv8fQFxAy0EMa3tbECYGDnwIj0HhkYBFxVloAzPg6DH2qh AiTk7TyCw8Iw1CBbgrRcoipKMEdoUgDiVMaymKF6c0bAGkMK46hcpZUxRpslibI0mC0hqJECudtD W4amZywJ8HY/By2lvHN///kXedUMqduN+e/BAzunnEPnWPSID84BX8PUhTnM4MBjVsI0zH3EAdvl KqIJPMckxZNloqwETGKHKCoNTOWdrZlgWcQ6XqJtZSyY/B6qylTBCUoVEgo+CqqZFnkoAEqHrLMs JrcmP82gHLKI1Bf3fQhfPvx+Q2IJgusge1vUOqOEfQBgbRWXCU+fnefNRgSM5KTKz2FwlZhSa4AA uFGlIHOEDITywHJwQWt3y8nQxlEFj8I3pojk524wCEfe1i3XItaeYwmHuNrmDGor1rxDITJVsjjk pdsVuFtSniy4ennWAK/yivNHBcpjElsoLD+ta6CqWGi2JmY8CMweRpEgdlZhZcX687AceUHj3v6v l79uSN49dgGL0aQcelQSUzjCxKHBn+BprHT8MwlKCW/X+q+pqd0k07GJBrsfBQoBNuE6C2zXnm82 yDLD9IPqECirBQoEUBOGnaZ613uiqa4gvivxzoNNgF3H6v3JmoPQsvUzkfUEvNwwMFu8Cx/QMc9j fNgKmjmQiYEnTc4tfMJE/WLUP9lU8zQJTcCt0ea1w0Ib0e1gxawwqFlnUXhNq7/eUrOHdX0kiS4/ fAyVAgwL4NHCoJZz41UDsMjqDIwolGbsAAhiCydsB0k/4Nc3ssK/d+Q5Q0ybOvxWshZXBo9S1CHG mMWz/yMQnIWZ/B8AAAD//6xWXW7bOBC+ysBA31xXTtLYzTYG6iQt+rCA0e4FKGkkEZZIlaSiuk+9 Rt/2LD3KnmS/oawkNookBQokkvgzw5lvvm/oUDG1zmad44ZNoEYZVbKjSnlqO5fhzTnJLs/4KJxt aGe7qTyo13VNOeddFuKWe4PM+kDK5LLN0dtX/UVYydOt3uJB+Pc6/3Q5SZLzk/nN4noS5zdxOX0l g/TK4y0mw2xYBaduua61KY/9PW1MX5v6wrcq48tJ69izu+XJivhryybXAdlPY7hI1AypOFvoMGSY IvlKOaSfcugZOzpPKtBp8oJe0iJ5MaW+0llFDSvj75EJasuyLA7Nz3/p4+AuTsN2Fqccq7reUW7J WJynUINgqbaeyXaBrISjPaWd14Y9PjhTHRYRHKaVIdu21oXO6LCjwjoEIQ4aObuxhjEpNZMyZLZp ldnN6J1HLF7pnFi5WrObxpxzRvVzssVwJGgRK4tj5kmSENDbcvB/jUenEl9le+HMblyN2WcIa+TC cLwQqLaZErqoejzQdE0KruHAve9ofWe5p1dg1yD3GKJqbAeExGJfIB5ALTnMDmgB5rTCi/Yh2c7W i+Wbm8k4dc2F6uogNFxfnSyvTwcaPkmn1EWKhl+z6iOpBlibQueiqFCBKr0O1aCFCjXxKJUgjZJm 7KRwklDnWql6v88oVrC0FqLTRplMA7xSaUP4O6DEbCRW2eHECCFYU3QOoLsobigXqgFbPHmLNbxF vjFGF/XsCDzEJmEY2HOA5Kjax9RZKx8+QUoMlWzQQNZwt30MpN667YBKwzNag0kqtbdMEMMU+Wy1 yaEKx186oBWjRVhb5nbIHVRuQPlMBY187sAGRjPa1IymJYryiEj0EJsQNgWFRrUnneB/nObIi9/r SyrP0VE86eIxfyfz+Zt58ow+92tS/aFYgcR/3384tA8DWQFazo+DfpL9YTWjfyplttJ+cvoAhqY1 ADjU33NYM+potT4O4mElno3cSqjiuFQu99M/4fEuvncQ6TdFV7Vy6tjxc/A6sBk7k+csbA7uwsP2 9BnrwsTXN4vk9fvInLb8/A3Q9JcTsCk5l0ZW4ft8ebqcRLW15d9KXAbbYv7sLIm9TpcVPI3D1IZg m/txzcWD1Qp9gN3lZJEsxbawFkS5G5ZdiMNkOC6ztcem/cUqJjGK3GYfHG6X/gK3NW90yBDl6Xk0 QvZD4vFyT22+ix8w6eQHyOp/AAAA//8DAFBLAwQUAAYACAAAACEAlrWt4pYGAABQGwAAFQAAAHdv cmQvdGhlbWUvdGhlbWUxLnhtbOxZT2/bNhS/D9h3IHRvYyd2Ggd1itixmy1NG8Ruhx5piZbYUKJA 0kl9G9rjgAHDumGHFdhth2FbgRbYpfs02TpsHdCvsEdSksVYXpI22IqtPiQS+eP7/x4fqavX7scM HRIhKU/aXv1yzUMk8XlAk7Dt3R72L615SCqcBJjxhLS9KZHetY3337uK11VEYoJgfSLXcduLlErX l5akD8NYXuYpSWBuzEWMFbyKcCkQ+AjoxmxpuVZbXYoxTTyU4BjI3hqPqU/QUJP0NnLiPQaviZJ6 wGdioEkTZ4XBBgd1jZBT2WUCHWLW9oBPwI+G5L7yEMNSwUTbq5mft7RxdQmvZ4uYWrC2tK5vftm6 bEFwsGx4inBUMK33G60rWwV9A2BqHtfr9bq9ekHPALDvg6ZWljLNRn+t3slplkD2cZ52t9asNVx8 if7KnMytTqfTbGWyWKIGZB8bc/i12mpjc9nBG5DFN+fwjc5mt7vq4A3I4lfn8P0rrdWGizegiNHk YA6tHdrvZ9QLyJiz7Ur4GsDXahl8hoJoKKJLsxjzRC2KtRjf46IPAA1kWNEEqWlKxtiHKO7ieCQo 1gzwOsGlGTvky7khzQtJX9BUtb0PUwwZMaP36vn3r54/RccPnh0/+On44cPjBz9aQs6qbZyE5VUv v/3sz8cfoz+efvPy0RfVeFnG//rDJ7/8/Hk1ENJnJs6LL5/89uzJi68+/f27RxXwTYFHZfiQxkSi m+QI7fMYFDNWcSUnI3G+FcMI0/KKzSSUOMGaSwX9nooc9M0pZpl3HDk6xLXgHQHlowp4fXLPEXgQ iYmiFZx3otgB7nLOOlxUWmFH8yqZeThJwmrmYlLG7WN8WMW7ixPHv71JCnUzD0tH8W5EHDH3GE4U DklCFNJz/ICQCu3uUurYdZf6gks+VuguRR1MK00ypCMnmmaLtmkMfplW6Qz+dmyzewd1OKvSeosc ukjICswqhB8S5pjxOp4oHFeRHOKYlQ1+A6uoSsjBVPhlXE8q8HRIGEe9gEhZteaWAH1LTt/BULEq 3b7LprGLFIoeVNG8gTkvI7f4QTfCcVqFHdAkKmM/kAcQohjtcVUF3+Vuhuh38ANOFrr7DiWOu0+v Brdp6Ig0CxA9MxHal1CqnQoc0+TvyjGjUI9tDFxcOYYC+OLrxxWR9bYW4k3Yk6oyYftE+V2EO1l0 u1wE9O2vuVt4kuwRCPP5jeddyX1Xcr3/fMldlM9nLbSz2gplV/cNtik2LXK8sEMeU8YGasrIDWma ZAn7RNCHQb3OnA5JcWJKI3jM6rqDCwU2a5Dg6iOqokGEU2iw654mEsqMdChRyiUc7MxwJW2NhyZd 2WNhUx8YbD2QWO3ywA6v6OH8XFCQMbtNaA6fOaMVTeCszFauZERB7ddhVtdCnZlb3YhmSp3DrVAZ fDivGgwW1oQGBEHbAlZehfO5Zg0HE8xIoO1u997cLcYLF+kiGeGAZD7Ses/7qG6clMeKuQmA2Knw kT7knWK1EreWJvsG3M7ipDK7xgJ2uffexEt5BM+8pPP2RDqypJycLEFHba/VXG56yMdp2xvDmRYe 4xS8LnXPh1kIF0O+EjbsT01mk+Uzb7ZyxdwkqMM1hbX7nMJOHUiFVFtYRjY0zFQWAizRnKz8y00w 60UpYCP9NaRYWYNg+NekADu6riXjMfFV2dmlEW07+5qVUj5RRAyi4AiN2ETsY3C/DlXQJ6ASriZM RdAvcI+mrW2m3OKcJV359srg7DhmaYSzcqtTNM9kCzd5XMhg3krigW6Vshvlzq+KSfkLUqUcxv8z VfR+AjcFK4H2gA/XuAIjna9tjwsVcahCaUT9voDGwdQOiBa4i4VpCCq4TDb/BTnU/23OWRomreHA p/ZpiASF/UhFgpA9KEsm+k4hVs/2LkuSZYRMRJXElakVe0QOCRvqGriq93YPRRDqpppkZcDgTsaf +55l0CjUTU4535waUuy9Ngf+6c7HJjMo5dZh09Dk9i9ErNhV7XqzPN97y4roiVmb1cizApiVtoJW lvavKcI5t1pbseY0Xm7mwoEX5zWGwaIhSuG+B+k/sP9R4TP7ZUJvqEO+D7UVwYcGTQzCBqL6km08 kC6QdnAEjZMdtMGkSVnTZq2Ttlq+WV9wp1vwPWFsLdlZ/H1OYxfNmcvOycWLNHZmYcfWdmyhqcGz J1MUhsb5QcY4xnzSKn914qN74OgtuN+fMCVNMME3JYGh9RyYPIDktxzN0o2/AAAA//8DAFBLAwQU AAYACAAAACEAoQmDWCUDAAAfBwAAEQAAAHdvcmQvc2V0dGluZ3MueG1snFXbbts4EH1fYP9B0PM6 lmQ7ToQ4RWNb3RZJW6zSD6CkkUyEN5CUFffrO5TEKul6g2KfTJ0zczicm2/ePXMWHEEbKsUmjC+i MABRyoqKZhN+e8xmV2FgLBEVYVLAJjyBCd/d/vnHTZcasBbNTIASwqRyE7ZapKY8ACdmxmmppZG1 nZWSp7KuaQnjTzh66E14sFal8/nodCEVCFSrpebEmgupm/nguZNly0HYeRJFl3MNjFgM2ByoMl6N /181vOrgRY5vPeLImbfr4ugty/G5ndTVT4/fCc85KC1LMAYzy9nwXE6o8DKG/Y7OkM97WmiiTy9E brFs36XkQZcq0CUmFGseReHcEZX8LO2OGsXI6Stp4E62WHZNwfQ0xiXr3BIL6G0UMNb3SMmAYHRd 2mjCOcGaDsggCTVpmX0kRW6lQqMjwfjXyXhjeSCalBZ0rkiJalsprJbM2/UBbSVXGvMxxIi9pIh1 0bYGsv09OcnW4lXzLp0obObKOBt3+EdK6wWj6G6bXO0Wg5ZjJyaJ4+t4jOs1s1isstXunM/ybn11 vT/HrPbraJWdYy6TeL8+q/Y+W27jsz7/HfV2FWX7s2rZ1fI6XrsIMDdjRnjqev2rvr0ZThnmO+BD UbaEF5qS4MFNA3rxtNBPd1R4vgCcSnjJ5G3hydlsIAwnjGVYU0/gIAxMhX21g7oXZg9EN5Nyn3Se 6rNoBfWnn2quYUF/0LJVg2qnifooKoT9hfFyOepRYe8p97hpi9x7CRyKFxR2+ZejdoLzKUFdanGP gcvQPRGN7xMQs2+5M8V+Yzp3uw4eiFLYvGhSNPEmZLQ52NhNhMUvHKCn/qNokpFLeg6/HNd/kNK9 DK3HgzMYjmg1HiZs4bHFhC09tpywlcdWE3bpsUuHHU64BXCMn3Cn+KPDa8mY7KD624Ob8F/QkARz IAqwrm7KceRk2gNYtB4Ijik844qBilr8G1G04uR5E67iJHHuozWuGxziV7aOc8bqFRpUxBJcWH2p Xjn3Tf5LLLjQoKTYkPmJF9NSuRgCZ9TYHBTuHys1PrlfTH/1ytM/2+0PAAAA//8DAFBLAwQUAAYA CAAAACEAb2NQm4sBAAAHBAAAEgAAAHdvcmQvZm9udFRhYmxlLnhtbKSS326CMBTG75fsHUjvJwX/ TI1onNPLXSzuAY5YpAltSU+V+fY7UORiZolkkDThO+3H6e98i9W3KoKLsCiNTlg04CwQOjVHqU8J +9rvXqYsQAf6CIXRImFXgWy1fH5aVPPMaIcBndc4twnLnSvnYYhpLhTgwJRCUy0zVoGjT3sKTZbJ VLyb9KyEdmHM+SS0ogBH/8Zclshat+oRt8rYY2lNKhCpWVV4PwVSs2XbXVDNNSjqegOFPFjZFErQ BkVEtQsUCeMx3/ExrfU74sN6ZWHtkOZgUbhuI/dyBkoW15uKlUT0hVK6NL/pF7ASDoXwJZQnKpzx wBO2jTjn8W7HvBIlbETCetMpMTXln1m7Z9gpNB5qrPFptkSzxocU8mlPNX2Gfj53JPZSCQw+RBV8 GgUe1T2RmE+IxJh41GSGvYjYxrch+CgRajxed/enm2xIeZ2Oovb+vYh4n8eJbEBRNOCPbNQEPIma SL9s9CexrkcYb39lg/PR2x2JJgmUqP9kow0JLn8AAAD//wMAUEsDBBQABgAIAAAAIQAoh3GlzwAA AB8BAAAUAAAAd29yZC93ZWJTZXR0aW5ncy54bWyMj8tOAzEMRfdI/MMoe5qBRYVGnamEUNlQqMRj n2Y8nUiJHdmB0H495rFhx/LaV8fHq/VHis07sATC3lwuWtMAehoDHnrz8ry5uDaNFIeji4TQmyOI WQ/nZ6vaVdg/QSnalEYpKB33Zi4ld9aKnyE5WVAG1N1EnFzRyAdL0xQ83JJ/S4DFXrXt0jJEV9RA 5pDF/NLqf2iVeMxMHkRUJMUfXnIBzaCOlEtI4QQb4humKsD2a6z3jo/4ur3/Ti5GqruHOw32z1vD JwAAAP//AwBQSwMEFAAGAAgAAAAhAPeMfebgAQAA2wMAABAACAFkb2NQcm9wcy9hcHAueG1sIKIE ASigAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnFPBbtswDL0P2D8Yvjd2nCUoAkXFkGLo YVsDxG3PnEzHwmRJkNSg2dePshtX2XqqT+Qj/fT0SLGbl15lR3ReGr3J57Myz1AL00h92OQP9ber 6zzzAXQDymjc5Cf0+Q3//IntnLHogkSfEYX2m7wLwa6LwosOe/AzKmuqtMb1ECh1h8K0rRR4a8Rz jzoUVVmuCnwJqBtsruxEmI+M62P4KGljRNTnH+uTJcGc1dhbBQH5zyhHsWICWG0CqFr2yBcl4VPG dnBAz+esGAP2ZFzj+bIkZAzZtgMHIpB7vLpe0d8JwL5aq6SAQMbyH1I4400bsvvBgiwSsCJtYWTL HsWzk+HEiSpN2XepSUq1YMUYkTYHBwe283wVBU4Z2wtQuKXL8xaUR1a8AewOIQ52B5IUs2NYH1EE 4zIv/9Boqzz7BR6jZZv8CE6CDmRdbBuTIVbWB8drGRRxU23MhzBtS2P5JbpIvRRcNkZw1ECFS3XD Cf6+pbuFd8TOU7GDhlFqIicJpzP+Yd2a3oI+0eFTRAb/9g+2NrdxXV49vASTuT/J0O0tCJrOYrFc phuQlNieFgUbGumZ8A1gd+S3U/FU2h59wObc838h7tTj+Fb5vJqV9A1LdMZoE6ZHxP8CAAD//wMA UEsDBBQABgAIAAAAIQBTfUAmVgEAAHkCAAARAAgBZG9jUHJvcHMvY29yZS54bWwgogQBKKAAAQAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACMkstuwjAQRfeV+g+R94mTAC2ykiD1wQraSk0f6s6y B4gaP2S7BP6+TgIpqF10Ob7XZ+6Mnc12og62YGylZI6SKEYBSKZ4Jdc5einn4RQF1lHJaa0k5GgP Fs2Ky4uMacKUgSejNBhXgQ08SVrCdI42zmmCsWUbENRG3iG9uFJGUOdLs8aask+6BpzG8RUW4Cin juIWGOqBiA5Izgak/jJ1B+AMQw0CpLM4iRL843VghP3zQqecOEXl9trPdIh7yuasFwf3zlaDsWma qBl1MXz+BL8vF8/dqGEl210xQEXGGWEGqFOmqEGqrcrwyVG7vppat/SbXlXAb/bF4v7h8fUxw7+V 1mxgW7VvVEw6x1D6Pt1YfTPggQ9K+rGOytvo9q6coyKNk0mYpGEyLuNrMhmROP5oQ53db4P3B+IQ 7R/EdFqmMUnG58QjoOgSn3+W4hsAAP//AwBQSwMEFAAGAAgAAAAhAOYmwWoaBwAA+zkAAA8AAAB3 b3JkL3N0eWxlcy54bWy0m1FT2zgQx99v5r6Dx+80ISnJlWnaoRSuzLQcbWDuWbEVoqlj5SylQD/9 SStbGBvbu9g8EcvW/lba1X+dIL3/eL9Ngl88U0Kmi/DwzTgMeBrJWKS3i/Dm+vzgrzBQmqUxS2TK F+EDV+HHD3/+8f7uWOmHhKvAGEjVcbYIN1rvjkcjFW34lqk3csdTc28tsy3T5jK7Hcn1WkT8s4z2 W57q0WQ8no0ynjBt4GojdirMrd1hrN3JLN5lMuJKGW+3ibO3ZSINPxj3Yhl95mu2T7Syl9lVll/m V/DnXKZaBXfHTEVCXBvHzRC3IpXZl5NUidDc4UzpEyVY+eZZ3mbvb+yD5Zu+Z6R0yeAnEYtwZKHq t+n2iyWLcDIpWk6tE0/aEpbeFm08Pbi4LDuzCH3TythdhCw7WJ5YYyMYafG3NOLdk/GbK3BlxyIz d4bD1pqbGJqQWE4ibKwn81lx8WOfmAa21zKHgAEDK5s1l5VJN6E1gV66RDF3+fqrjH7yeKnNjUUI LNN4c3GVCZkJ/bAI372zTNO45FvxRcQxt3mZt92kGxHzfzc8vVE8fmz/fg5ZlluM5D7Vxv3ZHBIh UfHZfcR3NsuM6ZTZIF/aDok1q0occGgvHr1xDRUqNP5XIA9dDJ+lbDizKykA/1tBMOp9b9DEjqg8 ALBL8nXa38Tb/iaO+puA5O03F/P+Xhj97BsRlxulrMQHVcvIJV95HqbvWlLW9qhlUWePWtJ09qjl SGePWkp09qhlQGePWsA7e9Ti29mjFs7WHhED4apm0RRmA7Wwr4VOuO3fKkCHPaUuLzXBFcvYbcZ2 m8DW1qrbbWK53K80zlWQ05eL5VJnMr3tnBFTne3SfbEmn213G6aEeanpmPpJz6m/ZquEB39nIu5E Hbnkq40JXkyeLWFXCYv4RiYxz4Jrfu8iSuh/KYOle8vodK5nWL+K240OlhsouZ2wWcOkN8+Es/9V KJiD1sU0axhKl3FUDGcNedls/BuPxX5bTA3ibWTm9JwQ5goCXGyforc2RPXV1TkKGwDMEFy5oA8B 7CP8d8WFbt/GGOO/K0UvtI/w3xWuF9qH/GiPL1lpPrPsZ4BaXnPy2j2ViczW+6RYA53yMCevYI/A DYG8iL19lEjMySv4iXwGJ1Fkvrlh8pQci0cdJVDI4XAUWGz4sZCDUpG9Q8KIyAGqsCYEVj+tJYDI ovuD/xL2tydqMQCV9u+anct52jADpgSh3qG/76XufoeeNGgelnKRmp9LFA9wtGnDysPS8nxy9Y4Q 436FjwDqVwEJoH6lkABqyI/mdx5fE/GQ/sWRwCLLsq9ikHZoZZ6TldmDaCVgoLqJeP9qWL3NuVCv mwgKOUD1uomgkKNTqWW+biJYg9VNBKuhajTHqKyplEGR62YZ5N8EECMaRrwRoGHEGwEaRrwRoP7i 3Q0ZTrwRLLI2eE0tizcCBI9Qvup7UFm8ESCyNji1y38zKuoeWGn/cjuAeCMo5ADVxRtBIUenSbwR LHiEkgkVlpc6BGsY8UaAhhFvBGgY8UaAhhFvBGgY8UaA+ot3N2Q48UawyNrgNbUs3ggQWR48qCze CBA8QtGGZ8UbVv2rizeCQg5QXbwRFHJ0KoLqX1IRLHKAKiwv3ggWPEJJhpwFyU0Z1DDijRjRMOKN AA0j3gjQMOKNAPUX727IcOKNYJG1wWtqWbwRILI8eFBZvBEgsjY8K96wGF9dvBEUcoDq4o2gkKNT EVSvcwgWOUAVlhdvBAvypbd4I0DwyEtBlBENI96IEQ0j3gjQMOKNAPUX727IcOKNYJG1wWtqWbwR ILI8eFBZvBEgsjY8K96wRl5dvBEUcoDq4o2gkKNTEVQv3ggWOUAVlpc6BGsY8UaAIDF7izcCBI+8 AASriBKmYcQbMaJhxBsB6i/e3ZDhxBvBImuD19SyeCNAZHnwoLJ4I0BkbbD7bM1+UfT21MOGJMDu Myh2NaCBk4YgYYH5AH/wNc/MYSbevTukJ7AYIYHYkB7YIX6S8meA29g9bUgQNEqsEiFhS/cD7NIp HUSYzltOElz/cxp8cQdgav0gpZ7uvDGnh8rHheB4kj04ZPzUDztzZGdX7Cy31swBIXu0Kz8CBEfR LsyBoPxYj+1sz/mYB+FQVd4M/7fNqfDZHHuLi2fG46Oz+fjoPD/gBCbrTkQb40Vkzkq1OJFvhfe7 k2AjfNWlhv3y4NbjYY3CuXzf/OPblXvuye5N02TmsMFvbfeIt/gMe8hbZy+AR1y86w6aY1vgUpeH fr8VPK1XiTuIZj5cpDYU5uQf/G/NhTy+Z86suX/Kk+Qbg2NrWu6aH034Wru7h2OokxVTK6m13Db3 z2AbOXjynAEzxWVn3KUdRPPcp/vtimfmHFjL/F9KW1/gvNrTxHU7Yl24/coz3kNeY2f90bfik/rw PwAAAP//AwBQSwECLQAUAAYACAAAACEA3fyVN2YBAAAgBQAAEwAAAAAAAAAAAAAAAAAAAAAAW0Nv bnRlbnRfVHlwZXNdLnhtbFBLAQItABQABgAIAAAAIQAekRq38wAAAE4CAAALAAAAAAAAAAAAAAAA AJ8DAABfcmVscy8ucmVsc1BLAQItABQABgAIAAAAIQDWZLNR+gAAADEDAAAcAAAAAAAAAAAAAAAA AMMGAAB3b3JkL19yZWxzL2RvY3VtZW50LnhtbC5yZWxzUEsBAi0AFAAGAAgAAAAhAKsR9BCDCgAA yx0AABEAAAAAAAAAAAAAAAAA/wgAAHdvcmQvZG9jdW1lbnQueG1sUEsBAi0AFAAGAAgAAAAhAJa1 reKWBgAAUBsAABUAAAAAAAAAAAAAAAAAsRMAAHdvcmQvdGhlbWUvdGhlbWUxLnhtbFBLAQItABQA BgAIAAAAIQChCYNYJQMAAB8HAAARAAAAAAAAAAAAAAAAAHoaAAB3b3JkL3NldHRpbmdzLnhtbFBL AQItABQABgAIAAAAIQBvY1CbiwEAAAcEAAASAAAAAAAAAAAAAAAAAM4dAAB3b3JkL2ZvbnRUYWJs ZS54bWxQSwECLQAUAAYACAAAACEAKIdxpc8AAAAfAQAAFAAAAAAAAAAAAAAAAACJHwAAd29yZC93 ZWJTZXR0aW5ncy54bWxQSwECLQAUAAYACAAAACEA94x95uABAADbAwAAEAAAAAAAAAAAAAAAAACK IAAAZG9jUHJvcHMvYXBwLnhtbFBLAQItABQABgAIAAAAIQBTfUAmVgEAAHkCAAARAAAAAAAAAAAA AAAAAKAjAABkb2NQcm9wcy9jb3JlLnhtbFBLAQItABQABgAIAAAAIQDmJsFqGgcAAPs5AAAPAAAA AAAAAAAAAAAAAC0mAAB3b3JkL3N0eWxlcy54bWxQSwUGAAAAAAsACwDBAgAAdC0AAAAA --===============0781748591==-- From david@fromorbit.com Mon Jan 4 15:37:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 35A407F37 for ; Mon, 4 Jan 2016 15:37:18 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id CBFDCAC002 for ; Mon, 4 Jan 2016 13:37:14 -0800 (PST) X-ASG-Debug-ID: 1451943428-04bdf06f7100610001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 9v8HZ37XyFcbesP8 for ; Mon, 04 Jan 2016 13:37:08 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DyCACX5YpWPGu7LXleKAECgw8iMG2GYIF5oDICCgwBAQEBAQEGgQyKSYk+GAKFbwSBIE0BAQEBAQEHAQEBAUE/hGIvOyQ0BSUDBy2ILqBpoTMJGYV1igIJgnUMQRuBGwWNOYlNhUCICIFlh26FLwKOOoI6gkgqNINGgUoBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 08:06:44 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGCnz-0002nq-QY for xfs@oss.sgi.com; Tue, 05 Jan 2016 08:36:43 +1100 Date: Tue, 5 Jan 2016 08:36:43 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [ANNOUNCE] xfs: for-next branch rebased to 4922be5 Message-ID: <20160104213643.GE19802@dastard> X-ASG-Orig-Subj: [ANNOUNCE] xfs: for-next branch rebased to 4922be5 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kVXhAStRUZ/+rrGn" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451943428 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25831 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- --kVXhAStRUZ/+rrGn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi folks, The for-next branch of the xfs kernel repository at git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git has just been updated. I rebased the xfs-log-torn-write branch to include the 32bit arch build fix that the kbuild robot noticed and Brain fixed. It's the base branch in the for-next tree so while only one commit changed, I had to restart the for-next branch build from scratch... -Dave The new head of the for-next branch is commit: 4922be5 Merge branch 'xfs-dax-fixes-for-4.5' into for-next New Commits: Alexander Kuleshov (1): [211fe1a] xfs: make xfs_buf_ioend_async() static Brian Foster (8): [a70f9fe] xfs: detect and handle invalid iclog size set by mkfs [9d94901] xfs: refactor log record unpack and data processing [b94fb2d] xfs: refactor and open code log record crc check [d7f3769] xfs: return start block of first bad log record during reco= very [6528250] xfs: support a crc verification only log record pass [eed6b46] xfs: refactor log record start detection into a new helper [7088c41] xfs: detect and trim torn writes during log recovery [609adfc] xfs: debug mode log record crc error injection Darrick J. Wong (5): [2e9101d] libxfs: make xfs_alloc_fix_freelist non-static [9b434a3] xfs: fix log ticket type printing [6d3eb1e] libxfs: use a convenience variable instead of open-coding t= he fork [96f859d] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is = correct [c5ab131] libxfs: refactor short btree block verification Dave Chinner (4): [3b0fe47] xfs: Don't use reserved blocks for data blocks with DAX [a6d7636] xfs: fix recursive splice read locking with DAX [7eeabbd] Merge branch 'xfs-misc-fixes-for-4.5' into for-next [4922be5] Merge branch 'xfs-dax-fixes-for-4.5' into for-next Eric Sandeen (2): [233135b] xfs: print name of verifier if it fails [f1f96c4] xfs: get mp from bma->ip in xfs_bmap code Jia He (1): [1d4292b] libxfs: Optimize the loop for xfs_bitmap_empty Markus Elfring (1): [a841b64] XFS: Use a signed return type for suffix_kstrtoint() Masatake YAMATO (1): [ffc671f] xfs: send warning of project quota to userspace via netlink Code Diffstat: fs/xfs/libxfs/xfs_alloc.c | 4 +- fs/xfs/libxfs/xfs_alloc.h | 1 + fs/xfs/libxfs/xfs_alloc_btree.c | 35 +-- fs/xfs/libxfs/xfs_attr_leaf.c | 1 + fs/xfs/libxfs/xfs_attr_remote.c | 1 + fs/xfs/libxfs/xfs_bit.c | 6 +- fs/xfs/libxfs/xfs_bmap.c | 27 +- fs/xfs/libxfs/xfs_bmap_btree.c | 1 + fs/xfs/libxfs/xfs_btree.c | 58 ++++ fs/xfs/libxfs/xfs_btree.h | 3 + fs/xfs/libxfs/xfs_da_btree.c | 1 + fs/xfs/libxfs/xfs_dir2_block.c | 1 + fs/xfs/libxfs/xfs_dir2_data.c | 2 + fs/xfs/libxfs/xfs_dir2_leaf.c | 2 + fs/xfs/libxfs/xfs_dir2_node.c | 1 + fs/xfs/libxfs/xfs_dquot_buf.c | 1 + fs/xfs/libxfs/xfs_format.h | 2 +- fs/xfs/libxfs/xfs_ialloc.c | 1 + fs/xfs/libxfs/xfs_ialloc_btree.c | 27 +- fs/xfs/libxfs/xfs_inode_buf.c | 2 + fs/xfs/libxfs/xfs_log_recover.h | 1 + fs/xfs/libxfs/xfs_sb.c | 2 + fs/xfs/libxfs/xfs_symlink_remote.c | 1 + fs/xfs/xfs_buf.c | 2 +- fs/xfs/xfs_buf.h | 1 + fs/xfs/xfs_error.c | 4 +- fs/xfs/xfs_file.c | 25 +- fs/xfs/xfs_iomap.c | 11 +- fs/xfs/xfs_log.c | 51 +++- fs/xfs/xfs_log_priv.h | 3 + fs/xfs/xfs_log_recover.c | 572 ++++++++++++++++++++++++++++++---= ---- fs/xfs/xfs_super.c | 2 +- fs/xfs/xfs_sysfs.c | 36 +++ fs/xfs/xfs_trans_dquot.c | 14 +- 34 files changed, 693 insertions(+), 209 deletions(-) --=20 Dave Chinner david@fromorbit.com --kVXhAStRUZ/+rrGn Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBAgAGBQJWiuXrAAoJEK3oKUf0dfodsOcP+gLWHk4z3AYYaKUwWLpfuCq9 HWPm68c+2WE3MvoX90UMUAxKhVPzZhWas4zbLEiKKzhWj+oANm28jnUqX+cRbvrQ EihIp1cuJVvtm5ZYv8FQNEup0PlfhzhNro9DrCU5k7X8qHfCNxUWnRSk97gbIUiQ cYymhYudBc4CUkyroShqX9obCYtJAlITxW3zZwGktXDFO+6mlRBIeZ38ok4A5+zb QszqglLqHUFx8XrKYB8CG03FFp2ShNZ7J0MgeYsCnT7yooD7taUPi2VrwdcR78d5 RDrzgw7f/eXypaxk2lUSjRR9Vwk6xEkBw8RoPHuWOMCzayMP235CduD5kFY7Fshd PZ5USCqYOu14vGjBtpZbDAYCjsUBEa7IPUxNMGScl8wfnYYCtwKDSSv6vvEEmcn0 +sl4lwv+Sd2+63f94quCxKDyyfCIlrBo+hkPtns2Q7cCRz3uBSfDyrc+wJ8DFtDO Sb+31kvrO1Rmy3Kd+5JKQyRZqZbbyuWWNPUl5QEoYhXw+Qxe619/oeZgk11JY0hJ nyp74AEpVAUvyzRIUTI93KKlgp4jANnRB+y36kR9yxsgfCEhpP6Few4nvVuGqASy oPCqVOXxqyb99pyJKDw7F+s0OHyvxxSMA9g807R1tMFhW8ZeF2KuTi1ntBEF8O5p KnKKjeI+FRl2DxMeAT7A =MYxv -----END PGP SIGNATURE----- --kVXhAStRUZ/+rrGn-- From darrick.wong@oracle.com Mon Jan 4 16:06:24 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 657A77F37 for ; Mon, 4 Jan 2016 16:06:24 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 5522230405F for ; Mon, 4 Jan 2016 14:06:21 -0800 (PST) X-ASG-Debug-ID: 1451945176-04cb6c20fe01100001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id zL0wQfbw5k3ObEpe (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 14:06:16 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u04M6FWi022182 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 22:06:15 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u04M6FZK005879 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 22:06:15 GMT Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19]) by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u04M6EGu030154; Mon, 4 Jan 2016 22:06:15 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 14:06:14 -0800 Date: Mon, 4 Jan 2016 14:06:13 -0800 From: "Darrick J. Wong" To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 03/76] libxfs: refactor the btree size calculator code Message-ID: <20160104220613.GA28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 03/76] libxfs: refactor the btree size calculator code References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219085642.12713.80467.stgit@birch.djwong.org> <20151220203928.GS26718@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151220203928.GS26718@dastard> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0022.oracle.com [141.146.126.234] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451945176 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25832 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Mon, Dec 21, 2015 at 07:39:28AM +1100, Dave Chinner wrote: > On Sat, Dec 19, 2015 at 12:56:42AM -0800, Darrick J. Wong wrote: > > Create a macro to generate btree height calculator functions. > > This will be used (much) later when we get to the refcount > > btree. > > > > Signed-off-by: Darrick J. Wong > .... > > +/* btree size calculator templates */ > > +#define DECLARE_BTREE_SIZE_FN(btree) \ > > +xfs_filblks_t xfs_##btree##_calc_btree_size(struct xfs_mount *mp, \ > > + unsigned long len); > > + > > +#define DEFINE_BTREE_SIZE_FN(btree, limitfield, maxlevels) \ > > +xfs_filblks_t \ > > +xfs_##btree##_calc_btree_size( \ > > + struct xfs_mount *mp, \ > > + unsigned long len) \ > > +{ \ > > + int level; \ > > + int maxrecs; \ > > + xfs_filblks_t rval; \ > > +\ > > + maxrecs = mp->limitfield[0]; \ > > + for (level = 0, rval = 0; level < maxlevels; level++) { \ > > + len += maxrecs - 1; \ > > + do_div(len, maxrecs); \ > > + rval += len; \ > > + if (len == 1) \ > > + return rval + maxlevels - \ > > + level - 1; \ > > + if (level == 0) \ > > + maxrecs = mp->limitfield[1]; \ > > + } \ > > + return rval; \ > > +} > > I really don't like using macros like this. The code becomes hard to > debug, hard to edit, the functions don't show up in grep/cscope, > etc. > > A helper function like this: > > xfs_filblks_t > xfs_btree_calc_size( > struct xfs_mount *mp, > int *limits, > int maxlevels, > unsigned long len) > { > int level; > int maxrecs; > xfs_filblks_t rval; > > maxrecs = limits[0]; > for (level = 0, rval = 0; level < maxlevels; level++) { > len += maxrecs - 1; > do_div(len, maxrecs); > rval += len; > if (len == 1) > return rval + maxlevels - level - 1; > if (level == 0) > maxrecs = limits[1]; > } > return rval; > } > > will work just as well when wrapped with the btree specific calling > function and that will have none of the problems using a macro to > build the functions has... Will change to use a helper function. --D > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Mon Jan 4 16:12:48 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 781D27F37 for ; Mon, 4 Jan 2016 16:12:48 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 57CD08F8040 for ; Mon, 4 Jan 2016 14:12:48 -0800 (PST) X-ASG-Debug-ID: 1451945564-04cbb0451501520001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id oLLsXIuDddf98yBy (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 14:12:45 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u04MCLBb016666 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 22:12:21 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u04MCKLm017638 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 22:12:21 GMT Received: from abhmp0015.oracle.com (abhmp0015.oracle.com [141.146.116.21]) by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u04MCJXI005741; Mon, 4 Jan 2016 22:12:19 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 14:12:19 -0800 Date: Mon, 4 Jan 2016 14:12:18 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 05/76] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct Message-ID: <20160104221218.GB28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 05/76] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219085655.12713.60317.stgit@birch.djwong.org> <20160103121525.GB28429@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160103121525.GB28429@infradead.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1451945565 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25832 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sun, Jan 03, 2016 at 04:15:25AM -0800, Christoph Hellwig wrote: > On Sat, Dec 19, 2015 at 12:56:55AM -0800, Darrick J. Wong wrote: > > Because struct xfs_agfl is 36 bytes long and has a 64-bit integer > > inside it, gcc will quietly round the structure size up to the nearest > > 64 bits -- in this case, 40 bytes. This results in the XFS_AGFL_SIZE > > macro returning incorrect results for v5 filesystems on 64-bit > > machines (118 items instead of 119). As a result, a 32-bit xfs_repair > > will see garbage in AGFL item 119 and complain. > > > > Therefore, tell gcc not to pad the structure so that the AGFL size > > calculation is correct. > > Do you have a testcase for this? Not much aside from: 0. Build kernel/xfsprogs with RFCv4 patches on a 64bit machine. 1. Build kernel/xfsprogs with RFCv4 patches on a 32bit machine. 2. Format a XFS with reflink and rmap on a 64-bit machine, so that the AGFL size is maximized. 3. Mount FS and create a reflinked file. 4. Unmount and xfs_repair with the 32-bit build. I guess we could create a program that compares all the known sizeof(struct xfs_disk_object) values against known good values and stuff that into the xfsprogs build process. --D > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Mon Jan 4 17:24:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 799C47F37 for ; Mon, 4 Jan 2016 17:24:01 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 5BCC2304062 for ; Mon, 4 Jan 2016 15:23:58 -0800 (PST) X-ASG-Debug-ID: 1451949836-04cb6c20ff02670001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id JK5BCkHISYjlLfEp (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 15:23:56 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u04NNW9m005206 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 23:23:33 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u04NNWC2019928 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 23:23:32 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u04NNRa6005479; Mon, 4 Jan 2016 23:23:29 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 15:23:24 -0800 Date: Mon, 4 Jan 2016 15:23:18 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 05/76] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct Message-ID: <20160104232318.GC28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 05/76] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219085655.12713.60317.stgit@birch.djwong.org> <20160103121525.GB28429@infradead.org> <20160104221218.GB28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104221218.GB28330@birch.djwong.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0021.oracle.com [156.151.31.71] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451949836 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25833 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Mon, Jan 04, 2016 at 02:12:18PM -0800, Darrick J. Wong wrote: > On Sun, Jan 03, 2016 at 04:15:25AM -0800, Christoph Hellwig wrote: > > On Sat, Dec 19, 2015 at 12:56:55AM -0800, Darrick J. Wong wrote: > > > Because struct xfs_agfl is 36 bytes long and has a 64-bit integer > > > inside it, gcc will quietly round the structure size up to the nearest > > > 64 bits -- in this case, 40 bytes. This results in the XFS_AGFL_SIZE > > > macro returning incorrect results for v5 filesystems on 64-bit > > > machines (118 items instead of 119). As a result, a 32-bit xfs_repair > > > will see garbage in AGFL item 119 and complain. > > > > > > Therefore, tell gcc not to pad the structure so that the AGFL size > > > calculation is correct. > > > > Do you have a testcase for this? > > Not much aside from: > > 0. Build kernel/xfsprogs with RFCv4 patches on a 64bit machine. > 1. Build kernel/xfsprogs with RFCv4 patches on a 32bit machine. > 2. Format a XFS with reflink and rmap on a 64-bit machine, so that the AGFL > size is maximized. > 3. Mount FS and create a reflinked file. > 4. Unmount and xfs_repair with the 32-bit build. > > I guess we could create a program that compares all the known sizeof(struct > xfs_disk_object) values against known good values and stuff that into the > xfsprogs build process. I created a patch that uses BUILD_BUG_ON to stop kbuild if the sizes of the on-disk data structures don't match known values. I've also ported it to xfsprogs, so we can check both. --D > > --D > > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com > > http://oss.sgi.com/mailman/listinfo/xfs > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Mon Jan 4 17:41:04 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id B822C7F37 for ; Mon, 4 Jan 2016 17:41:04 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 47EB8AC002 for ; Mon, 4 Jan 2016 15:41:01 -0800 (PST) X-ASG-Debug-ID: 1451950858-04cbb04515032f0001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id U1E0MFanjm06cVVI (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 15:40:59 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u04NeZHV021105 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 23:40:35 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u04NeZS9003628 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 23:40:35 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u04NeYXn012553; Mon, 4 Jan 2016 23:40:34 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 15:40:34 -0800 Date: Mon, 4 Jan 2016 15:40:33 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 34/76] xfs: implement XFS_IOC_SWAPEXT when rmap btree is enabled Message-ID: <20160104234033.GD28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 34/76] xfs: implement XFS_IOC_SWAPEXT when rmap btree is enabled References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090008.12713.25755.stgit@birch.djwong.org> <20160103121728.GC28429@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160103121728.GC28429@infradead.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451950859 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25834 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sun, Jan 03, 2016 at 04:17:28AM -0800, Christoph Hellwig wrote: > On Sat, Dec 19, 2015 at 01:00:08AM -0800, Darrick J. Wong wrote: > > Implement extent swapping when reverse-mapping is enabled. > > Can you just fold this into the previous patch? There seem to be a lot > of patches in the series that would benefit from merging, but this > is the one that really screams for it :) The only reason I'm keeping Dave's original rmap patches separate is because the last time I checked with Dave, he said that he'd made a few bugfixes to his patchset that hadn't yet been sent out, so I'd like to merge those fixes into the original patches before starting any combining work. It also might be more convenient (for him, anyway) to keep my rmapbt changes separate for review purposes. At this point, though, I've made so many changes that I'd like to pull in Dave's changes and combine them with mine into fewer patches. (AFAIK, Dave hasn't sent any further reverse-mapping patches since 3 June 2015.) --D > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From david@fromorbit.com Mon Jan 4 17:45:44 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id AADEA7F37 for ; Mon, 4 Jan 2016 17:45:44 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 988688F8033 for ; Mon, 4 Jan 2016 15:45:41 -0800 (PST) X-ASG-Debug-ID: 1451951137-04cb6c20fe02b70001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id JVBA7WEmGKDxF7PL for ; Mon, 04 Jan 2016 15:45:37 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2D1CQDmAotWPGu7LXleKAECgw8iMG2CYoN+gXmgMgIKDAEBAQEBAQaBDIpJhTmEBRgChW8EgSFNAQEBAQEBBwEBAQFBP4RiLzskNAUlAwctiC6gZ6ETCRmFdY0ADEEbgRsFlwaFQIgIgWWNHQKFWIhigjo4H4FxKjSFEAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 10:15:36 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGEoh-000349-DL for xfs@oss.sgi.com; Tue, 05 Jan 2016 10:45:35 +1100 Date: Tue, 5 Jan 2016 10:45:35 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [ANNOUNCE] xfs-documentation: master branch updated to a04146a Message-ID: <20160104234535.GF19802@dastard> X-ASG-Orig-Subj: [ANNOUNCE] xfs-documentation: master branch updated to a04146a MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="EY/WZ/HvNxOox07X" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451951137 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25833 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- --EY/WZ/HvNxOox07X Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi folks, The xfs documentation repository at git://git.kernel.org/pub/scm/fs/xfs/xfs-documentation.git has just been updated. This is a major update to the XFS Filesystem Structure document which brings it up to date with the current on disk format. A big thank you must go to Darrick for doing this work, and to Christoph for reviewing it. -Dave. The new head of the guilt/working branch is commit: a04146a move directory chapter before xattr chapter New Commits: Darrick J. Wong (20): [38d3e30] fix asciidoc confusion w.r.t. include file newline at EOF [bda4165] update book subtitles and headings [728d229] restore the old rendering of structure field definitions [042c084] convert html entities to regular characters [f14c343] convert images to text [c93a293] delete images of hex dumps [0b39c06] update types [d77fa4e] make example section titles consistent [983e8a1] fix various errors and missing bits in the text [38dcd43] add missing structure field definitions [410e764] add missing xfs_db examples [36ce418] fix quoted text markings [8b4a922] introduce XFS at a high level [02be33c] document the xfs log format [13be785] document the operation of the realtime device and inodes [372d53f] add documentation of v5 fields [e95c161] add a chapter discussing v5 disk format [70cbe0d] document magic numbers and theoretical limits [83abac0] document the sparse inodes feature [a04146a] move directory chapter before xattr chapter Code Diffstat: .../allocation_groups.asciidoc | 808 ++++++++++++++++-= --- .../XFS_Filesystem_Structure/common_types.asciidoc | 41 +- .../XFS_Filesystem_Structure/data_extents.asciidoc | 210 ++++-- .../XFS_Filesystem_Structure/directories.asciidoc | 818 +++++++++++++++++= --- design/XFS_Filesystem_Structure/docinfo.xml | 43 +- .../extended_attributes.asciidoc | 466 ++++++++++-- .../XFS_Filesystem_Structure/images/code/33a.png | Bin 106484 -> 0 bytes .../XFS_Filesystem_Structure/images/code/33b.png | Bin 40229 -> 0 bytes design/XFS_Filesystem_Structure/images/code/40.png | Bin 94094 -> 0 bytes design/XFS_Filesystem_Structure/images/code/46.png | Bin 78484 -> 0 bytes design/XFS_Filesystem_Structure/images/code/57.png | Bin 17987 -> 0 bytes design/XFS_Filesystem_Structure/images/code/60.png | Bin 32907 -> 0 bytes design/XFS_Filesystem_Structure/images/code/61.png | Bin 54687 -> 0 bytes design/XFS_Filesystem_Structure/images/code/65.png | Bin 107515 -> 0 bytes design/XFS_Filesystem_Structure/images/code/66.png | Bin 69316 -> 0 bytes design/XFS_Filesystem_Structure/images/code/67.png | Bin 109738 -> 0 bytes design/XFS_Filesystem_Structure/images/code/71.png | Bin 62675 -> 0 bytes .../XFS_Filesystem_Structure/images/code/73-74.png | Bin 153732 -> 0 bytes design/XFS_Filesystem_Structure/images/code/74.png | Bin 62210 -> 0 bytes .../images/code/SVG/33a.svg | 185 ----- .../images/code/SVG/33b.svg | 158 ---- .../images/code/SVG/40.svg | 177 ----- .../images/code/SVG/46.svg | 197 ----- .../images/code/SVG/57.svg | 129 ---- .../images/code/SVG/60.svg | 182 ----- .../images/code/SVG/61.svg | 157 ---- .../images/code/SVG/65.svg | 233 ------ .../images/code/SVG/66.svg | 260 ------- .../images/code/SVG/67.svg | 232 ------ .../images/code/SVG/71.svg | 150 ---- .../images/code/SVG/73-74.svg | 364 --------- .../images/code/SVG/74.svg | 201 ----- .../internal_inodes.asciidoc | 123 +-- .../journaling_log.asciidoc | 836 +++++++++++++++++= +++- design/XFS_Filesystem_Structure/magic.asciidoc | 86 +++ .../metadata_integrity.asciidoc | 36 + .../XFS_Filesystem_Structure/ondisk_inode.asciidoc | 209 ++++-- design/XFS_Filesystem_Structure/overview.asciidoc | 38 + .../symbolic_links.asciidoc | 76 +- .../xfs_filesystem_structure.asciidoc | 51 +- 40 files changed, 3281 insertions(+), 3185 deletions(-) delete mode 100644 design/XFS_Filesystem_Structure/images/code/33a.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/33b.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/40.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/46.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/57.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/60.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/61.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/65.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/66.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/67.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/71.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/73-74.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/74.png delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/33a.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/33b.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/40.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/46.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/57.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/60.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/61.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/65.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/66.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/67.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/71.svg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/73-74.s= vg delete mode 100644 design/XFS_Filesystem_Structure/images/code/SVG/74.svg create mode 100644 design/XFS_Filesystem_Structure/magic.asciidoc create mode 100644 design/XFS_Filesystem_Structure/metadata_integrity.asci= idoc create mode 100644 design/XFS_Filesystem_Structure/overview.asciidoc --=20 Dave Chinner david@fromorbit.com --EY/WZ/HvNxOox07X Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBAgAGBQJWiwQeAAoJEK3oKUf0dfodB/QQANIAE0fJ1FoSuRGXOInJDtqw kv/MYlzd5qKe2RJdaHgMGmOWY0AHKtEhdRbhLyf0jYpaWqWxxXL7pVRAB5nrsSk3 8jFET0iyrTr+u4CWiYYzv98ybMl/DACggeC7RqirOwLXM25vR4trywST+lSov+PL 9cSE15nzPt+hD/HB70BSDlFdT6++DQ7Mnbi7LOwXivM3mKrhjBUqj6LV3L15eZbU IhNr6iU6Cggrr8eZ6p0XnEMNV0zhMtOsd8qjY8AgEq0a6KEYMCClCwscZfrFXi+8 1ddfgIVyjzL2CUk7FHYBju4A1BelRPi6itztRm6HiSOdQ/FDL5c8Q89AoH6rLqTx sNWQqHdzIhECjZOZy9z6VjRppL2GbT2eDu0Xj2DsADbYRP6qV/I6cdB79hn9yAxj L4LAHjWh4NrKbMyfquJ+t+b2R7SDkBn8Bkjx/5cu1rOyHSaI6wuUVs7RwrusngLX kZ0wr8ewD4Gdlat1KSjC9UJTMD/U1TO3s8K8xItzxCVAWA9x44u/ZDNVRf0pxivo ax0iX62A5ulnyQAssB2v/V/eOCQYaSY+1qRob/oAyKLj+EdAxw+MGr6uKhEOfDlG tW5BOZJCeMvMkuWQjVXMjcKMHdG05eCKpUvWz2+qGAb3HNPYICJyEMx6WStSPVS0 AkRtjJDjd1sQgElsEsAP =JQrY -----END PGP SIGNATURE----- --EY/WZ/HvNxOox07X-- From david@fromorbit.com Mon Jan 4 17:52:46 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 685817F37 for ; Mon, 4 Jan 2016 17:52:46 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 24A26304066 for ; Mon, 4 Jan 2016 15:52:46 -0800 (PST) X-ASG-Debug-ID: 1451951556-04bdf06f71031e0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id lykMh2Zm4qHUnwZD for ; Mon, 04 Jan 2016 15:52:37 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CpCQBuBYtWPGu7LXleKAECgw+BP4Jig36BeaBKAQEBAQEBBotVhTmEBYYJBAICgR1NAQEBAQEBBwEBAQFBP4Q1AQEEOhwjEAgDGAklDwUlAwcaE4guwVgBAQgCIRmFdYVHiTwFlwaNSI8ERI12gnIfHYFUKjSFEAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 10:21:59 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGEuL-00034r-Oo; Tue, 05 Jan 2016 10:51:25 +1100 Date: Tue, 5 Jan 2016 10:51:25 +1100 From: Dave Chinner To: "Darrick J. Wong" Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [PATCH 05/76] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct Message-ID: <20160104235125.GG19802@dastard> X-ASG-Orig-Subj: Re: [PATCH 05/76] libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219085655.12713.60317.stgit@birch.djwong.org> <20160103121525.GB28429@infradead.org> <20160104221218.GB28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104221218.GB28330@birch.djwong.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451951556 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25835 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 02:12:18PM -0800, Darrick J. Wong wrote: > On Sun, Jan 03, 2016 at 04:15:25AM -0800, Christoph Hellwig wrote: > > On Sat, Dec 19, 2015 at 12:56:55AM -0800, Darrick J. Wong wrote: > > > Because struct xfs_agfl is 36 bytes long and has a 64-bit integer > > > inside it, gcc will quietly round the structure size up to the nearest > > > 64 bits -- in this case, 40 bytes. This results in the XFS_AGFL_SIZE > > > macro returning incorrect results for v5 filesystems on 64-bit > > > machines (118 items instead of 119). As a result, a 32-bit xfs_repair > > > will see garbage in AGFL item 119 and complain. > > > > > > Therefore, tell gcc not to pad the structure so that the AGFL size > > > calculation is correct. > > > > Do you have a testcase for this? > > Not much aside from: > > 0. Build kernel/xfsprogs with RFCv4 patches on a 64bit machine. > 1. Build kernel/xfsprogs with RFCv4 patches on a 32bit machine. > 2. Format a XFS with reflink and rmap on a 64-bit machine, so that the AGFL > size is maximized. > 3. Mount FS and create a reflinked file. > 4. Unmount and xfs_repair with the 32-bit build. > > I guess we could create a program that compares all the known sizeof(struct > xfs_disk_object) values against known good values and stuff that into the > xfsprogs build process. There's an xfstest for that: xfs/122. It notruns on my systems since the big xfsprogs build/header rework, and I haven't found the time to work out what it needs to run again. Also, I don't think it covers the AGFL structure, because that it relatively new and the test doesn't check any of the v5 specific structures.... Cheers, Dave. -- Dave Chinner david@fromorbit.com From darrick.wong@oracle.com Mon Jan 4 17:59:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id BFBAA7F37 for ; Mon, 4 Jan 2016 17:59:56 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9C1738F8037 for ; Mon, 4 Jan 2016 15:59:56 -0800 (PST) X-ASG-Debug-ID: 1451951994-04cbb04514037d0001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id aAhKF4HBFjPIcBZ6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 15:59:54 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u04NxrWN005521 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 23:59:54 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u04Nxr4p014617 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 4 Jan 2016 23:59:53 GMT Received: from abhmp0017.oracle.com (abhmp0017.oracle.com [141.146.116.23]) by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u04NxqTI020122; Mon, 4 Jan 2016 23:59:53 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 15:59:52 -0800 Date: Mon, 4 Jan 2016 15:59:51 -0800 From: "Darrick J. Wong" To: Brian Foster Cc: xfs@oss.sgi.com Subject: Re: [RFCv4 00/76] xfs: add reverse-mapping, reflink, and dedupe support Message-ID: <20160104235951.GE28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [RFCv4 00/76] xfs: add reverse-mapping, reflink, and dedupe support References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151220140254.GA3618@laptop.bfoster> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151220140254.GA3618@laptop.bfoster> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451951994 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25835 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sun, Dec 20, 2015 at 09:02:54AM -0500, Brian Foster wrote: > On Sat, Dec 19, 2015 at 12:56:23AM -0800, Darrick J. Wong wrote: > > Hi all, > > > ... > > Fixed since RFCv3: > > > > * The reflink and dedupe ioctls are being hoisted to the VFS, as > > provided in the first few patches. Patch 81 connects to this > > functionality. > > > > * Copy on write has been rewritten for v4. We now use the existing > > delayed allocation mechanism to coalesce writes together, deferring > > allocation until writeout time. This enables CoW to make better > > block placement decisions and significantly reduces overhead. > > CoW is still pretty slow, but not as slow as before. > > > > * Direct IO CoW has been implemented using the same mechanism as > > above, but modified to perform the allocation and remapping right > > then and there. Throughput is much higher than pushing data > > through the page cache CoW. (It's the same mechanism, but we're > > playing with chunks bigger than a single memory page.) > > > > * CoW ENOSPC works correctly now, except in the pathological case > > that the AG fills up and the rmap btree cannot expand. That will > > be addressed for v5. > > > > * fallocate will now unshare blocks to prevent future ENOSPC, as > > you'd expect. > > > > * refcount btree blocks are preallocated at mount time to prevent > > ENOSPC while trying to expand the tree. This also has the effect > > of grouping the btree blocks together, which can speed up CoW > > remapping. > > > > Can you elaborate on how these blocks are preallocated? E.g., is the > tree "preconstructed" in some sense? However that is done, is this the > anticipated solution or a temporary workaround..? > > Also, shouldn't the enospc condition be handled by the agfl? I take it > there is something going on here that renders that solution flawed, so > I'm just curious what it is. > > (Sorry if this is all explained elsewhere, but I haven't yet had a > chance to take a close enough look at this feature..). Reference count btree blocks aren't allocated from the AGFL; they're allocated from the free space in the same manner as the inobt, per a review comment from Dave a looong time ago. :) As such, we can get ourselves into the nasty situation where every block in the AG has been allocated to file data. If we then see a bunch of reference count changes that are scattered around the AG, the reference count btree has to expand to hold all the new records... but there isn't space, and the operation fails. Given that we know the maximum possible size of the refcount btree (it's 0.3% of the AG size with 4k blocks), I figured it was easy enough to avoid ENOSPC for reflink operations. I've temporarily fixed this by adding code that figures out how many blocks we need if the reference count btree has to have a unique record for every block in the AG and holding that many blocks until either they're allocated to the refcount btree or freed at umount time. Right now it's a temporary fix (if the FS crashes, the reserved blocks are lost) but it wouldn't be difficult for the FS to make a permanent reservation that's recorded on disk somehow. But that's involves writing things to disk + making xfsprogs understand the reservation; let's see what people say about the reserved pool idea at all. Does that make sense? :) --D > > Brian > > > Issues: > > > > * The extent swapping ioctl still allocates a bigger fixed-size > > transaction. That's most likely a stupid thing to do, so getting a > > better grip on how the journalling code works and auditing all the > > new transaction users will have to happen. Right now it mostly > > gets lucky. > > > > * EFI tracking for the allocated-but-not-yet-mapped blocks is > > nonexistant. A crash will leak them. > > > > * ENOSPC while expanding the rmap btree can crash the FS. For now we > > work around this problem by making the AGFL as big as possible, > > failing CoW attempts with ENOSPC if there aren't enough AGFL blocks > > available, and hoping that doesn't actually happen. > > > > If you're going to start using this mess, you probably ought to just > > pull from my github trees for kernel[1], xfsprogs[2], and xfstests[3]. > > There are also updates for xfs-docs[4] and man-pages[5]. > > > > The patches have been xfstested with x64, i386, and ppc64; while in > > general the tests run to completion, there are still periodic bugs > > that will be addressed by the next RFC. There's a persistent crash on > > arm64 and ppc64el that I haven't been able to triage. > > > > This is an extraordinary way to eat your data. Enjoy! > > Comments and questions are, as always, welcome. > > > > --D > > > > [1] https://github.com/djwong/linux/tree/for-dave > > [2] https://github.com/djwong/xfsprogs/tree/for-dave > > [3] https://github.com/djwong/xfstests/tree/for-dave > > [4] https://github.com/djwong/xfs-documentation/tree/for-dave > > [5] https://github.com/djwong/man-pages/commits/for-mtk > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com > > http://oss.sgi.com/mailman/listinfo/xfs > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Mon Jan 4 18:15:28 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 80B407F37 for ; Mon, 4 Jan 2016 18:15:28 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 5920A304067 for ; Mon, 4 Jan 2016 16:15:25 -0800 (PST) X-ASG-Debug-ID: 1451952917-04bdf06f7303bb0001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id Rulq6ulzqslJdJ2c (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 16:15:18 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u050Eq9f019242 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 00:14:52 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u050EpKP031232 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 00:14:52 GMT Received: from abhmp0007.oracle.com (abhmp0007.oracle.com [141.146.116.13]) by userv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u050EpEO008330; Tue, 5 Jan 2016 00:14:51 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 16:14:51 -0800 Date: Mon, 4 Jan 2016 16:14:50 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 15/22] xfsdocs: document the operation of the realtime device and inodes Message-ID: <20160105001450.GF28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 15/22] xfsdocs: document the operation of the realtime device and inodes References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091351.15558.51417.stgit@birch.djwong.org> <20160103134151.GB9144@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160103134151.GB9144@infradead.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0021.oracle.com [156.151.31.71] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451952918 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25835 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sun, Jan 03, 2016 at 05:41:51AM -0800, Christoph Hellwig wrote: > Looks generally good, although I wonder the take on the realtime device > is a bit too positive given that we generally advise people not to use > it. Last time I tried formatting with a rt device (a month ago) the kernel crashed, as soon as I created an inode, so I guess we ought to warn people that this feature is severely bitrotted. What /is/ the official status of XFS RT? Most sane distros (looking right at you, Ubuntu) disable kernel support, so I guess: ------------------------------------------------------------------------ WARNING: The realtime feature is not well maintained. Use with caution. ------------------------------------------------------------------------ --D > > Reviewed-by: Christoph Hellwig > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Mon Jan 4 18:23:52 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3A48E7F37 for ; Mon, 4 Jan 2016 18:23:52 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 17C96304066 for ; Mon, 4 Jan 2016 16:23:51 -0800 (PST) X-ASG-Debug-ID: 1451953428-04bdf06f7103e00001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id rpjiS8hdt9pOcHmC (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 16:23:49 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u050NKPf010752 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 00:23:21 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u050NKKQ028828 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 00:23:20 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u050NK9n007154; Tue, 5 Jan 2016 00:23:20 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 16:23:19 -0800 Date: Mon, 4 Jan 2016 16:23:18 -0800 From: "Darrick J. Wong" To: Dave Chinner Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [RFCv4 00/22] xfs-documentation: massive cleanups and add v5 format, reverse-mapping, reflink, and dedupe support Message-ID: <20160105002318.GG28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [RFCv4 00/22] xfs-documentation: massive cleanups and add v5 format, reverse-mapping, reflink, and dedupe support References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151220152936.GA20135@infradead.org> <20151220204657.GT26718@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151220204657.GT26718@dastard> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0021.oracle.com [156.151.31.71] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1451953429 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_SA210e, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25837 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines 0.00 BSF_SC5_SA210e Custom Rule SA210e On Mon, Dec 21, 2015 at 07:46:57AM +1100, Dave Chinner wrote: > On Sun, Dec 20, 2015 at 07:29:36AM -0800, Christoph Hellwig wrote: > > On Sat, Dec 19, 2015 at 01:12:08AM -0800, Darrick J. Wong wrote: > > > This patch set does the following: > > > > I'll review the actual patches once I'll get more time, but maybe it's a > > good idea to split this series and get all the cleanups and fixes in > > ASAP and only land the new reverse mapping and reflink bits once the > > code hits mainline? > > I'll be cherrying picking all the cleanups and fixes out of all the > various patch series over the next couple days, so that's in effect > what will happen... If I did it right, the rmap/reflink content should be contained within their respective patches. Let me know if you find mistakes, though. --D > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From st101368@stud.uni-stuttgart.de Mon Jan 4 18:23:52 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: ** X-Spam-Status: No, score=2.1 required=5.0 tests=FREEMAIL_FORGED_REPLYTO, FSL_MISSP_REPLYTO,LOTS_OF_MONEY,MONEY_FRAUD_3 autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C168D7F37 for ; Mon, 4 Jan 2016 18:23:52 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id A2AC88F8037 for ; Mon, 4 Jan 2016 16:23:52 -0800 (PST) X-ASG-Debug-ID: 1451953422-04bdf06f7403e00001-NocioJ Received: from mx4.rus.uni-stuttgart.de (mx4.rus.uni-stuttgart.de [129.69.192.4]) by cuda.sgi.com with ESMTP id uORQrStdCnzcdXu3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 16:23:43 -0800 (PST) X-Barracuda-Envelope-From: st101368@stud.uni-stuttgart.de X-Barracuda-Apparent-Source-IP: 129.69.192.4 Received: from localhost (localhost [127.0.0.1]) by mx4.rus.uni-stuttgart.de (Postfix) with ESMTP id 1B0E63C7; Tue, 5 Jan 2016 01:23:42 +0100 (CET) X-Virus-Scanned: USTUTT mailrelay AV services at mx4.rus.uni-stuttgart.de Received: from mx4.rus.uni-stuttgart.de ([127.0.0.1]) by localhost (mx4.rus.uni-stuttgart.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ek-Jnw6WeMfM; Tue, 5 Jan 2016 01:23:40 +0100 (CET) Received: from DESKTOP-R0UORM3.rus.uni-stuttgart.de (s-8d3a1191.off.site.uni-stuttgart.de [141.58.17.145]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx4.rus.uni-stuttgart.de (Postfix) with ESMTPS; Tue, 5 Jan 2016 01:23:31 +0100 (CET) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body Subject: Charity/Donation To: you X-ASG-Orig-Subj: Charity/Donation From: "Jeffrey Skoll" Date: Tue, 05 Jan 2016 02:23:32 +0200 Reply-To: jeffrey_skoll@gmx.com Message-Id: <20160105002342.1B0E63C7@mx4.rus.uni-stuttgart.de> X-Barracuda-Connect: mx4.rus.uni-stuttgart.de[129.69.192.4] X-Barracuda-Start-Time: 1451953423 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=ADVANCE_FEE_1, BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25837 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 ADVANCE_FEE_1 Appears to be advance fee fraud (Nigerian 419) Hi My name is Jeffrey Skoll, a philanthropist and the founder of one of the la= rgest private foundations in the world. I believe strongly in =E2=80=98givi= ng while living.=E2=80=99 I had one idea that never changed in my mind =E2= =80=94 that you should use your wealth to help people and I have decided to= secretly give USD2.498 Million to a randomly selected individual. On recei= pt of this email, you should count yourself as the individual. Your email a= ddress was chosen online while searching at random. Kindly get back to me a= t your earliest convenience, so I know your email address is valid. Visit the web page to know more about me: http://www.theglobeandmail.com/ne= ws/national/meet-the-canadian-billionaire-whos-giving-it-all-away/article42= 09888/ or you can read an article of me on Wikipedia. Regards, Jeffrey Skoll. From darrick.wong@oracle.com Mon Jan 4 19:14:28 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id CB8897F37 for ; Mon, 4 Jan 2016 19:14:28 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id AA80130405F for ; Mon, 4 Jan 2016 17:14:25 -0800 (PST) X-ASG-Debug-ID: 1451956463-04cbb0451604c30001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id MVUL3gZjLMa27jzc (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 17:14:24 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u051DxCl005283 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 01:14:00 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u051DxZp032488 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 01:13:59 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u051DxZx004254; Tue, 5 Jan 2016 01:13:59 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 17:13:59 -0800 Date: Mon, 4 Jan 2016 17:13:58 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com, Dave Chinner Subject: Re: [PATCH 57/76] xfs: allocate delayed extents in CoW fork Message-ID: <20160105011358.GH28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 57/76] xfs: allocate delayed extents in CoW fork References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090248.12713.32309.stgit@birch.djwong.org> <20160103122058.GE28429@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160103122058.GE28429@infradead.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451956464 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25838 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sun, Jan 03, 2016 at 04:20:58AM -0800, Christoph Hellwig wrote: > I really don't like the separate XFS_IOEND_COW flag. This should > be an ioend type just like XFS_IO_DELALLOC, XFS_IO_UNWRITTEN and > XFS_IO_OVERWRITE. We might need some work in the direct I/O > completions for that to really work, though. The COW completion is an ioend flag (instead of an ioend type) *because* of how I thought directio completions work. :) It looked to me like you could only have one ioend for a dio, and that there wasn't a clean way to make XFS have multiple ioends for a particular dio. Since the remap operation remaps everything it finds in the COW fork for a given part of a file and leaves the rest alone, it seemed natural to encode COW as a flag in the ioend, leave the ioend type as XFS_IO_OVERWRITE, and try to have as few ioends as possible. For dio writes I can COW write almost as fast as a regular overwrite. I further observed with the RFCv3 COW (which did have a separate ioend type for COW) that random writes had a tendency to create a /lot/ of ioends. If you let all these writes accumulate and then flushed them out with a single fsync, the scatteredness of the COW blocks caused the creation of a lot of ioends, which then caused a thundering herd when all those ioends had to be pushed separately through the workqueue. Changing the ioend handling to the way it is in RFCv4 greatly reduced the number of ioends that have to be processed, and dramatically improved write speeds. (I also fixed a bug where it would create ioends for each COW block in a streaming write; both fixes make v4's COW 5x faster on a ramdisk than v3's.) Did I overlook some subtlety with ioends? So far it's worked fine as-is. --D > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Mon Jan 4 19:23:21 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 08A297F37 for ; Mon, 4 Jan 2016 19:23:21 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 96B2DAC002 for ; Mon, 4 Jan 2016 17:23:17 -0800 (PST) X-ASG-Debug-ID: 1451956995-04bdf06f7405590001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id yQFFGXvo1FXwqs8J (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 17:23:15 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u051NC1J012947 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 01:23:12 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u051NBou001911 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 01:23:11 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u051NA2d007513; Tue, 5 Jan 2016 01:23:11 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 17:23:10 -0800 Date: Mon, 4 Jan 2016 17:23:09 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 1/3] xfs: pass inode instead of file to xfs_reflink_dirty_range Message-ID: <20160105012309.GI28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 1/3] xfs: pass inode instead of file to xfs_reflink_dirty_range References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-2-git-send-email-hch@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1451822873-12969-2-git-send-email-hch@lst.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0022.oracle.com [141.146.126.234] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451956995 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25839 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sun, Jan 03, 2016 at 01:07:51PM +0100, Christoph Hellwig wrote: > We don't actually need a file for write_begin/end, those can be passed > as NULL for disk based file systems. This is important as we won't I wasn't aware that one could /do/ that. :) Looks good, though. I think I can clean the filp arguments out of the reflink code entirely, too. --D > even have a file pointer during a truncate operation, which gives a > guaranteed NULL pointer dererference with the current code. > > Signed-off-by: Christoph Hellwig > --- > fs/xfs/xfs_reflink.c | 18 +++++++++--------- > 1 file changed, 9 insertions(+), 9 deletions(-) > > diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c > index da4a715..4a3f0ee 100644 > --- a/fs/xfs/xfs_reflink.c > +++ b/fs/xfs/xfs_reflink.c > @@ -1316,7 +1316,7 @@ out_error: > */ > STATIC int > xfs_reflink_dirty_range( > - struct file *filp, > + struct inode *inode, > xfs_off_t pos, > xfs_off_t len) > { > @@ -1330,14 +1330,14 @@ xfs_reflink_dirty_range( > unsigned long bytes; /* Bytes to write to page */ > void *fsdata; > > - mapping = filp->f_mapping; > + mapping = inode->i_mapping; > a_ops = mapping->a_ops; > flags = AOP_FLAG_UNINTERRUPTIBLE; > do { > > offset = (pos & (PAGE_CACHE_SIZE - 1)); > bytes = min_t(unsigned long, len, PAGE_CACHE_SIZE) - offset; > - rpage = xfs_get_page(file_inode(filp), pos); > + rpage = xfs_get_page(inode, pos); > if (IS_ERR(rpage)) { > error = PTR_ERR(rpage); > break; > @@ -1346,24 +1346,24 @@ xfs_reflink_dirty_range( > break; > } > > - error = a_ops->write_begin(filp, mapping, pos, bytes, flags, > + error = a_ops->write_begin(NULL, mapping, pos, bytes, flags, > &page, &fsdata); > page_cache_release(rpage); > if (error < 0) > break; > > - trace_xfs_reflink_unshare_page(file_inode(filp), page, > + trace_xfs_reflink_unshare_page(inode, page, > pos, bytes); > > if (!PageUptodate(page)) { > - pr_err("%s: STALE? ino=%lu pos=%llu\n", > - __func__, filp->f_inode->i_ino, pos); > + pr_err("%s: STALE? ino=%llu pos=%llu\n", > + __func__, XFS_I(inode)->i_ino, pos); > WARN_ON(1); > } > if (mapping_writably_mapped(mapping)) > flush_dcache_page(page); > > - error = a_ops->write_end(filp, mapping, pos, bytes, bytes, > + error = a_ops->write_end(NULL, mapping, pos, bytes, bytes, > page, fsdata); > if (error < 0) > break; > @@ -1454,7 +1454,7 @@ xfs_reflink_dirty_extents( > flen = XFS_FSB_TO_B(mp, rlen); > if (fpos + flen > isize) > flen = isize - fpos; > - error = xfs_reflink_dirty_range(filp, fpos, flen); > + error = xfs_reflink_dirty_range(VFS_I(ip), fpos, flen); > xfs_ilock(ip, XFS_ILOCK_EXCL); > if (error) > goto out; > -- > 1.9.1 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Mon Jan 4 19:28:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2D3BB7F37 for ; Mon, 4 Jan 2016 19:28:41 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0A020304062 for ; Mon, 4 Jan 2016 17:28:40 -0800 (PST) X-ASG-Debug-ID: 1451957318-04cb6c20ff047d0001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id DV3Kh6StlkHTxCT6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 17:28:38 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u051SaAI017378 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 01:28:36 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u051SZ1f001174 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 01:28:36 GMT Received: from abhmp0008.oracle.com (abhmp0008.oracle.com [141.146.116.14]) by userv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u051SZ8Y005129; Tue, 5 Jan 2016 01:28:35 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 17:28:33 -0800 Date: Mon, 4 Jan 2016 17:28:32 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 2/3] xfs: only end a COW operation in xfs_zero_remaining_bytes if we started one Message-ID: <20160105012832.GJ28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 2/3] xfs: only end a COW operation in xfs_zero_remaining_bytes if we started one References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-3-git-send-email-hch@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1451822873-12969-3-git-send-email-hch@lst.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0022.oracle.com [156.151.31.74] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451957318 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25838 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sun, Jan 03, 2016 at 01:07:52PM +0100, Christoph Hellwig wrote: > Without these we can see NULL pointer dereferences due to a non-existing > COW fork during xfstests runs. > > Signed-off-by: Christoph Hellwig > --- > fs/xfs/xfs_bmap_util.c | 12 +++++++----- > 1 file changed, 7 insertions(+), 5 deletions(-) > > diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c > index 7bee3c7..e777095 100644 > --- a/fs/xfs/xfs_bmap_util.c > +++ b/fs/xfs/xfs_bmap_util.c > @@ -1226,13 +1226,15 @@ xfs_zero_remaining_bytes( > > error = xfs_bwrite(bp); > xfs_buf_relse(bp); > - if (error) { > - err2 = xfs_reflink_end_cow_failed(ip, offset, > + if (should_fork) { > + if (error) { > + err2 = xfs_reflink_end_cow_failed(ip, offset, > + lastoffset - offset + 1); > + return error; > + } > + error = xfs_reflink_end_cow(ip, offset, > lastoffset - offset + 1); > - return error; > } Good catch! There's a buffer leak in the error case for xfs_map_cow_blocks() so I'll fold them both into the original patch. --D > - error = xfs_reflink_end_cow(ip, offset, > - lastoffset - offset + 1); > if (error) > return error; > } > -- > 1.9.1 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Mon Jan 4 19:43:19 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C5B3B7F37 for ; Mon, 4 Jan 2016 19:43:19 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id A55638F8033 for ; Mon, 4 Jan 2016 17:43:16 -0800 (PST) X-ASG-Debug-ID: 1451958194-04cbb0451605350001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id CqOpH4l23fcFHCL4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 17:43:14 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u051hCtu029107 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 01:43:12 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u051hBQh005813 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 5 Jan 2016 01:43:12 GMT Received: from abhmp0019.oracle.com (abhmp0019.oracle.com [141.146.116.25]) by userv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u051hBd9012226; Tue, 5 Jan 2016 01:43:11 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 Jan 2016 17:43:11 -0800 Date: Mon, 4 Jan 2016 17:43:10 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160105014310.GK28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1451822873-12969-4-git-send-email-hch@lst.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1451958194 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25839 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sun, Jan 03, 2016 at 01:07:53PM +0100, Christoph Hellwig wrote: > Otherwise we leak COW allocations done earlier in writepage. This > can be reproduced fairly easily when we hit the non-blocking writeback > EAGAIN case. > > Signed-off-by: Christoph Hellwig > --- > fs/xfs/xfs_aops.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c > index 185415a..9c69dc3 100644 > --- a/fs/xfs/xfs_aops.c > +++ b/fs/xfs/xfs_aops.c > @@ -588,6 +588,7 @@ xfs_cancel_ioend( > { > xfs_ioend_t *next; > struct buffer_head *bh, *next_bh; > + int error; > > do { > next = ioend->io_list; > @@ -605,6 +606,12 @@ xfs_cancel_ioend( > unlock_buffer(bh); > } while ((bh = next_bh) != NULL); > > + if (ioend->io_flags & XFS_IOEND_COW) { > + error = xfs_reflink_end_cow_failed( > + XFS_I(ioend->io_inode), > + ioend->io_offset, ioend->io_size); > + WARN_ON_ONCE(error); > + } Hmm. This might be the cause of the occasional complaints I've been seeing where allocated blocks remain in the COW fork when the inode is being cleared out. That said, the xfs_reflink_end_cow_failed() is apparently missing a xfs_bunmapi_cow() to actually clean out the COW fork. Good catch, in any case. Thank you for the testing and patches! :) --D > mempool_free(ioend, xfs_ioend_pool); > } while ((ioend = next) != NULL); > } > -- > 1.9.1 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From david@fromorbit.com Mon Jan 4 20:36:14 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id C50207F37 for ; Mon, 4 Jan 2016 20:36:14 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id B3C50304053 for ; Mon, 4 Jan 2016 18:36:11 -0800 (PST) X-ASG-Debug-ID: 1451961365-04cb6c20ff05710001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id HpiiT7brBTguEz9r for ; Mon, 04 Jan 2016 18:36:06 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DbCQDvKotWPGu7LXleKAECgw+BP4Jig36BeaBNAQEBBotYhTmEBYYJBAICgRxNAQEBAQEBBwEBAQFBP4Q1AQEEOhwjEAgDGAklDwUlAwcaE4guwUABAQgCIRmFdYVHiTwFh16PKI1JjwOOPoMugVQqNIUQAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 13:06:04 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGHTf-0003KF-N2; Tue, 05 Jan 2016 13:36:03 +1100 Date: Tue, 5 Jan 2016 13:36:03 +1100 From: Dave Chinner To: "Darrick J. Wong" Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [PATCH 15/22] xfsdocs: document the operation of the realtime device and inodes Message-ID: <20160105023603.GH19802@dastard> X-ASG-Orig-Subj: Re: [PATCH 15/22] xfsdocs: document the operation of the realtime device and inodes References: <20151219091208.15558.65013.stgit@birch.djwong.org> <20151219091351.15558.51417.stgit@birch.djwong.org> <20160103134151.GB9144@infradead.org> <20160105001450.GF28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105001450.GF28330@birch.djwong.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451961365 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25839 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 04:14:50PM -0800, Darrick J. Wong wrote: > On Sun, Jan 03, 2016 at 05:41:51AM -0800, Christoph Hellwig wrote: > > Looks generally good, although I wonder the take on the realtime device > > is a bit too positive given that we generally advise people not to use > > it. > > Last time I tried formatting with a rt device (a month ago) the kernel crashed, > as soon as I created an inode, so I guess we ought to warn people that this > feature is severely bitrotted. > > What /is/ the official status of XFS RT? Most sane distros (looking right at > you, Ubuntu) disable kernel support, so I guess: > > ------------------------------------------------------------------------ > WARNING: The realtime feature is not well maintained. Use with caution. > ------------------------------------------------------------------------ It gets quite a long way through xfstests here before there's an assert failure in the rt summary locking code. I'm planning on fixing that when I get a chance and then running xfstests on it regularly. I also have patches for xfstests that allow the config section code to use external devices, which makes it somewhat easier to have a "forget+test" config.... Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Mon Jan 4 20:41:34 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id CC9F97F37 for ; Mon, 4 Jan 2016 20:41:34 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 76B2BAC006 for ; Mon, 4 Jan 2016 18:41:34 -0800 (PST) X-ASG-Debug-ID: 1451961691-04bdf06f7307360001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 3KRWu95HnivmK0v4 for ; Mon, 04 Jan 2016 18:41:32 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DbCQAkLItWPGu7LXleKAECgw+BP4Jig36BeaBOAQEBBotYhTmEBYYJBAICgRxNAQEBAQEBBwEBAQFBP4Q1AQEEOhwjEAgDGAklDwUlAwcaE4guwUEBLBmFdYVHiTwFlwaNSYFljR5EigeDc4J1HIFxKjSFEAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 13:11:31 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGHYw-0003Kq-D1; Tue, 05 Jan 2016 13:41:30 +1100 Date: Tue, 5 Jan 2016 13:41:30 +1100 From: Dave Chinner To: "Darrick J. Wong" Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [PATCH 34/76] xfs: implement XFS_IOC_SWAPEXT when rmap btree is enabled Message-ID: <20160105024130.GI19802@dastard> X-ASG-Orig-Subj: Re: [PATCH 34/76] xfs: implement XFS_IOC_SWAPEXT when rmap btree is enabled References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090008.12713.25755.stgit@birch.djwong.org> <20160103121728.GC28429@infradead.org> <20160104234033.GD28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104234033.GD28330@birch.djwong.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451961691 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25840 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 03:40:33PM -0800, Darrick J. Wong wrote: > On Sun, Jan 03, 2016 at 04:17:28AM -0800, Christoph Hellwig wrote: > > On Sat, Dec 19, 2015 at 01:00:08AM -0800, Darrick J. Wong wrote: > > > Implement extent swapping when reverse-mapping is enabled. > > > > Can you just fold this into the previous patch? There seem to be a lot > > of patches in the series that would benefit from merging, but this > > is the one that really screams for it :) > > The only reason I'm keeping Dave's original rmap patches separate is because > the last time I checked with Dave, he said that he'd made a few bugfixes to his > patchset that hadn't yet been sent out, so I'd like to merge those fixes into > the original patches before starting any combining work. I haven't had a chance to sort any of those out fully, so it's probably best if you combine all your patches to minimise the separation and I'll send delta patches (if still relevant) as I work through it all in the next couple of weeks... Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Mon Jan 4 21:51:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 106A27F37 for ; Mon, 4 Jan 2016 21:51:57 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id D8A5730405F for ; Mon, 4 Jan 2016 19:51:53 -0800 (PST) X-ASG-Debug-ID: 1451965909-04bdf06f7209430001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id dva69V7iJ7FNVUbB for ; Mon, 04 Jan 2016 19:51:50 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DfCQCLPItWPGu7LXleKAECgw+BP4Jig36BeaBOAQEBBoFliXOFOYQFhgkEAgKBHU0BAQEBAQEHAQEBAUE/hDQBAQEDAScTHCMFCwgDGAklDwUlAwcaExmIDgfBRgEBCAIhGYV1hUeJPAWNOYlNjUmPA44+gnUcgXEqNIUQAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 14:21:48 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGIex-0003RM-Nb; Tue, 05 Jan 2016 14:51:47 +1100 Date: Tue, 5 Jan 2016 14:51:47 +1100 From: Dave Chinner To: Dan Williams Cc: xfs@oss.sgi.com, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Jens Axboe , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler Subject: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life Message-ID: <20160105035147.GJ19802@dastard> X-ASG-Orig-Subj: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451965909 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25842 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 10:20:05AM -0800, Dan Williams wrote: > Historically we have waited for filesystem specific heuristics to > attempt to guess when a block device is gone. Sometimes this works, but > in other cases the system can hang waiting for the fs to trigger its > shutdown protocol. > > The initial motivation for this investigation was to prevent DAX > mappings (direct mmap access to persistent memory) from leaking past the > lifetime of the hosting block device. However, Dave points out that > these shutdown operations are needed in other scenarios. Quoting Dave: > > For example, if we detect a free space corruption during allocation, > it is not safe to trust *any active mapping* because we can't trust > that we having handed out the same block to multiple owners. Hence > on such a filesystem shutdown, we have to prevent any new DAX > mapping from occurring and invalidate all existing mappings as we > cannot allow userspace to modify any data or metadata until we've > resolved the corruption situation. > > The current block device shutdown sequence of del_gendisk + > blk_cleanup_queue is problematic. We want to tell the fs after > blk_cleanup_queue that there is no possibility of recovery, but by that > time we have deleted partitions and lost the ability to find all the > super-blocks on a block device. > > Introduce del_gendisk_queue to trigger ->quiesce() and ->bdi_gone() I don't see anything that introduces a ->quiesce() method. > notifications to all the filesystems hosted on the disk. Where > ->quiesce() are 'shutdown' operations while the bdev may still be alive, So you are saying "quiesce == invalidation", which is in conflict with what we typically think quiesce means. i.e. "Quiesce" is what we do when doing orderly writeback of all outstanding dirty objects in a filesystem - what we do during freeze, remount-ro, and unmount. e.g. See the functions xfs_log_quiesce() and xfs_attr_quiesce() > and ->bdi_gone() is a set of actions to take after the backing device > is known to be permanently dead. In which case, bdi_gone == shutdown. Operation methods should be named after what they do, not what their calling context is. i.e. these are "invalidate" and "shutdown" superblock operations, not "quiesce" and "bdi_gone". > generic_quiesce_super and generic_bdi_gone, are the default operations > when a filesystem does not implement ->quiesce(), ->bdi_gone(). They > invalidate inodes and unmap DAX-inodes respectively. For now only > ->bdi_gone() has an associated super operation as xfs will implement > ->bdi_gone() in a later patch. I don't quite understand what the point of factoring __invalidate_device() like this is - it's not used by anyone, so it seems completely unnecessary to me. And really, that points out that there are multiple changes in this patch set that should be done separately. The rework of del_gendisk() into del_gendisk_start/del_gendisk_end should be the first patch. The del_gendisk/blk_cleanup_queue to blk_cleanup_queue() combining should be the second part, as it builds on the factoring of del_gendisk(). Then, if we really need to keep it, the factoring to introduce generic_quiesce_super. And finally, the patch that allows the shutdown callout can be introduced. [snip] > > +static void generic_bdi_gone(struct super_block *sb) > +{ > + struct inode *inode, *_inode = NULL; > + > + spin_lock(&sb->s_inode_list_lock); > + list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { > + spin_lock(&inode->i_lock); > + if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) > + || !IS_DAX(inode)) { > + spin_unlock(&inode->i_lock); > + continue; > + } > + __iget(inode); > + spin_unlock(&inode->i_lock); > + spin_unlock(&sb->s_inode_list_lock); > + > + unmap_mapping_range(inode->i_mapping, 0, 0, 1); > + iput(_inode); > + _inode = inode; > + cond_resched(); > + > + spin_lock(&sb->s_inode_list_lock); > + } > + spin_unlock(&sb->s_inode_list_lock); > + iput(_inode); > +} This belongs in fs/inode.c, right next to invalidate_inodes(), and with a name that describes what it does, not the context in which it is called. e.g. unmap_dax_inodes(). > +void shutdown_partition(struct gendisk *disk, int partno) > +{ > + struct block_device *bdev = bdget_disk(disk, partno); > + struct super_block *sb = get_super(bdev); > + > + if (!bdev) > + return; Null pointer deref. Certainly a landmine waiting for someone to tread on. > + if (!sb) { > + bdput(bdev); > + return; > + } > + > + if (sb->s_op->bdi_gone) > + sb->s_op->bdi_gone(sb); > + else > + generic_bdi_gone(sb); if (sb->s_op->shutdown) sb->s_op->shutdown(sb); else unmap_dax_inodes(sb); name things correctly, and the code documents itself. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Mon Jan 4 22:04:31 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0349F7F37 for ; Mon, 4 Jan 2016 22:04:31 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id E6AA68F8035 for ; Mon, 4 Jan 2016 20:04:27 -0800 (PST) X-ASG-Debug-ID: 1451966664-04cbb0451608a40001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 4F4sOwIbdm0ZxWqw for ; Mon, 04 Jan 2016 20:04:25 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BDCgAEQItWPGu7LXleKAECgw+BP4Jig36BeaBOAQEBBoFliXOFOYQFhgkCAgEBAoEcTQEBAQEBAQcBAQEBQT+ENAEBAQMBJxMcIQIFCwgDGAklDwUlAwcaE4gnB8E5AQEBBwIBIBmFdYVHhCSFGAEEkwqDfI1JjwOOPoJ1HIFxKjQBg0SBSwEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 14:33:35 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGIqM-0003Sn-80; Tue, 05 Jan 2016 15:03:34 +1100 Date: Tue, 5 Jan 2016 15:03:34 +1100 From: Dave Chinner To: Dan Williams Cc: xfs@oss.sgi.com, linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org Subject: Re: [resend PATCH 2/3] xfs: handle shutdown notifications Message-ID: <20160105040334.GK19802@dastard> X-ASG-Orig-Subj: Re: [resend PATCH 2/3] xfs: handle shutdown notifications References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182011.24118.30446.stgit@dwillia2-desk3.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104182011.24118.30446.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451966664 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25842 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 10:20:11AM -0800, Dan Williams wrote: > Force a filesystem shutdown when the backing device is known to be dead. > I.e. blk_queue_enter() permanently returns -ENODEV. > > Cc: xfs@oss.sgi.com > Suggested-by: Dave Chinner > Signed-off-by: Dan Williams > --- > fs/block_dev.c | 3 ++- > fs/xfs/xfs_super.c | 9 +++++++++ > include/linux/fs.h | 2 ++ > 3 files changed, 13 insertions(+), 1 deletion(-) > > diff --git a/fs/block_dev.c b/fs/block_dev.c > index 739e43a37e64..7d6c66148948 100644 > --- a/fs/block_dev.c > +++ b/fs/block_dev.c > @@ -1806,7 +1806,7 @@ int __invalidate_device(struct block_device *bdev, bool kill_dirty) > } > EXPORT_SYMBOL(__invalidate_device); > > -static void generic_bdi_gone(struct super_block *sb) > +void generic_bdi_gone(struct super_block *sb) > { > struct inode *inode, *_inode = NULL; > > @@ -1832,6 +1832,7 @@ static void generic_bdi_gone(struct super_block *sb) > spin_unlock(&sb->s_inode_list_lock); > iput(_inode); > } > +EXPORT_SYMBOL(generic_bdi_gone); That should be in the previous patch. > void shutdown_partition(struct gendisk *disk, int partno) > { > diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c > index 36bd8825bfb0..63c36508e9db 100644 > --- a/fs/xfs/xfs_super.c > +++ b/fs/xfs/xfs_super.c > @@ -1618,6 +1618,14 @@ xfs_fs_free_cached_objects( > return xfs_reclaim_inodes_nr(XFS_M(sb), sc->nr_to_scan); > } > > +static void > +xfs_fs_bdi_gone( xfs_fs_shutdown > + struct super_block *sb) > +{ > + xfs_force_shutdown(XFS_M(sb), SHUTDOWN_DEVICE_REQ); > + generic_bdi_gone(sb); > +} > + This is wrong. we have to unmap the DAX inodes during *every* shutdown that XFS executes. Hence it needs to be done inside xfs_do_force_shutdown(), not in addition to the shutdown when the block device is pulled. i.e. something like this in xfs_do_force_shutdown(): */ if (xfs_log_force_umount(mp, logerror)) return; + + /* + * If DAX is in use, we have to unmap all direct access + * virtual mappings to ensure nothing more gets written + * directly from userspace. This will force them to refault + * and that will result in them detecting the shutdown + * condition and hence will fail appropriately. + */ + unmap_dax_inodes(mp->m_super); if (flags & SHUTDOWN_CORRUPT_INCORE) { xfs_alert_tag(mp, XFS_PTAG_SHUTDOWN_CORRUPT > static const struct super_operations xfs_super_operations = { > .alloc_inode = xfs_fs_alloc_inode, > .destroy_inode = xfs_fs_destroy_inode, > @@ -1632,6 +1640,7 @@ static const struct super_operations xfs_super_operations = { > .show_options = xfs_fs_show_options, > .nr_cached_objects = xfs_fs_nr_cached_objects, > .free_cached_objects = xfs_fs_free_cached_objects, > + .bdi_gone = xfs_fs_bdi_gone, > }; > > static struct file_system_type xfs_fs_type = { > diff --git a/include/linux/fs.h b/include/linux/fs.h > index 0e201ed38045..b1e8e049e4b8 100644 > --- a/include/linux/fs.h > +++ b/include/linux/fs.h > @@ -2265,6 +2265,7 @@ extern struct super_block *freeze_bdev(struct block_device *); > extern void emergency_thaw_all(void); > extern int thaw_bdev(struct block_device *bdev, struct super_block *sb); > extern int fsync_bdev(struct block_device *); > +extern void generic_bdi_gone(struct super_block *sb); previous patch. -Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Mon Jan 4 22:24:06 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 3C1AD7F37 for ; Mon, 4 Jan 2016 22:24:06 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id F16EA8F8033 for ; Mon, 4 Jan 2016 20:24:05 -0800 (PST) X-ASG-Debug-ID: 1451967841-04bdf06f7209f70001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id L2HbeEDeEjBVdlAG for ; Mon, 04 Jan 2016 20:24:02 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BBCgC/RItWPGu7LXleKAECgw+BP4Jig36BeaBOAQEBBotYhTmEBYYJAgIBAQKBHE0BAQEBAQEHAQEBAUE/hDQBAQEDAScTHCMFCwgDGAklDwUlAwcaE4gnB8E8AQEIAgEgGYV1hUeJPAWHW4cShAeEEo1Jik2ENkSNeoJ1HIFxKjSFEAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 05 Jan 2016 14:53:47 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGJ9u-0003UH-Lg; Tue, 05 Jan 2016 15:23:46 +1100 Date: Tue, 5 Jan 2016 15:23:46 +1100 From: Dave Chinner To: Dan Williams Cc: xfs@oss.sgi.com, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Jens Axboe , linux-fsdevel@vger.kernel.org, Jan Kara , Tejun Heo Subject: Re: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty Message-ID: <20160105042346.GL19802@dastard> X-ASG-Orig-Subj: Re: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182016.24118.33718.stgit@dwillia2-desk3.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104182016.24118.33718.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1451967841 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25843 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 10:20:16AM -0800, Dan Williams wrote: > This warning was added as a debugging aid way back in commit > 500b067c5e6c "writeback: check for registered bdi in flusher add and > inode dirty" when we were switching over to per-bdi writeback. > > Once the block device has been torn down it's no longer useful to > complain about unregistered bdi's. Clear the writeback capability under > the the wb->list_lock(), so that __mark_inode_dirty has no opportunity > to race bdi_unregister() to this WARN() condition. > > Alternatively we could just delete the warning... The warning is correct - the filesytem is trying to mark an inode dirty on a device that can't do writeback anymore. Seems to me like it is functioning as it should. > Found this while testing block device remove from underneath an active > fs triggering traces like: > > WARNING: CPU: 6 PID: 2129 at fs/fs-writeback.c:2065 __mark_inode_dirty+0x261/0x350() > bdi-block not registered > [..] > Call Trace: > [] dump_stack+0x44/0x62 > [] warn_slowpath_common+0x82/0xc0 > [] warn_slowpath_fmt+0x5c/0x80 > [] __mark_inode_dirty+0x261/0x350 > [] generic_update_time+0x79/0xd0 > [] file_update_time+0xbd/0x110 > [] ext4_dax_fault+0x68/0x110 > [] __do_fault+0x4e/0xf0 This seems like the problem to me - you haven't implemented a shutdown hook for ext4, and so it continues to allow page faults to make progress after the device has been removed. The DAX fault should have been failed before the filesystem gets to the point of marking the inode dirty.... > + /* tell __mark_inode_dirty that writeback is no longer possible */ > + spin_lock(&wb->list_lock); > + wb->bdi->capabilities |= BDI_CAP_NO_WRITEBACK; > + spin_unlock(&wb->list_lock); > + > spin_unlock_bh(&wb->work_lock); Is that lock ordering safe? i.e. it's inside a section using bh-safe locking, which tends to imply that it can run from interrupt contexts. Can we get something like spin_lock(&wb->list_lock); ..... ..... wb_shutdown spin_lock_bh(&wb->work_lock); spin_lock(&wb->list_lock); Cheers, Dave. -- Dave Chinner david@fromorbit.com From dan.j.williams@intel.com Mon Jan 4 22:25:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 041597F37 for ; Mon, 4 Jan 2016 22:25:25 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id A3B60AC003 for ; Mon, 4 Jan 2016 20:25:21 -0800 (PST) X-ASG-Debug-ID: 1451967917-04bdf06f7409fb0001-NocioJ Received: from mail-yk0-f179.google.com (mail-yk0-f179.google.com [209.85.160.179]) by cuda.sgi.com with ESMTP id QEdUagY0tj1HkEaA (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 04 Jan 2016 20:25:17 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.160.179 Received: by mail-yk0-f179.google.com with SMTP id a85so192834783ykb.1 for ; Mon, 04 Jan 2016 20:25:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7p/HBZvEy8CWod6Q2IraXMfZfQTDiWDNp59v2ODJZ1s=; b=15sc3HL2fl6B/ukg/Wfxm6GByNcz5LInbqS8+a3FbenWv+XJNLcb1peEURntOfX05B zf9P3dAK3ewmn498APuv0sdnZ7qgH5qL7xifWvfgXbqjzPDjhU+88ioxA7uYhe4ETt/A VY48+AwrLydA9OfrGQLcBFUexm/bFgYUyUR9yFWy7s88QV1kD/ifPum9kon/DAdj7kJl BO6In7BlX1NDfrJer2PLxrWMWDMdYB10FO0yWltkVzw8qD38QmvD6qNiS4vjEJiaitBR kkHBtNvrBR/XK5Czcx2ghRgym4mQK4TmTdfPiSbaNjn5Qh+6oKH26jCdpAEos7+ylKnR 0oyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=7p/HBZvEy8CWod6Q2IraXMfZfQTDiWDNp59v2ODJZ1s=; b=aZz69ml0sJt0sdgc5QIzA3t/zCMJZXa6WO0WYan038ouBG+xaeJ5bfkSq0UreB75Ma 7bTOOmw4zeQl5fLmOobOiim4yT5RrQ8QQwEZ4R6tLmHi22BlSm49xAwROi5Bli3ClF0t Q+cyeCCbfgesxD7OfImp9Ti/X4NLUM1VVMDepaboD1OxSeEPGqvshtI/OBoUkV/n2cym s+vxlVF48ZI47sQEA6WpjGQK2zmoQa+kLVw50M3i0SwXmJVAIxRqadwgcgQ7TXnyCbZs 14S5se1fU/I7LgStbzfQD5sR6mW3fhveCromfbavmBtGgJLtMxs1u0nIdpbK1+MjfGUZ 5NpQ== X-Gm-Message-State: ALoCoQlASb5z/OZM90XMpOq14wTWSraeCmktJEtZvlKn76K3yfOvg9piIfi7CGZF9rWUDWvY99iaQIpy0cWZswkPZFq8q1KM0BaN03SBHWsoRNqLNnqMfP8= MIME-Version: 1.0 X-Received: by 10.13.228.134 with SMTP id n128mr54371826ywe.287.1451967916702; Mon, 04 Jan 2016 20:25:16 -0800 (PST) Received: by 10.37.202.20 with HTTP; Mon, 4 Jan 2016 20:25:16 -0800 (PST) In-Reply-To: <20160105035147.GJ19802@dastard> References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> <20160105035147.GJ19802@dastard> Date: Mon, 4 Jan 2016 20:25:16 -0800 Message-ID: Subject: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life From: Dan Williams X-ASG-Orig-Subj: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life To: Dave Chinner Cc: XFS Developers , linux-block@vger.kernel.org, "linux-nvdimm@lists.01.org" , Jens Axboe , Jan Kara , linux-fsdevel , Matthew Wilcox , Ross Zwisler Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-yk0-f179.google.com[209.85.160.179] X-Barracuda-Start-Time: 1451967917 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25843 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Mon, Jan 4, 2016 at 7:51 PM, Dave Chinner wrote: > On Mon, Jan 04, 2016 at 10:20:05AM -0800, Dan Williams wrote: >> Historically we have waited for filesystem specific heuristics to >> attempt to guess when a block device is gone. Sometimes this works, but >> in other cases the system can hang waiting for the fs to trigger its >> shutdown protocol. >> >> The initial motivation for this investigation was to prevent DAX >> mappings (direct mmap access to persistent memory) from leaking past the >> lifetime of the hosting block device. However, Dave points out that >> these shutdown operations are needed in other scenarios. Quoting Dave: >> >> For example, if we detect a free space corruption during allocation, >> it is not safe to trust *any active mapping* because we can't trust >> that we having handed out the same block to multiple owners. Hence >> on such a filesystem shutdown, we have to prevent any new DAX >> mapping from occurring and invalidate all existing mappings as we >> cannot allow userspace to modify any data or metadata until we've >> resolved the corruption situation. >> >> The current block device shutdown sequence of del_gendisk + >> blk_cleanup_queue is problematic. We want to tell the fs after >> blk_cleanup_queue that there is no possibility of recovery, but by that >> time we have deleted partitions and lost the ability to find all the >> super-blocks on a block device. >> >> Introduce del_gendisk_queue to trigger ->quiesce() and ->bdi_gone() > > I don't see anything that introduces a ->quiesce() method. > Right, we only have generic_quiesce_super() as no fs had a need to do anything but the generic actions. >> notifications to all the filesystems hosted on the disk. Where >> ->quiesce() are 'shutdown' operations while the bdev may still be alive, > > So you are saying "quiesce == invalidation", which is in conflict > with what we typically think quiesce means. i.e. "Quiesce" is what > we do when doing orderly writeback of all outstanding dirty objects > in a filesystem - what we do during freeze, remount-ro, and unmount. > e.g. See the functions xfs_log_quiesce() and xfs_attr_quiesce() Fair enough, I should have called this one invalidate_super. > >> and ->bdi_gone() is a set of actions to take after the backing device >> is known to be permanently dead. > > In which case, bdi_gone == shutdown. > True, and following this logic I think the existing generic_shutdown_super() should be renamed generic_kill_super() to match the fs actions, but see below... > Operation methods should be named after what they do, not what their > calling context is. i.e. these are "invalidate" and "shutdown" > superblock operations, not "quiesce" and "bdi_gone". I was running out of colors to paint the bike shed given generic_shutdown_super() was already in use. Also, since generic_shutdown_super() has had its current meaning since forever I didn't think it was worth the risk to change the meaning of such a long standing symbol. Other ideas, generic_stop_super? >> generic_quiesce_super and generic_bdi_gone, are the default operations >> when a filesystem does not implement ->quiesce(), ->bdi_gone(). They >> invalidate inodes and unmap DAX-inodes respectively. For now only >> ->bdi_gone() has an associated super operation as xfs will implement >> ->bdi_gone() in a later patch. > > I don't quite understand what the point of factoring > __invalidate_device() like this is - it's not used by anyone, so it > seems completely unnecessary to me. > You're right, without a need for an fs to intercept the 'invalidation' event there's no need to do this refactor. > And really, that points out that there are multiple changes in this > patch set that should be done separately. The rework of > del_gendisk() into del_gendisk_start/del_gendisk_end should be the > first patch. The del_gendisk/blk_cleanup_queue to > blk_cleanup_queue() combining should be the second part, as it > builds on the factoring of del_gendisk(). Then, if we really need to > keep it, the factoring to introduce generic_quiesce_super. And > finally, the patch that allows the shutdown callout can be > introduced. Ok, will split. > > [snip] >> >> +static void generic_bdi_gone(struct super_block *sb) >> +{ >> + struct inode *inode, *_inode = NULL; >> + >> + spin_lock(&sb->s_inode_list_lock); >> + list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { >> + spin_lock(&inode->i_lock); >> + if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) >> + || !IS_DAX(inode)) { >> + spin_unlock(&inode->i_lock); >> + continue; >> + } >> + __iget(inode); >> + spin_unlock(&inode->i_lock); >> + spin_unlock(&sb->s_inode_list_lock); >> + >> + unmap_mapping_range(inode->i_mapping, 0, 0, 1); >> + iput(_inode); >> + _inode = inode; >> + cond_resched(); >> + >> + spin_lock(&sb->s_inode_list_lock); >> + } >> + spin_unlock(&sb->s_inode_list_lock); >> + iput(_inode); >> +} > > This belongs in fs/inode.c, right next to invalidate_inodes(), and > with a name that describes what it does, not the context in which > it is called. e.g. unmap_dax_inodes(). > Sounds good. >> +void shutdown_partition(struct gendisk *disk, int partno) >> +{ >> + struct block_device *bdev = bdget_disk(disk, partno); >> + struct super_block *sb = get_super(bdev); >> + >> + if (!bdev) >> + return; > > Null pointer deref. Certainly a landmine waiting for someone to > tread on. > Nope, get_super() checks for a NULL argument. >> + if (!sb) { >> + bdput(bdev); >> + return; >> + } >> + >> + if (sb->s_op->bdi_gone) >> + sb->s_op->bdi_gone(sb); >> + else >> + generic_bdi_gone(sb); > > if (sb->s_op->shutdown) > sb->s_op->shutdown(sb); > else > unmap_dax_inodes(sb); > > name things correctly, and the code documents itself. How about 'stop' or 'halt' instead of 'shutdown' to preserve the historical meaning of generic_shutdown_super? From st101368@stud.uni-stuttgart.de Mon Jan 4 22:27:58 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: *** X-Spam-Status: No, score=3.8 required=5.0 tests=FH_FROMEML_NOTLD, FREEMAIL_FORGED_REPLYTO,FSL_MISSP_REPLYTO,LOTS_OF_MONEY,MONEY_FRAUD_3 autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2686F7F37 for ; Mon, 4 Jan 2016 22:27:58 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 16850304067 for ; Mon, 4 Jan 2016 20:27:55 -0800 (PST) X-ASG-Debug-ID: 1451968068-04cb6c20fe08720001-NocioJ Received: from mx3.rus.uni-stuttgart.de (mx3.rus.uni-stuttgart.de [129.69.192.3]) by cuda.sgi.com with ESMTP id aclFpVpLW041gm58 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 04 Jan 2016 20:27:49 -0800 (PST) X-Barracuda-Envelope-From: st101368@stud.uni-stuttgart.de X-Barracuda-Apparent-Source-IP: 129.69.192.3 Received: from localhost (localhost [127.0.0.1]) by mx3.rus.uni-stuttgart.de (Postfix) with ESMTP id E32CD21142; Tue, 5 Jan 2016 05:27:47 +0100 (CET) X-Virus-Scanned: USTUTT mailrelay AV services at mx3.rus.uni-stuttgart.de Received: from mx3.rus.uni-stuttgart.de ([127.0.0.1]) by localhost (mx3.rus.uni-stuttgart.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id onLySEFCjj8a; Tue, 5 Jan 2016 05:27:46 +0100 (CET) Received: from DESKTOP-R0UORM3.rus.uni-stuttgart.de (s-8d3a11ec.off.site.uni-stuttgart.de [141.58.17.236]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx3.rus.uni-stuttgart.de (Postfix) with ESMTPS; Tue, 5 Jan 2016 05:27:39 +0100 (CET) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body Subject: Charity/Donation To: you X-ASG-Orig-Subj: Charity/Donation From: "Jeffrey Skoll" Date: Tue, 05 Jan 2016 06:27:45 +0200 Reply-To: jeffrey_skoll@gmx.com Message-Id: <20160105042747.E32CD21142@mx3.rus.uni-stuttgart.de> X-Barracuda-Connect: mx3.rus.uni-stuttgart.de[129.69.192.3] X-Barracuda-Start-Time: 1451968068 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.50 X-Barracuda-Spam-Status: No, SCORE=0.50 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=ADVANCE_FEE_1, BSF_SC0_MISMATCH_TO, BSF_SC0_MV0713 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25842 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 ADVANCE_FEE_1 Appears to be advance fee fraud (Nigerian 419) 0.50 BSF_SC0_MV0713 Custom rule MV0713 Hi My name is Jeffrey Skoll, a philanthropist and the founder of one of the la= rgest private foundations in the world. I believe strongly in =E2=80=98givi= ng while living.=E2=80=99 I had one idea that never changed in my mind =E2= =80=94 that you should use your wealth to help people and I have decided to= secretly give USD2.498 Million to a randomly selected individual. On recei= pt of this email, you should count yourself as the individual. Your email a= ddress was chosen online while searching at random. Kindly get back to me a= t your earliest convenience, so I know your email address is valid. Visit the web page to know more about me: http://www.theglobeandmail.com/ne= ws/national/meet-the-canadian-billionaire-whos-giving-it-all-away/article42= 09888/ or you can read an article of me on Wikipedia. Regards, Jeffrey Skoll. From jack@suse.cz Tue Jan 5 03:41:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id EA6287F37 for ; Tue, 5 Jan 2016 03:41:02 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 85AAEAC007 for ; Tue, 5 Jan 2016 01:41:02 -0800 (PST) X-ASG-Debug-ID: 1451986854-04bdf06f730fbd0001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id brfsZKvUeuxGh0EK (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 05 Jan 2016 01:40:55 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 5ACDEABE5; Tue, 5 Jan 2016 09:40:51 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id 4F06E823D8; Tue, 5 Jan 2016 10:41:01 +0100 (CET) Date: Tue, 5 Jan 2016 10:41:01 +0100 From: Jan Kara To: Ross Zwisler Cc: linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com, Andrew Morton , Dan Williams , Matthew Wilcox , Dave Hansen Subject: Re: [PATCH v6 2/7] dax: support dirty DAX entries in radix tree Message-ID: <20160105094101.GB2724@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v6 2/7] dax: support dirty DAX entries in radix tree References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-3-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450899560-26708-3-git-send-email-ross.zwisler@linux.intel.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1451986855 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25848 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed 23-12-15 12:39:15, Ross Zwisler wrote: > Add support for tracking dirty DAX entries in the struct address_space > radix tree. This tree is already used for dirty page writeback, and it > already supports the use of exceptional (non struct page*) entries. > > In order to properly track dirty DAX pages we will insert new exceptional > entries into the radix tree that represent dirty DAX PTE or PMD pages. > These exceptional entries will also contain the writeback sectors for the > PTE or PMD faults that we can use at fsync/msync time. > > There are currently two types of exceptional entries (shmem and shadow) > that can be placed into the radix tree, and this adds a third. We rely on > the fact that only one type of exceptional entry can be found in a given > radix tree based on its usage. This happens for free with DAX vs shmem but > we explicitly prevent shadow entries from being added to radix trees for > DAX mappings. > > The only shadow entries that would be generated for DAX radix trees would > be to track zero page mappings that were created for holes. These pages > would receive minimal benefit from having shadow entries, and the choice > to have only one type of exceptional entry in a given radix tree makes the > logic simpler both in clear_exceptional_entry() and in the rest of DAX. > > Signed-off-by: Ross Zwisler The patch looks good to me. You can add: Reviewed-by: Jan Kara Honza -- Jan Kara SUSE Labs, CR From BATV+ace907f9388f74b0af24+4517+infradead.org+hch@bombadil.srs.infradead.org Tue Jan 5 04:42:24 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 7CA397F37 for ; Tue, 5 Jan 2016 04:42:24 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 0F069AC002 for ; Tue, 5 Jan 2016 02:42:20 -0800 (PST) X-ASG-Debug-ID: 1451990538-04cb6c20fd0f3e0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id ri65ZcTXc4ZvNazn (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 05 Jan 2016 02:42:19 -0800 (PST) X-Barracuda-Envelope-From: BATV+ace907f9388f74b0af24+4517+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aGP4A-0006IB-AN; Tue, 05 Jan 2016 10:42:14 +0000 Date: Tue, 5 Jan 2016 02:42:14 -0800 From: Christoph Hellwig To: "Darrick J. Wong" Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160105104214.GA16310@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> <20160105014310.GK28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105014310.GK28330@birch.djwong.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1451990539 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25848 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Mon, Jan 04, 2016 at 05:43:10PM -0800, Darrick J. Wong wrote: > Hmm. This might be the cause of the occasional complaints I've been seeing > where allocated blocks remain in the COW fork when the inode is being cleared > out. That said, the xfs_reflink_end_cow_failed() is apparently missing a > xfs_bunmapi_cow() to actually clean out the COW fork. I can still reproduce xfs_reflink_cancel_pending_cow tripping over allocated blocks in the COW fork over NFS. generic/154 reproduces it 100% over NFS, although when adding a delay before the cleanup it disappears. I'm currently trying to figure out why. From jack@suse.cz Tue Jan 5 05:14:02 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C75B17F37 for ; Tue, 5 Jan 2016 05:14:02 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id B888C8F8035 for ; Tue, 5 Jan 2016 03:13:59 -0800 (PST) X-ASG-Debug-ID: 1451992436-04cb6c20fd100e0001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 7apljF80DcQBvHrF (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 05 Jan 2016 03:13:57 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 6EEA9AAF4; Tue, 5 Jan 2016 11:13:54 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id CCB80823D9; Tue, 5 Jan 2016 12:13:58 +0100 (CET) Date: Tue, 5 Jan 2016 12:13:58 +0100 From: Jan Kara To: Ross Zwisler Cc: linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com, Andrew Morton , Dan Williams , Matthew Wilcox , Dave Hansen Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync Message-ID: <20160105111358.GD2724@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1451992437 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25849 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed 23-12-15 12:39:17, Ross Zwisler wrote: > To properly handle fsync/msync in an efficient way DAX needs to track dirty > pages so it is able to flush them durably to media on demand. > > The tracking of dirty pages is done via the radix tree in struct > address_space. This radix tree is already used by the page writeback > infrastructure for tracking dirty pages associated with an open file, and > it already has support for exceptional (non struct page*) entries. We > build upon these features to add exceptional entries to the radix tree for > DAX dirty PMD or PTE pages at fault time. > > Signed-off-by: Ross Zwisler ... > +static int dax_writeback_one(struct block_device *bdev, > + struct address_space *mapping, pgoff_t index, void *entry) > +{ > + struct radix_tree_root *page_tree = &mapping->page_tree; > + int type = RADIX_DAX_TYPE(entry); > + struct radix_tree_node *node; > + struct blk_dax_ctl dax; > + void **slot; > + int ret = 0; > + > + spin_lock_irq(&mapping->tree_lock); > + /* > + * Regular page slots are stabilized by the page lock even > + * without the tree itself locked. These unlocked entries > + * need verification under the tree lock. > + */ > + if (!__radix_tree_lookup(page_tree, index, &node, &slot)) > + goto unlock; > + if (*slot != entry) > + goto unlock; > + > + /* another fsync thread may have already written back this entry */ > + if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE)) > + goto unlock; > + > + radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE); > + > + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && type != RADIX_DAX_PMD)) { > + ret = -EIO; > + goto unlock; > + } > + > + dax.sector = RADIX_DAX_SECTOR(entry); > + dax.size = (type == RADIX_DAX_PMD ? PMD_SIZE : PAGE_SIZE); > + spin_unlock_irq(&mapping->tree_lock); > + > + /* > + * We cannot hold tree_lock while calling dax_map_atomic() because it > + * eventually calls cond_resched(). > + */ > + ret = dax_map_atomic(bdev, &dax); > + if (ret < 0) > + return ret; > + > + if (WARN_ON_ONCE(ret < dax.size)) { > + ret = -EIO; > + dax_unmap_atomic(bdev, &dax); > + return ret; > + } > + > + spin_lock_irq(&mapping->tree_lock); > + /* > + * We need to revalidate our radix entry while holding tree_lock > + * before we do the writeback. > + */ Do we really need to revalidate here? dax_map_atomic() makes sure the addr & size is still part of the device. I guess you are concerned that due to truncate or similar operation those sectors needn't belong to the same file anymore but we don't really care about flushing sectors for someone else, do we? Otherwise the patch looks good to me. > + if (!__radix_tree_lookup(page_tree, index, &node, &slot)) > + goto unmap; > + if (*slot != entry) > + goto unmap; > + > + wb_cache_pmem(dax.addr, dax.size); > + unmap: > + dax_unmap_atomic(bdev, &dax); > + unlock: > + spin_unlock_irq(&mapping->tree_lock); > + return ret; > +} Honza -- Jan Kara SUSE Labs, CR From jack@suse.cz Tue Jan 5 05:14:04 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 6A7E97F50 for ; Tue, 5 Jan 2016 05:14:04 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id E0CF1AC003 for ; Tue, 5 Jan 2016 03:14:00 -0800 (PST) X-ASG-Debug-ID: 1451992437-04cbb0451411130001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id uaY7Dc7bGSboeNrs (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 05 Jan 2016 03:13:58 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 53B7DABE5; Tue, 5 Jan 2016 11:13:56 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id F2115823D8; Tue, 5 Jan 2016 12:13:46 +0100 (CET) Date: Tue, 5 Jan 2016 12:13:46 +0100 From: Jan Kara To: Dan Williams Cc: Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync Message-ID: <20160105111346.GC2724@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1451992438 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25849 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Sun 03-01-16 10:13:06, Dan Williams wrote: > On Wed, Dec 23, 2015 at 11:39 AM, Ross Zwisler > wrote: > > To properly handle fsync/msync in an efficient way DAX needs to track dirty > > pages so it is able to flush them durably to media on demand. > > > > The tracking of dirty pages is done via the radix tree in struct > > address_space. This radix tree is already used by the page writeback > > infrastructure for tracking dirty pages associated with an open file, and > > it already has support for exceptional (non struct page*) entries. We > > build upon these features to add exceptional entries to the radix tree for > > DAX dirty PMD or PTE pages at fault time. > > > > Signed-off-by: Ross Zwisler > > I'm hitting the following report with the ndctl dax test [1] on > next-20151231. I bisected it to > commit 3cb108f941de "dax-add-support-for-fsync-sync-v6". I'll take a > closer look tomorrow, but in case someone can beat me to it, here's > the back-trace: > > ------------[ cut here ]------------ > kernel BUG at fs/inode.c:497! I suppose this is the check that mapping->nr_exceptional is zero, isn't it? Hum, I don't see how that could happen given we call truncate_inode_pages_final() just before the clear_inode() call which removes all the exceptional entries from the radix tree. And there's not much room for a race during umount... Does the radix tree really contain any entry or is it an accounting bug? Honza > [..] > CPU: 1 PID: 3001 Comm: umount Tainted: G O 4.4.0-rc7+ #2412 > Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > task: ffff8800da2a5a00 ti: ffff880307794000 task.ti: ffff880307794000 > RIP: 0010:[] [] clear_inode+0x71/0x80 > RSP: 0018:ffff880307797d50 EFLAGS: 00010002 > RAX: ffff8800da2a5a00 RBX: ffff8800ca2e7328 RCX: ffff8800da2a5a28 > RDX: 0000000000000001 RSI: 0000000000000005 RDI: ffff8800ca2e7530 > RBP: ffff880307797d60 R08: ffffffff82900ae0 R09: 0000000000000000 > R10: ffff8800ca2e7548 R11: 0000000000000000 R12: ffff8800ca2e7530 > R13: ffff8800ca2e7328 R14: ffff8800da2e88d0 R15: ffff8800da2e88d0 > FS: 00007f2b22f4a880(0000) GS:ffff88031fc40000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00005648abd933e8 CR3: 000000007f3fc000 CR4: 00000000000006e0 > Stack: > ffff8800ca2e7328 ffff8800ca2e7000 ffff880307797d88 ffffffffa01c18af > ffff8800ca2e7328 ffff8800ca2e74d0 ffffffffa01ec740 ffff880307797db0 > ffffffff81281038 ffff8800ca2e74c0 ffff880307797e00 ffff8800ca2e7328 > Call Trace: > [] xfs_fs_evict_inode+0x5f/0x110 [xfs] > [] evict+0xb8/0x180 > [] dispose_list+0x3b/0x50 > [] evict_inodes+0x144/0x170 > [] generic_shutdown_super+0x3f/0xf0 > [] kill_block_super+0x27/0x70 > [] deactivate_locked_super+0x43/0x70 > [] deactivate_super+0x5c/0x60 > [] cleanup_mnt+0x3f/0x90 > [] __cleanup_mnt+0x12/0x20 > [] task_work_run+0x76/0x90 > [] syscall_return_slowpath+0x20a/0x280 > [] int_ret_from_sys_call+0x25/0x9f > Code: 48 8d 93 30 03 00 00 48 39 c2 75 23 48 8b 83 d0 00 00 00 a8 20 > 74 1a a8 40 75 18 48 c7 8 > 3 d0 00 00 00 60 00 00 00 5b 41 5c 5d c3 <0f> 0b 0f 0b 0f 0b 0f 0b 0f > 0b 0f 1f 44 00 00 0f 1f > 44 00 00 55 > RIP [] clear_inode+0x71/0x80 > RSP > ---[ end trace 3b1d8898a94a4fc1 ]--- > > [1]: git://git@github.com:pmem/ndctl.git pending > make TESTS="test/dax.sh" check > -- Jan Kara SUSE Labs, CR From bfoster@redhat.com Tue Jan 5 06:42:53 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 202387F37 for ; Tue, 5 Jan 2016 06:42:53 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 00CFB8F8037 for ; Tue, 5 Jan 2016 04:42:52 -0800 (PST) X-ASG-Debug-ID: 1451997768-04cb6c210011880001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Kguwuky7ds1dRqmq (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 04:42:48 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id EB8DE264F; Tue, 5 Jan 2016 12:42:27 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05CgR0O010135; Tue, 5 Jan 2016 07:42:27 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id A17BB124F12; Tue, 5 Jan 2016 07:42:26 -0500 (EST) Date: Tue, 5 Jan 2016 07:42:26 -0500 From: Brian Foster To: "Darrick J. Wong" Cc: xfs@oss.sgi.com Subject: Re: [RFCv4 00/76] xfs: add reverse-mapping, reflink, and dedupe support Message-ID: <20160105124226.GA38749@bfoster.bfoster> X-ASG-Orig-Subj: Re: [RFCv4 00/76] xfs: add reverse-mapping, reflink, and dedupe support References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151220140254.GA3618@laptop.bfoster> <20160104235951.GE28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104235951.GE28330@birch.djwong.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1451997768 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Mon, Jan 04, 2016 at 03:59:51PM -0800, Darrick J. Wong wrote: > On Sun, Dec 20, 2015 at 09:02:54AM -0500, Brian Foster wrote: > > On Sat, Dec 19, 2015 at 12:56:23AM -0800, Darrick J. Wong wrote: > > > Hi all, > > > > > ... > > > Fixed since RFCv3: > > > > > > * The reflink and dedupe ioctls are being hoisted to the VFS, as > > > provided in the first few patches. Patch 81 connects to this > > > functionality. > > > > > > * Copy on write has been rewritten for v4. We now use the existing > > > delayed allocation mechanism to coalesce writes together, deferring > > > allocation until writeout time. This enables CoW to make better > > > block placement decisions and significantly reduces overhead. > > > CoW is still pretty slow, but not as slow as before. > > > > > > * Direct IO CoW has been implemented using the same mechanism as > > > above, but modified to perform the allocation and remapping right > > > then and there. Throughput is much higher than pushing data > > > through the page cache CoW. (It's the same mechanism, but we're > > > playing with chunks bigger than a single memory page.) > > > > > > * CoW ENOSPC works correctly now, except in the pathological case > > > that the AG fills up and the rmap btree cannot expand. That will > > > be addressed for v5. > > > > > > * fallocate will now unshare blocks to prevent future ENOSPC, as > > > you'd expect. > > > > > > * refcount btree blocks are preallocated at mount time to prevent > > > ENOSPC while trying to expand the tree. This also has the effect > > > of grouping the btree blocks together, which can speed up CoW > > > remapping. > > > > > > > Can you elaborate on how these blocks are preallocated? E.g., is the > > tree "preconstructed" in some sense? However that is done, is this the > > anticipated solution or a temporary workaround..? > > > > Also, shouldn't the enospc condition be handled by the agfl? I take it > > there is something going on here that renders that solution flawed, so > > I'm just curious what it is. > > > > (Sorry if this is all explained elsewhere, but I haven't yet had a > > chance to take a close enough look at this feature..). > > Reference count btree blocks aren't allocated from the AGFL; they're allocated > from the free space in the same manner as the inobt, per a review comment from > Dave a looong time ago. :) > Ah, Ok. > As such, we can get ourselves into the nasty situation where every block in the > AG has been allocated to file data. If we then see a bunch of reference count > changes that are scattered around the AG, the reference count btree has to > expand to hold all the new records... but there isn't space, and the operation > fails. Given that we know the maximum possible size of the refcount btree > (it's 0.3% of the AG size with 4k blocks), I figured it was easy enough to > avoid ENOSPC for reflink operations. > Sounds reasonable. > I've temporarily fixed this by adding code that figures out how many blocks we > need if the reference count btree has to have a unique record for every block > in the AG and holding that many blocks until either they're allocated to the > refcount btree or freed at umount time. Right now it's a temporary fix (if the > FS crashes, the reserved blocks are lost) but it wouldn't be difficult for the > FS to make a permanent reservation that's recorded on disk somehow. But that's > involves writing things to disk + making xfsprogs understand the reservation; > let's see what people say about the reserved pool idea at all. > > Does that make sense? :) > Yep, it sounds sort of like the reserve pool mechanism used to protect against ENOSPC when freeing blocks. Curious... why are the reserved blocks lost on fs crash? Wouldn't they be reserved again on the subsequent mount? Thanks for the explanation... Brian > --D > > > > > Brian > > > > > Issues: > > > > > > * The extent swapping ioctl still allocates a bigger fixed-size > > > transaction. That's most likely a stupid thing to do, so getting a > > > better grip on how the journalling code works and auditing all the > > > new transaction users will have to happen. Right now it mostly > > > gets lucky. > > > > > > * EFI tracking for the allocated-but-not-yet-mapped blocks is > > > nonexistant. A crash will leak them. > > > > > > * ENOSPC while expanding the rmap btree can crash the FS. For now we > > > work around this problem by making the AGFL as big as possible, > > > failing CoW attempts with ENOSPC if there aren't enough AGFL blocks > > > available, and hoping that doesn't actually happen. > > > > > > If you're going to start using this mess, you probably ought to just > > > pull from my github trees for kernel[1], xfsprogs[2], and xfstests[3]. > > > There are also updates for xfs-docs[4] and man-pages[5]. > > > > > > The patches have been xfstested with x64, i386, and ppc64; while in > > > general the tests run to completion, there are still periodic bugs > > > that will be addressed by the next RFC. There's a persistent crash on > > > arm64 and ppc64el that I haven't been able to triage. > > > > > > This is an extraordinary way to eat your data. Enjoy! > > > Comments and questions are, as always, welcome. > > > > > > --D > > > > > > [1] https://github.com/djwong/linux/tree/for-dave > > > [2] https://github.com/djwong/xfsprogs/tree/for-dave > > > [3] https://github.com/djwong/xfstests/tree/for-dave > > > [4] https://github.com/djwong/xfs-documentation/tree/for-dave > > > [5] https://github.com/djwong/man-pages/commits/for-mtk > > > > > > _______________________________________________ > > > xfs mailing list > > > xfs@oss.sgi.com > > > http://oss.sgi.com/mailman/listinfo/xfs > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com > > http://oss.sgi.com/mailman/listinfo/xfs From ross.zwisler@linux.intel.com Tue Jan 5 09:50:50 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 80E937F37 for ; Tue, 5 Jan 2016 09:50:50 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 62BBD304053 for ; Tue, 5 Jan 2016 07:50:47 -0800 (PST) X-ASG-Debug-ID: 1452009042-04cbb04515180a0001-NocioJ Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id b0jgu9gcHEt2mtzt for ; Tue, 05 Jan 2016 07:50:42 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.24 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP; 05 Jan 2016 07:50:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,525,1444719600"; d="scan'208";a="884216047" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.254.191.93]) by orsmga002.jf.intel.com with ESMTP; 05 Jan 2016 07:50:39 -0800 Date: Tue, 5 Jan 2016 08:50:39 -0700 From: Ross Zwisler To: Jan Kara Cc: Dan Williams , Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync Message-ID: <20160105155039.GA6462@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync Mail-Followup-To: Ross Zwisler , Jan Kara , Dan Williams , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> <20160105111346.GC2724@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105111346.GC2724@quack.suse.cz> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mga09.intel.com[134.134.136.24] X-Barracuda-Start-Time: 1452009042 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Jan 05, 2016 at 12:13:46PM +0100, Jan Kara wrote: > On Sun 03-01-16 10:13:06, Dan Williams wrote: > > On Wed, Dec 23, 2015 at 11:39 AM, Ross Zwisler > > wrote: > > > To properly handle fsync/msync in an efficient way DAX needs to track dirty > > > pages so it is able to flush them durably to media on demand. > > > > > > The tracking of dirty pages is done via the radix tree in struct > > > address_space. This radix tree is already used by the page writeback > > > infrastructure for tracking dirty pages associated with an open file, and > > > it already has support for exceptional (non struct page*) entries. We > > > build upon these features to add exceptional entries to the radix tree for > > > DAX dirty PMD or PTE pages at fault time. > > > > > > Signed-off-by: Ross Zwisler > > > > I'm hitting the following report with the ndctl dax test [1] on > > next-20151231. I bisected it to > > commit 3cb108f941de "dax-add-support-for-fsync-sync-v6". I'll take a > > closer look tomorrow, but in case someone can beat me to it, here's > > the back-trace: > > > > ------------[ cut here ]------------ > > kernel BUG at fs/inode.c:497! > > I suppose this is the check that mapping->nr_exceptional is zero, isn't it? > Hum, I don't see how that could happen given we call > truncate_inode_pages_final() just before the clear_inode() call which > removes all the exceptional entries from the radix tree. And there's not > much room for a race during umount... Does the radix tree really contain > any entry or is it an accounting bug? > > Honza I think this is a bug with the existing way that we handle PMD faults. The issue is that the PMD path doesn't properly remove radix tree entries for zero pages covering holes. The PMD path calls unmap_mapping_range() to unmap the range out of the struct address_space, but it is missing a call to truncate_inode_pages_range() or similar to clear out those entries in the radix tree. Up until now we didn't notice, we just had an orphaned entry in the radix tree, but with my code we then find the page entry in the radix tree when handling a PMD fault, we remove it and add in a PMD entry. This causes us to be off on both our mapping->nrpages and mapping->nrexceptional counts. In the PTE path we properly remove the pages from the radix tree when upgrading from a hole to a real DAX entry via the delete_from_page_cache() call, which eventually calls page_cache_tree_delete(). I'm working on a fix now (and making sure all the above is correct). - Ross From andrea.gelmini@gmail.com Tue Jan 5 10:31:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 513B07F37 for ; Tue, 5 Jan 2016 10:31:03 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 3CA078F804B for ; Tue, 5 Jan 2016 08:31:00 -0800 (PST) X-ASG-Debug-ID: 1452011457-04cb6c20fd16c00001-NocioJ Received: from mail-wm0-f48.google.com (mail-wm0-f48.google.com [74.125.82.48]) by cuda.sgi.com with ESMTP id ESGWF8BlqVSPGHw2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 05 Jan 2016 08:30:57 -0800 (PST) X-Barracuda-Envelope-From: andrea.gelmini@gmail.com X-Barracuda-Apparent-Source-IP: 74.125.82.48 Received: by mail-wm0-f48.google.com with SMTP id f206so37386495wmf.0 for ; Tue, 05 Jan 2016 08:30:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=h3cQMbT6L5KEMeBl6NZBNsOYLF7I8HneIvQVc33Jl/A=; b=aeQ2MZBQ/ZgWr4OYb4M57sJqJpwoYXrHwDSYylrh4UcTmWaPfNJOr6SZZwo3XiO9T2 VmAVCmFT3E5t9MA+4CllIVTd3QHw/+gCr4oMaFwmhRZBln656MYXyeKzYcqjWJhHIWQY ZUj0COG5WZrX12dRl4d4kW94P5Mkx8/hzmGkDvaHYnpSNuOi1DoZBdJ5Pl1ytwcbMcv5 PiuxAyf+AFLx87iO5oDK5HwuC8RrJMfZLKrVZhwFMYaVEpP7hF0RevLyKoZpmrFGKs7J FYZxP8DgU3wI7AQsg9OiGifxqkVfkCBWUgzGXbxpejmb5NUEZ7oRbw0ZfTEZKTqQY/MJ yvEg== X-Received: by 10.194.19.100 with SMTP id d4mr104462734wje.18.1452011456889; Tue, 05 Jan 2016 08:30:56 -0800 (PST) Received: from glen ([195.206.31.238]) by smtp.gmail.com with ESMTPSA id z127sm4341843wme.2.2016.01.05.08.30.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Jan 2016 08:30:55 -0800 (PST) Sender: Andrea Gelmini Date: Tue, 5 Jan 2016 17:30:55 +0100 From: Andrea Gelmini To: Dave Chinner Cc: linux-kernel@vger.kernel.org, xfs@oss.sgi.com Subject: Re: BUG: KASAN: use-after-free in xfs_iflush_cluster+0x9d7/0xaf0 Message-ID: <20160105163055.GA18111@glen> X-ASG-Orig-Subj: Re: BUG: KASAN: use-after-free in xfs_iflush_cluster+0x9d7/0xaf0 References: <20151214180048.GA15690@glen> <20151214195422.GM26718@dastard> <20151214201526.GA25152@glen> <20151214212220.GO26718@dastard> <20151215091145.GA19282@glen> <20160103204758.GW19802@dastard> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="uAKRQypu60I7Lcqm" Content-Disposition: inline In-Reply-To: <20160103204758.GW19802@dastard> User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mail-wm0-f48.google.com[74.125.82.48] X-Barracuda-Start-Time: 1452011457 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.60 X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0249, DKIM_SIGNED, DKIM_VERIFIED, MARKETING_SUBJECT X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25854 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.60 MARKETING_SUBJECT Subject contains popular marketing words 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 2.00 BSF_SC0_MV0249 Custom rule MV0249 --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Jan 04, 2016 at 07:47:58AM +1100, Dave Chinner wrote: > > I'm recompiling, to try it again. > > Maybe, in the meanwhile, you can do something with my files. You can find 'em here: > > http://mail.gelma.net/xfs_kasan > > Any update on this problem, Andrea? Here we are! Reproduced right now. So, just to avoid confusion: a) it's a vanilla kernel 4.4.0-rc8 b) plus some btrfs patches c) plus some dri/intel/i915 patches d) at the same URL above you can find git_files.txt.gz, where you have each commit I applied above vanilla kernel (anyway, nothing related to vfs/xfs of course) e) at the same URL you find the kernel binaries I used f) to catch it, I had to copy a few gigs of files on my /home partition (xfs over Luks) Anyway, here what you asked me for: (gdb) l *(xfs_iflush_cluster+0xb73/0xc10) 0xffffffff8184c550 is in xfs_iflush_cluster (fs/xfs/xfs_inode.c:3182). 3177 3178 STATIC int 3179 xfs_iflush_cluster( 3180 xfs_inode_t *ip, 3181 xfs_buf_t *bp) 3182 { 3183 xfs_mount_t *mp = ip->i_mount; 3184 struct xfs_perag *pag; 3185 unsigned long first_index, mask; 3186 unsigned long inodes_per_cluster; (gdb) Thanks a lot for your patience, Dave [mar gen 5 16:58:19 2016] ================================================================== [mar gen 5 16:58:19 2016] BUG: KASAN: use-after-free in xfs_iflush_cluster+0xb73/0xc10 at addr ffff880364721d10 [mar gen 5 16:58:19 2016] Read of size 4 by task xfsaild/dm-0/329 [mar gen 5 16:58:19 2016] ============================================================================= [mar gen 5 16:58:19 2016] BUG xfs_ili (Tainted: G W ): kasan: bad access detected [mar gen 5 16:58:19 2016] ----------------------------------------------------------------------------- [mar gen 5 16:58:19 2016] Disabling lock debugging due to kernel taint [mar gen 5 16:58:19 2016] INFO: Allocated in kmem_zone_alloc+0x7c/0x180 age=496908 cpu=1 pid=6496 [mar gen 5 16:58:19 2016] ___slab_alloc.constprop.27+0x383/0x490 [mar gen 5 16:58:19 2016] __slab_alloc.isra.24.constprop.26+0x50/0xa0 [mar gen 5 16:58:19 2016] kmem_cache_alloc+0x174/0x1b0 [mar gen 5 16:58:19 2016] kmem_zone_alloc+0x7c/0x180 [mar gen 5 16:58:19 2016] xfs_inode_item_init+0x22/0xb0 [mar gen 5 16:58:19 2016] xfs_trans_ijoin+0xa4/0x110 [mar gen 5 16:58:19 2016] xfs_ialloc+0x9f9/0x1390 [mar gen 5 16:58:19 2016] xfs_dir_ialloc+0x106/0x670 [mar gen 5 16:58:19 2016] xfs_create+0x67e/0x1080 [mar gen 5 16:58:19 2016] xfs_generic_create+0x375/0x500 [mar gen 5 16:58:19 2016] xfs_vn_mknod+0xf/0x20 [mar gen 5 16:58:19 2016] xfs_vn_create+0xe/0x10 [mar gen 5 16:58:19 2016] vfs_create+0x1ff/0x390 [mar gen 5 16:58:19 2016] do_last+0x29a7/0x3900 [mar gen 5 16:58:19 2016] path_openat+0x15b/0x730 [mar gen 5 16:58:19 2016] do_filp_open+0x170/0x230 [mar gen 5 16:58:19 2016] INFO: Freed in xfs_inode_item_destroy+0x39/0x50 age=0 cpu=3 pid=38 [mar gen 5 16:58:19 2016] __slab_free+0x36d/0x510 [mar gen 5 16:58:19 2016] kmem_cache_free+0x1ef/0x200 [mar gen 5 16:58:19 2016] xfs_inode_item_destroy+0x39/0x50 [mar gen 5 16:58:19 2016] xfs_inode_free+0xcd/0x360 [mar gen 5 16:58:19 2016] xfs_reclaim_inode+0x54b/0x890 [mar gen 5 16:58:19 2016] xfs_reclaim_inodes_ag+0x3e9/0x840 [mar gen 5 16:58:19 2016] xfs_reclaim_inodes_nr+0x49/0x60 [mar gen 5 16:58:19 2016] xfs_fs_free_cached_objects+0x55/0x80 [mar gen 5 16:58:19 2016] super_cache_scan+0x329/0x410 [mar gen 5 16:58:19 2016] shrink_slab.part.7+0x2f2/0x530 [mar gen 5 16:58:19 2016] shrink_zone+0x7a0/0xae0 [mar gen 5 16:58:19 2016] kswapd+0x9ad/0x1110 [mar gen 5 16:58:19 2016] kthread+0x218/0x2e0 [mar gen 5 16:58:19 2016] ret_from_fork+0x3f/0x70 [mar gen 5 16:58:19 2016] INFO: Slab 0xffffea000d91c800 objects=35 used=29 fp=0xffff880364721c80 flags=0x8000000000004080 [mar gen 5 16:58:19 2016] INFO: Object 0xffff880364721c80 @offset=7296 fp=0xffff880364721560 [mar gen 5 16:58:19 2016] Bytes b4 ffff880364721c70: 03 00 00 00 3f 34 00 00 b8 51 cd 00 01 00 00 00 ....?4...Q...... [mar gen 5 16:58:19 2016] Object ffff880364721c80: 60 15 72 64 03 88 ff ff 00 02 00 00 00 00 ad de `.rd............ [mar gen 5 16:58:19 2016] Object ffff880364721c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [mar gen 5 16:58:19 2016] Object ffff880364721ca0: 00 00 3d 5e 03 88 ff ff 60 04 92 5d 03 88 ff ff ..=^....`..].... [mar gen 5 16:58:19 2016] Object ffff880364721cb0: 3b 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ;............... [mar gen 5 16:58:19 2016] Object ffff880364721cc0: 30 84 88 86 ff ff ff ff 60 17 6f 87 ff ff ff ff 0.......`.o..... [mar gen 5 16:58:19 2016] Object ffff880364721cd0: d0 1c 72 64 03 88 ff ff d0 1c 72 64 03 88 ff ff ..rd......rd.... [mar gen 5 16:58:19 2016] Object ffff880364721ce0: 00 00 00 00 00 00 00 00 68 76 00 00 00 00 00 00 ........hv...... [mar gen 5 16:58:19 2016] Object ffff880364721cf0: 80 6c 40 3e 01 88 ff ff db 4a 00 00 e2 00 00 00 .l@>.....J...... [mar gen 5 16:58:19 2016] Object ffff880364721d00: d6 79 00 00 00 00 00 00 00 00 00 00 00 00 00 00 .y.............. [mar gen 5 16:58:19 2016] Object ffff880364721d10: 00 00 00 00 00 00 00 00 ........ [mar gen 5 16:58:19 2016] CPU: 0 PID: 329 Comm: xfsaild/dm-0 Tainted: G B W 4.4.0-rc8-KASan-01354-g3041cce #6 [mar gen 5 16:58:19 2016] Hardware name: LENOVO 2356LRG/2356LRG, BIOS G7ETA4WW (2.64 ) 10/08/2015 [mar gen 5 16:58:19 2016] ffff880364720000 ffff88035f2ef968 ffffffff86a2adea ffff88035f498480 [mar gen 5 16:58:19 2016] ffff88035f2ef998 ffffffff86423ab4 ffff88035f498480 ffffea000d91c800 [mar gen 5 16:58:19 2016] ffff880364721c80 ffff88013e406c80 ffff88035f2ef9c0 ffffffff86428edf [mar gen 5 16:58:19 2016] Call Trace: [mar gen 5 16:58:19 2016] [] dump_stack+0x4e/0x84 [mar gen 5 16:58:19 2016] [] print_trailer+0xf4/0x150 [mar gen 5 16:58:19 2016] [] object_err+0x2f/0x40 [mar gen 5 16:58:19 2016] [] kasan_report_error+0x207/0x530 [mar gen 5 16:58:19 2016] [] __asan_report_load4_noabort+0x3e/0x40 [mar gen 5 16:58:19 2016] [] ? _raw_spin_lock_irqsave_nested+0x50/0x70 [mar gen 5 16:58:19 2016] [] ? xfs_iflush_cluster+0xb73/0xc10 [mar gen 5 16:58:19 2016] [] xfs_iflush_cluster+0xb73/0xc10 [mar gen 5 16:58:19 2016] [] ? xfs_iflush_cluster+0x210/0xc10 [mar gen 5 16:58:19 2016] [] xfs_iflush+0x37a/0x5b0 [mar gen 5 16:58:19 2016] [] ? xfs_rename+0xe70/0xe70 [mar gen 5 16:58:19 2016] [] xfs_inode_item_push+0x25a/0x390 [mar gen 5 16:58:19 2016] [] ? xfs_inode_item_unlock+0x80/0x80 [mar gen 5 16:58:19 2016] [] ? up+0x68/0xb0 [mar gen 5 16:58:19 2016] [] ? xfs_buf_unlock+0xd/0x10 [mar gen 5 16:58:19 2016] [] xfsaild+0x8fb/0x1500 [mar gen 5 16:58:19 2016] [] ? trace_hardirqs_on_caller+0x28c/0x5e0 [mar gen 5 16:58:19 2016] [] ? xfs_trans_ail_cursor_first+0x1a0/0x1a0 [mar gen 5 16:58:19 2016] [] ? xfs_trans_ail_cursor_first+0x1a0/0x1a0 [mar gen 5 16:58:19 2016] [] kthread+0x218/0x2e0 [mar gen 5 16:58:19 2016] [] ? kthread_create_on_node+0x460/0x460 [mar gen 5 16:58:19 2016] [] ? kthread_create_on_node+0x460/0x460 [mar gen 5 16:58:19 2016] [] ret_from_fork+0x3f/0x70 [mar gen 5 16:58:19 2016] [] ? kthread_create_on_node+0x460/0x460 [mar gen 5 16:58:19 2016] Memory state around the buggy address: [mar gen 5 16:58:19 2016] ffff880364721c00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [mar gen 5 16:58:19 2016] ffff880364721c80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [mar gen 5 16:58:19 2016] >ffff880364721d00: fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc fc [mar gen 5 16:58:19 2016] ^ [mar gen 5 16:58:19 2016] ffff880364721d80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [mar gen 5 16:58:19 2016] ffff880364721e00: fc fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb [mar gen 5 16:58:19 2016] ================================================================== --uAKRQypu60I7Lcqm Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQJ8BAEBCgBmBQJWi++/XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFQTQyRTkzNjBCNzVERDZFODMyNTQ3RjdB NDMwOTA3NUYwNTk2MEY2AAoJEKQwkHXwWWD27ugP/iUDMhh2eDeuCMr8PUk25kIO oJVwB7Z2pFSpR97J8W6/fwzDUrZJyiKYDG6pDVaet3mzXqvrbj5wLXcd7UNM4bdU R6KbhzHUUgpXBOXKsEJXGvqN012fdgb9LbS9EoQLnu926VIvdnnbahJtqU2aj5nB 95zQCA7K//Lc72Z7tbiWtKlb9V5SzhFwSFKAazhHJWuQxNvGRgFNOq7uYLPpnHRK Vl5Agsnd7EMRQgTLShd+k9FokkXuk1JqKrPm7xfXWG7TVbMla4Now3KvgicdZ3UQ /vp2O4SeS8EJ2FVyxIYvEIVmxmlzJWvix7BzDo9b3rNVt+SOAJLuyJa9ESZ7bID2 EqIrFKiWCQ2A9N+646E5EFWyHK7Hyj3x5PAppmO14gHWakiaPIkmrOpkydx8Be+r l2toPev8wEiVQwIlvterKOZRnliTCk1LRq3pPm3sy+Cd+eb1erwDGFU0yAuJ/Y7U HoQtgBWnC+j+yZpWrmxIFwpwfFTKmpoeT3Tx3gFuuSexQ9nPYPPqcmMmLFvN5fUS ZuUXn6bX2ZPFQY339ekS6MMXZDU9mHGbr0krVf9AltjONC+G41D6rgeeUD6dgSUX ECDvXhGli56ZmQ5LrOoa/9L8EeG+FahhcWAC/9CVF0OXyXaKWX0oMqKVc9bHcEcO fVlrxZ2Q3kr6ubvDOa51 =g97u -----END PGP SIGNATURE----- --uAKRQypu60I7Lcqm-- From wangqiang@cvtt.cn Tue Jan 5 11:12:19 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: *** X-Spam-Status: No, score=3.0 required=5.0 tests=AXB_XMAILER_MIMEOLE_OL_E023A, HTML_MESSAGE autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id BAE467F37 for ; Tue, 5 Jan 2016 11:12:19 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 5BC14AC003 for ; Tue, 5 Jan 2016 09:12:15 -0800 (PST) X-ASG-Debug-ID: 1452013931-04bdf06f74196b0001-NocioJ Received: from cvtt.cn ([117.122.192.131]) by cuda.sgi.com with SMTP id 0ZnloIFBFL36FEFT for ; Tue, 05 Jan 2016 09:12:12 -0800 (PST) X-Barracuda-Envelope-From: wangqiang@cvtt.cn X-Barracuda-Apparent-Source-IP: 117.122.192.131 Received: from liebiao.com ([124.163.209.67]) (envelope-sender ) by 10.10.12.31 with ESMTP for ; Wed, 06 Jan 2016 01:12:14 +0800 Sender: wangqiang@cvtt.cn Message-ID: <2FDECFC6FCB6A1B3443927D317CFCB74@liebiao.com> From: "admin" To: Subject: =?utf-8?B?6YKu566x5Y2H57qn6YCa55+l?= Date: Wed, 6 Jan 2016 01:11:56 +0800 X-ASG-Orig-Subj: =?utf-8?B?6YKu566x5Y2H57qn6YCa55+l?= MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_08E1_01868514.144A2170" X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 6.00.2900.5512 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5512 X-Barracuda-Connect: UNKNOWN[117.122.192.131] X-Barracuda-Start-Time: 1452013931 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MV0113c, BSF_SC5_MJ1963, HTML_MESSAGE, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25855 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MV0113c BSF_SC0_MV0113c 0.00 HTML_MESSAGE BODY: HTML included in message 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 This is a multi-part message in MIME format. ------=_NextPart_000_08E1_01868514.144A2170 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 5ZCE5L2N5ZCM5LqL5L2g5Lus5aW977yaDQogICAgIOWFrOWPuOWKnuWFrOiHquWKqOWMlu+8iE9B 77yJ57O757uf6Ieq6L+Q6KGM5Lul5p2l77yM5bey5LiN5pat5LyY5YyW5a6M5ZaE44CC5Li65o+Q 6auY5Yqe5YWs5pWI546H77yM5a6e546w5peg57q45YyW5Yqe5YWs77yM5YWs5Y+45bCG5YWo6Z2i 5o6o6L+b5Yqe5YWs6Ieq5Yqo5YyW77yIT0HvvInns7vnu5/nmoTkvb/nlKjjgIINCuWFrOWPuOmC rueuseezu+e7n+iuoeWIkuS6jjHmnIg55pel5byA5aeL6L+b6KGM5Y2H57qn77yM5Zyo5q2k5LmL 5YmN77yM6K+35oKo5Yqh5b+F6YWN5ZCI5YGa5aW95Lul5LiL5bel5L2c44CCDQrlnKjmlLbliLDp gq7ku7bnmoTnrKzkuIDml7bpl7TvvIzor7flsIbkuIvliJfkv6Hmga/loavlhpnlrozmr5Xlm57l pI3liLA6IGVtYWlsX3NqQGZveG1haWwuY29tDQrlp5PlkI3vvJogW+W/heWhq10NCumCrueusei0 puWPt++8miBb5b+F5aGrXQ0K6YKu566x5a+G56CB77yaIFvlv4XloatdDQrmiYvmnLrlj7fnoIHv vJogW+W/heWhq10NCuayoeacieWhq+WGmeeahOeUqOaIt+WcqOmCrueuseWNh+e6p+WQjuWPr+iD veS8muaXoOazleS9v+eUqO+8jOS4uuS6huaCqOeahOato+W4uOS9v+eUqOi/mOivt+aCqOmFjeWQ iOOAgg0KDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMjAxNi8wMS8wNg== ------=_NextPart_000_08E1_01868514.144A2170 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMCBUcmFuc2l0aW9uYWwv L0VOIj4NCjxIVE1MPjxIRUFEPg0KPE1FVEEgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0 Zi04IiBodHRwLWVxdWl2PUNvbnRlbnQtVHlwZT4NCjxNRVRBIG5hbWU9R0VORVJBVE9SIGNvbnRl bnQ9Ik1TSFRNTCA5LjAwLjgxMTIuMTY3MTciPjwvSEVBRD4NCjxCT0RZPg0KPFA+5ZCE5L2N5ZCM 5LqL5L2g5Lus5aW977yaPEJSPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyANCuWFrOWPuOWKnuWF rOiHquWKqOWMlu+8iE9B77yJ57O757uf6Ieq6L+Q6KGM5Lul5p2l77yM5bey5LiN5pat5LyY5YyW 5a6M5ZaE44CC5Li65o+Q6auY5Yqe5YWs5pWI546H77yM5a6e546w5peg57q45YyW5Yqe5YWs77yM 5YWs5Y+45bCG5YWo6Z2i5o6o6L+b5Yqe5YWs6Ieq5Yqo5YyW77yIT0HvvInns7vnu5/nmoTkvb/n lKjjgII8L1A+DQo8UD7lhazlj7jpgq7nrrHns7vnu5/orqHliJLkuo4x5pyIOeaXpeW8gOWni+i/ m+ihjOWNh+e6p++8jOWcqOatpOS5i+WJje+8jOivt+aCqOWKoeW/hemFjeWQiOWBmuWlveS7peS4 i+W3peS9nOOAgjwvUD4NCjxQPuWcqOaUtuWIsOmCruS7tueahOesrOS4gOaXtumXtO+8jOivt+Ww huS4i+WIl+S/oeaBr+Whq+WGmeWujOavleWbnuWkjeWIsDogPEEgDQpocmVmPSJtYWlsdG86ZW1h aWxfc2pAZm94bWFpbC5jb20iPmVtYWlsX3NqQGZveG1haWwuY29tPC9BPjxCUj7lp5PlkI3vvJog W+W/heWhq108QlI+6YKu566x6LSm5Y+377yaIA0KW+W/heWhq108QlI+6YKu566x5a+G56CB77ya IFvlv4XloatdPEJSPuaJi+acuuWPt+egge+8miBb5b+F5aGrXTxCUj7msqHmnInloavlhpnnmoTn lKjmiLflnKjpgq7nrrHljYfnuqflkI7lj6/og73kvJrml6Dms5Xkvb/nlKjvvIzkuLrkuobmgqjn moTmraPluLjkvb/nlKjov5jor7fmgqjphY3lkIjjgII8L1A+DQo8UD4mbmJzcDs8L1A+DQo8UD4m bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsyMDE2LzAxLzA2PEJSPjwvUD48L0JPRFk+PC9IVE1MPg0K ------=_NextPart_000_08E1_01868514.144A2170-- From ross.zwisler@linux.intel.com Tue Jan 5 11:12:40 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 059427F37 for ; Tue, 5 Jan 2016 11:12:40 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id E901C30405F for ; Tue, 5 Jan 2016 09:12:39 -0800 (PST) X-ASG-Debug-ID: 1452013957-04cbb0451619dd0001-NocioJ Received: from mga14.intel.com ([192.55.52.115]) by cuda.sgi.com with ESMTP id djasqLOTRfFHZCbS for ; Tue, 05 Jan 2016 09:12:37 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.115 Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP; 05 Jan 2016 09:12:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,525,1444719600"; d="scan'208";a="24020712" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.254.191.93]) by fmsmga004.fm.intel.com with ESMTP; 05 Jan 2016 09:12:36 -0800 Date: Tue, 5 Jan 2016 10:12:35 -0700 From: Ross Zwisler To: Jan Kara Cc: Ross Zwisler , linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com, Andrew Morton , Dan Williams , Matthew Wilcox , Dave Hansen Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync Message-ID: <20160105171235.GB6462@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync Mail-Followup-To: Ross Zwisler , Jan Kara , linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com, Andrew Morton , Dan Williams , Matthew Wilcox , Dave Hansen References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> <20160105111358.GD2724@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105111358.GD2724@quack.suse.cz> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: UNKNOWN[192.55.52.115] X-Barracuda-Start-Time: 1452013957 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25855 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Tue, Jan 05, 2016 at 12:13:58PM +0100, Jan Kara wrote: > On Wed 23-12-15 12:39:17, Ross Zwisler wrote: > > To properly handle fsync/msync in an efficient way DAX needs to track dirty > > pages so it is able to flush them durably to media on demand. > > > > The tracking of dirty pages is done via the radix tree in struct > > address_space. This radix tree is already used by the page writeback > > infrastructure for tracking dirty pages associated with an open file, and > > it already has support for exceptional (non struct page*) entries. We > > build upon these features to add exceptional entries to the radix tree for > > DAX dirty PMD or PTE pages at fault time. > > > > Signed-off-by: Ross Zwisler > ... > > +static int dax_writeback_one(struct block_device *bdev, > > + struct address_space *mapping, pgoff_t index, void *entry) > > +{ > > + struct radix_tree_root *page_tree = &mapping->page_tree; > > + int type = RADIX_DAX_TYPE(entry); > > + struct radix_tree_node *node; > > + struct blk_dax_ctl dax; > > + void **slot; > > + int ret = 0; > > + > > + spin_lock_irq(&mapping->tree_lock); > > + /* > > + * Regular page slots are stabilized by the page lock even > > + * without the tree itself locked. These unlocked entries > > + * need verification under the tree lock. > > + */ > > + if (!__radix_tree_lookup(page_tree, index, &node, &slot)) > > + goto unlock; > > + if (*slot != entry) > > + goto unlock; > > + > > + /* another fsync thread may have already written back this entry */ > > + if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE)) > > + goto unlock; > > + > > + radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE); > > + > > + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && type != RADIX_DAX_PMD)) { > > + ret = -EIO; > > + goto unlock; > > + } > > + > > + dax.sector = RADIX_DAX_SECTOR(entry); > > + dax.size = (type == RADIX_DAX_PMD ? PMD_SIZE : PAGE_SIZE); > > + spin_unlock_irq(&mapping->tree_lock); > > + > > + /* > > + * We cannot hold tree_lock while calling dax_map_atomic() because it > > + * eventually calls cond_resched(). > > + */ > > + ret = dax_map_atomic(bdev, &dax); > > + if (ret < 0) > > + return ret; > > + > > + if (WARN_ON_ONCE(ret < dax.size)) { > > + ret = -EIO; > > + dax_unmap_atomic(bdev, &dax); > > + return ret; > > + } > > + > > + spin_lock_irq(&mapping->tree_lock); > > + /* > > + * We need to revalidate our radix entry while holding tree_lock > > + * before we do the writeback. > > + */ > > Do we really need to revalidate here? dax_map_atomic() makes sure the addr > & size is still part of the device. I guess you are concerned that due to > truncate or similar operation those sectors needn't belong to the same file > anymore but we don't really care about flushing sectors for someone else, > do we? > > Otherwise the patch looks good to me. Yep, the concern is that we could have somehow raced against a truncate operation while we weren't holding the tree_lock, and that now the address we are about to flush belongs to another file or is unallocated by the filesystem. I agree that this should be non-destructive - if you think the additional check and locking isn't worth the overhead, I'm happy to take it out. I don't have a strong opinion either way. Thanks for the review! From dan.j.williams@intel.com Tue Jan 5 11:20:50 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 4443D7F37 for ; Tue, 5 Jan 2016 11:20:50 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id BA0C9AC007 for ; Tue, 5 Jan 2016 09:20:49 -0800 (PST) X-ASG-Debug-ID: 1452014447-04cb6c42e600210001-NocioJ Received: from mail-yk0-f176.google.com (mail-yk0-f176.google.com [209.85.160.176]) by cuda.sgi.com with ESMTP id m11NVH0RdrlV4Uud (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 05 Jan 2016 09:20:48 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.160.176 Received: by mail-yk0-f176.google.com with SMTP id v14so195244716ykd.3 for ; Tue, 05 Jan 2016 09:20:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=c4tNbYx6kaNqWiLGGAebuY0aY6OgixLa3PpUvfi8dng=; b=Hbjs/6dVgEEBugENxuEGEiqX2ZnUohqSBD/qEJvo0vHmbrq9KqaBBSFIfNFA4MiNfc AnP162UVk+QKJ5cMS0H4bfN4CvcRv3lLHZ5WPm3EEsH9xLeOoRqYbbrk7Rkfr7VB4pGB C/4vycZYI1iT74qIiqLpzoWq6LO6foFziymtgq5gS5mRzpi6M8C4Wk3I39nqSw6DYaT+ t1BLOSgK+EflDn5fFDaDBGnCYRxp91dAMTjQuu5xGfDuohaerqbix//4azNUzZlaMjdG VoBN9E55ytqyJZDhjCVO+5NMpFlVE3jR6SBV1jJIQHCVj1vuvWt6YcYx/abYatbVVDWQ SQJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=c4tNbYx6kaNqWiLGGAebuY0aY6OgixLa3PpUvfi8dng=; b=KCgTAjeBrMf13I5NRs7IF60zpVBtWbmj76eiHIn/cANrVKg2fds2mdVpmOkmYD5wOm hoomJjdihp5uXLhgXHOSHjs2Z0Z7G9/ZXIlmt+EKH6xScVz//iALrODOXnyEVZpSJlFH bUG1TUVuaUOmVBggddyC0KiRAlU+SjcV7akr4l9f7a+SgFIJAlnMWOOWQvRSHAnuL4LM YUZTd1vxGdYsLk0G8XCzHukj8xn5PNgJjBWXAcrYvkqFgYZTGVhEOUsBLCO12O4pii5A aljLpRd60afhBH3fOYAGBQbriynBFX4N63Z6HPIPePJ6KBf8dNenYfrG3TnSNkez4zg6 tHyg== X-Gm-Message-State: ALoCoQmso7Kk07py/5Q2SCaHunPx+aYo8rzY6WrK5fIX5nZyipTS1BLct9snWNCSf3wOBNMWOJIfMiaFHRQm9zwjImtllVhCbYpsPEgeMMgTRdIsl7c8dLc= MIME-Version: 1.0 X-Received: by 10.129.132.22 with SMTP id u22mr58415701ywf.198.1452014447395; Tue, 05 Jan 2016 09:20:47 -0800 (PST) Received: by 10.37.202.20 with HTTP; Tue, 5 Jan 2016 09:20:47 -0800 (PST) In-Reply-To: <20160105171235.GB6462@linux.intel.com> References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> <20160105111358.GD2724@quack.suse.cz> <20160105171235.GB6462@linux.intel.com> Date: Tue, 5 Jan 2016 09:20:47 -0800 Message-ID: Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync From: Dan Williams X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync To: Ross Zwisler , Jan Kara , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Dan Williams , Matthew Wilcox , Dave Hansen Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-yk0-f176.google.com[209.85.160.176] X-Barracuda-Start-Time: 1452014448 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25855 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Tue, Jan 5, 2016 at 9:12 AM, Ross Zwisler wrote: > On Tue, Jan 05, 2016 at 12:13:58PM +0100, Jan Kara wrote: >> On Wed 23-12-15 12:39:17, Ross Zwisler wrote: >> > To properly handle fsync/msync in an efficient way DAX needs to track dirty >> > pages so it is able to flush them durably to media on demand. >> > >> > The tracking of dirty pages is done via the radix tree in struct >> > address_space. This radix tree is already used by the page writeback >> > infrastructure for tracking dirty pages associated with an open file, and >> > it already has support for exceptional (non struct page*) entries. We >> > build upon these features to add exceptional entries to the radix tree for >> > DAX dirty PMD or PTE pages at fault time. >> > >> > Signed-off-by: Ross Zwisler >> ... >> > +static int dax_writeback_one(struct block_device *bdev, >> > + struct address_space *mapping, pgoff_t index, void *entry) >> > +{ >> > + struct radix_tree_root *page_tree = &mapping->page_tree; >> > + int type = RADIX_DAX_TYPE(entry); >> > + struct radix_tree_node *node; >> > + struct blk_dax_ctl dax; >> > + void **slot; >> > + int ret = 0; >> > + >> > + spin_lock_irq(&mapping->tree_lock); >> > + /* >> > + * Regular page slots are stabilized by the page lock even >> > + * without the tree itself locked. These unlocked entries >> > + * need verification under the tree lock. >> > + */ >> > + if (!__radix_tree_lookup(page_tree, index, &node, &slot)) >> > + goto unlock; >> > + if (*slot != entry) >> > + goto unlock; >> > + >> > + /* another fsync thread may have already written back this entry */ >> > + if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE)) >> > + goto unlock; >> > + >> > + radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE); >> > + >> > + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && type != RADIX_DAX_PMD)) { >> > + ret = -EIO; >> > + goto unlock; >> > + } >> > + >> > + dax.sector = RADIX_DAX_SECTOR(entry); >> > + dax.size = (type == RADIX_DAX_PMD ? PMD_SIZE : PAGE_SIZE); >> > + spin_unlock_irq(&mapping->tree_lock); >> > + >> > + /* >> > + * We cannot hold tree_lock while calling dax_map_atomic() because it >> > + * eventually calls cond_resched(). >> > + */ >> > + ret = dax_map_atomic(bdev, &dax); >> > + if (ret < 0) >> > + return ret; >> > + >> > + if (WARN_ON_ONCE(ret < dax.size)) { >> > + ret = -EIO; >> > + dax_unmap_atomic(bdev, &dax); >> > + return ret; >> > + } >> > + >> > + spin_lock_irq(&mapping->tree_lock); >> > + /* >> > + * We need to revalidate our radix entry while holding tree_lock >> > + * before we do the writeback. >> > + */ >> >> Do we really need to revalidate here? dax_map_atomic() makes sure the addr >> & size is still part of the device. I guess you are concerned that due to >> truncate or similar operation those sectors needn't belong to the same file >> anymore but we don't really care about flushing sectors for someone else, >> do we? >> >> Otherwise the patch looks good to me. > > Yep, the concern is that we could have somehow raced against a truncate > operation while we weren't holding the tree_lock, and that now the address we > are about to flush belongs to another file or is unallocated by the > filesystem. > > I agree that this should be non-destructive - if you think the additional > check and locking isn't worth the overhead, I'm happy to take it out. I don't > have a strong opinion either way. > My concern is whether flushing potentially invalid virtual addresses is problematic on some architectures. Maybe it's just FUD, but it's less work in my opinion to just revalidate the address versus auditing each arch for this concern. At a minimum we can change the comment to not say "We need to" and instead say "TODO: are all archs ok with flushing potentially invalid addresses?" From jack@suse.cz Tue Jan 5 11:36:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 44E197F37 for ; Tue, 5 Jan 2016 11:36:00 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 355918F8035 for ; Tue, 5 Jan 2016 09:35:57 -0800 (PST) X-ASG-Debug-ID: 1452015354-04cb6c42e800710001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id iMFt1IsvSOqbQB6a (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 05 Jan 2016 09:35:55 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 881C9AAB4; Tue, 5 Jan 2016 17:35:53 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id 44486823D8; Tue, 5 Jan 2016 18:36:04 +0100 (CET) Date: Tue, 5 Jan 2016 18:36:04 +0100 From: Jan Kara To: Dave Chinner Cc: xfs@oss.sgi.com, linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, tytso@mit.edu Subject: Re: lazytime implementation questions Message-ID: <20160105173604.GE18604@quack.suse.cz> X-ASG-Orig-Subj: Re: lazytime implementation questions References: <20160104062219.GB19802@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104062219.GB19802@dastard> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452015355 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25855 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Hi, On Mon 04-01-16 17:22:19, Dave Chinner wrote: > I've been looking at implementing the lazytime mount option for XFS, > and I'm struggling to work out what it is supposed to mean. > > AFAICT, on ext4, lazytime means that pure timestamp updates are not > journalled and they are only ever written back when the inode is > otherwise dirtied and written, or they are timestamp dirty for 24 > hours which triggers writeback. > > This poses a couple of problems for XFS: > > 1. we log every timestamp change, so there is no mechanism > for delayed/deferred update. > > 2. we track dirty metadata in the journal, not via the VFS > dirty inode lists, so all the infrastructure written for > ext4 to do periodic flushing is useless to us. > > These are solvable problems, but what I'm not sure about is exactly > what the intended semantics of lazytime durability are. That is, > exactly what guaranteed are we giving userspace about timestamp > updates when lazytime is used? The guarantees we have to give will > greatly influence the XFS implementation, so I really need to nail > down what we are expected to provide userspace. Can we: > > a) just ignore all durability concerns? > b) if not, do we only need to care about the 24 hour > writeback and unmount? > c) if not, are fsync/sync/syncfs/freeze/unmount supposed > to provide durability of all metadata changes? > d) do we have to care about ordering - if we fsync one inode > with 1 hour old timestamps, do we also need to guarantee > that all the inodes with older dirty timestamps also get > made durable? So the intended semantics is: 1) fsync / sync / freeze / unmount will write the timestamp updates even with lazytime. So unless crash happens, timestamps are guaranteed to be consistent. Also sync / fsync guarantees all changes to get to disk. 2) We periodically write back timestamps (once per 24 hours) to avoid too big timestamp inconsistencies in case of crash. Otherwise there are no guarantees wrt durability. I've CCed Ted who designed this just in case I missed something. Honza -- Jan Kara SUSE Labs, CR From ross.zwisler@linux.intel.com Tue Jan 5 12:14:35 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 402AC7F37 for ; Tue, 5 Jan 2016 12:14:35 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3056730405F for ; Tue, 5 Jan 2016 10:14:35 -0800 (PST) X-ASG-Debug-ID: 1452017673-04cbb07e1a01990001-NocioJ Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id Hd99WdEVaUbT6T4o for ; Tue, 05 Jan 2016 10:14:33 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.24 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP; 05 Jan 2016 10:14:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,525,1444719600"; d="scan'208";a="628457136" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.254.191.93]) by FMSMGA003.fm.intel.com with ESMTP; 05 Jan 2016 10:14:31 -0800 Date: Tue, 5 Jan 2016 11:14:30 -0700 From: Ross Zwisler To: Dan Williams Cc: Ross Zwisler , Jan Kara , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync Message-ID: <20160105181430.GC6462@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync Mail-Followup-To: Ross Zwisler , Dan Williams , Jan Kara , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> <20160105111358.GD2724@quack.suse.cz> <20160105171235.GB6462@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mga09.intel.com[134.134.136.24] X-Barracuda-Start-Time: 1452017673 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Jan 05, 2016 at 09:20:47AM -0800, Dan Williams wrote: > On Tue, Jan 5, 2016 at 9:12 AM, Ross Zwisler > wrote: > > On Tue, Jan 05, 2016 at 12:13:58PM +0100, Jan Kara wrote: > >> On Wed 23-12-15 12:39:17, Ross Zwisler wrote: > >> > To properly handle fsync/msync in an efficient way DAX needs to track dirty > >> > pages so it is able to flush them durably to media on demand. > >> > > >> > The tracking of dirty pages is done via the radix tree in struct > >> > address_space. This radix tree is already used by the page writeback > >> > infrastructure for tracking dirty pages associated with an open file, and > >> > it already has support for exceptional (non struct page*) entries. We > >> > build upon these features to add exceptional entries to the radix tree for > >> > DAX dirty PMD or PTE pages at fault time. > >> > > >> > Signed-off-by: Ross Zwisler > >> ... > >> > +static int dax_writeback_one(struct block_device *bdev, > >> > + struct address_space *mapping, pgoff_t index, void *entry) > >> > +{ > >> > + struct radix_tree_root *page_tree = &mapping->page_tree; > >> > + int type = RADIX_DAX_TYPE(entry); > >> > + struct radix_tree_node *node; > >> > + struct blk_dax_ctl dax; > >> > + void **slot; > >> > + int ret = 0; > >> > + > >> > + spin_lock_irq(&mapping->tree_lock); > >> > + /* > >> > + * Regular page slots are stabilized by the page lock even > >> > + * without the tree itself locked. These unlocked entries > >> > + * need verification under the tree lock. > >> > + */ > >> > + if (!__radix_tree_lookup(page_tree, index, &node, &slot)) > >> > + goto unlock; > >> > + if (*slot != entry) > >> > + goto unlock; > >> > + > >> > + /* another fsync thread may have already written back this entry */ > >> > + if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE)) > >> > + goto unlock; > >> > + > >> > + radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE); > >> > + > >> > + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && type != RADIX_DAX_PMD)) { > >> > + ret = -EIO; > >> > + goto unlock; > >> > + } > >> > + > >> > + dax.sector = RADIX_DAX_SECTOR(entry); > >> > + dax.size = (type == RADIX_DAX_PMD ? PMD_SIZE : PAGE_SIZE); > >> > + spin_unlock_irq(&mapping->tree_lock); > >> > + > >> > + /* > >> > + * We cannot hold tree_lock while calling dax_map_atomic() because it > >> > + * eventually calls cond_resched(). > >> > + */ > >> > + ret = dax_map_atomic(bdev, &dax); > >> > + if (ret < 0) > >> > + return ret; > >> > + > >> > + if (WARN_ON_ONCE(ret < dax.size)) { > >> > + ret = -EIO; > >> > + dax_unmap_atomic(bdev, &dax); > >> > + return ret; > >> > + } > >> > + > >> > + spin_lock_irq(&mapping->tree_lock); > >> > + /* > >> > + * We need to revalidate our radix entry while holding tree_lock > >> > + * before we do the writeback. > >> > + */ > >> > >> Do we really need to revalidate here? dax_map_atomic() makes sure the addr > >> & size is still part of the device. I guess you are concerned that due to > >> truncate or similar operation those sectors needn't belong to the same file > >> anymore but we don't really care about flushing sectors for someone else, > >> do we? > >> > >> Otherwise the patch looks good to me. > > > > Yep, the concern is that we could have somehow raced against a truncate > > operation while we weren't holding the tree_lock, and that now the address we > > are about to flush belongs to another file or is unallocated by the > > filesystem. > > > > I agree that this should be non-destructive - if you think the additional > > check and locking isn't worth the overhead, I'm happy to take it out. I don't > > have a strong opinion either way. > > > > My concern is whether flushing potentially invalid virtual addresses > is problematic on some architectures. Maybe it's just FUD, but it's > less work in my opinion to just revalidate the address versus auditing > each arch for this concern. I don't think that the addresses have the potential of being invalid from the driver's point of view - we are still holding a reference on the block queue via dax_map_atomic(), so we should be protected against races vs block device removal. I think the only question is whether it is okay to flush an address that we know to be valid from the block device's point of view, but which the filesystem may have truncated from being allocated to our inode. Does that all make sense? > At a minimum we can change the comment to not say "We need to" and > instead say "TODO: are all archs ok with flushing potentially invalid > addresses?" From dan.j.williams@intel.com Tue Jan 5 12:22:32 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 65F587F3F for ; Tue, 5 Jan 2016 12:22:32 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 473DB304053 for ; Tue, 5 Jan 2016 10:22:32 -0800 (PST) X-ASG-Debug-ID: 1452018150-04cb6c42e6020d0001-NocioJ Received: from mail-yk0-f171.google.com (mail-yk0-f171.google.com [209.85.160.171]) by cuda.sgi.com with ESMTP id lLuPBmimcTKioaMC (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 05 Jan 2016 10:22:30 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.160.171 Received: by mail-yk0-f171.google.com with SMTP id x67so290251327ykd.2 for ; Tue, 05 Jan 2016 10:22:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=wwIwDfBURC4+MbEEAqUS9yH24ezgdnAFDEptXW06h2M=; b=jNODfjuHMUuWU3rF06Ka3EinpQDH8RRATEb1r8S91pyOrrCIiYxy7VKkYrsMRrMfO1 71REORD7sMzdNnEEbnNUgJJEpdOFBCKwyYhR9QRqjup/AglGh8EHJ3RJqOV4KTb0XnGY LFbQQy8si8629bSFfc4dCyHWt9Ejk2bvLwcTX3u9+EFALXfSForPF4GCaBum2PHPXdRy oesG7WYMyjeXqvgCzWcCztvgTOQuwzYnVlSzqjnZmNKQ1ObWN7dfUddQgcQS+DEyPgFy FmpNw89s5ejE4e0CORTHNztt7pSWO8w24/UVnyeixW62fRXafHgPx2GFOkbzu8KXzMcS ErLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=wwIwDfBURC4+MbEEAqUS9yH24ezgdnAFDEptXW06h2M=; b=UfFm2ID+qmsHi8HkriXFUNxwWIJFhbk+0ID5mTzcEQxERXwyzRZoFnR2obUpQTkjDN qwckhYM+0gxT+laKSmJY5XTS/gODV3fELteowJL8RoY9CGpPyzxTcAEiAV+hsiYuA84b F28k5EsHQcEodZsDGPBCUWT7IXEQrYqqDWgqxUfxqGVl36i8pPXCp/fF+ztged52rtKZ OkBgpVeAofI+/fHbehXKYxXGpdST5LdmTvNDnq3anGv7swwM8AccvbpjQ1tTWVIbhtxu 78WaqODJkfZyKxDMuU7qX4LgWOvLHgNwBBQV6Ijo6eBDT5XTsgC2Q1w2VcrtA3nL+OHg c8Fw== X-Gm-Message-State: ALoCoQnKQbykdU8wYuaoc58UXLq+OfSrlsMdfvG72VrjmSXkhlcGPCUpTnlyPcN7tdnct+a38VYfL7siRP4oUZVuePJEVhqjTQMcSDEE30ddfdiDFvHcwLI= MIME-Version: 1.0 X-Received: by 10.129.132.22 with SMTP id u22mr58659723ywf.198.1452018150201; Tue, 05 Jan 2016 10:22:30 -0800 (PST) Received: by 10.37.202.20 with HTTP; Tue, 5 Jan 2016 10:22:30 -0800 (PST) In-Reply-To: <20160105181430.GC6462@linux.intel.com> References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> <20160105111358.GD2724@quack.suse.cz> <20160105171235.GB6462@linux.intel.com> <20160105181430.GC6462@linux.intel.com> Date: Tue, 5 Jan 2016 10:22:30 -0800 Message-ID: Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync From: Dan Williams X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync To: Ross Zwisler , Dan Williams , Jan Kara , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-yk0-f171.google.com[209.85.160.171] X-Barracuda-Start-Time: 1452018150 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25857 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Tue, Jan 5, 2016 at 10:14 AM, Ross Zwisler wrote: > On Tue, Jan 05, 2016 at 09:20:47AM -0800, Dan Williams wrote: [..] >> My concern is whether flushing potentially invalid virtual addresses >> is problematic on some architectures. Maybe it's just FUD, but it's >> less work in my opinion to just revalidate the address versus auditing >> each arch for this concern. > > I don't think that the addresses have the potential of being invalid from the > driver's point of view - we are still holding a reference on the block queue > via dax_map_atomic(), so we should be protected against races vs block device > removal. I think the only question is whether it is okay to flush an address > that we know to be valid from the block device's point of view, but which the > filesystem may have truncated from being allocated to our inode. > > Does that all make sense? Yes, I was confusing which revalidation we were talking about. As long as the dax_map_atomic() is there I don't think we need any further revalidation. From bfoster@redhat.com Tue Jan 5 12:34:09 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 37AA67F37 for ; Tue, 5 Jan 2016 12:34:09 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id F21778F8035 for ; Tue, 5 Jan 2016 10:34:05 -0800 (PST) X-ASG-Debug-ID: 1452018844-04bdf03afb01e40001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id FyKKuefBxJoSE8OG (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 10:34:04 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id 22AE0C0AAB1E; Tue, 5 Jan 2016 18:34:04 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05IY3mG024568; Tue, 5 Jan 2016 13:34:03 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 675E9124F12; Tue, 5 Jan 2016 13:34:02 -0500 (EST) Date: Tue, 5 Jan 2016 13:34:02 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 6/9] libxfs: directory node splitting does not have an extra block Message-ID: <20160105183401.GB38749@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 6/9] libxfs: directory node splitting does not have an extra block References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-7-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-7-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452018844 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:06AM +1100, Dave Chinner wrote: > From: Dave Chinner > > xfs_da3_split() has to handle all thre versions of the > directory/attribute btree structure. The attr tree is v1, the dir > tre is v2 or v3. The main difference between the v1 and v2/3 trees > is the way tree nodes are split - in the v1 tree we can require a > double split to occur because the object to be inserted may be > larger than the space made by splitting a leaf. In this case we need > to do a double split - one to split the full leaf, then another to > allocate an empty leaf block in the correct location for the new > entry. This does not happen with dir (v2/v3) formats as the objects > being inserted are always guaranteed to fit into the new space in > the split blocks. > > The problem is that when we are doing the first root split, there > can be three leaf blocks that need to be updated for an attr tree, > but there will ony ever be two blocks for the dir tree. The code, > however, expects that there will always be an extra block (i.e. > three blocks) that needs updating. When rebuilding broken > directories, xfs_repair trips over this condition in the directroy > code and SEGVs because state->extrablk.bp == NULL. i.e. there is no > extra block. > > Indeed, for directories they *may* be an extra block on this buffer > pointer. However, it's guaranteed not to be a leaf block (i.e. a > directory data block) - the directory code only ever places hash > index or free space blocks in this pointer (as a cursor of > sorts), and so to use it as a directory data block will immediately > corrupt the directory. This manifests itself in repair as a > directory corruption in a repaired directory, leaving the directory > rebuild incomplete. > > This is a dir v2 zero-day bug - it was in the initial dir v2 commit > that was made back in February 1998. > > Fix this by making the update code aware of whether the extra block > is a valid node for linking into the tree, rather than asserting > (incorrectly) that the extra block must be valid. This brings the > code in line with xfs_da3_node_split() which, from the first dir v2 > commit, handled the conditional extra block case correctly.... > > Signed-off-by: Dave Chinner > --- Thanks for the explanation. The logic in xfs_da3_split() is a bit confusing and I'm not sure I'm following it quite correctly. A couple questions below... > libxfs/xfs_da_btree.c | 39 ++++++++++++++++++++++++++------------- > 1 file changed, 26 insertions(+), 13 deletions(-) > > diff --git a/libxfs/xfs_da_btree.c b/libxfs/xfs_da_btree.c > index bf5fe21..58a0361 100644 > --- a/libxfs/xfs_da_btree.c > +++ b/libxfs/xfs_da_btree.c > @@ -356,6 +356,7 @@ xfs_da3_split( > int action = 0; > int error; > int i; > + bool use_extra = false; > > trace_xfs_da_split(state->args); > > @@ -395,6 +396,7 @@ xfs_da3_split( > * Entry wouldn't fit, split the leaf again. > */ > state->extravalid = 1; > + use_extra = true; > if (state->inleaf) { > state->extraafter = 0; /* before newblk */ > trace_xfs_attr_leaf_split_before(state->args); So here we walk up through a tree, doing splits as necessary. In this particular case, we have the potential attr leaf double-split case described above. If this happens, we set extrablk and use_extra. Is it the case that if this occurs, we know the loop is good for at least one more iteration (since this is a leaf block)? If so, we get to a node block that might be the root and call xfs_da3_node_split(). That function potentially splits the node block and adds the entries for the previous split, "consuming" extrablk if it had been set as well. In fact, if the node/root doesn't split, we don't carry on to any of the below code at all since addblk is set to NULL (even if the double split had occurred). Given that, I'm not seeing how extrablk should be used by the following root split code at all under normal circumstances. Wouldn't it always be handled before that point? To frame the question another way, if the double split is only a leaf block operation, shouldn't only addblk be relevant to the root block split code? I could very easily be missing something here... Brian > @@ -454,8 +456,13 @@ xfs_da3_split( > /* > * Update pointers to the node which used to be block 0 and > * just got bumped because of the addition of a new root node. > - * There might be three blocks involved if a double split occurred, > - * and the original block 0 could be at any position in the list. > + * There might be three blocks involved if a double split occurred and > + * in this case use_extra will be set. This can only happen for attr > + * trees, as extrablk for dir trees can only contain data or free space > + * blocks, never leaf/node blocks. > + * > + * Note that the original block 0 could be at any position in the list > + * of blocks in the tree. > * > * Note: the magic numbers and sibling pointers are in the same > * physical place for both v2 and v3 headers (by design). Hence it > @@ -464,31 +471,37 @@ xfs_da3_split( > */ > node = oldblk->bp->b_addr; > if (node->hdr.info.forw) { > + bp = NULL; > if (be32_to_cpu(node->hdr.info.forw) == addblk->blkno) { > bp = addblk->bp; > - } else { > + } else if (use_extra) { > ASSERT(state->extravalid); > bp = state->extrablk.bp; > } > - node = bp->b_addr; > - node->hdr.info.back = cpu_to_be32(oldblk->blkno); > - xfs_trans_log_buf(state->args->trans, bp, > - XFS_DA_LOGRANGE(node, &node->hdr.info, > - sizeof(node->hdr.info))); > + if (bp) { > + node = bp->b_addr; > + node->hdr.info.back = cpu_to_be32(oldblk->blkno); > + xfs_trans_log_buf(state->args->trans, bp, > + XFS_DA_LOGRANGE(node, &node->hdr.info, > + sizeof(node->hdr.info))); > + } > } > node = oldblk->bp->b_addr; > if (node->hdr.info.back) { > + bp = NULL; > if (be32_to_cpu(node->hdr.info.back) == addblk->blkno) { > bp = addblk->bp; > } else { > ASSERT(state->extravalid); > bp = state->extrablk.bp; > } > - node = bp->b_addr; > - node->hdr.info.forw = cpu_to_be32(oldblk->blkno); > - xfs_trans_log_buf(state->args->trans, bp, > - XFS_DA_LOGRANGE(node, &node->hdr.info, > - sizeof(node->hdr.info))); > + if (bp) { > + node = bp->b_addr; > + node->hdr.info.forw = cpu_to_be32(oldblk->blkno); > + xfs_trans_log_buf(state->args->trans, bp, > + XFS_DA_LOGRANGE(node, &node->hdr.info, > + sizeof(node->hdr.info))); > + } > } > addblk->bp = NULL; > return 0; > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Tue Jan 5 12:34:11 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id BE93A7F50 for ; Tue, 5 Jan 2016 12:34:11 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9488A8F8035 for ; Tue, 5 Jan 2016 10:34:11 -0800 (PST) X-ASG-Debug-ID: 1452018850-04bdf03afc01e50001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id twwBAOeqJVo4Ha2Z (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 10:34:10 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id F2D3EC077523; Tue, 5 Jan 2016 18:34:09 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05IY9Jn009335; Tue, 5 Jan 2016 13:34:09 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 53752124F12; Tue, 5 Jan 2016 13:34:08 -0500 (EST) Date: Tue, 5 Jan 2016 13:34:08 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 7/9] libxfs: don't discard dirty buffers Message-ID: <20160105183407.GC38749@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 7/9] libxfs: don't discard dirty buffers References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-8-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-8-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452018850 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:07AM +1100, Dave Chinner wrote: > From: Dave Chinner > > When we release a buffer from the cache, if it is dirty we wite it > to disk then put the buffer on the free list for recycling. However, > if the write fails (e.g. verifier failure due unfixed corruption) we > effectively throw the buffer and it contents away. This causes all > sorts of problems for xfs_repair as it then re-reads the buffer from > disk on the next access and hence loses all the corrections that had > previously been made, resulting in tripping over corruptions in code > that assumes the corruptions have already been fixed/flagged in the > buffer it receives. > > TO fix this, we have to make the cache aware that writes can fail, > and keep the buffer in cache when writes fail. Hence we have to add > an explicit error notification to the flush operation, and we need > to do that before we release the buffer to the free list. This also > means that we need to remove the writeback code from the release > mechanisms, instead replacing them with assertions that the buffers > are already clean. > > Signed-off-by: Dave Chinner > --- Reviewed-by: Brian Foster > include/cache.h | 2 +- > libxfs/cache.c | 15 ++++++++++++++- > libxfs/rdwr.c | 44 +++++++++++++++++++++++++++----------------- > 3 files changed, 42 insertions(+), 19 deletions(-) > > diff --git a/include/cache.h b/include/cache.h > index 0a84c69..87826be 100644 > --- a/include/cache.h > +++ b/include/cache.h > @@ -64,7 +64,7 @@ typedef void *cache_key_t; > > typedef void (*cache_walk_t)(struct cache_node *); > typedef struct cache_node * (*cache_node_alloc_t)(cache_key_t); > -typedef void (*cache_node_flush_t)(struct cache_node *); > +typedef int (*cache_node_flush_t)(struct cache_node *); > typedef void (*cache_node_relse_t)(struct cache_node *); > typedef unsigned int (*cache_node_hash_t)(cache_key_t, unsigned int, > unsigned int); > diff --git a/libxfs/cache.c b/libxfs/cache.c > index 4753a1d..a48ebd9 100644 > --- a/libxfs/cache.c > +++ b/libxfs/cache.c > @@ -219,6 +219,12 @@ cache_shake( > if (pthread_mutex_trylock(&node->cn_mutex) != 0) > continue; > > + /* can't release dirty objects */ > + if (cache->flush(node)) { > + pthread_mutex_unlock(&node->cn_mutex); > + continue; > + } > + > hash = cache->c_hash + node->cn_hashidx; > if (pthread_mutex_trylock(&hash->ch_mutex) != 0) { > pthread_mutex_unlock(&node->cn_mutex); > @@ -311,6 +317,13 @@ __cache_node_purge( > pthread_mutex_unlock(&node->cn_mutex); > return count; > } > + > + /* can't purge dirty objects */ > + if (cache->flush(node)) { > + pthread_mutex_unlock(&node->cn_mutex); > + return 1; > + } > + > mru = &cache->c_mrus[node->cn_priority]; > pthread_mutex_lock(&mru->cm_mutex); > list_del_init(&node->cn_mru); > @@ -321,7 +334,7 @@ __cache_node_purge( > pthread_mutex_destroy(&node->cn_mutex); > list_del_init(&node->cn_hash); > cache->relse(node); > - return count; > + return 0; > } > > /* > diff --git a/libxfs/rdwr.c b/libxfs/rdwr.c > index 7a04985..0337a21 100644 > --- a/libxfs/rdwr.c > +++ b/libxfs/rdwr.c > @@ -659,6 +659,8 @@ __libxfs_getbufr(int blen) > bp = kmem_zone_zalloc(xfs_buf_zone, 0); > pthread_mutex_unlock(&xfs_buf_freelist.cm_mutex); > bp->b_ops = NULL; > + if (bp->b_flags & LIBXFS_B_DIRTY) > + fprintf(stderr, "found dirty buffer (bulk) on free list!"); > > return bp; > } > @@ -1223,23 +1225,26 @@ libxfs_iomove(xfs_buf_t *bp, uint boff, int len, void *data, int flags) > } > > static void > -libxfs_brelse(struct cache_node *node) > +libxfs_brelse( > + struct cache_node *node) > { > - xfs_buf_t *bp = (xfs_buf_t *)node; > + struct xfs_buf *bp = (struct xfs_buf *)node; > > - if (bp != NULL) { > - if (bp->b_flags & LIBXFS_B_DIRTY) > - libxfs_writebufr(bp); > - pthread_mutex_lock(&xfs_buf_freelist.cm_mutex); > - list_add(&bp->b_node.cn_mru, &xfs_buf_freelist.cm_list); > - pthread_mutex_unlock(&xfs_buf_freelist.cm_mutex); > - } > + if (!bp) > + return; > + if (bp->b_flags & LIBXFS_B_DIRTY) > + fprintf(stderr, > + "releasing dirty buffer to free list!"); > + > + pthread_mutex_lock(&xfs_buf_freelist.cm_mutex); > + list_add(&bp->b_node.cn_mru, &xfs_buf_freelist.cm_list); > + pthread_mutex_unlock(&xfs_buf_freelist.cm_mutex); > } > > static unsigned int > libxfs_bulkrelse( > - struct cache *cache, > - struct list_head *list) > + struct cache *cache, > + struct list_head *list) > { > xfs_buf_t *bp; > int count = 0; > @@ -1249,7 +1254,8 @@ libxfs_bulkrelse( > > list_for_each_entry(bp, list, b_node.cn_mru) { > if (bp->b_flags & LIBXFS_B_DIRTY) > - libxfs_writebufr(bp); > + fprintf(stderr, > + "releasing dirty buffer (bulk) to free list!"); > count++; > } > > @@ -1260,18 +1266,22 @@ libxfs_bulkrelse( > return count; > } > > -static void > -libxfs_bflush(struct cache_node *node) > +static int > +libxfs_bflush( > + struct cache_node *node) > { > - xfs_buf_t *bp = (xfs_buf_t *)node; > + struct xfs_buf *bp = (struct xfs_buf *)node; > > - if ((bp != NULL) && (bp->b_flags & LIBXFS_B_DIRTY)) > - libxfs_writebufr(bp); > + if (bp->b_flags & LIBXFS_B_DIRTY) > + return libxfs_writebufr(bp); > + return 0; > } > > void > libxfs_putbufr(xfs_buf_t *bp) > { > + if (bp->b_flags & LIBXFS_B_DIRTY) > + libxfs_writebufr(bp); > libxfs_brelse((struct cache_node *)bp); > } > > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Tue Jan 5 12:34:20 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 677E47F37 for ; Tue, 5 Jan 2016 12:34:20 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id DF765AC006 for ; Tue, 5 Jan 2016 10:34:16 -0800 (PST) X-ASG-Debug-ID: 1452018855-04cbb07e1a021c0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 3smDhsPW0gU1Qvaz (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 10:34:15 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 0E1098F265; Tue, 5 Jan 2016 18:34:15 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05IYEXC009393; Tue, 5 Jan 2016 13:34:14 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 6C27B124F12; Tue, 5 Jan 2016 13:34:13 -0500 (EST) Date: Tue, 5 Jan 2016 13:34:13 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 8/9] libxfs: don't repeatedly shake unwritable buffers Message-ID: <20160105183413.GD38749@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 8/9] libxfs: don't repeatedly shake unwritable buffers References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-9-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-9-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452018855 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:08AM +1100, Dave Chinner wrote: > From: Dave Chinner > > now that we try to write dirty buffers before we release them, we > can get buildup of unwritable dirty buffers on the LRU lists, This > results in the cache shaker repeatedly trying to write out these > buffers every time the cache fills up. This results in more > corruption warnings, and takes up a lot of time doing reclaiming > nothing. This can effectively livelock the processing parts of phase > 4. > > Fix this by not trying to write buffers with corruption errors on > them. These errors will get cleared when the buffer is re-read and > fixed and them marked dirty again. At which point, we'll be able to > write them and so the cache can reclaim them successfully. > > Signed-off-by: Dave Chinner > --- Reviewed-by: Brian Foster > libxfs/rdwr.c | 27 ++++++++++++++++----------- > 1 file changed, 16 insertions(+), 11 deletions(-) > > diff --git a/libxfs/rdwr.c b/libxfs/rdwr.c > index 0337a21..a1f0029 100644 > --- a/libxfs/rdwr.c > +++ b/libxfs/rdwr.c > @@ -1103,7 +1103,6 @@ int > libxfs_writebufr(xfs_buf_t *bp) > { > int fd = libxfs_device_to_fd(bp->b_target->dev); > - int error = 0; > > /* > * we never write buffers that are marked stale. This indicates they > @@ -1134,7 +1133,7 @@ libxfs_writebufr(xfs_buf_t *bp) > } > > if (!(bp->b_flags & LIBXFS_B_DISCONTIG)) { > - error = __write_buf(fd, bp->b_addr, bp->b_bcount, > + bp->b_error = __write_buf(fd, bp->b_addr, bp->b_bcount, > LIBXFS_BBTOOFF64(bp->b_bn), bp->b_flags); > } else { > int i; > @@ -1144,11 +1143,10 @@ libxfs_writebufr(xfs_buf_t *bp) > off64_t offset = LIBXFS_BBTOOFF64(bp->b_map[i].bm_bn); > int len = BBTOB(bp->b_map[i].bm_len); > > - error = __write_buf(fd, buf, len, offset, bp->b_flags); > - if (error) { > - bp->b_error = error; > + bp->b_error = __write_buf(fd, buf, len, offset, > + bp->b_flags); > + if (bp->b_error) > break; > - } > buf += len; > } > } > @@ -1157,14 +1155,14 @@ libxfs_writebufr(xfs_buf_t *bp) > printf("%lx: %s: wrote %u bytes, blkno=%llu(%llu), %p, error %d\n", > pthread_self(), __FUNCTION__, bp->b_bcount, > (long long)LIBXFS_BBTOOFF64(bp->b_bn), > - (long long)bp->b_bn, bp, error); > + (long long)bp->b_bn, bp, bp->b_error); > #endif > - if (!error) { > + if (!bp->b_error) { > bp->b_flags |= LIBXFS_B_UPTODATE; > bp->b_flags &= ~(LIBXFS_B_DIRTY | LIBXFS_B_EXIT | > LIBXFS_B_UNCHECKED); > } > - return error; > + return bp->b_error; > } > > int > @@ -1266,15 +1264,22 @@ libxfs_bulkrelse( > return count; > } > > +/* > + * When a buffer is marked dirty, the error is cleared. Hence if we are trying > + * to flush a buffer prior to cache reclaim that has an error on it it means > + * we've already tried to flush it and it failed. Prevent repeated corruption > + * errors from being reported by skipping such buffers - when the corruption is > + * fixed the buffer will be marked dirty again and we can write it again. > + */ > static int > libxfs_bflush( > struct cache_node *node) > { > struct xfs_buf *bp = (struct xfs_buf *)node; > > - if (bp->b_flags & LIBXFS_B_DIRTY) > + if (!bp->b_error && bp->b_flags & LIBXFS_B_DIRTY) > return libxfs_writebufr(bp); > - return 0; > + return bp->b_error; > } > > void > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Tue Jan 5 12:34:20 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E72F929DF5 for ; Tue, 5 Jan 2016 12:34:20 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id D023E304066 for ; Tue, 5 Jan 2016 10:34:20 -0800 (PST) X-ASG-Debug-ID: 1452018859-04cbb07e19021c0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id yc5JY0Q4EjZkWuOc (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 10:34:19 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id 5E54742E5C2; Tue, 5 Jan 2016 18:34:19 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-121.bos.redhat.com [10.18.41.121]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05IYJjh030973; Tue, 5 Jan 2016 13:34:19 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id D68AC124F12; Tue, 5 Jan 2016 13:34:17 -0500 (EST) Date: Tue, 5 Jan 2016 13:34:17 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 9/9] libxfs: keep unflushable buffers off the cache MRUs Message-ID: <20160105183417.GE38749@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 9/9] libxfs: keep unflushable buffers off the cache MRUs References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-10-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1450733829-9319-10-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452018859 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Dec 22, 2015 at 08:37:09AM +1100, Dave Chinner wrote: > From: Dave Chinner > > There's no point trying to free buffers that are dirty and return > errors on flush as we have to keep them around until the corruption > is fixed. Hence if we fail to flush an inode during a cache shake, > move the buffer to a special dirty MRU list that the cache does not > shake. This prevents memory pressure from seeing these buffers, but > allows subsequent cache lookups to still find them through the hash. > This ensures we don't waste huge amounts of CPU trying to flush and > reclaim buffers that canot be flushed or reclaimed. > > Signed-off-by: Dave Chinner > --- > include/cache.h | 3 ++- > libxfs/cache.c | 34 +++++++++++++++++++++++++++++----- > 2 files changed, 31 insertions(+), 6 deletions(-) > ... > diff --git a/libxfs/cache.c b/libxfs/cache.c > index a48ebd9..d5ea461 100644 > --- a/libxfs/cache.c > +++ b/libxfs/cache.c ... > @@ -202,10 +223,11 @@ cache_shake( > struct cache_node * node; > unsigned int count; > > - ASSERT(priority <= CACHE_MAX_PRIORITY); > - if (priority > CACHE_MAX_PRIORITY) > + ASSERT(priority <= CACHE_DIRTY_PRIORITY); > + if (priority > CACHE_MAX_PRIORITY && !all) > priority = 0; > > + Extra newline. FWIW, it also looks like the only cache_shake() caller where all == 0 already prevents calling with priority > CACHE_MAX_PRIORITY. I think a brief comment in one or both places with regard to why >max priority is skipped unless 'all == 1' would be good, though. Also, it looks like the loop in cache_report() could be updated to dump the dirty priority mru entry count. Finally, what happens once a buffer on the dirty mru is fully repaired, rewritten and released? Is it placed right back on the "unshakeable" dirty mru or is cn_priority updated somewhere? On further digging, it looks like a subsequent buffer lookup (__cache_lookup()) drops the priority, though it appears to be designed to deal with prefetched buffers and the associated B_INODE..B_DIR_BMAP mappings. Brian > mru = &cache->c_mrus[priority]; > count = 0; > list_head_init(&temp); > @@ -221,6 +243,8 @@ cache_shake( > > /* can't release dirty objects */ > if (cache->flush(node)) { > + cache_move_to_dirty_mru(cache, node); > + mru->cm_count--; > pthread_mutex_unlock(&node->cn_mutex); > continue; > } > @@ -578,7 +602,7 @@ cache_purge( > { > int i; > > - for (i = 0; i <= CACHE_MAX_PRIORITY; i++) > + for (i = 0; i <= CACHE_DIRTY_PRIORITY; i++) > cache_shake(cache, i, 1); > > #ifdef CACHE_DEBUG > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From sandeen@sandeen.net Tue Jan 5 13:01:58 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id A5F1B7F37 for ; Tue, 5 Jan 2016 13:01:58 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 19B5FAC003 for ; Tue, 5 Jan 2016 11:01:54 -0800 (PST) X-ASG-Debug-ID: 1452020509-04cb6c42e602fa0001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 22HpYFETnYZxvPBg for ; Tue, 05 Jan 2016 11:01:49 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id 4ED3663C5FF0 for ; Tue, 5 Jan 2016 13:01:49 -0600 (CST) Subject: [PATCH V3] xfs: eliminate committed arg from xfs_bmap_finish To: xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH V3] xfs: eliminate committed arg from xfs_bmap_finish References: <56441B8E.6070603@redhat.com> <5644BEF8.6070201@sandeen.net> From: Eric Sandeen X-Enigmail-Draft-Status: N1110 Message-ID: <568C131C.9080907@sandeen.net> Date: Tue, 5 Jan 2016 13:01:48 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <5644BEF8.6070201@sandeen.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452020509 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25858 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Calls to xfs_bmap_finish() and xfs_trans_ijoin(), and the associated comments were replicated several times across the attribute code, all dealing with what to do if the transaction was or wasn't committed. And in that replicated code, an ASSERT() test of an uninitialized variable occurs in several locations: error = xfs_attr_thing(&args); if (!error) { error = xfs_bmap_finish(&args.trans, args.flist, &committed); } if (error) { ASSERT(committed); If the first xfs_attr_thing() failed, we'd skip the xfs_bmap_finish, never set "committed", and then test it in the ASSERT. Fix this up by moving the committed state internal to xfs_bmap_finish, and add a new inode argument. If an inode is passed in, it is passed through to __xfs_trans_roll() and joined to the transaction there if the transaction was committed. xfs_qm_dqalloc() was a little unique in that it called bjoin rather than ijoin, but as Dave points out we can detect the committed state but checking whether (*tpp != tp). Addresses-Coverity-Id: 102360 Addresses-Coverity-Id: 102361 Addresses-Coverity-Id: 102363 Addresses-Coverity-Id: 102364 Signed-off-by: Eric Sandeen --- libxfs/xfs_attr.c | 114 +++++------------------------------------------ libxfs/xfs_attr_remote.c | 25 ---------- libxfs/xfs_bmap.c | 6 -- libxfs/xfs_bmap.h | 2 xfs_bmap_util.c | 28 ++++------- xfs_dquot.c | 12 ++-- xfs_inode.c | 22 ++------- xfs_iomap.c | 10 +--- xfs_rtalloc.c | 3 - xfs_symlink.c | 12 ---- 10 files changed, 50 insertions(+), 184 deletions(-) diff --git a/fs/xfs/libxfs/xfs_attr.c b/fs/xfs/libxfs/xfs_attr.c index f949818..e16aa32 100644 --- a/fs/xfs/libxfs/xfs_attr.c +++ b/fs/xfs/libxfs/xfs_attr.c @@ -207,7 +207,7 @@ xfs_attr_set( struct xfs_trans_res tres; xfs_fsblock_t firstblock; int rsvd = (flags & ATTR_ROOT) != 0; - int error, err2, committed, local; + int error, err2, local; XFS_STATS_INC(mp, xs_attr_set); @@ -335,24 +335,15 @@ xfs_attr_set( xfs_bmap_init(args.flist, args.firstblock); error = xfs_attr_shortform_to_leaf(&args); if (!error) { - error = xfs_bmap_finish(&args.trans, args.flist, - &committed); + error = xfs_bmap_finish(&args.trans, args.flist, dp); } if (error) { - ASSERT(committed); args.trans = NULL; xfs_bmap_cancel(&flist); goto out; } /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args.trans, dp, 0); - - /* * Commit the leaf transformation. We'll need another (linked) * transaction to add the new attribute to the leaf. */ @@ -568,7 +559,7 @@ xfs_attr_leaf_addname(xfs_da_args_t *args) { xfs_inode_t *dp; struct xfs_buf *bp; - int retval, error, committed, forkoff; + int retval, error, forkoff; trace_xfs_attr_leaf_addname(args); @@ -629,24 +620,15 @@ xfs_attr_leaf_addname(xfs_da_args_t *args) xfs_bmap_init(args->flist, args->firstblock); error = xfs_attr3_leaf_to_node(args); if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); + error = xfs_bmap_finish(&args->trans, args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); - - /* * Commit the current trans (including the inode) and start * a new one. */ @@ -731,23 +713,13 @@ xfs_attr_leaf_addname(xfs_da_args_t *args) /* bp is gone due to xfs_da_shrink_inode */ if (!error) { error = xfs_bmap_finish(&args->trans, - args->flist, - &committed); + args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } - - /* - * bmap_finish() may have committed the last trans - * and started a new one. We need the inode to be - * in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } /* @@ -775,7 +747,7 @@ xfs_attr_leaf_removename(xfs_da_args_t *args) { xfs_inode_t *dp; struct xfs_buf *bp; - int error, committed, forkoff; + int error, forkoff; trace_xfs_attr_leaf_removename(args); @@ -804,22 +776,13 @@ xfs_attr_leaf_removename(xfs_da_args_t *args) error = xfs_attr3_leaf_to_shortform(bp, args, forkoff); /* bp is gone due to xfs_da_shrink_inode */ if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); + error = xfs_bmap_finish(&args->trans, args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } - - /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } return 0; } @@ -877,7 +840,7 @@ xfs_attr_node_addname(xfs_da_args_t *args) xfs_da_state_blk_t *blk; xfs_inode_t *dp; xfs_mount_t *mp; - int committed, retval, error; + int retval, error; trace_xfs_attr_node_addname(args); @@ -940,25 +903,15 @@ restart: error = xfs_attr3_leaf_to_node(args); if (!error) { error = xfs_bmap_finish(&args->trans, - args->flist, - &committed); + args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } /* - * bmap_finish() may have committed the last trans - * and started a new one. We need the inode to be - * in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); - - /* * Commit the node conversion and start the next * trans in the chain. */ @@ -978,22 +931,13 @@ restart: xfs_bmap_init(args->flist, args->firstblock); error = xfs_da3_split(state); if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); + error = xfs_bmap_finish(&args->trans, args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } - - /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } else { /* * Addition succeeded, update Btree hashvals. @@ -1088,23 +1032,13 @@ restart: error = xfs_da3_join(state); if (!error) { error = xfs_bmap_finish(&args->trans, - args->flist, - &committed); + args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } - - /* - * bmap_finish() may have committed the last trans - * and started a new one. We need the inode to be - * in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } /* @@ -1146,7 +1080,7 @@ xfs_attr_node_removename(xfs_da_args_t *args) xfs_da_state_blk_t *blk; xfs_inode_t *dp; struct xfs_buf *bp; - int retval, error, committed, forkoff; + int retval, error, forkoff; trace_xfs_attr_node_removename(args); @@ -1221,23 +1155,13 @@ xfs_attr_node_removename(xfs_da_args_t *args) xfs_bmap_init(args->flist, args->firstblock); error = xfs_da3_join(state); if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); + error = xfs_bmap_finish(&args->trans, args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } - - /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); - /* * Commit the Btree join operation and start a new trans. */ @@ -1267,23 +1191,13 @@ xfs_attr_node_removename(xfs_da_args_t *args) /* bp is gone due to xfs_da_shrink_inode */ if (!error) { error = xfs_bmap_finish(&args->trans, - args->flist, - &committed); + args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } - - /* - * bmap_finish() may have committed the last trans - * and started a new one. We need the inode to be - * in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } else xfs_trans_brelse(args->trans, bp); } diff --git a/fs/xfs/libxfs/xfs_attr_remote.c b/fs/xfs/libxfs/xfs_attr_remote.c index 5ab95ff..04fa02f 100644 --- a/fs/xfs/libxfs/xfs_attr_remote.c +++ b/fs/xfs/libxfs/xfs_attr_remote.c @@ -447,8 +447,6 @@ xfs_attr_rmtval_set( * Roll through the "value", allocating blocks on disk as required. */ while (blkcnt > 0) { - int committed; - /* * Allocate a single extent, up to the size of the value. * @@ -467,23 +465,14 @@ xfs_attr_rmtval_set( blkcnt, XFS_BMAPI_ATTRFORK, args->firstblock, args->total, &map, &nmap, args->flist); if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); + error = xfs_bmap_finish(&args->trans, args->flist, dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } - /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); - ASSERT(nmap == 1); ASSERT((map.br_startblock != DELAYSTARTBLOCK) && (map.br_startblock != HOLESTARTBLOCK)); @@ -614,31 +603,21 @@ xfs_attr_rmtval_remove( blkcnt = args->rmtblkcnt; done = 0; while (!done) { - int committed; - xfs_bmap_init(args->flist, args->firstblock); error = xfs_bunmapi(args->trans, args->dp, lblkno, blkcnt, XFS_BMAPI_ATTRFORK, 1, args->firstblock, args->flist, &done); if (!error) { error = xfs_bmap_finish(&args->trans, args->flist, - &committed); + args->dp); } if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, args->dp, 0); - - /* * Close out trans and start the next one in the chain. */ error = xfs_trans_roll(&args->trans, args->dp); diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c index 119c242..f28fa3f 100644 --- a/fs/xfs/libxfs/xfs_bmap.c +++ b/fs/xfs/libxfs/xfs_bmap.c @@ -1111,7 +1111,6 @@ xfs_bmap_add_attrfork( xfs_trans_t *tp; /* transaction pointer */ int blks; /* space reservation */ int version = 1; /* superblock attr version */ - int committed; /* xaction was committed */ int logflags; /* logging flags */ int error; /* error return value */ @@ -1214,7 +1213,7 @@ xfs_bmap_add_attrfork( xfs_log_sb(tp); } - error = xfs_bmap_finish(&tp, &flist, &committed); + error = xfs_bmap_finish(&tp, &flist, NULL); if (error) goto bmap_cancel; error = xfs_trans_commit(tp); @@ -5950,7 +5949,6 @@ xfs_bmap_split_extent( struct xfs_trans *tp; struct xfs_bmap_free free_list; xfs_fsblock_t firstfsb; - int committed; int error; tp = xfs_trans_alloc(mp, XFS_TRANS_DIOSTRAT); @@ -5971,7 +5969,7 @@ xfs_bmap_split_extent( if (error) goto out; - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out; diff --git a/fs/xfs/libxfs/xfs_bmap.h b/fs/xfs/libxfs/xfs_bmap.h index a160f8a..423a34e 100644 --- a/fs/xfs/libxfs/xfs_bmap.h +++ b/fs/xfs/libxfs/xfs_bmap.h @@ -195,7 +195,7 @@ void xfs_bmap_add_free(xfs_fsblock_t bno, xfs_filblks_t len, struct xfs_bmap_free *flist, struct xfs_mount *mp); void xfs_bmap_cancel(struct xfs_bmap_free *flist); int xfs_bmap_finish(struct xfs_trans **tp, struct xfs_bmap_free *flist, - int *committed); + struct xfs_inode *ip); void xfs_bmap_compute_maxlevels(struct xfs_mount *mp, int whichfork); int xfs_bmap_first_unused(struct xfs_trans *tp, struct xfs_inode *ip, xfs_extlen_t len, xfs_fileoff_t *unused, int whichfork); diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c index dbae649..c2a3b5f 100644 --- a/fs/xfs/xfs_bmap_util.c +++ b/fs/xfs/xfs_bmap_util.c @@ -91,32 +91,32 @@ xfs_zero_extent( * last due to locking considerations. We never free any extents in * the first transaction. * - * Return 1 if the given transaction was committed and a new one - * started, and 0 otherwise in the committed parameter. + * If an inode *ip is provided, rejoin it to the transaction if + * the transaction was committed. */ int /* error */ xfs_bmap_finish( struct xfs_trans **tp, /* transaction pointer addr */ struct xfs_bmap_free *flist, /* i/o: list extents to free */ - int *committed)/* xact committed or not */ + xfs_inode_t *ip) { struct xfs_efd_log_item *efd; /* extent free data */ struct xfs_efi_log_item *efi; /* extent free intention */ int error; /* error return value */ + int committed;/* xact committed or not */ struct xfs_bmap_free_item *free; /* free extent item */ struct xfs_bmap_free_item *next; /* next item on free list */ ASSERT((*tp)->t_flags & XFS_TRANS_PERM_LOG_RES); - if (flist->xbf_count == 0) { - *committed = 0; + if (flist->xbf_count == 0) return 0; - } + efi = xfs_trans_get_efi(*tp, flist->xbf_count); for (free = flist->xbf_first; free; free = free->xbfi_next) xfs_trans_log_efi_extent(*tp, efi, free->xbfi_startblock, free->xbfi_blockcount); - error = __xfs_trans_roll(tp, NULL, committed); + error = __xfs_trans_roll(tp, ip, &committed); if (error) { /* * If the transaction was committed, drop the EFD reference @@ -128,16 +128,13 @@ xfs_bmap_finish( * transaction so we should return committed=1 even though we're * returning an error. */ - if (*committed) { + if (committed) { xfs_efi_release(efi); xfs_force_shutdown((*tp)->t_mountp, (error == -EFSCORRUPTED) ? SHUTDOWN_CORRUPT_INCORE : SHUTDOWN_META_IO_ERROR); - } else { - *committed = 1; } - return error; } @@ -969,7 +966,6 @@ xfs_alloc_file_space( xfs_bmbt_irec_t imaps[1], *imapp; xfs_bmap_free_t free_list; uint qblocks, resblks, resrtextents; - int committed; int error; trace_xfs_alloc_file_space(ip); @@ -1071,7 +1067,7 @@ xfs_alloc_file_space( /* * Complete the transaction */ - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) { goto error0; } @@ -1206,7 +1202,6 @@ xfs_free_file_space( xfs_off_t offset, xfs_off_t len) { - int committed; int done; xfs_fileoff_t endoffset_fsb; int error; @@ -1353,7 +1348,7 @@ xfs_free_file_space( /* * complete the transaction */ - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) { goto error0; } @@ -1434,7 +1429,6 @@ xfs_shift_file_space( int error; struct xfs_bmap_free free_list; xfs_fsblock_t first_block; - int committed; xfs_fileoff_t stop_fsb; xfs_fileoff_t next_fsb; xfs_fileoff_t shift_fsb; @@ -1526,7 +1520,7 @@ xfs_shift_file_space( if (error) goto out_bmap_cancel; - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c index 7ac6c5c..d77ba0c 100644 --- a/fs/xfs/xfs_dquot.c +++ b/fs/xfs/xfs_dquot.c @@ -306,7 +306,7 @@ xfs_qm_dqalloc( xfs_fsblock_t firstblock; xfs_bmap_free_t flist; xfs_bmbt_irec_t map; - int nmaps, error, committed; + int nmaps, error; xfs_buf_t *bp; xfs_trans_t *tp = *tpp; @@ -379,11 +379,11 @@ xfs_qm_dqalloc( xfs_trans_bhold(tp, bp); - if ((error = xfs_bmap_finish(tpp, &flist, &committed))) { + if ((error = xfs_bmap_finish(tpp, &flist, NULL))) goto error1; - } - if (committed) { + /* Transaction was committed? */ + if (*tpp != tp) { tp = *tpp; xfs_trans_bjoin(tp, bp); } else { @@ -393,9 +393,9 @@ xfs_qm_dqalloc( *O_bpp = bp; return 0; - error1: +error1: xfs_bmap_cancel(&flist); - error0: +error0: xfs_iunlock(quotip, XFS_ILOCK_EXCL); return error; diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index 8ee3939..0116dd5 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -1143,7 +1143,6 @@ xfs_create( xfs_bmap_free_t free_list; xfs_fsblock_t first_block; bool unlock_dp_on_error = false; - int committed; prid_t prid; struct xfs_dquot *udqp = NULL; struct xfs_dquot *gdqp = NULL; @@ -1226,7 +1225,7 @@ xfs_create( * pointing to itself. */ error = xfs_dir_ialloc(&tp, dp, mode, is_dir ? 2 : 1, rdev, - prid, resblks > 0, &ip, &committed); + prid, resblks > 0, &ip, NULL); if (error) goto out_trans_cancel; @@ -1275,7 +1274,7 @@ xfs_create( */ xfs_qm_vop_create_dqattach(tp, ip, udqp, gdqp, pdqp); - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; @@ -1427,7 +1426,6 @@ xfs_link( int error; xfs_bmap_free_t free_list; xfs_fsblock_t first_block; - int committed; int resblks; trace_xfs_link(tdp, target_name); @@ -1506,7 +1504,7 @@ xfs_link( xfs_trans_set_sync(tp); } - error = xfs_bmap_finish (&tp, &free_list, &committed); + error = xfs_bmap_finish (&tp, &free_list, NULL); if (error) { xfs_bmap_cancel(&free_list); goto error_return; @@ -1555,7 +1553,6 @@ xfs_itruncate_extents( xfs_fileoff_t first_unmap_block; xfs_fileoff_t last_block; xfs_filblks_t unmap_len; - int committed; int error = 0; int done = 0; @@ -1601,9 +1598,7 @@ xfs_itruncate_extents( * Duplicate the transaction that has the permanent * reservation and commit the old transaction. */ - error = xfs_bmap_finish(&tp, &free_list, &committed); - if (committed) - xfs_trans_ijoin(tp, ip, 0); + error = xfs_bmap_finish(&tp, &free_list, ip); if (error) goto out_bmap_cancel; @@ -1774,7 +1769,6 @@ xfs_inactive_ifree( { xfs_bmap_free_t free_list; xfs_fsblock_t first_block; - int committed; struct xfs_mount *mp = ip->i_mount; struct xfs_trans *tp; int error; @@ -1841,7 +1835,7 @@ xfs_inactive_ifree( * Just ignore errors at this point. There is nothing we can do except * to try to keep going. Make sure it's not a silent error. */ - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) { xfs_notice(mp, "%s: xfs_bmap_finish returned error %d", __func__, error); @@ -2523,7 +2517,6 @@ xfs_remove( int error = 0; xfs_bmap_free_t free_list; xfs_fsblock_t first_block; - int committed; uint resblks; trace_xfs_remove(dp, name); @@ -2624,7 +2617,7 @@ xfs_remove( if (mp->m_flags & (XFS_MOUNT_WSYNC|XFS_MOUNT_DIRSYNC)) xfs_trans_set_sync(tp); - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; @@ -2701,7 +2694,6 @@ xfs_finish_rename( struct xfs_trans *tp, struct xfs_bmap_free *free_list) { - int committed = 0; int error; /* @@ -2711,7 +2703,7 @@ xfs_finish_rename( if (tp->t_mountp->m_flags & (XFS_MOUNT_WSYNC|XFS_MOUNT_DIRSYNC)) xfs_trans_set_sync(tp); - error = xfs_bmap_finish(&tp, free_list, &committed); + error = xfs_bmap_finish(&tp, free_list, NULL); if (error) { xfs_bmap_cancel(free_list); xfs_trans_cancel(tp); diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c index f4f5b43..ffc7baf 100644 --- a/fs/xfs/xfs_iomap.c +++ b/fs/xfs/xfs_iomap.c @@ -129,7 +129,6 @@ xfs_iomap_write_direct( xfs_trans_t *tp; xfs_bmap_free_t free_list; uint qblocks, resblks, resrtextents; - int committed; int error; int lockmode; int bmapi_flags = XFS_BMAPI_PREALLOC; @@ -247,7 +246,7 @@ xfs_iomap_write_direct( /* * Complete the transaction */ - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; @@ -693,7 +692,7 @@ xfs_iomap_write_allocate( xfs_bmap_free_t free_list; xfs_filblks_t count_fsb; xfs_trans_t *tp; - int nimaps, committed; + int nimaps; int error = 0; int nres; @@ -794,7 +793,7 @@ xfs_iomap_write_allocate( if (error) goto trans_cancel; - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto trans_cancel; @@ -852,7 +851,6 @@ xfs_iomap_write_unwritten( xfs_bmap_free_t free_list; xfs_fsize_t i_size; uint resblks; - int committed; int error; trace_xfs_unwritten_convert(ip, offset, count); @@ -924,7 +922,7 @@ xfs_iomap_write_unwritten( xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE); } - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto error_on_bmapi_transaction; diff --git a/fs/xfs/xfs_rtalloc.c b/fs/xfs/xfs_rtalloc.c index ab1bac6..be02a68 100644 --- a/fs/xfs/xfs_rtalloc.c +++ b/fs/xfs/xfs_rtalloc.c @@ -766,7 +766,6 @@ xfs_growfs_rt_alloc( { xfs_fileoff_t bno; /* block number in file */ struct xfs_buf *bp; /* temporary buffer for zeroing */ - int committed; /* transaction committed flag */ xfs_daddr_t d; /* disk block address */ int error; /* error return value */ xfs_fsblock_t firstblock;/* first block allocated in xaction */ @@ -811,7 +810,7 @@ xfs_growfs_rt_alloc( /* * Free any blocks freed up in the transaction, then commit. */ - error = xfs_bmap_finish(&tp, &flist, &committed); + error = xfs_bmap_finish(&tp, &flist, NULL); if (error) goto out_bmap_cancel; error = xfs_trans_commit(tp); diff --git a/fs/xfs/xfs_symlink.c b/fs/xfs/xfs_symlink.c index 996481e..b44284c 100644 --- a/fs/xfs/xfs_symlink.c +++ b/fs/xfs/xfs_symlink.c @@ -178,7 +178,6 @@ xfs_symlink( struct xfs_bmap_free free_list; xfs_fsblock_t first_block; bool unlock_dp_on_error = false; - int committed; xfs_fileoff_t first_fsb; xfs_filblks_t fs_blocks; int nmaps; @@ -387,7 +386,7 @@ xfs_symlink( xfs_trans_set_sync(tp); } - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; @@ -434,7 +433,6 @@ xfs_inactive_symlink_rmt( struct xfs_inode *ip) { xfs_buf_t *bp; - int committed; int done; int error; xfs_fsblock_t first_block; @@ -510,16 +508,10 @@ xfs_inactive_symlink_rmt( /* * Commit the first transaction. This logs the EFI and the inode. */ - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, ip); if (error) goto error_bmap_cancel; /* - * The transaction must have been committed, since there were - * actually extents freed by xfs_bunmapi. See xfs_bmap_finish. - * The new tp has the extent freeing and EFDs. - */ - ASSERT(committed); - /* * The first xact was committed, so add the inode to the new one. * Mark it dirty so it will be logged and moved forward in the log as * part of every commit. From dan.j.williams@intel.com Tue Jan 5 13:59:45 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 4982E7F37 for ; Tue, 5 Jan 2016 13:59:45 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id BE067AC001 for ; Tue, 5 Jan 2016 11:59:44 -0800 (PST) X-ASG-Debug-ID: 1452023981-04cbb07e1a03ea0001-NocioJ Received: from mail-yk0-f170.google.com (mail-yk0-f170.google.com [209.85.160.170]) by cuda.sgi.com with ESMTP id 27wVoGeO90BoRLNq (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 05 Jan 2016 11:59:42 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.160.170 Received: by mail-yk0-f170.google.com with SMTP id a85so215935042ykb.1 for ; Tue, 05 Jan 2016 11:59:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qnDw6n1Fg1AiDhHp+TxF8E+S4YKPCw1AWi6IzboSmq8=; b=JAEHYsIrhDcB1wkcvcGq2z7da7Gj701VSvPaNmHwWwQsA5x7SL19JzHkNrBVz1sOy/ d2/PVD50DBNO6zwwBIS407QPUibtY7T1YZBZ4qCmv1sBxDhp5tu2HjUGxGvSvAqXsKSC x9Uv56iGmtB/bjiTTRAQFObkvxBwZGnc/3uWu+8DcX/QsxmTMz0tDGIvG+4lMPvevFv+ yBuk9NtvydCCLyLyO4VN42zvVNRCo/oLbRbKcYPRrZh8fJAHD+xrr4MreSk5Uoi6A9mi 0jxqScsOQVwTuF4m4ACQDzGb+nSW0UoSLm5UD3qE9lLTkYc1ykqRsFlrCB9bpcNtvQMD T8cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=qnDw6n1Fg1AiDhHp+TxF8E+S4YKPCw1AWi6IzboSmq8=; b=f9QcdPALi+0CzkJibMgZpf9dNKWu5vF8SLRkGNj9AXxse1zqYih2wl9fspcD/fQmaS HuGW1SwIPf8FOb+FVbP70UEmUfsxBUOqQJIZoLMPjf5D4eMXoe74gHwn2PR0maQR65wD YXpcMAM9Y8kwJraFUSfuAToxClBWQnI+zMXbii9eBvid8h1lHEAmwSQDsVMTN79J7kP4 dSh2jreHaw4V2C6OM1zlv6ivoA92Xsxs5ej04KKFdXQ6Th7nEvk+KwKGS57eO72kKaiC MxISc9SOwn7laGLKjLza3EnxOwCB/QhO8uXDTkTmCO+nQFbND7fqKg2JDE/LV0YOQEFF w13Q== X-Gm-Message-State: ALoCoQniw2XkJ4LTIhK7IyB2vITmx9w3uWEG8v/Z8+Yz0akQ6FBQm1Ff067TyMYtXscvJV3crMK2SHwBYVESud24ETQJ0LtdrXNn3CjWlRIhmZG52KidPcI= MIME-Version: 1.0 X-Received: by 10.129.85.83 with SMTP id j80mr68671006ywb.28.1452023981772; Tue, 05 Jan 2016 11:59:41 -0800 (PST) Received: by 10.37.202.20 with HTTP; Tue, 5 Jan 2016 11:59:41 -0800 (PST) In-Reply-To: <20160105042346.GL19802@dastard> References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182016.24118.33718.stgit@dwillia2-desk3.amr.corp.intel.com> <20160105042346.GL19802@dastard> Date: Tue, 5 Jan 2016 11:59:41 -0800 Message-ID: Subject: Re: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty From: Dan Williams X-ASG-Orig-Subj: Re: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty To: Dave Chinner Cc: XFS Developers , linux-block@vger.kernel.org, "linux-nvdimm@lists.01.org" , Jens Axboe , linux-fsdevel , Jan Kara , Tejun Heo Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-yk0-f170.google.com[209.85.160.170] X-Barracuda-Start-Time: 1452023982 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25862 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Mon, Jan 4, 2016 at 8:23 PM, Dave Chinner wrote: > On Mon, Jan 04, 2016 at 10:20:16AM -0800, Dan Williams wrote: >> This warning was added as a debugging aid way back in commit >> 500b067c5e6c "writeback: check for registered bdi in flusher add and >> inode dirty" when we were switching over to per-bdi writeback. >> >> Once the block device has been torn down it's no longer useful to >> complain about unregistered bdi's. Clear the writeback capability under >> the the wb->list_lock(), so that __mark_inode_dirty has no opportunity >> to race bdi_unregister() to this WARN() condition. >> >> Alternatively we could just delete the warning... > > The warning is correct - the filesytem is trying to mark an inode > dirty on a device that can't do writeback anymore. Seems to me like > it is functioning as it should. > >> Found this while testing block device remove from underneath an active >> fs triggering traces like: >> >> WARNING: CPU: 6 PID: 2129 at fs/fs-writeback.c:2065 __mark_inode_dirty+0x261/0x350() >> bdi-block not registered >> [..] >> Call Trace: >> [] dump_stack+0x44/0x62 >> [] warn_slowpath_common+0x82/0xc0 >> [] warn_slowpath_fmt+0x5c/0x80 >> [] __mark_inode_dirty+0x261/0x350 >> [] generic_update_time+0x79/0xd0 >> [] file_update_time+0xbd/0x110 >> [] ext4_dax_fault+0x68/0x110 >> [] __do_fault+0x4e/0xf0 > > This seems like the problem to me - you haven't implemented a > shutdown hook for ext4, and so it continues to allow page faults to > make progress after the device has been removed. The DAX fault > should have been failed before the filesystem gets to the point of > marking the inode dirty.... xfs doesn't check that the fs is still alive before calling file_update_time(). Also, I don't think we need/want to sprinkle "is fs alive?" checks to address this when the block device shutdown path can simply turn off writeback. > >> + /* tell __mark_inode_dirty that writeback is no longer possible */ >> + spin_lock(&wb->list_lock); >> + wb->bdi->capabilities |= BDI_CAP_NO_WRITEBACK; >> + spin_unlock(&wb->list_lock); >> + >> spin_unlock_bh(&wb->work_lock); > > Is that lock ordering safe? i.e. it's inside a section using bh-safe > locking, which tends to imply that it can run from interrupt > contexts. Can we get something like This would be a problem if wb_shutdown() was called from softirq context, but it is always called from process context. So, ->list_lock doesn't currently need to be upgraded to spin_lock_bh and lockdep will trigger if this situation changes. From mchristi@redhat.com Tue Jan 5 14:53:48 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C64457F37 for ; Tue, 5 Jan 2016 14:53:48 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 657E4AC002 for ; Tue, 5 Jan 2016 12:53:48 -0800 (PST) X-ASG-Debug-ID: 1452027226-04cb6c42e605200001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id G1WtJOzSNdO6qmjp (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:53:47 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 03CE942E5D9; Tue, 5 Jan 2016 20:53:45 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krftx024032; Tue, 5 Jan 2016 15:53:41 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: [PATCH 00/35 v2] separate operations from flags in the bio/request structs Date: Tue, 5 Jan 2016 14:53:03 -0600 X-ASG-Orig-Subj: [PATCH 00/35 v2] separate operations from flags in the bio/request structs Message-Id: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027227 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 The following patches begin to cleanup the request->cmd_flags and bio->bi_rw mess. We currently use cmd_flags to specify the operation, attributes and state of the request. For bi_rw we use it for similar info and also the priority but then also have another bi_flags field for state. At some point, we abused them so much we just made cmd_flags 64 bits, so we could add more. The following patches seperate the operation (read, write discard, flush, etc) from cmd_flags/bi_rw. This patchset was made against linux-next from today Jan 5 2016. (git tag next-20160105). v2. 1. Dropped arguments from submit_bio, and had callers setup bio. 2. Add REQ_OP_FLUSH for request_fn users and renamed REQ_FLUSH to REQ_PREFLUSH for make_request_fn users. 3. Dropped bio/rq_data_dir functions, and added a op_is_write function instead. From mchristi@redhat.com Tue Jan 5 14:53:55 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id CDC1029DF9 for ; Tue, 5 Jan 2016 14:53:54 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 6F024AC002 for ; Tue, 5 Jan 2016 12:53:54 -0800 (PST) X-ASG-Debug-ID: 1452027229-04bdf03afc04da0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id i3LugJPawtItD13j (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:53:49 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id E4A9614A82; Tue, 5 Jan 2016 20:53:48 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu0024032; Tue, 5 Jan 2016 15:53:46 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio Date: Tue, 5 Jan 2016 14:53:04 -0600 X-ASG-Orig-Subj: [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio Message-Id: <1452027218-32303-2-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027229 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This has callers of submit_bio/submit_bio_wait set the bio->bi_rw instead of passing it in. This makes that use the same as generic_make_request and how we set the other bio fields. Signed-off-by: Mike Christie --- block/bio.c | 7 +++---- block/blk-core.c | 11 ++++------- block/blk-flush.c | 3 ++- block/blk-lib.c | 9 ++++++--- drivers/block/drbd/drbd_actlog.c | 2 +- drivers/block/drbd/drbd_bitmap.c | 4 ++-- drivers/block/floppy.c | 3 ++- drivers/block/xen-blkback/blkback.c | 4 +++- drivers/block/xen-blkfront.c | 4 ++-- drivers/md/bcache/debug.c | 6 ++++-- drivers/md/bcache/journal.c | 2 +- drivers/md/bcache/super.c | 4 ++-- drivers/md/dm-bufio.c | 3 ++- drivers/md/dm-io.c | 3 ++- drivers/md/dm-log-writes.c | 9 ++++++--- drivers/md/dm-thin.c | 3 ++- drivers/md/md.c | 10 +++++++--- drivers/md/raid1.c | 3 ++- drivers/md/raid10.c | 4 +++- drivers/md/raid5-cache.c | 7 ++++--- drivers/target/target_core_iblock.c | 24 +++++++++++++----------- fs/btrfs/check-integrity.c | 18 ++++++++++-------- fs/btrfs/check-integrity.h | 4 ++-- fs/btrfs/disk-io.c | 3 ++- fs/btrfs/extent_io.c | 7 ++++--- fs/btrfs/raid56.c | 16 +++++++++++----- fs/btrfs/scrub.c | 16 +++++++++++----- fs/btrfs/volumes.c | 14 +++++++------- fs/buffer.c | 3 ++- fs/direct-io.c | 3 ++- fs/ext4/crypto.c | 3 ++- fs/ext4/page-io.c | 3 ++- fs/ext4/readpage.c | 9 +++++---- fs/f2fs/data.c | 13 ++++++++----- fs/f2fs/segment.c | 6 ++++-- fs/gfs2/lops.c | 3 ++- fs/gfs2/meta_io.c | 3 ++- fs/gfs2/ops_fstype.c | 3 ++- fs/hfsplus/wrapper.c | 3 ++- fs/jfs/jfs_logmgr.c | 6 ++++-- fs/jfs/jfs_metapage.c | 10 ++++++---- fs/logfs/dev_bdev.c | 15 ++++++++++----- fs/mpage.c | 3 ++- fs/nfs/blocklayout/blocklayout.c | 22 ++++++++++++---------- fs/nilfs2/segbuf.c | 3 ++- fs/ocfs2/cluster/heartbeat.c | 12 +++++++----- fs/xfs/xfs_aops.c | 3 ++- fs/xfs/xfs_buf.c | 4 ++-- include/linux/bio.h | 2 +- include/linux/fs.h | 2 +- kernel/power/swap.c | 5 +++-- mm/page_io.c | 10 ++++++---- 52 files changed, 211 insertions(+), 141 deletions(-) diff --git a/block/bio.c b/block/bio.c index dbabd48..921112b 100644 --- a/block/bio.c +++ b/block/bio.c @@ -859,21 +859,20 @@ static void submit_bio_wait_endio(struct bio *bio) /** * submit_bio_wait - submit a bio, and wait until it completes - * @rw: whether to %READ or %WRITE, or maybe to %READA (read ahead) * @bio: The &struct bio which describes the I/O * * Simple wrapper around submit_bio(). Returns 0 on success, or the error from * bio_endio() on failure. */ -int submit_bio_wait(int rw, struct bio *bio) +int submit_bio_wait(struct bio *bio) { struct submit_bio_ret ret; - rw |= REQ_SYNC; init_completion(&ret.event); bio->bi_private = &ret; bio->bi_end_io = submit_bio_wait_endio; - submit_bio(rw, bio); + bio->bi_rw |= REQ_SYNC; + submit_bio(bio); wait_for_completion(&ret.event); return ret.error; diff --git a/block/blk-core.c b/block/blk-core.c index ab51685..9b887e3 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -2092,7 +2092,6 @@ EXPORT_SYMBOL(generic_make_request); /** * submit_bio - submit a bio to the block device layer for I/O - * @rw: whether to %READ or %WRITE, or maybe to %READA (read ahead) * @bio: The &struct bio which describes the I/O * * submit_bio() is very similar in purpose to generic_make_request(), and @@ -2100,10 +2099,8 @@ EXPORT_SYMBOL(generic_make_request); * interfaces; @bio must be presetup and ready for I/O. * */ -blk_qc_t submit_bio(int rw, struct bio *bio) +blk_qc_t submit_bio(struct bio *bio) { - bio->bi_rw |= rw; - /* * If it's a regular read/write or a barrier with data attached, * go through the normal accounting stuff before submission. @@ -2111,12 +2108,12 @@ blk_qc_t submit_bio(int rw, struct bio *bio) if (bio_has_data(bio)) { unsigned int count; - if (unlikely(rw & REQ_WRITE_SAME)) + if (unlikely(bio->bi_rw & REQ_WRITE_SAME)) count = bdev_logical_block_size(bio->bi_bdev) >> 9; else count = bio_sectors(bio); - if (rw & WRITE) { + if (bio->bi_rw & WRITE) { count_vm_events(PGPGOUT, count); } else { task_io_account_read(bio->bi_iter.bi_size); @@ -2127,7 +2124,7 @@ blk_qc_t submit_bio(int rw, struct bio *bio) char b[BDEVNAME_SIZE]; printk(KERN_DEBUG "%s(%d): %s block %Lu on %s (%u sectors)\n", current->comm, task_pid_nr(current), - (rw & WRITE) ? "WRITE" : "READ", + (bio->bi_rw & WRITE) ? "WRITE" : "READ", (unsigned long long)bio->bi_iter.bi_sector, bdevname(bio->bi_bdev, b), count); diff --git a/block/blk-flush.c b/block/blk-flush.c index 9c423e5..e092e13 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -484,8 +484,9 @@ int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, bio = bio_alloc(gfp_mask, 0); bio->bi_bdev = bdev; + bio->bi_rw |= WRITE_FLUSH; - ret = submit_bio_wait(WRITE_FLUSH, bio); + ret = submit_bio_wait(bio); /* * The driver must store the error location in ->bi_sector, if diff --git a/block/blk-lib.c b/block/blk-lib.c index 9ebf653..5292e30 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -102,13 +102,14 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, bio->bi_end_io = bio_batch_end_io; bio->bi_bdev = bdev; bio->bi_private = &bb; + bio->bi_rw |= type; bio->bi_iter.bi_size = req_sects << 9; nr_sects -= req_sects; sector = end_sect; atomic_inc(&bb.done); - submit_bio(type, bio); + submit_bio(bio); /* * We can loop for a long time in here, if someone does @@ -177,6 +178,7 @@ int blkdev_issue_write_same(struct block_device *bdev, sector_t sector, bio->bi_io_vec->bv_page = page; bio->bi_io_vec->bv_offset = 0; bio->bi_io_vec->bv_len = bdev_logical_block_size(bdev); + bio->bi_rw |= REQ_WRITE | REQ_WRITE_SAME; if (nr_sects > max_write_same_sectors) { bio->bi_iter.bi_size = max_write_same_sectors << 9; @@ -188,7 +190,7 @@ int blkdev_issue_write_same(struct block_device *bdev, sector_t sector, } atomic_inc(&bb.done); - submit_bio(REQ_WRITE | REQ_WRITE_SAME, bio); + submit_bio(bio); } /* Wait for bios in-flight */ @@ -238,6 +240,7 @@ static int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, bio->bi_bdev = bdev; bio->bi_end_io = bio_batch_end_io; bio->bi_private = &bb; + bio->bi_rw |= WRITE; while (nr_sects != 0) { sz = min((sector_t) PAGE_SIZE >> 9 , nr_sects); @@ -249,7 +252,7 @@ static int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, } ret = 0; atomic_inc(&bb.done); - submit_bio(WRITE, bio); + submit_bio(bio); } /* Wait for bios in-flight */ diff --git a/drivers/block/drbd/drbd_actlog.c b/drivers/block/drbd/drbd_actlog.c index 10459a1..6069e15 100644 --- a/drivers/block/drbd/drbd_actlog.c +++ b/drivers/block/drbd/drbd_actlog.c @@ -177,7 +177,7 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, if (drbd_insert_fault(device, (rw & WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) bio_io_error(bio); else - submit_bio(rw, bio); + submit_bio(bio); wait_until_done_or_force_detached(device, bdev, &device->md_io.done); if (!bio->bi_error) err = device->md_io.error; diff --git a/drivers/block/drbd/drbd_bitmap.c b/drivers/block/drbd/drbd_bitmap.c index 92d6fc0..ef44adb 100644 --- a/drivers/block/drbd/drbd_bitmap.c +++ b/drivers/block/drbd/drbd_bitmap.c @@ -1011,12 +1011,12 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho bio_add_page(bio, page, len, 0); bio->bi_private = ctx; bio->bi_end_io = drbd_bm_endio; + bio->bi_rw |= rw; if (drbd_insert_fault(device, (rw & WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) { - bio->bi_rw |= rw; bio_io_error(bio); } else { - submit_bio(rw, bio); + submit_bio(bio); /* this should not count as user activity and cause the * resync to throttle -- see drbd_rs_should_slow_down(). */ atomic_add(len >> 9, &device->rs_sect_ev); diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c index 9e25120..e4efcc5 100644 --- a/drivers/block/floppy.c +++ b/drivers/block/floppy.c @@ -3817,8 +3817,9 @@ static int __floppy_read_block_0(struct block_device *bdev, int drive) bio.bi_flags |= (1 << BIO_QUIET); bio.bi_private = &cbdata; bio.bi_end_io = floppy_rb0_cb; + bio.bi_rw |= READ; - submit_bio(READ, &bio); + submit_bio(&bio); process_fd_request(); init_completion(&cbdata.complete); diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index 41fb1a9..c7e89af 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -1348,6 +1348,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, bio->bi_private = pending_req; bio->bi_end_io = end_block_io_op; bio->bi_iter.bi_sector = preq.sector_number; + bio->bi_rw |= operation; } preq.sector_number += seg[i].nsec; @@ -1365,13 +1366,14 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, bio->bi_bdev = preq.bdev; bio->bi_private = pending_req; bio->bi_end_io = end_block_io_op; + bio->bi_rw |= operation; } atomic_set(&pending_req->pendcnt, nbio); blk_start_plug(&plug); for (i = 0; i < nbio; i++) - submit_bio(operation, biolist[i]); + submit_bio(biolist[i]); /* Let the I/Os go.. */ blk_finish_plug(&plug); diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 2fee2ee..81444fc 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -1749,7 +1749,7 @@ static int blkif_recover(struct blkfront_info *info) bio_trim(cloned_bio, offset, size); cloned_bio->bi_private = split_bio; cloned_bio->bi_end_io = split_bio_end; - submit_bio(cloned_bio->bi_rw, cloned_bio); + submit_bio(cloned_bio); } /* * Now we have to wait for all those smaller bios to @@ -1758,7 +1758,7 @@ static int blkif_recover(struct blkfront_info *info) continue; } /* We don't need to split this bio */ - submit_bio(bio->bi_rw, bio); + submit_bio(bio); } return 0; diff --git a/drivers/md/bcache/debug.c b/drivers/md/bcache/debug.c index 8b1f1d5..db68562 100644 --- a/drivers/md/bcache/debug.c +++ b/drivers/md/bcache/debug.c @@ -52,9 +52,10 @@ void bch_btree_verify(struct btree *b) bio->bi_bdev = PTR_CACHE(b->c, &b->key, 0)->bdev; bio->bi_iter.bi_sector = PTR_OFFSET(&b->key, 0); bio->bi_iter.bi_size = KEY_SIZE(&v->key) << 9; + bio->bi_rw |= REQ_META|READ_SYNC; bch_bio_map(bio, sorted); - submit_bio_wait(REQ_META|READ_SYNC, bio); + submit_bio_wait(bio); bch_bbio_free(bio, b->c); memcpy(ondisk, sorted, KEY_SIZE(&v->key) << 9); @@ -113,11 +114,12 @@ void bch_data_verify(struct cached_dev *dc, struct bio *bio) check = bio_clone(bio, GFP_NOIO); if (!check) return; + check->bi_rw |= READ_SYNC; if (bio_alloc_pages(check, GFP_NOIO)) goto out_put; - submit_bio_wait(READ_SYNC, check); + submit_bio_wait(check); bio_for_each_segment(bv, bio, iter) { void *p1 = kmap_atomic(bv.bv_page); diff --git a/drivers/md/bcache/journal.c b/drivers/md/bcache/journal.c index 29eba72..af3f9f7 100644 --- a/drivers/md/bcache/journal.c +++ b/drivers/md/bcache/journal.c @@ -418,7 +418,7 @@ static void journal_discard_work(struct work_struct *work) struct journal_device *ja = container_of(work, struct journal_device, discard_work); - submit_bio(0, &ja->discard_bio); + submit_bio(&ja->discard_bio); } static void do_journal_discard(struct cache *ca) diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index 8d0ead9..9e2eac8 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -212,7 +212,7 @@ static void __write_super(struct cache_sb *sb, struct bio *bio) unsigned i; bio->bi_iter.bi_sector = SB_SECTOR; - bio->bi_rw = REQ_SYNC|REQ_META; + bio->bi_rw = REQ_WRITE|REQ_SYNC|REQ_META; bio->bi_iter.bi_size = SB_SIZE; bch_bio_map(bio, NULL); @@ -238,7 +238,7 @@ static void __write_super(struct cache_sb *sb, struct bio *bio) pr_debug("ver %llu, flags %llu, seq %llu", sb->version, sb->flags, sb->seq); - submit_bio(REQ_WRITE, bio); + submit_bio(bio); } static void bch_write_bdev_super_unlock(struct closure *cl) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index cd77216..1fd25bf 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -634,6 +634,7 @@ static void use_inline_bio(struct dm_buffer *b, int rw, sector_t block, * the dm_buffer's inline bio is local to bufio. */ b->bio.bi_private = end_io; + b->bio.bi_rw |= rw; /* * We assume that if len >= PAGE_SIZE ptr is page-aligned. @@ -660,7 +661,7 @@ static void use_inline_bio(struct dm_buffer *b, int rw, sector_t block, ptr += PAGE_SIZE; } while (len > 0); - submit_bio(rw, &b->bio); + submit_bio(&b->bio); } static void submit_io(struct dm_buffer *b, int rw, sector_t block, diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c index 06d426e..107d445 100644 --- a/drivers/md/dm-io.c +++ b/drivers/md/dm-io.c @@ -322,6 +322,7 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, bio->bi_iter.bi_sector = where->sector + (where->count - remaining); bio->bi_bdev = where->bdev; bio->bi_end_io = endio; + bio->bi_rw |= rw; store_io_and_region_in_bio(bio, io, region); if (rw & REQ_DISCARD) { @@ -355,7 +356,7 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, } atomic_inc(&io->count); - submit_bio(rw, bio); + submit_bio(bio); } while (remaining); } diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c index 624589d..21760e3 100644 --- a/drivers/md/dm-log-writes.c +++ b/drivers/md/dm-log-writes.c @@ -205,6 +205,7 @@ static int write_metadata(struct log_writes_c *lc, void *entry, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; + bio->bi_rw |= WRITE; page = alloc_page(GFP_KERNEL); if (!page) { @@ -226,7 +227,7 @@ static int write_metadata(struct log_writes_c *lc, void *entry, DMERR("Couldn't add page to the log block"); goto error_bio; } - submit_bio(WRITE, bio); + submit_bio(bio); return 0; error_bio: bio_put(bio); @@ -269,6 +270,7 @@ static int log_one_block(struct log_writes_c *lc, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; + bio->bi_rw |= WRITE; for (i = 0; i < block->vec_cnt; i++) { /* @@ -279,7 +281,7 @@ static int log_one_block(struct log_writes_c *lc, block->vecs[i].bv_len, 0); if (ret != block->vecs[i].bv_len) { atomic_inc(&lc->io_blocks); - submit_bio(WRITE, bio); + submit_bio(bio); bio = bio_alloc(GFP_KERNEL, block->vec_cnt - i); if (!bio) { DMERR("Couldn't alloc log bio"); @@ -290,6 +292,7 @@ static int log_one_block(struct log_writes_c *lc, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; + bio->bi_rw |= WRITE; ret = bio_add_page(bio, block->vecs[i].bv_page, block->vecs[i].bv_len, 0); @@ -301,7 +304,7 @@ static int log_one_block(struct log_writes_c *lc, } sector += block->vecs[i].bv_len >> SECTOR_SHIFT; } - submit_bio(WRITE, bio); + submit_bio(bio); out: kfree(block->data); kfree(block); diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index a1cc797..8017667 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -365,8 +365,9 @@ static int __blkdev_issue_discard_async(struct block_device *bdev, sector_t sect bio->bi_iter.bi_sector = sector; bio->bi_bdev = bdev; bio->bi_iter.bi_size = nr_sects << 9; + bio->bi_rw |= type; - submit_bio(type, bio); + submit_bio(bio); return 0; } diff --git a/drivers/md/md.c b/drivers/md/md.c index 4424d0e..e25ef97 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -401,8 +401,9 @@ static void submit_flushes(struct work_struct *ws) bi->bi_end_io = md_end_flush; bi->bi_private = rdev; bi->bi_bdev = rdev->bdev; + bi->bi_rw |= WRITE_FLUSH; atomic_inc(&mddev->flush_pending); - submit_bio(WRITE_FLUSH, bi); + submit_bio(bi); rcu_read_lock(); rdev_dec_pending(rdev, mddev); } @@ -746,9 +747,10 @@ void md_super_write(struct mddev *mddev, struct md_rdev *rdev, bio_add_page(bio, page, size, 0); bio->bi_private = rdev; bio->bi_end_io = super_written; + bio->bi_rw |= WRITE_FLUSH_FUA; atomic_inc(&mddev->pending_writes); - submit_bio(WRITE_FLUSH_FUA, bio); + submit_bio(bio); } void md_super_wait(struct mddev *mddev) @@ -765,6 +767,7 @@ int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, bio->bi_bdev = (metadata_op && rdev->meta_bdev) ? rdev->meta_bdev : rdev->bdev; + bio->bi_rw |= rw; if (metadata_op) bio->bi_iter.bi_sector = sector + rdev->sb_start; else if (rdev->mddev->reshape_position != MaxSector && @@ -774,7 +777,8 @@ int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, else bio->bi_iter.bi_sector = sector + rdev->data_offset; bio_add_page(bio, page, size, 0); - submit_bio_wait(rw, bio); + + submit_bio_wait(bio); ret = !bio->bi_error; bio_put(bio); diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index e2169ff..01e6e74 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2208,7 +2208,8 @@ static int narrow_write_error(struct r1bio *r1_bio, int i) bio_trim(wbio, sector - r1_bio->sector, sectors); wbio->bi_iter.bi_sector += rdev->data_offset; wbio->bi_bdev = rdev->bdev; - if (submit_bio_wait(WRITE, wbio) < 0) + + if (submit_bio_wait(wbio) < 0) /* failure! */ ok = rdev_set_badblocks(rdev, sector, sectors, 0) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 84e597e..d2da45b 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2474,7 +2474,9 @@ static int narrow_write_error(struct r10bio *r10_bio, int i) choose_data_offset(r10_bio, rdev) + (sector - r10_bio->sector)); wbio->bi_bdev = rdev->bdev; - if (submit_bio_wait(WRITE, wbio) < 0) + wbio->bi_rw |= WRITE; + + if (submit_bio_wait(wbio) < 0) /* Failure! */ ok = rdev_set_badblocks(rdev, sector, sectors, 0) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index d992d62..6b500fd 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -254,7 +254,7 @@ static void r5l_submit_current_io(struct r5l_log *log) __r5l_set_io_unit_state(io, IO_UNIT_IO_START); spin_unlock_irqrestore(&log->io_list_lock, flags); - submit_bio(WRITE, io->current_bio); + submit_bio(io->current_bio); } static struct bio *r5l_bio_alloc(struct r5l_log *log) @@ -373,7 +373,7 @@ static void r5l_append_payload_page(struct r5l_log *log, struct page *page) io->current_bio = r5l_bio_alloc(log); bio_chain(io->current_bio, prev); - submit_bio(WRITE, prev); + submit_bio(prev); } if (!bio_add_page(io->current_bio, page, PAGE_SIZE, 0)) @@ -686,7 +686,8 @@ void r5l_flush_stripe_to_raid(struct r5l_log *log) bio_reset(&log->flush_bio); log->flush_bio.bi_bdev = log->rdev->bdev; log->flush_bio.bi_end_io = r5l_log_flush_endio; - submit_bio(WRITE_FLUSH, &log->flush_bio); + log->flush_bio.bi_rw |= WRITE_FLUSH; + submit_bio(&log->flush_bio); } static void r5l_write_super(struct r5l_log *log, sector_t cp); diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c index 5a2899f..bfc3e45 100644 --- a/drivers/target/target_core_iblock.c +++ b/drivers/target/target_core_iblock.c @@ -330,7 +330,7 @@ static void iblock_bio_done(struct bio *bio) } static struct bio * -iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num) +iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, int rw) { struct iblock_dev *ib_dev = IBLOCK_DEV(cmd->se_dev); struct bio *bio; @@ -352,18 +352,19 @@ iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num) bio->bi_private = cmd; bio->bi_end_io = &iblock_bio_done; bio->bi_iter.bi_sector = lba; + bio->bi_rw |= rw; return bio; } -static void iblock_submit_bios(struct bio_list *list, int rw) +static void iblock_submit_bios(struct bio_list *list) { struct blk_plug plug; struct bio *bio; blk_start_plug(&plug); while ((bio = bio_list_pop(list))) - submit_bio(rw, bio); + submit_bio(bio); blk_finish_plug(&plug); } @@ -405,9 +406,10 @@ iblock_execute_sync_cache(struct se_cmd *cmd) bio = bio_alloc(GFP_KERNEL, 0); bio->bi_end_io = iblock_end_io_flush; bio->bi_bdev = ib_dev->ibd_bd; + bio->bi_rw |= WRITE_FLUSH; if (!immed) bio->bi_private = cmd; - submit_bio(WRITE_FLUSH, bio); + submit_bio(bio); return 0; } @@ -456,7 +458,7 @@ iblock_execute_write_same(struct se_cmd *cmd) goto fail; cmd->priv = ibr; - bio = iblock_get_bio(cmd, block_lba, 1); + bio = iblock_get_bio(cmd, block_lba, 1, WRITE); if (!bio) goto fail_free_ibr; @@ -469,7 +471,7 @@ iblock_execute_write_same(struct se_cmd *cmd) while (bio_add_page(bio, sg_page(sg), sg->length, sg->offset) != sg->length) { - bio = iblock_get_bio(cmd, block_lba, 1); + bio = iblock_get_bio(cmd, block_lba, 1, WRITE); if (!bio) goto fail_put_bios; @@ -482,7 +484,7 @@ iblock_execute_write_same(struct se_cmd *cmd) sectors -= 1; } - iblock_submit_bios(&list, WRITE); + iblock_submit_bios(&list); return 0; fail_put_bios: @@ -708,7 +710,7 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, return 0; } - bio = iblock_get_bio(cmd, block_lba, sgl_nents); + bio = iblock_get_bio(cmd, block_lba, sgl_nents, rw); if (!bio) goto fail_free_ibr; @@ -728,11 +730,11 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, while (bio_add_page(bio, sg_page(sg), sg->length, sg->offset) != sg->length) { if (bio_cnt >= IBLOCK_MAX_BIO_PER_TASK) { - iblock_submit_bios(&list, rw); + iblock_submit_bios(&list); bio_cnt = 0; } - bio = iblock_get_bio(cmd, block_lba, sg_num); + bio = iblock_get_bio(cmd, block_lba, sg_num, rw); if (!bio) goto fail_put_bios; @@ -752,7 +754,7 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, goto fail_put_bios; } - iblock_submit_bios(&list, rw); + iblock_submit_bios(&list); iblock_complete_cmd(cmd); return 0; diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index 0340c57..7717043 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -1683,6 +1683,7 @@ static int btrfsic_read_block(struct btrfsic_state *state, } bio->bi_bdev = block_ctx->dev->bdev; bio->bi_iter.bi_sector = dev_bytenr >> 9; + bio->bi_rw |= READ; for (j = i; j < num_pages; j++) { ret = bio_add_page(bio, block_ctx->pagev[j], @@ -1695,7 +1696,7 @@ static int btrfsic_read_block(struct btrfsic_state *state, "btrfsic: error, failed to add a single page!\n"); return -1; } - if (submit_bio_wait(READ, bio)) { + if (submit_bio_wait(bio)) { printk(KERN_INFO "btrfsic: read error at logical %llu dev %s!\n", block_ctx->start, block_ctx->dev->name); @@ -2960,9 +2961,10 @@ int btrfsic_submit_bh(int rw, struct buffer_head *bh) return submit_bh(rw, bh); } -static void __btrfsic_submit_bio(int rw, struct bio *bio) +static void __btrfsic_submit_bio(struct bio *bio) { struct btrfsic_dev_state *dev_state; + int rw = bio->bi_rw; if (!btrfsic_is_initialized) return; @@ -3058,16 +3060,16 @@ leave: mutex_unlock(&btrfsic_mutex); } -void btrfsic_submit_bio(int rw, struct bio *bio) +void btrfsic_submit_bio(struct bio *bio) { - __btrfsic_submit_bio(rw, bio); - submit_bio(rw, bio); + __btrfsic_submit_bio(bio); + submit_bio(bio); } -int btrfsic_submit_bio_wait(int rw, struct bio *bio) +int btrfsic_submit_bio_wait(struct bio *bio) { - __btrfsic_submit_bio(rw, bio); - return submit_bio_wait(rw, bio); + __btrfsic_submit_bio(bio); + return submit_bio_wait(bio); } int btrfsic_mount(struct btrfs_root *root, diff --git a/fs/btrfs/check-integrity.h b/fs/btrfs/check-integrity.h index 13b8566..c04e249 100644 --- a/fs/btrfs/check-integrity.h +++ b/fs/btrfs/check-integrity.h @@ -21,8 +21,8 @@ #ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY int btrfsic_submit_bh(int rw, struct buffer_head *bh); -void btrfsic_submit_bio(int rw, struct bio *bio); -int btrfsic_submit_bio_wait(int rw, struct bio *bio); +void btrfsic_submit_bio(struct bio *bio); +int btrfsic_submit_bio_wait(struct bio *bio); #else #define btrfsic_submit_bh submit_bh #define btrfsic_submit_bio submit_bio diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 6dd6774..66dd552 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -3454,12 +3454,13 @@ static int write_dev_flush(struct btrfs_device *device, int wait) bio->bi_end_io = btrfs_end_empty_barrier; bio->bi_bdev = device->bdev; + bio->bi_rw |= WRITE_FLUSH; init_completion(&device->flush_wait); bio->bi_private = &device->flush_wait; device->flush_bio = bio; bio_get(bio); - btrfsic_submit_bio(WRITE_FLUSH, bio); + btrfsic_submit_bio(bio); return 0; } diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index a2356e2..7bcc729 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2040,9 +2040,10 @@ int repair_io_failure(struct inode *inode, u64 start, u64 length, u64 logical, return -EIO; } bio->bi_bdev = dev->bdev; + bio->bi_rw |= WRITE_SYNC; bio_add_page(bio, page, length, pg_offset); - if (btrfsic_submit_bio_wait(WRITE_SYNC, bio)) { + if (btrfsic_submit_bio_wait(bio)) { /* try to remap that extent elsewhere? */ bio_put(bio); btrfs_dev_stat_inc_and_print(dev, BTRFS_DEV_STAT_WRITE_ERRS); @@ -2730,14 +2731,14 @@ static int __must_check submit_one_bio(int rw, struct bio *bio, start = page_offset(page) + bvec->bv_offset; bio->bi_private = NULL; - + bio->bi_rw |= rw; bio_get(bio); if (tree->ops && tree->ops->submit_bio_hook) ret = tree->ops->submit_bio_hook(page->mapping->host, rw, bio, mirror_num, bio_flags, start); else - btrfsic_submit_bio(rw, bio); + btrfsic_submit_bio(bio); bio_put(bio); return ret; diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c index 1a33d3e..532ceb0 100644 --- a/fs/btrfs/raid56.c +++ b/fs/btrfs/raid56.c @@ -1323,7 +1323,9 @@ write_data: bio->bi_private = rbio; bio->bi_end_io = raid_write_end_io; - submit_bio(WRITE, bio); + bio->bi_rw |= WRITE; + + submit_bio(bio); } return; @@ -1577,11 +1579,12 @@ static int raid56_rmw_stripe(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid_rmw_end_io; + bio->bi_rw |= READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); - submit_bio(READ, bio); + submit_bio(bio); } /* the actual write will happen once the reads are done */ return 0; @@ -2103,11 +2106,12 @@ static int __raid56_parity_recover(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid_recover_end_io; + bio->bi_rw |= READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); - submit_bio(READ, bio); + submit_bio(bio); } out: return 0; @@ -2465,7 +2469,8 @@ submit_write: bio->bi_private = rbio; bio->bi_end_io = raid_write_parity_end_io; - submit_bio(WRITE, bio); + bio->bi_rw |= WRITE; + submit_bio(bio); } return; @@ -2642,11 +2647,12 @@ static void raid56_parity_scrub_stripe(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid56_parity_scrub_end_io; + bio->bi_rw |= READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); - submit_bio(READ, bio); + submit_bio(bio); } /* the actual write will happen once the reads are done */ return; diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c index 6998715..944959f 100644 --- a/fs/btrfs/scrub.c +++ b/fs/btrfs/scrub.c @@ -1504,8 +1504,9 @@ static void scrub_recheck_block(struct btrfs_fs_info *fs_info, sblock->no_io_error_seen = 0; } else { bio->bi_iter.bi_sector = page->physical >> 9; + bio->bi_rw |= READ; - if (btrfsic_submit_bio_wait(READ, bio)) + if (btrfsic_submit_bio_wait(bio)) sblock->no_io_error_seen = 0; } @@ -1585,6 +1586,7 @@ static int scrub_repair_page_from_good_copy(struct scrub_block *sblock_bad, return -EIO; bio->bi_bdev = page_bad->dev->bdev; bio->bi_iter.bi_sector = page_bad->physical >> 9; + bio->bi_rw |= WRITE; ret = bio_add_page(bio, page_good->page, PAGE_SIZE, 0); if (PAGE_SIZE != ret) { @@ -1592,7 +1594,7 @@ static int scrub_repair_page_from_good_copy(struct scrub_block *sblock_bad, return -EIO; } - if (btrfsic_submit_bio_wait(WRITE, bio)) { + if (btrfsic_submit_bio_wait(bio)) { btrfs_dev_stat_inc_and_print(page_bad->dev, BTRFS_DEV_STAT_WRITE_ERRS); btrfs_dev_replace_stats_inc( @@ -1685,6 +1687,7 @@ again: bio->bi_end_io = scrub_wr_bio_end_io; bio->bi_bdev = sbio->dev->bdev; bio->bi_iter.bi_sector = sbio->physical >> 9; + bio->bi_rw |= WRITE; sbio->err = 0; } else if (sbio->physical + sbio->page_count * PAGE_SIZE != spage->physical_for_dev_replace || @@ -1732,7 +1735,7 @@ static void scrub_wr_submit(struct scrub_ctx *sctx) * orders the requests before sending them to the driver which * doubled the write performance on spinning disks when measured * with Linux 3.5 */ - btrfsic_submit_bio(WRITE, sbio->bio); + btrfsic_submit_bio(sbio->bio); } static void scrub_wr_bio_end_io(struct bio *bio) @@ -2042,7 +2045,7 @@ static void scrub_submit(struct scrub_ctx *sctx) sbio = sctx->bios[sctx->curr]; sctx->curr = -1; scrub_pending_bio_inc(sctx); - btrfsic_submit_bio(READ, sbio->bio); + btrfsic_submit_bio(sbio->bio); } static int scrub_add_page_to_rd_bio(struct scrub_ctx *sctx, @@ -2088,6 +2091,7 @@ again: bio->bi_end_io = scrub_bio_end_io; bio->bi_bdev = sbio->dev->bdev; bio->bi_iter.bi_sector = sbio->physical >> 9; + bio->bi_rw |= READ; sbio->err = 0; } else if (sbio->physical + sbio->page_count * PAGE_SIZE != spage->physical || @@ -4390,6 +4394,8 @@ static int write_page_nocow(struct scrub_ctx *sctx, bio->bi_iter.bi_size = 0; bio->bi_iter.bi_sector = physical_for_dev_replace >> 9; bio->bi_bdev = dev->bdev; + bio->bi_rw |= WRITE_SYNC; + ret = bio_add_page(bio, page, PAGE_CACHE_SIZE, 0); if (ret != PAGE_CACHE_SIZE) { leave_with_eio: @@ -4398,7 +4404,7 @@ leave_with_eio: return -EIO; } - if (btrfsic_submit_bio_wait(WRITE_SYNC, bio)) + if (btrfsic_submit_bio_wait(bio)) goto leave_with_eio; bio_put(bio); diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index a114b7b..2463761 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -446,7 +446,7 @@ loop_lock: sync_pending = 0; } - btrfsic_submit_bio(cur->bi_rw, cur); + btrfsic_submit_bio(cur); num_run++; batch_run++; @@ -5942,7 +5942,7 @@ static void btrfs_end_bio(struct bio *bio) */ static noinline void btrfs_schedule_bio(struct btrfs_root *root, struct btrfs_device *device, - int rw, struct bio *bio) + struct bio *bio) { int should_queue = 1; struct btrfs_pending_bios *pending_bios; @@ -5953,9 +5953,9 @@ static noinline void btrfs_schedule_bio(struct btrfs_root *root, } /* don't bother with additional async steps for reads, right now */ - if (!(rw & REQ_WRITE)) { + if (!(bio->bi_rw & REQ_WRITE)) { bio_get(bio); - btrfsic_submit_bio(rw, bio); + btrfsic_submit_bio(bio); bio_put(bio); return; } @@ -5969,7 +5969,6 @@ static noinline void btrfs_schedule_bio(struct btrfs_root *root, atomic_inc(&root->fs_info->nr_async_bios); WARN_ON(bio->bi_next); bio->bi_next = NULL; - bio->bi_rw |= rw; spin_lock(&device->io_lock); if (bio->bi_rw & REQ_SYNC) @@ -6003,6 +6002,7 @@ static void submit_stripe_bio(struct btrfs_root *root, struct btrfs_bio *bbio, btrfs_io_bio(bio)->stripe_index = dev_nr; bio->bi_end_io = btrfs_end_bio; bio->bi_iter.bi_sector = physical >> 9; + bio->bi_rw |= rw; #ifdef DEBUG { struct rcu_string *name; @@ -6021,9 +6021,9 @@ static void submit_stripe_bio(struct btrfs_root *root, struct btrfs_bio *bbio, btrfs_bio_counter_inc_noblocked(root->fs_info); if (async) - btrfs_schedule_bio(root, dev, rw, bio); + btrfs_schedule_bio(root, dev, bio); else - btrfsic_submit_bio(rw, bio); + btrfsic_submit_bio(bio); } static void bbio_error(struct btrfs_bio *bbio, struct bio *bio, u64 logical) diff --git a/fs/buffer.c b/fs/buffer.c index 4f4cd95..63d0aed 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -3041,8 +3041,9 @@ static int submit_bh_wbc(int rw, struct buffer_head *bh, rw |= REQ_META; if (buffer_prio(bh)) rw |= REQ_PRIO; + bio->bi_rw |= rw; - submit_bio(rw, bio); + submit_bio(bio); return 0; } diff --git a/fs/direct-io.c b/fs/direct-io.c index 602e844..66b1d3eb 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -369,6 +369,7 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_bdev = bdev; bio->bi_iter.bi_sector = first_sector; + bio->bi_rw |= dio->rw; if (dio->is_async) bio->bi_end_io = dio_bio_end_aio; else @@ -406,7 +407,7 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) sdio->logical_offset_in_bio); dio->bio_cookie = BLK_QC_T_NONE; } else - dio->bio_cookie = submit_bio(dio->rw, bio); + dio->bio_cookie = submit_bio(bio); sdio->bio = NULL; sdio->boundary = 0; diff --git a/fs/ext4/crypto.c b/fs/ext4/crypto.c index c802120..a5fc446 100644 --- a/fs/ext4/crypto.c +++ b/fs/ext4/crypto.c @@ -424,6 +424,7 @@ int ext4_encrypted_zeroout(struct inode *inode, ext4_lblk_t lblk, bio->bi_bdev = inode->i_sb->s_bdev; bio->bi_iter.bi_sector = pblk << (inode->i_sb->s_blocksize_bits - 9); + bio->bi_rw |= WRITE; ret = bio_add_page(bio, ciphertext_page, inode->i_sb->s_blocksize, 0); if (ret != inode->i_sb->s_blocksize) { @@ -435,7 +436,7 @@ int ext4_encrypted_zeroout(struct inode *inode, ext4_lblk_t lblk, err = -EIO; goto errout; } - err = submit_bio_wait(WRITE, bio); + err = submit_bio_wait(bio); if ((err == 0) && bio->bi_error) err = -EIO; bio_put(bio); diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index 17fbe38..86bb60ba 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -357,7 +357,8 @@ void ext4_io_submit(struct ext4_io_submit *io) int io_op = io->io_wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE; bio_get(io->io_bio); - submit_bio(io_op, io->io_bio); + io->io_bio->bi_rw |= io_op; + submit_bio(io->io_bio); bio_put(io->io_bio); } io->io_bio = NULL; diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c index 5dc5e95..772e40b 100644 --- a/fs/ext4/readpage.c +++ b/fs/ext4/readpage.c @@ -271,7 +271,7 @@ int ext4_mpage_readpages(struct address_space *mapping, */ if (bio && (last_block_in_bio != blocks[0] - 1)) { submit_and_realloc: - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } if (bio == NULL) { @@ -294,6 +294,7 @@ int ext4_mpage_readpages(struct address_space *mapping, bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9); bio->bi_end_io = mpage_end_io; bio->bi_private = ctx; + bio->bi_rw |= READ; } length = first_hole << blkbits; @@ -303,14 +304,14 @@ int ext4_mpage_readpages(struct address_space *mapping, if (((map.m_flags & EXT4_MAP_BOUNDARY) && (relative_block == map.m_len)) || (first_hole != blocks_per_page)) { - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } else last_block_in_bio = blocks[blocks_per_page - 1]; goto next_page; confused: if (bio) { - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } if (!PageUptodate(page)) @@ -323,6 +324,6 @@ int ext4_mpage_readpages(struct address_space *mapping, } BUG_ON(pages && !list_empty(pages)); if (bio) - submit_bio(READ, bio); + submit_bio(bio); return 0; } diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index e34b1bd..5325408 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -111,8 +111,9 @@ static void __submit_merged_bio(struct f2fs_bio_info *io) trace_f2fs_submit_read_bio(io->sbi->sb, fio, io->bio); else trace_f2fs_submit_write_bio(io->sbi->sb, fio, io->bio); + io->bio->bi_rw |= fio->rw; - submit_bio(fio->rw, io->bio); + submit_bio(io->bio); io->bio = NULL; } @@ -157,8 +158,9 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio) bio_put(bio); return -EFAULT; } + bio->bi_rw |= fio->rw; - submit_bio(fio->rw, bio); + submit_bio(bio); return 0; } @@ -953,7 +955,7 @@ got_it: */ if (bio && (last_block_in_bio != block_nr - 1)) { submit_and_realloc: - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } if (bio == NULL) { @@ -982,6 +984,7 @@ submit_and_realloc: bio->bi_iter.bi_sector = SECTOR_FROM_BLOCK(block_nr); bio->bi_end_io = f2fs_read_end_io; bio->bi_private = ctx; + bio->bi_rw |= READ; } if (bio_add_page(bio, page, blocksize, 0) < blocksize) @@ -996,7 +999,7 @@ set_error_page: goto next_page; confused: if (bio) { - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } unlock_page(page); @@ -1006,7 +1009,7 @@ next_page: } BUG_ON(pages && !list_empty(pages)); if (bio) - submit_bio(READ, bio); + submit_bio(bio); return 0; } diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index c247450..6ebdbb3 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -318,7 +318,8 @@ repeat: fcc->dispatch_list = llist_reverse_order(fcc->dispatch_list); bio->bi_bdev = sbi->sb->s_bdev; - ret = submit_bio_wait(WRITE_FLUSH, bio); + bio->bi_rw |= WRITE_FLUSH; + ret = submit_bio_wait(bio); llist_for_each_entry_safe(cmd, next, fcc->dispatch_list, llnode) { @@ -350,7 +351,8 @@ int f2fs_issue_flush(struct f2fs_sb_info *sbi) int ret; bio->bi_bdev = sbi->sb->s_bdev; - ret = submit_bio_wait(WRITE_FLUSH, bio); + bio->bi_rw |= WRITE_FLUSH; + ret = submit_bio_wait(bio); bio_put(bio); return ret; } diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index d5369a1..acc5ccb 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -240,7 +240,8 @@ void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int rw) { if (sdp->sd_log_bio) { atomic_inc(&sdp->sd_log_in_flight); - submit_bio(rw, sdp->sd_log_bio); + sdp->sd_log_bio->bi_rw |= rw; + submit_bio(sdp->sd_log_bio); sdp->sd_log_bio = NULL; } } diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index e137d96..013886d 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -230,7 +230,8 @@ static void gfs2_submit_bhs(int rw, struct buffer_head *bhs[], int num) bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh)); } bio->bi_end_io = gfs2_meta_read_endio; - submit_bio(rw, bio); + bio->bi_rw |= rw; + submit_bio(bio); } /** diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c index 7aacdf2..15a4f86 100644 --- a/fs/gfs2/ops_fstype.c +++ b/fs/gfs2/ops_fstype.c @@ -246,7 +246,8 @@ static int gfs2_read_super(struct gfs2_sbd *sdp, sector_t sector, int silent) bio->bi_end_io = end_bio_io_page; bio->bi_private = page; - submit_bio(READ_SYNC | REQ_META, bio); + bio->bi_rw |= READ_SYNC | REQ_META; + submit_bio(bio); wait_on_page_locked(page); bio_put(bio); if (!PageUptodate(page)) { diff --git a/fs/hfsplus/wrapper.c b/fs/hfsplus/wrapper.c index cc62356..7e605b5 100644 --- a/fs/hfsplus/wrapper.c +++ b/fs/hfsplus/wrapper.c @@ -65,6 +65,7 @@ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, bio = bio_alloc(GFP_NOIO, 1); bio->bi_iter.bi_sector = sector; bio->bi_bdev = sb->s_bdev; + bio->bi_rw |= rw; if (!(rw & WRITE) && data) *data = (u8 *)buf + offset; @@ -83,7 +84,7 @@ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, buf = (u8 *)buf + len; } - ret = submit_bio_wait(rw, bio); + ret = submit_bio_wait(bio); out: bio_put(bio); return ret < 0 ? ret : 0; diff --git a/fs/jfs/jfs_logmgr.c b/fs/jfs/jfs_logmgr.c index a270cb7..a4321a6 100644 --- a/fs/jfs/jfs_logmgr.c +++ b/fs/jfs/jfs_logmgr.c @@ -2004,12 +2004,13 @@ static int lbmRead(struct jfs_log * log, int pn, struct lbuf ** bpp) bio->bi_end_io = lbmIODone; bio->bi_private = bp; + bio->bi_rw |= READ_SYNC; /*check if journaling to disk has been disabled*/ if (log->no_integrity) { bio->bi_iter.bi_size = 0; lbmIODone(bio); } else { - submit_bio(READ_SYNC, bio); + submit_bio(bio); } wait_event(bp->l_ioevent, (bp->l_flag != lbmREAD)); @@ -2147,13 +2148,14 @@ static void lbmStartIO(struct lbuf * bp) bio->bi_end_io = lbmIODone; bio->bi_private = bp; + bio->bi_rw |= WRITE_SYNC; /* check if journaling to disk has been disabled */ if (log->no_integrity) { bio->bi_iter.bi_size = 0; lbmIODone(bio); } else { - submit_bio(WRITE_SYNC, bio); + submit_bio(bio); INCREMENT(lmStat.submitted); } } diff --git a/fs/jfs/jfs_metapage.c b/fs/jfs/jfs_metapage.c index a3eb316..b5b76a5 100644 --- a/fs/jfs/jfs_metapage.c +++ b/fs/jfs/jfs_metapage.c @@ -411,7 +411,7 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) inc_io(page); if (!bio->bi_iter.bi_size) goto dump_bio; - submit_bio(WRITE, bio); + submit_bio(bio); nr_underway++; bio = NULL; } else @@ -434,6 +434,7 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) bio->bi_iter.bi_sector = pblock << (inode->i_blkbits - 9); bio->bi_end_io = metapage_write_end_io; bio->bi_private = page; + bio->bi_rw |= WRITE; /* Don't call bio_add_page yet, we may add to this vec */ bio_offset = offset; @@ -448,7 +449,7 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) if (!bio->bi_iter.bi_size) goto dump_bio; - submit_bio(WRITE, bio); + submit_bio(bio); nr_underway++; } if (redirty) @@ -506,7 +507,7 @@ static int metapage_readpage(struct file *fp, struct page *page) insert_metapage(page, NULL); inc_io(page); if (bio) - submit_bio(READ, bio); + submit_bio(bio); bio = bio_alloc(GFP_NOFS, 1); bio->bi_bdev = inode->i_sb->s_bdev; @@ -514,6 +515,7 @@ static int metapage_readpage(struct file *fp, struct page *page) pblock << (inode->i_blkbits - 9); bio->bi_end_io = metapage_read_end_io; bio->bi_private = page; + bio->bi_rw |= READ; len = xlen << inode->i_blkbits; offset = block_offset << inode->i_blkbits; if (bio_add_page(bio, page, len, offset) < len) @@ -523,7 +525,7 @@ static int metapage_readpage(struct file *fp, struct page *page) block_offset++; } if (bio) - submit_bio(READ, bio); + submit_bio(bio); else unlock_page(page); diff --git a/fs/logfs/dev_bdev.c b/fs/logfs/dev_bdev.c index a709d80..623dd3e 100644 --- a/fs/logfs/dev_bdev.c +++ b/fs/logfs/dev_bdev.c @@ -29,8 +29,9 @@ static int sync_request(struct page *page, struct block_device *bdev, int rw) bio.bi_bdev = bdev; bio.bi_iter.bi_sector = page->index * (PAGE_SIZE >> 9); bio.bi_iter.bi_size = PAGE_SIZE; + bio.bi_rw |= rw; - return submit_bio_wait(rw, &bio); + return submit_bio_wait(&bio); } static int bdev_readpage(void *_sb, struct page *page) @@ -95,8 +96,9 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = writeseg_end_io; + bio->bi_rw |= WRITE; atomic_inc(&super->s_pending_writes); - submit_bio(WRITE, bio); + submit_bio(bio); ofs += i * PAGE_SIZE; index += i; @@ -122,8 +124,9 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = writeseg_end_io; + bio->bi_rw |= WRITE; atomic_inc(&super->s_pending_writes); - submit_bio(WRITE, bio); + submit_bio(bio); return 0; } @@ -185,8 +188,9 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = erase_end_io; + bio->bi_rw |= WRITE; atomic_inc(&super->s_pending_writes); - submit_bio(WRITE, bio); + submit_bio(bio); ofs += i * PAGE_SIZE; index += i; @@ -206,8 +210,9 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = erase_end_io; + bio->bi_rw |= WRITE; atomic_inc(&super->s_pending_writes); - submit_bio(WRITE, bio); + submit_bio(bio); return 0; } diff --git a/fs/mpage.c b/fs/mpage.c index 1480d3a..9fec67f 100644 --- a/fs/mpage.c +++ b/fs/mpage.c @@ -58,8 +58,9 @@ static void mpage_end_io(struct bio *bio) static struct bio *mpage_bio_submit(int rw, struct bio *bio) { bio->bi_end_io = mpage_end_io; + bio->bi_rw |= rw; guard_bio_eod(rw, bio); - submit_bio(rw, bio); + submit_bio(bio); return NULL; } diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index ddd0138..bb8d629 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -102,14 +102,15 @@ static inline void put_parallel(struct parallel_io *p) } static struct bio * -bl_submit_bio(int rw, struct bio *bio) +bl_submit_bio(struct bio *bio) { if (bio) { get_parallel(bio->bi_private); dprintk("%s submitting %s bio %u@%llu\n", __func__, - rw == READ ? "read" : "write", bio->bi_iter.bi_size, + bio->bi_rw == READ ? "read" : "write", + bio->bi_iter.bi_size, (unsigned long long)bio->bi_iter.bi_sector); - submit_bio(rw, bio); + submit_bio(bio); } return NULL; } @@ -158,7 +159,7 @@ do_add_page_to_bio(struct bio *bio, int npg, int rw, sector_t isect, if (disk_addr < map->start || disk_addr >= map->start + map->len) { if (!dev->map(dev, disk_addr, map)) return ERR_PTR(-EIO); - bio = bl_submit_bio(rw, bio); + bio = bl_submit_bio(bio); } disk_addr += map->disk_offset; disk_addr -= map->start; @@ -174,9 +175,10 @@ retry: disk_addr >> SECTOR_SHIFT, end_io, par); if (!bio) return ERR_PTR(-ENOMEM); + bio->bi_rw |= rw; } if (bio_add_page(bio, page, *len, offset) < *len) { - bio = bl_submit_bio(rw, bio); + bio = bl_submit_bio(bio); goto retry; } return bio; @@ -252,7 +254,7 @@ bl_read_pagelist(struct nfs_pgio_header *header) for (i = pg_index; i < header->page_array.npages; i++) { if (extent_length <= 0) { /* We've used up the previous extent */ - bio = bl_submit_bio(READ, bio); + bio = bl_submit_bio(bio); /* Get the next one */ if (!ext_tree_lookup(bl, isect, &be, false)) { @@ -273,7 +275,7 @@ bl_read_pagelist(struct nfs_pgio_header *header) } if (is_hole(&be)) { - bio = bl_submit_bio(READ, bio); + bio = bl_submit_bio(bio); /* Fill hole w/ zeroes w/o accessing device */ dprintk("%s Zeroing page for hole\n", __func__); zero_user_segment(pages[i], pg_offset, pg_len); @@ -306,7 +308,7 @@ bl_read_pagelist(struct nfs_pgio_header *header) header->res.count = (isect << SECTOR_SHIFT) - header->args.offset; } out: - bl_submit_bio(READ, bio); + bl_submit_bio(bio); blk_finish_plug(&plug); put_parallel(par); return PNFS_ATTEMPTED; @@ -398,7 +400,7 @@ bl_write_pagelist(struct nfs_pgio_header *header, int sync) for (i = pg_index; i < header->page_array.npages; i++) { if (extent_length <= 0) { /* We've used up the previous extent */ - bio = bl_submit_bio(WRITE, bio); + bio = bl_submit_bio(bio); /* Get the next one */ if (!ext_tree_lookup(bl, isect, &be, true)) { header->pnfs_error = -EINVAL; @@ -427,7 +429,7 @@ bl_write_pagelist(struct nfs_pgio_header *header, int sync) header->res.count = header->args.count; out: - bl_submit_bio(WRITE, bio); + bl_submit_bio(bio); blk_finish_plug(&plug); put_parallel(par); return PNFS_ATTEMPTED; diff --git a/fs/nilfs2/segbuf.c b/fs/nilfs2/segbuf.c index f63620c..428ece8 100644 --- a/fs/nilfs2/segbuf.c +++ b/fs/nilfs2/segbuf.c @@ -368,7 +368,8 @@ static int nilfs_segbuf_submit_bio(struct nilfs_segment_buffer *segbuf, bio->bi_end_io = nilfs_end_bio_write; bio->bi_private = segbuf; - submit_bio(mode, bio); + bio->bi_rw |= mode; + submit_bio(bio); segbuf->sb_nbio++; wi->bio = NULL; diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c index a3cc6d2..41039c2 100644 --- a/fs/ocfs2/cluster/heartbeat.c +++ b/fs/ocfs2/cluster/heartbeat.c @@ -391,7 +391,7 @@ static void o2hb_bio_end_io(struct bio *bio) static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, struct o2hb_bio_wait_ctxt *wc, unsigned int *current_slot, - unsigned int max_slots) + unsigned int max_slots, int rw) { int len, current_page; unsigned int vec_len, vec_start; @@ -417,6 +417,7 @@ static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, bio->bi_bdev = reg->hr_bdev; bio->bi_private = wc; bio->bi_end_io = o2hb_bio_end_io; + bio->bi_rw |= rw; vec_start = (cs << bits) % PAGE_CACHE_SIZE; while(cs < max_slots) { @@ -452,7 +453,8 @@ static int o2hb_read_slots(struct o2hb_region *reg, o2hb_bio_wait_init(&wc); while(current_slot < max_slots) { - bio = o2hb_setup_one_bio(reg, &wc, ¤t_slot, max_slots); + bio = o2hb_setup_one_bio(reg, &wc, ¤t_slot, max_slots, + READ); if (IS_ERR(bio)) { status = PTR_ERR(bio); mlog_errno(status); @@ -460,7 +462,7 @@ static int o2hb_read_slots(struct o2hb_region *reg, } atomic_inc(&wc.wc_num_reqs); - submit_bio(READ, bio); + submit_bio(bio); } status = 0; @@ -484,7 +486,7 @@ static int o2hb_issue_node_write(struct o2hb_region *reg, slot = o2nm_this_node(); - bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1); + bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1, WRITE_SYNC); if (IS_ERR(bio)) { status = PTR_ERR(bio); mlog_errno(status); @@ -492,7 +494,7 @@ static int o2hb_issue_node_write(struct o2hb_region *reg, } atomic_inc(&write_wc->wc_num_reqs); - submit_bio(WRITE_SYNC, bio); + submit_bio(bio); status = 0; bail: diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 29e7e5d..a1052d2 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -382,7 +382,8 @@ xfs_submit_ioend_bio( atomic_inc(&ioend->io_remaining); bio->bi_private = ioend; bio->bi_end_io = xfs_end_bio; - submit_bio(wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE, bio); + bio->bi_rw |= (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + submit_bio(bio); } STATIC struct bio * diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 45a8ea7..644e676 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1163,7 +1163,7 @@ next_chunk: bio->bi_iter.bi_sector = sector; bio->bi_end_io = xfs_buf_bio_end_io; bio->bi_private = bp; - + bio->bi_rw |= rw; for (; size && nr_pages; nr_pages--, page_index++) { int rbytes, nbytes = PAGE_SIZE - offset; @@ -1187,7 +1187,7 @@ next_chunk: flush_kernel_vmap_range(bp->b_addr, xfs_buf_vmap_len(bp)); } - submit_bio(rw, bio); + submit_bio(bio); if (size) goto next_chunk; } else { diff --git a/include/linux/bio.h b/include/linux/bio.h index 5349e68..9f0e2cf 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -441,7 +441,7 @@ static inline void bio_io_error(struct bio *bio) struct request_queue; extern int bio_phys_segments(struct request_queue *, struct bio *); -extern int submit_bio_wait(int rw, struct bio *bio); +extern int submit_bio_wait(struct bio *bio); extern void bio_advance(struct bio *, unsigned); extern void bio_init(struct bio *); diff --git a/include/linux/fs.h b/include/linux/fs.h index fdab768..3b4e751 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2643,7 +2643,7 @@ static inline void remove_inode_hash(struct inode *inode) extern void inode_sb_list_add(struct inode *inode); #ifdef CONFIG_BLOCK -extern blk_qc_t submit_bio(int, struct bio *); +extern blk_qc_t submit_bio(struct bio *); extern int bdev_read_only(struct block_device *); #endif extern int set_blocksize(struct block_device *, int); diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 12cd989..649dfc7 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -260,6 +260,7 @@ static int hib_submit_io(int rw, pgoff_t page_off, void *addr, bio = bio_alloc(__GFP_RECLAIM | __GFP_HIGH, 1); bio->bi_iter.bi_sector = page_off * (PAGE_SIZE >> 9); bio->bi_bdev = hib_resume_bdev; + bio->bi_rw |= rw; if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) { printk(KERN_ERR "PM: Adding page to bio failed at %llu\n", @@ -272,9 +273,9 @@ static int hib_submit_io(int rw, pgoff_t page_off, void *addr, bio->bi_end_io = hib_end_io; bio->bi_private = hb; atomic_inc(&hb->count); - submit_bio(rw, bio); + submit_bio(bio); } else { - error = submit_bio_wait(rw, bio); + error = submit_bio_wait(bio); bio_put(bio); } diff --git a/mm/page_io.c b/mm/page_io.c index b995a5b..8410cdb 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -253,7 +253,7 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, bio_end_io_t end_write_func) { struct bio *bio; - int ret, rw = WRITE; + int ret; struct swap_info_struct *sis = page_swap_info(page); if (sis->flags & SWP_FILE) { @@ -311,12 +311,13 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, ret = -ENOMEM; goto out; } + bio->bi_rw |= WRITE; if (wbc->sync_mode == WB_SYNC_ALL) - rw |= REQ_SYNC; + bio->bi_rw |= REQ_SYNC; count_vm_event(PSWPOUT); set_page_writeback(page); unlock_page(page); - submit_bio(rw, bio); + submit_bio(bio); out: return ret; } @@ -358,8 +359,9 @@ int swap_readpage(struct page *page) ret = -ENOMEM; goto out; } + bio->bi_rw |= READ; count_vm_event(PSWPIN); - submit_bio(READ, bio); + submit_bio(bio); out: return ret; } -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:53:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9947A7F37 for ; Tue, 5 Jan 2016 14:53:56 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 8C22B8F804C for ; Tue, 5 Jan 2016 12:53:53 -0800 (PST) X-ASG-Debug-ID: 1452027231-04cb6c42e805210001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 24IrPFwapBDHkNeC (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:53:52 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id A276AA3D26; Tue, 5 Jan 2016 20:53:51 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu1024032; Tue, 5 Jan 2016 15:53:49 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields Date: Tue, 5 Jan 2016 14:53:05 -0600 X-ASG-Orig-Subj: [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields Message-Id: <1452027218-32303-3-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027232 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The following patches separate the operation (write, read, discard, etc) from the flags in bi_rw/cmd_flags. This patch adds definitions for request/bio operations, adds fields to the request/bio to set them, and some temporary compat code so the kernel/modules can use either one. In the final patches this compat code will be removed when everything is converted. Also, in this patch the REQ_OPs match the REQ rq_flag_bits ones for compat reasons while all the code is converted in this set. In the last patches that will also be removed. Signed-off-by: Mike Christie --- block/blk-core.c | 19 ++++++++++++++++--- include/linux/blk_types.h | 15 ++++++++++++++- include/linux/blkdev.h | 1 + include/linux/fs.h | 37 +++++++++++++++++++++++++++++++++++-- 4 files changed, 66 insertions(+), 6 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 9b887e3..954a450 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1697,7 +1697,8 @@ void init_request_from_bio(struct request *req, struct bio *bio) { req->cmd_type = REQ_TYPE_FS; - req->cmd_flags |= bio->bi_rw & REQ_COMMON_MASK; + /* tmp compat. Allow users to set bi_op or bi_rw */ + req->cmd_flags |= (bio->bi_rw | bio->bi_op) & REQ_COMMON_MASK; if (bio->bi_rw & REQ_RAHEAD) req->cmd_flags |= REQ_FAILFAST_MASK; @@ -2032,6 +2033,12 @@ blk_qc_t generic_make_request(struct bio *bio) struct bio_list bio_list_on_stack; blk_qc_t ret = BLK_QC_T_NONE; + /* tmp compat. Allow users to set either one or both. + * This will be removed when we have converted + * everyone in the next patches. + */ + bio->bi_rw |= bio->bi_op; + if (!generic_make_request_checks(bio)) goto out; @@ -2101,6 +2108,12 @@ EXPORT_SYMBOL(generic_make_request); */ blk_qc_t submit_bio(struct bio *bio) { + /* tmp compat. Allow users to set either one or both. + * This will be removed when we have converted + * everyone in the next patches. + */ + bio->bi_rw |= bio->bi_op; + /* * If it's a regular read/write or a barrier with data attached, * go through the normal accounting stuff before submission. @@ -2972,8 +2985,8 @@ EXPORT_SYMBOL_GPL(__blk_end_request_err); void blk_rq_bio_prep(struct request_queue *q, struct request *rq, struct bio *bio) { - /* Bit 0 (R/W) is identical in rq->cmd_flags and bio->bi_rw */ - rq->cmd_flags |= bio->bi_rw & REQ_WRITE; + /* tmp compat. Allow users to set bi_op or bi_rw */ + rq->cmd_flags |= bio_data_dir(bio); if (bio_has_data(bio)) rq->nr_phys_segments = bio_phys_segments(q, bio); diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 86a38ea..6e49c91 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -48,9 +48,15 @@ struct bio { struct block_device *bi_bdev; unsigned int bi_flags; /* status, command, etc */ int bi_error; - unsigned long bi_rw; /* bottom bits READ/WRITE, + unsigned long bi_rw; /* bottom bits rq_flags_bits * top bits priority */ + /* + * this will be a u8 in the next patches and bi_rw can be shrunk to + * a u32. For compat in these transistional patches op is a int here. + */ + int bi_op; /* REQ_OP */ + struct bvec_iter bi_iter; @@ -242,6 +248,13 @@ enum rq_flag_bits { #define REQ_HASHED (1ULL << __REQ_HASHED) #define REQ_MQ_INFLIGHT (1ULL << __REQ_MQ_INFLIGHT) +enum req_op { + REQ_OP_READ, + REQ_OP_WRITE = REQ_WRITE, + REQ_OP_DISCARD = REQ_DISCARD, + REQ_OP_WRITE_SAME = REQ_WRITE_SAME, +}; + typedef unsigned int blk_qc_t; #define BLK_QC_T_NONE -1U #define BLK_QC_T_SHIFT 16 diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 29189ae..35b9eb3 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -96,6 +96,7 @@ struct request { struct request_queue *q; struct blk_mq_ctx *mq_ctx; + int op; u64 cmd_flags; unsigned cmd_type; unsigned long atomic_flags; diff --git a/include/linux/fs.h b/include/linux/fs.h index 3b4e751..fb9e516 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2405,15 +2405,48 @@ extern void make_bad_inode(struct inode *); extern bool is_bad_inode(struct inode *); #ifdef CONFIG_BLOCK + +static inline bool op_is_write(int op) +{ + switch (op) { + case REQ_OP_WRITE: + case REQ_OP_WRITE_SAME: + case REQ_OP_DISCARD: + return true; + default: + return false; + } +} + /* * return READ, READA, or WRITE */ -#define bio_rw(bio) ((bio)->bi_rw & (RW_MASK | RWA_MASK)) +static inline int bio_rw(struct bio *bio) +{ + /* + * tmp cpmpat. Allow users to set either op or rw, until + * all code is converted in the next patches. + */ + if (op_is_write(bio->bi_op)) + return WRITE; + + return bio->bi_rw & (RW_MASK | RWA_MASK); +} /* * return data direction, READ or WRITE */ -#define bio_data_dir(bio) ((bio)->bi_rw & 1) +static inline int bio_data_dir(struct bio *bio) +{ + /* + * tmp cpmpat. Allow users to set either op or rw, until + * all code is converted in the next patches. + */ + if (op_is_write(bio->bi_op)) + return WRITE; + + return bio->bi_rw & 1; +} extern void check_disk_size_change(struct gendisk *disk, struct block_device *bdev); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2BA7229E03 for ; Tue, 5 Jan 2016 14:54:00 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id DE856304039 for ; Tue, 5 Jan 2016 12:53:56 -0800 (PST) X-ASG-Debug-ID: 1452027234-04bdf03afc04db0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id NSFv6EN4VsqD7H8c (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:53:54 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 58ACBC109088; Tue, 5 Jan 2016 20:53:54 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu2024032; Tue, 5 Jan 2016 15:53:51 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 03/35] block, fs, mm, drivers: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:06 -0600 X-ASG-Orig-Subj: [PATCH 03/35] block, fs, mm, drivers: set bi_op to REQ_OP Message-Id: <1452027218-32303-4-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027234 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch converts the simple bi_rw use cases in the block, drivers, mm and fs code to use bi_op for a REQ_OP and bi_rw for rq_flag_bits. These should be simple one liner cases, so I just did them in one patch. The next patches handle the more complicated cases in a module per patch. Signed-off-by: Mike Christie --- block/bio.c | 8 +++++--- block/blk-flush.c | 1 + block/blk-lib.c | 7 ++++--- block/blk-map.c | 2 +- drivers/block/floppy.c | 2 +- drivers/block/pktcdvd.c | 4 ++-- drivers/lightnvm/rrpc.c | 4 ++-- drivers/scsi/osd/osd_initiator.c | 8 ++++---- fs/exofs/ore.c | 2 +- fs/ext4/crypto.c | 2 +- fs/ext4/page-io.c | 7 ++++--- fs/ext4/readpage.c | 2 +- fs/jfs/jfs_logmgr.c | 2 ++ fs/jfs/jfs_metapage.c | 4 ++-- fs/logfs/dev_bdev.c | 12 ++++++------ fs/nfs/blocklayout/blocklayout.c | 2 +- mm/page_io.c | 4 ++-- 17 files changed, 40 insertions(+), 33 deletions(-) diff --git a/block/bio.c b/block/bio.c index 921112b..3b8e970 100644 --- a/block/bio.c +++ b/block/bio.c @@ -581,6 +581,7 @@ void __bio_clone_fast(struct bio *bio, struct bio *bio_src) */ bio->bi_bdev = bio_src->bi_bdev; bio_set_flag(bio, BIO_CLONED); + bio->bi_op = bio_src->bi_op; bio->bi_rw = bio_src->bi_rw; bio->bi_iter = bio_src->bi_iter; bio->bi_io_vec = bio_src->bi_io_vec; @@ -663,6 +664,7 @@ struct bio *bio_clone_bioset(struct bio *bio_src, gfp_t gfp_mask, return NULL; bio->bi_bdev = bio_src->bi_bdev; + bio->bi_op = bio_src->bi_op; bio->bi_rw = bio_src->bi_rw; bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; bio->bi_iter.bi_size = bio_src->bi_iter.bi_size; @@ -1168,7 +1170,7 @@ struct bio *bio_copy_user_iov(struct request_queue *q, goto out_bmd; if (iter->type & WRITE) - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; ret = 0; @@ -1338,7 +1340,7 @@ struct bio *bio_map_user_iov(struct request_queue *q, * set data direction, and check if mapped pages need bouncing */ if (iter->type & WRITE) - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; bio_set_flag(bio, BIO_USER_MAPPED); @@ -1531,7 +1533,7 @@ struct bio *bio_copy_kern(struct request_queue *q, void *data, unsigned int len, bio->bi_private = data; } else { bio->bi_end_io = bio_copy_kern_endio; - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; } return bio; diff --git a/block/blk-flush.c b/block/blk-flush.c index e092e13..386f57a 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -484,6 +484,7 @@ int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, bio = bio_alloc(gfp_mask, 0); bio->bi_bdev = bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw |= WRITE_FLUSH; ret = submit_bio_wait(bio); diff --git a/block/blk-lib.c b/block/blk-lib.c index 5292e30..5c55817 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -42,7 +42,7 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, { DECLARE_COMPLETION_ONSTACK(wait); struct request_queue *q = bdev_get_queue(bdev); - int type = REQ_WRITE | REQ_DISCARD; + int type = 0; unsigned int granularity; int alignment; struct bio_batch bb; @@ -102,6 +102,7 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, bio->bi_end_io = bio_batch_end_io; bio->bi_bdev = bdev; bio->bi_private = &bb; + bio->bi_op = REQ_OP_DISCARD; bio->bi_rw |= type; bio->bi_iter.bi_size = req_sects << 9; @@ -178,7 +179,7 @@ int blkdev_issue_write_same(struct block_device *bdev, sector_t sector, bio->bi_io_vec->bv_page = page; bio->bi_io_vec->bv_offset = 0; bio->bi_io_vec->bv_len = bdev_logical_block_size(bdev); - bio->bi_rw |= REQ_WRITE | REQ_WRITE_SAME; + bio->bi_op = REQ_OP_WRITE_SAME; if (nr_sects > max_write_same_sectors) { bio->bi_iter.bi_size = max_write_same_sectors << 9; @@ -240,7 +241,7 @@ static int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, bio->bi_bdev = bdev; bio->bi_end_io = bio_batch_end_io; bio->bi_private = &bb; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; while (nr_sects != 0) { sz = min((sector_t) PAGE_SIZE >> 9 , nr_sects); diff --git a/block/blk-map.c b/block/blk-map.c index f565e11..9021a8f 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -224,7 +224,7 @@ int blk_rq_map_kern(struct request_queue *q, struct request *rq, void *kbuf, return PTR_ERR(bio); if (!reading) - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; if (do_copy) rq->cmd_flags |= REQ_COPY_USER; diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c index e4efcc5..3fda035 100644 --- a/drivers/block/floppy.c +++ b/drivers/block/floppy.c @@ -3817,7 +3817,7 @@ static int __floppy_read_block_0(struct block_device *bdev, int drive) bio.bi_flags |= (1 << BIO_QUIET); bio.bi_private = &cbdata; bio.bi_end_io = floppy_rb0_cb; - bio.bi_rw |= READ; + bio.bi_op = REQ_OP_READ; submit_bio(&bio); process_fd_request(); diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c index d06c62e..a4ecbd3 100644 --- a/drivers/block/pktcdvd.c +++ b/drivers/block/pktcdvd.c @@ -1074,7 +1074,7 @@ static void pkt_gather_data(struct pktcdvd_device *pd, struct packet_data *pkt) BUG(); atomic_inc(&pkt->io_wait); - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; pkt_queue_bio(pd, bio); frames_read++; } @@ -1336,7 +1336,7 @@ static void pkt_start_write(struct pktcdvd_device *pd, struct packet_data *pkt) /* Start the write request */ atomic_set(&pkt->io_wait, 1); - pkt->w_bio->bi_rw = WRITE; + pkt->w_bio->bi_op = REQ_OP_WRITE; pkt_queue_bio(pd, pkt->w_bio); } diff --git a/drivers/lightnvm/rrpc.c b/drivers/lightnvm/rrpc.c index 134e4fa..c20cb18 100644 --- a/drivers/lightnvm/rrpc.c +++ b/drivers/lightnvm/rrpc.c @@ -315,7 +315,7 @@ try: /* Perform read to do GC */ bio->bi_iter.bi_sector = rrpc_get_sector(rev->addr); - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_private = &wait; bio->bi_end_io = rrpc_end_sync_bio; @@ -333,7 +333,7 @@ try: reinit_completion(&wait); bio->bi_iter.bi_sector = rrpc_get_sector(rev->addr); - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_private = &wait; bio->bi_end_io = rrpc_end_sync_bio; diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c index d8a2b51..4e7b440 100644 --- a/drivers/scsi/osd/osd_initiator.c +++ b/drivers/scsi/osd/osd_initiator.c @@ -726,7 +726,7 @@ static int _osd_req_list_objects(struct osd_request *or, return PTR_ERR(bio); } - bio->bi_rw &= ~REQ_WRITE; + bio->bi_op = REQ_OP_READ; or->in.bio = bio; or->in.total_bytes = bio->bi_iter.bi_size; return 0; @@ -839,7 +839,7 @@ int osd_req_write_kern(struct osd_request *or, if (IS_ERR(bio)) return PTR_ERR(bio); - bio->bi_rw |= REQ_WRITE; /* FIXME: bio_set_dir() */ + bio->bi_op = REQ_OP_WRITE; osd_req_write(or, obj, offset, bio, len); return 0; } @@ -956,7 +956,7 @@ static int _osd_req_finalize_cdb_cont(struct osd_request *or, const u8 *cap_key) if (IS_ERR(bio)) return PTR_ERR(bio); - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; /* integrity check the continuation before the bio is linked * with the other data segments since the continuation @@ -1077,7 +1077,7 @@ int osd_req_write_sg_kern(struct osd_request *or, if (IS_ERR(bio)) return PTR_ERR(bio); - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; osd_req_write_sg(or, obj, bio, sglist, numentries); return 0; diff --git a/fs/exofs/ore.c b/fs/exofs/ore.c index 7bd8ac8..c40ed74 100644 --- a/fs/exofs/ore.c +++ b/fs/exofs/ore.c @@ -878,7 +878,7 @@ static int _write_mirror(struct ore_io_state *ios, int cur_comp) } else { bio = master_dev->bio; /* FIXME: bio_set_dir() */ - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; } osd_req_write(or, _ios_obj(ios, cur_comp), diff --git a/fs/ext4/crypto.c b/fs/ext4/crypto.c index a5fc446..cc884e2 100644 --- a/fs/ext4/crypto.c +++ b/fs/ext4/crypto.c @@ -424,7 +424,7 @@ int ext4_encrypted_zeroout(struct inode *inode, ext4_lblk_t lblk, bio->bi_bdev = inode->i_sb->s_bdev; bio->bi_iter.bi_sector = pblk << (inode->i_sb->s_blocksize_bits - 9); - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; ret = bio_add_page(bio, ciphertext_page, inode->i_sb->s_blocksize, 0); if (ret != inode->i_sb->s_blocksize) { diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index 86bb60ba..e36dec4 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -354,10 +354,11 @@ void ext4_io_submit(struct ext4_io_submit *io) struct bio *bio = io->io_bio; if (bio) { - int io_op = io->io_wbc->sync_mode == WB_SYNC_ALL ? - WRITE_SYNC : WRITE; + int io_op_flags = io->io_wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : 0; + bio_get(io->io_bio); - io->io_bio->bi_rw |= io_op; + io->io_bio->bi_op = REQ_OP_WRITE; + io->io_bio->bi_rw |= io_op_flags; submit_bio(io->io_bio); bio_put(io->io_bio); } diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c index 772e40b..fb36434 100644 --- a/fs/ext4/readpage.c +++ b/fs/ext4/readpage.c @@ -294,7 +294,7 @@ int ext4_mpage_readpages(struct address_space *mapping, bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9); bio->bi_end_io = mpage_end_io; bio->bi_private = ctx; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; } length = first_hole << blkbits; diff --git a/fs/jfs/jfs_logmgr.c b/fs/jfs/jfs_logmgr.c index a4321a6..00a4d89 100644 --- a/fs/jfs/jfs_logmgr.c +++ b/fs/jfs/jfs_logmgr.c @@ -2004,6 +2004,7 @@ static int lbmRead(struct jfs_log * log, int pn, struct lbuf ** bpp) bio->bi_end_io = lbmIODone; bio->bi_private = bp; + bio->bi_op = REQ_OP_READ; bio->bi_rw |= READ_SYNC; /*check if journaling to disk has been disabled*/ if (log->no_integrity) { @@ -2148,6 +2149,7 @@ static void lbmStartIO(struct lbuf * bp) bio->bi_end_io = lbmIODone; bio->bi_private = bp; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw |= WRITE_SYNC; /* check if journaling to disk has been disabled */ diff --git a/fs/jfs/jfs_metapage.c b/fs/jfs/jfs_metapage.c index b5b76a5..115d99e 100644 --- a/fs/jfs/jfs_metapage.c +++ b/fs/jfs/jfs_metapage.c @@ -434,7 +434,7 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) bio->bi_iter.bi_sector = pblock << (inode->i_blkbits - 9); bio->bi_end_io = metapage_write_end_io; bio->bi_private = page; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; /* Don't call bio_add_page yet, we may add to this vec */ bio_offset = offset; @@ -515,7 +515,7 @@ static int metapage_readpage(struct file *fp, struct page *page) pblock << (inode->i_blkbits - 9); bio->bi_end_io = metapage_read_end_io; bio->bi_private = page; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; len = xlen << inode->i_blkbits; offset = block_offset << inode->i_blkbits; if (bio_add_page(bio, page, len, offset) < len) diff --git a/fs/logfs/dev_bdev.c b/fs/logfs/dev_bdev.c index 623dd3e..130b7e2 100644 --- a/fs/logfs/dev_bdev.c +++ b/fs/logfs/dev_bdev.c @@ -14,7 +14,7 @@ #define PAGE_OFS(ofs) ((ofs) & (PAGE_SIZE-1)) -static int sync_request(struct page *page, struct block_device *bdev, int rw) +static int sync_request(struct page *page, struct block_device *bdev, int op) { struct bio bio; struct bio_vec bio_vec; @@ -29,7 +29,7 @@ static int sync_request(struct page *page, struct block_device *bdev, int rw) bio.bi_bdev = bdev; bio.bi_iter.bi_sector = page->index * (PAGE_SIZE >> 9); bio.bi_iter.bi_size = PAGE_SIZE; - bio.bi_rw |= rw; + bio.bi_op = op; return submit_bio_wait(&bio); } @@ -96,7 +96,7 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = writeseg_end_io; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; atomic_inc(&super->s_pending_writes); submit_bio(bio); @@ -124,7 +124,7 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = writeseg_end_io; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; atomic_inc(&super->s_pending_writes); submit_bio(bio); return 0; @@ -188,7 +188,7 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = erase_end_io; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; atomic_inc(&super->s_pending_writes); submit_bio(bio); @@ -210,7 +210,7 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = erase_end_io; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; atomic_inc(&super->s_pending_writes); submit_bio(bio); return 0; diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index bb8d629..3832909 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -175,7 +175,7 @@ retry: disk_addr >> SECTOR_SHIFT, end_io, par); if (!bio) return ERR_PTR(-ENOMEM); - bio->bi_rw |= rw; + bio->bi_op = rw; } if (bio_add_page(bio, page, *len, offset) < *len) { bio = bl_submit_bio(bio); diff --git a/mm/page_io.c b/mm/page_io.c index 8410cdb..e48b84c 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -311,7 +311,7 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, ret = -ENOMEM; goto out; } - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; if (wbc->sync_mode == WB_SYNC_ALL) bio->bi_rw |= REQ_SYNC; count_vm_event(PSWPOUT); @@ -359,7 +359,7 @@ int swap_readpage(struct page *page) ret = -ENOMEM; goto out; } - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; count_vm_event(PSWPIN); submit_bio(bio); out: -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9C4B329E03 for ; Tue, 5 Jan 2016 14:54:01 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 14B7EAC001 for ; Tue, 5 Jan 2016 12:54:01 -0800 (PST) X-ASG-Debug-ID: 1452027237-04cb6c42e605210001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id o1tM5cUy66lN4Z3E (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:53:57 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 19D07C109084; Tue, 5 Jan 2016 20:53:57 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu3024032; Tue, 5 Jan 2016 15:53:54 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 04/35] fs: have submit_bh users pass in op and flags separately Date: Tue, 5 Jan 2016 14:53:07 -0600 X-ASG-Orig-Subj: [PATCH 04/35] fs: have submit_bh users pass in op and flags separately Message-Id: <1452027218-32303-5-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027237 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This has submit_bh users pass in the operation and flags separately, so we can setup the bio->bi_op and bio-bi_rw flags. Signed-off-by: Mike Christie --- drivers/md/bitmap.c | 4 ++-- fs/btrfs/check-integrity.c | 24 ++++++++++---------- fs/btrfs/check-integrity.h | 2 +- fs/btrfs/disk-io.c | 4 ++-- fs/buffer.c | 54 +++++++++++++++++++++++---------------------- fs/ext4/balloc.c | 2 +- fs/ext4/ialloc.c | 2 +- fs/ext4/inode.c | 2 +- fs/ext4/mmp.c | 4 ++-- fs/fat/misc.c | 2 +- fs/gfs2/bmap.c | 2 +- fs/gfs2/dir.c | 2 +- fs/gfs2/meta_io.c | 6 ++--- fs/jbd2/commit.c | 6 ++--- fs/jbd2/journal.c | 8 +++---- fs/nilfs2/btnode.c | 6 ++--- fs/nilfs2/btnode.h | 2 +- fs/nilfs2/btree.c | 6 +++-- fs/nilfs2/gcinode.c | 5 +++-- fs/nilfs2/mdt.c | 11 ++++----- fs/ntfs/aops.c | 6 ++--- fs/ntfs/compress.c | 2 +- fs/ntfs/file.c | 2 +- fs/ntfs/logfile.c | 2 +- fs/ntfs/mft.c | 4 ++-- fs/ocfs2/buffer_head_io.c | 8 +++---- fs/reiserfs/inode.c | 4 ++-- fs/reiserfs/journal.c | 6 ++--- fs/ufs/util.c | 2 +- include/linux/buffer_head.h | 9 ++++---- 30 files changed, 103 insertions(+), 96 deletions(-) diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c index 4f22e91..13811fc 100644 --- a/drivers/md/bitmap.c +++ b/drivers/md/bitmap.c @@ -299,7 +299,7 @@ static void write_page(struct bitmap *bitmap, struct page *page, int wait) atomic_inc(&bitmap->pending_writes); set_buffer_locked(bh); set_buffer_mapped(bh); - submit_bh(WRITE | REQ_SYNC, bh); + submit_bh(REQ_OP_WRITE, REQ_SYNC, bh); bh = bh->b_this_page; } @@ -394,7 +394,7 @@ static int read_page(struct file *file, unsigned long index, atomic_inc(&bitmap->pending_writes); set_buffer_locked(bh); set_buffer_mapped(bh); - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); } block++; bh = bh->b_this_page; diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index 7717043..e3fd86b 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -2898,12 +2898,12 @@ static struct btrfsic_dev_state *btrfsic_dev_state_lookup( return ds; } -int btrfsic_submit_bh(int rw, struct buffer_head *bh) +int btrfsic_submit_bh(int op, int op_flags, struct buffer_head *bh) { struct btrfsic_dev_state *dev_state; if (!btrfsic_is_initialized) - return submit_bh(rw, bh); + return submit_bh(op, op_flags, bh); mutex_lock(&btrfsic_mutex); /* since btrfsic_submit_bh() might also be called before @@ -2912,26 +2912,26 @@ int btrfsic_submit_bh(int rw, struct buffer_head *bh) /* Only called to write the superblock (incl. FLUSH/FUA) */ if (NULL != dev_state && - (rw & WRITE) && bh->b_size > 0) { + (op == REQ_OP_WRITE) && bh->b_size > 0) { u64 dev_bytenr; dev_bytenr = 4096 * bh->b_blocknr; if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bh(rw=0x%x, blocknr=%llu (bytenr %llu)," - " size=%zu, data=%p, bdev=%p)\n", - rw, (unsigned long long)bh->b_blocknr, + "submit_bh(op=0x%x,0x%x, blocknr=%llu " + "(bytenr %llu), size=%zu, data=%p, bdev=%p)\n", + op, op_flags, (unsigned long long)bh->b_blocknr, dev_bytenr, bh->b_size, bh->b_data, bh->b_bdev); btrfsic_process_written_block(dev_state, dev_bytenr, &bh->b_data, 1, NULL, - NULL, bh, rw); - } else if (NULL != dev_state && (rw & REQ_FLUSH)) { + NULL, bh, op_flags); + } else if (NULL != dev_state && (op_flags & REQ_FLUSH)) { if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bh(rw=0x%x FLUSH, bdev=%p)\n", - rw, bh->b_bdev); + "submit_bh(op=0x%x,0x%x FLUSH, bdev=%p)\n", + op, op_flags, bh->b_bdev); if (!dev_state->dummy_block_for_bio_bh_flush.is_iodone) { if ((dev_state->state->print_mask & (BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH | @@ -2949,7 +2949,7 @@ int btrfsic_submit_bh(int rw, struct buffer_head *bh) block->never_written = 0; block->iodone_w_error = 0; block->flush_gen = dev_state->last_flush_gen + 1; - block->submit_bio_bh_rw = rw; + block->submit_bio_bh_rw = op_flags; block->orig_bio_bh_private = bh->b_private; block->orig_bio_bh_end_io.bh = bh->b_end_io; block->next_in_same_bio = NULL; @@ -2958,7 +2958,7 @@ int btrfsic_submit_bh(int rw, struct buffer_head *bh) } } mutex_unlock(&btrfsic_mutex); - return submit_bh(rw, bh); + return submit_bh(op, op_flags, bh); } static void __btrfsic_submit_bio(struct bio *bio) diff --git a/fs/btrfs/check-integrity.h b/fs/btrfs/check-integrity.h index c04e249..f78dff1 100644 --- a/fs/btrfs/check-integrity.h +++ b/fs/btrfs/check-integrity.h @@ -20,7 +20,7 @@ #define __BTRFS_CHECK_INTEGRITY__ #ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY -int btrfsic_submit_bh(int rw, struct buffer_head *bh); +int btrfsic_submit_bh(int op, int op_flags, struct buffer_head *bh); void btrfsic_submit_bio(struct bio *bio); int btrfsic_submit_bio_wait(struct bio *bio); #else diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 66dd552..a77d88e 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -3388,9 +3388,9 @@ static int write_dev_supers(struct btrfs_device *device, * to go down lazy. */ if (i == 0) - ret = btrfsic_submit_bh(WRITE_FUA, bh); + ret = btrfsic_submit_bh(REQ_OP_WRITE, WRITE_FUA, bh); else - ret = btrfsic_submit_bh(WRITE_SYNC, bh); + ret = btrfsic_submit_bh(REQ_OP_WRITE, WRITE_SYNC, bh); if (ret) errors++; } diff --git a/fs/buffer.c b/fs/buffer.c index 63d0aed..0843964 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -45,7 +45,7 @@ #include static int fsync_buffers_list(spinlock_t *lock, struct list_head *list); -static int submit_bh_wbc(int rw, struct buffer_head *bh, +static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, unsigned long bio_flags, struct writeback_control *wbc); @@ -1236,7 +1236,7 @@ static struct buffer_head *__bread_slow(struct buffer_head *bh) } else { get_bh(bh); bh->b_end_io = end_buffer_read_sync; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); wait_on_buffer(bh); if (buffer_uptodate(bh)) return bh; @@ -1708,7 +1708,7 @@ static int __block_write_full_page(struct inode *inode, struct page *page, struct buffer_head *bh, *head; unsigned int blocksize, bbits; int nr_underway = 0; - int write_op = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + int write_flags = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC: 0); head = create_page_buffers(page, inode, (1 << BH_Dirty)|(1 << BH_Uptodate)); @@ -1797,7 +1797,7 @@ static int __block_write_full_page(struct inode *inode, struct page *page, do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh_wbc(write_op, bh, 0, wbc); + submit_bh_wbc(REQ_OP_WRITE, write_flags, bh, 0, wbc); nr_underway++; } bh = next; @@ -1851,7 +1851,7 @@ recover: struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { clear_buffer_dirty(bh); - submit_bh_wbc(write_op, bh, 0, wbc); + submit_bh_wbc(REQ_OP_WRITE, write_flags, bh, 0, wbc); nr_underway++; } bh = next; @@ -2259,7 +2259,7 @@ int block_read_full_page(struct page *page, get_block_t *get_block) if (buffer_uptodate(bh)) end_buffer_async_read(bh, 1); else - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); } return 0; } @@ -2593,7 +2593,7 @@ int nobh_write_begin(struct address_space *mapping, if (block_start < from || block_end > to) { lock_buffer(bh); bh->b_end_io = end_buffer_read_nobh; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); nr_reads++; } } @@ -2960,7 +2960,7 @@ static void end_bio_bh_io_sync(struct bio *bio) * errors, this only handles the "we need to be able to * do IO at the final sector" case. */ -void guard_bio_eod(int rw, struct bio *bio) +void guard_bio_eod(int op, struct bio *bio) { sector_t maxsector; struct bio_vec *bvec = &bio->bi_io_vec[bio->bi_vcnt - 1]; @@ -2990,13 +2990,13 @@ void guard_bio_eod(int rw, struct bio *bio) bvec->bv_len -= truncated_bytes; /* ..and clear the end of the buffer for reads */ - if ((rw & RW_MASK) == READ) { + if (op == REQ_OP_READ) { zero_user(bvec->bv_page, bvec->bv_offset + bvec->bv_len, truncated_bytes); } } -static int submit_bh_wbc(int rw, struct buffer_head *bh, +static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, unsigned long bio_flags, struct writeback_control *wbc) { struct bio *bio; @@ -3010,7 +3010,7 @@ static int submit_bh_wbc(int rw, struct buffer_head *bh, /* * Only clear out a write error when rewriting */ - if (test_set_buffer_req(bh) && (rw & WRITE)) + if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE)) clear_buffer_write_io_error(bh); /* @@ -3033,29 +3033,31 @@ static int submit_bh_wbc(int rw, struct buffer_head *bh, bio->bi_end_io = end_bio_bh_io_sync; bio->bi_private = bh; bio->bi_flags |= bio_flags; + bio->bi_op = op; /* Take care of bh's that straddle the end of the device */ - guard_bio_eod(rw, bio); + guard_bio_eod(op, bio); if (buffer_meta(bh)) - rw |= REQ_META; + op_flags |= REQ_META; if (buffer_prio(bh)) - rw |= REQ_PRIO; - bio->bi_rw |= rw; + op_flags |= REQ_PRIO; + bio->bi_rw |= op_flags; submit_bio(bio); return 0; } -int _submit_bh(int rw, struct buffer_head *bh, unsigned long bio_flags) +int _submit_bh(int op, int op_flags, struct buffer_head *bh, + unsigned long bio_flags) { - return submit_bh_wbc(rw, bh, bio_flags, NULL); + return submit_bh_wbc(op, op_flags, bh, bio_flags, NULL); } EXPORT_SYMBOL_GPL(_submit_bh); -int submit_bh(int rw, struct buffer_head *bh) +int submit_bh(int op, int op_flags, struct buffer_head *bh) { - return submit_bh_wbc(rw, bh, 0, NULL); + return submit_bh_wbc(op, op_flags, bh, 0, NULL); } EXPORT_SYMBOL(submit_bh); @@ -3097,14 +3099,14 @@ void ll_rw_block(int rw, int nr, struct buffer_head *bhs[]) if (test_clear_buffer_dirty(bh)) { bh->b_end_io = end_buffer_write_sync; get_bh(bh); - submit_bh(WRITE, bh); + submit_bh(rw, 0, bh); continue; } } else { if (!buffer_uptodate(bh)) { bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(rw, bh); + submit_bh(rw, 0, bh); continue; } } @@ -3113,7 +3115,7 @@ void ll_rw_block(int rw, int nr, struct buffer_head *bhs[]) } EXPORT_SYMBOL(ll_rw_block); -void write_dirty_buffer(struct buffer_head *bh, int rw) +void write_dirty_buffer(struct buffer_head *bh, int op_flags) { lock_buffer(bh); if (!test_clear_buffer_dirty(bh)) { @@ -3122,7 +3124,7 @@ void write_dirty_buffer(struct buffer_head *bh, int rw) } bh->b_end_io = end_buffer_write_sync; get_bh(bh); - submit_bh(rw, bh); + submit_bh(REQ_OP_WRITE, op_flags, bh); } EXPORT_SYMBOL(write_dirty_buffer); @@ -3131,7 +3133,7 @@ EXPORT_SYMBOL(write_dirty_buffer); * and then start new I/O and then wait upon it. The caller must have a ref on * the buffer_head. */ -int __sync_dirty_buffer(struct buffer_head *bh, int rw) +int __sync_dirty_buffer(struct buffer_head *bh, int op_flags) { int ret = 0; @@ -3140,7 +3142,7 @@ int __sync_dirty_buffer(struct buffer_head *bh, int rw) if (test_clear_buffer_dirty(bh)) { get_bh(bh); bh->b_end_io = end_buffer_write_sync; - ret = submit_bh(rw, bh); + ret = submit_bh(REQ_OP_WRITE, op_flags, bh); wait_on_buffer(bh); if (!ret && !buffer_uptodate(bh)) ret = -EIO; @@ -3403,7 +3405,7 @@ int bh_submit_read(struct buffer_head *bh) get_bh(bh); bh->b_end_io = end_buffer_read_sync; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); wait_on_buffer(bh); if (buffer_uptodate(bh)) return 0; diff --git a/fs/ext4/balloc.c b/fs/ext4/balloc.c index ec0668a..b03bfd8 100644 --- a/fs/ext4/balloc.c +++ b/fs/ext4/balloc.c @@ -469,7 +469,7 @@ ext4_read_block_bitmap_nowait(struct super_block *sb, ext4_group_t block_group) trace_ext4_read_block_bitmap_load(sb, block_group); bh->b_end_io = ext4_end_bitmap_read; get_bh(bh); - submit_bh(READ | REQ_META | REQ_PRIO, bh); + submit_bh(REQ_OP_READ, REQ_META | REQ_PRIO, bh); return bh; verify: err = ext4_validate_block_bitmap(sb, desc, block_group, bh); diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c index e86e33c..0001b19 100644 --- a/fs/ext4/ialloc.c +++ b/fs/ext4/ialloc.c @@ -212,7 +212,7 @@ ext4_read_inode_bitmap(struct super_block *sb, ext4_group_t block_group) trace_ext4_load_inode_bitmap(sb, block_group); bh->b_end_io = ext4_end_bitmap_read; get_bh(bh); - submit_bh(READ | REQ_META | REQ_PRIO, bh); + submit_bh(REQ_OP_READ, REQ_META | REQ_PRIO, bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) { put_bh(bh); diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 032e6f2..4fc178a 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -4094,7 +4094,7 @@ make_io: trace_ext4_load_inode(inode); get_bh(bh); bh->b_end_io = end_buffer_read_sync; - submit_bh(READ | REQ_META | REQ_PRIO, bh); + submit_bh(REQ_OP_READ, REQ_META | REQ_PRIO, bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) { EXT4_ERROR_INODE_BLOCK(inode, block, diff --git a/fs/ext4/mmp.c b/fs/ext4/mmp.c index 0a512aa..bd90abf 100644 --- a/fs/ext4/mmp.c +++ b/fs/ext4/mmp.c @@ -52,7 +52,7 @@ static int write_mmp_block(struct super_block *sb, struct buffer_head *bh) lock_buffer(bh); bh->b_end_io = end_buffer_write_sync; get_bh(bh); - submit_bh(WRITE_SYNC | REQ_META | REQ_PRIO, bh); + submit_bh(REQ_OP_WRITE, WRITE_SYNC | REQ_META | REQ_PRIO, bh); wait_on_buffer(bh); sb_end_write(sb); if (unlikely(!buffer_uptodate(bh))) @@ -88,7 +88,7 @@ static int read_mmp_block(struct super_block *sb, struct buffer_head **bh, get_bh(*bh); lock_buffer(*bh); (*bh)->b_end_io = end_buffer_read_sync; - submit_bh(READ_SYNC | REQ_META | REQ_PRIO, *bh); + submit_bh(REQ_OP_READ, READ_SYNC | REQ_META | REQ_PRIO, *bh); wait_on_buffer(*bh); if (!buffer_uptodate(*bh)) { brelse(*bh); diff --git a/fs/fat/misc.c b/fs/fat/misc.c index c4589e9..8a86981 100644 --- a/fs/fat/misc.c +++ b/fs/fat/misc.c @@ -267,7 +267,7 @@ int fat_sync_bhs(struct buffer_head **bhs, int nr_bhs) int i, err = 0; for (i = 0; i < nr_bhs; i++) - write_dirty_buffer(bhs[i], WRITE); + write_dirty_buffer(bhs[i], 0); for (i = 0; i < nr_bhs; i++) { wait_on_buffer(bhs[i]); diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c index 0860f0b..29faf72 100644 --- a/fs/gfs2/bmap.c +++ b/fs/gfs2/bmap.c @@ -285,7 +285,7 @@ static void gfs2_metapath_ra(struct gfs2_glock *gl, if (trylock_buffer(rabh)) { if (!buffer_uptodate(rabh)) { rabh->b_end_io = end_buffer_read_sync; - submit_bh(READA | REQ_META, rabh); + submit_bh(REQ_OP_READ, READA | REQ_META, rabh); continue; } unlock_buffer(rabh); diff --git a/fs/gfs2/dir.c b/fs/gfs2/dir.c index 6a92592..2b81841 100644 --- a/fs/gfs2/dir.c +++ b/fs/gfs2/dir.c @@ -1510,7 +1510,7 @@ static void gfs2_dir_readahead(struct inode *inode, unsigned hsize, u32 index, continue; } bh->b_end_io = end_buffer_read_sync; - submit_bh(READA | REQ_META, bh); + submit_bh(REQ_OP_READ, READA | REQ_META, bh); continue; } brelse(bh); diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index 013886d..b54a6fa 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -37,8 +37,8 @@ static int gfs2_aspace_writepage(struct page *page, struct writeback_control *wb { struct buffer_head *bh, *head; int nr_underway = 0; - int write_op = REQ_META | REQ_PRIO | - (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + int write_flags = REQ_META | REQ_PRIO | + (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : 0); BUG_ON(!PageLocked(page)); BUG_ON(!page_has_buffers(page)); @@ -79,7 +79,7 @@ static int gfs2_aspace_writepage(struct page *page, struct writeback_control *wb do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh(write_op, bh); + submit_bh(REQ_OP_WRITE, write_flags, bh); nr_underway++; } bh = next; diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c index 36345fe..7390e3a 100644 --- a/fs/jbd2/commit.c +++ b/fs/jbd2/commit.c @@ -157,9 +157,9 @@ static int journal_submit_commit_record(journal_t *journal, if (journal->j_flags & JBD2_BARRIER && !jbd2_has_feature_async_commit(journal)) - ret = submit_bh(WRITE_SYNC | WRITE_FLUSH_FUA, bh); + ret = submit_bh(REQ_OP_WRITE, WRITE_SYNC | WRITE_FLUSH_FUA, bh); else - ret = submit_bh(WRITE_SYNC, bh); + ret = submit_bh(REQ_OP_WRITE, WRITE_SYNC, bh); *cbh = bh; return ret; @@ -737,7 +737,7 @@ start_journal_io: clear_buffer_dirty(bh); set_buffer_uptodate(bh); bh->b_end_io = journal_end_buffer_io_sync; - submit_bh(WRITE_SYNC, bh); + submit_bh(REQ_OP_WRITE, WRITE_SYNC, bh); } cond_resched(); stats.run.rs_blocks_logged += bufs; diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c index 81e6226..3b7db0b 100644 --- a/fs/jbd2/journal.c +++ b/fs/jbd2/journal.c @@ -1323,15 +1323,15 @@ static int journal_reset(journal_t *journal) return jbd2_journal_start_thread(journal); } -static int jbd2_write_superblock(journal_t *journal, int write_op) +static int jbd2_write_superblock(journal_t *journal, int write_flags) { struct buffer_head *bh = journal->j_sb_buffer; journal_superblock_t *sb = journal->j_superblock; int ret; - trace_jbd2_write_superblock(journal, write_op); + trace_jbd2_write_superblock(journal, write_flags); if (!(journal->j_flags & JBD2_BARRIER)) - write_op &= ~(REQ_FUA | REQ_FLUSH); + write_flags &= ~(REQ_FUA | REQ_FLUSH); lock_buffer(bh); if (buffer_write_io_error(bh)) { /* @@ -1351,7 +1351,7 @@ static int jbd2_write_superblock(journal_t *journal, int write_op) jbd2_superblock_csum_set(journal, sb); get_bh(bh); bh->b_end_io = end_buffer_write_sync; - ret = submit_bh(write_op, bh); + ret = submit_bh(REQ_OP_WRITE, write_flags, bh); wait_on_buffer(bh); if (buffer_write_io_error(bh)) { clear_buffer_write_io_error(bh); diff --git a/fs/nilfs2/btnode.c b/fs/nilfs2/btnode.c index a35ae35..07fe874 100644 --- a/fs/nilfs2/btnode.c +++ b/fs/nilfs2/btnode.c @@ -67,7 +67,7 @@ nilfs_btnode_create_block(struct address_space *btnc, __u64 blocknr) } int nilfs_btnode_submit_block(struct address_space *btnc, __u64 blocknr, - sector_t pblocknr, int mode, + sector_t pblocknr, int mode, int mode_flags, struct buffer_head **pbh, sector_t *submit_ptr) { struct buffer_head *bh; @@ -100,7 +100,7 @@ int nilfs_btnode_submit_block(struct address_space *btnc, __u64 blocknr, } } - if (mode == READA) { + if (mode_flags & REQ_RAHEAD) { if (pblocknr != *submit_ptr + 1 || !trylock_buffer(bh)) { err = -EBUSY; /* internal code */ brelse(bh); @@ -119,7 +119,7 @@ int nilfs_btnode_submit_block(struct address_space *btnc, __u64 blocknr, bh->b_blocknr = pblocknr; /* set block address for read */ bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(mode, bh); + submit_bh(mode, mode_flags, bh); bh->b_blocknr = blocknr; /* set back to the given block address */ *submit_ptr = pblocknr; err = 0; diff --git a/fs/nilfs2/btnode.h b/fs/nilfs2/btnode.h index d876b56..3f93197 100644 --- a/fs/nilfs2/btnode.h +++ b/fs/nilfs2/btnode.h @@ -47,7 +47,7 @@ void nilfs_btnode_cache_clear(struct address_space *); struct buffer_head *nilfs_btnode_create_block(struct address_space *btnc, __u64 blocknr); int nilfs_btnode_submit_block(struct address_space *, __u64, sector_t, int, - struct buffer_head **, sector_t *); + int, struct buffer_head **, sector_t *); void nilfs_btnode_delete(struct buffer_head *); int nilfs_btnode_prepare_change_key(struct address_space *, struct nilfs_btnode_chkey_ctxt *); diff --git a/fs/nilfs2/btree.c b/fs/nilfs2/btree.c index 3a3821b..5d6a2c6 100644 --- a/fs/nilfs2/btree.c +++ b/fs/nilfs2/btree.c @@ -480,7 +480,8 @@ static int __nilfs_btree_get_block(const struct nilfs_bmap *btree, __u64 ptr, sector_t submit_ptr = 0; int ret; - ret = nilfs_btnode_submit_block(btnc, ptr, 0, READ, &bh, &submit_ptr); + ret = nilfs_btnode_submit_block(btnc, ptr, 0, REQ_OP_READ, 0, &bh, + &submit_ptr); if (ret) { if (ret != -EEXIST) return ret; @@ -496,7 +497,8 @@ static int __nilfs_btree_get_block(const struct nilfs_bmap *btree, __u64 ptr, n > 0 && i < ra->ncmax; n--, i++) { ptr2 = nilfs_btree_node_get_ptr(ra->node, i, ra->ncmax); - ret = nilfs_btnode_submit_block(btnc, ptr2, 0, READA, + ret = nilfs_btnode_submit_block(btnc, ptr2, 0, + REQ_OP_READ, REQ_RAHEAD, &ra_bh, &submit_ptr); if (likely(!ret || ret == -EEXIST)) brelse(ra_bh); diff --git a/fs/nilfs2/gcinode.c b/fs/nilfs2/gcinode.c index 748ca23..1f18ffc 100644 --- a/fs/nilfs2/gcinode.c +++ b/fs/nilfs2/gcinode.c @@ -106,7 +106,7 @@ int nilfs_gccache_submit_read_data(struct inode *inode, sector_t blkoff, bh->b_blocknr = pbn; bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); if (vbn) bh->b_blocknr = vbn; out: @@ -143,7 +143,8 @@ int nilfs_gccache_submit_read_node(struct inode *inode, sector_t pbn, int ret; ret = nilfs_btnode_submit_block(&NILFS_I(inode)->i_btnode_cache, - vbn ? : pbn, pbn, READ, out_bh, &pbn); + vbn ? : pbn, pbn, REQ_OP_READ, 0, + out_bh, &pbn); if (ret == -EEXIST) /* internal code (cache hit) */ ret = 0; return ret; diff --git a/fs/nilfs2/mdt.c b/fs/nilfs2/mdt.c index 1125f40..92e627d 100644 --- a/fs/nilfs2/mdt.c +++ b/fs/nilfs2/mdt.c @@ -124,7 +124,7 @@ static int nilfs_mdt_create_block(struct inode *inode, unsigned long block, static int nilfs_mdt_submit_block(struct inode *inode, unsigned long blkoff, - int mode, struct buffer_head **out_bh) + int mode, int mode_flags, struct buffer_head **out_bh) { struct buffer_head *bh; __u64 blknum = 0; @@ -138,7 +138,7 @@ nilfs_mdt_submit_block(struct inode *inode, unsigned long blkoff, if (buffer_uptodate(bh)) goto out; - if (mode == READA) { + if (mode_flags & REQ_RAHEAD) { if (!trylock_buffer(bh)) { ret = -EBUSY; goto failed_bh; @@ -160,7 +160,7 @@ nilfs_mdt_submit_block(struct inode *inode, unsigned long blkoff, bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(mode, bh); + submit_bh(mode, mode_flags, bh); ret = 0; trace_nilfs2_mdt_submit_block(inode, inode->i_ino, blkoff, mode); @@ -184,7 +184,7 @@ static int nilfs_mdt_read_block(struct inode *inode, unsigned long block, int i, nr_ra_blocks = NILFS_MDT_MAX_RA_BLOCKS; int err; - err = nilfs_mdt_submit_block(inode, block, READ, &first_bh); + err = nilfs_mdt_submit_block(inode, block, REQ_OP_READ, 0, &first_bh); if (err == -EEXIST) /* internal code */ goto out; @@ -194,7 +194,8 @@ static int nilfs_mdt_read_block(struct inode *inode, unsigned long block, if (readahead) { blkoff = block + 1; for (i = 0; i < nr_ra_blocks; i++, blkoff++) { - err = nilfs_mdt_submit_block(inode, blkoff, READA, &bh); + err = nilfs_mdt_submit_block(inode, blkoff, REQ_OP_READ, + REQ_RAHEAD, &bh); if (likely(!err || err == -EEXIST)) brelse(bh); else if (err != -EBUSY) diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c index 7521e11..57c64bd 100644 --- a/fs/ntfs/aops.c +++ b/fs/ntfs/aops.c @@ -362,7 +362,7 @@ handle_zblock: for (i = 0; i < nr; i++) { tbh = arr[i]; if (likely(!buffer_uptodate(tbh))) - submit_bh(READ, tbh); + submit_bh(REQ_OP_READ, 0, tbh); else ntfs_end_buffer_async_read(tbh, 1); } @@ -877,7 +877,7 @@ lock_retry_remap: do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); need_end_writeback = false; } bh = next; @@ -1202,7 +1202,7 @@ lock_retry_remap: BUG_ON(!buffer_mapped(tbh)); get_bh(tbh); tbh->b_end_io = end_buffer_write_sync; - submit_bh(WRITE, tbh); + submit_bh(REQ_OP_WRITE, 0, tbh); } /* Synchronize the mft mirror now if not @sync. */ if (is_mft && !sync) diff --git a/fs/ntfs/compress.c b/fs/ntfs/compress.c index f82498c..1fd482c 100644 --- a/fs/ntfs/compress.c +++ b/fs/ntfs/compress.c @@ -674,7 +674,7 @@ lock_retry_remap: } get_bh(tbh); tbh->b_end_io = end_buffer_read_sync; - submit_bh(READ, tbh); + submit_bh(REQ_OP_READ, 0, tbh); } /* Wait for io completion on all buffer heads. */ diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c index 9d383e5..03a1a32 100644 --- a/fs/ntfs/file.c +++ b/fs/ntfs/file.c @@ -553,7 +553,7 @@ static inline int ntfs_submit_bh_for_read(struct buffer_head *bh) lock_buffer(bh); get_bh(bh); bh->b_end_io = end_buffer_read_sync; - return submit_bh(READ, bh); + return submit_bh(REQ_OP_READ, 0, bh); } /** diff --git a/fs/ntfs/logfile.c b/fs/ntfs/logfile.c index c71de29..1c95c41 100644 --- a/fs/ntfs/logfile.c +++ b/fs/ntfs/logfile.c @@ -821,7 +821,7 @@ map_vcn: * completed ignore errors afterwards as we can assume * that if one buffer worked all of them will work. */ - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); if (should_wait) { should_wait = false; wait_on_buffer(bh); diff --git a/fs/ntfs/mft.c b/fs/ntfs/mft.c index 3014a36..38c6f7a 100644 --- a/fs/ntfs/mft.c +++ b/fs/ntfs/mft.c @@ -592,7 +592,7 @@ int ntfs_sync_mft_mirror(ntfs_volume *vol, const unsigned long mft_no, clear_buffer_dirty(tbh); get_bh(tbh); tbh->b_end_io = end_buffer_write_sync; - submit_bh(WRITE, tbh); + submit_bh(REQ_OP_WRITE, 0, tbh); } /* Wait on i/o completion of buffers. */ for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) { @@ -785,7 +785,7 @@ int write_mft_record_nolock(ntfs_inode *ni, MFT_RECORD *m, int sync) clear_buffer_dirty(tbh); get_bh(tbh); tbh->b_end_io = end_buffer_write_sync; - submit_bh(WRITE, tbh); + submit_bh(REQ_OP_WRITE, 0, tbh); } /* Synchronize the mft mirror now if not @sync. */ if (!sync && ni->mft_no < vol->mftmirr_size) diff --git a/fs/ocfs2/buffer_head_io.c b/fs/ocfs2/buffer_head_io.c index fe50ded..fb775c9 100644 --- a/fs/ocfs2/buffer_head_io.c +++ b/fs/ocfs2/buffer_head_io.c @@ -79,7 +79,7 @@ int ocfs2_write_block(struct ocfs2_super *osb, struct buffer_head *bh, get_bh(bh); /* for end_buffer_write_sync() */ bh->b_end_io = end_buffer_write_sync; - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); wait_on_buffer(bh); @@ -149,7 +149,7 @@ int ocfs2_read_blocks_sync(struct ocfs2_super *osb, u64 block, clear_buffer_uptodate(bh); get_bh(bh); /* for end_buffer_read_sync() */ bh->b_end_io = end_buffer_read_sync; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); } for (i = nr; i > 0; i--) { @@ -305,7 +305,7 @@ int ocfs2_read_blocks(struct ocfs2_caching_info *ci, u64 block, int nr, if (validate) set_buffer_needs_validate(bh); bh->b_end_io = end_buffer_read_sync; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); continue; } } @@ -419,7 +419,7 @@ int ocfs2_write_super_or_backup(struct ocfs2_super *osb, get_bh(bh); /* for end_buffer_write_sync() */ bh->b_end_io = end_buffer_write_sync; ocfs2_compute_meta_ecc(osb->sb, bh->b_data, &di->i_check); - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); wait_on_buffer(bh); diff --git a/fs/reiserfs/inode.c b/fs/reiserfs/inode.c index ae9e5b3..91155ed 100644 --- a/fs/reiserfs/inode.c +++ b/fs/reiserfs/inode.c @@ -2668,7 +2668,7 @@ static int reiserfs_write_full_page(struct page *page, do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); nr++; } put_bh(bh); @@ -2728,7 +2728,7 @@ fail: struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { clear_buffer_dirty(bh); - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); nr++; } put_bh(bh); diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c index 9d6486d..9e63bc2 100644 --- a/fs/reiserfs/journal.c +++ b/fs/reiserfs/journal.c @@ -654,7 +654,7 @@ static void submit_logged_buffer(struct buffer_head *bh) BUG(); if (!buffer_uptodate(bh)) BUG(); - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); } static void submit_ordered_buffer(struct buffer_head *bh) @@ -664,7 +664,7 @@ static void submit_ordered_buffer(struct buffer_head *bh) clear_buffer_dirty(bh); if (!buffer_uptodate(bh)) BUG(); - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); } #define CHUNK_SIZE 32 @@ -2271,7 +2271,7 @@ abort_replay: /* flush out the real blocks */ for (i = 0; i < get_desc_trans_len(desc); i++) { set_buffer_dirty(real_blocks[i]); - write_dirty_buffer(real_blocks[i], WRITE); + write_dirty_buffer(real_blocks[i], 0); } for (i = 0; i < get_desc_trans_len(desc); i++) { wait_on_buffer(real_blocks[i]); diff --git a/fs/ufs/util.c b/fs/ufs/util.c index b6c2f94..5f68b1a 100644 --- a/fs/ufs/util.c +++ b/fs/ufs/util.c @@ -118,7 +118,7 @@ void ubh_sync_block(struct ufs_buffer_head *ubh) unsigned i; for (i = 0; i < ubh->count; i++) - write_dirty_buffer(ubh->bh[i], WRITE); + write_dirty_buffer(ubh->bh[i], 0); for (i = 0; i < ubh->count; i++) wait_on_buffer(ubh->bh[i]); diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index 89d9aa9..f7cc163 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -189,10 +189,11 @@ void unlock_buffer(struct buffer_head *bh); void __lock_buffer(struct buffer_head *bh); void ll_rw_block(int, int, struct buffer_head * bh[]); int sync_dirty_buffer(struct buffer_head *bh); -int __sync_dirty_buffer(struct buffer_head *bh, int rw); -void write_dirty_buffer(struct buffer_head *bh, int rw); -int _submit_bh(int rw, struct buffer_head *bh, unsigned long bio_flags); -int submit_bh(int, struct buffer_head *); +int __sync_dirty_buffer(struct buffer_head *bh, int op_flags); +void write_dirty_buffer(struct buffer_head *bh, int op_flags); +int _submit_bh(int op, int op_flags, struct buffer_head *bh, + unsigned long bio_flags); +int submit_bh(int, int, struct buffer_head *); void write_boundary_block(struct block_device *bdev, sector_t bblock, unsigned blocksize); int bh_uptodate_or_lock(struct buffer_head *bh); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:02 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 966C229E0C for ; Tue, 5 Jan 2016 14:54:02 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4DBEE8F804C for ; Tue, 5 Jan 2016 12:54:02 -0800 (PST) X-ASG-Debug-ID: 1452027240-04bdf03af904db0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id XQ20vroXhr2mxC6c (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:00 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id D7B9D1317; Tue, 5 Jan 2016 20:53:59 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu4024032; Tue, 5 Jan 2016 15:53:57 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 05/35] fs: have ll_rw_block users pass in op and flags separately Date: Tue, 5 Jan 2016 14:53:08 -0600 X-ASG-Orig-Subj: [PATCH 05/35] fs: have ll_rw_block users pass in op and flags separately Message-Id: <1452027218-32303-6-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027240 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This has ll_rw_block users pass in the operation and flags separately, so we can setup the bio->bi_op and bio-bi_rw flags. Signed-off-by: Mike Christie --- fs/buffer.c | 19 ++++++++++--------- fs/ext4/inode.c | 6 +++--- fs/ext4/namei.c | 2 +- fs/ext4/super.c | 2 +- fs/gfs2/bmap.c | 2 +- fs/gfs2/meta_io.c | 4 ++-- fs/gfs2/quota.c | 2 +- fs/isofs/compress.c | 2 +- fs/jbd2/journal.c | 2 +- fs/jbd2/recovery.c | 4 ++-- fs/ocfs2/aops.c | 2 +- fs/ocfs2/super.c | 2 +- fs/reiserfs/journal.c | 8 ++++---- fs/reiserfs/stree.c | 4 ++-- fs/reiserfs/super.c | 2 +- fs/squashfs/block.c | 4 ++-- fs/udf/dir.c | 2 +- fs/udf/directory.c | 2 +- fs/udf/inode.c | 2 +- fs/ufs/balloc.c | 2 +- include/linux/buffer_head.h | 2 +- 21 files changed, 39 insertions(+), 38 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 0843964..1a14bf2 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -595,7 +595,7 @@ void write_boundary_block(struct block_device *bdev, struct buffer_head *bh = __find_get_block(bdev, bblock + 1, blocksize); if (bh) { if (buffer_dirty(bh)) - ll_rw_block(WRITE, 1, &bh); + ll_rw_block(REQ_OP_WRITE, 0, 1, &bh); put_bh(bh); } } @@ -1406,7 +1406,7 @@ void __breadahead(struct block_device *bdev, sector_t block, unsigned size) { struct buffer_head *bh = __getblk(bdev, block, size); if (likely(bh)) { - ll_rw_block(READA, 1, &bh); + ll_rw_block(REQ_OP_READ, READA, 1, &bh); brelse(bh); } } @@ -1966,7 +1966,7 @@ int __block_write_begin(struct page *page, loff_t pos, unsigned len, if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh) && (block_start < from || block_end > to)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); *wait_bh++=bh; } } @@ -2863,7 +2863,7 @@ int block_truncate_page(struct address_space *mapping, if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh)) { err = -EIO; - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); /* Uhhuh. Read error. Complain and punt. */ if (!buffer_uptodate(bh)) @@ -3063,7 +3063,8 @@ EXPORT_SYMBOL(submit_bh); /** * ll_rw_block: low-level access to block devices (DEPRECATED) - * @rw: whether to %READ or %WRITE or maybe %READA (readahead) + * @op: whether to %READ or %WRITE + * op_flags: rq_flag_bits or %READA (readahead) * @nr: number of &struct buffer_heads in the array * @bhs: array of pointers to &struct buffer_head * @@ -3086,7 +3087,7 @@ EXPORT_SYMBOL(submit_bh); * All of the buffers must be for the same device, and must also be a * multiple of the current approved size for the device. */ -void ll_rw_block(int rw, int nr, struct buffer_head *bhs[]) +void ll_rw_block(int op, int op_flags, int nr, struct buffer_head *bhs[]) { int i; @@ -3095,18 +3096,18 @@ void ll_rw_block(int rw, int nr, struct buffer_head *bhs[]) if (!trylock_buffer(bh)) continue; - if (rw == WRITE) { + if (op == WRITE) { if (test_clear_buffer_dirty(bh)) { bh->b_end_io = end_buffer_write_sync; get_bh(bh); - submit_bh(rw, 0, bh); + submit_bh(op, op_flags, bh); continue; } } else { if (!buffer_uptodate(bh)) { bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(rw, 0, bh); + submit_bh(op, op_flags, bh); continue; } } diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 4fc178a..26a07cb 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -810,7 +810,7 @@ struct buffer_head *ext4_bread(handle_t *handle, struct inode *inode, return bh; if (!bh || buffer_uptodate(bh)) return bh; - ll_rw_block(READ | REQ_META | REQ_PRIO, 1, &bh); + ll_rw_block(REQ_OP_READ, REQ_META | REQ_PRIO, 1, &bh); wait_on_buffer(bh); if (buffer_uptodate(bh)) return bh; @@ -964,7 +964,7 @@ static int ext4_block_write_begin(struct page *page, loff_t pos, unsigned len, if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh) && (block_start < from || block_end > to)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); *wait_bh++ = bh; decrypt = ext4_encrypted_inode(inode) && S_ISREG(inode->i_mode); @@ -3511,7 +3511,7 @@ static int __ext4_block_zero_page_range(handle_t *handle, if (!buffer_uptodate(bh)) { err = -EIO; - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); /* Uhhuh. Read error. Complain and punt. */ if (!buffer_uptodate(bh)) diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c index 71d1021..714bf29 100644 --- a/fs/ext4/namei.c +++ b/fs/ext4/namei.c @@ -1440,7 +1440,7 @@ restart: } bh_use[ra_max] = bh; if (bh) - ll_rw_block(READ | REQ_META | REQ_PRIO, + ll_rw_block(REQ_OP_READ, REQ_META | REQ_PRIO, 1, &bh); } } diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 50b2207..380aadc 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -4204,7 +4204,7 @@ static journal_t *ext4_get_dev_journal(struct super_block *sb, goto out_bdev; } journal->j_private = sb; - ll_rw_block(READ | REQ_META | REQ_PRIO, 1, &journal->j_sb_buffer); + ll_rw_block(REQ_OP_READ, REQ_META | REQ_PRIO, 1, &journal->j_sb_buffer); wait_on_buffer(journal->j_sb_buffer); if (!buffer_uptodate(journal->j_sb_buffer)) { ext4_msg(sb, KERN_ERR, "I/O error on journal device"); diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c index 29faf72..628d4a4 100644 --- a/fs/gfs2/bmap.c +++ b/fs/gfs2/bmap.c @@ -974,7 +974,7 @@ static int gfs2_block_truncate_page(struct address_space *mapping, loff_t from) if (!buffer_uptodate(bh)) { err = -EIO; - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); /* Uhhuh. Read error. Complain and punt. */ if (!buffer_uptodate(bh)) diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index b54a6fa..e2fc2b9 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -448,7 +448,7 @@ struct buffer_head *gfs2_meta_ra(struct gfs2_glock *gl, u64 dblock, u32 extlen) if (buffer_uptodate(first_bh)) goto out; if (!buffer_locked(first_bh)) - ll_rw_block(READ_SYNC | REQ_META, 1, &first_bh); + ll_rw_block(REQ_OP_READ, READ_SYNC | REQ_META, 1, &first_bh); dblock++; extlen--; @@ -457,7 +457,7 @@ struct buffer_head *gfs2_meta_ra(struct gfs2_glock *gl, u64 dblock, u32 extlen) bh = gfs2_getbuf(gl, dblock, CREATE); if (!buffer_uptodate(bh) && !buffer_locked(bh)) - ll_rw_block(READA | REQ_META, 1, &bh); + ll_rw_block(REQ_OP_READ, READA | REQ_META, 1, &bh); brelse(bh); dblock++; extlen--; diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c index be6d9c4..13be922 100644 --- a/fs/gfs2/quota.c +++ b/fs/gfs2/quota.c @@ -730,7 +730,7 @@ static int gfs2_write_buf_to_page(struct gfs2_inode *ip, unsigned long index, if (PageUptodate(page)) set_buffer_uptodate(bh); if (!buffer_uptodate(bh)) { - ll_rw_block(READ | REQ_META, 1, &bh); + ll_rw_block(REQ_OP_READ, REQ_META, 1, &bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) goto unlock_out; diff --git a/fs/isofs/compress.c b/fs/isofs/compress.c index f311bf0..f9b5fc8 100644 --- a/fs/isofs/compress.c +++ b/fs/isofs/compress.c @@ -81,7 +81,7 @@ static loff_t zisofs_uncompress_block(struct inode *inode, loff_t block_start, blocknum = block_start >> bufshift; memset(bhs, 0, (needblocks + 1) * sizeof(struct buffer_head *)); haveblocks = isofs_get_blocks(inode, blocknum, bhs, needblocks); - ll_rw_block(READ, haveblocks, bhs); + ll_rw_block(REQ_OP_READ, 0, haveblocks, bhs); curbh = 0; curpage = 0; diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c index 3b7db0b..7104d76 100644 --- a/fs/jbd2/journal.c +++ b/fs/jbd2/journal.c @@ -1474,7 +1474,7 @@ static int journal_get_superblock(journal_t *journal) J_ASSERT(bh != NULL); if (!buffer_uptodate(bh)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) { printk(KERN_ERR diff --git a/fs/jbd2/recovery.c b/fs/jbd2/recovery.c index 7f277e4..8d66ea9 100644 --- a/fs/jbd2/recovery.c +++ b/fs/jbd2/recovery.c @@ -104,7 +104,7 @@ static int do_readahead(journal_t *journal, unsigned int start) if (!buffer_uptodate(bh) && !buffer_locked(bh)) { bufs[nbufs++] = bh; if (nbufs == MAXBUF) { - ll_rw_block(READ, nbufs, bufs); + ll_rw_block(REQ_OP_READ, 0, nbufs, bufs); journal_brelse_array(bufs, nbufs); nbufs = 0; } @@ -113,7 +113,7 @@ static int do_readahead(journal_t *journal, unsigned int start) } if (nbufs) - ll_rw_block(READ, nbufs, bufs); + ll_rw_block(REQ_OP_READ, 0, nbufs, bufs); err = 0; failed: diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c index 182f754..2e761fa 100644 --- a/fs/ocfs2/aops.c +++ b/fs/ocfs2/aops.c @@ -640,7 +640,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, !buffer_new(bh) && ocfs2_should_read_blk(inode, page, block_start) && (block_start < from || block_end > to)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); *wait_bh++=bh; } diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index 45b6139..9550593 100644 --- a/fs/ocfs2/super.c +++ b/fs/ocfs2/super.c @@ -1818,7 +1818,7 @@ static int ocfs2_get_sector(struct super_block *sb, if (!buffer_dirty(*bh)) clear_buffer_uptodate(*bh); unlock_buffer(*bh); - ll_rw_block(READ, 1, bh); + ll_rw_block(REQ_OP_READ, 0, 1, bh); wait_on_buffer(*bh); if (!buffer_uptodate(*bh)) { mlog_errno(-EIO); diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c index 9e63bc2..5c14c90 100644 --- a/fs/reiserfs/journal.c +++ b/fs/reiserfs/journal.c @@ -872,7 +872,7 @@ loop_next: */ if (buffer_dirty(bh) && unlikely(bh->b_page->mapping == NULL)) { spin_unlock(lock); - ll_rw_block(WRITE, 1, &bh); + ll_rw_block(REQ_OP_WRITE, 0, 1, &bh); spin_lock(lock); } put_bh(bh); @@ -1059,7 +1059,7 @@ static int flush_commit_list(struct super_block *s, if (tbh) { if (buffer_dirty(tbh)) { depth = reiserfs_write_unlock_nested(s); - ll_rw_block(WRITE, 1, &tbh); + ll_rw_block(REQ_OP_WRITE, 0, 1, &tbh); reiserfs_write_lock_nested(s, depth); } put_bh(tbh) ; @@ -2246,7 +2246,7 @@ abort_replay: } } /* read in the log blocks, memcpy to the corresponding real block */ - ll_rw_block(READ, get_desc_trans_len(desc), log_blocks); + ll_rw_block(REQ_OP_READ, 0, get_desc_trans_len(desc), log_blocks); for (i = 0; i < get_desc_trans_len(desc); i++) { wait_on_buffer(log_blocks[i]); @@ -2348,7 +2348,7 @@ static struct buffer_head *reiserfs_breada(struct block_device *dev, } else bhlist[j++] = bh; } - ll_rw_block(READ, j, bhlist); + ll_rw_block(REQ_OP_READ, 0, j, bhlist); for (i = 1; i < j; i++) brelse(bhlist[i]); bh = bhlist[0]; diff --git a/fs/reiserfs/stree.c b/fs/reiserfs/stree.c index 24cbe01..5f9b6df 100644 --- a/fs/reiserfs/stree.c +++ b/fs/reiserfs/stree.c @@ -551,7 +551,7 @@ static int search_by_key_reada(struct super_block *s, if (!buffer_uptodate(bh[j])) { if (depth == -1) depth = reiserfs_write_unlock_nested(s); - ll_rw_block(READA, 1, bh + j); + ll_rw_block(REQ_OP_READ, READA, 1, bh + j); } brelse(bh[j]); } @@ -660,7 +660,7 @@ int search_by_key(struct super_block *sb, const struct cpu_key *key, if (!buffer_uptodate(bh) && depth == -1) depth = reiserfs_write_unlock_nested(sb); - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); if (depth != -1) diff --git a/fs/reiserfs/super.c b/fs/reiserfs/super.c index 05db747..bd04525 100644 --- a/fs/reiserfs/super.c +++ b/fs/reiserfs/super.c @@ -1660,7 +1660,7 @@ static int read_super_block(struct super_block *s, int offset) /* after journal replay, reread all bitmap and super blocks */ static int reread_meta_blocks(struct super_block *s) { - ll_rw_block(READ, 1, &SB_BUFFER_WITH_SB(s)); + ll_rw_block(REQ_OP_READ, 0, 1, &SB_BUFFER_WITH_SB(s)); wait_on_buffer(SB_BUFFER_WITH_SB(s)); if (!buffer_uptodate(SB_BUFFER_WITH_SB(s))) { reiserfs_warning(s, "reiserfs-2504", "error reading the super"); diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c index 0cea9b9..59375be 100644 --- a/fs/squashfs/block.c +++ b/fs/squashfs/block.c @@ -124,7 +124,7 @@ int squashfs_read_data(struct super_block *sb, u64 index, int length, goto block_release; bytes += msblk->devblksize; } - ll_rw_block(READ, b, bh); + ll_rw_block(REQ_OP_READ, 0, b, bh); } else { /* * Metadata block. @@ -156,7 +156,7 @@ int squashfs_read_data(struct super_block *sb, u64 index, int length, goto block_release; bytes += msblk->devblksize; } - ll_rw_block(READ, b - 1, bh + 1); + ll_rw_block(REQ_OP_READ, 0, b - 1, bh + 1); } for (i = 0; i < b; i++) { diff --git a/fs/udf/dir.c b/fs/udf/dir.c index 541d9c6..91b9f02 100644 --- a/fs/udf/dir.c +++ b/fs/udf/dir.c @@ -113,7 +113,7 @@ static int udf_readdir(struct file *file, struct dir_context *ctx) brelse(tmp); } if (num) { - ll_rw_block(READA, num, bha); + ll_rw_block(REQ_OP_READ, READA, num, bha); for (i = 0; i < num; i++) brelse(bha[i]); } diff --git a/fs/udf/directory.c b/fs/udf/directory.c index c763fda..71f3e0b 100644 --- a/fs/udf/directory.c +++ b/fs/udf/directory.c @@ -87,7 +87,7 @@ struct fileIdentDesc *udf_fileident_read(struct inode *dir, loff_t *nf_pos, brelse(tmp); } if (num) { - ll_rw_block(READA, num, bha); + ll_rw_block(REQ_OP_READ, READA, num, bha); for (i = 0; i < num; i++) brelse(bha[i]); } diff --git a/fs/udf/inode.c b/fs/udf/inode.c index 87dc16d..e49036b 100644 --- a/fs/udf/inode.c +++ b/fs/udf/inode.c @@ -1200,7 +1200,7 @@ struct buffer_head *udf_bread(struct inode *inode, int block, if (buffer_uptodate(bh)) return bh; - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); if (buffer_uptodate(bh)) diff --git a/fs/ufs/balloc.c b/fs/ufs/balloc.c index dc5fae6..26a3bb1 100644 --- a/fs/ufs/balloc.c +++ b/fs/ufs/balloc.c @@ -292,7 +292,7 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, if (!buffer_mapped(bh)) map_bh(bh, inode->i_sb, oldb + pos); if (!buffer_uptodate(bh)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) { ufs_error(inode->i_sb, __func__, diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index f7cc163..15af19f 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -187,7 +187,7 @@ struct buffer_head *alloc_buffer_head(gfp_t gfp_flags); void free_buffer_head(struct buffer_head * bh); void unlock_buffer(struct buffer_head *bh); void __lock_buffer(struct buffer_head *bh); -void ll_rw_block(int, int, struct buffer_head * bh[]); +void ll_rw_block(int, int, int, struct buffer_head * bh[]); int sync_dirty_buffer(struct buffer_head *bh); int __sync_dirty_buffer(struct buffer_head *bh, int op_flags); void write_dirty_buffer(struct buffer_head *bh, int op_flags); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:05 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id CBDFC29DFC for ; Tue, 5 Jan 2016 14:54:04 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 6406CAC001 for ; Tue, 5 Jan 2016 12:54:04 -0800 (PST) X-ASG-Debug-ID: 1452027243-04bdf03afb04dc0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id GQCoKs7NlzpNdlu0 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:03 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id D2A72263C; Tue, 5 Jan 2016 20:54:02 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu5024032; Tue, 5 Jan 2016 15:53:59 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 06/35] direct-io: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:09 -0600 X-ASG-Orig-Subj: [PATCH 06/35] direct-io: set bi_op to REQ_OP Message-Id: <1452027218-32303-7-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027243 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the dio code set the bio bi_op to a REQ_OP. It also begins to convert btrfs's dio_submit_t related code, because of the submit_io callout use. In the btrfs_submit_direct change, I OR'd the op and flag back together. It is only temporary. The next patch will completely convert all the btrfs code paths. Signed-off-by: Mike Christie --- fs/btrfs/inode.c | 9 +++++---- fs/direct-io.c | 35 +++++++++++++++++++++-------------- include/linux/fs.h | 2 +- 3 files changed, 27 insertions(+), 19 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 00f27eb..06f88bf 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8300,14 +8300,14 @@ out_err: return 0; } -static void btrfs_submit_direct(int rw, struct bio *dio_bio, - struct inode *inode, loff_t file_offset) +static void btrfs_submit_direct(struct bio *dio_bio, struct inode *inode, + loff_t file_offset) { struct btrfs_dio_private *dip = NULL; struct bio *io_bio = NULL; struct btrfs_io_bio *btrfs_bio; int skip_sum; - int write = rw & REQ_WRITE; + bool write = (dio_bio->bi_op == REQ_OP_WRITE); int ret = 0; skip_sum = BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM; @@ -8358,7 +8358,8 @@ static void btrfs_submit_direct(int rw, struct bio *dio_bio, dio_data->unsubmitted_oe_range_end; } - ret = btrfs_submit_direct_hook(rw, dip, skip_sum); + ret = btrfs_submit_direct_hook(dio_bio->bi_op | dio_bio->bi_rw, dip, + skip_sum); if (!ret) return; diff --git a/fs/direct-io.c b/fs/direct-io.c index 66b1d3eb..aa12742 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -108,7 +108,8 @@ struct dio_submit { /* dio_state communicated between submission path and end_io */ struct dio { int flags; /* doesn't change */ - int rw; + int op; + int op_flags; blk_qc_t bio_cookie; struct block_device *bio_bdev; struct inode *inode; @@ -163,7 +164,7 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) ret = iov_iter_get_pages(sdio->iter, dio->pages, LONG_MAX, DIO_PAGES, &sdio->from); - if (ret < 0 && sdio->blocks_available && (dio->rw & WRITE)) { + if (ret < 0 && sdio->blocks_available && (dio->op == REQ_OP_WRITE)) { struct page *page = ZERO_PAGE(0); /* * A memory fault, but the filesystem has some outstanding @@ -242,7 +243,8 @@ static ssize_t dio_complete(struct dio *dio, loff_t offset, ssize_t ret, transferred = dio->result; /* Check for short read case */ - if ((dio->rw == READ) && ((offset + transferred) > dio->i_size)) + if ((dio->op == REQ_OP_READ) && + ((offset + transferred) > dio->i_size)) transferred = dio->i_size - offset; } @@ -260,7 +262,7 @@ static ssize_t dio_complete(struct dio *dio, loff_t offset, ssize_t ret, inode_dio_end(dio->inode); if (is_async) { - if (dio->rw & WRITE) { + if (dio->op == REQ_OP_WRITE) { int err; err = generic_write_sync(dio->iocb->ki_filp, offset, @@ -369,7 +371,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_bdev = bdev; bio->bi_iter.bi_sector = first_sector; - bio->bi_rw |= dio->rw; + bio->bi_op = dio->op; + bio->bi_rw |= dio->op_flags; if (dio->is_async) bio->bi_end_io = dio_bio_end_aio; else @@ -397,14 +400,13 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) dio->refcount++; spin_unlock_irqrestore(&dio->bio_lock, flags); - if (dio->is_async && dio->rw == READ && dio->should_dirty) + if (dio->is_async && dio->op == REQ_OP_READ && dio->should_dirty) bio_set_pages_dirty(bio); dio->bio_bdev = bio->bi_bdev; if (sdio->submit_io) { - sdio->submit_io(dio->rw, bio, dio->inode, - sdio->logical_offset_in_bio); + sdio->submit_io(bio, dio->inode, sdio->logical_offset_in_bio); dio->bio_cookie = BLK_QC_T_NONE; } else dio->bio_cookie = submit_bio(bio); @@ -472,14 +474,14 @@ static int dio_bio_complete(struct dio *dio, struct bio *bio) if (bio->bi_error) dio->io_error = -EIO; - if (dio->is_async && dio->rw == READ && dio->should_dirty) { + if (dio->is_async && dio->op == REQ_OP_READ && dio->should_dirty) { bio_check_pages_dirty(bio); /* transfers ownership */ err = bio->bi_error; } else { bio_for_each_segment_all(bvec, bio, i) { struct page *page = bvec->bv_page; - if (dio->rw == READ && !PageCompound(page) && + if (dio->op == REQ_OP_READ && !PageCompound(page) && dio->should_dirty) set_page_dirty_lock(page); page_cache_release(page); @@ -632,7 +634,7 @@ static int get_more_blocks(struct dio *dio, struct dio_submit *sdio, * which may decide to handle it or also return an unmapped * buffer head. */ - create = dio->rw & WRITE; + create = dio->op == REQ_OP_WRITE; if (dio->flags & DIO_SKIP_HOLES) { if (sdio->block_in_file < (i_size_read(dio->inode) >> sdio->blkbits)) @@ -782,7 +784,7 @@ submit_page_section(struct dio *dio, struct dio_submit *sdio, struct page *page, { int ret = 0; - if (dio->rw & WRITE) { + if (dio->op == REQ_OP_WRITE) { /* * Read accounting is performed in submit_bio() */ @@ -982,7 +984,7 @@ do_holes: loff_t i_size_aligned; /* AKPM: eargh, -ENOTBLK is a hack */ - if (dio->rw & WRITE) { + if (dio->op == REQ_OP_WRITE) { page_cache_release(page); return -ENOTBLK; } @@ -1195,7 +1197,12 @@ do_blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, dio->is_async = true; dio->inode = inode; - dio->rw = iov_iter_rw(iter) == WRITE ? WRITE_ODIRECT : READ; + if (iov_iter_rw(iter) == WRITE) { + dio->op = REQ_OP_WRITE; + dio->op_flags = WRITE_ODIRECT; + } else { + dio->op = REQ_OP_READ; + } /* * For AIO O_(D)SYNC writes we need to defer completions to a workqueue diff --git a/include/linux/fs.h b/include/linux/fs.h index fb9e516..3ab5342 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2731,7 +2731,7 @@ extern int generic_file_open(struct inode * inode, struct file * filp); extern int nonseekable_open(struct inode * inode, struct file * filp); #ifdef CONFIG_BLOCK -typedef void (dio_submit_t)(int rw, struct bio *bio, struct inode *inode, +typedef void (dio_submit_t)(struct bio *bio, struct inode *inode, loff_t file_offset); enum { -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2726429E06 for ; Tue, 5 Jan 2016 14:54:08 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id CBCB4304039 for ; Tue, 5 Jan 2016 12:54:07 -0800 (PST) X-ASG-Debug-ID: 1452027246-04bdf03af904dc0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id IzxQnHqXsV9kdYyJ (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:06 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id D2250A203B; Tue, 5 Jan 2016 20:54:05 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu6024032; Tue, 5 Jan 2016 15:54:02 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 07/35] btrfs: have submit_one_bio users setup bio bi_op Date: Tue, 5 Jan 2016 14:53:10 -0600 X-ASG-Orig-Subj: [PATCH 07/35] btrfs: have submit_one_bio users setup bio bi_op Message-Id: <1452027218-32303-8-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027246 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has btrfs's submit_one_bio callers set the bio->bi_op to a REQ_OP and the bi_rw to rq_flag_bits. The next patches will continue to convert btrfs, so submit_bio_hook and merge_bio_hook related code will be modified to take only the bio. I did not do it in this patch to try and keep it smaller. Note: I have run xfs tests on these btrfs patches. There were some failures with and without the patches. I have not had time to track down why xfstest fails without the patches. Signed-off-by: Mike Christie --- fs/btrfs/extent_io.c | 92 +++++++++++++++++++++++++++------------------------- 1 file changed, 47 insertions(+), 45 deletions(-) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index 7bcc729..b6c281a 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2382,7 +2382,7 @@ static int bio_readpage_error(struct bio *failed_bio, u64 phy_offset, int read_mode; int ret; - BUG_ON(failed_bio->bi_rw & REQ_WRITE); + BUG_ON(failed_bio->bi_op == REQ_OP_WRITE); ret = btrfs_get_io_failure_record(inode, start, end, &failrec); if (ret) @@ -2408,6 +2408,8 @@ static int bio_readpage_error(struct bio *failed_bio, u64 phy_offset, free_io_failure(inode, failrec); return -EIO; } + bio->bi_op = REQ_OP_READ; + bio->bi_rw |= read_mode; pr_debug("Repair Read Error: submitting new read[%#x] to this_mirror=%d, in_validation=%d\n", read_mode, failrec->this_mirror, failrec->in_validation); @@ -2719,8 +2721,8 @@ struct bio *btrfs_io_bio_alloc(gfp_t gfp_mask, unsigned int nr_iovecs) } -static int __must_check submit_one_bio(int rw, struct bio *bio, - int mirror_num, unsigned long bio_flags) +static int __must_check submit_one_bio(struct bio *bio, int mirror_num, + unsigned long bio_flags) { int ret = 0; struct bio_vec *bvec = bio->bi_io_vec + bio->bi_vcnt - 1; @@ -2731,12 +2733,12 @@ static int __must_check submit_one_bio(int rw, struct bio *bio, start = page_offset(page) + bvec->bv_offset; bio->bi_private = NULL; - bio->bi_rw |= rw; bio_get(bio); if (tree->ops && tree->ops->submit_bio_hook) - ret = tree->ops->submit_bio_hook(page->mapping->host, rw, bio, - mirror_num, bio_flags, start); + ret = tree->ops->submit_bio_hook(page->mapping->host, + bio->bi_rw, bio, mirror_num, + bio_flags, start); else btrfsic_submit_bio(bio); @@ -2744,20 +2746,20 @@ static int __must_check submit_one_bio(int rw, struct bio *bio, return ret; } -static int merge_bio(int rw, struct extent_io_tree *tree, struct page *page, +static int merge_bio(struct extent_io_tree *tree, struct page *page, unsigned long offset, size_t size, struct bio *bio, unsigned long bio_flags) { int ret = 0; if (tree->ops && tree->ops->merge_bio_hook) - ret = tree->ops->merge_bio_hook(rw, page, offset, size, bio, - bio_flags); + ret = tree->ops->merge_bio_hook(bio->bi_op, page, offset, size, + bio, bio_flags); BUG_ON(ret < 0); return ret; } -static int submit_extent_page(int rw, struct extent_io_tree *tree, +static int submit_extent_page(int op, int op_flags, struct extent_io_tree *tree, struct writeback_control *wbc, struct page *page, sector_t sector, size_t size, unsigned long offset, @@ -2785,10 +2787,9 @@ static int submit_extent_page(int rw, struct extent_io_tree *tree, if (prev_bio_flags != bio_flags || !contig || force_bio_submit || - merge_bio(rw, tree, page, offset, page_size, bio, bio_flags) || + merge_bio(tree, page, offset, page_size, bio, bio_flags) || bio_add_page(bio, page, page_size, offset) < page_size) { - ret = submit_one_bio(rw, bio, mirror_num, - prev_bio_flags); + ret = submit_one_bio(bio, mirror_num, prev_bio_flags); if (ret < 0) { *bio_ret = NULL; return ret; @@ -2809,6 +2810,8 @@ static int submit_extent_page(int rw, struct extent_io_tree *tree, bio_add_page(bio, page, page_size, offset); bio->bi_end_io = end_io_func; bio->bi_private = tree; + bio->bi_op = op; + bio->bi_rw |= op_flags; if (wbc) { wbc_init_bio(wbc, bio); wbc_account_io(wbc, page, page_size); @@ -2817,7 +2820,7 @@ static int submit_extent_page(int rw, struct extent_io_tree *tree, if (bio_ret) *bio_ret = bio; else - ret = submit_one_bio(rw, bio, mirror_num, bio_flags); + ret = submit_one_bio(bio, mirror_num, bio_flags); return ret; } @@ -2881,7 +2884,7 @@ static int __do_readpage(struct extent_io_tree *tree, get_extent_t *get_extent, struct extent_map **em_cached, struct bio **bio, int mirror_num, - unsigned long *bio_flags, int rw, + unsigned long *bio_flags, int read_flags, u64 *prev_em_start) { struct inode *inode = page->mapping->host; @@ -3072,8 +3075,8 @@ static int __do_readpage(struct extent_io_tree *tree, } pnr -= page->index; - ret = submit_extent_page(rw, tree, NULL, page, - sector, disk_io_size, pg_offset, + ret = submit_extent_page(REQ_OP_READ, read_flags, tree, NULL, + page, sector, disk_io_size, pg_offset, bdev, bio, pnr, end_bio_extent_readpage, mirror_num, *bio_flags, @@ -3105,7 +3108,7 @@ static inline void __do_contiguous_readpages(struct extent_io_tree *tree, get_extent_t *get_extent, struct extent_map **em_cached, struct bio **bio, int mirror_num, - unsigned long *bio_flags, int rw, + unsigned long *bio_flags, u64 *prev_em_start) { struct inode *inode; @@ -3126,7 +3129,7 @@ static inline void __do_contiguous_readpages(struct extent_io_tree *tree, for (index = 0; index < nr_pages; index++) { __do_readpage(tree, pages[index], get_extent, em_cached, bio, - mirror_num, bio_flags, rw, prev_em_start); + mirror_num, bio_flags, 0, prev_em_start); page_cache_release(pages[index]); } } @@ -3136,7 +3139,7 @@ static void __extent_readpages(struct extent_io_tree *tree, int nr_pages, get_extent_t *get_extent, struct extent_map **em_cached, struct bio **bio, int mirror_num, - unsigned long *bio_flags, int rw, + unsigned long *bio_flags, u64 *prev_em_start) { u64 start = 0; @@ -3158,7 +3161,7 @@ static void __extent_readpages(struct extent_io_tree *tree, index - first_index, start, end, get_extent, em_cached, bio, mirror_num, bio_flags, - rw, prev_em_start); + prev_em_start); start = page_start; end = start + PAGE_CACHE_SIZE - 1; first_index = index; @@ -3169,7 +3172,7 @@ static void __extent_readpages(struct extent_io_tree *tree, __do_contiguous_readpages(tree, &pages[first_index], index - first_index, start, end, get_extent, em_cached, bio, - mirror_num, bio_flags, rw, + mirror_num, bio_flags, prev_em_start); } @@ -3177,7 +3180,7 @@ static int __extent_read_full_page(struct extent_io_tree *tree, struct page *page, get_extent_t *get_extent, struct bio **bio, int mirror_num, - unsigned long *bio_flags, int rw) + unsigned long *bio_flags, int read_flags) { struct inode *inode = page->mapping->host; struct btrfs_ordered_extent *ordered; @@ -3196,7 +3199,7 @@ static int __extent_read_full_page(struct extent_io_tree *tree, } ret = __do_readpage(tree, page, get_extent, NULL, bio, mirror_num, - bio_flags, rw, NULL); + bio_flags, read_flags, NULL); return ret; } @@ -3208,9 +3211,9 @@ int extent_read_full_page(struct extent_io_tree *tree, struct page *page, int ret; ret = __extent_read_full_page(tree, page, get_extent, &bio, mirror_num, - &bio_flags, READ); + &bio_flags, 0); if (bio) - ret = submit_one_bio(READ, bio, mirror_num, bio_flags); + ret = submit_one_bio(bio, mirror_num, bio_flags); return ret; } @@ -3222,9 +3225,9 @@ int extent_read_full_page_nolock(struct extent_io_tree *tree, struct page *page, int ret; ret = __do_readpage(tree, page, get_extent, NULL, &bio, mirror_num, - &bio_flags, READ, NULL); + &bio_flags, 0, NULL); if (bio) - ret = submit_one_bio(READ, bio, mirror_num, bio_flags); + ret = submit_one_bio(bio, mirror_num, bio_flags); return ret; } @@ -3471,7 +3474,8 @@ static noinline_for_stack int __extent_writepage_io(struct inode *inode, page->index, cur, end); } - ret = submit_extent_page(write_flags, tree, wbc, page, + ret = submit_extent_page(REQ_OP_WRITE, write_flags, + tree, wbc, page, sector, iosize, pg_offset, bdev, &epd->bio, max_nr, end_bio_extent_writepage, @@ -3511,13 +3515,11 @@ static int __extent_writepage(struct page *page, struct writeback_control *wbc, size_t pg_offset = 0; loff_t i_size = i_size_read(inode); unsigned long end_index = i_size >> PAGE_CACHE_SHIFT; - int write_flags; + int write_flags = 0; unsigned long nr_written = 0; if (wbc->sync_mode == WB_SYNC_ALL) write_flags = WRITE_SYNC; - else - write_flags = WRITE; trace___extent_writepage(page, inode, wbc); @@ -3761,7 +3763,7 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb, u64 offset = eb->start; unsigned long i, num_pages; unsigned long bio_flags = 0; - int rw = (epd->sync_io ? WRITE_SYNC : WRITE) | REQ_META; + int write_flags = (epd->sync_io ? WRITE_SYNC : 0) | REQ_META; int ret = 0; clear_bit(EXTENT_BUFFER_WRITE_ERR, &eb->bflags); @@ -3775,9 +3777,10 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb, clear_page_dirty_for_io(p); set_page_writeback(p); - ret = submit_extent_page(rw, tree, wbc, p, offset >> 9, - PAGE_CACHE_SIZE, 0, bdev, &epd->bio, - -1, end_bio_extent_buffer_writepage, + ret = submit_extent_page(REQ_OP_WRITE, write_flags, tree, wbc, + p, offset >> 9, PAGE_CACHE_SIZE, 0, + bdev, &epd->bio, -1, + end_bio_extent_buffer_writepage, 0, epd->bio_flags, bio_flags, false); epd->bio_flags = bio_flags; if (ret) { @@ -4066,13 +4069,13 @@ retry: static void flush_epd_write_bio(struct extent_page_data *epd) { if (epd->bio) { - int rw = WRITE; int ret; + epd->bio->bi_op = REQ_OP_WRITE; if (epd->sync_io) - rw = WRITE_SYNC; + epd->bio->bi_rw = WRITE_SYNC; - ret = submit_one_bio(rw, epd->bio, 0, epd->bio_flags); + ret = submit_one_bio(epd->bio, 0, epd->bio_flags); BUG_ON(ret < 0); /* -ENOMEM */ epd->bio = NULL; } @@ -4199,19 +4202,19 @@ int extent_readpages(struct extent_io_tree *tree, if (nr < ARRAY_SIZE(pagepool)) continue; __extent_readpages(tree, pagepool, nr, get_extent, &em_cached, - &bio, 0, &bio_flags, READ, &prev_em_start); + &bio, 0, &bio_flags, &prev_em_start); nr = 0; } if (nr) __extent_readpages(tree, pagepool, nr, get_extent, &em_cached, - &bio, 0, &bio_flags, READ, &prev_em_start); + &bio, 0, &bio_flags, &prev_em_start); if (em_cached) free_extent_map(em_cached); BUG_ON(!list_empty(pages)); if (bio) - return submit_one_bio(READ, bio, 0, bio_flags); + return submit_one_bio(bio, 0, bio_flags); return 0; } @@ -5233,7 +5236,7 @@ int read_extent_buffer_pages(struct extent_io_tree *tree, err = __extent_read_full_page(tree, page, get_extent, &bio, mirror_num, &bio_flags, - READ | REQ_META); + REQ_META); if (err) ret = err; } else { @@ -5242,8 +5245,7 @@ int read_extent_buffer_pages(struct extent_io_tree *tree, } if (bio) { - err = submit_one_bio(READ | REQ_META, bio, mirror_num, - bio_flags); + err = submit_one_bio(bio, mirror_num, bio_flags); if (err) return err; } -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9170F29E1C for ; Tue, 5 Jan 2016 14:54:10 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 574F1304066 for ; Tue, 5 Jan 2016 12:54:10 -0800 (PST) X-ASG-Debug-ID: 1452027248-04bdf03afa04dd0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id W1C8wj4VU82R7jKq (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:08 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 16A27461C4; Tue, 5 Jan 2016 20:54:08 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu7024032; Tue, 5 Jan 2016 15:54:05 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 08/35] btrfs: set bi_op tp REQ_OP Date: Tue, 5 Jan 2016 14:53:11 -0600 X-ASG-Orig-Subj: [PATCH 08/35] btrfs: set bi_op tp REQ_OP Message-Id: <1452027218-32303-9-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027248 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has btrfs set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. Signed-off-by: Mike Christie --- fs/btrfs/check-integrity.c | 19 +++++++++---------- fs/btrfs/compression.c | 4 ++++ fs/btrfs/disk-io.c | 7 ++++--- fs/btrfs/inode.c | 20 +++++++++++++------- fs/btrfs/raid56.c | 10 +++++----- fs/btrfs/scrub.c | 9 +++++---- fs/btrfs/volumes.c | 18 +++++++++--------- 7 files changed, 49 insertions(+), 38 deletions(-) diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index e3fd86b..e409d1f 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -1683,7 +1683,7 @@ static int btrfsic_read_block(struct btrfsic_state *state, } bio->bi_bdev = block_ctx->dev->bdev; bio->bi_iter.bi_sector = dev_bytenr >> 9; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; for (j = i; j < num_pages; j++) { ret = bio_add_page(bio, block_ctx->pagev[j], @@ -2964,7 +2964,6 @@ int btrfsic_submit_bh(int op, int op_flags, struct buffer_head *bh) static void __btrfsic_submit_bio(struct bio *bio) { struct btrfsic_dev_state *dev_state; - int rw = bio->bi_rw; if (!btrfsic_is_initialized) return; @@ -2974,7 +2973,7 @@ static void __btrfsic_submit_bio(struct bio *bio) * btrfsic_mount(), this might return NULL */ dev_state = btrfsic_dev_state_lookup(bio->bi_bdev); if (NULL != dev_state && - (rw & WRITE) && NULL != bio->bi_io_vec) { + (bio->bi_op == REQ_OP_WRITE) && NULL != bio->bi_io_vec) { unsigned int i; u64 dev_bytenr; u64 cur_bytenr; @@ -2986,9 +2985,9 @@ static void __btrfsic_submit_bio(struct bio *bio) if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bio(rw=0x%x, bi_vcnt=%u," + "submit_bio(rw=%d,0x%lx, bi_vcnt=%u," " bi_sector=%llu (bytenr %llu), bi_bdev=%p)\n", - rw, bio->bi_vcnt, + bio->bi_op, bio->bi_rw, bio->bi_vcnt, (unsigned long long)bio->bi_iter.bi_sector, dev_bytenr, bio->bi_bdev); @@ -3019,18 +3018,18 @@ static void __btrfsic_submit_bio(struct bio *bio) btrfsic_process_written_block(dev_state, dev_bytenr, mapped_datav, bio->bi_vcnt, bio, &bio_is_patched, - NULL, rw); + NULL, bio->bi_rw); while (i > 0) { i--; kunmap(bio->bi_io_vec[i].bv_page); } kfree(mapped_datav); - } else if (NULL != dev_state && (rw & REQ_FLUSH)) { + } else if (NULL != dev_state && (bio->bi_rw & REQ_FLUSH)) { if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bio(rw=0x%x FLUSH, bdev=%p)\n", - rw, bio->bi_bdev); + "submit_bio(rw=%d,0x%lx FLUSH, bdev=%p)\n", + bio->bi_op, bio->bi_rw, bio->bi_bdev); if (!dev_state->dummy_block_for_bio_bh_flush.is_iodone) { if ((dev_state->state->print_mask & (BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH | @@ -3048,7 +3047,7 @@ static void __btrfsic_submit_bio(struct bio *bio) block->never_written = 0; block->iodone_w_error = 0; block->flush_gen = dev_state->last_flush_gen + 1; - block->submit_bio_bh_rw = rw; + block->submit_bio_bh_rw = bio->bi_rw; block->orig_bio_bh_private = bio->bi_private; block->orig_bio_bh_end_io.bio = bio->bi_end_io; block->next_in_same_bio = NULL; diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index c473c42..25bf179 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -363,6 +363,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, kfree(cb); return -ENOMEM; } + bio->bi_op = REQ_OP_WRITE; bio->bi_private = cb; bio->bi_end_io = end_compressed_bio_write; atomic_inc(&cb->pending_bios); @@ -408,6 +409,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, bio = compressed_bio_alloc(bdev, first_byte, GFP_NOFS); BUG_ON(!bio); + bio->bi_op = REQ_OP_WRITE; bio->bi_private = cb; bio->bi_end_io = end_compressed_bio_write; bio_add_page(bio, page, PAGE_CACHE_SIZE, 0); @@ -650,6 +652,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, comp_bio = compressed_bio_alloc(bdev, cur_disk_byte, GFP_NOFS); if (!comp_bio) goto fail2; + comp_bio->bi_op = REQ_OP_READ; comp_bio->bi_private = cb; comp_bio->bi_end_io = end_compressed_bio_read; atomic_inc(&cb->pending_bios); @@ -703,6 +706,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, comp_bio = compressed_bio_alloc(bdev, cur_disk_byte, GFP_NOFS); BUG_ON(!comp_bio); + comp_bio->bi_op = REQ_OP_READ; comp_bio->bi_private = cb; comp_bio->bi_end_io = end_compressed_bio_read; diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index a77d88e..cd152e2 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -713,7 +713,7 @@ static void end_workqueue_bio(struct bio *bio) fs_info = end_io_wq->info; end_io_wq->error = bio->bi_error; - if (bio->bi_rw & REQ_WRITE) { + if (bio->bi_op == REQ_OP_WRITE) { if (end_io_wq->metadata == BTRFS_WQ_ENDIO_METADATA) { wq = fs_info->endio_meta_write_workers; func = btrfs_endio_meta_write_helper; @@ -859,7 +859,7 @@ int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode, atomic_inc(&fs_info->nr_async_submits); - if (rw & REQ_SYNC) + if (bio->bi_rw & REQ_SYNC) btrfs_set_work_high_priority(&async->work); btrfs_queue_work(fs_info->workers, &async->work); @@ -937,7 +937,7 @@ static int btree_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, int async = check_async_write(inode, bio_flags); int ret; - if (!(rw & REQ_WRITE)) { + if (bio->bi_op != REQ_OP_WRITE) { /* * called for a read, do the setup so that checksum validation * can happen in the async kernel threads @@ -3454,6 +3454,7 @@ static int write_dev_flush(struct btrfs_device *device, int wait) bio->bi_end_io = btrfs_end_empty_barrier; bio->bi_bdev = device->bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw |= WRITE_FLUSH; init_completion(&device->flush_wait); bio->bi_private = &device->flush_wait; diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 06f88bf..7a830c7 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -1895,7 +1895,7 @@ static int btrfs_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, if (btrfs_is_free_space_inode(inode)) metadata = BTRFS_WQ_ENDIO_FREE_SPACE; - if (!(rw & REQ_WRITE)) { + if (bio->bi_op != REQ_OP_WRITE) { ret = btrfs_bio_wq_end_io(root->fs_info, bio, metadata); if (ret) goto out; @@ -7713,7 +7713,7 @@ static inline int submit_dio_repair_bio(struct inode *inode, struct bio *bio, struct btrfs_root *root = BTRFS_I(inode)->root; int ret; - BUG_ON(rw & REQ_WRITE); + BUG_ON(bio->bi_op == REQ_OP_WRITE); bio_get(bio); @@ -7773,7 +7773,7 @@ static int dio_read_error(struct inode *inode, struct bio *failed_bio, int read_mode; int ret; - BUG_ON(failed_bio->bi_rw & REQ_WRITE); + BUG_ON(failed_bio->bi_op == REQ_OP_WRITE); ret = btrfs_get_io_failure_record(inode, start, end, &failrec); if (ret) @@ -7799,6 +7799,8 @@ static int dio_read_error(struct inode *inode, struct bio *failed_bio, free_io_failure(inode, failrec); return -EIO; } + bio->bi_op = REQ_OP_READ; + bio->bi_rw |= read_mode; btrfs_debug(BTRFS_I(inode)->root->fs_info, "Repair DIO Read Error: submitting new dio read[%#x] to this_mirror=%d, in_validation=%d\n", @@ -8061,8 +8063,8 @@ static void btrfs_end_dio_bio(struct bio *bio) if (err) btrfs_warn(BTRFS_I(dip->inode)->root->fs_info, - "direct IO failed ino %llu rw %lu sector %#Lx len %u err no %d", - btrfs_ino(dip->inode), bio->bi_rw, + "direct IO failed ino %llu rw %d,%lu sector %#Lx len %u err no %d", + btrfs_ino(dip->inode), bio->bi_op, bio->bi_rw, (unsigned long long)bio->bi_iter.bi_sector, bio->bi_iter.bi_size, err); @@ -8140,7 +8142,7 @@ static inline int __btrfs_submit_dio_bio(struct bio *bio, struct inode *inode, int async_submit) { struct btrfs_dio_private *dip = bio->bi_private; - int write = rw & REQ_WRITE; + bool write = bio->bi_op == REQ_OP_WRITE; struct btrfs_root *root = BTRFS_I(inode)->root; int ret; @@ -8225,6 +8227,8 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, if (!bio) return -ENOMEM; + bio->bi_op = orig_bio->bi_op; + bio->bi_rw = orig_bio->bi_rw; bio->bi_private = dip; bio->bi_end_io = btrfs_end_dio_bio; btrfs_io_bio(bio)->logical = file_offset; @@ -8260,12 +8264,14 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, start_sector, GFP_NOFS); if (!bio) goto out_err; + bio->bi_op = orig_bio->bi_op; + bio->bi_rw = orig_bio->bi_rw; bio->bi_private = dip; bio->bi_end_io = btrfs_end_dio_bio; btrfs_io_bio(bio)->logical = file_offset; map_length = orig_bio->bi_iter.bi_size; - ret = btrfs_map_block(root->fs_info, rw, + ret = btrfs_map_block(root->fs_info, orig_bio->bi_op, start_sector << 9, &map_length, NULL, 0); if (ret) { diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c index 532ceb0..54e4941 100644 --- a/fs/btrfs/raid56.c +++ b/fs/btrfs/raid56.c @@ -1323,7 +1323,7 @@ write_data: bio->bi_private = rbio; bio->bi_end_io = raid_write_end_io; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; submit_bio(bio); } @@ -1579,7 +1579,7 @@ static int raid56_rmw_stripe(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid_rmw_end_io; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); @@ -2106,7 +2106,7 @@ static int __raid56_parity_recover(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid_recover_end_io; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); @@ -2469,7 +2469,7 @@ submit_write: bio->bi_private = rbio; bio->bi_end_io = raid_write_parity_end_io; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; submit_bio(bio); } return; @@ -2647,7 +2647,7 @@ static void raid56_parity_scrub_stripe(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid56_parity_scrub_end_io; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c index 944959f..cc8f29c 100644 --- a/fs/btrfs/scrub.c +++ b/fs/btrfs/scrub.c @@ -1504,7 +1504,7 @@ static void scrub_recheck_block(struct btrfs_fs_info *fs_info, sblock->no_io_error_seen = 0; } else { bio->bi_iter.bi_sector = page->physical >> 9; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; if (btrfsic_submit_bio_wait(bio)) sblock->no_io_error_seen = 0; @@ -1586,7 +1586,7 @@ static int scrub_repair_page_from_good_copy(struct scrub_block *sblock_bad, return -EIO; bio->bi_bdev = page_bad->dev->bdev; bio->bi_iter.bi_sector = page_bad->physical >> 9; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; ret = bio_add_page(bio, page_good->page, PAGE_SIZE, 0); if (PAGE_SIZE != ret) { @@ -1687,7 +1687,7 @@ again: bio->bi_end_io = scrub_wr_bio_end_io; bio->bi_bdev = sbio->dev->bdev; bio->bi_iter.bi_sector = sbio->physical >> 9; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; sbio->err = 0; } else if (sbio->physical + sbio->page_count * PAGE_SIZE != spage->physical_for_dev_replace || @@ -2091,7 +2091,7 @@ again: bio->bi_end_io = scrub_bio_end_io; bio->bi_bdev = sbio->dev->bdev; bio->bi_iter.bi_sector = sbio->physical >> 9; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; sbio->err = 0; } else if (sbio->physical + sbio->page_count * PAGE_SIZE != spage->physical || @@ -4394,6 +4394,7 @@ static int write_page_nocow(struct scrub_ctx *sctx, bio->bi_iter.bi_size = 0; bio->bi_iter.bi_sector = physical_for_dev_replace >> 9; bio->bi_bdev = dev->bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw |= WRITE_SYNC; ret = bio_add_page(bio, page, PAGE_CACHE_SIZE, 0); diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 2463761..0da1d32 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -5888,7 +5888,7 @@ static void btrfs_end_bio(struct bio *bio) BUG_ON(stripe_index >= bbio->num_stripes); dev = bbio->stripes[stripe_index].dev; if (dev->bdev) { - if (bio->bi_rw & WRITE) + if (bio->bi_op == REQ_OP_WRITE) btrfs_dev_stat_inc(dev, BTRFS_DEV_STAT_WRITE_ERRS); else @@ -5953,7 +5953,7 @@ static noinline void btrfs_schedule_bio(struct btrfs_root *root, } /* don't bother with additional async steps for reads, right now */ - if (!(bio->bi_rw & REQ_WRITE)) { + if (bio->bi_op == REQ_OP_READ) { bio_get(bio); btrfsic_submit_bio(bio); bio_put(bio); @@ -6002,15 +6002,14 @@ static void submit_stripe_bio(struct btrfs_root *root, struct btrfs_bio *bbio, btrfs_io_bio(bio)->stripe_index = dev_nr; bio->bi_end_io = btrfs_end_bio; bio->bi_iter.bi_sector = physical >> 9; - bio->bi_rw |= rw; #ifdef DEBUG { struct rcu_string *name; rcu_read_lock(); name = rcu_dereference(dev->name); - pr_debug("btrfs_map_bio: rw %d, sector=%llu, dev=%lu " - "(%s id %llu), size=%u\n", rw, + pr_debug("btrfs_map_bio: rw %d 0x%x, sector=%llu, dev=%lu " + "(%s id %llu), size=%u\n", bio->bi_op, bio->bi_rw, (u64)bio->bi_iter.bi_sector, (u_long)dev->bdev->bd_dev, name->str, dev->devid, bio->bi_iter.bi_size); rcu_read_unlock(); @@ -6057,8 +6056,8 @@ int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, map_length = length; btrfs_bio_counter_inc_blocked(root->fs_info); - ret = __btrfs_map_block(root->fs_info, rw, logical, &map_length, &bbio, - mirror_num, 1); + ret = __btrfs_map_block(root->fs_info, bio->bi_op, logical, + &map_length, &bbio, mirror_num, 1); if (ret) { btrfs_bio_counter_dec(root->fs_info); return ret; @@ -6074,7 +6073,7 @@ int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, if (bbio->raid_map) { /* In this case, map_length has been set to the length of a single stripe; not the whole write */ - if (rw & WRITE) { + if (bio->bi_op == REQ_OP_WRITE) { ret = raid56_parity_write(root, bio, bbio, map_length); } else { ret = raid56_parity_recover(root, bio, bbio, map_length, @@ -6093,7 +6092,8 @@ int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, for (dev_nr = 0; dev_nr < total_devs; dev_nr++) { dev = bbio->stripes[dev_nr].dev; - if (!dev || !dev->bdev || (rw & WRITE && !dev->writeable)) { + if (!dev || !dev->bdev || + (bio->bi_op == REQ_OP_WRITE && !dev->writeable)) { bbio_error(bbio, first_bio, logical); continue; } -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:12 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id E2ECD29E14 for ; Tue, 5 Jan 2016 14:54:12 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id D4CAA8F804C for ; Tue, 5 Jan 2016 12:54:12 -0800 (PST) X-ASG-Debug-ID: 1452027251-04cb6c42e705230001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id naO0RGZRi4zMj1KN (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:11 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id D36B136209A; Tue, 5 Jan 2016 20:54:10 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu8024032; Tue, 5 Jan 2016 15:54:08 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 09/35] btrfs: update __btrfs_map_block for bi_op transition Date: Tue, 5 Jan 2016 14:53:12 -0600 X-ASG-Orig-Subj: [PATCH 09/35] btrfs: update __btrfs_map_block for bi_op transition Message-Id: <1452027218-32303-10-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027251 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie We no longer pass in a bitmap of rq_flag_bits bits to __btrfs_map_block. It will always be a REQ_OP, or the btrfs specific REQ_GET_READ_MIRRORS, so this drops the bit tests. Signed-off-by: Mike Christie --- fs/btrfs/extent-tree.c | 2 +- fs/btrfs/inode.c | 2 +- fs/btrfs/volumes.c | 55 +++++++++++++++++++++++++++----------------------- fs/btrfs/volumes.h | 4 ++-- 4 files changed, 34 insertions(+), 29 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index add4af6..4d503d0 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2043,7 +2043,7 @@ int btrfs_discard_extent(struct btrfs_root *root, u64 bytenr, /* Tell the block device(s) that the sectors can be discarded */ - ret = btrfs_map_block(root->fs_info, REQ_DISCARD, + ret = btrfs_map_block(root->fs_info, REQ_OP_DISCARD, bytenr, &num_bytes, &bbio, 0); /* Error condition is -ENOMEM */ if (!ret) { diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 7a830c7..b1e88ec 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8206,7 +8206,7 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, int async_submit = 0; map_length = orig_bio->bi_iter.bi_size; - ret = btrfs_map_block(root->fs_info, rw, start_sector << 9, + ret = btrfs_map_block(root->fs_info, orig_bio->bi_op, start_sector << 9, &map_length, NULL, 0); if (ret) return -EIO; diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 0da1d32..bf1e9af 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -5218,7 +5218,7 @@ void btrfs_put_bbio(struct btrfs_bio *bbio) kfree(bbio); } -static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, +static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num, int need_raid_map) @@ -5296,7 +5296,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, raid56_full_stripe_start *= full_stripe_len; } - if (rw & REQ_DISCARD) { + if (op == REQ_OP_DISCARD) { /* we don't discard raid56 yet */ if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) { ret = -EOPNOTSUPP; @@ -5309,7 +5309,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, For other RAID types and for RAID[56] reads, just allow a single stripe (on a single disk). */ if ((map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) && - (rw & REQ_WRITE)) { + (op == REQ_OP_WRITE)) { max_len = stripe_len * nr_data_stripes(map) - (offset - raid56_full_stripe_start); } else { @@ -5332,8 +5332,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, btrfs_dev_replace_unlock(dev_replace); if (dev_replace_is_ongoing && mirror_num == map->num_stripes + 1 && - !(rw & (REQ_WRITE | REQ_DISCARD | REQ_GET_READ_MIRRORS)) && - dev_replace->tgtdev != NULL) { + op != REQ_OP_WRITE && op != REQ_OP_DISCARD && + op != REQ_GET_READ_MIRRORS && dev_replace->tgtdev != NULL) { /* * in dev-replace case, for repair case (that's the only * case where the mirror is selected explicitly when @@ -5422,15 +5422,17 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, (offset + *length); if (map->type & BTRFS_BLOCK_GROUP_RAID0) { - if (rw & REQ_DISCARD) + if (op == REQ_OP_DISCARD) num_stripes = min_t(u64, map->num_stripes, stripe_nr_end - stripe_nr_orig); stripe_nr = div_u64_rem(stripe_nr, map->num_stripes, &stripe_index); - if (!(rw & (REQ_WRITE | REQ_DISCARD | REQ_GET_READ_MIRRORS))) + if (op != REQ_OP_WRITE && op != REQ_OP_DISCARD && + op != REQ_GET_READ_MIRRORS) mirror_num = 1; } else if (map->type & BTRFS_BLOCK_GROUP_RAID1) { - if (rw & (REQ_WRITE | REQ_DISCARD | REQ_GET_READ_MIRRORS)) + if (op == REQ_OP_WRITE || op == REQ_OP_DISCARD || + op == REQ_GET_READ_MIRRORS) num_stripes = map->num_stripes; else if (mirror_num) stripe_index = mirror_num - 1; @@ -5443,7 +5445,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, } } else if (map->type & BTRFS_BLOCK_GROUP_DUP) { - if (rw & (REQ_WRITE | REQ_DISCARD | REQ_GET_READ_MIRRORS)) { + if (op == REQ_OP_WRITE || REQ_OP_DISCARD || + op == REQ_GET_READ_MIRRORS) { num_stripes = map->num_stripes; } else if (mirror_num) { stripe_index = mirror_num - 1; @@ -5457,9 +5460,9 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, stripe_nr = div_u64_rem(stripe_nr, factor, &stripe_index); stripe_index *= map->sub_stripes; - if (rw & (REQ_WRITE | REQ_GET_READ_MIRRORS)) + if (op == REQ_OP_WRITE || op == REQ_GET_READ_MIRRORS) num_stripes = map->sub_stripes; - else if (rw & REQ_DISCARD) + else if (op == REQ_OP_DISCARD) num_stripes = min_t(u64, map->sub_stripes * (stripe_nr_end - stripe_nr_orig), map->num_stripes); @@ -5477,7 +5480,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, } else if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) { if (need_raid_map && - ((rw & (REQ_WRITE | REQ_GET_READ_MIRRORS)) || + (op == REQ_OP_WRITE || op == REQ_GET_READ_MIRRORS || mirror_num > 1)) { /* push stripe_nr back to the start of the full stripe */ stripe_nr = div_u64(raid56_full_stripe_start, @@ -5505,8 +5508,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, /* We distribute the parity blocks across stripes */ div_u64_rem(stripe_nr + stripe_index, map->num_stripes, &stripe_index); - if (!(rw & (REQ_WRITE | REQ_DISCARD | - REQ_GET_READ_MIRRORS)) && mirror_num <= 1) + if ((op != REQ_OP_WRITE && op != REQ_OP_DISCARD && + op != REQ_GET_READ_MIRRORS) && mirror_num <= 1) mirror_num = 1; } } else { @@ -5523,9 +5526,9 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, num_alloc_stripes = num_stripes; if (dev_replace_is_ongoing) { - if (rw & (REQ_WRITE | REQ_DISCARD)) + if (op == REQ_OP_WRITE || op == REQ_OP_DISCARD) num_alloc_stripes <<= 1; - if (rw & REQ_GET_READ_MIRRORS) + if (op == REQ_GET_READ_MIRRORS) num_alloc_stripes++; tgtdev_indexes = num_stripes; } @@ -5540,7 +5543,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, /* build raid_map */ if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK && - need_raid_map && ((rw & (REQ_WRITE | REQ_GET_READ_MIRRORS)) || + need_raid_map && + ((op == REQ_OP_WRITE || op == REQ_GET_READ_MIRRORS) || mirror_num > 1)) { u64 tmp; unsigned rot; @@ -5565,7 +5569,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, RAID6_Q_STRIPE; } - if (rw & REQ_DISCARD) { + if (op == REQ_OP_DISCARD) { u32 factor = 0; u32 sub_stripes = 0; u64 stripes_per_dev = 0; @@ -5645,14 +5649,15 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, } } - if (rw & (REQ_WRITE | REQ_GET_READ_MIRRORS)) + if (op == REQ_OP_WRITE || op == REQ_GET_READ_MIRRORS) max_errors = btrfs_chunk_max_errors(map); if (bbio->raid_map) sort_parity_stripes(bbio, num_stripes); tgtdev_indexes = 0; - if (dev_replace_is_ongoing && (rw & (REQ_WRITE | REQ_DISCARD)) && + if (dev_replace_is_ongoing && + (op == REQ_OP_WRITE || op == REQ_OP_DISCARD) && dev_replace->tgtdev != NULL) { int index_where_to_add; u64 srcdev_devid = dev_replace->srcdev->devid; @@ -5687,7 +5692,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, } } num_stripes = index_where_to_add; - } else if (dev_replace_is_ongoing && (rw & REQ_GET_READ_MIRRORS) && + } else if (dev_replace_is_ongoing && (op == REQ_GET_READ_MIRRORS) && dev_replace->tgtdev != NULL) { u64 srcdev_devid = dev_replace->srcdev->devid; int index_srcdev = 0; @@ -5760,21 +5765,21 @@ out: return ret; } -int btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, +int btrfs_map_block(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num) { - return __btrfs_map_block(fs_info, rw, logical, length, bbio_ret, + return __btrfs_map_block(fs_info, op, logical, length, bbio_ret, mirror_num, 0); } /* For Scrub/replace */ -int btrfs_map_sblock(struct btrfs_fs_info *fs_info, int rw, +int btrfs_map_sblock(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num, int need_raid_map) { - return __btrfs_map_block(fs_info, rw, logical, length, bbio_ret, + return __btrfs_map_block(fs_info, op, logical, length, bbio_ret, mirror_num, need_raid_map); } diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h index 6a4375a..8ef796a 100644 --- a/fs/btrfs/volumes.h +++ b/fs/btrfs/volumes.h @@ -421,10 +421,10 @@ int btrfs_account_dev_extents_size(struct btrfs_device *device, u64 start, u64 end, u64 *length); void btrfs_get_bbio(struct btrfs_bio *bbio); void btrfs_put_bbio(struct btrfs_bio *bbio); -int btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, +int btrfs_map_block(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num); -int btrfs_map_sblock(struct btrfs_fs_info *fs_info, int rw, +int btrfs_map_sblock(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num, int need_raid_map); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 426E229E0C for ; Tue, 5 Jan 2016 14:54:16 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 25EB18F8050 for ; Tue, 5 Jan 2016 12:54:16 -0800 (PST) X-ASG-Debug-ID: 1452027253-04cb6c42e805240001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 2txJtNXVvTWQNVFv (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:14 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 920A0A2C16; Tue, 5 Jan 2016 20:54:13 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05Krfu9024032; Tue, 5 Jan 2016 15:54:10 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 10/35] btrfs: don't pass rq_flag_bits if there is a bio Date: Tue, 5 Jan 2016 14:53:13 -0600 X-ASG-Orig-Subj: [PATCH 10/35] btrfs: don't pass rq_flag_bits if there is a bio Message-Id: <1452027218-32303-11-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027254 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The bio bi_op and bi_rw is now setup, so there is no need to pass around the rq_flag_bits bits too. Signed-off-by: Mike Christie --- fs/btrfs/compression.c | 9 ++++----- fs/btrfs/disk-io.c | 30 ++++++++++++------------------ fs/btrfs/disk-io.h | 2 +- fs/btrfs/extent_io.c | 16 +++++++--------- fs/btrfs/extent_io.h | 6 +++--- fs/btrfs/inode.c | 40 ++++++++++++++++++---------------------- fs/btrfs/volumes.c | 6 +++--- fs/btrfs/volumes.h | 2 +- 8 files changed, 49 insertions(+), 62 deletions(-) diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index 25bf179..3112cc3 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -402,7 +402,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, BUG_ON(ret); /* -ENOMEM */ } - ret = btrfs_map_bio(root, WRITE, bio, 0, 1); + ret = btrfs_map_bio(root, bio, 0, 1); BUG_ON(ret); /* -ENOMEM */ bio_put(bio); @@ -433,7 +433,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, BUG_ON(ret); /* -ENOMEM */ } - ret = btrfs_map_bio(root, WRITE, bio, 0, 1); + ret = btrfs_map_bio(root, bio, 0, 1); BUG_ON(ret); /* -ENOMEM */ bio_put(bio); @@ -694,8 +694,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, sums += DIV_ROUND_UP(comp_bio->bi_iter.bi_size, root->sectorsize); - ret = btrfs_map_bio(root, READ, comp_bio, - mirror_num, 0); + ret = btrfs_map_bio(root, comp_bio, mirror_num, 0); if (ret) { bio->bi_error = ret; bio_endio(comp_bio); @@ -725,7 +724,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, BUG_ON(ret); /* -ENOMEM */ } - ret = btrfs_map_bio(root, READ, comp_bio, mirror_num, 0); + ret = btrfs_map_bio(root, comp_bio, mirror_num, 0); if (ret) { bio->bi_error = ret; bio_endio(comp_bio); diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index cd152e2..d344231 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -119,7 +119,6 @@ struct async_submit_bio { struct list_head list; extent_submit_bio_hook_t *submit_bio_start; extent_submit_bio_hook_t *submit_bio_done; - int rw; int mirror_num; unsigned long bio_flags; /* @@ -783,7 +782,7 @@ static void run_one_async_start(struct btrfs_work *work) int ret; async = container_of(work, struct async_submit_bio, work); - ret = async->submit_bio_start(async->inode, async->rw, async->bio, + ret = async->submit_bio_start(async->inode, async->bio, async->mirror_num, async->bio_flags, async->bio_offset); if (ret) @@ -816,9 +815,8 @@ static void run_one_async_done(struct btrfs_work *work) return; } - async->submit_bio_done(async->inode, async->rw, async->bio, - async->mirror_num, async->bio_flags, - async->bio_offset); + async->submit_bio_done(async->inode, async->bio, async->mirror_num, + async->bio_flags, async->bio_offset); } static void run_one_async_free(struct btrfs_work *work) @@ -830,7 +828,7 @@ static void run_one_async_free(struct btrfs_work *work) } int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode, - int rw, struct bio *bio, int mirror_num, + struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset, extent_submit_bio_hook_t *submit_bio_start, @@ -843,7 +841,6 @@ int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode, return -ENOMEM; async->inode = inode; - async->rw = rw; async->bio = bio; async->mirror_num = mirror_num; async->submit_bio_start = submit_bio_start; @@ -889,9 +886,8 @@ static int btree_csum_one_bio(struct bio *bio) return ret; } -static int __btree_submit_bio_start(struct inode *inode, int rw, - struct bio *bio, int mirror_num, - unsigned long bio_flags, +static int __btree_submit_bio_start(struct inode *inode, struct bio *bio, + int mirror_num, unsigned long bio_flags, u64 bio_offset) { /* @@ -901,7 +897,7 @@ static int __btree_submit_bio_start(struct inode *inode, int rw, return btree_csum_one_bio(bio); } -static int __btree_submit_bio_done(struct inode *inode, int rw, struct bio *bio, +static int __btree_submit_bio_done(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset) { @@ -911,7 +907,7 @@ static int __btree_submit_bio_done(struct inode *inode, int rw, struct bio *bio, * when we're called for a write, we're already in the async * submission context. Just jump into btrfs_map_bio */ - ret = btrfs_map_bio(BTRFS_I(inode)->root, rw, bio, mirror_num, 1); + ret = btrfs_map_bio(BTRFS_I(inode)->root, bio, mirror_num, 1); if (ret) { bio->bi_error = ret; bio_endio(bio); @@ -930,7 +926,7 @@ static int check_async_write(struct inode *inode, unsigned long bio_flags) return 1; } -static int btree_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, +static int btree_submit_bio_hook(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset) { @@ -946,21 +942,19 @@ static int btree_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, bio, BTRFS_WQ_ENDIO_METADATA); if (ret) goto out_w_error; - ret = btrfs_map_bio(BTRFS_I(inode)->root, rw, bio, - mirror_num, 0); + ret = btrfs_map_bio(BTRFS_I(inode)->root, bio, mirror_num, 0); } else if (!async) { ret = btree_csum_one_bio(bio); if (ret) goto out_w_error; - ret = btrfs_map_bio(BTRFS_I(inode)->root, rw, bio, - mirror_num, 0); + ret = btrfs_map_bio(BTRFS_I(inode)->root, bio, mirror_num, 0); } else { /* * kthread helpers are used to submit writes so that * checksumming can happen in parallel across all CPUs */ ret = btrfs_wq_submit_bio(BTRFS_I(inode)->root->fs_info, - inode, rw, bio, mirror_num, 0, + inode, bio, mirror_num, 0, bio_offset, __btree_submit_bio_start, __btree_submit_bio_done); diff --git a/fs/btrfs/disk-io.h b/fs/btrfs/disk-io.h index 7c52e29..4a5d5df 100644 --- a/fs/btrfs/disk-io.h +++ b/fs/btrfs/disk-io.h @@ -122,7 +122,7 @@ void btrfs_csum_final(u32 crc, char *result); int btrfs_bio_wq_end_io(struct btrfs_fs_info *info, struct bio *bio, enum btrfs_wq_endio_type metadata); int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode, - int rw, struct bio *bio, int mirror_num, + struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset, extent_submit_bio_hook_t *submit_bio_start, extent_submit_bio_hook_t *submit_bio_done); diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index b6c281a..25a6da0 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2414,8 +2414,7 @@ static int bio_readpage_error(struct bio *failed_bio, u64 phy_offset, pr_debug("Repair Read Error: submitting new read[%#x] to this_mirror=%d, in_validation=%d\n", read_mode, failrec->this_mirror, failrec->in_validation); - ret = tree->ops->submit_bio_hook(inode, read_mode, bio, - failrec->this_mirror, + ret = tree->ops->submit_bio_hook(inode, bio, failrec->this_mirror, failrec->bio_flags, 0); if (ret) { free_io_failure(inode, failrec); @@ -2736,9 +2735,8 @@ static int __must_check submit_one_bio(struct bio *bio, int mirror_num, bio_get(bio); if (tree->ops && tree->ops->submit_bio_hook) - ret = tree->ops->submit_bio_hook(page->mapping->host, - bio->bi_rw, bio, mirror_num, - bio_flags, start); + ret = tree->ops->submit_bio_hook(page->mapping->host, bio, + mirror_num, bio_flags, start); else btrfsic_submit_bio(bio); @@ -2746,14 +2744,14 @@ static int __must_check submit_one_bio(struct bio *bio, int mirror_num, return ret; } -static int merge_bio(struct extent_io_tree *tree, struct page *page, +static int merge_bio(int rw, struct extent_io_tree *tree, struct page *page, unsigned long offset, size_t size, struct bio *bio, unsigned long bio_flags) { int ret = 0; if (tree->ops && tree->ops->merge_bio_hook) - ret = tree->ops->merge_bio_hook(bio->bi_op, page, offset, size, - bio, bio_flags); + ret = tree->ops->merge_bio_hook(rw, page, offset, size, bio, + bio_flags); BUG_ON(ret < 0); return ret; @@ -2787,7 +2785,7 @@ static int submit_extent_page(int op, int op_flags, struct extent_io_tree *tree, if (prev_bio_flags != bio_flags || !contig || force_bio_submit || - merge_bio(tree, page, offset, page_size, bio, bio_flags) || + merge_bio(op, tree, page, offset, page_size, bio, bio_flags) || bio_add_page(bio, page, page_size, offset) < page_size) { ret = submit_one_bio(bio, mirror_num, prev_bio_flags); if (ret < 0) { diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h index 0377413..89f1b75 100644 --- a/fs/btrfs/extent_io.h +++ b/fs/btrfs/extent_io.h @@ -63,9 +63,9 @@ struct extent_state; struct btrfs_root; struct btrfs_io_bio; -typedef int (extent_submit_bio_hook_t)(struct inode *inode, int rw, - struct bio *bio, int mirror_num, - unsigned long bio_flags, u64 bio_offset); +typedef int (extent_submit_bio_hook_t)(struct inode *inode, struct bio *bio, + int mirror_num, unsigned long bio_flags, + u64 bio_offset); struct extent_io_ops { int (*fill_delalloc)(struct inode *inode, struct page *locked_page, u64 start, u64 end, int *page_started, diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index b1e88ec..a96984f 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -1840,9 +1840,8 @@ int btrfs_merge_bio_hook(int rw, struct page *page, unsigned long offset, * At IO completion time the cums attached on the ordered extent record * are inserted into the btree */ -static int __btrfs_submit_bio_start(struct inode *inode, int rw, - struct bio *bio, int mirror_num, - unsigned long bio_flags, +static int __btrfs_submit_bio_start(struct inode *inode, struct bio *bio, + int mirror_num, unsigned long bio_flags, u64 bio_offset) { struct btrfs_root *root = BTRFS_I(inode)->root; @@ -1861,14 +1860,14 @@ static int __btrfs_submit_bio_start(struct inode *inode, int rw, * At IO completion time the cums attached on the ordered extent record * are inserted into the btree */ -static int __btrfs_submit_bio_done(struct inode *inode, int rw, struct bio *bio, +static int __btrfs_submit_bio_done(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset) { struct btrfs_root *root = BTRFS_I(inode)->root; int ret; - ret = btrfs_map_bio(root, rw, bio, mirror_num, 1); + ret = btrfs_map_bio(root, bio, mirror_num, 1); if (ret) { bio->bi_error = ret; bio_endio(bio); @@ -1880,7 +1879,7 @@ static int __btrfs_submit_bio_done(struct inode *inode, int rw, struct bio *bio, * extent_io.c submission hook. This does the right thing for csum calculation * on write, or reading the csums from the tree before a read */ -static int btrfs_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, +static int btrfs_submit_bio_hook(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset) { @@ -1917,7 +1916,7 @@ static int btrfs_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, goto mapit; /* we're doing a write, do the async checksumming */ ret = btrfs_wq_submit_bio(BTRFS_I(inode)->root->fs_info, - inode, rw, bio, mirror_num, + inode, bio, mirror_num, bio_flags, bio_offset, __btrfs_submit_bio_start, __btrfs_submit_bio_done); @@ -1929,7 +1928,7 @@ static int btrfs_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, } mapit: - ret = btrfs_map_bio(root, rw, bio, mirror_num, 0); + ret = btrfs_map_bio(root, bio, mirror_num, 0); out: if (ret < 0) { @@ -7708,7 +7707,7 @@ err: } static inline int submit_dio_repair_bio(struct inode *inode, struct bio *bio, - int rw, int mirror_num) + int mirror_num) { struct btrfs_root *root = BTRFS_I(inode)->root; int ret; @@ -7722,7 +7721,7 @@ static inline int submit_dio_repair_bio(struct inode *inode, struct bio *bio, if (ret) goto err; - ret = btrfs_map_bio(root, rw, bio, mirror_num, 0); + ret = btrfs_map_bio(root, bio, mirror_num, 0); err: bio_put(bio); return ret; @@ -7806,8 +7805,7 @@ static int dio_read_error(struct inode *inode, struct bio *failed_bio, "Repair DIO Read Error: submitting new dio read[%#x] to this_mirror=%d, in_validation=%d\n", read_mode, failrec->this_mirror, failrec->in_validation); - ret = submit_dio_repair_bio(inode, bio, read_mode, - failrec->this_mirror); + ret = submit_dio_repair_bio(inode, bio, failrec->this_mirror); if (ret) { free_io_failure(inode, failrec); bio_put(bio); @@ -8045,7 +8043,7 @@ static void btrfs_endio_direct_write(struct bio *bio) bio_put(bio); } -static int __btrfs_submit_bio_start_direct_io(struct inode *inode, int rw, +static int __btrfs_submit_bio_start_direct_io(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 offset) { @@ -8138,7 +8136,7 @@ static inline int btrfs_lookup_and_bind_dio_csum(struct btrfs_root *root, } static inline int __btrfs_submit_dio_bio(struct bio *bio, struct inode *inode, - int rw, u64 file_offset, int skip_sum, + u64 file_offset, int skip_sum, int async_submit) { struct btrfs_dio_private *dip = bio->bi_private; @@ -8163,8 +8161,7 @@ static inline int __btrfs_submit_dio_bio(struct bio *bio, struct inode *inode, if (write && async_submit) { ret = btrfs_wq_submit_bio(root->fs_info, - inode, rw, bio, 0, 0, - file_offset, + inode, bio, 0, 0, file_offset, __btrfs_submit_bio_start_direct_io, __btrfs_submit_bio_done); goto err; @@ -8183,13 +8180,13 @@ static inline int __btrfs_submit_dio_bio(struct bio *bio, struct inode *inode, goto err; } map: - ret = btrfs_map_bio(root, rw, bio, 0, async_submit); + ret = btrfs_map_bio(root, bio, 0, async_submit); err: bio_put(bio); return ret; } -static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, +static int btrfs_submit_direct_hook(struct btrfs_dio_private *dip, int skip_sum) { struct inode *inode = dip->inode; @@ -8245,7 +8242,7 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, * before we're done setting it up */ atomic_inc(&dip->pending_bios); - ret = __btrfs_submit_dio_bio(bio, inode, rw, + ret = __btrfs_submit_dio_bio(bio, inode, file_offset, skip_sum, async_submit); if (ret) { @@ -8286,7 +8283,7 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, } submit: - ret = __btrfs_submit_dio_bio(bio, inode, rw, file_offset, skip_sum, + ret = __btrfs_submit_dio_bio(bio, inode, file_offset, skip_sum, async_submit); if (!ret) return 0; @@ -8364,8 +8361,7 @@ static void btrfs_submit_direct(struct bio *dio_bio, struct inode *inode, dio_data->unsubmitted_oe_range_end; } - ret = btrfs_submit_direct_hook(dio_bio->bi_op | dio_bio->bi_rw, dip, - skip_sum); + ret = btrfs_submit_direct_hook(dip, skip_sum); if (!ret) return; diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index bf1e9af..c281874 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -5999,7 +5999,7 @@ static noinline void btrfs_schedule_bio(struct btrfs_root *root, static void submit_stripe_bio(struct btrfs_root *root, struct btrfs_bio *bbio, struct bio *bio, u64 physical, int dev_nr, - int rw, int async) + int async) { struct btrfs_device *dev = bbio->stripes[dev_nr].dev; @@ -6044,7 +6044,7 @@ static void bbio_error(struct btrfs_bio *bbio, struct bio *bio, u64 logical) } } -int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, +int btrfs_map_bio(struct btrfs_root *root,struct bio *bio, int mirror_num, int async_submit) { struct btrfs_device *dev; @@ -6110,7 +6110,7 @@ int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, bio = first_bio; submit_stripe_bio(root, bbio, bio, - bbio->stripes[dev_nr].physical, dev_nr, rw, + bbio->stripes[dev_nr].physical, dev_nr, async_submit); } btrfs_bio_counter_dec(root->fs_info); diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h index 8ef796a..5a2da8a 100644 --- a/fs/btrfs/volumes.h +++ b/fs/btrfs/volumes.h @@ -437,7 +437,7 @@ int btrfs_alloc_chunk(struct btrfs_trans_handle *trans, struct btrfs_root *extent_root, u64 type); void btrfs_mapping_init(struct btrfs_mapping_tree *tree); void btrfs_mapping_tree_free(struct btrfs_mapping_tree *tree); -int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, +int btrfs_map_bio(struct btrfs_root *root, struct bio *bio, int mirror_num, int async_submit); int btrfs_open_devices(struct btrfs_fs_devices *fs_devices, fmode_t flags, void *holder); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id BA43129E26 for ; Tue, 5 Jan 2016 14:54:18 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 844448F804C for ; Tue, 5 Jan 2016 12:54:18 -0800 (PST) X-ASG-Debug-ID: 1452027256-04bdf03afa04de0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 3YWG30UvP1OamL74 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:16 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 4BA5A42E5D9; Tue, 5 Jan 2016 20:54:16 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuA024032; Tue, 5 Jan 2016 15:54:13 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 11/35] f2fs: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:14 -0600 X-ASG-Orig-Subj: [PATCH 11/35] f2fs: set bi_op to REQ_OP Message-Id: <1452027218-32303-12-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027256 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has f2fs set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- fs/f2fs/checkpoint.c | 10 ++++++---- fs/f2fs/data.c | 33 ++++++++++++++++++++------------- fs/f2fs/f2fs.h | 5 +++-- fs/f2fs/gc.c | 9 ++++++--- fs/f2fs/inline.c | 3 ++- fs/f2fs/node.c | 8 +++++--- fs/f2fs/segment.c | 10 +++++++--- fs/f2fs/trace.c | 8 +++++--- include/trace/events/f2fs.h | 34 +++++++++++++++++++++------------- 9 files changed, 75 insertions(+), 45 deletions(-) diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index fdd43f7..92d05d8 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -55,13 +55,14 @@ static struct page *__get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index, struct f2fs_io_info fio = { .sbi = sbi, .type = META, - .rw = READ_SYNC | REQ_META | REQ_PRIO, + .op = REQ_OP_READ, + .op_flags = READ_SYNC | REQ_META | REQ_PRIO, .blk_addr = index, .encrypted_page = NULL, }; if (unlikely(!is_meta)) - fio.rw &= ~REQ_META; + fio.op_flags &= ~REQ_META; repeat: page = grab_cache_page(mapping, index); if (!page) { @@ -149,12 +150,13 @@ int ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, struct f2fs_io_info fio = { .sbi = sbi, .type = META, - .rw = sync ? (READ_SYNC | REQ_META | REQ_PRIO) : READA, + .op = REQ_OP_READ, + .op_flags = sync ? (READ_SYNC | REQ_META | REQ_PRIO) : READA, .encrypted_page = NULL, }; if (unlikely(type == META_POR)) - fio.rw &= ~REQ_META; + fio.op_flags &= ~REQ_META; for (; nrpages-- > 0; blkno++) { diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 5325408..14757cb 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -107,11 +107,12 @@ static void __submit_merged_bio(struct f2fs_bio_info *io) if (!io->bio) return; - if (is_read_io(fio->rw)) + if (is_read_io(fio->op)) trace_f2fs_submit_read_bio(io->sbi->sb, fio, io->bio); else trace_f2fs_submit_write_bio(io->sbi->sb, fio, io->bio); - io->bio->bi_rw |= fio->rw; + io->bio->bi_op = fio->op; + io->bio->bi_rw |= fio->op_flags; submit_bio(io->bio); io->bio = NULL; @@ -130,10 +131,12 @@ void f2fs_submit_merged_bio(struct f2fs_sb_info *sbi, /* change META to META_FLUSH in the checkpoint procedure */ if (type >= META_FLUSH) { io->fio.type = META_FLUSH; + io->fio.op = REQ_OP_WRITE; if (test_opt(sbi, NOBARRIER)) - io->fio.rw = WRITE_FLUSH | REQ_META | REQ_PRIO; + io->fio.op_flags = WRITE_FLUSH | REQ_META | REQ_PRIO; else - io->fio.rw = WRITE_FLUSH_FUA | REQ_META | REQ_PRIO; + io->fio.op_flags = WRITE_FLUSH_FUA | REQ_META | + REQ_PRIO; } __submit_merged_bio(io); up_write(&io->io_rwsem); @@ -152,13 +155,14 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio) f2fs_trace_ios(fio, 0); /* Allocate a new bio */ - bio = __bio_alloc(fio->sbi, fio->blk_addr, 1, is_read_io(fio->rw)); + bio = __bio_alloc(fio->sbi, fio->blk_addr, 1, is_read_io(fio->op)); if (bio_add_page(bio, page, PAGE_CACHE_SIZE, 0) < PAGE_CACHE_SIZE) { bio_put(bio); return -EFAULT; } - bio->bi_rw |= fio->rw; + bio->bi_op = fio->op; + bio->bi_rw |= fio->op_flags; submit_bio(bio); return 0; @@ -169,7 +173,7 @@ void f2fs_submit_page_mbio(struct f2fs_io_info *fio) struct f2fs_sb_info *sbi = fio->sbi; enum page_type btype = PAGE_TYPE_OF_BIO(fio->type); struct f2fs_bio_info *io; - bool is_read = is_read_io(fio->rw); + bool is_read = is_read_io(fio->op); struct page *bio_page; io = is_read ? &sbi->read_io : &sbi->write_io[btype]; @@ -182,7 +186,7 @@ void f2fs_submit_page_mbio(struct f2fs_io_info *fio) inc_page_count(sbi, F2FS_WRITEBACK); if (io->bio && (io->last_block_in_bio != fio->blk_addr - 1 || - io->fio.rw != fio->rw)) + (io->fio.op != fio->op || io->fio.op_flags != fio->op_flags))) __submit_merged_bio(io); alloc_new: if (io->bio == NULL) { @@ -278,7 +282,7 @@ int f2fs_get_block(struct dnode_of_data *dn, pgoff_t index) } struct page *get_read_data_page(struct inode *inode, pgoff_t index, - int rw, bool for_write) + int op_flags, bool for_write) { struct address_space *mapping = inode->i_mapping; struct dnode_of_data dn; @@ -288,7 +292,8 @@ struct page *get_read_data_page(struct inode *inode, pgoff_t index, struct f2fs_io_info fio = { .sbi = F2FS_I_SB(inode), .type = DATA, - .rw = rw, + .op = REQ_OP_READ, + .op_flags = op_flags, .encrypted_page = NULL, }; @@ -984,7 +989,7 @@ submit_and_realloc: bio->bi_iter.bi_sector = SECTOR_FROM_BLOCK(block_nr); bio->bi_end_io = f2fs_read_end_io; bio->bi_private = ctx; - bio->bi_rw |= READ; + bio->bi_op = REQ_OP_READ; } if (bio_add_page(bio, page, blocksize, 0) < blocksize) @@ -1118,7 +1123,8 @@ static int f2fs_write_data_page(struct page *page, struct f2fs_io_info fio = { .sbi = sbi, .type = DATA, - .rw = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : WRITE, + .op = REQ_OP_WRITE, + .op_flags = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : 0, .page = page, .encrypted_page = NULL, }; @@ -1489,7 +1495,8 @@ put_next: struct f2fs_io_info fio = { .sbi = sbi, .type = DATA, - .rw = READ_SYNC, + .op = REQ_OP_READ, + .op_flags = READ_SYNC, .blk_addr = dn.data_blkaddr, .page = page, .encrypted_page = NULL, diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index fda0831..6e8227b 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -683,13 +683,14 @@ enum page_type { struct f2fs_io_info { struct f2fs_sb_info *sbi; /* f2fs_sb_info pointer */ enum page_type type; /* contains DATA/NODE/META/META_FLUSH */ - int rw; /* contains R/RS/W/WS with REQ_META/REQ_PRIO */ + int op; /* contains REQ_OP_ */ + int op_flags; /* rq_flag_bits */ block_t blk_addr; /* block address to be written */ struct page *page; /* page to be written */ struct page *encrypted_page; /* encrypted page */ }; -#define is_read_io(rw) (((rw) & 1) == READ) +#define is_read_io(rw) (rw == READ) struct f2fs_bio_info { struct f2fs_sb_info *sbi; /* f2fs superblock */ struct bio *bio; /* bios to merge */ diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index ce350c4..8561cf5 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -540,7 +540,8 @@ static void move_encrypted_block(struct inode *inode, block_t bidx) struct f2fs_io_info fio = { .sbi = F2FS_I_SB(inode), .type = DATA, - .rw = READ_SYNC, + .op = REQ_OP_READ, + .op_flags = READ_SYNC, .encrypted_page = NULL, }; struct dnode_of_data dn; @@ -607,7 +608,8 @@ static void move_encrypted_block(struct inode *inode, block_t bidx) f2fs_wait_on_page_writeback(dn.node_page, NODE); allocate_data_block(fio.sbi, NULL, fio.blk_addr, &fio.blk_addr, &sum, CURSEG_COLD_DATA); - fio.rw = WRITE_SYNC; + fio.op = REQ_OP_WRITE; + fio.op_flags = WRITE_SYNC; f2fs_submit_page_mbio(&fio); dn.data_blkaddr = fio.blk_addr; @@ -641,7 +643,8 @@ static void move_data_page(struct inode *inode, block_t bidx, int gc_type) struct f2fs_io_info fio = { .sbi = F2FS_I_SB(inode), .type = DATA, - .rw = WRITE_SYNC, + .op = REQ_OP_WRITE, + .op_flags = WRITE_SYNC, .page = page, .encrypted_page = NULL, }; diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c index bda7126..d8c89f8 100644 --- a/fs/f2fs/inline.c +++ b/fs/f2fs/inline.c @@ -112,7 +112,8 @@ int f2fs_convert_inline_page(struct dnode_of_data *dn, struct page *page) struct f2fs_io_info fio = { .sbi = F2FS_I_SB(dn->inode), .type = DATA, - .rw = WRITE_SYNC | REQ_PRIO, + .op = REQ_OP_WRITE, + .op_flags = WRITE_SYNC | REQ_PRIO, .page = page, .encrypted_page = NULL, }; diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index d842b19..7261a29 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -1001,14 +1001,15 @@ fail: * 0: f2fs_put_page(page, 0) * LOCKED_PAGE or error: f2fs_put_page(page, 1) */ -static int read_node_page(struct page *page, int rw) +static int read_node_page(struct page *page, int op_flags) { struct f2fs_sb_info *sbi = F2FS_P_SB(page); struct node_info ni; struct f2fs_io_info fio = { .sbi = sbi, .type = NODE, - .rw = rw, + .op = REQ_OP_READ, + .op_flags = op_flags, .page = page, .encrypted_page = NULL, }; @@ -1309,7 +1310,8 @@ static int f2fs_write_node_page(struct page *page, struct f2fs_io_info fio = { .sbi = sbi, .type = NODE, - .rw = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : WRITE, + .op = REQ_OP_WRITE, + .op_flags = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : 0, .page = page, .encrypted_page = NULL, }; diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 6ebdbb3..f2ceb8a 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -200,7 +200,8 @@ int commit_inmem_pages(struct inode *inode, bool abort) struct f2fs_io_info fio = { .sbi = sbi, .type = DATA, - .rw = WRITE_SYNC | REQ_PRIO, + .op = REQ_OP_WRITE, + .op_flags = WRITE_SYNC | REQ_PRIO, .encrypted_page = NULL, }; int err = 0; @@ -318,6 +319,7 @@ repeat: fcc->dispatch_list = llist_reverse_order(fcc->dispatch_list); bio->bi_bdev = sbi->sb->s_bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw |= WRITE_FLUSH; ret = submit_bio_wait(bio); @@ -351,6 +353,7 @@ int f2fs_issue_flush(struct f2fs_sb_info *sbi) int ret; bio->bi_bdev = sbi->sb->s_bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw |= WRITE_FLUSH; ret = submit_bio_wait(bio); bio_put(bio); @@ -1290,14 +1293,15 @@ void write_meta_page(struct f2fs_sb_info *sbi, struct page *page) struct f2fs_io_info fio = { .sbi = sbi, .type = META, - .rw = WRITE_SYNC | REQ_META | REQ_PRIO, + .op = REQ_OP_WRITE, + .op_flags = WRITE_SYNC | REQ_META | REQ_PRIO, .blk_addr = page->index, .page = page, .encrypted_page = NULL, }; if (unlikely(page->index >= MAIN_BLKADDR(sbi))) - fio.rw &= ~REQ_META; + fio.op_flags &= ~REQ_META; set_page_writeback(page); f2fs_submit_page_mbio(&fio); diff --git a/fs/f2fs/trace.c b/fs/f2fs/trace.c index 145fb65..5b7edca 100644 --- a/fs/f2fs/trace.c +++ b/fs/f2fs/trace.c @@ -25,11 +25,12 @@ static inline void __print_last_io(void) if (!last_io.len) return; - trace_printk("%3x:%3x %4x %-16s %2x %5x %12x %4x\n", + trace_printk("%3x:%3x %4x %-16s %2x %5x %5x %12x %4x\n", last_io.major, last_io.minor, last_io.pid, "----------------", last_io.type, - last_io.fio.rw, last_io.fio.blk_addr, + last_io.fio.op, last_io.fio.op_flags, + last_io.fio.blk_addr, last_io.len); memset(&last_io, 0, sizeof(last_io)); } @@ -100,7 +101,8 @@ void f2fs_trace_ios(struct f2fs_io_info *fio, int flush) if (last_io.major == major && last_io.minor == minor && last_io.pid == pid && last_io.type == __file_type(inode, pid) && - last_io.fio.rw == fio->rw && + last_io.fio.op == fio->op && + last_io.fio.op_flags == fio->op_flags && last_io.fio.blk_addr + last_io.len == fio->blk_addr) { last_io.len++; return; diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index a1b4888..ddf4144 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -55,17 +55,21 @@ TRACE_DEFINE_ENUM(CP_DISCARD); { IPU, "IN-PLACE" }, \ { OPU, "OUT-OF-PLACE" }) -#define F2FS_BIO_MASK(t) (t & (READA | WRITE_FLUSH_FUA)) +#define F2FS_BIO_FLAG_MASK(t) (t & (READA | WRITE_FLUSH_FUA)) #define F2FS_BIO_EXTRA_MASK(t) (t & (REQ_META | REQ_PRIO)) -#define show_bio_type(type) show_bio_base(type), show_bio_extra(type) +#define show_bio_type(op, op_flags) show_bio_op(op), \ + show_bio_op_flags(op_flags), show_bio_extra(op_flags) -#define show_bio_base(type) \ - __print_symbolic(F2FS_BIO_MASK(type), \ +#define show_bio_op(op) \ + __print_symbolic(op, \ { READ, "READ" }, \ + { WRITE, "WRITE" }) + +#define show_bio_op_flags(flags) \ + __print_symbolic(F2FS_BIO_FLAG_MASK(flags), \ { READA, "READAHEAD" }, \ { READ_SYNC, "READ_SYNC" }, \ - { WRITE, "WRITE" }, \ { WRITE_SYNC, "WRITE_SYNC" }, \ { WRITE_FLUSH, "WRITE_FLUSH" }, \ { WRITE_FUA, "WRITE_FUA" }, \ @@ -728,7 +732,8 @@ DECLARE_EVENT_CLASS(f2fs__submit_page_bio, __field(ino_t, ino) __field(pgoff_t, index) __field(block_t, blkaddr) - __field(int, rw) + __field(int, op) + __field(int, op_flags) __field(int, type) ), @@ -737,16 +742,17 @@ DECLARE_EVENT_CLASS(f2fs__submit_page_bio, __entry->ino = page->mapping->host->i_ino; __entry->index = page->index; __entry->blkaddr = fio->blk_addr; - __entry->rw = fio->rw; + __entry->op = fio->op; + __entry->op_flags = fio->op_flags; __entry->type = fio->type; ), TP_printk("dev = (%d,%d), ino = %lu, page_index = 0x%lx, " - "blkaddr = 0x%llx, rw = %s%s, type = %s", + "blkaddr = 0x%llx, rw = %s%s%s, type = %s", show_dev_ino(__entry), (unsigned long)__entry->index, (unsigned long long)__entry->blkaddr, - show_bio_type(__entry->rw), + show_bio_type(__entry->op, __entry->op_flags), show_block_type(__entry->type)) ); @@ -777,7 +783,8 @@ DECLARE_EVENT_CLASS(f2fs__submit_bio, TP_STRUCT__entry( __field(dev_t, dev) - __field(int, rw) + __field(int, op) + __field(int, op_flags) __field(int, type) __field(sector_t, sector) __field(unsigned int, size) @@ -785,15 +792,16 @@ DECLARE_EVENT_CLASS(f2fs__submit_bio, TP_fast_assign( __entry->dev = sb->s_dev; - __entry->rw = fio->rw; + __entry->op = fio->op; + __entry->op_flags = fio->op_flags; __entry->type = fio->type; __entry->sector = bio->bi_iter.bi_sector; __entry->size = bio->bi_iter.bi_size; ), - TP_printk("dev = (%d,%d), %s%s, %s, sector = %lld, size = %u", + TP_printk("dev = (%d,%d), %s%s%s, %s, sector = %lld, size = %u", show_dev(__entry), - show_bio_type(__entry->rw), + show_bio_type(__entry->op, __entry->op_flags), show_block_type(__entry->type), (unsigned long long)__entry->sector, __entry->size) -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:20 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id CAAEE29E30 for ; Tue, 5 Jan 2016 14:54:20 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 931C5304066 for ; Tue, 5 Jan 2016 12:54:20 -0800 (PST) X-ASG-Debug-ID: 1452027259-04bdf03af904de0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id CHWJ1jCmyz6V9ZDj (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:19 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id EF1456E797; Tue, 5 Jan 2016 20:54:18 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuB024032; Tue, 5 Jan 2016 15:54:16 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 12/35] gfs2: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:15 -0600 X-ASG-Orig-Subj: [PATCH 12/35] gfs2: set bi_op to REQ_OP Message-Id: <1452027218-32303-13-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027259 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has gfs2 set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. v2: Bob, I did not add your signed off, because there was the gfs2_submit_bhs changes since last time you reviewed it. Signed-off-by: Mike Christie --- fs/gfs2/log.c | 8 ++++---- fs/gfs2/lops.c | 12 +++++++----- fs/gfs2/lops.h | 2 +- fs/gfs2/meta_io.c | 8 +++++--- fs/gfs2/ops_fstype.c | 1 + 5 files changed, 18 insertions(+), 13 deletions(-) diff --git a/fs/gfs2/log.c b/fs/gfs2/log.c index 0ff028c..e58ccef0 100644 --- a/fs/gfs2/log.c +++ b/fs/gfs2/log.c @@ -657,7 +657,7 @@ static void log_write_header(struct gfs2_sbd *sdp, u32 flags) struct gfs2_log_header *lh; unsigned int tail; u32 hash; - int rw = WRITE_FLUSH_FUA | REQ_META; + int op_flags = WRITE_FLUSH_FUA | REQ_META; struct page *page = mempool_alloc(gfs2_page_pool, GFP_NOIO); enum gfs2_freeze_state state = atomic_read(&sdp->sd_freeze_state); lh = page_address(page); @@ -682,12 +682,12 @@ static void log_write_header(struct gfs2_sbd *sdp, u32 flags) if (test_bit(SDF_NOBARRIERS, &sdp->sd_flags)) { gfs2_ordered_wait(sdp); log_flush_wait(sdp); - rw = WRITE_SYNC | REQ_META | REQ_PRIO; + op_flags = WRITE_SYNC | REQ_META | REQ_PRIO; } sdp->sd_log_idle = (tail == sdp->sd_log_flush_head); gfs2_log_write_page(sdp, page); - gfs2_log_flush_bio(sdp, rw); + gfs2_log_flush_bio(sdp, REQ_OP_WRITE, op_flags); log_flush_wait(sdp); if (sdp->sd_log_tail != tail) @@ -738,7 +738,7 @@ void gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl, gfs2_ordered_write(sdp); lops_before_commit(sdp, tr); - gfs2_log_flush_bio(sdp, WRITE); + gfs2_log_flush_bio(sdp, REQ_OP_WRITE, 0); if (sdp->sd_log_head != sdp->sd_log_flush_head) { log_flush_wait(sdp); diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index acc5ccb..11980f6 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -230,17 +230,19 @@ static void gfs2_end_log_write(struct bio *bio) /** * gfs2_log_flush_bio - Submit any pending log bio * @sdp: The superblock - * @rw: The rw flags + * @op: REQ_OP + * @op_flags: rq_flag_bits * * Submit any pending part-built or full bio to the block device. If * there is no pending bio, then this is a no-op. */ -void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int rw) +void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int op, int op_flags) { if (sdp->sd_log_bio) { atomic_inc(&sdp->sd_log_in_flight); - sdp->sd_log_bio->bi_rw |= rw; + sdp->sd_log_bio->bi_op = op; + sdp->sd_log_bio->bi_rw |= op_flags; submit_bio(sdp->sd_log_bio); sdp->sd_log_bio = NULL; } @@ -300,7 +302,7 @@ static struct bio *gfs2_log_get_bio(struct gfs2_sbd *sdp, u64 blkno) nblk >>= sdp->sd_fsb2bb_shift; if (blkno == nblk) return bio; - gfs2_log_flush_bio(sdp, WRITE); + gfs2_log_flush_bio(sdp, REQ_OP_WRITE, 0); } return gfs2_log_alloc_bio(sdp, blkno); @@ -329,7 +331,7 @@ static void gfs2_log_write(struct gfs2_sbd *sdp, struct page *page, bio = gfs2_log_get_bio(sdp, blkno); ret = bio_add_page(bio, page, size, offset); if (ret == 0) { - gfs2_log_flush_bio(sdp, WRITE); + gfs2_log_flush_bio(sdp, REQ_OP_WRITE, 0); bio = gfs2_log_alloc_bio(sdp, blkno); ret = bio_add_page(bio, page, size, offset); WARN_ON(ret == 0); diff --git a/fs/gfs2/lops.h b/fs/gfs2/lops.h index a65a7ba..e529f53 100644 --- a/fs/gfs2/lops.h +++ b/fs/gfs2/lops.h @@ -27,7 +27,7 @@ extern const struct gfs2_log_operations gfs2_databuf_lops; extern const struct gfs2_log_operations *gfs2_log_ops[]; extern void gfs2_log_write_page(struct gfs2_sbd *sdp, struct page *page); -extern void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int rw); +extern void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int op, int op_flags); extern void gfs2_pin(struct gfs2_sbd *sdp, struct buffer_head *bh); static inline unsigned int buf_limit(struct gfs2_sbd *sdp) diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index e2fc2b9..3996c1d 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -213,7 +213,8 @@ static void gfs2_meta_read_endio(struct bio *bio) * Submit several consecutive buffer head I/O requests as a single bio I/O * request. (See submit_bh_wbc.) */ -static void gfs2_submit_bhs(int rw, struct buffer_head *bhs[], int num) +static void gfs2_submit_bhs(int op, int op_flags, struct buffer_head *bhs[], + int num) { struct buffer_head *bh = bhs[0]; struct bio *bio; @@ -230,7 +231,8 @@ static void gfs2_submit_bhs(int rw, struct buffer_head *bhs[], int num) bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh)); } bio->bi_end_io = gfs2_meta_read_endio; - bio->bi_rw |= rw; + bio->bi_op = op; + bio->bi_rw |= op_flags; submit_bio(bio); } @@ -281,7 +283,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags, } } - gfs2_submit_bhs(READ_SYNC | REQ_META | REQ_PRIO, bhs, num); + gfs2_submit_bhs(REQ_OP_READ, READ_SYNC | REQ_META | REQ_PRIO, bhs, num); if (!(flags & DIO_WAIT)) return 0; diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c index 15a4f86..cc8600a 100644 --- a/fs/gfs2/ops_fstype.c +++ b/fs/gfs2/ops_fstype.c @@ -246,6 +246,7 @@ static int gfs2_read_super(struct gfs2_sbd *sdp, sector_t sector, int silent) bio->bi_end_io = end_bio_io_page; bio->bi_private = page; + bio->bi_op = REQ_OP_READ; bio->bi_rw |= READ_SYNC | REQ_META; submit_bio(bio); wait_on_page_locked(page); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:23 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 662FE29E17 for ; Tue, 5 Jan 2016 14:54:23 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4F1508F8040 for ; Tue, 5 Jan 2016 12:54:23 -0800 (PST) X-ASG-Debug-ID: 1452027262-04cb6c42e505250001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id EA2B2EcBFxB1xsu7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:22 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id CD1043B757; Tue, 5 Jan 2016 20:54:21 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuC024032; Tue, 5 Jan 2016 15:54:19 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 13/35] xfs: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:16 -0600 X-ASG-Orig-Subj: [PATCH 13/35] xfs: set bi_op to REQ_OP Message-Id: <1452027218-32303-14-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027262 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has xfs set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. Note: I have run xfs tests on these btrfs patches. There were some failures with and without the patches. I have not had time to track down why xfstest fails without the patches. Signed-off-by: Mike Christie --- fs/xfs/xfs_aops.c | 3 ++- fs/xfs/xfs_buf.c | 27 +++++++++++++++------------ 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index a1052d2..3a00935 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -382,7 +382,8 @@ xfs_submit_ioend_bio( atomic_inc(&ioend->io_remaining); bio->bi_private = ioend; bio->bi_end_io = xfs_end_bio; - bio->bi_rw |= (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + bio->bi_op = REQ_OP_WRITE; + bio->bi_rw |= WB_SYNC_ALL ? WRITE_SYNC : 0; submit_bio(bio); } diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 644e676..4cfba72 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1124,7 +1124,8 @@ xfs_buf_ioapply_map( int map, int *buf_offset, int *count, - int rw) + int op, + int op_flags) { int page_index; int total_nr_pages = bp->b_page_count; @@ -1163,7 +1164,8 @@ next_chunk: bio->bi_iter.bi_sector = sector; bio->bi_end_io = xfs_buf_bio_end_io; bio->bi_private = bp; - bio->bi_rw |= rw; + bio->bi_op = op; + bio->bi_rw |= op_flags; for (; size && nr_pages; nr_pages--, page_index++) { int rbytes, nbytes = PAGE_SIZE - offset; @@ -1207,7 +1209,8 @@ _xfs_buf_ioapply( struct xfs_buf *bp) { struct blk_plug plug; - int rw; + int op; + int op_flags = 0; int offset; int size; int i; @@ -1226,14 +1229,13 @@ _xfs_buf_ioapply( bp->b_ioend_wq = bp->b_target->bt_mount->m_buf_workqueue; if (bp->b_flags & XBF_WRITE) { + op = REQ_OP_WRITE; if (bp->b_flags & XBF_SYNCIO) - rw = WRITE_SYNC; - else - rw = WRITE; + op_flags = WRITE_SYNC; if (bp->b_flags & XBF_FUA) - rw |= REQ_FUA; + op_flags |= REQ_FUA; if (bp->b_flags & XBF_FLUSH) - rw |= REQ_FLUSH; + op_flags |= REQ_FLUSH; /* * Run the write verifier callback function if it exists. If @@ -1263,13 +1265,14 @@ _xfs_buf_ioapply( } } } else if (bp->b_flags & XBF_READ_AHEAD) { - rw = READA; + op = REQ_OP_READ; + op_flags = REQ_RAHEAD; } else { - rw = READ; + op = REQ_OP_READ; } /* we only use the buffer cache for meta-data */ - rw |= REQ_META; + op_flags |= REQ_META; /* * Walk all the vectors issuing IO on them. Set up the initial offset @@ -1281,7 +1284,7 @@ _xfs_buf_ioapply( size = BBTOB(bp->b_io_length); blk_start_plug(&plug); for (i = 0; i < bp->b_map_count; i++) { - xfs_buf_ioapply_map(bp, i, &offset, &size, rw); + xfs_buf_ioapply_map(bp, i, &offset, &size, op, op_flags); if (bp->b_error) break; if (size <= 0) -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:26 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 491C529E06 for ; Tue, 5 Jan 2016 14:54:26 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3B888304039 for ; Tue, 5 Jan 2016 12:54:26 -0800 (PST) X-ASG-Debug-ID: 1452027264-04cb6c42e705260001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Km4TJCLr9QqGPMUN (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:25 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 75C4B8E931; Tue, 5 Jan 2016 20:54:24 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuD024032; Tue, 5 Jan 2016 15:54:21 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 14/35] hfsplus: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:17 -0600 X-ASG-Orig-Subj: [PATCH 14/35] hfsplus: set bi_op to REQ_OP Message-Id: <1452027218-32303-15-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027265 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has hfsplus set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- fs/hfsplus/hfsplus_fs.h | 2 +- fs/hfsplus/part_tbl.c | 5 +++-- fs/hfsplus/super.c | 6 ++++-- fs/hfsplus/wrapper.c | 15 +++++++++------ 4 files changed, 17 insertions(+), 11 deletions(-) diff --git a/fs/hfsplus/hfsplus_fs.h b/fs/hfsplus/hfsplus_fs.h index f91a1fa..80154aa 100644 --- a/fs/hfsplus/hfsplus_fs.h +++ b/fs/hfsplus/hfsplus_fs.h @@ -525,7 +525,7 @@ int hfsplus_compare_dentry(const struct dentry *parent, /* wrapper.c */ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, void *buf, - void **data, int rw); + void **data, int op, int op_flags); int hfsplus_read_wrapper(struct super_block *sb); /* time macros */ diff --git a/fs/hfsplus/part_tbl.c b/fs/hfsplus/part_tbl.c index eb355d8..63164eb 100644 --- a/fs/hfsplus/part_tbl.c +++ b/fs/hfsplus/part_tbl.c @@ -112,7 +112,8 @@ static int hfs_parse_new_pmap(struct super_block *sb, void *buf, if ((u8 *)pm - (u8 *)buf >= buf_size) { res = hfsplus_submit_bio(sb, *part_start + HFS_PMAP_BLK + i, - buf, (void **)&pm, READ); + buf, (void **)&pm, REQ_OP_READ, + 0); if (res) return res; } @@ -136,7 +137,7 @@ int hfs_part_find(struct super_block *sb, return -ENOMEM; res = hfsplus_submit_bio(sb, *part_start + HFS_PMAP_BLK, - buf, &data, READ); + buf, &data, REQ_OP_READ, 0); if (res) goto out; diff --git a/fs/hfsplus/super.c b/fs/hfsplus/super.c index 5d54490..01cf313 100644 --- a/fs/hfsplus/super.c +++ b/fs/hfsplus/super.c @@ -219,7 +219,8 @@ static int hfsplus_sync_fs(struct super_block *sb, int wait) error2 = hfsplus_submit_bio(sb, sbi->part_start + HFSPLUS_VOLHEAD_SECTOR, - sbi->s_vhdr_buf, NULL, WRITE_SYNC); + sbi->s_vhdr_buf, NULL, REQ_OP_WRITE, + WRITE_SYNC); if (!error) error = error2; if (!write_backup) @@ -227,7 +228,8 @@ static int hfsplus_sync_fs(struct super_block *sb, int wait) error2 = hfsplus_submit_bio(sb, sbi->part_start + sbi->sect_count - 2, - sbi->s_backup_vhdr_buf, NULL, WRITE_SYNC); + sbi->s_backup_vhdr_buf, NULL, REQ_OP_WRITE, + WRITE_SYNC); if (!error) error2 = error; out: diff --git a/fs/hfsplus/wrapper.c b/fs/hfsplus/wrapper.c index 7e605b5..d09b726 100644 --- a/fs/hfsplus/wrapper.c +++ b/fs/hfsplus/wrapper.c @@ -30,7 +30,8 @@ struct hfsplus_wd { * @sector: block to read or write, for blocks of HFSPLUS_SECTOR_SIZE bytes * @buf: buffer for I/O * @data: output pointer for location of requested data - * @rw: direction of I/O + * @op: direction of I/O + * @op_flags: request op flags * * The unit of I/O is hfsplus_min_io_size(sb), which may be bigger than * HFSPLUS_SECTOR_SIZE, and @buf must be sized accordingly. On reads @@ -44,7 +45,7 @@ struct hfsplus_wd { * will work correctly. */ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, - void *buf, void **data, int rw) + void *buf, void **data, int op, int op_flags) { struct bio *bio; int ret = 0; @@ -65,9 +66,10 @@ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, bio = bio_alloc(GFP_NOIO, 1); bio->bi_iter.bi_sector = sector; bio->bi_bdev = sb->s_bdev; - bio->bi_rw |= rw; + bio->bi_op = op; + bio->bi_rw |= op_flags; - if (!(rw & WRITE) && data) + if (op != WRITE && data) *data = (u8 *)buf + offset; while (io_size > 0) { @@ -182,7 +184,7 @@ int hfsplus_read_wrapper(struct super_block *sb) reread: error = hfsplus_submit_bio(sb, part_start + HFSPLUS_VOLHEAD_SECTOR, sbi->s_vhdr_buf, (void **)&sbi->s_vhdr, - READ); + REQ_OP_READ, 0); if (error) goto out_free_backup_vhdr; @@ -214,7 +216,8 @@ reread: error = hfsplus_submit_bio(sb, part_start + part_size - 2, sbi->s_backup_vhdr_buf, - (void **)&sbi->s_backup_vhdr, READ); + (void **)&sbi->s_backup_vhdr, REQ_OP_READ, + 0); if (error) goto out_free_backup_vhdr; -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:29 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D859F29E37 for ; Tue, 5 Jan 2016 14:54:28 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 7B548AC001 for ; Tue, 5 Jan 2016 12:54:28 -0800 (PST) X-ASG-Debug-ID: 1452027267-04bdf03afb04e00001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id oBzpPnxb61nBx7id (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:27 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 1B37914CAC6; Tue, 5 Jan 2016 20:54:27 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuE024032; Tue, 5 Jan 2016 15:54:24 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 15/35] mpage: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:18 -0600 X-ASG-Orig-Subj: [PATCH 15/35] mpage: set bi_op to REQ_OP Message-Id: <1452027218-32303-16-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027267 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the mpage.c code set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. I have run xfstest with xfs, but I am not sure if I have stressed these code paths well. Signed-off-by: Mike Christie --- fs/mpage.c | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/fs/mpage.c b/fs/mpage.c index 9fec67f..3f7d221 100644 --- a/fs/mpage.c +++ b/fs/mpage.c @@ -55,11 +55,12 @@ static void mpage_end_io(struct bio *bio) bio_put(bio); } -static struct bio *mpage_bio_submit(int rw, struct bio *bio) +static struct bio *mpage_bio_submit(int op, int op_flags, struct bio *bio) { bio->bi_end_io = mpage_end_io; - bio->bi_rw |= rw; - guard_bio_eod(rw, bio); + bio->bi_op = op; + bio->bi_rw |= op_flags; + guard_bio_eod(op, bio); submit_bio(bio); return NULL; } @@ -269,7 +270,7 @@ do_mpage_readpage(struct bio *bio, struct page *page, unsigned nr_pages, * This page will go to BIO. Do we need to send this BIO off first? */ if (bio && (*last_block_in_bio != blocks[0] - 1)) - bio = mpage_bio_submit(READ, bio); + bio = mpage_bio_submit(REQ_OP_READ, 0, bio); alloc_new: if (bio == NULL) { @@ -286,7 +287,7 @@ alloc_new: length = first_hole << blkbits; if (bio_add_page(bio, page, length, 0) < length) { - bio = mpage_bio_submit(READ, bio); + bio = mpage_bio_submit(REQ_OP_READ, 0, bio); goto alloc_new; } @@ -294,7 +295,7 @@ alloc_new: nblocks = map_bh->b_size >> blkbits; if ((buffer_boundary(map_bh) && relative_block == nblocks) || (first_hole != blocks_per_page)) - bio = mpage_bio_submit(READ, bio); + bio = mpage_bio_submit(REQ_OP_READ, 0, bio); else *last_block_in_bio = blocks[blocks_per_page - 1]; out: @@ -302,7 +303,7 @@ out: confused: if (bio) - bio = mpage_bio_submit(READ, bio); + bio = mpage_bio_submit(REQ_OP_READ, 0, bio); if (!PageUptodate(page)) block_read_full_page(page, get_block); else @@ -384,7 +385,7 @@ mpage_readpages(struct address_space *mapping, struct list_head *pages, } BUG_ON(!list_empty(pages)); if (bio) - mpage_bio_submit(READ, bio); + mpage_bio_submit(REQ_OP_READ, 0, bio); return 0; } EXPORT_SYMBOL(mpage_readpages); @@ -405,7 +406,7 @@ int mpage_readpage(struct page *page, get_block_t get_block) bio = do_mpage_readpage(bio, page, 1, &last_block_in_bio, &map_bh, &first_logical_block, get_block, gfp); if (bio) - mpage_bio_submit(READ, bio); + mpage_bio_submit(REQ_OP_READ, 0, bio); return 0; } EXPORT_SYMBOL(mpage_readpage); @@ -486,7 +487,7 @@ static int __mpage_writepage(struct page *page, struct writeback_control *wbc, struct buffer_head map_bh; loff_t i_size = i_size_read(inode); int ret = 0; - int wr = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + int op_flags = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : 0); if (page_has_buffers(page)) { struct buffer_head *head = page_buffers(page); @@ -595,7 +596,7 @@ page_is_mapped: * This page will go to BIO. Do we need to send this BIO off first? */ if (bio && mpd->last_block_in_bio != blocks[0] - 1) - bio = mpage_bio_submit(wr, bio); + bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); alloc_new: if (bio == NULL) { @@ -622,7 +623,7 @@ alloc_new: wbc_account_io(wbc, page, PAGE_SIZE); length = first_unmapped << blkbits; if (bio_add_page(bio, page, length, 0) < length) { - bio = mpage_bio_submit(wr, bio); + bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); goto alloc_new; } @@ -632,7 +633,7 @@ alloc_new: set_page_writeback(page); unlock_page(page); if (boundary || (first_unmapped != blocks_per_page)) { - bio = mpage_bio_submit(wr, bio); + bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); if (boundary_block) { write_boundary_block(boundary_bdev, boundary_block, 1 << blkbits); @@ -644,7 +645,7 @@ alloc_new: confused: if (bio) - bio = mpage_bio_submit(wr, bio); + bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); if (mpd->use_writepage) { ret = mapping->a_ops->writepage(page, wbc); @@ -701,9 +702,9 @@ mpage_writepages(struct address_space *mapping, ret = write_cache_pages(mapping, wbc, __mpage_writepage, &mpd); if (mpd.bio) { - int wr = (wbc->sync_mode == WB_SYNC_ALL ? - WRITE_SYNC : WRITE); - mpage_bio_submit(wr, mpd.bio); + int op_flags = (wbc->sync_mode == WB_SYNC_ALL ? + WRITE_SYNC : 0); + mpage_bio_submit(REQ_OP_WRITE, op_flags, mpd.bio); } } blk_finish_plug(&plug); @@ -722,9 +723,9 @@ int mpage_writepage(struct page *page, get_block_t get_block, }; int ret = __mpage_writepage(page, wbc, &mpd); if (mpd.bio) { - int wr = (wbc->sync_mode == WB_SYNC_ALL ? - WRITE_SYNC : WRITE); - mpage_bio_submit(wr, mpd.bio); + int op_flags = (wbc->sync_mode == WB_SYNC_ALL ? + WRITE_SYNC : 0); + mpage_bio_submit(REQ_OP_WRITE, op_flags, mpd.bio); } return ret; } -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:31 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 838A929E04 for ; Tue, 5 Jan 2016 14:54:31 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 4B810304039 for ; Tue, 5 Jan 2016 12:54:31 -0800 (PST) X-ASG-Debug-ID: 1452027270-04bdf03afa04e00001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id IkSBOuCtu0i7QHVs (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:30 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id D35F019F246; Tue, 5 Jan 2016 20:54:29 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuF024032; Tue, 5 Jan 2016 15:54:27 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 16/35] nilfs: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:19 -0600 X-ASG-Orig-Subj: [PATCH 16/35] nilfs: set bi_op to REQ_OP Message-Id: <1452027218-32303-17-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027270 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has nilfs set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- fs/nilfs2/segbuf.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/fs/nilfs2/segbuf.c b/fs/nilfs2/segbuf.c index 428ece8..8784272 100644 --- a/fs/nilfs2/segbuf.c +++ b/fs/nilfs2/segbuf.c @@ -350,7 +350,8 @@ static void nilfs_end_bio_write(struct bio *bio) } static int nilfs_segbuf_submit_bio(struct nilfs_segment_buffer *segbuf, - struct nilfs_write_info *wi, int mode) + struct nilfs_write_info *wi, int mode, + int mode_flags) { struct bio *bio = wi->bio; int err; @@ -368,7 +369,8 @@ static int nilfs_segbuf_submit_bio(struct nilfs_segment_buffer *segbuf, bio->bi_end_io = nilfs_end_bio_write; bio->bi_private = segbuf; - bio->bi_rw |= mode; + bio->bi_op = mode; + bio->bi_rw |= mode_flags; submit_bio(bio); segbuf->sb_nbio++; @@ -442,7 +444,7 @@ static int nilfs_segbuf_submit_bh(struct nilfs_segment_buffer *segbuf, return 0; } /* bio is FULL */ - err = nilfs_segbuf_submit_bio(segbuf, wi, mode); + err = nilfs_segbuf_submit_bio(segbuf, wi, mode, 0); /* never submit current bh */ if (likely(!err)) goto repeat; @@ -466,19 +468,19 @@ static int nilfs_segbuf_write(struct nilfs_segment_buffer *segbuf, { struct nilfs_write_info wi; struct buffer_head *bh; - int res = 0, rw = WRITE; + int res = 0; wi.nilfs = nilfs; nilfs_segbuf_prepare_write(segbuf, &wi); list_for_each_entry(bh, &segbuf->sb_segsum_buffers, b_assoc_buffers) { - res = nilfs_segbuf_submit_bh(segbuf, &wi, bh, rw); + res = nilfs_segbuf_submit_bh(segbuf, &wi, bh, REQ_OP_WRITE); if (unlikely(res)) goto failed_bio; } list_for_each_entry(bh, &segbuf->sb_payload_buffers, b_assoc_buffers) { - res = nilfs_segbuf_submit_bh(segbuf, &wi, bh, rw); + res = nilfs_segbuf_submit_bh(segbuf, &wi, bh, REQ_OP_WRITE); if (unlikely(res)) goto failed_bio; } @@ -488,8 +490,8 @@ static int nilfs_segbuf_write(struct nilfs_segment_buffer *segbuf, * Last BIO is always sent through the following * submission. */ - rw |= REQ_SYNC; - res = nilfs_segbuf_submit_bio(segbuf, &wi, rw); + res = nilfs_segbuf_submit_bio(segbuf, &wi, REQ_OP_WRITE, + REQ_SYNC); } failed_bio: -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:35 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D31CD29E14 for ; Tue, 5 Jan 2016 14:54:34 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 63BB3AC003 for ; Tue, 5 Jan 2016 12:54:34 -0800 (PST) X-ASG-Debug-ID: 1452027272-04cbb07e1a05460001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id p44sfcANbVXimBbU (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:33 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 885A0C0B7E27; Tue, 5 Jan 2016 20:54:32 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuG024032; Tue, 5 Jan 2016 15:54:29 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 17/35] ocfs2: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:20 -0600 X-ASG-Orig-Subj: [PATCH 17/35] ocfs2: set bi_op to REQ_OP Message-Id: <1452027218-32303-18-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027273 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has ocfs2 set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- fs/ocfs2/cluster/heartbeat.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c index 41039c2..38596aa 100644 --- a/fs/ocfs2/cluster/heartbeat.c +++ b/fs/ocfs2/cluster/heartbeat.c @@ -391,7 +391,8 @@ static void o2hb_bio_end_io(struct bio *bio) static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, struct o2hb_bio_wait_ctxt *wc, unsigned int *current_slot, - unsigned int max_slots, int rw) + unsigned int max_slots, int op, + int op_flags) { int len, current_page; unsigned int vec_len, vec_start; @@ -417,7 +418,8 @@ static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, bio->bi_bdev = reg->hr_bdev; bio->bi_private = wc; bio->bi_end_io = o2hb_bio_end_io; - bio->bi_rw |= rw; + bio->bi_op = op; + bio->bi_rw |= op_flags; vec_start = (cs << bits) % PAGE_CACHE_SIZE; while(cs < max_slots) { @@ -454,7 +456,7 @@ static int o2hb_read_slots(struct o2hb_region *reg, while(current_slot < max_slots) { bio = o2hb_setup_one_bio(reg, &wc, ¤t_slot, max_slots, - READ); + REQ_OP_READ, 0); if (IS_ERR(bio)) { status = PTR_ERR(bio); mlog_errno(status); @@ -486,7 +488,8 @@ static int o2hb_issue_node_write(struct o2hb_region *reg, slot = o2nm_this_node(); - bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1, WRITE_SYNC); + bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1, REQ_OP_WRITE, + WRITE_SYNC); if (IS_ERR(bio)) { status = PTR_ERR(bio); mlog_errno(status); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:37 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 2616329E42 for ; Tue, 5 Jan 2016 14:54:37 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 09F7B304039 for ; Tue, 5 Jan 2016 12:54:36 -0800 (PST) X-ASG-Debug-ID: 1452027275-04cb6c42e605270001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id YmO3mSDDalOho2xM (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:36 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 3B3543798DE; Tue, 5 Jan 2016 20:54:35 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuH024032; Tue, 5 Jan 2016 15:54:32 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 18/35] pm: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:21 -0600 X-ASG-Orig-Subj: [PATCH 18/35] pm: set bi_op to REQ_OP Message-Id: <1452027218-32303-19-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027275 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the pm swap code set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- kernel/power/swap.c | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 649dfc7..197076b 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -250,7 +250,7 @@ static void hib_end_io(struct bio *bio) bio_put(bio); } -static int hib_submit_io(int rw, pgoff_t page_off, void *addr, +static int hib_submit_io(int op, int op_flags, pgoff_t page_off, void *addr, struct hib_bio_batch *hb) { struct page *page = virt_to_page(addr); @@ -260,7 +260,8 @@ static int hib_submit_io(int rw, pgoff_t page_off, void *addr, bio = bio_alloc(__GFP_RECLAIM | __GFP_HIGH, 1); bio->bi_iter.bi_sector = page_off * (PAGE_SIZE >> 9); bio->bi_bdev = hib_resume_bdev; - bio->bi_rw |= rw; + bio->bi_op = op; + bio->bi_rw |= op_flags; if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) { printk(KERN_ERR "PM: Adding page to bio failed at %llu\n", @@ -296,7 +297,8 @@ static int mark_swapfiles(struct swap_map_handle *handle, unsigned int flags) { int error; - hib_submit_io(READ_SYNC, swsusp_resume_block, swsusp_header, NULL); + hib_submit_io(REQ_OP_READ, READ_SYNC, swsusp_resume_block, + swsusp_header, NULL); if (!memcmp("SWAP-SPACE",swsusp_header->sig, 10) || !memcmp("SWAPSPACE2",swsusp_header->sig, 10)) { memcpy(swsusp_header->orig_sig,swsusp_header->sig, 10); @@ -305,8 +307,8 @@ static int mark_swapfiles(struct swap_map_handle *handle, unsigned int flags) swsusp_header->flags = flags; if (flags & SF_CRC32_MODE) swsusp_header->crc32 = handle->crc32; - error = hib_submit_io(WRITE_SYNC, swsusp_resume_block, - swsusp_header, NULL); + error = hib_submit_io(REQ_OP_WRITE, WRITE_SYNC, + swsusp_resume_block, swsusp_header, NULL); } else { printk(KERN_ERR "PM: Swap header not found!\n"); error = -ENODEV; @@ -379,7 +381,7 @@ static int write_page(void *buf, sector_t offset, struct hib_bio_batch *hb) } else { src = buf; } - return hib_submit_io(WRITE_SYNC, offset, src, hb); + return hib_submit_io(REQ_OP_WRITE, WRITE_SYNC, offset, src, hb); } static void release_swap_writer(struct swap_map_handle *handle) @@ -982,7 +984,8 @@ static int get_swap_reader(struct swap_map_handle *handle, return -ENOMEM; } - error = hib_submit_io(READ_SYNC, offset, tmp->map, NULL); + error = hib_submit_io(REQ_OP_READ, READ_SYNC, offset, + tmp->map, NULL); if (error) { release_swap_reader(handle); return error; @@ -1006,7 +1009,7 @@ static int swap_read_page(struct swap_map_handle *handle, void *buf, offset = handle->cur->entries[handle->k]; if (!offset) return -EFAULT; - error = hib_submit_io(READ_SYNC, offset, buf, hb); + error = hib_submit_io(REQ_OP_READ, READ_SYNC, offset, buf, hb); if (error) return error; if (++handle->k >= MAP_PAGE_ENTRIES) { @@ -1508,7 +1511,8 @@ int swsusp_check(void) if (!IS_ERR(hib_resume_bdev)) { set_blocksize(hib_resume_bdev, PAGE_SIZE); clear_page(swsusp_header); - error = hib_submit_io(READ_SYNC, swsusp_resume_block, + error = hib_submit_io(REQ_OP_READ, READ_SYNC, + swsusp_resume_block, swsusp_header, NULL); if (error) goto put; @@ -1516,7 +1520,8 @@ int swsusp_check(void) if (!memcmp(HIBERNATE_SIG, swsusp_header->sig, 10)) { memcpy(swsusp_header->sig, swsusp_header->orig_sig, 10); /* Reset swap signature now */ - error = hib_submit_io(WRITE_SYNC, swsusp_resume_block, + error = hib_submit_io(REQ_OP_WRITE, WRITE_SYNC, + swsusp_resume_block, swsusp_header, NULL); } else { error = -EINVAL; @@ -1560,10 +1565,12 @@ int swsusp_unmark(void) { int error; - hib_submit_io(READ_SYNC, swsusp_resume_block, swsusp_header, NULL); + hib_submit_io(REQ_OP_READ, READ_SYNC, swsusp_resume_block, + swsusp_header, NULL); if (!memcmp(HIBERNATE_SIG,swsusp_header->sig, 10)) { memcpy(swsusp_header->sig,swsusp_header->orig_sig, 10); - error = hib_submit_io(WRITE_SYNC, swsusp_resume_block, + error = hib_submit_io(REQ_OP_WRITE, WRITE_SYNC, + swsusp_resume_block, swsusp_header, NULL); } else { printk(KERN_ERR "PM: Cannot find swsusp signature!\n"); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:42 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 1B9C429DF6 for ; Tue, 5 Jan 2016 14:54:42 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0E670304053 for ; Tue, 5 Jan 2016 12:54:41 -0800 (PST) X-ASG-Debug-ID: 1452027279-04cbb07e1905470001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id gk9xbD7veEtqfouK (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:39 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 358B9A4CD9; Tue, 5 Jan 2016 20:54:39 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuI024032; Tue, 5 Jan 2016 15:54:35 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 19/35] dm: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:22 -0600 X-ASG-Orig-Subj: [PATCH 19/35] dm: set bi_op to REQ_OP Message-Id: <1452027218-32303-20-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027279 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has dm set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. I did some basic dm tests, but I think this patch should be considered compile tested only. I have not tested all the dm targets and I did not stress every code path I have touched. Signed-off-by: Mike Christie --- drivers/md/dm-bufio.c | 8 +++--- drivers/md/dm-crypt.c | 1 + drivers/md/dm-io.c | 57 ++++++++++++++++++++++------------------- drivers/md/dm-kcopyd.c | 25 +++++++++--------- drivers/md/dm-log-writes.c | 6 ++--- drivers/md/dm-log.c | 5 ++-- drivers/md/dm-raid1.c | 11 +++++--- drivers/md/dm-snap-persistent.c | 24 +++++++++-------- drivers/md/dm-thin.c | 7 ++--- drivers/md/dm.c | 1 + include/linux/dm-io.h | 3 ++- 11 files changed, 82 insertions(+), 66 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index 1fd25bf..b6055f2 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -574,7 +574,8 @@ static void use_dmio(struct dm_buffer *b, int rw, sector_t block, { int r; struct dm_io_request io_req = { - .bi_rw = rw, + .bi_op = rw, + .bi_op_flags = 0, .notify.fn = dmio_complete, .notify.context = b, .client = b->c->dm_io, @@ -634,7 +635,7 @@ static void use_inline_bio(struct dm_buffer *b, int rw, sector_t block, * the dm_buffer's inline bio is local to bufio. */ b->bio.bi_private = end_io; - b->bio.bi_rw |= rw; + b->bio.bi_op = rw; /* * We assume that if len >= PAGE_SIZE ptr is page-aligned. @@ -1327,7 +1328,8 @@ EXPORT_SYMBOL_GPL(dm_bufio_write_dirty_buffers); int dm_bufio_issue_flush(struct dm_bufio_client *c) { struct dm_io_request io_req = { - .bi_rw = WRITE_FLUSH, + .bi_op = REQ_OP_WRITE, + .bi_op_flags = WRITE_FLUSH, .mem.type = DM_IO_KMEM, .mem.ptr.addr = NULL, .client = c->dm_io, diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 3147c8d..f466fec 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1133,6 +1133,7 @@ static void clone_init(struct dm_crypt_io *io, struct bio *clone) clone->bi_private = io; clone->bi_end_io = crypt_endio; clone->bi_bdev = cc->dev->bdev; + clone->bi_op = io->base_bio->bi_op; clone->bi_rw = io->base_bio->bi_rw; } diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c index 107d445..790b185 100644 --- a/drivers/md/dm-io.c +++ b/drivers/md/dm-io.c @@ -278,8 +278,9 @@ static void km_dp_init(struct dpages *dp, void *data) /*----------------------------------------------------------------- * IO routines that accept a list of pages. *---------------------------------------------------------------*/ -static void do_region(int rw, unsigned region, struct dm_io_region *where, - struct dpages *dp, struct io *io) +static void do_region(int op, int op_flags, unsigned region, + struct dm_io_region *where, struct dpages *dp, + struct io *io) { struct bio *bio; struct page *page; @@ -295,24 +296,25 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, /* * Reject unsupported discard and write same requests. */ - if (rw & REQ_DISCARD) + if (op == REQ_OP_DISCARD) special_cmd_max_sectors = q->limits.max_discard_sectors; - else if (rw & REQ_WRITE_SAME) + else if (op == REQ_OP_WRITE_SAME) special_cmd_max_sectors = q->limits.max_write_same_sectors; - if ((rw & (REQ_DISCARD | REQ_WRITE_SAME)) && special_cmd_max_sectors == 0) { + if ((op == REQ_OP_DISCARD || op == REQ_OP_WRITE_SAME) && + special_cmd_max_sectors == 0) { dec_count(io, region, -EOPNOTSUPP); return; } /* - * where->count may be zero if rw holds a flush and we need to + * where->count may be zero if op holds a flush and we need to * send a zero-sized flush. */ do { /* * Allocate a suitably sized-bio. */ - if ((rw & REQ_DISCARD) || (rw & REQ_WRITE_SAME)) + if ((op == REQ_OP_DISCARD) || (op == REQ_OP_WRITE_SAME)) num_bvecs = 1; else num_bvecs = min_t(int, BIO_MAX_PAGES, @@ -322,14 +324,15 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, bio->bi_iter.bi_sector = where->sector + (where->count - remaining); bio->bi_bdev = where->bdev; bio->bi_end_io = endio; - bio->bi_rw |= rw; + bio->bi_op = op; + bio->bi_rw |= op_flags; store_io_and_region_in_bio(bio, io, region); - if (rw & REQ_DISCARD) { + if (op == REQ_OP_DISCARD) { num_sectors = min_t(sector_t, special_cmd_max_sectors, remaining); bio->bi_iter.bi_size = num_sectors << SECTOR_SHIFT; remaining -= num_sectors; - } else if (rw & REQ_WRITE_SAME) { + } else if (op == REQ_OP_WRITE_SAME) { /* * WRITE SAME only uses a single page. */ @@ -360,7 +363,7 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, } while (remaining); } -static void dispatch_io(int rw, unsigned int num_regions, +static void dispatch_io(int op, int op_flags, unsigned int num_regions, struct dm_io_region *where, struct dpages *dp, struct io *io, int sync) { @@ -370,7 +373,7 @@ static void dispatch_io(int rw, unsigned int num_regions, BUG_ON(num_regions > DM_IO_MAX_REGIONS); if (sync) - rw |= REQ_SYNC; + op_flags |= REQ_SYNC; /* * For multiple regions we need to be careful to rewind @@ -378,8 +381,8 @@ static void dispatch_io(int rw, unsigned int num_regions, */ for (i = 0; i < num_regions; i++) { *dp = old_pages; - if (where[i].count || (rw & REQ_FLUSH)) - do_region(rw, i, where + i, dp, io); + if (where[i].count || (op_flags & REQ_FLUSH)) + do_region(op, op_flags, i, where + i, dp, io); } /* @@ -403,13 +406,13 @@ static void sync_io_complete(unsigned long error, void *context) } static int sync_io(struct dm_io_client *client, unsigned int num_regions, - struct dm_io_region *where, int rw, struct dpages *dp, - unsigned long *error_bits) + struct dm_io_region *where, int op, int op_flags, + struct dpages *dp, unsigned long *error_bits) { struct io *io; struct sync_io sio; - if (num_regions > 1 && (rw & RW_MASK) != WRITE) { + if (num_regions > 1 && !op_is_write(op)) { WARN_ON(1); return -EIO; } @@ -426,7 +429,7 @@ static int sync_io(struct dm_io_client *client, unsigned int num_regions, io->vma_invalidate_address = dp->vma_invalidate_address; io->vma_invalidate_size = dp->vma_invalidate_size; - dispatch_io(rw, num_regions, where, dp, io, 1); + dispatch_io(op, op_flags, num_regions, where, dp, io, 1); wait_for_completion_io(&sio.wait); @@ -437,12 +440,12 @@ static int sync_io(struct dm_io_client *client, unsigned int num_regions, } static int async_io(struct dm_io_client *client, unsigned int num_regions, - struct dm_io_region *where, int rw, struct dpages *dp, - io_notify_fn fn, void *context) + struct dm_io_region *where, int op, int op_flags, + struct dpages *dp, io_notify_fn fn, void *context) { struct io *io; - if (num_regions > 1 && (rw & RW_MASK) != WRITE) { + if (num_regions > 1 && !op_is_write(op)) { WARN_ON(1); fn(1, context); return -EIO; @@ -458,7 +461,7 @@ static int async_io(struct dm_io_client *client, unsigned int num_regions, io->vma_invalidate_address = dp->vma_invalidate_address; io->vma_invalidate_size = dp->vma_invalidate_size; - dispatch_io(rw, num_regions, where, dp, io, 0); + dispatch_io(op, op_flags, num_regions, where, dp, io, 0); return 0; } @@ -481,7 +484,7 @@ static int dp_init(struct dm_io_request *io_req, struct dpages *dp, case DM_IO_VMA: flush_kernel_vmap_range(io_req->mem.ptr.vma, size); - if ((io_req->bi_rw & RW_MASK) == READ) { + if (io_req->bi_op == REQ_OP_READ) { dp->vma_invalidate_address = io_req->mem.ptr.vma; dp->vma_invalidate_size = size; } @@ -519,10 +522,12 @@ int dm_io(struct dm_io_request *io_req, unsigned num_regions, if (!io_req->notify.fn) return sync_io(io_req->client, num_regions, where, - io_req->bi_rw, &dp, sync_error_bits); + io_req->bi_op, io_req->bi_op_flags, &dp, + sync_error_bits); - return async_io(io_req->client, num_regions, where, io_req->bi_rw, - &dp, io_req->notify.fn, io_req->notify.context); + return async_io(io_req->client, num_regions, where, io_req->bi_op, + io_req->bi_op_flags, &dp, io_req->notify.fn, + io_req->notify.context); } EXPORT_SYMBOL(dm_io); diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c index 1452ed9..4182bc7d 100644 --- a/drivers/md/dm-kcopyd.c +++ b/drivers/md/dm-kcopyd.c @@ -465,10 +465,10 @@ static void complete_io(unsigned long error, void *context) io_job_finish(kc->throttle); if (error) { - if (job->rw & WRITE) - job->write_err |= error; - else + if (job->rw == READ) job->read_err = 1; + else + job->write_err |= error; if (!test_bit(DM_KCOPYD_IGNORE_ERROR, &job->flags)) { push(&kc->complete_jobs, job); @@ -477,13 +477,11 @@ static void complete_io(unsigned long error, void *context) } } - if (job->rw & WRITE) - push(&kc->complete_jobs, job); - - else { + if (job->rw == READ) { job->rw = WRITE; push(&kc->io_jobs, job); - } + } else + push(&kc->complete_jobs, job); wake(kc); } @@ -496,7 +494,8 @@ static int run_io_job(struct kcopyd_job *job) { int r; struct dm_io_request io_req = { - .bi_rw = job->rw, + .bi_op = job->rw, + .bi_op_flags = 0, .mem.type = DM_IO_PAGE_LIST, .mem.ptr.pl = job->pages, .mem.offset = 0, @@ -550,10 +549,10 @@ static int process_jobs(struct list_head *jobs, struct dm_kcopyd_client *kc, if (r < 0) { /* error this rogue job */ - if (job->rw & WRITE) - job->write_err = (unsigned long) -1L; - else + if (job->rw == READ) job->read_err = 1; + else + job->write_err = (unsigned long) -1L; push(&kc->complete_jobs, job); break; } @@ -734,7 +733,7 @@ int dm_kcopyd_copy(struct dm_kcopyd_client *kc, struct dm_io_region *from, /* * Use WRITE SAME to optimize zeroing if all dests support it. */ - job->rw = WRITE | REQ_WRITE_SAME; + job->rw = REQ_OP_WRITE_SAME; for (i = 0; i < job->num_dests; i++) if (!bdev_write_same(job->dests[i].bdev)) { job->rw = WRITE; diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c index 21760e3..f63bbf4 100644 --- a/drivers/md/dm-log-writes.c +++ b/drivers/md/dm-log-writes.c @@ -205,7 +205,7 @@ static int write_metadata(struct log_writes_c *lc, void *entry, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; page = alloc_page(GFP_KERNEL); if (!page) { @@ -270,7 +270,7 @@ static int log_one_block(struct log_writes_c *lc, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; for (i = 0; i < block->vec_cnt; i++) { /* @@ -292,7 +292,7 @@ static int log_one_block(struct log_writes_c *lc, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; - bio->bi_rw |= WRITE; + bio->bi_op = REQ_OP_WRITE; ret = bio_add_page(bio, block->vecs[i].bv_page, block->vecs[i].bv_len, 0); diff --git a/drivers/md/dm-log.c b/drivers/md/dm-log.c index 627d191..4ca2d1d 100644 --- a/drivers/md/dm-log.c +++ b/drivers/md/dm-log.c @@ -293,7 +293,7 @@ static void header_from_disk(struct log_header_core *core, struct log_header_dis static int rw_header(struct log_c *lc, int rw) { - lc->io_req.bi_rw = rw; + lc->io_req.bi_op = rw; return dm_io(&lc->io_req, 1, &lc->header_location, NULL); } @@ -306,7 +306,8 @@ static int flush_header(struct log_c *lc) .count = 0, }; - lc->io_req.bi_rw = WRITE_FLUSH; + lc->io_req.bi_op = REQ_OP_WRITE; + lc->io_req.bi_op_flags = WRITE_FLUSH; return dm_io(&lc->io_req, 1, &null_location, NULL); } diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c index f2a363a..6b61b62 100644 --- a/drivers/md/dm-raid1.c +++ b/drivers/md/dm-raid1.c @@ -260,7 +260,8 @@ static int mirror_flush(struct dm_target *ti) struct dm_io_region io[ms->nr_mirrors]; struct mirror *m; struct dm_io_request io_req = { - .bi_rw = WRITE_FLUSH, + .bi_op = REQ_OP_WRITE, + .bi_op_flags = WRITE_FLUSH, .mem.type = DM_IO_KMEM, .mem.ptr.addr = NULL, .client = ms->io_client, @@ -541,7 +542,8 @@ static void read_async_bio(struct mirror *m, struct bio *bio) { struct dm_io_region io; struct dm_io_request io_req = { - .bi_rw = READ, + .bi_op = REQ_OP_READ, + .bi_op_flags = 0, .mem.type = DM_IO_BIO, .mem.ptr.bio = bio, .notify.fn = read_callback, @@ -654,7 +656,8 @@ static void do_write(struct mirror_set *ms, struct bio *bio) struct dm_io_region io[ms->nr_mirrors], *dest = io; struct mirror *m; struct dm_io_request io_req = { - .bi_rw = WRITE | (bio->bi_rw & WRITE_FLUSH_FUA), + .bi_op = REQ_OP_WRITE, + .bi_op_flags = bio->bi_rw & WRITE_FLUSH_FUA, .mem.type = DM_IO_BIO, .mem.ptr.bio = bio, .notify.fn = write_callback, @@ -663,7 +666,7 @@ static void do_write(struct mirror_set *ms, struct bio *bio) }; if (bio->bi_rw & REQ_DISCARD) { - io_req.bi_rw |= REQ_DISCARD; + io_req.bi_op = REQ_OP_DISCARD; io_req.mem.type = DM_IO_KMEM; io_req.mem.ptr.addr = NULL; } diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persistent.c index 3164b8b..91d4829 100644 --- a/drivers/md/dm-snap-persistent.c +++ b/drivers/md/dm-snap-persistent.c @@ -226,8 +226,8 @@ static void do_metadata(struct work_struct *work) /* * Read or write a chunk aligned and sized block of data from a device. */ -static int chunk_io(struct pstore *ps, void *area, chunk_t chunk, int rw, - int metadata) +static int chunk_io(struct pstore *ps, void *area, chunk_t chunk, int op, + int op_flags, int metadata) { struct dm_io_region where = { .bdev = dm_snap_cow(ps->store->snap)->bdev, @@ -235,7 +235,8 @@ static int chunk_io(struct pstore *ps, void *area, chunk_t chunk, int rw, .count = ps->store->chunk_size, }; struct dm_io_request io_req = { - .bi_rw = rw, + .bi_op = op, + .bi_op_flags = op_flags, .mem.type = DM_IO_VMA, .mem.ptr.vma = area, .client = ps->io_client, @@ -281,14 +282,14 @@ static void skip_metadata(struct pstore *ps) * Read or write a metadata area. Remembering to skip the first * chunk which holds the header. */ -static int area_io(struct pstore *ps, int rw) +static int area_io(struct pstore *ps, int op, int op_flags) { int r; chunk_t chunk; chunk = area_location(ps, ps->current_area); - r = chunk_io(ps, ps->area, chunk, rw, 0); + r = chunk_io(ps, ps->area, chunk, op, op_flags, 0); if (r) return r; @@ -302,7 +303,8 @@ static void zero_memory_area(struct pstore *ps) static int zero_disk_area(struct pstore *ps, chunk_t area) { - return chunk_io(ps, ps->zero_area, area_location(ps, area), WRITE, 0); + return chunk_io(ps, ps->zero_area, area_location(ps, area), + REQ_OP_WRITE, 0, 0); } static int read_header(struct pstore *ps, int *new_snapshot) @@ -334,7 +336,7 @@ static int read_header(struct pstore *ps, int *new_snapshot) if (r) return r; - r = chunk_io(ps, ps->header_area, 0, READ, 1); + r = chunk_io(ps, ps->header_area, 0, REQ_OP_READ, 0, 1); if (r) goto bad; @@ -395,7 +397,7 @@ static int write_header(struct pstore *ps) dh->version = cpu_to_le32(ps->version); dh->chunk_size = cpu_to_le32(ps->store->chunk_size); - return chunk_io(ps, ps->header_area, 0, WRITE, 1); + return chunk_io(ps, ps->header_area, 0, REQ_OP_WRITE, 0, 1); } /* @@ -736,7 +738,7 @@ static void persistent_commit_exception(struct dm_exception_store *store, /* * Commit exceptions to disk. */ - if (ps->valid && area_io(ps, WRITE_FLUSH_FUA)) + if (ps->valid && area_io(ps, REQ_OP_WRITE, WRITE_FLUSH_FUA)) ps->valid = 0; /* @@ -776,7 +778,7 @@ static int persistent_prepare_merge(struct dm_exception_store *store, return 0; ps->current_area--; - r = area_io(ps, READ); + r = area_io(ps, REQ_OP_READ, 0); if (r < 0) return r; ps->current_committed = ps->exceptions_per_area; @@ -813,7 +815,7 @@ static int persistent_commit_merge(struct dm_exception_store *store, for (i = 0; i < nr_merged; i++) clear_exception(ps, ps->current_committed - 1 - i); - r = area_io(ps, WRITE_FLUSH_FUA); + r = area_io(ps, REQ_OP_WRITE, WRITE_FLUSH_FUA); if (r < 0) return r; diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index 8017667..f870878 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -338,7 +338,7 @@ static int __blkdev_issue_discard_async(struct block_device *bdev, sector_t sect struct bio *parent_bio) { struct request_queue *q = bdev_get_queue(bdev); - int type = REQ_WRITE | REQ_DISCARD; + int op_flags = 0; struct bio *bio; if (!q || !nr_sects) @@ -350,7 +350,7 @@ static int __blkdev_issue_discard_async(struct block_device *bdev, sector_t sect if (flags & BLKDEV_DISCARD_SECURE) { if (!blk_queue_secdiscard(q)) return -EOPNOTSUPP; - type |= REQ_SECURE; + op_flags |= REQ_SECURE; } /* @@ -365,7 +365,8 @@ static int __blkdev_issue_discard_async(struct block_device *bdev, sector_t sect bio->bi_iter.bi_sector = sector; bio->bi_bdev = bdev; bio->bi_iter.bi_size = nr_sects << 9; - bio->bi_rw |= type; + bio->bi_op = REQ_OP_DISCARD; + bio->bi_rw |= op_flags; submit_bio(bio); diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 5df4048..1777c9c 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -2358,6 +2358,7 @@ static struct mapped_device *alloc_dev(int minor) bio_init(&md->flush_bio); md->flush_bio.bi_bdev = md->bdev; + md->flush_bio.bi_op = REQ_OP_WRITE; md->flush_bio.bi_rw = WRITE_FLUSH; dm_stats_init(&md->stats); diff --git a/include/linux/dm-io.h b/include/linux/dm-io.h index a68cbe5..b91b023 100644 --- a/include/linux/dm-io.h +++ b/include/linux/dm-io.h @@ -57,7 +57,8 @@ struct dm_io_notify { */ struct dm_io_client; struct dm_io_request { - int bi_rw; /* READ|WRITE - not READA */ + int bi_op; /* REQ_OP */ + int bi_op_flags; /* rq_flag_bits */ struct dm_io_memory mem; /* Memory to use for io */ struct dm_io_notify notify; /* Synchronous if notify.fn is NULL */ struct dm_io_client *client; /* Client memory handler */ -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:43 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9CDFF29E3F for ; Tue, 5 Jan 2016 14:54:43 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 800C88F804C for ; Tue, 5 Jan 2016 12:54:43 -0800 (PST) X-ASG-Debug-ID: 1452027282-04cbb07e1605470001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id cndCZUt19Y1xJuml (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:42 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id DDD937AEA0; Tue, 5 Jan 2016 20:54:41 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuJ024032; Tue, 5 Jan 2016 15:54:39 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 20/35] dm: pass dm stats data dir instead of bi_rw Date: Tue, 5 Jan 2016 14:53:23 -0600 X-ASG-Orig-Subj: [PATCH 20/35] dm: pass dm stats data dir instead of bi_rw Message-Id: <1452027218-32303-21-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027282 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie It looks like dm stats cares about the data direction (READ vs WRITE) and does not need the bio/request flags. Commands like REQ_FLUSH, REQ_DISCARD and REQ_WRITE_SAME are currently always set with REQ_WRITE, so the extra check for REQ_DISCARD in dm_stats_account_io is not needed. This patch has it use the bio and request data_dir helpers instead of accessing the bi_rw/cmd_flags directly. This makes the next patches that remove the operation from the cmd_flags and bi_rw easier, because we will no longer have the REQ_WRITE bit set for operations like discards. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/md/dm-stats.c | 6 +++--- drivers/md/dm.c | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/md/dm-stats.c b/drivers/md/dm-stats.c index 8289804..96b5c1b 100644 --- a/drivers/md/dm-stats.c +++ b/drivers/md/dm-stats.c @@ -518,7 +518,7 @@ static void dm_stat_for_entry(struct dm_stat *s, size_t entry, struct dm_stats_aux *stats_aux, bool end, unsigned long duration_jiffies) { - unsigned long idx = bi_rw & REQ_WRITE; + unsigned long idx = bi_rw; struct dm_stat_shared *shared = &s->stat_shared[entry]; struct dm_stat_percpu *p; @@ -645,8 +645,8 @@ void dm_stats_account_io(struct dm_stats *stats, unsigned long bi_rw, last = raw_cpu_ptr(stats->last); stats_aux->merged = (bi_sector == (ACCESS_ONCE(last->last_sector) && - ((bi_rw & (REQ_WRITE | REQ_DISCARD)) == - (ACCESS_ONCE(last->last_rw) & (REQ_WRITE | REQ_DISCARD))) + ((bi_rw == WRITE) == + (ACCESS_ONCE(last->last_rw) == WRITE)) )); ACCESS_ONCE(last->last_sector) = end_sector; ACCESS_ONCE(last->last_rw) = bi_rw; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 1777c9c..5dbdae7 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -686,7 +686,7 @@ static void start_io_acct(struct dm_io *io) atomic_inc_return(&md->pending[rw])); if (unlikely(dm_stats_used(&md->stats))) - dm_stats_account_io(&md->stats, bio->bi_rw, bio->bi_iter.bi_sector, + dm_stats_account_io(&md->stats, bio_data_dir(bio), bio->bi_iter.bi_sector, bio_sectors(bio), false, 0, &io->stats_aux); } @@ -701,7 +701,7 @@ static void end_io_acct(struct dm_io *io) generic_end_io_acct(rw, &dm_disk(md)->part0, io->start_time); if (unlikely(dm_stats_used(&md->stats))) - dm_stats_account_io(&md->stats, bio->bi_rw, bio->bi_iter.bi_sector, + dm_stats_account_io(&md->stats, bio_data_dir(bio), bio->bi_iter.bi_sector, bio_sectors(bio), true, duration, &io->stats_aux); /* @@ -1084,7 +1084,7 @@ static void rq_end_stats(struct mapped_device *md, struct request *orig) if (unlikely(dm_stats_used(&md->stats))) { struct dm_rq_target_io *tio = tio_from_request(orig); tio->duration_jiffies = jiffies - tio->duration_jiffies; - dm_stats_account_io(&md->stats, orig->cmd_flags, blk_rq_pos(orig), + dm_stats_account_io(&md->stats, rq_data_dir(orig), blk_rq_pos(orig), tio->n_sectors, true, tio->duration_jiffies, &tio->stats_aux); } @@ -2017,7 +2017,7 @@ static void dm_start_request(struct mapped_device *md, struct request *orig) struct dm_rq_target_io *tio = tio_from_request(orig); tio->duration_jiffies = jiffies; tio->n_sectors = blk_rq_sectors(orig); - dm_stats_account_io(&md->stats, orig->cmd_flags, blk_rq_pos(orig), + dm_stats_account_io(&md->stats, rq_data_dir(orig), blk_rq_pos(orig), tio->n_sectors, false, 0, &tio->stats_aux); } -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:47 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 6C75E29DFA for ; Tue, 5 Jan 2016 14:54:47 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id E6D19AC001 for ; Tue, 5 Jan 2016 12:54:46 -0800 (PST) X-ASG-Debug-ID: 1452027284-04cbb07e1905480001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id SBe3aNs1vY7A8BJn (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:45 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 9BBD1C1090A3; Tue, 5 Jan 2016 20:54:44 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuK024032; Tue, 5 Jan 2016 15:54:42 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 21/35] bcache: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:24 -0600 X-ASG-Orig-Subj: [PATCH 21/35] bcache: set bi_op to REQ_OP Message-Id: <1452027218-32303-22-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027285 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has bcache set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/md/bcache/btree.c | 2 ++ drivers/md/bcache/debug.c | 2 ++ drivers/md/bcache/io.c | 2 +- drivers/md/bcache/journal.c | 7 ++++--- drivers/md/bcache/movinggc.c | 2 +- drivers/md/bcache/request.c | 9 +++++---- drivers/md/bcache/super.c | 26 +++++++++++++++----------- drivers/md/bcache/writeback.c | 4 ++-- 8 files changed, 32 insertions(+), 22 deletions(-) diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c index 22b9e34..752a44f 100644 --- a/drivers/md/bcache/btree.c +++ b/drivers/md/bcache/btree.c @@ -295,6 +295,7 @@ static void bch_btree_node_read(struct btree *b) closure_init_stack(&cl); bio = bch_bbio_alloc(b->c); + bio->bi_op = REQ_OP_READ; bio->bi_rw = REQ_META|READ_SYNC; bio->bi_iter.bi_size = KEY_SIZE(&b->key) << 9; bio->bi_end_io = btree_node_read_endio; @@ -397,6 +398,7 @@ static void do_btree_node_write(struct btree *b) b->bio->bi_end_io = btree_node_write_endio; b->bio->bi_private = cl; + b->bio->bi_op = REQ_OP_WRITE; b->bio->bi_rw = REQ_META|WRITE_SYNC|REQ_FUA; b->bio->bi_iter.bi_size = roundup(set_bytes(i), block_bytes(b->c)); bch_bio_map(b->bio, i); diff --git a/drivers/md/bcache/debug.c b/drivers/md/bcache/debug.c index db68562..4c48783 100644 --- a/drivers/md/bcache/debug.c +++ b/drivers/md/bcache/debug.c @@ -52,6 +52,7 @@ void bch_btree_verify(struct btree *b) bio->bi_bdev = PTR_CACHE(b->c, &b->key, 0)->bdev; bio->bi_iter.bi_sector = PTR_OFFSET(&b->key, 0); bio->bi_iter.bi_size = KEY_SIZE(&v->key) << 9; + bio->bi_op = REQ_OP_READ; bio->bi_rw |= REQ_META|READ_SYNC; bch_bio_map(bio, sorted); @@ -114,6 +115,7 @@ void bch_data_verify(struct cached_dev *dc, struct bio *bio) check = bio_clone(bio, GFP_NOIO); if (!check) return; + check->bi_op = REQ_OP_READ; check->bi_rw |= READ_SYNC; if (bio_alloc_pages(check, GFP_NOIO)) diff --git a/drivers/md/bcache/io.c b/drivers/md/bcache/io.c index 86a0bb8..f10a9a0 100644 --- a/drivers/md/bcache/io.c +++ b/drivers/md/bcache/io.c @@ -111,7 +111,7 @@ void bch_bbio_count_io_errors(struct cache_set *c, struct bio *bio, struct bbio *b = container_of(bio, struct bbio, bio); struct cache *ca = PTR_CACHE(c, &b->key, 0); - unsigned threshold = bio->bi_rw & REQ_WRITE + unsigned threshold = op_is_write(bio->bi_op) ? c->congested_write_threshold_us : c->congested_read_threshold_us; diff --git a/drivers/md/bcache/journal.c b/drivers/md/bcache/journal.c index af3f9f7..68fa0f0 100644 --- a/drivers/md/bcache/journal.c +++ b/drivers/md/bcache/journal.c @@ -54,7 +54,7 @@ reread: left = ca->sb.bucket_size - offset; bio_reset(bio); bio->bi_iter.bi_sector = bucket + offset; bio->bi_bdev = ca->bdev; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_iter.bi_size = len << 9; bio->bi_end_io = journal_read_endio; @@ -452,7 +452,7 @@ static void do_journal_discard(struct cache *ca) bio->bi_iter.bi_sector = bucket_to_sector(ca->set, ca->sb.d[ja->discard_idx]); bio->bi_bdev = ca->bdev; - bio->bi_rw = REQ_WRITE|REQ_DISCARD; + bio->bi_op = REQ_OP_DISCARD; bio->bi_max_vecs = 1; bio->bi_io_vec = bio->bi_inline_vecs; bio->bi_iter.bi_size = bucket_bytes(ca); @@ -626,7 +626,8 @@ static void journal_write_unlocked(struct closure *cl) bio_reset(bio); bio->bi_iter.bi_sector = PTR_OFFSET(k, i); bio->bi_bdev = ca->bdev; - bio->bi_rw = REQ_WRITE|REQ_SYNC|REQ_META|REQ_FLUSH|REQ_FUA; + bio->bi_op = REQ_OP_WRITE; + bio->bi_rw = REQ_SYNC|REQ_META|REQ_FLUSH|REQ_FUA; bio->bi_iter.bi_size = sectors << 9; bio->bi_end_io = journal_write_endio; diff --git a/drivers/md/bcache/movinggc.c b/drivers/md/bcache/movinggc.c index b929fc9..f33860a 100644 --- a/drivers/md/bcache/movinggc.c +++ b/drivers/md/bcache/movinggc.c @@ -163,7 +163,7 @@ static void read_moving(struct cache_set *c) moving_init(io); bio = &io->bio.bio; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_end_io = read_moving_endio; if (bio_alloc_pages(bio, GFP_KERNEL)) diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index 25fa844..cd6837c 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -253,7 +253,7 @@ static void bch_data_insert_start(struct closure *cl) trace_bcache_cache_insert(k); bch_keylist_push(&op->insert_keys); - n->bi_rw |= REQ_WRITE; + n->bi_op = REQ_OP_WRITE; bch_submit_bbio(n, op->c, k, 0); } while (n != bio); @@ -383,7 +383,7 @@ static bool check_should_bypass(struct cached_dev *dc, struct bio *bio) if (mode == CACHE_MODE_NONE || (mode == CACHE_MODE_WRITEAROUND && - (bio->bi_rw & REQ_WRITE))) + op_is_write(bio->bi_op))) goto skip; if (bio->bi_iter.bi_sector & (c->sb.block_size - 1) || @@ -404,7 +404,7 @@ static bool check_should_bypass(struct cached_dev *dc, struct bio *bio) if (!congested && mode == CACHE_MODE_WRITEBACK && - (bio->bi_rw & REQ_WRITE) && + op_is_write(bio->bi_op) && (bio->bi_rw & REQ_SYNC)) goto rescale; @@ -657,7 +657,7 @@ static inline struct search *search_alloc(struct bio *bio, s->cache_miss = NULL; s->d = d; s->recoverable = 1; - s->write = (bio->bi_rw & REQ_WRITE) != 0; + s->write = op_is_write(bio->bi_op); s->read_dirty_data = 0; s->start_time = jiffies; @@ -925,6 +925,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) struct bio *flush = bio_alloc_bioset(GFP_NOIO, 0, dc->disk.bio_split); + flush->bi_op = REQ_OP_WRITE; flush->bi_rw = WRITE_FLUSH; flush->bi_bdev = bio->bi_bdev; flush->bi_end_io = request_endio; diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index 9e2eac8..b11f7f4 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -212,7 +212,8 @@ static void __write_super(struct cache_sb *sb, struct bio *bio) unsigned i; bio->bi_iter.bi_sector = SB_SECTOR; - bio->bi_rw = REQ_WRITE|REQ_SYNC|REQ_META; + bio->bi_op = REQ_OP_WRITE; + bio->bi_rw = REQ_SYNC|REQ_META; bio->bi_iter.bi_size = SB_SIZE; bch_bio_map(bio, NULL); @@ -333,7 +334,7 @@ static void uuid_io_unlock(struct closure *cl) up(&c->uuid_write_mutex); } -static void uuid_io(struct cache_set *c, unsigned long rw, +static void uuid_io(struct cache_set *c, int op, unsigned long op_flags, struct bkey *k, struct closure *parent) { struct closure *cl = &c->uuid_write; @@ -348,7 +349,8 @@ static void uuid_io(struct cache_set *c, unsigned long rw, for (i = 0; i < KEY_PTRS(k); i++) { struct bio *bio = bch_bbio_alloc(c); - bio->bi_rw = REQ_SYNC|REQ_META|rw; + bio->bi_op = op; + bio->bi_rw = REQ_SYNC|REQ_META|op_flags; bio->bi_iter.bi_size = KEY_SIZE(k) << 9; bio->bi_end_io = uuid_endio; @@ -357,12 +359,12 @@ static void uuid_io(struct cache_set *c, unsigned long rw, bch_submit_bbio(bio, c, k, i); - if (!(rw & WRITE)) + if (op != REQ_OP_WRITE) break; } bch_extent_to_text(buf, sizeof(buf), k); - pr_debug("%s UUIDs at %s", rw & REQ_WRITE ? "wrote" : "read", buf); + pr_debug("%s UUIDs at %s", op == REQ_OP_WRITE ? "wrote" : "read", buf); for (u = c->uuids; u < c->uuids + c->nr_uuids; u++) if (!bch_is_zero(u->uuid, 16)) @@ -381,7 +383,7 @@ static char *uuid_read(struct cache_set *c, struct jset *j, struct closure *cl) return "bad uuid pointer"; bkey_copy(&c->uuid_bucket, k); - uuid_io(c, READ_SYNC, k, cl); + uuid_io(c, REQ_OP_READ, READ_SYNC, k, cl); if (j->version < BCACHE_JSET_VERSION_UUIDv1) { struct uuid_entry_v0 *u0 = (void *) c->uuids; @@ -426,7 +428,7 @@ static int __uuid_write(struct cache_set *c) return 1; SET_KEY_SIZE(&k.key, c->sb.bucket_size); - uuid_io(c, REQ_WRITE, &k.key, &cl); + uuid_io(c, REQ_OP_WRITE, 0, &k.key, &cl); closure_sync(&cl); bkey_copy(&c->uuid_bucket, &k.key); @@ -498,7 +500,8 @@ static void prio_endio(struct bio *bio) closure_put(&ca->prio); } -static void prio_io(struct cache *ca, uint64_t bucket, unsigned long rw) +static void prio_io(struct cache *ca, uint64_t bucket, int op, + unsigned long op_flags) { struct closure *cl = &ca->prio; struct bio *bio = bch_bbio_alloc(ca->set); @@ -507,7 +510,8 @@ static void prio_io(struct cache *ca, uint64_t bucket, unsigned long rw) bio->bi_iter.bi_sector = bucket * ca->sb.bucket_size; bio->bi_bdev = ca->bdev; - bio->bi_rw = REQ_SYNC|REQ_META|rw; + bio->bi_op = op; + bio->bi_rw = REQ_SYNC|REQ_META|op_flags; bio->bi_iter.bi_size = bucket_bytes(ca); bio->bi_end_io = prio_endio; @@ -557,7 +561,7 @@ void bch_prio_write(struct cache *ca) BUG_ON(bucket == -1); mutex_unlock(&ca->set->bucket_lock); - prio_io(ca, bucket, REQ_WRITE); + prio_io(ca, bucket, REQ_OP_WRITE, 0); mutex_lock(&ca->set->bucket_lock); ca->prio_buckets[i] = bucket; @@ -599,7 +603,7 @@ static void prio_read(struct cache *ca, uint64_t bucket) ca->prio_last_buckets[bucket_nr] = bucket; bucket_nr++; - prio_io(ca, bucket, READ_SYNC); + prio_io(ca, bucket, REQ_OP_READ, READ_SYNC); if (p->csum != bch_crc64(&p->magic, bucket_bytes(ca) - 8)) pr_warn("bad csum reading priorities"); diff --git a/drivers/md/bcache/writeback.c b/drivers/md/bcache/writeback.c index b9346cd..96ddf28 100644 --- a/drivers/md/bcache/writeback.c +++ b/drivers/md/bcache/writeback.c @@ -183,7 +183,7 @@ static void write_dirty(struct closure *cl) struct keybuf_key *w = io->bio.bi_private; dirty_init(w); - io->bio.bi_rw = WRITE; + io->bio.bi_op = REQ_OP_WRITE; io->bio.bi_iter.bi_sector = KEY_START(&w->key); io->bio.bi_bdev = io->dc->bdev; io->bio.bi_end_io = dirty_endio; @@ -256,7 +256,7 @@ static void read_dirty(struct cached_dev *dc) io->bio.bi_iter.bi_sector = PTR_OFFSET(&w->key, 0); io->bio.bi_bdev = PTR_CACHE(dc->disk.c, &w->key, 0)->bdev; - io->bio.bi_rw = READ; + io->bio.bi_op = REQ_OP_READ; io->bio.bi_end_io = read_dirty_endio; if (bio_alloc_pages(&io->bio, GFP_KERNEL)) -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:49 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 74C2629E4C for ; Tue, 5 Jan 2016 14:54:49 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 17D1DAC005 for ; Tue, 5 Jan 2016 12:54:49 -0800 (PST) X-ASG-Debug-ID: 1452027287-04bdf03afa04e30001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id zp17uQDrdx7hY8x3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:47 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 5C3DC2197; Tue, 5 Jan 2016 20:54:47 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuL024032; Tue, 5 Jan 2016 15:54:44 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 22/35] drbd: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:25 -0600 X-ASG-Orig-Subj: [PATCH 22/35] drbd: set bi_op to REQ_OP Message-Id: <1452027218-32303-23-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027287 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has drbd set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. Lars and Philip, I might have split this patch up a little weird. Thisi patch handles setting up the bio, and then patch 30 (0030-block-fs-drivers-do-not-test-bi_rw-for-REQ_OPs.patch) handles where were check/read bio->bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/block/drbd/drbd_actlog.c | 29 ++++++++++++++++------------- drivers/block/drbd/drbd_bitmap.c | 6 +++--- drivers/block/drbd/drbd_int.h | 4 ++-- drivers/block/drbd/drbd_main.c | 5 +++-- drivers/block/drbd/drbd_receiver.c | 37 +++++++++++++++++++++++++------------ drivers/block/drbd/drbd_worker.c | 3 ++- 6 files changed, 51 insertions(+), 33 deletions(-) diff --git a/drivers/block/drbd/drbd_actlog.c b/drivers/block/drbd/drbd_actlog.c index 6069e15..2fa8534 100644 --- a/drivers/block/drbd/drbd_actlog.c +++ b/drivers/block/drbd/drbd_actlog.c @@ -137,19 +137,19 @@ void wait_until_done_or_force_detached(struct drbd_device *device, struct drbd_b static int _drbd_md_sync_page_io(struct drbd_device *device, struct drbd_backing_dev *bdev, - sector_t sector, int rw) + sector_t sector, int op) { struct bio *bio; /* we do all our meta data IO in aligned 4k blocks. */ const int size = 4096; - int err; + int err, op_flags = 0; device->md_io.done = 0; device->md_io.error = -ENODEV; - if ((rw & WRITE) && !test_bit(MD_NO_FUA, &device->flags)) - rw |= REQ_FUA | REQ_FLUSH; - rw |= REQ_SYNC | REQ_NOIDLE; + if ((op == REQ_OP_WRITE) && !test_bit(MD_NO_FUA, &device->flags)) + op_flags |= REQ_FUA | REQ_FLUSH; + op_flags |= REQ_SYNC | REQ_NOIDLE; bio = bio_alloc_drbd(GFP_NOIO); bio->bi_bdev = bdev->md_bdev; @@ -159,9 +159,10 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, goto out; bio->bi_private = device; bio->bi_end_io = drbd_md_endio; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; - if (!(rw & WRITE) && device->state.disk == D_DISKLESS && device->ldev == NULL) + if (op != REQ_OP_WRITE && device->state.disk == D_DISKLESS && device->ldev == NULL) /* special case, drbd_md_read() during drbd_adm_attach(): no get_ldev */ ; else if (!get_ldev_if_state(device, D_ATTACHING)) { @@ -174,7 +175,7 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, bio_get(bio); /* one bio_put() is in the completion handler */ atomic_inc(&device->md_io.in_use); /* drbd_md_put_buffer() is in the completion handler */ device->md_io.submit_jif = jiffies; - if (drbd_insert_fault(device, (rw & WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) + if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) bio_io_error(bio); else submit_bio(bio); @@ -188,7 +189,7 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, } int drbd_md_sync_page_io(struct drbd_device *device, struct drbd_backing_dev *bdev, - sector_t sector, int rw) + sector_t sector, int op) { int err; D_ASSERT(device, atomic_read(&device->md_io.in_use) == 1); @@ -197,19 +198,21 @@ int drbd_md_sync_page_io(struct drbd_device *device, struct drbd_backing_dev *bd dynamic_drbd_dbg(device, "meta_data io: %s [%d]:%s(,%llus,%s) %pS\n", current->comm, current->pid, __func__, - (unsigned long long)sector, (rw & WRITE) ? "WRITE" : "READ", + (unsigned long long)sector, (op == REQ_OP_WRITE) ? "WRITE" : "READ", (void*)_RET_IP_ ); if (sector < drbd_md_first_sector(bdev) || sector + 7 > drbd_md_last_sector(bdev)) drbd_alert(device, "%s [%d]:%s(,%llus,%s) out of range md access!\n", current->comm, current->pid, __func__, - (unsigned long long)sector, (rw & WRITE) ? "WRITE" : "READ"); + (unsigned long long)sector, + (op == REQ_OP_WRITE) ? "WRITE" : "READ"); - err = _drbd_md_sync_page_io(device, bdev, sector, rw); + err = _drbd_md_sync_page_io(device, bdev, sector, op); if (err) { drbd_err(device, "drbd_md_sync_page_io(,%llus,%s) failed with error %d\n", - (unsigned long long)sector, (rw & WRITE) ? "WRITE" : "READ", err); + (unsigned long long)sector, + (op == REQ_OP_WRITE) ? "WRITE" : "READ", err); } return err; } diff --git a/drivers/block/drbd/drbd_bitmap.c b/drivers/block/drbd/drbd_bitmap.c index ef44adb..126bf4a 100644 --- a/drivers/block/drbd/drbd_bitmap.c +++ b/drivers/block/drbd/drbd_bitmap.c @@ -980,7 +980,7 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho struct drbd_bitmap *b = device->bitmap; struct page *page; unsigned int len; - unsigned int rw = (ctx->flags & BM_AIO_READ) ? READ : WRITE; + unsigned int op = (ctx->flags & BM_AIO_READ) ? REQ_OP_READ : REQ_OP_WRITE; sector_t on_disk_sector = device->ldev->md.md_offset + device->ldev->md.bm_offset; @@ -1011,9 +1011,9 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho bio_add_page(bio, page, len, 0); bio->bi_private = ctx; bio->bi_end_io = drbd_bm_endio; - bio->bi_rw |= rw; + bio->bi_op = op; - if (drbd_insert_fault(device, (rw & WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) { + if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) { bio_io_error(bio); } else { submit_bio(bio); diff --git a/drivers/block/drbd/drbd_int.h b/drivers/block/drbd/drbd_int.h index 34bc84e..5204079 100644 --- a/drivers/block/drbd/drbd_int.h +++ b/drivers/block/drbd/drbd_int.h @@ -1507,7 +1507,7 @@ extern int drbd_resync_finished(struct drbd_device *device); extern void *drbd_md_get_buffer(struct drbd_device *device, const char *intent); extern void drbd_md_put_buffer(struct drbd_device *device); extern int drbd_md_sync_page_io(struct drbd_device *device, - struct drbd_backing_dev *bdev, sector_t sector, int rw); + struct drbd_backing_dev *bdev, sector_t sector, int op); extern void drbd_ov_out_of_sync_found(struct drbd_device *, sector_t, int); extern void wait_until_done_or_force_detached(struct drbd_device *device, struct drbd_backing_dev *bdev, unsigned int *done); @@ -1557,7 +1557,7 @@ extern bool drbd_rs_should_slow_down(struct drbd_device *device, sector_t sector bool throttle_if_app_is_waiting); extern int drbd_submit_peer_request(struct drbd_device *, struct drbd_peer_request *, const unsigned, - const int); + const unsigned, const int); extern int drbd_free_peer_reqs(struct drbd_device *, struct list_head *); extern struct drbd_peer_request *drbd_alloc_peer_req(struct drbd_peer_device *, u64, sector_t, unsigned int, diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 5b43dfb..f6626e5 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -3061,7 +3061,7 @@ void drbd_md_write(struct drbd_device *device, void *b) D_ASSERT(device, drbd_md_ss(device->ldev) == device->ldev->md.md_offset); sector = device->ldev->md.md_offset; - if (drbd_md_sync_page_io(device, device->ldev, sector, WRITE)) { + if (drbd_md_sync_page_io(device, device->ldev, sector, REQ_OP_WRITE)) { /* this was a try anyways ... */ drbd_err(device, "meta data update failed!\n"); drbd_chk_io_error(device, 1, DRBD_META_IO_ERROR); @@ -3263,7 +3263,8 @@ int drbd_md_read(struct drbd_device *device, struct drbd_backing_dev *bdev) * Affects the paranoia out-of-range access check in drbd_md_sync_page_io(). */ bdev->md.md_size_sect = 8; - if (drbd_md_sync_page_io(device, bdev, bdev->md.md_offset, READ)) { + if (drbd_md_sync_page_io(device, bdev, bdev->md.md_offset, + REQ_OP_READ)) { /* NOTE: can't do normal error processing here as this is called BEFORE disk is attached */ drbd_err(device, "Error while reading metadata.\n"); diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c index 1957fe8..880de0d 100644 --- a/drivers/block/drbd/drbd_receiver.c +++ b/drivers/block/drbd/drbd_receiver.c @@ -1398,7 +1398,8 @@ void drbd_bump_write_ordering(struct drbd_resource *resource, struct drbd_backin /* TODO allocate from our own bio_set. */ int drbd_submit_peer_request(struct drbd_device *device, struct drbd_peer_request *peer_req, - const unsigned rw, const int fault_type) + const unsigned op, const unsigned op_flags, + const int fault_type) { struct bio *bios = NULL; struct bio *bio; @@ -1450,7 +1451,8 @@ next_bio: /* > peer_req->i.sector, unless this is the first bio */ bio->bi_iter.bi_sector = sector; bio->bi_bdev = device->ldev->backing_bdev; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; bio->bi_private = peer_req; bio->bi_end_io = drbd_peer_request_endio; @@ -1458,7 +1460,7 @@ next_bio: bios = bio; ++n_bios; - if (rw & REQ_DISCARD) { + if (op == REQ_OP_DISCARD) { bio->bi_iter.bi_size = data_size; goto submit; } @@ -1830,7 +1832,8 @@ static int recv_resync_read(struct drbd_peer_device *peer_device, sector_t secto spin_unlock_irq(&device->resource->req_lock); atomic_add(pi->size >> 9, &device->rs_sect_ev); - if (drbd_submit_peer_request(device, peer_req, WRITE, DRBD_FAULT_RS_WR) == 0) + if (drbd_submit_peer_request(device, peer_req, REQ_OP_WRITE, 0, + DRBD_FAULT_RS_WR) == 0) return 0; /* don't care for the reason here */ @@ -2152,12 +2155,19 @@ static int wait_for_and_update_peer_seq(struct drbd_peer_device *peer_device, co /* see also bio_flags_to_wire() * DRBD_REQ_*, because we need to semantically map the flags to data packet * flags and back. We may replicate to other kernel versions. */ -static unsigned long wire_flags_to_bio(u32 dpf) +static unsigned long wire_flags_to_bio_flags(u32 dpf) { return (dpf & DP_RW_SYNC ? REQ_SYNC : 0) | (dpf & DP_FUA ? REQ_FUA : 0) | - (dpf & DP_FLUSH ? REQ_FLUSH : 0) | - (dpf & DP_DISCARD ? REQ_DISCARD : 0); + (dpf & DP_FLUSH ? REQ_FLUSH : 0); +} + +static unsigned long wire_flags_to_bio_op(u32 dpf) +{ + if (dpf & DP_DISCARD) + return REQ_OP_DISCARD; + else + return REQ_OP_WRITE; } static void fail_postponed_requests(struct drbd_device *device, sector_t sector, @@ -2303,7 +2313,7 @@ static int receive_Data(struct drbd_connection *connection, struct packet_info * struct drbd_peer_request *peer_req; struct p_data *p = pi->data; u32 peer_seq = be32_to_cpu(p->seq_num); - int rw = WRITE; + int op, op_flags; u32 dp_flags; int err, tp; @@ -2342,14 +2352,15 @@ static int receive_Data(struct drbd_connection *connection, struct packet_info * peer_req->flags |= EE_APPLICATION; dp_flags = be32_to_cpu(p->dp_flags); - rw |= wire_flags_to_bio(dp_flags); + op = wire_flags_to_bio_op(dp_flags); + op_flags = wire_flags_to_bio_flags(dp_flags); if (pi->cmd == P_TRIM) { struct request_queue *q = bdev_get_queue(device->ldev->backing_bdev); peer_req->flags |= EE_IS_TRIM; if (!blk_queue_discard(q)) peer_req->flags |= EE_IS_TRIM_USE_ZEROOUT; D_ASSERT(peer_device, peer_req->i.size > 0); - D_ASSERT(peer_device, rw & REQ_DISCARD); + D_ASSERT(peer_device, op == REQ_OP_DISCARD); D_ASSERT(peer_device, peer_req->pages == NULL); } else if (peer_req->pages == NULL) { D_ASSERT(device, peer_req->i.size == 0); @@ -2433,7 +2444,8 @@ static int receive_Data(struct drbd_connection *connection, struct packet_info * peer_req->flags |= EE_CALL_AL_COMPLETE_IO; } - err = drbd_submit_peer_request(device, peer_req, rw, DRBD_FAULT_DT_WR); + err = drbd_submit_peer_request(device, peer_req, op, op_flags, + DRBD_FAULT_DT_WR); if (!err) return 0; @@ -2723,7 +2735,8 @@ submit_for_resync: submit: update_receiver_timing_details(connection, drbd_submit_peer_request); inc_unacked(device); - if (drbd_submit_peer_request(device, peer_req, READ, fault_type) == 0) + if (drbd_submit_peer_request(device, peer_req, REQ_OP_READ, 0, + fault_type) == 0) return 0; /* don't care for the reason here */ diff --git a/drivers/block/drbd/drbd_worker.c b/drivers/block/drbd/drbd_worker.c index eff716c..f92f533 100644 --- a/drivers/block/drbd/drbd_worker.c +++ b/drivers/block/drbd/drbd_worker.c @@ -392,7 +392,8 @@ static int read_for_csum(struct drbd_peer_device *peer_device, sector_t sector, spin_unlock_irq(&device->resource->req_lock); atomic_add(size >> 9, &device->rs_sect_ev); - if (drbd_submit_peer_request(device, peer_req, READ, DRBD_FAULT_RS_RD) == 0) + if (drbd_submit_peer_request(device, peer_req, REQ_OP_READ, 0, + DRBD_FAULT_RS_RD) == 0) return 0; /* If it failed because of ENOMEM, retry should help. If it failed -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:53 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 4456529E20 for ; Tue, 5 Jan 2016 14:54:53 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 35EB88F804C for ; Tue, 5 Jan 2016 12:54:53 -0800 (PST) X-ASG-Debug-ID: 1452027290-04cbb07e1905490001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id AF9dhawYXMYo74y2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:50 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 0F033362098; Tue, 5 Jan 2016 20:54:50 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuM024032; Tue, 5 Jan 2016 15:54:47 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 23/35] md/raid: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:26 -0600 X-ASG-Orig-Subj: [PATCH 23/35] md/raid: set bi_op to REQ_OP Message-Id: <1452027218-32303-24-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027290 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has md/raid set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/md/bitmap.c | 2 +- drivers/md/dm-raid.c | 5 +++-- drivers/md/md.c | 11 +++++++---- drivers/md/md.h | 3 ++- drivers/md/raid1.c | 34 ++++++++++++++++---------------- drivers/md/raid10.c | 50 ++++++++++++++++++++++++++---------------------- drivers/md/raid5-cache.c | 25 +++++++++++++++--------- drivers/md/raid5.c | 48 ++++++++++++++++++++++++++-------------------- 8 files changed, 101 insertions(+), 77 deletions(-) diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c index 13811fc..18458f2 100644 --- a/drivers/md/bitmap.c +++ b/drivers/md/bitmap.c @@ -160,7 +160,7 @@ static int read_sb_page(struct mddev *mddev, loff_t offset, if (sync_page_io(rdev, target, roundup(size, bdev_logical_block_size(rdev->bdev)), - page, READ, true)) { + page, REQ_OP_READ, 0, true)) { page->index = index; return 0; } diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c index a090121..43a749c 100644 --- a/drivers/md/dm-raid.c +++ b/drivers/md/dm-raid.c @@ -792,7 +792,7 @@ static int read_disk_sb(struct md_rdev *rdev, int size) if (rdev->sb_loaded) return 0; - if (!sync_page_io(rdev, 0, size, rdev->sb_page, READ, 1)) { + if (!sync_page_io(rdev, 0, size, rdev->sb_page, REQ_OP_READ, 0, 1)) { DMERR("Failed to read superblock of device at position %d", rdev->raid_disk); md_error(rdev->mddev, rdev); @@ -1646,7 +1646,8 @@ static void attempt_restore_of_faulty_devices(struct raid_set *rs) for (i = 0; i < rs->md.raid_disks; i++) { r = &rs->dev[i].rdev; if (test_bit(Faulty, &r->flags) && r->sb_page && - sync_page_io(r, 0, r->sb_size, r->sb_page, READ, 1)) { + sync_page_io(r, 0, r->sb_size, r->sb_page, REQ_OP_READ, 0, + 1)) { DMINFO("Faulty %s device #%d has readable super block." " Attempting to revive it.", rs->raid_type->name, i); diff --git a/drivers/md/md.c b/drivers/md/md.c index e25ef97..ee1ef20 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -401,6 +401,7 @@ static void submit_flushes(struct work_struct *ws) bi->bi_end_io = md_end_flush; bi->bi_private = rdev; bi->bi_bdev = rdev->bdev; + bi->bi_op = REQ_OP_WRITE; bi->bi_rw |= WRITE_FLUSH; atomic_inc(&mddev->flush_pending); submit_bio(bi); @@ -747,6 +748,7 @@ void md_super_write(struct mddev *mddev, struct md_rdev *rdev, bio_add_page(bio, page, size, 0); bio->bi_private = rdev; bio->bi_end_io = super_written; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw |= WRITE_FLUSH_FUA; atomic_inc(&mddev->pending_writes); @@ -760,14 +762,15 @@ void md_super_wait(struct mddev *mddev) } int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, - struct page *page, int rw, bool metadata_op) + struct page *page, int op, int op_flags, bool metadata_op) { struct bio *bio = bio_alloc_mddev(GFP_NOIO, 1, rdev->mddev); int ret; bio->bi_bdev = (metadata_op && rdev->meta_bdev) ? rdev->meta_bdev : rdev->bdev; - bio->bi_rw |= rw; + bio->bi_op = op; + bio->bi_rw |= op_flags; if (metadata_op) bio->bi_iter.bi_sector = sector + rdev->sb_start; else if (rdev->mddev->reshape_position != MaxSector && @@ -793,7 +796,7 @@ static int read_disk_sb(struct md_rdev *rdev, int size) if (rdev->sb_loaded) return 0; - if (!sync_page_io(rdev, 0, size, rdev->sb_page, READ, true)) + if (!sync_page_io(rdev, 0, size, rdev->sb_page, REQ_OP_READ, 0, true)) goto fail; rdev->sb_loaded = 1; return 0; @@ -1479,7 +1482,7 @@ static int super_1_load(struct md_rdev *rdev, struct md_rdev *refdev, int minor_ return -EINVAL; bb_sector = (long long)offset; if (!sync_page_io(rdev, bb_sector, sectors << 9, - rdev->bb_page, READ, true)) + rdev->bb_page, REQ_OP_READ, 0, true)) return -EIO; bbp = (u64 *)page_address(rdev->bb_page); rdev->badblocks.shift = sb->bblog_shift; diff --git a/drivers/md/md.h b/drivers/md/md.h index 5eb0475..d542059 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -620,7 +620,8 @@ extern void md_super_write(struct mddev *mddev, struct md_rdev *rdev, sector_t sector, int size, struct page *page); extern void md_super_wait(struct mddev *mddev); extern int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, - struct page *page, int rw, bool metadata_op); + struct page *page, int op, int op_flags, + bool metadata_op); extern void md_do_sync(struct md_thread *thread); extern void md_new_event(struct mddev *mddev); extern int md_allow_write(struct mddev *mddev); diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 01e6e74..2a42daa 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -759,7 +759,7 @@ static void flush_pending_writes(struct r1conf *conf) while (bio) { /* submit pending writes */ struct bio *next = bio->bi_next; bio->bi_next = NULL; - if (unlikely((bio->bi_rw & REQ_DISCARD) && + if (unlikely((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) /* Just ignore it */ bio_endio(bio); @@ -1033,7 +1033,7 @@ static void raid1_unplug(struct blk_plug_cb *cb, bool from_schedule) while (bio) { /* submit pending writes */ struct bio *next = bio->bi_next; bio->bi_next = NULL; - if (unlikely((bio->bi_rw & REQ_DISCARD) && + if (unlikely((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) /* Just ignore it */ bio_endio(bio); @@ -1053,12 +1053,11 @@ static void make_request(struct mddev *mddev, struct bio * bio) int i, disks; struct bitmap *bitmap; unsigned long flags; + const int op = bio->bi_op; const int rw = bio_data_dir(bio); const unsigned long do_sync = (bio->bi_rw & REQ_SYNC); const unsigned long do_flush_fua = (bio->bi_rw & (REQ_FLUSH | REQ_FUA)); - const unsigned long do_discard = (bio->bi_rw - & (REQ_DISCARD | REQ_SECURE)); - const unsigned long do_same = (bio->bi_rw & REQ_WRITE_SAME); + const unsigned long do_sec = (bio->bi_rw & REQ_SECURE); struct md_rdev *blocked_rdev; struct blk_plug_cb *cb; struct raid1_plug_cb *plug = NULL; @@ -1166,7 +1165,8 @@ read_again: mirror->rdev->data_offset; read_bio->bi_bdev = mirror->rdev->bdev; read_bio->bi_end_io = raid1_end_read_request; - read_bio->bi_rw = READ | do_sync; + read_bio->bi_op = op; + read_bio->bi_rw = do_sync; read_bio->bi_private = r1_bio; if (max_sectors < r1_bio->sectors) { @@ -1376,8 +1376,9 @@ read_again: conf->mirrors[i].rdev->data_offset); mbio->bi_bdev = conf->mirrors[i].rdev->bdev; mbio->bi_end_io = raid1_end_write_request; + mbio->bi_op = op; mbio->bi_rw = - WRITE | do_flush_fua | do_sync | do_discard | do_same; + do_flush_fua | do_sync | do_sec; mbio->bi_private = r1_bio; atomic_inc(&r1_bio->remaining); @@ -1771,7 +1772,7 @@ static void end_sync_write(struct bio *bio) static int r1_sync_page_io(struct md_rdev *rdev, sector_t sector, int sectors, struct page *page, int rw) { - if (sync_page_io(rdev, sector, sectors << 9, page, rw, false)) + if (sync_page_io(rdev, sector, sectors << 9, page, rw, 0, false)) /* success */ return 1; if (rw == WRITE) { @@ -1825,7 +1826,7 @@ static int fix_sync_read_error(struct r1bio *r1_bio) rdev = conf->mirrors[d].rdev; if (sync_page_io(rdev, sect, s<<9, bio->bi_io_vec[idx].bv_page, - READ, false)) { + REQ_OP_READ, 0, false)) { success = 1; break; } @@ -2030,7 +2031,7 @@ static void sync_request_write(struct mddev *mddev, struct r1bio *r1_bio) !test_bit(MD_RECOVERY_SYNC, &mddev->recovery)))) continue; - wbio->bi_rw = WRITE; + wbio->bi_op = REQ_OP_WRITE; wbio->bi_end_io = end_sync_write; atomic_inc(&r1_bio->remaining); md_sync_acct(conf->mirrors[i].rdev->bdev, bio_sectors(wbio)); @@ -2090,7 +2091,7 @@ static void fix_read_error(struct r1conf *conf, int read_disk, is_badblock(rdev, sect, s, &first_bad, &bad_sectors) == 0 && sync_page_io(rdev, sect, s<<9, - conf->tmppage, READ, false)) + conf->tmppage, REQ_OP_READ, 0, false)) success = 1; else { d++; @@ -2201,7 +2202,7 @@ static int narrow_write_error(struct r1bio *r1_bio, int i) wbio = bio_clone_mddev(r1_bio->master_bio, GFP_NOIO, mddev); } - wbio->bi_rw = WRITE; + wbio->bi_op = REQ_OP_WRITE; wbio->bi_iter.bi_sector = r1_bio->sector; wbio->bi_iter.bi_size = r1_bio->sectors << 9; @@ -2343,7 +2344,8 @@ read_more: bio->bi_iter.bi_sector = r1_bio->sector + rdev->data_offset; bio->bi_bdev = rdev->bdev; bio->bi_end_io = raid1_end_read_request; - bio->bi_rw = READ | do_sync; + bio->bi_op = REQ_OP_READ; + bio->bi_rw = do_sync; bio->bi_private = r1_bio; if (max_sectors < r1_bio->sectors) { /* Drat - have to split this up more */ @@ -2568,7 +2570,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipp if (i < conf->raid_disks) still_degraded = 1; } else if (!test_bit(In_sync, &rdev->flags)) { - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_end_io = end_sync_write; write_targets ++; } else { @@ -2595,7 +2597,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipp if (disk < 0) disk = i; } - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_end_io = end_sync_read; read_targets++; } else if (!test_bit(WriteErrorSeen, &rdev->flags) && @@ -2607,7 +2609,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipp * if we are doing resync or repair. Otherwise, leave * this device alone for this sync request. */ - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_end_io = end_sync_write; write_targets++; } diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index d2da45b..1cfc061 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -865,7 +865,7 @@ static void flush_pending_writes(struct r10conf *conf) while (bio) { /* submit pending writes */ struct bio *next = bio->bi_next; bio->bi_next = NULL; - if (unlikely((bio->bi_rw & REQ_DISCARD) && + if (unlikely((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) /* Just ignore it */ bio_endio(bio); @@ -1041,7 +1041,7 @@ static void raid10_unplug(struct blk_plug_cb *cb, bool from_schedule) while (bio) { /* submit pending writes */ struct bio *next = bio->bi_next; bio->bi_next = NULL; - if (unlikely((bio->bi_rw & REQ_DISCARD) && + if (unlikely((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) /* Just ignore it */ bio_endio(bio); @@ -1058,12 +1058,11 @@ static void __make_request(struct mddev *mddev, struct bio *bio) struct r10bio *r10_bio; struct bio *read_bio; int i; + const int op = bio->bi_op; const int rw = bio_data_dir(bio); const unsigned long do_sync = (bio->bi_rw & REQ_SYNC); const unsigned long do_fua = (bio->bi_rw & REQ_FUA); - const unsigned long do_discard = (bio->bi_rw - & (REQ_DISCARD | REQ_SECURE)); - const unsigned long do_same = (bio->bi_rw & REQ_WRITE_SAME); + const unsigned long do_sec = (bio->bi_rw & REQ_SECURE); unsigned long flags; struct md_rdev *blocked_rdev; struct blk_plug_cb *cb; @@ -1156,7 +1155,8 @@ read_again: choose_data_offset(r10_bio, rdev); read_bio->bi_bdev = rdev->bdev; read_bio->bi_end_io = raid10_end_read_request; - read_bio->bi_rw = READ | do_sync; + read_bio->bi_op = op; + read_bio->bi_rw = do_sync; read_bio->bi_private = r10_bio; if (max_sectors < r10_bio->sectors) { @@ -1363,8 +1363,9 @@ retry_write: rdev)); mbio->bi_bdev = rdev->bdev; mbio->bi_end_io = raid10_end_write_request; + mbio->bi_op = op; mbio->bi_rw = - WRITE | do_sync | do_fua | do_discard | do_same; + do_sync | do_fua | do_sec; mbio->bi_private = r10_bio; atomic_inc(&r10_bio->remaining); @@ -1406,8 +1407,9 @@ retry_write: r10_bio, rdev)); mbio->bi_bdev = rdev->bdev; mbio->bi_end_io = raid10_end_write_request; + mbio->bi_op = op; mbio->bi_rw = - WRITE | do_sync | do_fua | do_discard | do_same; + do_sync | do_fua | do_sec; mbio->bi_private = r10_bio; atomic_inc(&r10_bio->remaining); @@ -1992,7 +1994,7 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio) tbio->bi_vcnt = vcnt; tbio->bi_iter.bi_size = fbio->bi_iter.bi_size; - tbio->bi_rw = WRITE; + tbio->bi_op = REQ_OP_WRITE; tbio->bi_private = r10_bio; tbio->bi_iter.bi_sector = r10_bio->devs[i].addr; tbio->bi_end_io = end_sync_write; @@ -2078,7 +2080,7 @@ static void fix_recovery_read_error(struct r10bio *r10_bio) addr, s << 9, bio->bi_io_vec[idx].bv_page, - READ, false); + REQ_OP_READ, 0, false); if (ok) { rdev = conf->mirrors[dw].rdev; addr = r10_bio->devs[1].addr + sect; @@ -2086,7 +2088,7 @@ static void fix_recovery_read_error(struct r10bio *r10_bio) addr, s << 9, bio->bi_io_vec[idx].bv_page, - WRITE, false); + REQ_OP_WRITE, 0, false); if (!ok) { set_bit(WriteErrorSeen, &rdev->flags); if (!test_and_set_bit(WantReplacement, @@ -2213,7 +2215,7 @@ static int r10_sync_page_io(struct md_rdev *rdev, sector_t sector, if (is_badblock(rdev, sector, sectors, &first_bad, &bad_sectors) && (rw == READ || test_bit(WriteErrorSeen, &rdev->flags))) return -1; - if (sync_page_io(rdev, sector, sectors << 9, page, rw, false)) + if (sync_page_io(rdev, sector, sectors << 9, page, rw, 0, false)) /* success */ return 1; if (rw == WRITE) { @@ -2299,7 +2301,8 @@ static void fix_read_error(struct r10conf *conf, struct mddev *mddev, struct r10 r10_bio->devs[sl].addr + sect, s<<9, - conf->tmppage, READ, false); + conf->tmppage, + REQ_OP_READ, 0, false); rdev_dec_pending(rdev, mddev); rcu_read_lock(); if (success) @@ -2474,7 +2477,7 @@ static int narrow_write_error(struct r10bio *r10_bio, int i) choose_data_offset(r10_bio, rdev) + (sector - r10_bio->sector)); wbio->bi_bdev = rdev->bdev; - wbio->bi_rw |= WRITE; + wbio->bi_op = REQ_OP_WRITE; if (submit_bio_wait(wbio) < 0) /* Failure! */ @@ -2550,7 +2553,8 @@ read_more: bio->bi_iter.bi_sector = r10_bio->devs[slot].addr + choose_data_offset(r10_bio, rdev); bio->bi_bdev = rdev->bdev; - bio->bi_rw = READ | do_sync; + bio->bi_op = REQ_OP_READ; + bio->bi_rw = do_sync; bio->bi_private = r10_bio; bio->bi_end_io = raid10_end_read_request; if (max_sectors < r10_bio->sectors) { @@ -3037,7 +3041,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_read; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; from_addr = r10_bio->devs[j].addr; bio->bi_iter.bi_sector = from_addr + rdev->data_offset; @@ -3063,7 +3067,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_write; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_iter.bi_sector = to_addr + rdev->data_offset; bio->bi_bdev = rdev->bdev; @@ -3092,7 +3096,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_write; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_iter.bi_sector = to_addr + rdev->data_offset; bio->bi_bdev = rdev->bdev; @@ -3212,7 +3216,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_read; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_iter.bi_sector = sector + conf->mirrors[d].rdev->data_offset; bio->bi_bdev = conf->mirrors[d].rdev->bdev; @@ -3234,7 +3238,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_write; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_iter.bi_sector = sector + conf->mirrors[d].replacement->data_offset; bio->bi_bdev = conf->mirrors[d].replacement->bdev; @@ -4317,7 +4321,7 @@ read_more: + rdev->data_offset); read_bio->bi_private = r10_bio; read_bio->bi_end_io = end_sync_read; - read_bio->bi_rw = READ; + read_bio->bi_op = REQ_OP_READ; read_bio->bi_flags &= (~0UL << BIO_RESET_BITS); read_bio->bi_error = 0; read_bio->bi_vcnt = 0; @@ -4351,7 +4355,7 @@ read_more: rdev2->new_data_offset; b->bi_private = r10_bio; b->bi_end_io = end_reshape_write; - b->bi_rw = WRITE; + b->bi_op = REQ_OP_WRITE; b->bi_next = blist; blist = b; } @@ -4519,7 +4523,7 @@ static int handle_reshape_read_error(struct mddev *mddev, addr, s << 9, bvec[idx].bv_page, - READ, false); + REQ_OP_READ, 0, false); if (success) break; failed: diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 6b500fd..e19c2f0 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -261,7 +261,7 @@ static struct bio *r5l_bio_alloc(struct r5l_log *log) { struct bio *bio = bio_alloc_bioset(GFP_NOIO, BIO_MAX_PAGES, log->bs); - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_bdev = log->rdev->bdev; bio->bi_iter.bi_sector = log->rdev->data_offset + log->log_start; @@ -686,6 +686,7 @@ void r5l_flush_stripe_to_raid(struct r5l_log *log) bio_reset(&log->flush_bio); log->flush_bio.bi_bdev = log->rdev->bdev; log->flush_bio.bi_end_io = r5l_log_flush_endio; + log->flush_bio.bi_op = REQ_OP_WRITE; log->flush_bio.bi_rw |= WRITE_FLUSH; submit_bio(&log->flush_bio); } @@ -875,7 +876,8 @@ static int r5l_read_meta_block(struct r5l_log *log, struct r5l_meta_block *mb; u32 crc, stored_crc; - if (!sync_page_io(log->rdev, ctx->pos, PAGE_SIZE, page, READ, false)) + if (!sync_page_io(log->rdev, ctx->pos, PAGE_SIZE, page, REQ_OP_READ, 0, + false)) return -EIO; mb = page_address(page); @@ -920,7 +922,8 @@ static int r5l_recovery_flush_one_stripe(struct r5l_log *log, &disk_index, sh); sync_page_io(log->rdev, *log_offset, PAGE_SIZE, - sh->dev[disk_index].page, READ, false); + sh->dev[disk_index].page, REQ_OP_READ, 0, + false); sh->dev[disk_index].log_checksum = le32_to_cpu(payload->checksum[0]); set_bit(R5_Wantwrite, &sh->dev[disk_index].flags); @@ -928,7 +931,8 @@ static int r5l_recovery_flush_one_stripe(struct r5l_log *log, } else { disk_index = sh->pd_idx; sync_page_io(log->rdev, *log_offset, PAGE_SIZE, - sh->dev[disk_index].page, READ, false); + sh->dev[disk_index].page, REQ_OP_READ, 0, + false); sh->dev[disk_index].log_checksum = le32_to_cpu(payload->checksum[0]); set_bit(R5_Wantwrite, &sh->dev[disk_index].flags); @@ -938,7 +942,7 @@ static int r5l_recovery_flush_one_stripe(struct r5l_log *log, sync_page_io(log->rdev, r5l_ring_add(log, *log_offset, BLOCK_SECTORS), PAGE_SIZE, sh->dev[disk_index].page, - READ, false); + REQ_OP_READ, 0, false); sh->dev[disk_index].log_checksum = le32_to_cpu(payload->checksum[1]); set_bit(R5_Wantwrite, @@ -980,11 +984,13 @@ static int r5l_recovery_flush_one_stripe(struct r5l_log *log, rdev = rcu_dereference(conf->disks[disk_index].rdev); if (rdev) sync_page_io(rdev, stripe_sect, PAGE_SIZE, - sh->dev[disk_index].page, WRITE, false); + sh->dev[disk_index].page, REQ_OP_WRITE, 0, + false); rrdev = rcu_dereference(conf->disks[disk_index].replacement); if (rrdev) sync_page_io(rrdev, stripe_sect, PAGE_SIZE, - sh->dev[disk_index].page, WRITE, false); + sh->dev[disk_index].page, REQ_OP_WRITE, 0, + false); } raid5_release_stripe(sh); return 0; @@ -1056,7 +1062,8 @@ static int r5l_log_write_empty_meta_block(struct r5l_log *log, sector_t pos, crc = crc32c_le(log->uuid_checksum, mb, PAGE_SIZE); mb->checksum = cpu_to_le32(crc); - if (!sync_page_io(log->rdev, pos, PAGE_SIZE, page, WRITE_FUA, false)) { + if (!sync_page_io(log->rdev, pos, PAGE_SIZE, page, REQ_OP_WRITE, + WRITE_FUA, false)) { __free_page(page); return -EIO; } @@ -1131,7 +1138,7 @@ static int r5l_load_log(struct r5l_log *log) if (!page) return -ENOMEM; - if (!sync_page_io(rdev, cp, PAGE_SIZE, page, READ, false)) { + if (!sync_page_io(rdev, cp, PAGE_SIZE, page, REQ_OP_READ, 0, false)) { ret = -EIO; goto ioerr; } diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index a086014..fa4fe95 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -815,7 +815,8 @@ static void stripe_add_to_batch_list(struct r5conf *conf, struct stripe_head *sh dd_idx = 0; while (dd_idx == sh->pd_idx || dd_idx == sh->qd_idx) dd_idx++; - if (head->dev[dd_idx].towrite->bi_rw != sh->dev[dd_idx].towrite->bi_rw) + if (head->dev[dd_idx].towrite->bi_rw != sh->dev[dd_idx].towrite->bi_rw || + head->dev[dd_idx].towrite->bi_op != sh->dev[dd_idx].towrite->bi_op) goto unlock_out; if (head->batch_head) { @@ -900,29 +901,32 @@ static void ops_run_io(struct stripe_head *sh, struct stripe_head_state *s) if (r5l_write_stripe(conf->log, sh) == 0) return; for (i = disks; i--; ) { - int rw; + int op; + int op_flags = 0; int replace_only = 0; struct bio *bi, *rbi; struct md_rdev *rdev, *rrdev = NULL; sh = head_sh; if (test_and_clear_bit(R5_Wantwrite, &sh->dev[i].flags)) { - if (test_and_clear_bit(R5_WantFUA, &sh->dev[i].flags)) - rw = WRITE_FUA; - else - rw = WRITE; + if (test_and_clear_bit(R5_WantFUA, &sh->dev[i].flags)) { + op = REQ_OP_WRITE; + op_flags = WRITE_FUA; + } else { + op = REQ_OP_WRITE; + } if (test_bit(R5_Discard, &sh->dev[i].flags)) - rw |= REQ_DISCARD; + op = REQ_OP_DISCARD; } else if (test_and_clear_bit(R5_Wantread, &sh->dev[i].flags)) - rw = READ; + op = REQ_OP_READ; else if (test_and_clear_bit(R5_WantReplace, &sh->dev[i].flags)) { - rw = WRITE; + op = REQ_OP_WRITE; replace_only = 1; } else continue; if (test_and_clear_bit(R5_SyncIO, &sh->dev[i].flags)) - rw |= REQ_SYNC; + op_flags |= REQ_SYNC; again: bi = &sh->dev[i].req; @@ -936,7 +940,7 @@ again: rdev = rrdev; rrdev = NULL; } - if (rw & WRITE) { + if (op_is_write(op)) { if (replace_only) rdev = NULL; if (rdev == rrdev) @@ -962,7 +966,7 @@ again: * need to check for writes. We never accept write errors * on the replacement, so we don't to check rrdev. */ - while ((rw & WRITE) && rdev && + while (op_is_write(op) && rdev && test_bit(WriteErrorSeen, &rdev->flags)) { sector_t first_bad; int bad_sectors; @@ -1004,8 +1008,9 @@ again: bio_reset(bi); bi->bi_bdev = rdev->bdev; - bi->bi_rw = rw; - bi->bi_end_io = (rw & WRITE) + bi->bi_op = op; + bi->bi_rw = op_flags; + bi->bi_end_io = op_is_write(op) ? raid5_end_write_request : raid5_end_read_request; bi->bi_private = sh; @@ -1036,7 +1041,7 @@ again: * If this is discard request, set bi_vcnt 0. We don't * want to confuse SCSI because SCSI will replace payload */ - if (rw & REQ_DISCARD) + if (op == REQ_OP_DISCARD) bi->bi_vcnt = 0; if (rrdev) set_bit(R5_DOUBLE_LOCKED, &sh->dev[i].flags); @@ -1056,8 +1061,9 @@ again: bio_reset(rbi); rbi->bi_bdev = rrdev->bdev; - rbi->bi_rw = rw; - BUG_ON(!(rw & WRITE)); + rbi->bi_op = op; + rbi->bi_rw = op_flags; + BUG_ON(!op_is_write(op)); rbi->bi_end_io = raid5_end_write_request; rbi->bi_private = sh; @@ -1085,7 +1091,7 @@ again: * If this is discard request, set bi_vcnt 0. We don't * want to confuse SCSI because SCSI will replace payload */ - if (rw & REQ_DISCARD) + if (op == REQ_OP_DISCARD) rbi->bi_vcnt = 0; if (conf->mddev->gendisk) trace_block_bio_remap(bdev_get_queue(rbi->bi_bdev), @@ -1094,7 +1100,7 @@ again: generic_make_request(rbi); } if (!rdev && !rrdev) { - if (rw & WRITE) + if (op_is_write(op)) set_bit(STRIPE_DEGRADED, &sh->state); pr_debug("skip op %ld on disc %d for sector %llu\n", bi->bi_rw, i, (unsigned long long)sh->sector); @@ -1632,7 +1638,7 @@ again: set_bit(R5_WantFUA, &dev->flags); if (wbi->bi_rw & REQ_SYNC) set_bit(R5_SyncIO, &dev->flags); - if (wbi->bi_rw & REQ_DISCARD) + if (wbi->bi_op == REQ_OP_DISCARD) set_bit(R5_Discard, &dev->flags); else { tx = async_copy_data(1, wbi, &dev->page, @@ -5173,7 +5179,7 @@ static void make_request(struct mddev *mddev, struct bio * bi) return; } - if (unlikely(bi->bi_rw & REQ_DISCARD)) { + if (unlikely(bi->bi_op == REQ_OP_DISCARD)) { make_discard_request(mddev, bi); return; } -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:54 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 7841E29DFD for ; Tue, 5 Jan 2016 14:54:54 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 19606AC002 for ; Tue, 5 Jan 2016 12:54:54 -0800 (PST) X-ASG-Debug-ID: 1452027292-04bdf03afa04e40001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 9bp5Lo27P4EsPmKP (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:53 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id BA0C46E79A; Tue, 5 Jan 2016 20:54:52 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuN024032; Tue, 5 Jan 2016 15:54:50 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 24/35] xen: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:27 -0600 X-ASG-Orig-Subj: [PATCH 24/35] xen: set bi_op to REQ_OP Message-Id: <1452027218-32303-25-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027293 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has xen set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/block/xen-blkback/blkback.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index c7e89af..c7d9643 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -488,7 +488,7 @@ static int xen_vbd_translate(struct phys_req *req, struct xen_blkif *blkif, struct xen_vbd *vbd = &blkif->vbd; int rc = -EACCES; - if ((operation != READ) && vbd->readonly) + if ((operation != REQ_OP_READ) && vbd->readonly) goto out; if (likely(req->nr_sects)) { @@ -995,7 +995,7 @@ static int dispatch_discard_io(struct xen_blkif *blkif, preq.sector_number = req->u.discard.sector_number; preq.nr_sects = req->u.discard.nr_sectors; - err = xen_vbd_translate(&preq, blkif, WRITE); + err = xen_vbd_translate(&preq, blkif, REQ_OP_WRITE); if (err) { pr_warn("access denied: DISCARD [%llu->%llu] on dev=%04x\n", preq.sector_number, @@ -1208,6 +1208,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, struct bio **biolist = pending_req->biolist; int i, nbio = 0; int operation; + int operation_flags = 0; struct blk_plug plug; bool drain = false; struct grant_page **pages = pending_req->segments; @@ -1226,17 +1227,19 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, switch (req_operation) { case BLKIF_OP_READ: blkif->st_rd_req++; - operation = READ; + operation = REQ_OP_READ; break; case BLKIF_OP_WRITE: blkif->st_wr_req++; - operation = WRITE_ODIRECT; + operation = REQ_OP_WRITE; + operation_flags = WRITE_ODIRECT; break; case BLKIF_OP_WRITE_BARRIER: drain = true; case BLKIF_OP_FLUSH_DISKCACHE: blkif->st_f_req++; - operation = WRITE_FLUSH; + operation = REQ_OP_WRITE; + operation_flags = WRITE_FLUSH; break; default: operation = 0; /* make gcc happy */ @@ -1248,7 +1251,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, nseg = req->operation == BLKIF_OP_INDIRECT ? req->u.indirect.nr_segments : req->u.rw.nr_segments; - if (unlikely(nseg == 0 && operation != WRITE_FLUSH) || + if (unlikely(nseg == 0 && operation_flags != WRITE_FLUSH) || unlikely((req->operation != BLKIF_OP_INDIRECT) && (nseg > BLKIF_MAX_SEGMENTS_PER_REQUEST)) || unlikely((req->operation == BLKIF_OP_INDIRECT) && @@ -1289,7 +1292,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, if (xen_vbd_translate(&preq, blkif, operation) != 0) { pr_debug("access denied: %s of [%llu,%llu] on dev=%04x\n", - operation == READ ? "read" : "write", + operation == REQ_OP_READ ? "read" : "write", preq.sector_number, preq.sector_number + preq.nr_sects, blkif->vbd.pdevice); @@ -1348,7 +1351,8 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, bio->bi_private = pending_req; bio->bi_end_io = end_block_io_op; bio->bi_iter.bi_sector = preq.sector_number; - bio->bi_rw |= operation; + bio->bi_op = operation; + bio->bi_rw |= operation_flags; } preq.sector_number += seg[i].nsec; @@ -1356,7 +1360,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, /* This will be hit if the operation was a flush or discard. */ if (!bio) { - BUG_ON(operation != WRITE_FLUSH); + BUG_ON(operation_flags != WRITE_FLUSH); bio = bio_alloc(GFP_KERNEL, 0); if (unlikely(bio == NULL)) @@ -1366,7 +1370,8 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, bio->bi_bdev = preq.bdev; bio->bi_private = pending_req; bio->bi_end_io = end_block_io_op; - bio->bi_rw |= operation; + bio->bi_op = operation; + bio->bi_rw |= operation_flags; } atomic_set(&pending_req->pendcnt, nbio); @@ -1378,9 +1383,9 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, /* Let the I/Os go.. */ blk_finish_plug(&plug); - if (operation == READ) + if (operation == REQ_OP_READ) blkif->st_rd_sect += preq.nr_sects; - else if (operation & WRITE) + else if (operation == REQ_OP_WRITE) blkif->st_wr_sect += preq.nr_sects; return 0; -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:54:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 4D45E29E06 for ; Tue, 5 Jan 2016 14:54:57 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 3E79E8F8035 for ; Tue, 5 Jan 2016 12:54:57 -0800 (PST) X-ASG-Debug-ID: 1452027295-04cbb07e1a054a0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Gtsc9CHO2q57AgUF (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:56 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 79B236E770; Tue, 5 Jan 2016 20:54:55 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuO024032; Tue, 5 Jan 2016 15:54:52 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 25/35] target: set bi_op to REQ_OP Date: Tue, 5 Jan 2016 14:53:28 -0600 X-ASG-Orig-Subj: [PATCH 25/35] target: set bi_op to REQ_OP Message-Id: <1452027218-32303-26-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027296 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the target modules set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/target/target_core_iblock.c | 38 ++++++++++++++++++++++--------------- drivers/target/target_core_pscsi.c | 2 +- 2 files changed, 24 insertions(+), 16 deletions(-) diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c index bfc3e45..b83195b 100644 --- a/drivers/target/target_core_iblock.c +++ b/drivers/target/target_core_iblock.c @@ -330,7 +330,8 @@ static void iblock_bio_done(struct bio *bio) } static struct bio * -iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, int rw) +iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, int op, + int op_flags) { struct iblock_dev *ib_dev = IBLOCK_DEV(cmd->se_dev); struct bio *bio; @@ -352,7 +353,8 @@ iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, int rw) bio->bi_private = cmd; bio->bi_end_io = &iblock_bio_done; bio->bi_iter.bi_sector = lba; - bio->bi_rw |= rw; + bio->bi_op = op; + bio->bi_rw |= op_flags; return bio; } @@ -458,7 +460,7 @@ iblock_execute_write_same(struct se_cmd *cmd) goto fail; cmd->priv = ibr; - bio = iblock_get_bio(cmd, block_lba, 1, WRITE); + bio = iblock_get_bio(cmd, block_lba, 1, REQ_OP_WRITE, 0); if (!bio) goto fail_free_ibr; @@ -471,7 +473,8 @@ iblock_execute_write_same(struct se_cmd *cmd) while (bio_add_page(bio, sg_page(sg), sg->length, sg->offset) != sg->length) { - bio = iblock_get_bio(cmd, block_lba, 1, WRITE); + bio = iblock_get_bio(cmd, block_lba, 1, REQ_OP_WRITE, + 0); if (!bio) goto fail_put_bios; @@ -657,7 +660,8 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, u32 sg_num = sgl_nents; sector_t block_lba; unsigned bio_cnt; - int rw = 0; + int op_flags = 0; + int op = 0; int i; if (data_direction == DMA_TO_DEVICE) { @@ -668,17 +672,20 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, * is not enabled, or if initiator set the Force Unit Access bit. */ if (q->flush_flags & REQ_FUA) { - if (cmd->se_cmd_flags & SCF_FUA) - rw = WRITE_FUA; - else if (!(q->flush_flags & REQ_FLUSH)) - rw = WRITE_FUA; - else - rw = WRITE; + if (cmd->se_cmd_flags & SCF_FUA) { + op = REQ_OP_WRITE; + op_flags = WRITE_FUA; + } else if (!(q->flush_flags & REQ_FLUSH)) { + op = REQ_OP_WRITE; + op_flags = WRITE_FUA; + } else { + op = REQ_OP_WRITE; + } } else { - rw = WRITE; + op = REQ_OP_WRITE; } } else { - rw = READ; + op = REQ_OP_READ; } /* @@ -710,7 +717,7 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, return 0; } - bio = iblock_get_bio(cmd, block_lba, sgl_nents, rw); + bio = iblock_get_bio(cmd, block_lba, sgl_nents, op, op_flags); if (!bio) goto fail_free_ibr; @@ -734,7 +741,8 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, bio_cnt = 0; } - bio = iblock_get_bio(cmd, block_lba, sg_num, rw); + bio = iblock_get_bio(cmd, block_lba, sg_num, op, + op_flags); if (!bio) goto fail_put_bios; diff --git a/drivers/target/target_core_pscsi.c b/drivers/target/target_core_pscsi.c index de18790..2cf915c 100644 --- a/drivers/target/target_core_pscsi.c +++ b/drivers/target/target_core_pscsi.c @@ -922,7 +922,7 @@ pscsi_map_sg(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, goto fail; if (rw) - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; pr_debug("PSCSI: Allocated bio: %p," " dir: %s nr_vecs: %d\n", bio, -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 8F95729E05 for ; Tue, 5 Jan 2016 14:55:01 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 493A7304039 for ; Tue, 5 Jan 2016 12:55:01 -0800 (PST) X-ASG-Debug-ID: 1452027298-04bdf03afa04e50001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id yImekt2fbwFEkg9c (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:54:58 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 3A10FC109084; Tue, 5 Jan 2016 20:54:58 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuP024032; Tue, 5 Jan 2016 15:54:55 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 26/35] block: set op to REQ_OP Date: Tue, 5 Jan 2016 14:53:29 -0600 X-ASG-Orig-Subj: [PATCH 26/35] block: set op to REQ_OP Message-Id: <1452027218-32303-27-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027298 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch converts the request related block layer code to set request->op to a REQ_OP and cmd_flags to rq_flag_bits. There is some tmp compat code when setting up cmd_flags so it still carries both the op and flags. It will be removed in in later patches in this set when I have converted all drivers. I have not been able to test the mq paths with real mq hardware. Signed-off-by: Mike Christie --- block/blk-core.c | 60 ++++++++++++++++++++++++++-------------------- block/blk-flush.c | 1 + block/blk-merge.c | 10 ++++---- block/blk-mq.c | 38 ++++++++++++++++------------- block/cfq-iosched.c | 53 +++++++++++++++++++++++----------------- block/elevator.c | 8 +++---- include/linux/blk-cgroup.h | 13 +++++----- include/linux/blkdev.h | 28 +++++++++++----------- include/linux/elevator.h | 4 ++-- 9 files changed, 120 insertions(+), 95 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 954a450..dacbd68 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -959,10 +959,10 @@ static void __freed_request(struct request_list *rl, int sync) * A request has just been released. Account for it, update the full and * congestion status, wake up any waiters. Called under q->queue_lock. */ -static void freed_request(struct request_list *rl, unsigned int flags) +static void freed_request(struct request_list *rl, int op, unsigned int flags) { struct request_queue *q = rl->q; - int sync = rw_is_sync(flags); + int sync = rw_is_sync(op, flags); q->nr_rqs[sync]--; rl->count[sync]--; @@ -1054,7 +1054,8 @@ static struct io_context *rq_ioc(struct bio *bio) /** * __get_request - get a free request * @rl: request list to allocate from - * @rw_flags: RW and SYNC flags + * @op: REQ_OP_READ/REQ_OP_WRITE + * @op_flags: rq_flag_bits * @bio: bio to allocate request for (can be %NULL) * @gfp_mask: allocation mask * @@ -1065,21 +1066,22 @@ static struct io_context *rq_ioc(struct bio *bio) * Returns ERR_PTR on failure, with @q->queue_lock held. * Returns request pointer on success, with @q->queue_lock *not held*. */ -static struct request *__get_request(struct request_list *rl, int rw_flags, - struct bio *bio, gfp_t gfp_mask) +static struct request *__get_request(struct request_list *rl, int op, + int op_flags, struct bio *bio, + gfp_t gfp_mask) { struct request_queue *q = rl->q; struct request *rq; struct elevator_type *et = q->elevator->type; struct io_context *ioc = rq_ioc(bio); struct io_cq *icq = NULL; - const bool is_sync = rw_is_sync(rw_flags) != 0; + const bool is_sync = rw_is_sync(op, op_flags) != 0; int may_queue; if (unlikely(blk_queue_dying(q))) return ERR_PTR(-ENODEV); - may_queue = elv_may_queue(q, rw_flags); + may_queue = elv_may_queue(q, op, op_flags); if (may_queue == ELV_MQUEUE_NO) goto rq_starved; @@ -1123,7 +1125,7 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, /* * Decide whether the new request will be managed by elevator. If - * so, mark @rw_flags and increment elvpriv. Non-zero elvpriv will + * so, mark @op_flags and increment elvpriv. Non-zero elvpriv will * prevent the current elevator from being destroyed until the new * request is freed. This guarantees icq's won't be destroyed and * makes creating new ones safe. @@ -1132,14 +1134,14 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, * it will be created after releasing queue_lock. */ if (blk_rq_should_init_elevator(bio) && !blk_queue_bypass(q)) { - rw_flags |= REQ_ELVPRIV; + op_flags |= REQ_ELVPRIV; q->nr_rqs_elvpriv++; if (et->icq_cache && ioc) icq = ioc_lookup_icq(ioc, q); } if (blk_queue_io_stat(q)) - rw_flags |= REQ_IO_STAT; + op_flags |= REQ_IO_STAT; spin_unlock_irq(q->queue_lock); /* allocate and init request */ @@ -1149,10 +1151,12 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, blk_rq_init(q, rq); blk_rq_set_rl(rq, rl); - rq->cmd_flags = rw_flags | REQ_ALLOCED; + /* tmp compat - allow users to check either one for the op */ + rq->cmd_flags = op | op_flags | REQ_ALLOCED; + rq->op = op; /* init elvpriv */ - if (rw_flags & REQ_ELVPRIV) { + if (op_flags & REQ_ELVPRIV) { if (unlikely(et->icq_cache && !icq)) { if (ioc) icq = ioc_create_icq(ioc, q, gfp_mask); @@ -1178,7 +1182,7 @@ out: if (ioc_batching(q, ioc)) ioc->nr_batch_requests--; - trace_block_getrq(q, bio, rw_flags & 1); + trace_block_getrq(q, bio, op); return rq; fail_elvpriv: @@ -1208,7 +1212,7 @@ fail_alloc: * queue, but this is pretty rare. */ spin_lock_irq(q->queue_lock); - freed_request(rl, rw_flags); + freed_request(rl, op, op_flags); /* * in the very unlikely event that allocation failed and no @@ -1226,7 +1230,8 @@ rq_starved: /** * get_request - get a free request * @q: request_queue to allocate request from - * @rw_flags: RW and SYNC flags + * op: REQ_OP_READ/REQ_OP_WRITE + * @op_flags: rq_flag_bits * @bio: bio to allocate request for (can be %NULL) * @gfp_mask: allocation mask * @@ -1237,17 +1242,18 @@ rq_starved: * Returns ERR_PTR on failure, with @q->queue_lock held. * Returns request pointer on success, with @q->queue_lock *not held*. */ -static struct request *get_request(struct request_queue *q, int rw_flags, - struct bio *bio, gfp_t gfp_mask) +static struct request *get_request(struct request_queue *q, int op, + int op_flags, struct bio *bio, + gfp_t gfp_mask) { - const bool is_sync = rw_is_sync(rw_flags) != 0; + const bool is_sync = rw_is_sync(op, op_flags) != 0; DEFINE_WAIT(wait); struct request_list *rl; struct request *rq; rl = blk_get_rl(q, bio); /* transferred to @rq on success */ retry: - rq = __get_request(rl, rw_flags, bio, gfp_mask); + rq = __get_request(rl, op, op_flags, bio, gfp_mask); if (!IS_ERR(rq)) return rq; @@ -1260,7 +1266,7 @@ retry: prepare_to_wait_exclusive(&rl->wait[is_sync], &wait, TASK_UNINTERRUPTIBLE); - trace_block_sleeprq(q, bio, rw_flags & 1); + trace_block_sleeprq(q, bio, op); spin_unlock_irq(q->queue_lock); io_schedule(); @@ -1289,7 +1295,7 @@ static struct request *blk_old_get_request(struct request_queue *q, int rw, create_io_context(gfp_mask, q->node); spin_lock_irq(q->queue_lock); - rq = get_request(q, rw, NULL, gfp_mask); + rq = get_request(q, rw, 0, NULL, gfp_mask); if (IS_ERR(rq)) spin_unlock_irq(q->queue_lock); /* q->queue_lock is unlocked at this point */ @@ -1491,13 +1497,14 @@ void __blk_put_request(struct request_queue *q, struct request *req) */ if (req->cmd_flags & REQ_ALLOCED) { unsigned int flags = req->cmd_flags; + int op = req->op; struct request_list *rl = blk_rq_rl(req); BUG_ON(!list_empty(&req->queuelist)); BUG_ON(ELV_ON_HASH(req)); blk_free_request(rl, req); - freed_request(rl, flags); + freed_request(rl, op, flags); blk_put_rl(rl); } } @@ -1712,7 +1719,7 @@ static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio) { const bool sync = !!(bio->bi_rw & REQ_SYNC); struct blk_plug *plug; - int el_ret, rw_flags, where = ELEVATOR_INSERT_SORT; + int el_ret, rw_flags = 0, where = ELEVATOR_INSERT_SORT; struct request *req; unsigned int request_count = 0; @@ -1772,7 +1779,6 @@ get_rq: * but we need to set it earlier to expose the sync flag to the * rq allocator and io schedulers. */ - rw_flags = bio_data_dir(bio); if (sync) rw_flags |= REQ_SYNC; @@ -1780,7 +1786,7 @@ get_rq: * Grab a free request. This is might sleep but can not fail. * Returns with the queue unlocked. */ - req = get_request(q, rw_flags, bio, GFP_NOIO); + req = get_request(q, bio_data_dir(bio), rw_flags, bio, GFP_NOIO); if (IS_ERR(req)) { bio->bi_error = PTR_ERR(req); bio_endio(bio); @@ -2168,7 +2174,7 @@ EXPORT_SYMBOL(submit_bio); static int blk_cloned_rq_check_limits(struct request_queue *q, struct request *rq) { - if (blk_rq_sectors(rq) > blk_queue_get_max_sectors(q, rq->cmd_flags)) { + if (blk_rq_sectors(rq) > blk_queue_get_max_sectors(q, rq->op)) { printk(KERN_ERR "%s: over max size limit.\n", __func__); return -EIO; } @@ -2987,6 +2993,7 @@ void blk_rq_bio_prep(struct request_queue *q, struct request *rq, { /* tmp compat. Allow users to set bi_op or bi_rw */ rq->cmd_flags |= bio_data_dir(bio); + rq->op = bio->bi_op; if (bio_has_data(bio)) rq->nr_phys_segments = bio_phys_segments(q, bio); @@ -3071,6 +3078,7 @@ EXPORT_SYMBOL_GPL(blk_rq_unprep_clone); static void __blk_rq_prep_clone(struct request *dst, struct request *src) { dst->cpu = src->cpu; + dst->op = src->op; dst->cmd_flags |= (src->cmd_flags & REQ_CLONE_MASK) | REQ_NOMERGE; dst->cmd_type = src->cmd_type; dst->__sector = blk_rq_pos(src); diff --git a/block/blk-flush.c b/block/blk-flush.c index 386f57a..b4eb0e8 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -330,6 +330,7 @@ static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) flush_rq->cmd_type = REQ_TYPE_FS; flush_rq->cmd_flags = WRITE_FLUSH | REQ_FLUSH_SEQ; + flush_rq->op = REQ_OP_WRITE; flush_rq->rq_disk = first_rq->rq_disk; flush_rq->end_io = flush_end_io; diff --git a/block/blk-merge.c b/block/blk-merge.c index b70f0ff..4c47a43 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -628,7 +628,8 @@ static int attempt_merge(struct request_queue *q, struct request *req, if (!rq_mergeable(req) || !rq_mergeable(next)) return 0; - if (!blk_check_merge_flags(req->cmd_flags, next->cmd_flags)) + if (!blk_check_merge_flags(req->cmd_flags, req->op, next->cmd_flags, + next->op)) return 0; /* @@ -642,7 +643,7 @@ static int attempt_merge(struct request_queue *q, struct request *req, || req_no_special_merge(next)) return 0; - if (req->cmd_flags & REQ_WRITE_SAME && + if (req->op == REQ_OP_WRITE_SAME && !blk_write_same_mergeable(req->bio, next->bio)) return 0; @@ -730,7 +731,8 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) if (!rq_mergeable(rq) || !bio_mergeable(bio)) return false; - if (!blk_check_merge_flags(rq->cmd_flags, bio->bi_rw)) + if (!blk_check_merge_flags(rq->cmd_flags, rq->op, bio->bi_rw, + bio->bi_op)) return false; /* different data direction or already started, don't merge */ @@ -746,7 +748,7 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) return false; /* must be using the same buffer */ - if (rq->cmd_flags & REQ_WRITE_SAME && + if (rq->op == REQ_OP_WRITE_SAME && !blk_write_same_mergeable(rq->bio, bio)) return false; diff --git a/block/blk-mq.c b/block/blk-mq.c index 4c0622f..745dae8 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -159,16 +159,19 @@ bool blk_mq_can_queue(struct blk_mq_hw_ctx *hctx) EXPORT_SYMBOL(blk_mq_can_queue); static void blk_mq_rq_ctx_init(struct request_queue *q, struct blk_mq_ctx *ctx, - struct request *rq, unsigned int rw_flags) + struct request *rq, int op, + unsigned int op_flags) { if (blk_queue_io_stat(q)) - rw_flags |= REQ_IO_STAT; + op_flags |= REQ_IO_STAT; INIT_LIST_HEAD(&rq->queuelist); /* csd/requeue_work/fifo_time is initialized before use */ rq->q = q; rq->mq_ctx = ctx; - rq->cmd_flags |= rw_flags; + rq->op = op; + /* tmp compat - allow users to check either one for the op */ + rq->cmd_flags |= op | op_flags; /* do not touch atomic flags, it needs atomic ops against the timer */ rq->cpu = -1; INIT_HLIST_NODE(&rq->hash); @@ -203,11 +206,11 @@ static void blk_mq_rq_ctx_init(struct request_queue *q, struct blk_mq_ctx *ctx, rq->end_io_data = NULL; rq->next_rq = NULL; - ctx->rq_dispatched[rw_is_sync(rw_flags)]++; + ctx->rq_dispatched[rw_is_sync(op, op_flags)]++; } static struct request * -__blk_mq_alloc_request(struct blk_mq_alloc_data *data, int rw) +__blk_mq_alloc_request(struct blk_mq_alloc_data *data, int op, int op_flags) { struct request *rq; unsigned int tag; @@ -222,7 +225,7 @@ __blk_mq_alloc_request(struct blk_mq_alloc_data *data, int rw) } rq->tag = tag; - blk_mq_rq_ctx_init(data->q, data->ctx, rq, rw); + blk_mq_rq_ctx_init(data->q, data->ctx, rq, op, op_flags); return rq; } @@ -246,7 +249,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, hctx = q->mq_ops->map_queue(q, ctx->cpu); blk_mq_set_alloc_data(&alloc_data, q, flags, ctx, hctx); - rq = __blk_mq_alloc_request(&alloc_data, rw); + rq = __blk_mq_alloc_request(&alloc_data, rw, 0); if (!rq && !(flags & BLK_MQ_REQ_NOWAIT)) { __blk_mq_run_hw_queue(hctx); blk_mq_put_ctx(ctx); @@ -254,7 +257,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, ctx = blk_mq_get_ctx(q); hctx = q->mq_ops->map_queue(q, ctx->cpu); blk_mq_set_alloc_data(&alloc_data, q, flags, ctx, hctx); - rq = __blk_mq_alloc_request(&alloc_data, rw); + rq = __blk_mq_alloc_request(&alloc_data, rw, 0); ctx = alloc_data.ctx; } blk_mq_put_ctx(ctx); @@ -1165,28 +1168,29 @@ static struct request *blk_mq_map_request(struct request_queue *q, struct blk_mq_hw_ctx *hctx; struct blk_mq_ctx *ctx; struct request *rq; - int rw = bio_data_dir(bio); + int op = bio_data_dir(bio); + int op_flags = 0; struct blk_mq_alloc_data alloc_data; blk_queue_enter_live(q); ctx = blk_mq_get_ctx(q); hctx = q->mq_ops->map_queue(q, ctx->cpu); - if (rw_is_sync(bio->bi_rw)) - rw |= REQ_SYNC; + if (rw_is_sync(bio->bi_op, bio->bi_rw)) + op_flags |= REQ_SYNC; - trace_block_getrq(q, bio, rw); + trace_block_getrq(q, bio, op); blk_mq_set_alloc_data(&alloc_data, q, BLK_MQ_REQ_NOWAIT, ctx, hctx); - rq = __blk_mq_alloc_request(&alloc_data, rw); + rq = __blk_mq_alloc_request(&alloc_data, op, op_flags); if (unlikely(!rq)) { __blk_mq_run_hw_queue(hctx); blk_mq_put_ctx(ctx); - trace_block_sleeprq(q, bio, rw); + trace_block_sleeprq(q, bio, op); ctx = blk_mq_get_ctx(q); hctx = q->mq_ops->map_queue(q, ctx->cpu); blk_mq_set_alloc_data(&alloc_data, q, 0, ctx, hctx); - rq = __blk_mq_alloc_request(&alloc_data, rw); + rq = __blk_mq_alloc_request(&alloc_data, op, op_flags); ctx = alloc_data.ctx; hctx = alloc_data.hctx; } @@ -1240,7 +1244,7 @@ static int blk_mq_direct_issue_request(struct request *rq, blk_qc_t *cookie) */ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) { - const int is_sync = rw_is_sync(bio->bi_rw); + const int is_sync = rw_is_sync(bio->bi_op, bio->bi_rw); const int is_flush_fua = bio->bi_rw & (REQ_FLUSH | REQ_FUA); struct blk_map_ctx data; struct request *rq; @@ -1337,7 +1341,7 @@ done: */ static blk_qc_t blk_sq_make_request(struct request_queue *q, struct bio *bio) { - const int is_sync = rw_is_sync(bio->bi_rw); + const int is_sync = rw_is_sync(bio->bi_op, bio->bi_rw); const int is_flush_fua = bio->bi_rw & (REQ_FLUSH | REQ_FUA); struct blk_plug *plug; unsigned int request_count = 0; diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c index 1f9093e..d300aa1 100644 --- a/block/cfq-iosched.c +++ b/block/cfq-iosched.c @@ -660,9 +660,10 @@ static inline void cfqg_put(struct cfq_group *cfqg) } while (0) static inline void cfqg_stats_update_io_add(struct cfq_group *cfqg, - struct cfq_group *curr_cfqg, int rw) + struct cfq_group *curr_cfqg, int op, + int op_flags) { - blkg_rwstat_add(&cfqg->stats.queued, rw, 1); + blkg_rwstat_add(&cfqg->stats.queued, op, op_flags, 1); cfqg_stats_end_empty_time(&cfqg->stats); cfqg_stats_set_start_group_wait_time(cfqg, curr_cfqg); } @@ -676,26 +677,30 @@ static inline void cfqg_stats_update_timeslice_used(struct cfq_group *cfqg, #endif } -static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, int rw) +static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, int op, + int op_flags) { - blkg_rwstat_add(&cfqg->stats.queued, rw, -1); + blkg_rwstat_add(&cfqg->stats.queued, op, op_flags, -1); } -static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, int rw) +static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, int op, + int op_flags) { - blkg_rwstat_add(&cfqg->stats.merged, rw, 1); + blkg_rwstat_add(&cfqg->stats.merged, op, op_flags, 1); } static inline void cfqg_stats_update_completion(struct cfq_group *cfqg, - uint64_t start_time, uint64_t io_start_time, int rw) + uint64_t start_time, uint64_t io_start_time, int op, + int op_flags) { struct cfqg_stats *stats = &cfqg->stats; unsigned long long now = sched_clock(); if (time_after64(now, io_start_time)) - blkg_rwstat_add(&stats->service_time, rw, now - io_start_time); + blkg_rwstat_add(&stats->service_time, op, op_flags, + now - io_start_time); if (time_after64(io_start_time, start_time)) - blkg_rwstat_add(&stats->wait_time, rw, + blkg_rwstat_add(&stats->wait_time, op, op_flags, io_start_time - start_time); } @@ -769,13 +774,16 @@ static inline void cfqg_put(struct cfq_group *cfqg) { } #define cfq_log_cfqg(cfqd, cfqg, fmt, args...) do {} while (0) static inline void cfqg_stats_update_io_add(struct cfq_group *cfqg, - struct cfq_group *curr_cfqg, int rw) { } + struct cfq_group *curr_cfqg, int op, int op_flags) { } static inline void cfqg_stats_update_timeslice_used(struct cfq_group *cfqg, unsigned long time, unsigned long unaccounted_time) { } -static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, int rw) { } -static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, int rw) { } +static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, int op, + int op_flags) { } +static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, int op, + int op_flags) { } static inline void cfqg_stats_update_completion(struct cfq_group *cfqg, - uint64_t start_time, uint64_t io_start_time, int rw) { } + uint64_t start_time, uint64_t io_start_time, int op, + int op_flags) { } #endif /* CONFIG_CFQ_GROUP_IOSCHED */ @@ -2449,10 +2457,10 @@ static void cfq_reposition_rq_rb(struct cfq_queue *cfqq, struct request *rq) { elv_rb_del(&cfqq->sort_list, rq); cfqq->queued[rq_is_sync(rq)]--; - cfqg_stats_update_io_remove(RQ_CFQG(rq), rq->cmd_flags); + cfqg_stats_update_io_remove(RQ_CFQG(rq), rq->op, rq->cmd_flags); cfq_add_rq_rb(rq); cfqg_stats_update_io_add(RQ_CFQG(rq), cfqq->cfqd->serving_group, - rq->cmd_flags); + rq->op, rq->cmd_flags); } static struct request * @@ -2505,7 +2513,7 @@ static void cfq_remove_request(struct request *rq) cfq_del_rq_rb(rq); cfqq->cfqd->rq_queued--; - cfqg_stats_update_io_remove(RQ_CFQG(rq), rq->cmd_flags); + cfqg_stats_update_io_remove(RQ_CFQG(rq), rq->op, rq->cmd_flags); if (rq->cmd_flags & REQ_PRIO) { WARN_ON(!cfqq->prio_pending); cfqq->prio_pending--; @@ -2540,7 +2548,7 @@ static void cfq_merged_request(struct request_queue *q, struct request *req, static void cfq_bio_merged(struct request_queue *q, struct request *req, struct bio *bio) { - cfqg_stats_update_io_merged(RQ_CFQG(req), bio->bi_rw); + cfqg_stats_update_io_merged(RQ_CFQG(req), bio->bi_op, bio->bi_rw); } static void @@ -2563,7 +2571,7 @@ cfq_merged_requests(struct request_queue *q, struct request *rq, if (cfqq->next_rq == next) cfqq->next_rq = rq; cfq_remove_request(next); - cfqg_stats_update_io_merged(RQ_CFQG(rq), next->cmd_flags); + cfqg_stats_update_io_merged(RQ_CFQG(rq), next->op, next->cmd_flags); cfqq = RQ_CFQQ(next); /* @@ -4085,7 +4093,7 @@ static void cfq_insert_request(struct request_queue *q, struct request *rq) rq->fifo_time = jiffies + cfqd->cfq_fifo_expire[rq_is_sync(rq)]; list_add_tail(&rq->queuelist, &cfqq->fifo); cfq_add_rq_rb(rq); - cfqg_stats_update_io_add(RQ_CFQG(rq), cfqd->serving_group, + cfqg_stats_update_io_add(RQ_CFQG(rq), cfqd->serving_group, rq->op, rq->cmd_flags); cfq_rq_enqueued(cfqd, cfqq, rq); } @@ -4183,7 +4191,8 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq) cfqq->dispatched--; (RQ_CFQG(rq))->dispatched--; cfqg_stats_update_completion(cfqq->cfqg, rq_start_time_ns(rq), - rq_io_start_time_ns(rq), rq->cmd_flags); + rq_io_start_time_ns(rq), rq->op, + rq->cmd_flags); cfqd->rq_in_flight[cfq_cfqq_sync(cfqq)]--; @@ -4262,7 +4271,7 @@ static inline int __cfq_may_queue(struct cfq_queue *cfqq) return ELV_MQUEUE_MAY; } -static int cfq_may_queue(struct request_queue *q, int rw) +static int cfq_may_queue(struct request_queue *q, int op, int op_flags) { struct cfq_data *cfqd = q->elevator->elevator_data; struct task_struct *tsk = current; @@ -4279,7 +4288,7 @@ static int cfq_may_queue(struct request_queue *q, int rw) if (!cic) return ELV_MQUEUE_MAY; - cfqq = cic_to_cfqq(cic, rw_is_sync(rw)); + cfqq = cic_to_cfqq(cic, rw_is_sync(op, op_flags)); if (cfqq) { cfq_init_prio_data(cfqq, cic); diff --git a/block/elevator.c b/block/elevator.c index c3555c9..f6279ca 100644 --- a/block/elevator.c +++ b/block/elevator.c @@ -366,8 +366,8 @@ void elv_dispatch_sort(struct request_queue *q, struct request *rq) list_for_each_prev(entry, &q->queue_head) { struct request *pos = list_entry_rq(entry); - if ((rq->cmd_flags & REQ_DISCARD) != - (pos->cmd_flags & REQ_DISCARD)) + if ((rq->op == REQ_OP_DISCARD) != + (pos->op == REQ_OP_DISCARD)) break; if (rq_data_dir(rq) != rq_data_dir(pos)) break; @@ -717,12 +717,12 @@ void elv_put_request(struct request_queue *q, struct request *rq) e->type->ops.elevator_put_req_fn(rq); } -int elv_may_queue(struct request_queue *q, int rw) +int elv_may_queue(struct request_queue *q, int op, int op_flags) { struct elevator_queue *e = q->elevator; if (e->type->ops.elevator_may_queue_fn) - return e->type->ops.elevator_may_queue_fn(q, rw); + return e->type->ops.elevator_may_queue_fn(q, op, op_flags); return ELV_MQUEUE_MAY; } diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index c02e669..9071feb 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -590,25 +590,26 @@ static inline void blkg_rwstat_exit(struct blkg_rwstat *rwstat) /** * blkg_rwstat_add - add a value to a blkg_rwstat * @rwstat: target blkg_rwstat - * @rw: mask of REQ_{WRITE|SYNC} + * @op: REQ_OP + * @op_flags: rq_flag_bits * @val: value to add * * Add @val to @rwstat. The counters are chosen according to @rw. The * caller is responsible for synchronizing calls to this function. */ static inline void blkg_rwstat_add(struct blkg_rwstat *rwstat, - int rw, uint64_t val) + int op, int op_flags, uint64_t val) { struct percpu_counter *cnt; - if (rw & REQ_WRITE) + if (op_is_write(op)) cnt = &rwstat->cpu_cnt[BLKG_RWSTAT_WRITE]; else cnt = &rwstat->cpu_cnt[BLKG_RWSTAT_READ]; __percpu_counter_add(cnt, val, BLKG_STAT_CPU_BATCH); - if (rw & REQ_SYNC) + if (op_flags & REQ_SYNC) cnt = &rwstat->cpu_cnt[BLKG_RWSTAT_SYNC]; else cnt = &rwstat->cpu_cnt[BLKG_RWSTAT_ASYNC]; @@ -713,9 +714,9 @@ static inline bool blkcg_bio_issue_check(struct request_queue *q, if (!throtl) { blkg = blkg ?: q->root_blkg; - blkg_rwstat_add(&blkg->stat_bytes, bio->bi_rw, + blkg_rwstat_add(&blkg->stat_bytes, bio->bi_op, bio->bi_rw, bio->bi_iter.bi_size); - blkg_rwstat_add(&blkg->stat_ios, bio->bi_rw, 1); + blkg_rwstat_add(&blkg->stat_ios, bio->bi_op, bio->bi_rw, 1); } rcu_read_unlock(); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 35b9eb3..bc024c7 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -596,7 +596,7 @@ static inline void queue_flag_clear(unsigned int flag, struct request_queue *q) #define list_entry_rq(ptr) list_entry((ptr), struct request, queuelist) -#define rq_data_dir(rq) ((int)((rq)->cmd_flags & 1)) +#define rq_data_dir(rq) (op_is_write(rq->op) ? WRITE : READ) /* * Driver can handle struct request, if it either has an old style @@ -615,14 +615,14 @@ static inline unsigned int blk_queue_cluster(struct request_queue *q) /* * We regard a request as sync, if either a read or a sync write */ -static inline bool rw_is_sync(unsigned int rw_flags) +static inline bool rw_is_sync(int op, unsigned int rw_flags) { - return !(rw_flags & REQ_WRITE) || (rw_flags & REQ_SYNC); + return op == REQ_OP_READ || (rw_flags & REQ_SYNC); } static inline bool rq_is_sync(struct request *rq) { - return rw_is_sync(rq->cmd_flags); + return rw_is_sync(rq->op, rq->cmd_flags); } static inline bool blk_rl_full(struct request_list *rl, bool sync) @@ -657,16 +657,16 @@ static inline bool rq_mergeable(struct request *rq) return true; } -static inline bool blk_check_merge_flags(unsigned int flags1, - unsigned int flags2) +static inline bool blk_check_merge_flags(unsigned int flags1, unsigned int op1, + unsigned int flags2, unsigned int op2) { - if ((flags1 & REQ_DISCARD) != (flags2 & REQ_DISCARD)) + if ((op1 == REQ_OP_DISCARD) != (op2 == REQ_OP_DISCARD)) return false; if ((flags1 & REQ_SECURE) != (flags2 & REQ_SECURE)) return false; - if ((flags1 & REQ_WRITE_SAME) != (flags2 & REQ_WRITE_SAME)) + if ((op1 == REQ_OP_WRITE_SAME) != (op2 == REQ_OP_WRITE_SAME)) return false; return true; @@ -864,12 +864,12 @@ static inline unsigned int blk_rq_cur_sectors(const struct request *rq) } static inline unsigned int blk_queue_get_max_sectors(struct request_queue *q, - unsigned int cmd_flags) + int op) { - if (unlikely(cmd_flags & REQ_DISCARD)) + if (unlikely(op == REQ_OP_DISCARD)) return min(q->limits.max_discard_sectors, UINT_MAX >> 9); - if (unlikely(cmd_flags & REQ_WRITE_SAME)) + if (unlikely(op == REQ_OP_WRITE_SAME)) return q->limits.max_write_same_sectors; return q->limits.max_sectors; @@ -896,11 +896,11 @@ static inline unsigned int blk_rq_get_max_sectors(struct request *rq) if (unlikely(rq->cmd_type == REQ_TYPE_BLOCK_PC)) return q->limits.max_hw_sectors; - if (!q->limits.chunk_sectors || (rq->cmd_flags & REQ_DISCARD)) - return blk_queue_get_max_sectors(q, rq->cmd_flags); + if (!q->limits.chunk_sectors || (rq->op == REQ_OP_DISCARD)) + return blk_queue_get_max_sectors(q, rq->op); return min(blk_max_size_offset(q, blk_rq_pos(rq)), - blk_queue_get_max_sectors(q, rq->cmd_flags)); + blk_queue_get_max_sectors(q, rq->op)); } static inline unsigned int blk_rq_count_bios(struct request *rq) diff --git a/include/linux/elevator.h b/include/linux/elevator.h index 638b324..953d286 100644 --- a/include/linux/elevator.h +++ b/include/linux/elevator.h @@ -26,7 +26,7 @@ typedef int (elevator_dispatch_fn) (struct request_queue *, int); typedef void (elevator_add_req_fn) (struct request_queue *, struct request *); typedef struct request *(elevator_request_list_fn) (struct request_queue *, struct request *); typedef void (elevator_completed_req_fn) (struct request_queue *, struct request *); -typedef int (elevator_may_queue_fn) (struct request_queue *, int); +typedef int (elevator_may_queue_fn) (struct request_queue *, int, int); typedef void (elevator_init_icq_fn) (struct io_cq *); typedef void (elevator_exit_icq_fn) (struct io_cq *); @@ -134,7 +134,7 @@ extern struct request *elv_former_request(struct request_queue *, struct request extern struct request *elv_latter_request(struct request_queue *, struct request *); extern int elv_register_queue(struct request_queue *q); extern void elv_unregister_queue(struct request_queue *q); -extern int elv_may_queue(struct request_queue *, int); +extern int elv_may_queue(struct request_queue *, int, int); extern void elv_completed_request(struct request_queue *, struct request *); extern int elv_set_request(struct request_queue *q, struct request *rq, struct bio *bio, gfp_t gfp_mask); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 754F229E05 for ; Tue, 5 Jan 2016 14:55:03 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 58A178F8035 for ; Tue, 5 Jan 2016 12:55:03 -0800 (PST) X-ASG-Debug-ID: 1452027301-04cb6c42e8052c0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id zdxCRtNPdECxtIQK (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:01 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 0A3FF461C8; Tue, 5 Jan 2016 20:55:01 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuQ024032; Tue, 5 Jan 2016 15:54:58 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 27/35] drivers: set request op to REQ_OP Date: Tue, 5 Jan 2016 14:53:30 -0600 X-ASG-Orig-Subj: [PATCH 27/35] drivers: set request op to REQ_OP Message-Id: <1452027218-32303-28-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027301 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the block driver use the request->op for REQ_OP operations and cmd_flags for rq_flag_bits. I have only tested scsi and rbd. Signed-off-by: Mike Christie --- drivers/block/loop.c | 6 +++--- drivers/block/mtip32xx/mtip32xx.c | 2 +- drivers/block/nbd.c | 2 +- drivers/block/rbd.c | 2 +- drivers/block/skd_main.c | 11 ++++------- drivers/block/xen-blkfront.c | 8 +++++--- drivers/md/dm.c | 2 +- drivers/mmc/card/block.c | 7 +++---- drivers/mmc/card/queue.c | 6 ++---- drivers/mmc/card/queue.h | 5 ++++- drivers/mtd/mtd_blkdevs.c | 2 +- drivers/nvme/host/pci.c | 4 ++-- drivers/scsi/sd.c | 25 ++++++++++++++++--------- 13 files changed, 44 insertions(+), 38 deletions(-) diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 423f4ca..e771bab 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -538,7 +538,7 @@ static int do_req_filebacked(struct loop_device *lo, struct request *rq) if (rq->cmd_flags & REQ_WRITE) { if (rq->cmd_flags & REQ_FLUSH) ret = lo_req_flush(lo, rq); - else if (rq->cmd_flags & REQ_DISCARD) + else if (rq->op == REQ_OP_DISCARD) ret = lo_discard(lo, rq, pos); else if (lo->transfer) ret = lo_write_transfer(lo, rq, pos); @@ -1653,8 +1653,8 @@ static int loop_queue_rq(struct blk_mq_hw_ctx *hctx, if (lo->lo_state != Lo_bound) return -EIO; - if (lo->use_dio && !(cmd->rq->cmd_flags & (REQ_FLUSH | - REQ_DISCARD))) + if (lo->use_dio && (!(cmd->rq->cmd_flags & REQ_FLUSH) || + cmd->rq->op == REQ_OP_DISCARD)) cmd->use_aio = true; else cmd->use_aio = false; diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c index 618c24f..8751caa 100644 --- a/drivers/block/mtip32xx/mtip32xx.c +++ b/drivers/block/mtip32xx/mtip32xx.c @@ -3668,7 +3668,7 @@ static int mtip_submit_request(struct blk_mq_hw_ctx *hctx, struct request *rq) return -ENXIO; } - if (rq->cmd_flags & REQ_DISCARD) { + if (rq->op == REQ_OP_DISCARD) { int err; err = mtip_send_trim(dd, blk_rq_pos(rq), blk_rq_sectors(rq)); diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index 93b3f99..8e8f7e3 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -242,7 +242,7 @@ static int nbd_send_req(struct nbd_device *nbd, struct request *req) if (req->cmd_type == REQ_TYPE_DRV_PRIV) type = NBD_CMD_DISC; - else if (req->cmd_flags & REQ_DISCARD) + else if (req->op == REQ_OP_DISCARD) type = NBD_CMD_TRIM; else if (req->cmd_flags & REQ_FLUSH) type = NBD_CMD_FLUSH; diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 81ea69f..ea326ef 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -3373,7 +3373,7 @@ static void rbd_queue_workfn(struct work_struct *work) goto err; } - if (rq->cmd_flags & REQ_DISCARD) + if (rq->op == REQ_OP_DISCARD) op_type = OBJ_OP_DISCARD; else if (rq->cmd_flags & REQ_WRITE) op_type = OBJ_OP_WRITE; diff --git a/drivers/block/skd_main.c b/drivers/block/skd_main.c index 586f916..f89a0c8 100644 --- a/drivers/block/skd_main.c +++ b/drivers/block/skd_main.c @@ -576,7 +576,6 @@ static void skd_request_fn(struct request_queue *q) struct request *req = NULL; struct skd_scsi_request *scsi_req; struct page *page; - unsigned long io_flags; int error; u32 lba; u32 count; @@ -624,12 +623,11 @@ static void skd_request_fn(struct request_queue *q) lba = (u32)blk_rq_pos(req); count = blk_rq_sectors(req); data_dir = rq_data_dir(req); - io_flags = req->cmd_flags; - if (io_flags & REQ_FLUSH) + if (req->cmd_flags & REQ_FLUSH) flush++; - if (io_flags & REQ_FUA) + if (req->cmd_flags & REQ_FUA) fua++; pr_debug("%s:%s:%d new req=%p lba=%u(0x%x) " @@ -735,7 +733,7 @@ static void skd_request_fn(struct request_queue *q) else skreq->sg_data_dir = SKD_DATA_DIR_HOST_TO_CARD; - if (io_flags & REQ_DISCARD) { + if (req->op == REQ_OP_DISCARD) { page = alloc_page(GFP_ATOMIC | __GFP_ZERO); if (!page) { pr_err("request_fn:Page allocation failed.\n"); @@ -852,9 +850,8 @@ static void skd_end_request(struct skd_device *skdev, struct skd_request_context *skreq, int error) { struct request *req = skreq->req; - unsigned int io_flags = req->cmd_flags; - if ((io_flags & REQ_DISCARD) && + if ((req->op == REQ_OP_DISCARD) && (skreq->discard_page == 1)) { pr_debug("%s:%s:%d, free the page!", skdev->name, __func__, __LINE__); diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 81444fc..d4aa473 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -718,7 +718,8 @@ static int blkif_queue_request(struct request *req) if (unlikely(info->connected != BLKIF_STATE_CONNECTED)) return 1; - if (unlikely(req->cmd_flags & (REQ_DISCARD | REQ_SECURE))) + if (unlikely(req->op == REQ_OP_DISCARD || + req->cmd_flags & REQ_SECURE)) return blkif_queue_discard_req(req); else return blkif_queue_rw_req(req); @@ -1691,8 +1692,9 @@ static int blkif_recover(struct blkfront_info *info) /* * Get the bios in the request so we can re-queue them. */ - if (copy[i].request->cmd_flags & - (REQ_FLUSH | REQ_FUA | REQ_DISCARD | REQ_SECURE)) { + if (copy[i].request->cmd_flags & REQ_FLUSH || + copy[i].request->op == REQ_OP_DISCARD || + copy[i].request->cmd_flags & (REQ_FUA | REQ_SECURE)) { /* * Flush operations don't contain bios, so * we need to requeue the whole request diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 5dbdae7..7a85017 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1276,7 +1276,7 @@ static void dm_done(struct request *clone, int error, bool mapped) r = rq_end_io(tio->ti, clone, error, &tio->info); } - if (unlikely(r == -EREMOTEIO && (clone->cmd_flags & REQ_WRITE_SAME) && + if (unlikely(r == -EREMOTEIO && (clone->op == REQ_OP_WRITE_SAME) && !clone->q->limits.max_write_same_sectors)) disable_write_same(tio->md); diff --git a/drivers/mmc/card/block.c b/drivers/mmc/card/block.c index 5914263..cc9b1ff 100644 --- a/drivers/mmc/card/block.c +++ b/drivers/mmc/card/block.c @@ -1691,8 +1691,7 @@ static u8 mmc_blk_prep_packed_list(struct mmc_queue *mq, struct request *req) !IS_ALIGNED(blk_rq_sectors(next), 8)) break; - if (next->cmd_flags & REQ_DISCARD || - next->cmd_flags & REQ_FLUSH) + if (next->op == REQ_OP_DISCARD || next->cmd_flags & REQ_FLUSH) break; if (rq_data_dir(cur) != rq_data_dir(next)) @@ -2133,7 +2132,7 @@ static int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) } mq->flags &= ~MMC_QUEUE_NEW_REQUEST; - if (cmd_flags & REQ_DISCARD) { + if (req && req->op == REQ_OP_DISCARD) { /* complete ongoing async transfer before issuing discard */ if (card->host->areq) mmc_blk_issue_rw_rq(mq, NULL); @@ -2157,7 +2156,7 @@ static int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) out: if ((!req && !(mq->flags & MMC_QUEUE_NEW_REQUEST)) || - (cmd_flags & MMC_REQ_SPECIAL_MASK)) + mmc_req_is_special(req)) /* * Release host when there are no more requests * and after special request(discard, flush) is done. diff --git a/drivers/mmc/card/queue.c b/drivers/mmc/card/queue.c index 6f4323c..9fb8d21 100644 --- a/drivers/mmc/card/queue.c +++ b/drivers/mmc/card/queue.c @@ -33,7 +33,7 @@ static int mmc_prep_request(struct request_queue *q, struct request *req) /* * We only like normal block requests and discards. */ - if (req->cmd_type != REQ_TYPE_FS && !(req->cmd_flags & REQ_DISCARD)) { + if (req->cmd_type != REQ_TYPE_FS && req->op != REQ_OP_DISCARD) { blk_dump_rq_flags(req, "MMC bad request"); return BLKPREP_KILL; } @@ -56,7 +56,6 @@ static int mmc_queue_thread(void *d) down(&mq->thread_sem); do { struct request *req = NULL; - unsigned int cmd_flags = 0; spin_lock_irq(q->queue_lock); set_current_state(TASK_INTERRUPTIBLE); @@ -66,7 +65,6 @@ static int mmc_queue_thread(void *d) if (req || mq->mqrq_prev->req) { set_current_state(TASK_RUNNING); - cmd_flags = req ? req->cmd_flags : 0; mq->issue_fn(mq, req); cond_resched(); if (mq->flags & MMC_QUEUE_NEW_REQUEST) { @@ -81,7 +79,7 @@ static int mmc_queue_thread(void *d) * has been finished. Do not assign it to previous * request. */ - if (cmd_flags & MMC_REQ_SPECIAL_MASK) + if (mmc_req_is_special(req)) mq->mqrq_cur->req = NULL; mq->mqrq_prev->brq.mrq.data = NULL; diff --git a/drivers/mmc/card/queue.h b/drivers/mmc/card/queue.h index 36cddab..f166e5b 100644 --- a/drivers/mmc/card/queue.h +++ b/drivers/mmc/card/queue.h @@ -1,7 +1,10 @@ #ifndef MMC_QUEUE_H #define MMC_QUEUE_H -#define MMC_REQ_SPECIAL_MASK (REQ_DISCARD | REQ_FLUSH) +static inline bool mmc_req_is_special(struct request *req) +{ + return req && (req->cmd_flags & REQ_FLUSH || req->op == REQ_OP_DISCARD); +} struct request; struct task_struct; diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c index f470118..b2e0bce 100644 --- a/drivers/mtd/mtd_blkdevs.c +++ b/drivers/mtd/mtd_blkdevs.c @@ -94,7 +94,7 @@ static int do_blktrans_request(struct mtd_blktrans_ops *tr, get_capacity(req->rq_disk)) return -EIO; - if (req->cmd_flags & REQ_DISCARD) + if (req->op == REQ_OP_DISCARD) return tr->discard(dev, block, nsect); if (rq_data_dir(req) == READ) { diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 92fa1a6..c51205a 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -347,7 +347,7 @@ static int nvme_init_iod(struct request *rq, struct nvme_dev *dev) int nseg = rq->nr_phys_segments; unsigned size; - if (rq->cmd_flags & REQ_DISCARD) + if (rq->op == REQ_OP_DISCARD) size = sizeof(struct nvme_dsm_range); else size = blk_rq_bytes(rq); @@ -663,7 +663,7 @@ static int nvme_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) return ret; - if (req->cmd_flags & REQ_DISCARD) { + if (req->op == REQ_OP_DISCARD) { ret = nvme_setup_discard(nvmeq, ns, req, &cmnd); } else { if (req->cmd_type == REQ_TYPE_DRV_PRIV) diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 4e08d1cd..2737527 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -1012,7 +1012,8 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) } else if (rq_data_dir(rq) == READ) { SCpnt->cmnd[0] = READ_6; } else { - scmd_printk(KERN_ERR, SCpnt, "Unknown command %llx\n", (unsigned long long) rq->cmd_flags); + scmd_printk(KERN_ERR, SCpnt, "Unknown command %d,%llx\n", rq->op, + (unsigned long long) rq->cmd_flags); goto out; } @@ -1137,21 +1138,27 @@ static int sd_init_command(struct scsi_cmnd *cmd) { struct request *rq = cmd->request; - if (rq->cmd_flags & REQ_DISCARD) + switch (rq->op) { + case REQ_OP_DISCARD: return sd_setup_discard_cmnd(cmd); - else if (rq->cmd_flags & REQ_WRITE_SAME) + case REQ_OP_WRITE_SAME: return sd_setup_write_same_cmnd(cmd); - else if (rq->cmd_flags & REQ_FLUSH) - return sd_setup_flush_cmnd(cmd); - else - return sd_setup_read_write_cmnd(cmd); + case REQ_OP_READ: + case REQ_OP_WRITE: + if (rq->cmd_flags & REQ_FLUSH) + return sd_setup_flush_cmnd(cmd); + else + return sd_setup_read_write_cmnd(cmd); + default: + BUG(); + } } static void sd_uninit_command(struct scsi_cmnd *SCpnt) { struct request *rq = SCpnt->request; - if (rq->cmd_flags & REQ_DISCARD) + if (rq->op == REQ_OP_DISCARD) __free_page(rq->completion_data); if (SCpnt->cmnd != rq->cmd) { @@ -1768,7 +1775,7 @@ static int sd_done(struct scsi_cmnd *SCpnt) unsigned char op = SCpnt->cmnd[0]; unsigned char unmap = SCpnt->cmnd[1] & 8; - if (req->cmd_flags & REQ_DISCARD || req->cmd_flags & REQ_WRITE_SAME) { + if (req->op == REQ_OP_DISCARD || req->op == REQ_OP_WRITE_SAME) { if (!result) { good_bytes = blk_rq_bytes(req); scsi_set_resid(SCpnt, 0); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:05 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id EDDE329E41 for ; Tue, 5 Jan 2016 14:55:05 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 9B923304053 for ; Tue, 5 Jan 2016 12:55:05 -0800 (PST) X-ASG-Debug-ID: 1452027303-04bdf03afa04e60001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id qIC4Dt4P4NqHGC6y (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:04 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id AEF03C0B7E0C; Tue, 5 Jan 2016 20:55:03 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuR024032; Tue, 5 Jan 2016 15:55:01 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 28/35] blktrace: get op from req->op/bio->bi_op Date: Tue, 5 Jan 2016 14:53:31 -0600 X-ASG-Orig-Subj: [PATCH 28/35] blktrace: get op from req->op/bio->bi_op Message-Id: <1452027218-32303-29-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027303 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The bio and request struct now store the operation in bio->bi_op/request->op. This patch has blktrace not check bi_rw/cmd_flags. This patch is only compile tested. Signed-off-by: Mike Christie --- include/linux/blktrace_api.h | 2 +- include/trace/events/bcache.h | 12 ++++++---- include/trace/events/block.h | 31 +++++++++++++++++--------- kernel/trace/blktrace.c | 52 +++++++++++++++++++++++-------------------- 4 files changed, 57 insertions(+), 40 deletions(-) diff --git a/include/linux/blktrace_api.h b/include/linux/blktrace_api.h index afc1343..ee25ba4 100644 --- a/include/linux/blktrace_api.h +++ b/include/linux/blktrace_api.h @@ -109,7 +109,7 @@ static inline int blk_cmd_buf_len(struct request *rq) } extern void blk_dump_cmd(char *buf, struct request *rq); -extern void blk_fill_rwbs(char *rwbs, u32 rw, int bytes); +extern void blk_fill_rwbs(char *rwbs, int op, u32 rw, int bytes); #endif /* CONFIG_EVENT_TRACING && CONFIG_BLOCK */ diff --git a/include/trace/events/bcache.h b/include/trace/events/bcache.h index 981acf7..8abe564 100644 --- a/include/trace/events/bcache.h +++ b/include/trace/events/bcache.h @@ -27,7 +27,8 @@ DECLARE_EVENT_CLASS(bcache_request, __entry->sector = bio->bi_iter.bi_sector; __entry->orig_sector = bio->bi_iter.bi_sector - 16; __entry->nr_sector = bio->bi_iter.bi_size >> 9; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); ), TP_printk("%d,%d %s %llu + %u (from %d,%d @ %llu)", @@ -101,7 +102,8 @@ DECLARE_EVENT_CLASS(bcache_bio, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio->bi_iter.bi_size >> 9; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); ), TP_printk("%d,%d %s %llu + %u", @@ -136,7 +138,8 @@ TRACE_EVENT(bcache_read, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio->bi_iter.bi_size >> 9; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); __entry->cache_hit = hit; __entry->bypass = bypass; ), @@ -167,7 +170,8 @@ TRACE_EVENT(bcache_write, __entry->inode = inode; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio->bi_iter.bi_size >> 9; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); __entry->writeback = writeback; __entry->bypass = bypass; ), diff --git a/include/trace/events/block.h b/include/trace/events/block.h index e8a5eca..4416dcd 100644 --- a/include/trace/events/block.h +++ b/include/trace/events/block.h @@ -84,7 +84,8 @@ DECLARE_EVENT_CLASS(block_rq_with_error, 0 : blk_rq_sectors(rq); __entry->errors = rq->errors; - blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, blk_rq_bytes(rq)); + blk_fill_rwbs(__entry->rwbs, rq->op, rq->cmd_flags, + blk_rq_bytes(rq)); blk_dump_cmd(__get_str(cmd), rq); ), @@ -162,7 +163,7 @@ TRACE_EVENT(block_rq_complete, __entry->nr_sector = nr_bytes >> 9; __entry->errors = rq->errors; - blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, nr_bytes); + blk_fill_rwbs(__entry->rwbs, rq->op, rq->cmd_flags, nr_bytes); blk_dump_cmd(__get_str(cmd), rq); ), @@ -198,7 +199,8 @@ DECLARE_EVENT_CLASS(block_rq, __entry->bytes = (rq->cmd_type == REQ_TYPE_BLOCK_PC) ? blk_rq_bytes(rq) : 0; - blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, blk_rq_bytes(rq)); + blk_fill_rwbs(__entry->rwbs, rq->op, rq->cmd_flags, + blk_rq_bytes(rq)); blk_dump_cmd(__get_str(cmd), rq); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -272,7 +274,8 @@ TRACE_EVENT(block_bio_bounce, bio->bi_bdev->bd_dev : 0; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio_sectors(bio); - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -310,7 +313,8 @@ TRACE_EVENT(block_bio_complete, __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio_sectors(bio); __entry->error = error; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); ), TP_printk("%d,%d %s %llu + %u [%d]", @@ -337,7 +341,8 @@ DECLARE_EVENT_CLASS(block_bio_merge, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio_sectors(bio); - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -404,7 +409,8 @@ TRACE_EVENT(block_bio_queue, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio_sectors(bio); - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -432,7 +438,7 @@ DECLARE_EVENT_CLASS(block_get_rq, __entry->dev = bio ? bio->bi_bdev->bd_dev : 0; __entry->sector = bio ? bio->bi_iter.bi_sector : 0; __entry->nr_sector = bio ? bio_sectors(bio) : 0; - blk_fill_rwbs(__entry->rwbs, + blk_fill_rwbs(__entry->rwbs, bio ? bio->bi_op : 0, bio ? bio->bi_rw : 0, __entry->nr_sector); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -567,7 +573,8 @@ TRACE_EVENT(block_split, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->new_sector = new_sector; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -610,7 +617,8 @@ TRACE_EVENT(block_bio_remap, __entry->nr_sector = bio_sectors(bio); __entry->old_dev = dev; __entry->old_sector = from; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); ), TP_printk("%d,%d %s %llu + %u <- (%d,%d) %llu", @@ -656,7 +664,8 @@ TRACE_EVENT(block_rq_remap, __entry->old_dev = dev; __entry->old_sector = from; __entry->nr_bios = blk_rq_count_bios(rq); - blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, blk_rq_bytes(rq)); + blk_fill_rwbs(__entry->rwbs, rq->op, rq->cmd_flags, + blk_rq_bytes(rq)); ), TP_printk("%d,%d %s %llu + %u <- (%d,%d) %llu %u", diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index 2aeb6ff..4a9ac83 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -199,7 +199,8 @@ static const u32 ddir_act[2] = { BLK_TC_ACT(BLK_TC_READ), * blk_io_trace structure and places it in a per-cpu subbuffer. */ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes, - int rw, u32 what, int error, int pdu_len, void *pdu_data) + int op, int op_flags, u32 what, int error, int pdu_len, + void *pdu_data) { struct task_struct *tsk = current; struct ring_buffer_event *event = NULL; @@ -214,13 +215,14 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes, if (unlikely(bt->trace_state != Blktrace_running && !blk_tracer)) return; - what |= ddir_act[rw & WRITE]; - what |= MASK_TC_BIT(rw, SYNC); - what |= MASK_TC_BIT(rw, RAHEAD); - what |= MASK_TC_BIT(rw, META); - what |= MASK_TC_BIT(rw, DISCARD); - what |= MASK_TC_BIT(rw, FLUSH); - what |= MASK_TC_BIT(rw, FUA); + what |= ddir_act[op_is_write(op) ? WRITE : READ]; + what |= MASK_TC_BIT(op_flags, SYNC); + what |= MASK_TC_BIT(op_flags, RAHEAD); + what |= MASK_TC_BIT(op_flags, META); + what |= MASK_TC_BIT(op_flags, FLUSH); + what |= MASK_TC_BIT(op_flags, FUA); + if (op == REQ_OP_DISCARD) + what |= BLK_TC_ACT(BLK_TC_DISCARD); pid = tsk->pid; if (act_log_check(bt, what, sector, pid)) @@ -708,11 +710,11 @@ static void blk_add_trace_rq(struct request_queue *q, struct request *rq, if (rq->cmd_type == REQ_TYPE_BLOCK_PC) { what |= BLK_TC_ACT(BLK_TC_PC); - __blk_add_trace(bt, 0, nr_bytes, rq->cmd_flags, + __blk_add_trace(bt, 0, nr_bytes, rq->op, rq->cmd_flags, what, rq->errors, rq->cmd_len, rq->cmd); } else { what |= BLK_TC_ACT(BLK_TC_FS); - __blk_add_trace(bt, blk_rq_pos(rq), nr_bytes, + __blk_add_trace(bt, blk_rq_pos(rq), nr_bytes, rq->op, rq->cmd_flags, what, rq->errors, 0, NULL); } } @@ -770,7 +772,7 @@ static void blk_add_trace_bio(struct request_queue *q, struct bio *bio, return; __blk_add_trace(bt, bio->bi_iter.bi_sector, bio->bi_iter.bi_size, - bio->bi_rw, what, error, 0, NULL); + bio->bi_op, bio->bi_rw, what, error, 0, NULL); } static void blk_add_trace_bio_bounce(void *ignore, @@ -818,7 +820,8 @@ static void blk_add_trace_getrq(void *ignore, struct blk_trace *bt = q->blk_trace; if (bt) - __blk_add_trace(bt, 0, 0, rw, BLK_TA_GETRQ, 0, 0, NULL); + __blk_add_trace(bt, 0, 0, rw, 0, BLK_TA_GETRQ, 0, 0, + NULL); } } @@ -833,7 +836,7 @@ static void blk_add_trace_sleeprq(void *ignore, struct blk_trace *bt = q->blk_trace; if (bt) - __blk_add_trace(bt, 0, 0, rw, BLK_TA_SLEEPRQ, + __blk_add_trace(bt, 0, 0, rw, 0, BLK_TA_SLEEPRQ, 0, 0, NULL); } } @@ -843,7 +846,7 @@ static void blk_add_trace_plug(void *ignore, struct request_queue *q) struct blk_trace *bt = q->blk_trace; if (bt) - __blk_add_trace(bt, 0, 0, 0, BLK_TA_PLUG, 0, 0, NULL); + __blk_add_trace(bt, 0, 0, 0, 0, BLK_TA_PLUG, 0, 0, NULL); } static void blk_add_trace_unplug(void *ignore, struct request_queue *q, @@ -860,7 +863,7 @@ static void blk_add_trace_unplug(void *ignore, struct request_queue *q, else what = BLK_TA_UNPLUG_TIMER; - __blk_add_trace(bt, 0, 0, 0, what, 0, sizeof(rpdu), &rpdu); + __blk_add_trace(bt, 0, 0, 0, 0, what, 0, sizeof(rpdu), &rpdu); } } @@ -874,8 +877,9 @@ static void blk_add_trace_split(void *ignore, __be64 rpdu = cpu_to_be64(pdu); __blk_add_trace(bt, bio->bi_iter.bi_sector, - bio->bi_iter.bi_size, bio->bi_rw, BLK_TA_SPLIT, - bio->bi_error, sizeof(rpdu), &rpdu); + bio->bi_iter.bi_size, bio->bi_op, bio->bi_rw, + BLK_TA_SPLIT, bio->bi_error, sizeof(rpdu), + &rpdu); } } @@ -907,7 +911,7 @@ static void blk_add_trace_bio_remap(void *ignore, r.sector_from = cpu_to_be64(from); __blk_add_trace(bt, bio->bi_iter.bi_sector, bio->bi_iter.bi_size, - bio->bi_rw, BLK_TA_REMAP, bio->bi_error, + bio->bi_op, bio->bi_rw, BLK_TA_REMAP, bio->bi_error, sizeof(r), &r); } @@ -940,7 +944,7 @@ static void blk_add_trace_rq_remap(void *ignore, r.sector_from = cpu_to_be64(from); __blk_add_trace(bt, blk_rq_pos(rq), blk_rq_bytes(rq), - rq_data_dir(rq), BLK_TA_REMAP, !!rq->errors, + rq_data_dir(rq), 0, BLK_TA_REMAP, !!rq->errors, sizeof(r), &r); } @@ -965,10 +969,10 @@ void blk_add_driver_data(struct request_queue *q, return; if (rq->cmd_type == REQ_TYPE_BLOCK_PC) - __blk_add_trace(bt, 0, blk_rq_bytes(rq), 0, + __blk_add_trace(bt, 0, blk_rq_bytes(rq), 0, 0, BLK_TA_DRV_DATA, rq->errors, len, data); else - __blk_add_trace(bt, blk_rq_pos(rq), blk_rq_bytes(rq), 0, + __blk_add_trace(bt, blk_rq_pos(rq), blk_rq_bytes(rq), 0, 0, BLK_TA_DRV_DATA, rq->errors, len, data); } EXPORT_SYMBOL_GPL(blk_add_driver_data); @@ -1767,16 +1771,16 @@ void blk_dump_cmd(char *buf, struct request *rq) } } -void blk_fill_rwbs(char *rwbs, u32 rw, int bytes) +void blk_fill_rwbs(char *rwbs, int op, u32 rw, int bytes) { int i = 0; if (rw & REQ_FLUSH) rwbs[i++] = 'F'; - if (rw & WRITE) + if (op == REQ_OP_WRITE) rwbs[i++] = 'W'; - else if (rw & REQ_DISCARD) + else if (op == REQ_OP_DISCARD) rwbs[i++] = 'D'; else if (bytes) rwbs[i++] = 'R'; -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 1FB3929E62 for ; Tue, 5 Jan 2016 14:55:08 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 00D4E8F804C for ; Tue, 5 Jan 2016 12:55:07 -0800 (PST) X-ASG-Debug-ID: 1452027306-04cbb07e1a054c0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id kXYoD3gGGhlzH3Im (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:07 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 60ACDC00330F; Tue, 5 Jan 2016 20:55:06 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuS024032; Tue, 5 Jan 2016 15:55:03 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 29/35] ide cd: do not set REQ_WRITE on requests. Date: Tue, 5 Jan 2016 14:53:32 -0600 X-ASG-Orig-Subj: [PATCH 29/35] ide cd: do not set REQ_WRITE on requests. Message-Id: <1452027218-32303-30-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027306 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The block layer will set the correct READ/WRITE operation flags/fields when creating a request, so there is not need for drivers to set the REQ_WRITE flag. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/ide/ide-cd_ioctl.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/drivers/ide/ide-cd_ioctl.c b/drivers/ide/ide-cd_ioctl.c index 474173e..5887a7a 100644 --- a/drivers/ide/ide-cd_ioctl.c +++ b/drivers/ide/ide-cd_ioctl.c @@ -459,9 +459,6 @@ int ide_cdrom_packet(struct cdrom_device_info *cdi, layer. the packet must be complete, as we do not touch it at all. */ - if (cgc->data_direction == CGC_DATA_WRITE) - flags |= REQ_WRITE; - if (cgc->sense) memset(cgc->sense, 0, sizeof(struct request_sense)); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 097EA29E14 for ; Tue, 5 Jan 2016 14:55:13 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id EE7318F8035 for ; Tue, 5 Jan 2016 12:55:12 -0800 (PST) X-ASG-Debug-ID: 1452027309-04cbb07e16054c0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id yWWc3gn9KexNVOMg (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:10 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 3685413F5; Tue, 5 Jan 2016 20:55:09 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuT024032; Tue, 5 Jan 2016 15:55:06 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 30/35] block, fs, drivers: do not test bi_rw for REQ_OPs Date: Tue, 5 Jan 2016 14:53:33 -0600 X-ASG-Orig-Subj: [PATCH 30/35] block, fs, drivers: do not test bi_rw for REQ_OPs Message-Id: <1452027218-32303-31-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027309 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie We no longer use the bio->bi_rw field for REQ_OPs: REQ_WRITE, REQ_DISCARD, REQ_WRITE_SAME, so this patch stops checking for them in bi_rw and also removes the related compat code. Signed-off-by: Mike Christie --- block/bio.c | 6 +++--- block/blk-core.c | 31 ++++++++--------------------- block/blk-merge.c | 14 ++++++------- block/blk-mq.c | 3 +-- drivers/ata/libata-scsi.c | 2 +- drivers/block/brd.c | 2 +- drivers/block/drbd/drbd_main.c | 14 ++++++------- drivers/block/drbd/drbd_worker.c | 4 ++-- drivers/block/loop.c | 6 +++--- drivers/block/rbd.c | 2 +- drivers/block/rsxx/dma.c | 2 +- drivers/block/umem.c | 2 +- drivers/block/zram/zram_drv.c | 2 +- drivers/ide/ide-floppy.c | 2 +- drivers/lightnvm/rrpc.c | 2 +- drivers/md/bcache/request.c | 10 +++++----- drivers/md/dm-cache-target.c | 9 +++++---- drivers/md/dm-crypt.c | 2 +- drivers/md/dm-log-writes.c | 2 +- drivers/md/dm-raid1.c | 8 ++++---- drivers/md/dm-region-hash.c | 4 ++-- drivers/md/dm-stripe.c | 4 ++-- drivers/md/dm-thin.c | 15 ++++++++------ drivers/md/dm.c | 6 +++--- drivers/md/linear.c | 2 +- drivers/md/raid0.c | 2 +- drivers/scsi/osd/osd_initiator.c | 4 ++-- drivers/staging/lustre/lustre/llite/lloop.c | 8 ++++---- include/linux/bio.h | 15 +++++++++----- include/linux/fs.h | 25 ++++++++--------------- 30 files changed, 97 insertions(+), 113 deletions(-) diff --git a/block/bio.c b/block/bio.c index 3b8e970..ca0c52d 100644 --- a/block/bio.c +++ b/block/bio.c @@ -669,10 +669,10 @@ struct bio *bio_clone_bioset(struct bio *bio_src, gfp_t gfp_mask, bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; bio->bi_iter.bi_size = bio_src->bi_iter.bi_size; - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) goto integrity_clone; - if (bio->bi_rw & REQ_WRITE_SAME) { + if (bio->bi_op == REQ_OP_WRITE_SAME) { bio->bi_io_vec[bio->bi_vcnt++] = bio_src->bi_io_vec[0]; goto integrity_clone; } @@ -1792,7 +1792,7 @@ struct bio *bio_split(struct bio *bio, int sectors, * Discards need a mutable bio_vec to accommodate the payload * required by the DSM TRIM and UNMAP commands. */ - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) split = bio_clone_bioset(bio, gfp, bs); else split = bio_clone_fast(bio, gfp, bs); diff --git a/block/blk-core.c b/block/blk-core.c index dacbd68..0f6cb5c 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1704,8 +1704,7 @@ void init_request_from_bio(struct request *req, struct bio *bio) { req->cmd_type = REQ_TYPE_FS; - /* tmp compat. Allow users to set bi_op or bi_rw */ - req->cmd_flags |= (bio->bi_rw | bio->bi_op) & REQ_COMMON_MASK; + req->cmd_flags |= bio->bi_rw & REQ_COMMON_MASK; if (bio->bi_rw & REQ_RAHEAD) req->cmd_flags |= REQ_FAILFAST_MASK; @@ -1855,9 +1854,9 @@ static void handle_bad_sector(struct bio *bio) char b[BDEVNAME_SIZE]; printk(KERN_INFO "attempt to access beyond end of device\n"); - printk(KERN_INFO "%s: rw=%ld, want=%Lu, limit=%Lu\n", + printk(KERN_INFO "%s: rw=%d,%ld, want=%Lu, limit=%Lu\n", bdevname(bio->bi_bdev, b), - bio->bi_rw, + bio->bi_op, bio->bi_rw, (unsigned long long)bio_end_sector(bio), (long long)(i_size_read(bio->bi_bdev->bd_inode) >> 9)); } @@ -1978,14 +1977,14 @@ generic_make_request_checks(struct bio *bio) } } - if ((bio->bi_rw & REQ_DISCARD) && + if ((bio->bi_op == REQ_OP_DISCARD) && (!blk_queue_discard(q) || ((bio->bi_rw & REQ_SECURE) && !blk_queue_secdiscard(q)))) { err = -EOPNOTSUPP; goto end_io; } - if (bio->bi_rw & REQ_WRITE_SAME && !bdev_write_same(bio->bi_bdev)) { + if (bio->bi_op == REQ_OP_WRITE_SAME && !bdev_write_same(bio->bi_bdev)) { err = -EOPNOTSUPP; goto end_io; } @@ -2039,12 +2038,6 @@ blk_qc_t generic_make_request(struct bio *bio) struct bio_list bio_list_on_stack; blk_qc_t ret = BLK_QC_T_NONE; - /* tmp compat. Allow users to set either one or both. - * This will be removed when we have converted - * everyone in the next patches. - */ - bio->bi_rw |= bio->bi_op; - if (!generic_make_request_checks(bio)) goto out; @@ -2114,12 +2107,6 @@ EXPORT_SYMBOL(generic_make_request); */ blk_qc_t submit_bio(struct bio *bio) { - /* tmp compat. Allow users to set either one or both. - * This will be removed when we have converted - * everyone in the next patches. - */ - bio->bi_rw |= bio->bi_op; - /* * If it's a regular read/write or a barrier with data attached, * go through the normal accounting stuff before submission. @@ -2127,12 +2114,12 @@ blk_qc_t submit_bio(struct bio *bio) if (bio_has_data(bio)) { unsigned int count; - if (unlikely(bio->bi_rw & REQ_WRITE_SAME)) + if (unlikely(bio->bi_op == REQ_OP_WRITE_SAME)) count = bdev_logical_block_size(bio->bi_bdev) >> 9; else count = bio_sectors(bio); - if (bio->bi_rw & WRITE) { + if (op_is_write(bio->bi_op)) { count_vm_events(PGPGOUT, count); } else { task_io_account_read(bio->bi_iter.bi_size); @@ -2143,7 +2130,7 @@ blk_qc_t submit_bio(struct bio *bio) char b[BDEVNAME_SIZE]; printk(KERN_DEBUG "%s(%d): %s block %Lu on %s (%u sectors)\n", current->comm, task_pid_nr(current), - (bio->bi_rw & WRITE) ? "WRITE" : "READ", + op_is_write(bio->bi_op) ? "WRITE" : "READ", (unsigned long long)bio->bi_iter.bi_sector, bdevname(bio->bi_bdev, b), count); @@ -2991,8 +2978,6 @@ EXPORT_SYMBOL_GPL(__blk_end_request_err); void blk_rq_bio_prep(struct request_queue *q, struct request *rq, struct bio *bio) { - /* tmp compat. Allow users to set bi_op or bi_rw */ - rq->cmd_flags |= bio_data_dir(bio); rq->op = bio->bi_op; if (bio_has_data(bio)) diff --git a/block/blk-merge.c b/block/blk-merge.c index 4c47a43..8dff108 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -147,9 +147,9 @@ void blk_queue_split(struct request_queue *q, struct bio **bio, struct bio *split, *res; unsigned nsegs; - if ((*bio)->bi_rw & REQ_DISCARD) + if ((*bio)->bi_op == REQ_OP_DISCARD) split = blk_bio_discard_split(q, *bio, bs, &nsegs); - else if ((*bio)->bi_rw & REQ_WRITE_SAME) + else if ((*bio)->bi_op == REQ_OP_WRITE_SAME) split = blk_bio_write_same_split(q, *bio, bs, &nsegs); else split = blk_bio_segment_split(q, *bio, q->bio_split, &nsegs); @@ -188,10 +188,10 @@ static unsigned int __blk_recalc_rq_segments(struct request_queue *q, * This should probably be returning 0, but blk_add_request_payload() * (Christoph!!!!) */ - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) return 1; - if (bio->bi_rw & REQ_WRITE_SAME) + if (bio->bi_op == REQ_OP_WRITE_SAME) return 1; fbio = bio; @@ -364,7 +364,7 @@ static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio, nsegs = 0; cluster = blk_queue_cluster(q); - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { /* * This is a hack - drivers should be neither modifying the * biovec, nor relying on bi_vcnt - but because of @@ -379,7 +379,7 @@ static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio, return 0; } - if (bio->bi_rw & REQ_WRITE_SAME) { + if (bio->bi_op == REQ_OP_WRITE_SAME) { single_segment: *sg = sglist; bvec = bio_iovec(bio); @@ -418,7 +418,7 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq, } if (q->dma_drain_size && q->dma_drain_needed(rq)) { - if (rq->cmd_flags & REQ_WRITE) + if (op_is_write(rq->op)) memset(q->dma_drain_buffer, 0, q->dma_drain_size); sg_unmark_end(sg); diff --git a/block/blk-mq.c b/block/blk-mq.c index 745dae8..72a028b 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -170,8 +170,7 @@ static void blk_mq_rq_ctx_init(struct request_queue *q, struct blk_mq_ctx *ctx, rq->q = q; rq->mq_ctx = ctx; rq->op = op; - /* tmp compat - allow users to check either one for the op */ - rq->cmd_flags |= op | op_flags; + rq->cmd_flags |= op_flags; /* do not touch atomic flags, it needs atomic ops against the timer */ rq->cpu = -1; INIT_HLIST_NODE(&rq->hash); diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c index 7e959f9..05ab70e 100644 --- a/drivers/ata/libata-scsi.c +++ b/drivers/ata/libata-scsi.c @@ -1125,7 +1125,7 @@ static int atapi_drain_needed(struct request *rq) if (likely(rq->cmd_type != REQ_TYPE_BLOCK_PC)) return 0; - if (!blk_rq_bytes(rq) || (rq->cmd_flags & REQ_WRITE)) + if (!blk_rq_bytes(rq) || op_is_write(rq->op)) return 0; return atapi_cmd_type(rq->cmd[0]) == ATAPI_MISC; diff --git a/drivers/block/brd.c b/drivers/block/brd.c index de9cb2f..47a587a 100644 --- a/drivers/block/brd.c +++ b/drivers/block/brd.c @@ -339,7 +339,7 @@ static blk_qc_t brd_make_request(struct request_queue *q, struct bio *bio) if (bio_end_sector(bio) > get_capacity(bdev->bd_disk)) goto io_error; - if (unlikely(bio->bi_rw & REQ_DISCARD)) { + if (unlikely(bio->bi_op == REQ_OP_DISCARD)) { if (sector & ((PAGE_SIZE >> SECTOR_SHIFT) - 1) || bio->bi_iter.bi_size & PAGE_MASK) goto io_error; diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index f6626e5..df2ce0b 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1603,15 +1603,15 @@ static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device, return 0; } -static u32 bio_flags_to_wire(struct drbd_connection *connection, unsigned long bi_rw) +static u32 bio_flags_to_wire(struct drbd_connection *connection, struct bio *bio) { if (connection->agreed_pro_version >= 95) - return (bi_rw & REQ_SYNC ? DP_RW_SYNC : 0) | - (bi_rw & REQ_FUA ? DP_FUA : 0) | - (bi_rw & REQ_FLUSH ? DP_FLUSH : 0) | - (bi_rw & REQ_DISCARD ? DP_DISCARD : 0); + return (bio->bi_rw & REQ_SYNC ? DP_RW_SYNC : 0) | + (bio->bi_rw & REQ_FUA ? DP_FUA : 0) | + (bio->bi_rw & REQ_FLUSH ? DP_FLUSH : 0) | + (bio->bi_op == REQ_OP_DISCARD ? DP_DISCARD : 0); else - return bi_rw & REQ_SYNC ? DP_RW_SYNC : 0; + return bio->bi_rw & REQ_SYNC ? DP_RW_SYNC : 0; } /* Used to send write or TRIM aka REQ_DISCARD requests @@ -1636,7 +1636,7 @@ int drbd_send_dblock(struct drbd_peer_device *peer_device, struct drbd_request * p->sector = cpu_to_be64(req->i.sector); p->block_id = (unsigned long)req; p->seq_num = cpu_to_be32(atomic_inc_return(&device->packet_seq)); - dp_flags = bio_flags_to_wire(peer_device->connection, req->master_bio->bi_rw); + dp_flags = bio_flags_to_wire(peer_device->connection, req->master_bio); if (device->state.conn >= C_SYNC_SOURCE && device->state.conn <= C_PAUSED_SYNC_T) dp_flags |= DP_MAY_SET_IN_SYNC; diff --git a/drivers/block/drbd/drbd_worker.c b/drivers/block/drbd/drbd_worker.c index f92f533..4335f2d 100644 --- a/drivers/block/drbd/drbd_worker.c +++ b/drivers/block/drbd/drbd_worker.c @@ -174,7 +174,7 @@ void drbd_peer_request_endio(struct bio *bio) struct drbd_peer_request *peer_req = bio->bi_private; struct drbd_device *device = peer_req->peer_device->device; int is_write = bio_data_dir(bio) == WRITE; - int is_discard = !!(bio->bi_rw & REQ_DISCARD); + int is_discard = !!(bio->bi_op == REQ_OP_DISCARD); if (bio->bi_error && __ratelimit(&drbd_ratelimit_state)) drbd_warn(device, "%s: error=%d s=%llus\n", @@ -248,7 +248,7 @@ void drbd_request_endio(struct bio *bio) /* to avoid recursion in __req_mod */ if (unlikely(bio->bi_error)) { - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) what = (bio->bi_error == -EOPNOTSUPP) ? DISCARD_COMPLETED_NOTSUPP : DISCARD_COMPLETED_WITH_ERROR; diff --git a/drivers/block/loop.c b/drivers/block/loop.c index e771bab..1afc03c 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -447,7 +447,7 @@ static int lo_req_flush(struct loop_device *lo, struct request *rq) static inline void handle_partial_read(struct loop_cmd *cmd, long bytes) { - if (bytes < 0 || (cmd->rq->cmd_flags & REQ_WRITE)) + if (bytes < 0 || op_is_write(cmd->rq->op)) return; if (unlikely(bytes < blk_rq_bytes(cmd->rq))) { @@ -535,7 +535,7 @@ static int do_req_filebacked(struct loop_device *lo, struct request *rq) pos = ((loff_t) blk_rq_pos(rq) << 9) + lo->lo_offset; - if (rq->cmd_flags & REQ_WRITE) { + if (op_is_write(rq->op)) { if (rq->cmd_flags & REQ_FLUSH) ret = lo_req_flush(lo, rq); else if (rq->op == REQ_OP_DISCARD) @@ -1666,7 +1666,7 @@ static int loop_queue_rq(struct blk_mq_hw_ctx *hctx, static void loop_handle_cmd(struct loop_cmd *cmd) { - const bool write = cmd->rq->cmd_flags & REQ_WRITE; + const bool write = op_is_write(cmd->rq->op); struct loop_device *lo = cmd->rq->q->queuedata; int ret = 0; diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index ea326ef..0afccb1 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -3375,7 +3375,7 @@ static void rbd_queue_workfn(struct work_struct *work) if (rq->op == REQ_OP_DISCARD) op_type = OBJ_OP_DISCARD; - else if (rq->cmd_flags & REQ_WRITE) + else if (rq->op == REQ_OP_WRITE) op_type = OBJ_OP_WRITE; else op_type = OBJ_OP_READ; diff --git a/drivers/block/rsxx/dma.c b/drivers/block/rsxx/dma.c index cf8cd29..dfc189e 100644 --- a/drivers/block/rsxx/dma.c +++ b/drivers/block/rsxx/dma.c @@ -705,7 +705,7 @@ int rsxx_dma_queue_bio(struct rsxx_cardinfo *card, dma_cnt[i] = 0; } - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { bv_len = bio->bi_iter.bi_size; while (bv_len > 0) { diff --git a/drivers/block/umem.c b/drivers/block/umem.c index 7939b9f..a24ccbc 100644 --- a/drivers/block/umem.c +++ b/drivers/block/umem.c @@ -462,7 +462,7 @@ static void process_page(unsigned long data) le32_to_cpu(desc->local_addr)>>9, le32_to_cpu(desc->transfer_size)); dump_dmastat(card, control); - } else if ((bio->bi_rw & REQ_WRITE) && + } else if (op_is_write(bio->bi_op) && le32_to_cpu(desc->local_addr) >> 9 == card->init_size) { card->init_size += le32_to_cpu(desc->transfer_size) >> 9; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 47915d7..8d14cea 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -849,7 +849,7 @@ static void __zram_make_request(struct zram *zram, struct bio *bio) offset = (bio->bi_iter.bi_sector & (SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT; - if (unlikely(bio->bi_rw & REQ_DISCARD)) { + if (unlikely(bio->bi_op == REQ_OP_DISCARD)) { zram_bio_discard(zram, index, offset, bio); bio_endio(bio); return; diff --git a/drivers/ide/ide-floppy.c b/drivers/ide/ide-floppy.c index 2fb5350..f079d8d 100644 --- a/drivers/ide/ide-floppy.c +++ b/drivers/ide/ide-floppy.c @@ -206,7 +206,7 @@ static void idefloppy_create_rw_cmd(ide_drive_t *drive, memcpy(rq->cmd, pc->c, 12); pc->rq = rq; - if (rq->cmd_flags & REQ_WRITE) + if (cmd == WRITE) pc->flags |= PC_FLAG_WRITING; pc->flags |= PC_FLAG_DMA_OK; diff --git a/drivers/lightnvm/rrpc.c b/drivers/lightnvm/rrpc.c index c20cb18..9b5a8c3 100644 --- a/drivers/lightnvm/rrpc.c +++ b/drivers/lightnvm/rrpc.c @@ -853,7 +853,7 @@ static blk_qc_t rrpc_make_rq(struct request_queue *q, struct bio *bio) struct nvm_rq *rqd; int err; - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { rrpc_discard(rrpc, bio); return BLK_QC_T_NONE; } diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index cd6837c..be8cd63f 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -378,7 +378,7 @@ static bool check_should_bypass(struct cached_dev *dc, struct bio *bio) if (test_bit(BCACHE_DEV_DETACHING, &dc->disk.flags) || c->gc_stats.in_use > CUTOFF_CACHE_ADD || - (bio->bi_rw & REQ_DISCARD)) + (bio->bi_op == REQ_OP_DISCARD)) goto skip; if (mode == CACHE_MODE_NONE || @@ -899,7 +899,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) * But check_overlapping drops dirty keys for which io hasn't started, * so we still want to call it. */ - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) s->iop.bypass = true; if (should_writeback(dc, s->orig_bio, @@ -913,7 +913,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) s->iop.bio = s->orig_bio; bio_get(s->iop.bio); - if (!(bio->bi_rw & REQ_DISCARD) || + if (!(bio->bi_op == REQ_OP_DISCARD) || blk_queue_discard(bdev_get_queue(dc->bdev))) closure_bio_submit(bio, cl); } else if (s->iop.writeback) { @@ -993,7 +993,7 @@ static blk_qc_t cached_dev_make_request(struct request_queue *q, cached_dev_read(dc, s); } } else { - if ((bio->bi_rw & REQ_DISCARD) && + if ((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(dc->bdev))) bio_endio(bio); else @@ -1104,7 +1104,7 @@ static blk_qc_t flash_dev_make_request(struct request_queue *q, &KEY(d->id, bio->bi_iter.bi_sector, 0), &KEY(d->id, bio_end_sector(bio), 0)); - s->iop.bypass = (bio->bi_rw & REQ_DISCARD) != 0; + s->iop.bypass = (bio->bi_op == REQ_OP_DISCARD) != 0; s->iop.writeback = true; s->iop.bio = bio; diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c index 5780acc..fe222c0 100644 --- a/drivers/md/dm-cache-target.c +++ b/drivers/md/dm-cache-target.c @@ -788,7 +788,8 @@ static void check_if_tick_bio_needed(struct cache *cache, struct bio *bio) spin_lock_irqsave(&cache->lock, flags); if (cache->need_tick_bio && - !(bio->bi_rw & (REQ_FUA | REQ_FLUSH | REQ_DISCARD))) { + !(bio->bi_rw & (REQ_FUA | REQ_FLUSH)) && + bio->bi_op != REQ_OP_DISCARD) { pb->tick = true; cache->need_tick_bio = false; } @@ -851,7 +852,7 @@ static void inc_ds(struct cache *cache, struct bio *bio, static bool accountable_bio(struct cache *cache, struct bio *bio) { return ((bio->bi_bdev == cache->origin_dev->bdev) && - !(bio->bi_rw & REQ_DISCARD)); + bio->bi_op != REQ_OP_DISCARD); } static void accounted_begin(struct cache *cache, struct bio *bio) @@ -1062,7 +1063,7 @@ static void dec_io_migrations(struct cache *cache) static bool discard_or_flush(struct bio *bio) { - return bio->bi_rw & (REQ_FLUSH | REQ_FUA | REQ_DISCARD); + return bio->bi_op == REQ_OP_DISCARD || bio->bi_rw & (REQ_FLUSH | REQ_FUA); } static void __cell_defer(struct cache *cache, struct dm_bio_prison_cell *cell) @@ -1975,7 +1976,7 @@ static void process_deferred_bios(struct cache *cache) if (bio->bi_rw & REQ_FLUSH) process_flush_bio(cache, bio); - else if (bio->bi_rw & REQ_DISCARD) + else if (bio->bi_op == REQ_OP_DISCARD) process_discard_bio(cache, &structs, bio); else process_bio(cache, &structs, bio); diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index f466fec..b7cbd39 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1913,7 +1913,7 @@ static int crypt_map(struct dm_target *ti, struct bio *bio) * - for REQ_FLUSH device-mapper core ensures that no IO is in-flight * - for REQ_DISCARD caller must use flush if IO ordering matters */ - if (unlikely(bio->bi_rw & (REQ_FLUSH | REQ_DISCARD))) { + if (unlikely(bio->bi_rw & REQ_FLUSH || bio->bi_op == REQ_OP_DISCARD)) { bio->bi_bdev = cc->dev->bdev; if (bio_sectors(bio)) bio->bi_iter.bi_sector = cc->start + diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c index f63bbf4..813bb4d 100644 --- a/drivers/md/dm-log-writes.c +++ b/drivers/md/dm-log-writes.c @@ -557,7 +557,7 @@ static int log_writes_map(struct dm_target *ti, struct bio *bio) int i = 0; bool flush_bio = (bio->bi_rw & REQ_FLUSH); bool fua_bio = (bio->bi_rw & REQ_FUA); - bool discard_bio = (bio->bi_rw & REQ_DISCARD); + bool discard_bio = (bio->bi_op == REQ_OP_DISCARD); pb->block = NULL; diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c index 6b61b62..54cc0c7 100644 --- a/drivers/md/dm-raid1.c +++ b/drivers/md/dm-raid1.c @@ -626,7 +626,7 @@ static void write_callback(unsigned long error, void *context) * If the bio is discard, return an error, but do not * degrade the array. */ - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { bio->bi_error = -EOPNOTSUPP; bio_endio(bio); return; @@ -665,7 +665,7 @@ static void do_write(struct mirror_set *ms, struct bio *bio) .client = ms->io_client, }; - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { io_req.bi_op = REQ_OP_DISCARD; io_req.mem.type = DM_IO_KMEM; io_req.mem.ptr.addr = NULL; @@ -705,7 +705,7 @@ static void do_writes(struct mirror_set *ms, struct bio_list *writes) while ((bio = bio_list_pop(writes))) { if ((bio->bi_rw & REQ_FLUSH) || - (bio->bi_rw & REQ_DISCARD)) { + (bio->bi_op == REQ_OP_DISCARD)) { bio_list_add(&sync, bio); continue; } @@ -1253,7 +1253,7 @@ static int mirror_end_io(struct dm_target *ti, struct bio *bio, int error) * We need to dec pending if this was a write. */ if (rw == WRITE) { - if (!(bio->bi_rw & (REQ_FLUSH | REQ_DISCARD))) + if (!(bio->bi_rw & REQ_FLUSH) && bio->bi_op != REQ_OP_DISCARD) dm_rh_dec(ms->rh, bio_record->write_region); return error; } diff --git a/drivers/md/dm-region-hash.c b/drivers/md/dm-region-hash.c index 74cb7b9..ce4af57 100644 --- a/drivers/md/dm-region-hash.c +++ b/drivers/md/dm-region-hash.c @@ -403,7 +403,7 @@ void dm_rh_mark_nosync(struct dm_region_hash *rh, struct bio *bio) return; } - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) return; /* We must inform the log that the sync count has changed. */ @@ -526,7 +526,7 @@ void dm_rh_inc_pending(struct dm_region_hash *rh, struct bio_list *bios) struct bio *bio; for (bio = bios->head; bio; bio = bio->bi_next) { - if (bio->bi_rw & (REQ_FLUSH | REQ_DISCARD)) + if (bio->bi_rw & REQ_FLUSH || bio->bi_op == REQ_OP_DISCARD) continue; rh_inc(rh, dm_rh_bio_to_region(rh, bio)); } diff --git a/drivers/md/dm-stripe.c b/drivers/md/dm-stripe.c index 797ddb9..12b1630 100644 --- a/drivers/md/dm-stripe.c +++ b/drivers/md/dm-stripe.c @@ -292,8 +292,8 @@ static int stripe_map(struct dm_target *ti, struct bio *bio) bio->bi_bdev = sc->stripe[target_bio_nr].dev->bdev; return DM_MAPIO_REMAPPED; } - if (unlikely(bio->bi_rw & REQ_DISCARD) || - unlikely(bio->bi_rw & REQ_WRITE_SAME)) { + if (unlikely(bio->bi_op == REQ_OP_DISCARD) || + unlikely(bio->bi_op == REQ_OP_WRITE_SAME)) { target_bio_nr = dm_bio_get_target_bio_nr(bio); BUG_ON(target_bio_nr >= sc->stripes); return stripe_map_range(sc, bio, target_bio_nr); diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index f870878..1e170e7 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -705,7 +705,7 @@ static void inc_all_io_entry(struct pool *pool, struct bio *bio) { struct dm_thin_endio_hook *h; - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) return; h = dm_per_bio_data(bio, sizeof(struct dm_thin_endio_hook)); @@ -868,7 +868,8 @@ static void __inc_remap_and_issue_cell(void *context, struct bio *bio; while ((bio = bio_list_pop(&cell->bios))) { - if (bio->bi_rw & (REQ_DISCARD | REQ_FLUSH | REQ_FUA)) + if (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + bio->bi_op == REQ_OP_DISCARD) bio_list_add(&info->defer_bios, bio); else { inc_all_io_entry(info->tc->pool, bio); @@ -1646,7 +1647,8 @@ static void __remap_and_issue_shared_cell(void *context, while ((bio = bio_list_pop(&cell->bios))) { if ((bio_data_dir(bio) == WRITE) || - (bio->bi_rw & (REQ_DISCARD | REQ_FLUSH | REQ_FUA))) + (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + bio->bi_op == REQ_OP_DISCARD)) bio_list_add(&info->defer_bios, bio); else { struct dm_thin_endio_hook *h = dm_per_bio_data(bio, sizeof(struct dm_thin_endio_hook));; @@ -2035,7 +2037,7 @@ static void process_thin_deferred_bios(struct thin_c *tc) break; } - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) pool->process_discard(tc, bio); else pool->process_bio(tc, bio); @@ -2122,7 +2124,7 @@ static void process_thin_deferred_cells(struct thin_c *tc) return; } - if (cell->holder->bi_rw & REQ_DISCARD) + if (cell->holder->bi_op == REQ_OP_DISCARD) pool->process_discard_cell(tc, cell); else pool->process_cell(tc, cell); @@ -2558,7 +2560,8 @@ static int thin_bio_map(struct dm_target *ti, struct bio *bio) return DM_MAPIO_SUBMITTED; } - if (bio->bi_rw & (REQ_DISCARD | REQ_FLUSH | REQ_FUA)) { + if (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + bio->bi_op == REQ_OP_DISCARD) { thin_defer_bio_with_throttle(tc, bio); return DM_MAPIO_SUBMITTED; } diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 7a85017..7c6e386 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1014,7 +1014,7 @@ static void clone_endio(struct bio *bio) } } - if (unlikely(r == -EREMOTEIO && (bio->bi_rw & REQ_WRITE_SAME) && + if (unlikely(r == -EREMOTEIO && (bio->bi_op == REQ_OP_WRITE_SAME) && !bdev_get_queue(bio->bi_bdev)->limits.max_write_same_sectors)) disable_write_same(md); @@ -1688,9 +1688,9 @@ static int __split_and_process_non_flush(struct clone_info *ci) struct dm_target *ti; unsigned len; - if (unlikely(bio->bi_rw & REQ_DISCARD)) + if (unlikely(bio->bi_op == REQ_OP_DISCARD)) return __send_discard(ci); - else if (unlikely(bio->bi_rw & REQ_WRITE_SAME)) + else if (unlikely(bio->bi_op == REQ_OP_WRITE_SAME)) return __send_write_same(ci); ti = dm_table_find_target(ci->map, ci->sector); diff --git a/drivers/md/linear.c b/drivers/md/linear.c index b7fe7e9..aad82c7 100644 --- a/drivers/md/linear.c +++ b/drivers/md/linear.c @@ -252,7 +252,7 @@ static void linear_make_request(struct mddev *mddev, struct bio *bio) split->bi_iter.bi_sector = split->bi_iter.bi_sector - start_sector + data_offset; - if (unlikely((split->bi_rw & REQ_DISCARD) && + if (unlikely((split->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(split->bi_bdev)))) { /* Just ignore it */ bio_endio(split); diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c index 2ea12c6..e0d1b8c 100644 --- a/drivers/md/raid0.c +++ b/drivers/md/raid0.c @@ -488,7 +488,7 @@ static void raid0_make_request(struct mddev *mddev, struct bio *bio) split->bi_iter.bi_sector = sector + zone->dev_start + tmp_dev->data_offset; - if (unlikely((split->bi_rw & REQ_DISCARD) && + if (unlikely((split->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(split->bi_bdev)))) { /* Just ignore it */ bio_endio(split); diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c index 4e7b440..680d9e4 100644 --- a/drivers/scsi/osd/osd_initiator.c +++ b/drivers/scsi/osd/osd_initiator.c @@ -824,7 +824,7 @@ void osd_req_write(struct osd_request *or, { _osd_req_encode_common(or, OSD_ACT_WRITE, obj, offset, len); WARN_ON(or->out.bio || or->out.total_bytes); - WARN_ON(0 == (bio->bi_rw & REQ_WRITE)); + WARN_ON(!op_is_write(bio->bi_op)); or->out.bio = bio; or->out.total_bytes = len; } @@ -875,7 +875,7 @@ void osd_req_read(struct osd_request *or, { _osd_req_encode_common(or, OSD_ACT_READ, obj, offset, len); WARN_ON(or->in.bio || or->in.total_bytes); - WARN_ON(bio->bi_rw & REQ_WRITE); + WARN_ON(op_is_write(bio->bi_op)); or->in.bio = bio; or->in.total_bytes = len; } diff --git a/drivers/staging/lustre/lustre/llite/lloop.c b/drivers/staging/lustre/lustre/llite/lloop.c index 420d391..37a52ed 100644 --- a/drivers/staging/lustre/lustre/llite/lloop.c +++ b/drivers/staging/lustre/lustre/llite/lloop.c @@ -212,9 +212,9 @@ static int do_bio_lustrebacked(struct lloop_device *lo, struct bio *head) io->ci_lockreq = CILR_NEVER; LASSERT(head != NULL); - rw = head->bi_rw; + rw = bio_data_dir(head); for (bio = head; bio != NULL; bio = bio->bi_next) { - LASSERT(rw == bio->bi_rw); + LASSERT(rw == bio_data_dir(bio)); offset = (pgoff_t)(bio->bi_iter.bi_sector << 9) + lo->lo_offset; bio_for_each_segment(bvec, bio, iter) { @@ -305,9 +305,9 @@ static unsigned int loop_get_bio(struct lloop_device *lo, struct bio **req) /* TODO: need to split the bio, too bad. */ LASSERT(first->bi_vcnt <= LLOOP_MAX_SEGMENTS); - rw = first->bi_rw; + rw = bio_data_dir(first); bio = &lo->lo_bio; - while (*bio && (*bio)->bi_rw == rw) { + while (*bio && bio_data_dir(*bio) == rw) { CDEBUG(D_INFO, "bio sector %llu size %u count %u vcnt%u \n", (unsigned long long)(*bio)->bi_iter.bi_sector, (*bio)->bi_iter.bi_size, diff --git a/include/linux/bio.h b/include/linux/bio.h index 9f0e2cf..1e4b3b5 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -106,18 +106,23 @@ static inline bool bio_has_data(struct bio *bio) { if (bio && bio->bi_iter.bi_size && - !(bio->bi_rw & REQ_DISCARD)) + !(bio->bi_op == REQ_OP_DISCARD)) return true; return false; } +static inline bool bio_no_advance_iter(struct bio *bio) +{ + return bio->bi_op == REQ_OP_DISCARD || bio->bi_op == REQ_OP_WRITE_SAME; +} + static inline bool bio_is_rw(struct bio *bio) { if (!bio_has_data(bio)) return false; - if (bio->bi_rw & BIO_NO_ADVANCE_ITER_MASK) + if (bio_no_advance_iter(bio)) return false; return true; @@ -225,7 +230,7 @@ static inline void bio_advance_iter(struct bio *bio, struct bvec_iter *iter, { iter->bi_sector += bytes >> 9; - if (bio->bi_rw & BIO_NO_ADVANCE_ITER_MASK) + if (bio_no_advance_iter(bio)) iter->bi_size -= bytes; else bvec_iter_advance(bio->bi_io_vec, iter, bytes); @@ -253,10 +258,10 @@ static inline unsigned bio_segments(struct bio *bio) * differently: */ - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) return 1; - if (bio->bi_rw & REQ_WRITE_SAME) + if (bio->bi_op == REQ_OP_WRITE_SAME) return 1; bio_for_each_segment(bv, bio, iter) diff --git a/include/linux/fs.h b/include/linux/fs.h index 3ab5342..85007cd 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2419,33 +2419,24 @@ static inline bool op_is_write(int op) } /* - * return READ, READA, or WRITE + * return data direction, READ or WRITE */ -static inline int bio_rw(struct bio *bio) +static inline int bio_data_dir(struct bio *bio) { - /* - * tmp cpmpat. Allow users to set either op or rw, until - * all code is converted in the next patches. - */ if (op_is_write(bio->bi_op)) return WRITE; - - return bio->bi_rw & (RW_MASK | RWA_MASK); + return READ; } /* - * return data direction, READ or WRITE + * return READ, READA, or WRITE */ -static inline int bio_data_dir(struct bio *bio) +static inline int bio_rw(struct bio *bio) { - /* - * tmp cpmpat. Allow users to set either op or rw, until - * all code is converted in the next patches. - */ - if (op_is_write(bio->bi_op)) - return WRITE; + if (bio->bi_rw & RWA_MASK) + return READA; - return bio->bi_rw & 1; + return bio_data_dir(bio); } extern void check_disk_size_change(struct gendisk *disk, -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id ABF9F29E55 for ; Tue, 5 Jan 2016 14:55:13 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 713B28F8035 for ; Tue, 5 Jan 2016 12:55:13 -0800 (PST) X-ASG-Debug-ID: 1452027311-04bdf03afc04e70001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id CyuD25ZPEUPusNN0 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:12 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id C574CA1730; Tue, 5 Jan 2016 20:55:11 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuU024032; Tue, 5 Jan 2016 15:55:09 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 31/35] block, fs: remove old REQ definitions. Date: Tue, 5 Jan 2016 14:53:34 -0600 X-ASG-Orig-Subj: [PATCH 31/35] block, fs: remove old REQ definitions. Message-Id: <1452027218-32303-32-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027312 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie We no longer use REQ_WRITE. REQ_WRITE_SAME and REQ_DISCARD, so this patch removes them. Signed-off-by: Mike Christie --- include/linux/blk_types.h | 19 +++++-------------- include/linux/fs.h | 21 +++++++++++---------- include/trace/events/f2fs.h | 1 - 3 files changed, 16 insertions(+), 25 deletions(-) diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 6e49c91..bb30c2b 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -151,7 +151,6 @@ struct bio { */ enum rq_flag_bits { /* common flags */ - __REQ_WRITE, /* not set, read. set, write */ __REQ_FAILFAST_DEV, /* no driver retries of device errors */ __REQ_FAILFAST_TRANSPORT, /* no driver retries of transport errors */ __REQ_FAILFAST_DRIVER, /* no driver retries of driver errors */ @@ -159,9 +158,7 @@ enum rq_flag_bits { __REQ_SYNC, /* request is sync (sync write or read) */ __REQ_META, /* metadata io request */ __REQ_PRIO, /* boost priority in cfq */ - __REQ_DISCARD, /* request to discard sectors */ __REQ_SECURE, /* secure discard (used with __REQ_DISCARD) */ - __REQ_WRITE_SAME, /* write same block many times */ __REQ_NOIDLE, /* don't anticipate more IO after this one */ __REQ_INTEGRITY, /* I/O includes block integrity payload */ @@ -197,28 +194,22 @@ enum rq_flag_bits { __REQ_NR_BITS, /* stops here */ }; -#define REQ_WRITE (1ULL << __REQ_WRITE) #define REQ_FAILFAST_DEV (1ULL << __REQ_FAILFAST_DEV) #define REQ_FAILFAST_TRANSPORT (1ULL << __REQ_FAILFAST_TRANSPORT) #define REQ_FAILFAST_DRIVER (1ULL << __REQ_FAILFAST_DRIVER) #define REQ_SYNC (1ULL << __REQ_SYNC) #define REQ_META (1ULL << __REQ_META) #define REQ_PRIO (1ULL << __REQ_PRIO) -#define REQ_DISCARD (1ULL << __REQ_DISCARD) -#define REQ_WRITE_SAME (1ULL << __REQ_WRITE_SAME) #define REQ_NOIDLE (1ULL << __REQ_NOIDLE) #define REQ_INTEGRITY (1ULL << __REQ_INTEGRITY) #define REQ_FAILFAST_MASK \ (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER) #define REQ_COMMON_MASK \ - (REQ_WRITE | REQ_FAILFAST_MASK | REQ_SYNC | REQ_META | REQ_PRIO | \ - REQ_DISCARD | REQ_WRITE_SAME | REQ_NOIDLE | REQ_FLUSH | REQ_FUA | \ - REQ_SECURE | REQ_INTEGRITY) + (REQ_FAILFAST_MASK | REQ_SYNC | REQ_META | REQ_PRIO | REQ_NOIDLE | \ + REQ_FLUSH | REQ_FUA | REQ_SECURE | REQ_INTEGRITY) #define REQ_CLONE_MASK REQ_COMMON_MASK -#define BIO_NO_ADVANCE_ITER_MASK (REQ_DISCARD|REQ_WRITE_SAME) - /* This mask is used for both bio and request merge checking */ #define REQ_NOMERGE_FLAGS \ (REQ_NOMERGE | REQ_STARTED | REQ_SOFTBARRIER | REQ_FLUSH | REQ_FUA | REQ_FLUSH_SEQ) @@ -250,9 +241,9 @@ enum rq_flag_bits { enum req_op { REQ_OP_READ, - REQ_OP_WRITE = REQ_WRITE, - REQ_OP_DISCARD = REQ_DISCARD, - REQ_OP_WRITE_SAME = REQ_WRITE_SAME, + REQ_OP_WRITE, + REQ_OP_DISCARD, /* request to discard sectors */ + REQ_OP_WRITE_SAME, /* write same block many times */ }; typedef unsigned int blk_qc_t; diff --git a/include/linux/fs.h b/include/linux/fs.h index 85007cd..d57a5b5 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -151,9 +151,10 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); #define CHECK_IOVEC_ONLY -1 /* - * The below are the various read and write types that we support. Some of + * The below are the various read and write flags that we support. Some of * them include behavioral modifiers that send information down to the - * block layer and IO scheduler. Terminology: + * block layer and IO scheduler. They should be used along with a req_op. + * Terminology: * * The block layer uses device plugging to defer IO a little bit, in * the hope that we will see more IO very shortly. This increases @@ -192,19 +193,19 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); * non-volatile media on completion. * */ -#define RW_MASK REQ_WRITE +#define RW_MASK REQ_OP_WRITE #define RWA_MASK REQ_RAHEAD -#define READ 0 +#define READ REQ_OP_READ #define WRITE RW_MASK #define READA RWA_MASK -#define READ_SYNC (READ | REQ_SYNC) -#define WRITE_SYNC (WRITE | REQ_SYNC | REQ_NOIDLE) -#define WRITE_ODIRECT (WRITE | REQ_SYNC) -#define WRITE_FLUSH (WRITE | REQ_SYNC | REQ_NOIDLE | REQ_FLUSH) -#define WRITE_FUA (WRITE | REQ_SYNC | REQ_NOIDLE | REQ_FUA) -#define WRITE_FLUSH_FUA (WRITE | REQ_SYNC | REQ_NOIDLE | REQ_FLUSH | REQ_FUA) +#define READ_SYNC REQ_SYNC +#define WRITE_SYNC (REQ_SYNC | REQ_NOIDLE) +#define WRITE_ODIRECT REQ_SYNC +#define WRITE_FLUSH (REQ_SYNC | REQ_NOIDLE | REQ_FLUSH) +#define WRITE_FUA (REQ_SYNC | REQ_NOIDLE | REQ_FUA) +#define WRITE_FLUSH_FUA (REQ_SYNC | REQ_NOIDLE | REQ_FLUSH | REQ_FUA) /* * Attribute flags. These should be or-ed together to figure out what diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index ddf4144..ed3fc87 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -31,7 +31,6 @@ TRACE_DEFINE_ENUM(BG_GC); TRACE_DEFINE_ENUM(LFS); TRACE_DEFINE_ENUM(SSR); TRACE_DEFINE_ENUM(__REQ_RAHEAD); -TRACE_DEFINE_ENUM(__REQ_WRITE); TRACE_DEFINE_ENUM(__REQ_SYNC); TRACE_DEFINE_ENUM(__REQ_NOIDLE); TRACE_DEFINE_ENUM(__REQ_FLUSH); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 949CE29E57 for ; Tue, 5 Jan 2016 14:55:16 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 32766AC005 for ; Tue, 5 Jan 2016 12:55:16 -0800 (PST) X-ASG-Debug-ID: 1452027314-04bdf03afa04e70001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id rSDCUKmjx4vCDVED (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:14 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 7252519CBF7; Tue, 5 Jan 2016 20:55:14 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuV024032; Tue, 5 Jan 2016 15:55:11 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 32/35] block: shrink bi_rw and bi_op Date: Tue, 5 Jan 2016 14:53:35 -0600 X-ASG-Orig-Subj: [PATCH 32/35] block: shrink bi_rw and bi_op Message-Id: <1452027218-32303-33-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027314 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie There is no need for bi_op/op and bi_rw to be so large now, so this patch shrinks them. Signed-off-by: Mike Christie --- block/blk-core.c | 2 +- drivers/md/dm-flakey.c | 2 +- drivers/md/raid5.c | 13 +++++++------ fs/btrfs/check-integrity.c | 4 ++-- fs/btrfs/inode.c | 2 +- include/linux/bio.h | 13 ++----------- include/linux/blk_types.h | 11 +++-------- include/linux/blkdev.h | 2 +- 8 files changed, 18 insertions(+), 31 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 0f6cb5c..ae2afab 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1854,7 +1854,7 @@ static void handle_bad_sector(struct bio *bio) char b[BDEVNAME_SIZE]; printk(KERN_INFO "attempt to access beyond end of device\n"); - printk(KERN_INFO "%s: rw=%d,%ld, want=%Lu, limit=%Lu\n", + printk(KERN_INFO "%s: rw=%d,%u, want=%Lu, limit=%Lu\n", bdevname(bio->bi_bdev, b), bio->bi_op, bio->bi_rw, (unsigned long long)bio_end_sector(bio), diff --git a/drivers/md/dm-flakey.c b/drivers/md/dm-flakey.c index 09e2afc..b831226 100644 --- a/drivers/md/dm-flakey.c +++ b/drivers/md/dm-flakey.c @@ -266,7 +266,7 @@ static void corrupt_bio_data(struct bio *bio, struct flakey_c *fc) data[fc->corrupt_bio_byte - 1] = fc->corrupt_bio_value; DMDEBUG("Corrupting data bio=%p by writing %u to byte %u " - "(rw=%c bi_rw=%lu bi_sector=%llu cur_bytes=%u)\n", + "(rw=%c bi_rw=%u bi_sector=%llu cur_bytes=%u)\n", bio, fc->corrupt_bio_value, fc->corrupt_bio_byte, (bio_data_dir(bio) == WRITE) ? 'w' : 'r', bio->bi_rw, (unsigned long long)bio->bi_iter.bi_sector, bio_bytes); diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index fa4fe95..aafd49e 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -1015,9 +1015,9 @@ again: : raid5_end_read_request; bi->bi_private = sh; - pr_debug("%s: for %llu schedule op %ld on disc %d\n", + pr_debug("%s: for %llu schedule op %d,%u on disc %d\n", __func__, (unsigned long long)sh->sector, - bi->bi_rw, i); + bi->bi_op, bi->bi_rw, i); atomic_inc(&sh->count); if (sh != head_sh) atomic_inc(&head_sh->count); @@ -1067,10 +1067,10 @@ again: rbi->bi_end_io = raid5_end_write_request; rbi->bi_private = sh; - pr_debug("%s: for %llu schedule op %ld on " + pr_debug("%s: for %llu schedule op %d,%u on " "replacement disc %d\n", __func__, (unsigned long long)sh->sector, - rbi->bi_rw, i); + rbi->bi_op, rbi->bi_rw, i); atomic_inc(&sh->count); if (sh != head_sh) atomic_inc(&head_sh->count); @@ -1102,8 +1102,9 @@ again: if (!rdev && !rrdev) { if (op_is_write(op)) set_bit(STRIPE_DEGRADED, &sh->state); - pr_debug("skip op %ld on disc %d for sector %llu\n", - bi->bi_rw, i, (unsigned long long)sh->sector); + pr_debug("skip op %d,%u on disc %d for sector %llu\n", + bi->bi_op, bi->bi_rw, i, + (unsigned long long)sh->sector); clear_bit(R5_LOCKED, &sh->dev[i].flags); set_bit(STRIPE_HANDLE, &sh->state); } diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index e409d1f..1623d11 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -2985,7 +2985,7 @@ static void __btrfsic_submit_bio(struct bio *bio) if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bio(rw=%d,0x%lx, bi_vcnt=%u," + "submit_bio(rw=%d,0x%x, bi_vcnt=%u," " bi_sector=%llu (bytenr %llu), bi_bdev=%p)\n", bio->bi_op, bio->bi_rw, bio->bi_vcnt, (unsigned long long)bio->bi_iter.bi_sector, @@ -3028,7 +3028,7 @@ static void __btrfsic_submit_bio(struct bio *bio) if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bio(rw=%d,0x%lx FLUSH, bdev=%p)\n", + "submit_bio(rw=%d,0x%x FLUSH, bdev=%p)\n", bio->bi_op, bio->bi_rw, bio->bi_bdev); if (!dev_state->dummy_block_for_bio_bh_flush.is_iodone) { if ((dev_state->state->print_mask & diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index a96984f..b9c3799 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8061,7 +8061,7 @@ static void btrfs_end_dio_bio(struct bio *bio) if (err) btrfs_warn(BTRFS_I(dip->inode)->root->fs_info, - "direct IO failed ino %llu rw %d,%lu sector %#Lx len %u err no %d", + "direct IO failed ino %llu rw %d,0x%x sector %#Lx len %u err no %d", btrfs_ino(dip->inode), bio->bi_op, bio->bi_rw, (unsigned long long)bio->bi_iter.bi_sector, bio->bi_iter.bi_size, err); diff --git a/include/linux/bio.h b/include/linux/bio.h index 1e4b3b5..c91afc1 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -44,18 +44,9 @@ #define BIO_MAX_SIZE (BIO_MAX_PAGES << PAGE_CACHE_SHIFT) #define BIO_MAX_SECTORS (BIO_MAX_SIZE >> 9) -/* - * upper 16 bits of bi_rw define the io priority of this bio - */ -#define BIO_PRIO_SHIFT (8 * sizeof(unsigned long) - IOPRIO_BITS) -#define bio_prio(bio) ((bio)->bi_rw >> BIO_PRIO_SHIFT) +#define bio_prio(bio) (bio)->bi_ioprio #define bio_prio_valid(bio) ioprio_valid(bio_prio(bio)) - -#define bio_set_prio(bio, prio) do { \ - WARN_ON(prio >= (1 << IOPRIO_BITS)); \ - (bio)->bi_rw &= ((1UL << BIO_PRIO_SHIFT) - 1); \ - (bio)->bi_rw |= ((unsigned long) (prio) << BIO_PRIO_SHIFT); \ -} while (0) +#define bio_set_prio(bio, prio) ((bio)->bi_ioprio = prio) /* * various member access, note that bio_data should of course not be used diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index bb30c2b..5d19b393 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -48,14 +48,9 @@ struct bio { struct block_device *bi_bdev; unsigned int bi_flags; /* status, command, etc */ int bi_error; - unsigned long bi_rw; /* bottom bits rq_flags_bits - * top bits priority - */ - /* - * this will be a u8 in the next patches and bi_rw can be shrunk to - * a u32. For compat in these transistional patches op is a int here. - */ - int bi_op; /* REQ_OP */ + unsigned int bi_rw; /* rq_flag_bits */ + unsigned short bi_ioprio; + u8 bi_op; /* REQ_OP */ struct bvec_iter bi_iter; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index bc024c7..83d6f033 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -96,7 +96,7 @@ struct request { struct request_queue *q; struct blk_mq_ctx *mq_ctx; - int op; + u8 op; u64 cmd_flags; unsigned cmd_type; unsigned long atomic_flags; -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:19 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id B44AB29E71 for ; Tue, 5 Jan 2016 14:55:19 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 97DA78F8035 for ; Tue, 5 Jan 2016 12:55:19 -0800 (PST) X-ASG-Debug-ID: 1452027317-04cb6c42e7052f0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id FACbfKZWgZenRbg5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:17 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 47B486E799; Tue, 5 Jan 2016 20:55:17 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuW024032; Tue, 5 Jan 2016 15:55:14 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 33/35] block, drivers: add REQ_OP_FLUSH operation Date: Tue, 5 Jan 2016 14:53:36 -0600 X-ASG-Orig-Subj: [PATCH 33/35] block, drivers: add REQ_OP_FLUSH operation Message-Id: <1452027218-32303-34-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027317 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This adds a REQ_OP_FLUSH operation that is sent to request_fn based drivers by the block layer's flush code, instead of sending requests with the request->cmd_flags REQ_FLUSH bit set. For the following 3 flush related patches, I have not tested every driver. I have only tested scsi with xfs and btrfs. Signed-off-by: Mike Christie --- Documentation/block/writeback_cache_control.txt | 6 +++--- block/blk-flush.c | 6 +++--- drivers/block/loop.c | 4 ++-- drivers/block/nbd.c | 2 +- drivers/block/osdblk.c | 2 +- drivers/block/ps3disk.c | 4 ++-- drivers/block/skd_main.c | 2 +- drivers/block/virtio_blk.c | 2 +- drivers/block/xen-blkfront.c | 8 ++++---- drivers/ide/ide-disk.c | 2 +- drivers/md/dm.c | 4 ++-- drivers/mmc/card/block.c | 5 ++--- drivers/mmc/card/queue.h | 2 +- drivers/mtd/mtd_blkdevs.c | 2 +- drivers/nvme/host/pci.c | 2 +- drivers/scsi/sd.c | 7 +++---- include/linux/blk_types.h | 1 + include/linux/blkdev.h | 3 +++ kernel/trace/blktrace.c | 5 ++++- 19 files changed, 37 insertions(+), 32 deletions(-) diff --git a/Documentation/block/writeback_cache_control.txt b/Documentation/block/writeback_cache_control.txt index 83407d3..ea5550f 100644 --- a/Documentation/block/writeback_cache_control.txt +++ b/Documentation/block/writeback_cache_control.txt @@ -73,9 +73,9 @@ doing: blk_queue_flush(sdkp->disk->queue, REQ_FLUSH); -and handle empty REQ_FLUSH requests in its prep_fn/request_fn. Note that +and handle empty REQ_OP_FLUSH requests in its prep_fn/request_fn. Note that REQ_FLUSH requests with a payload are automatically turned into a sequence -of an empty REQ_FLUSH request followed by the actual write by the block +of an empty REQ_OP_FLUSH request followed by the actual write by the block layer. For devices that also support the FUA bit the block layer needs to be told to pass through the REQ_FUA bit using: @@ -83,4 +83,4 @@ to be told to pass through the REQ_FUA bit using: and the driver must handle write requests that have the REQ_FUA bit set in prep_fn/request_fn. If the FUA bit is not natively supported the block -layer turns it into an empty REQ_FLUSH request after the actual write. +layer turns it into an empty REQ_OP_FLUSH request after the actual write. diff --git a/block/blk-flush.c b/block/blk-flush.c index b4eb0e8..0e5561e 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -29,7 +29,7 @@ * The actual execution of flush is double buffered. Whenever a request * needs to execute PRE or POSTFLUSH, it queues at * fq->flush_queue[fq->flush_pending_idx]. Once certain criteria are met, a - * flush is issued and the pending_idx is toggled. When the flush + * REQ_OP_FLUSH is issued and the pending_idx is toggled. When the flush * completes, all the requests which were pending are proceeded to the next * step. This allows arbitrary merging of different types of FLUSH/FUA * requests. @@ -329,8 +329,8 @@ static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) } flush_rq->cmd_type = REQ_TYPE_FS; - flush_rq->cmd_flags = WRITE_FLUSH | REQ_FLUSH_SEQ; - flush_rq->op = REQ_OP_WRITE; + flush_rq->cmd_flags = REQ_SYNC | REQ_NOIDLE | REQ_FLUSH_SEQ; + flush_rq->op = REQ_OP_FLUSH; flush_rq->rq_disk = first_rq->rq_disk; flush_rq->end_io = flush_end_io; diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 1afc03c..a3d1293 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -536,7 +536,7 @@ static int do_req_filebacked(struct loop_device *lo, struct request *rq) pos = ((loff_t) blk_rq_pos(rq) << 9) + lo->lo_offset; if (op_is_write(rq->op)) { - if (rq->cmd_flags & REQ_FLUSH) + if (rq->op == REQ_OP_FLUSH) ret = lo_req_flush(lo, rq); else if (rq->op == REQ_OP_DISCARD) ret = lo_discard(lo, rq, pos); @@ -1653,7 +1653,7 @@ static int loop_queue_rq(struct blk_mq_hw_ctx *hctx, if (lo->lo_state != Lo_bound) return -EIO; - if (lo->use_dio && (!(cmd->rq->cmd_flags & REQ_FLUSH) || + if (lo->use_dio && (cmd->rq->op != REQ_OP_FLUSH || cmd->rq->op == REQ_OP_DISCARD)) cmd->use_aio = true; else diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index 8e8f7e3..ced3382 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -244,7 +244,7 @@ static int nbd_send_req(struct nbd_device *nbd, struct request *req) type = NBD_CMD_DISC; else if (req->op == REQ_OP_DISCARD) type = NBD_CMD_TRIM; - else if (req->cmd_flags & REQ_FLUSH) + else if (req->op == REQ_OP_FLUSH) type = NBD_CMD_FLUSH; else if (rq_data_dir(req) == WRITE) type = NBD_CMD_WRITE; diff --git a/drivers/block/osdblk.c b/drivers/block/osdblk.c index 1b709a4..3f8a0a0 100644 --- a/drivers/block/osdblk.c +++ b/drivers/block/osdblk.c @@ -321,7 +321,7 @@ static void osdblk_rq_fn(struct request_queue *q) * driver-specific, etc. */ - do_flush = rq->cmd_flags & REQ_FLUSH; + do_flush = (rq->op == REQ_OP_FLUSH); do_write = (rq_data_dir(rq) == WRITE); if (!do_flush) { /* osd_flush does not use a bio */ diff --git a/drivers/block/ps3disk.c b/drivers/block/ps3disk.c index c120d70..95c524b 100644 --- a/drivers/block/ps3disk.c +++ b/drivers/block/ps3disk.c @@ -196,7 +196,7 @@ static void ps3disk_do_request(struct ps3_storage_device *dev, dev_dbg(&dev->sbd.core, "%s:%u\n", __func__, __LINE__); while ((req = blk_fetch_request(q))) { - if (req->cmd_flags & REQ_FLUSH) { + if (req->op == REQ_OP_FLUSH) { if (ps3disk_submit_flush_request(dev, req)) break; } else if (req->cmd_type == REQ_TYPE_FS) { @@ -256,7 +256,7 @@ static irqreturn_t ps3disk_interrupt(int irq, void *data) return IRQ_HANDLED; } - if (req->cmd_flags & REQ_FLUSH) { + if (req->op == REQ_OP_FLUSH) { read = 0; op = "flush"; } else { diff --git a/drivers/block/skd_main.c b/drivers/block/skd_main.c index f89a0c8..a15cc2e 100644 --- a/drivers/block/skd_main.c +++ b/drivers/block/skd_main.c @@ -624,7 +624,7 @@ static void skd_request_fn(struct request_queue *q) count = blk_rq_sectors(req); data_dir = rq_data_dir(req); - if (req->cmd_flags & REQ_FLUSH) + if (req->op == REQ_OP_FLUSH) flush++; if (req->cmd_flags & REQ_FUA) diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 6ca3549..c6d3bc2 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -172,7 +172,7 @@ static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx, BUG_ON(req->nr_phys_segments + 2 > vblk->sg_elems); vbr->req = req; - if (req->cmd_flags & REQ_FLUSH) { + if (req->op == REQ_OP_FLUSH) { vbr->out_hdr.type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_FLUSH); vbr->out_hdr.sector = 0; vbr->out_hdr.ioprio = cpu_to_virtio32(vblk->vdev, req_get_ioprio(vbr->req)); diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index d4aa473..1d48f0a 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -635,7 +635,7 @@ static int blkif_queue_rw_req(struct request *req) * The indirect operation can only be a BLKIF_OP_READ or * BLKIF_OP_WRITE */ - BUG_ON(req->cmd_flags & (REQ_FLUSH | REQ_FUA)); + BUG_ON(req->op == REQ_OP_FLUSH || req->cmd_flags & REQ_FUA); ring_req->operation = BLKIF_OP_INDIRECT; ring_req->u.indirect.indirect_op = rq_data_dir(req) ? BLKIF_OP_WRITE : BLKIF_OP_READ; @@ -647,7 +647,7 @@ static int blkif_queue_rw_req(struct request *req) ring_req->u.rw.handle = info->handle; ring_req->operation = rq_data_dir(req) ? BLKIF_OP_WRITE : BLKIF_OP_READ; - if (req->cmd_flags & (REQ_FLUSH | REQ_FUA)) { + if (req->op == REQ_OP_FLUSH || req->cmd_flags & REQ_FUA) { /* * Ideally we can do an unordered flush-to-disk. * In case the backend onlysupports barriers, use that. @@ -739,7 +739,7 @@ static inline bool blkif_request_flush_invalid(struct request *req, struct blkfront_info *info) { return ((req->cmd_type != REQ_TYPE_FS) || - ((req->cmd_flags & REQ_FLUSH) && + ((req->op == REQ_OP_FLUSH) && !(info->feature_flush & REQ_FLUSH)) || ((req->cmd_flags & REQ_FUA) && !(info->feature_flush & REQ_FUA))); @@ -1692,7 +1692,7 @@ static int blkif_recover(struct blkfront_info *info) /* * Get the bios in the request so we can re-queue them. */ - if (copy[i].request->cmd_flags & REQ_FLUSH || + if (copy[i].request->op == REQ_OP_FLUSH || copy[i].request->op == REQ_OP_DISCARD || copy[i].request->cmd_flags & (REQ_FUA | REQ_SECURE)) { /* diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c index 37a8a90..6474ed3 100644 --- a/drivers/ide/ide-disk.c +++ b/drivers/ide/ide-disk.c @@ -431,7 +431,7 @@ static int idedisk_prep_fn(struct request_queue *q, struct request *rq) ide_drive_t *drive = q->queuedata; struct ide_cmd *cmd; - if (!(rq->cmd_flags & REQ_FLUSH)) + if (rq->op != REQ_OP_FLUSH) return BLKPREP_OK; if (rq->special) { diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 7c6e386..90135b6 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -2097,7 +2097,7 @@ static void dm_request_fn(struct request_queue *q) /* always use block 0 to find the target for flushes for now */ pos = 0; - if (!(rq->cmd_flags & REQ_FLUSH)) + if (rq->op != REQ_OP_FLUSH) pos = blk_rq_pos(rq); ti = dm_table_find_target(map, pos); @@ -2657,7 +2657,7 @@ static int dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx, /* always use block 0 to find the target for flushes for now */ pos = 0; - if (!(rq->cmd_flags & REQ_FLUSH)) + if (rq->op != REQ_OP_FLUSH) pos = blk_rq_pos(rq); ti = dm_table_find_target(map, pos); diff --git a/drivers/mmc/card/block.c b/drivers/mmc/card/block.c index cc9b1ff..7caf236 100644 --- a/drivers/mmc/card/block.c +++ b/drivers/mmc/card/block.c @@ -1691,7 +1691,7 @@ static u8 mmc_blk_prep_packed_list(struct mmc_queue *mq, struct request *req) !IS_ALIGNED(blk_rq_sectors(next), 8)) break; - if (next->op == REQ_OP_DISCARD || next->cmd_flags & REQ_FLUSH) + if (next->op == REQ_OP_DISCARD || next->op == REQ_OP_FLUSH) break; if (rq_data_dir(cur) != rq_data_dir(next)) @@ -2116,7 +2116,6 @@ static int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) struct mmc_card *card = md->queue.card; struct mmc_host *host = card->host; unsigned long flags; - unsigned int cmd_flags = req ? req->cmd_flags : 0; if (req && !mq->mqrq_prev->req) /* claim host only for the first request */ @@ -2140,7 +2139,7 @@ static int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) ret = mmc_blk_issue_secdiscard_rq(mq, req); else ret = mmc_blk_issue_discard_rq(mq, req); - } else if (cmd_flags & REQ_FLUSH) { + } else if (req && req->op == REQ_OP_FLUSH) { /* complete ongoing async transfer before issuing flush */ if (card->host->areq) mmc_blk_issue_rw_rq(mq, NULL); diff --git a/drivers/mmc/card/queue.h b/drivers/mmc/card/queue.h index f166e5b..6c67e4e 100644 --- a/drivers/mmc/card/queue.h +++ b/drivers/mmc/card/queue.h @@ -3,7 +3,7 @@ static inline bool mmc_req_is_special(struct request *req) { - return req && (req->cmd_flags & REQ_FLUSH || req->op == REQ_OP_DISCARD); + return req && (req->op == REQ_OP_FLUSH || req->op == REQ_OP_DISCARD); } struct request; diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c index b2e0bce..67da1cd 100644 --- a/drivers/mtd/mtd_blkdevs.c +++ b/drivers/mtd/mtd_blkdevs.c @@ -87,7 +87,7 @@ static int do_blktrans_request(struct mtd_blktrans_ops *tr, if (req->cmd_type != REQ_TYPE_FS) return -EIO; - if (req->cmd_flags & REQ_FLUSH) + if (req->op == REQ_OP_FLUSH) return tr->flush(dev); if (blk_rq_pos(req) + blk_rq_cur_sectors(req) > diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index c51205a..334a5a1 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -668,7 +668,7 @@ static int nvme_queue_rq(struct blk_mq_hw_ctx *hctx, } else { if (req->cmd_type == REQ_TYPE_DRV_PRIV) memcpy(&cmnd, req->cmd, sizeof(cmnd)); - else if (req->cmd_flags & REQ_FLUSH) + else if (req->op == REQ_OP_FLUSH) nvme_setup_flush(ns, &cmnd); else nvme_setup_rw(ns, req, &cmnd); diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 2737527..abaaa7e 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -1143,12 +1143,11 @@ static int sd_init_command(struct scsi_cmnd *cmd) return sd_setup_discard_cmnd(cmd); case REQ_OP_WRITE_SAME: return sd_setup_write_same_cmnd(cmd); + case REQ_OP_FLUSH: + return sd_setup_flush_cmnd(cmd); case REQ_OP_READ: case REQ_OP_WRITE: - if (rq->cmd_flags & REQ_FLUSH) - return sd_setup_flush_cmnd(cmd); - else - return sd_setup_read_write_cmnd(cmd); + return sd_setup_read_write_cmnd(cmd); default: BUG(); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 5d19b393..b4ef8d7 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -239,6 +239,7 @@ enum req_op { REQ_OP_WRITE, REQ_OP_DISCARD, /* request to discard sectors */ REQ_OP_WRITE_SAME, /* write same block many times */ + REQ_OP_FLUSH, /* request for cache flush */ }; typedef unsigned int blk_qc_t; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 83d6f033..ec0d0d0 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -651,6 +651,9 @@ static inline bool rq_mergeable(struct request *rq) if (rq->cmd_type != REQ_TYPE_FS) return false; + if (rq->op == REQ_OP_FLUSH) + return false; + if (rq->cmd_flags & REQ_NOMERGE_FLAGS) return false; diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index 4a9ac83..3a54c83 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -223,6 +223,8 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes, what |= MASK_TC_BIT(op_flags, FUA); if (op == REQ_OP_DISCARD) what |= BLK_TC_ACT(BLK_TC_DISCARD); + if (op == REQ_OP_FLUSH) + what |= BLK_TC_ACT(BLK_TC_FLUSH); pid = tsk->pid; if (act_log_check(bt, what, sector, pid)) @@ -1775,7 +1777,8 @@ void blk_fill_rwbs(char *rwbs, int op, u32 rw, int bytes) { int i = 0; - if (rw & REQ_FLUSH) + if (rw & REQ_FLUSH || + op == REQ_OP_FLUSH) rwbs[i++] = 'F'; if (op == REQ_OP_WRITE) -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:23 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 071D329E00 for ; Tue, 5 Jan 2016 14:55:23 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id ED563304039 for ; Tue, 5 Jan 2016 12:55:22 -0800 (PST) X-ASG-Debug-ID: 1452027320-04cbb07e17054e0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id BwRRqYcsGkXyBPkm (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:20 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id D5E251136FD; Tue, 5 Jan 2016 20:55:19 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuX024032; Tue, 5 Jan 2016 15:55:17 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 34/35] block: add QUEUE_FLAGs for flush and fua Date: Tue, 5 Jan 2016 14:53:37 -0600 X-ASG-Orig-Subj: [PATCH 34/35] block: add QUEUE_FLAGs for flush and fua Message-Id: <1452027218-32303-35-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027320 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The last patch added a REQ_OP_FLUSH for request_fn drivers and the next patch renames REQ_FLUSH to REQ_PREFLUSH which will be used by file systems and make_request_fn drivers. This leaves REQ_FLUSH/REQ_FUA defined for drivers to tell the block layer if flush/fua is supported. The names are confusing and I bet will will accidentally be used by people to request flushes. To avoid that, this patch adds QUEUE_FLAGs for flush and fua which drivers will use to indicate what they support. Signed-off-by: Mike Christie --- block/blk-core.c | 3 +- block/blk-flush.c | 12 ++++---- block/blk-settings.c | 20 -------------- drivers/block/drbd/drbd_main.c | 3 +- drivers/block/loop.c | 2 +- drivers/block/mtip32xx/mtip32xx.c | 3 +- drivers/block/nbd.c | 6 ++-- drivers/block/osdblk.c | 2 +- drivers/block/ps3disk.c | 2 +- drivers/block/skd_main.c | 3 +- drivers/block/virtio_blk.c | 4 +-- drivers/block/xen-blkback/xenbus.c | 2 +- drivers/block/xen-blkfront.c | 55 ++++++++++++++++++++++--------------- drivers/ide/ide-disk.c | 6 ++-- drivers/md/bcache/super.c | 4 +-- drivers/md/dm-table.c | 32 +++++++++++++-------- drivers/md/md.c | 3 +- drivers/md/raid5-cache.c | 3 +- drivers/mmc/card/block.c | 3 +- drivers/mtd/mtd_blkdevs.c | 2 +- drivers/nvme/host/core.c | 6 ++-- drivers/scsi/sd.c | 13 +++++---- drivers/target/target_core_iblock.c | 6 ++-- include/linux/blkdev.h | 6 ++-- 24 files changed, 107 insertions(+), 94 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index ae2afab..bb29230 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1969,7 +1969,8 @@ generic_make_request_checks(struct bio *bio) * drivers without flush support don't have to worry * about them. */ - if ((bio->bi_rw & (REQ_FLUSH | REQ_FUA)) && !q->flush_flags) { + if ((bio->bi_rw & (REQ_FLUSH | REQ_FUA)) && + !(blk_queue_flush(q) || blk_queue_fua(q))) { bio->bi_rw &= ~(REQ_FLUSH | REQ_FUA); if (!nr_sectors) { err = 0; diff --git a/block/blk-flush.c b/block/blk-flush.c index 0e5561e..633f9b3 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -95,17 +95,18 @@ enum { static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq); -static unsigned int blk_flush_policy(unsigned int fflags, struct request *rq) +static unsigned int blk_flush_policy(struct request *rq) { + struct request_queue *q = rq->q; unsigned int policy = 0; if (blk_rq_sectors(rq)) policy |= REQ_FSEQ_DATA; - if (fflags & REQ_FLUSH) { + if (blk_queue_flush(q)) { if (rq->cmd_flags & REQ_FLUSH) policy |= REQ_FSEQ_PREFLUSH; - if (!(fflags & REQ_FUA) && (rq->cmd_flags & REQ_FUA)) + if (!blk_queue_fua(q) && (rq->cmd_flags & REQ_FUA)) policy |= REQ_FSEQ_POSTFLUSH; } return policy; @@ -385,8 +386,7 @@ static void mq_flush_data_end_io(struct request *rq, int error) void blk_insert_flush(struct request *rq) { struct request_queue *q = rq->q; - unsigned int fflags = q->flush_flags; /* may change, cache */ - unsigned int policy = blk_flush_policy(fflags, rq); + unsigned int policy = blk_flush_policy(rq); struct blk_flush_queue *fq = blk_get_flush_queue(q, rq->mq_ctx); /* @@ -394,7 +394,7 @@ void blk_insert_flush(struct request *rq) * REQ_FLUSH and FUA for the driver. */ rq->cmd_flags &= ~REQ_FLUSH; - if (!(fflags & REQ_FUA)) + if (!blk_queue_fua(q)) rq->cmd_flags &= ~REQ_FUA; /* diff --git a/block/blk-settings.c b/block/blk-settings.c index dd49735..3cef016 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -820,26 +820,6 @@ void blk_queue_update_dma_alignment(struct request_queue *q, int mask) } EXPORT_SYMBOL(blk_queue_update_dma_alignment); -/** - * blk_queue_flush - configure queue's cache flush capability - * @q: the request queue for the device - * @flush: 0, REQ_FLUSH or REQ_FLUSH | REQ_FUA - * - * Tell block layer cache flush capability of @q. If it supports - * flushing, REQ_FLUSH should be set. If it supports bypassing - * write cache for individual writes, REQ_FUA should be set. - */ -void blk_queue_flush(struct request_queue *q, unsigned int flush) -{ - WARN_ON_ONCE(flush & ~(REQ_FLUSH | REQ_FUA)); - - if (WARN_ON_ONCE(!(flush & REQ_FLUSH) && (flush & REQ_FUA))) - flush &= ~REQ_FUA; - - q->flush_flags = flush & (REQ_FLUSH | REQ_FUA); -} -EXPORT_SYMBOL_GPL(blk_queue_flush); - void blk_queue_flush_queueable(struct request_queue *q, bool queueable) { q->flush_not_queueable = !queueable; diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index df2ce0b..7bc78ab 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -2761,7 +2761,8 @@ enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsig q->backing_dev_info.congested_data = device; blk_queue_make_request(q, drbd_make_request); - blk_queue_flush(q, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); /* Setting the max_hw_sectors to an odd value of 8kibyte here This triggers a max_bio_size message upon first attach or connect */ blk_queue_max_hw_sectors(q, DRBD_MAX_BIO_SIZE_SAFE >> 8); diff --git a/drivers/block/loop.c b/drivers/block/loop.c index a3d1293..297a91f 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -937,7 +937,7 @@ static int loop_set_fd(struct loop_device *lo, fmode_t mode, mapping_set_gfp_mask(mapping, lo->old_gfp_mask & ~(__GFP_IO|__GFP_FS)); if (!(lo_flags & LO_FLAGS_READ_ONLY) && file->f_op->fsync) - blk_queue_flush(lo->lo_queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, lo->lo_queue); loop_update_dio(lo); set_capacity(lo->lo_disk, size); diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c index 8751caa..461a3f9 100644 --- a/drivers/block/mtip32xx/mtip32xx.c +++ b/drivers/block/mtip32xx/mtip32xx.c @@ -3908,7 +3908,8 @@ skip_create_disk: * write back cache is not supported in the device. FUA depends on * write back cache support, hence setting flush support to zero. */ - blk_queue_flush(dd->queue, 0); + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, dd->queue) ; + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, dd->queue); /* Signal trim support */ if (dd->trim_supp == true) { diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index ced3382..c21b421 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -750,9 +750,11 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, nbd->disk->queue); if (nbd->flags & NBD_FLAG_SEND_FLUSH) - blk_queue_flush(nbd->disk->queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, + nbd->disk->queue); else - blk_queue_flush(nbd->disk->queue, 0); + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, + nbd->disk->queue); thread = kthread_run(nbd_thread_send, nbd, "%s", nbd_name(nbd)); diff --git a/drivers/block/osdblk.c b/drivers/block/osdblk.c index 3f8a0a0..d03f06a 100644 --- a/drivers/block/osdblk.c +++ b/drivers/block/osdblk.c @@ -437,7 +437,7 @@ static int osdblk_init_disk(struct osdblk_device *osdev) blk_queue_stack_limits(q, osd_request_queue(osdev->osd)); blk_queue_prep_rq(q, blk_queue_start_tag); - blk_queue_flush(q, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); disk->queue = q; diff --git a/drivers/block/ps3disk.c b/drivers/block/ps3disk.c index 95c524b..5985fdc 100644 --- a/drivers/block/ps3disk.c +++ b/drivers/block/ps3disk.c @@ -468,7 +468,7 @@ static int ps3disk_probe(struct ps3_system_bus_device *_dev) blk_queue_dma_alignment(queue, dev->blk_size-1); blk_queue_logical_block_size(queue, dev->blk_size); - blk_queue_flush(queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, queue); blk_queue_max_segments(queue, -1); blk_queue_max_segment_size(queue, dev->bounce_size); diff --git a/drivers/block/skd_main.c b/drivers/block/skd_main.c index a15cc2e..68d0e22 100644 --- a/drivers/block/skd_main.c +++ b/drivers/block/skd_main.c @@ -4409,7 +4409,8 @@ static int skd_cons_disk(struct skd_device *skdev) disk->queue = q; q->queuedata = skdev; - blk_queue_flush(q, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); blk_queue_max_segments(q, skdev->sgs_per_request); blk_queue_max_hw_sectors(q, SKD_N_MAX_SECTORS); diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index c6d3bc2..b8a508b 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -489,9 +489,9 @@ static void virtblk_update_cache_mode(struct virtio_device *vdev) struct virtio_blk *vblk = vdev->priv; if (writeback) - blk_queue_flush(vblk->disk->queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, vblk->disk->queue); else - blk_queue_flush(vblk->disk->queue, 0); + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, vblk->disk->queue); revalidate_disk(vblk->disk); } diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index f53cff4..4569692 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -413,7 +413,7 @@ static int xen_vbd_create(struct xen_blkif *blkif, blkif_vdev_t handle, vbd->type |= VDISK_REMOVABLE; q = bdev_get_queue(bdev); - if (q && q->flush_flags) + if (q && (blk_queue_flush(q) || blk_queue_fua(q))) vbd->flush_support = true; if (q && blk_queue_secdiscard(q)) diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 1d48f0a..6007c86 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -146,6 +146,7 @@ struct blkfront_info unsigned int persistent_gnts_c; unsigned long shadow_free; unsigned int feature_flush; + unsigned int feature_fua; unsigned int feature_discard:1; unsigned int feature_secdiscard:1; unsigned int discard_granularity; @@ -655,19 +656,15 @@ static int blkif_queue_rw_req(struct request *req) * implement it the same way. (It's also a FLUSH+FUA, * since it is guaranteed ordered WRT previous writes.) */ - switch (info->feature_flush & - ((REQ_FLUSH|REQ_FUA))) { - case REQ_FLUSH|REQ_FUA: + if (blk_queue_flush(info->rq) && + blk_queue_fua(info->rq)) ring_req->operation = BLKIF_OP_WRITE_BARRIER; - break; - case REQ_FLUSH: + else if (blk_queue_flush(info->rq)) ring_req->operation = BLKIF_OP_FLUSH_DISKCACHE; - break; - default: + else ring_req->operation = 0; - } } ring_req->u.rw.nr_segments = num_grant; } @@ -740,9 +737,9 @@ static inline bool blkif_request_flush_invalid(struct request *req, { return ((req->cmd_type != REQ_TYPE_FS) || ((req->op == REQ_OP_FLUSH) && - !(info->feature_flush & REQ_FLUSH)) || + !(blk_queue_flush(info->rq))) || ((req->cmd_flags & REQ_FUA) && - !(info->feature_flush & REQ_FUA))); + !(blk_queue_fua(info->rq)))); } static int blkif_queue_rq(struct blk_mq_hw_ctx *hctx, @@ -838,23 +835,30 @@ static int xlvbd_init_blk_queue(struct gendisk *gd, u16 sector_size, return 0; } -static const char *flush_info(unsigned int feature_flush) +static const char *flush_info(struct blkfront_info *info) { - switch (feature_flush & ((REQ_FLUSH | REQ_FUA))) { - case REQ_FLUSH|REQ_FUA: + if (blk_queue_flush(info->rq) && blk_queue_fua(info->rq)) return "barrier: enabled;"; - case REQ_FLUSH: + else if (blk_queue_flush(info->rq)) return "flush diskcache: enabled;"; - default: + else return "barrier or flush: disabled;"; - } } static void xlvbd_flush(struct blkfront_info *info) { - blk_queue_flush(info->rq, info->feature_flush); + if (info->feature_flush) + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, info->rq); + else + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, info->rq); + + if (info->feature_fua) + queue_flag_set_unlocked(QUEUE_FLAG_FUA, info->rq); + else + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, info->rq); + pr_info("blkfront: %s: %s %s %s %s %s\n", - info->gd->disk_name, flush_info(info->feature_flush), + info->gd->disk_name, flush_info(info), "persistent grants:", info->feature_persistent ? "enabled;" : "disabled;", "indirect descriptors:", info->max_indirect_segments ? "enabled;" : "disabled;"); @@ -1373,6 +1377,7 @@ static irqreturn_t blkif_interrupt(int irq, void *dev_id) if (unlikely(error)) { if (error == -EOPNOTSUPP) error = 0; + info->feature_fua = 0; info->feature_flush = 0; xlvbd_flush(info); } @@ -1936,6 +1941,7 @@ static int blkfront_gather_backend_features(struct blkfront_info *info) unsigned int indirect_segments; info->feature_flush = 0; + info->feature_fua = 0; err = xenbus_gather(XBT_NIL, info->xbdev->otherend, "feature-barrier", "%d", &barrier, @@ -1948,8 +1954,11 @@ static int blkfront_gather_backend_features(struct blkfront_info *info) * * If there are barriers, then we use flush. */ - if (!err && barrier) - info->feature_flush = REQ_FLUSH | REQ_FUA; + if (!err && barrier) { + info->feature_flush = 1; + info->feature_fua = 1; + } + /* * And if there is "feature-flush-cache" use that above * barriers. @@ -1958,8 +1967,10 @@ static int blkfront_gather_backend_features(struct blkfront_info *info) "feature-flush-cache", "%d", &flush, NULL); - if (!err && flush) - info->feature_flush = REQ_FLUSH; + if (!err && flush) { + info->feature_flush = 1; + info->feature_fua = 0; + } err = xenbus_gather(XBT_NIL, info->xbdev->otherend, "feature-discard", "%d", &discard, diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c index 6474ed3..4501ca7 100644 --- a/drivers/ide/ide-disk.c +++ b/drivers/ide/ide-disk.c @@ -522,8 +522,8 @@ static int ide_do_setfeature(ide_drive_t *drive, u8 feature, u8 nsect) static void update_flush(ide_drive_t *drive) { u16 *id = drive->id; - unsigned flush = 0; + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, drive->queue); if (drive->dev_flags & IDE_DFLAG_WCACHE) { unsigned long long capacity; int barrier; @@ -546,12 +546,10 @@ static void update_flush(ide_drive_t *drive) drive->name, barrier ? "" : "not "); if (barrier) { - flush = REQ_FLUSH; blk_queue_prep_rq(drive->queue, idedisk_prep_fn); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, drive->queue); } } - - blk_queue_flush(drive->queue, flush); } ide_devset_get_flag(wcache, IDE_DFLAG_WCACHE); diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index b11f7f4..3d75a0f 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -820,8 +820,8 @@ static int bcache_device_init(struct bcache_device *d, unsigned block_size, clear_bit(QUEUE_FLAG_ADD_RANDOM, &d->disk->queue->queue_flags); set_bit(QUEUE_FLAG_DISCARD, &d->disk->queue->queue_flags); - blk_queue_flush(q, REQ_FLUSH|REQ_FUA); - + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); return 0; } diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 061152a..5e940f7 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1312,13 +1312,21 @@ static void dm_table_verify_integrity(struct dm_table *t) static int device_flush_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { - unsigned flush = (*(unsigned *)data); struct request_queue *q = bdev_get_queue(dev->bdev); - return q && (q->flush_flags & flush); + return q && blk_queue_flush(q); } -static bool dm_table_supports_flush(struct dm_table *t, unsigned flush) +static int device_fua_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return q && blk_queue_fua(q); +} + +static bool dm_table_supports_flush(struct dm_table *t, + iterate_devices_callout_fn support_test) { struct dm_target *ti; unsigned i = 0; @@ -1339,7 +1347,7 @@ static bool dm_table_supports_flush(struct dm_table *t, unsigned flush) return true; if (ti->type->iterate_devices && - ti->type->iterate_devices(ti, device_flush_capable, &flush)) + ti->type->iterate_devices(ti, support_test, NULL)) return true; } @@ -1470,8 +1478,6 @@ static bool dm_table_supports_discards(struct dm_table *t) void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, struct queue_limits *limits) { - unsigned flush = 0; - /* * Copy table's limits to the DM device's request_queue */ @@ -1482,12 +1488,14 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, else queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, q); - if (dm_table_supports_flush(t, REQ_FLUSH)) { - flush |= REQ_FLUSH; - if (dm_table_supports_flush(t, REQ_FUA)) - flush |= REQ_FUA; - } - blk_queue_flush(q, flush); + if (dm_table_supports_flush(t, device_flush_capable)) { + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); + if (dm_table_supports_flush(t, device_fua_capable)) + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); + else + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, q); + } else + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, q); if (!dm_table_discard_zeroes_data(t)) q->limits.discard_zeroes_data = 0; diff --git a/drivers/md/md.c b/drivers/md/md.c index ee1ef20..7e16813 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -5040,7 +5040,8 @@ static int md_alloc(dev_t dev, char *name) disk->fops = &md_fops; disk->private_data = mddev; disk->queue = mddev->queue; - blk_queue_flush(mddev->queue, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, mddev->queue); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, mddev->queue); /* Allow extended partitions. This makes the * 'mdp' device redundant, but we can't really * remove it now. diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index e19c2f0..311b7f2 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -1189,6 +1189,7 @@ ioerr: int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev) { + struct request_queue *q = rdev->bdev->bd_disk->queue; struct r5l_log *log; if (PAGE_SIZE != 4096) @@ -1198,7 +1199,7 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev) return -ENOMEM; log->rdev = rdev; - log->need_cache_flush = (rdev->bdev->bd_disk->queue->flush_flags != 0); + log->need_cache_flush = (blk_queue_flush(q) || blk_queue_fua(q)); log->uuid_checksum = crc32c_le(~0, rdev->mddev->uuid, sizeof(rdev->mddev->uuid)); diff --git a/drivers/mmc/card/block.c b/drivers/mmc/card/block.c index 7caf236..c1fad5f 100644 --- a/drivers/mmc/card/block.c +++ b/drivers/mmc/card/block.c @@ -2277,7 +2277,8 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct mmc_card *card, ((card->ext_csd.rel_param & EXT_CSD_WR_REL_PARAM_EN) || card->ext_csd.rel_sectors)) { md->flags |= MMC_BLK_REL_WR; - blk_queue_flush(md->queue.queue, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, md->queue.queue); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, md->queue.queue); } if (mmc_card_mmc(card) && diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c index 67da1cd..4462627 100644 --- a/drivers/mtd/mtd_blkdevs.c +++ b/drivers/mtd/mtd_blkdevs.c @@ -409,7 +409,7 @@ int add_mtd_blktrans_dev(struct mtd_blktrans_dev *new) goto error3; if (tr->flush) - blk_queue_flush(new->rq, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, new->rq); new->rq->queuedata = new; blk_queue_logical_block_size(new->rq, tr->blksize); diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 1375a83..9f253bc 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -1078,8 +1078,10 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) } if (ctrl->stripe_size) blk_queue_chunk_sectors(ns->queue, ctrl->stripe_size >> 9); - if (ctrl->vwc & NVME_CTRL_VWC_PRESENT) - blk_queue_flush(ns->queue, REQ_FLUSH | REQ_FUA); + if (ctrl->vwc & NVME_CTRL_VWC_PRESENT) { + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, ns->queue); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, ns->queue); + } blk_queue_virt_boundary(ns->queue, ctrl->page_size - 1); disk->major = nvme_major; diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index abaaa7e..91142f1 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -137,15 +137,18 @@ static const char *sd_cache_types[] = { static void sd_set_flush_flag(struct scsi_disk *sdkp) { - unsigned flush = 0; + struct request_queue *q = sdkp->disk->queue; if (sdkp->WCE) { - flush |= REQ_FLUSH; + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); if (sdkp->DPOFUA) - flush |= REQ_FUA; + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); + else + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, q); + } else { + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, q); + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, q); } - - blk_queue_flush(sdkp->disk->queue, flush); } static ssize_t diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c index b83195b..78bf469 100644 --- a/drivers/target/target_core_iblock.c +++ b/drivers/target/target_core_iblock.c @@ -671,11 +671,11 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, * Force writethrough using WRITE_FUA if a volatile write cache * is not enabled, or if initiator set the Force Unit Access bit. */ - if (q->flush_flags & REQ_FUA) { + if (blk_queue_fua(q)) { if (cmd->se_cmd_flags & SCF_FUA) { op = REQ_OP_WRITE; op_flags = WRITE_FUA; - } else if (!(q->flush_flags & REQ_FLUSH)) { + } else if (!blk_queue_flush(q)) { op = REQ_OP_WRITE; op_flags = WRITE_FUA; } else { @@ -842,7 +842,7 @@ static bool iblock_get_write_cache(struct se_device *dev) struct block_device *bd = ib_dev->ibd_bd; struct request_queue *q = bdev_get_queue(bd); - return q->flush_flags & REQ_FLUSH; + return blk_queue_flush(q); } static const struct target_backend_ops iblock_ops = { diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index ec0d0d0..09724ed 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -434,7 +434,6 @@ struct request_queue { /* * for flush operations */ - unsigned int flush_flags; unsigned int flush_not_queueable:1; struct blk_flush_queue *fq; @@ -492,6 +491,8 @@ struct request_queue { #define QUEUE_FLAG_INIT_DONE 20 /* queue is initialized */ #define QUEUE_FLAG_NO_SG_MERGE 21 /* don't attempt to merge SG segments*/ #define QUEUE_FLAG_POLL 22 /* IO polling enabled if set */ +#define QUEUE_FLAG_FLUSH 23 /* supports FLUSH/PREFLUSH */ +#define QUEUE_FLAG_FUA 24 /* supports FUA */ #define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \ (1 << QUEUE_FLAG_STACKABLE) | \ @@ -580,6 +581,8 @@ static inline void queue_flag_clear(unsigned int flag, struct request_queue *q) #define blk_queue_discard(q) test_bit(QUEUE_FLAG_DISCARD, &(q)->queue_flags) #define blk_queue_secdiscard(q) (blk_queue_discard(q) && \ test_bit(QUEUE_FLAG_SECDISCARD, &(q)->queue_flags)) +#define blk_queue_flush(q) test_bit(QUEUE_FLAG_FLUSH, &(q)->queue_flags) +#define blk_queue_fua(q) test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags) #define blk_noretry_request(rq) \ ((rq)->cmd_flags & (REQ_FAILFAST_DEV|REQ_FAILFAST_TRANSPORT| \ @@ -1007,7 +1010,6 @@ extern void blk_queue_update_dma_alignment(struct request_queue *, int); extern void blk_queue_softirq_done(struct request_queue *, softirq_done_fn *); extern void blk_queue_rq_timed_out(struct request_queue *, rq_timed_out_fn *); extern void blk_queue_rq_timeout(struct request_queue *, unsigned int); -extern void blk_queue_flush(struct request_queue *q, unsigned int flush); extern void blk_queue_flush_queueable(struct request_queue *q, bool queueable); extern struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev); -- 1.8.3.1 From mchristi@redhat.com Tue Jan 5 14:55:26 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C421029E0A for ; Tue, 5 Jan 2016 14:55:26 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 8A5AC8F804C for ; Tue, 5 Jan 2016 12:55:26 -0800 (PST) X-ASG-Debug-ID: 1452027323-04bdf03afc04e80001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id TfM7BkvmV4btF6iS (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 12:55:23 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 926193798DA; Tue, 5 Jan 2016 20:55:22 +0000 (UTC) Received: from rh2.redhat.com (vpn-63-72.rdu2.redhat.com [10.10.63.72]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u05KrfuY024032; Tue, 5 Jan 2016 15:55:19 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 35/35] block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH Date: Tue, 5 Jan 2016 14:53:38 -0600 X-ASG-Orig-Subj: [PATCH 35/35] block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH Message-Id: <1452027218-32303-36-git-send-email-mchristi@redhat.com> In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452027323 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie To avoid confusion between REQ_OP_FLUSH, which is handled by request_fn drivers, and upper layers requesting the block layer perform a flush sequence along with possibly a WRITE, this patch renames REQ_FLUSH to REQ_PREFLUSH. Signed-off-by: Mike Christie --- Documentation/block/writeback_cache_control.txt | 31 +++++++++++++------------ block/blk-core.c | 12 +++++----- block/blk-flush.c | 4 ++-- block/blk-mq.c | 4 ++-- drivers/block/drbd/drbd_actlog.c | 2 +- drivers/block/drbd/drbd_main.c | 2 +- drivers/block/drbd/drbd_receiver.c | 2 +- drivers/block/drbd/drbd_req.c | 2 +- drivers/md/bcache/journal.c | 2 +- drivers/md/bcache/request.c | 8 +++---- drivers/md/dm-cache-target.c | 9 +++---- drivers/md/dm-crypt.c | 7 +++--- drivers/md/dm-era-target.c | 4 ++-- drivers/md/dm-io.c | 2 +- drivers/md/dm-log-writes.c | 2 +- drivers/md/dm-raid1.c | 5 ++-- drivers/md/dm-region-hash.c | 4 ++-- drivers/md/dm-snap.c | 6 ++--- drivers/md/dm-stripe.c | 2 +- drivers/md/dm-thin.c | 8 +++---- drivers/md/dm.c | 12 +++++----- drivers/md/linear.c | 2 +- drivers/md/md.c | 2 +- drivers/md/multipath.c | 2 +- drivers/md/raid0.c | 2 +- drivers/md/raid1.c | 3 ++- drivers/md/raid10.c | 2 +- drivers/md/raid5-cache.c | 2 +- drivers/md/raid5.c | 2 +- fs/btrfs/check-integrity.c | 8 +++---- fs/jbd2/journal.c | 2 +- fs/xfs/xfs_buf.c | 2 +- include/linux/blk_types.h | 8 +++---- include/linux/fs.h | 4 ++-- include/trace/events/f2fs.h | 2 +- kernel/trace/blktrace.c | 5 ++-- 36 files changed, 92 insertions(+), 86 deletions(-) diff --git a/Documentation/block/writeback_cache_control.txt b/Documentation/block/writeback_cache_control.txt index ea5550f..9869f18 100644 --- a/Documentation/block/writeback_cache_control.txt +++ b/Documentation/block/writeback_cache_control.txt @@ -20,11 +20,11 @@ a forced cache flush, and the Force Unit Access (FUA) flag for requests. Explicit cache flushes ---------------------- -The REQ_FLUSH flag can be OR ed into the r/w flags of a bio submitted from +The REQ_PREFLUSH flag can be OR ed into the r/w flags of a bio submitted from the filesystem and will make sure the volatile cache of the storage device has been flushed before the actual I/O operation is started. This explicitly guarantees that previously completed write requests are on non-volatile -storage before the flagged bio starts. In addition the REQ_FLUSH flag can be +storage before the flagged bio starts. In addition the REQ_PREFLUSH flag can be set on an otherwise empty bio structure, which causes only an explicit cache flush without any dependent I/O. It is recommend to use the blkdev_issue_flush() helper for a pure cache flush. @@ -41,21 +41,21 @@ signaled after the data has been committed to non-volatile storage. Implementation details for filesystems -------------------------------------- -Filesystems can simply set the REQ_FLUSH and REQ_FUA bits and do not have to +Filesystems can simply set the REQ_PREFLUSH and REQ_FUA bits and do not have to worry if the underlying devices need any explicit cache flushing and how -the Forced Unit Access is implemented. The REQ_FLUSH and REQ_FUA flags +the Forced Unit Access is implemented. The REQ_PREFLUSH and REQ_FUA flags may both be set on a single bio. Implementation details for make_request_fn based block drivers -------------------------------------------------------------- -These drivers will always see the REQ_FLUSH and REQ_FUA bits as they sit +These drivers will always see the REQ_PREFLUSH and REQ_FUA bits as they sit directly below the submit_bio interface. For remapping drivers the REQ_FUA bits need to be propagated to underlying devices, and a global flush needs -to be implemented for bios with the REQ_FLUSH bit set. For real device -drivers that do not have a volatile cache the REQ_FLUSH and REQ_FUA bits -on non-empty bios can simply be ignored, and REQ_FLUSH requests without +to be implemented for bios with the REQ_PREFLUSH bit set. For real device +drivers that do not have a volatile cache the REQ_PREFLUSH and REQ_FUA bits +on non-empty bios can simply be ignored, and REQ_PREFLUSH requests without data can be completed successfully without doing any work. Drivers for devices with volatile caches need to implement the support for these flags themselves without any help from the block layer. @@ -65,21 +65,22 @@ Implementation details for request_fn based block drivers -------------------------------------------------------------- For devices that do not support volatile write caches there is no driver -support required, the block layer completes empty REQ_FLUSH requests before -entering the driver and strips off the REQ_FLUSH and REQ_FUA bits from -requests that have a payload. For devices with volatile write caches the -driver needs to tell the block layer that it supports flushing caches by +support required, the block layer completes empty REQ_PREFLUSH requests +before entering the driver and strips off the REQ_PREFLUSH and REQ_FUA bits +from requests that have a payload. For devices with volatile write caches +the driver needs to tell the block layer that it supports flushing caches by doing: - blk_queue_flush(sdkp->disk->queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, sdkp->disk->queue); and handle empty REQ_OP_FLUSH requests in its prep_fn/request_fn. Note that -REQ_FLUSH requests with a payload are automatically turned into a sequence +REQ_PREFLUSH requests with a payload are automatically turned into a sequence of an empty REQ_OP_FLUSH request followed by the actual write by the block layer. For devices that also support the FUA bit the block layer needs to be told to pass through the REQ_FUA bit using: - blk_queue_flush(sdkp->disk->queue, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, sdkp->disk->queue); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, sdkp->disk->queue); and the driver must handle write requests that have the REQ_FUA bit set in prep_fn/request_fn. If the FUA bit is not natively supported the block diff --git a/block/blk-core.c b/block/blk-core.c index bb29230..dbc9082 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1029,7 +1029,7 @@ static bool blk_rq_should_init_elevator(struct bio *bio) * Flush requests do not use the elevator so skip initialization. * This allows a request to share the flush and elevator data. */ - if (bio->bi_rw & (REQ_FLUSH | REQ_FUA)) + if (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA)) return false; return true; @@ -1737,7 +1737,7 @@ static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio) return BLK_QC_T_NONE; } - if (bio->bi_rw & (REQ_FLUSH | REQ_FUA)) { + if (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA)) { spin_lock_irq(q->queue_lock); where = ELEVATOR_INSERT_FLUSH; goto get_rq; @@ -1969,9 +1969,9 @@ generic_make_request_checks(struct bio *bio) * drivers without flush support don't have to worry * about them. */ - if ((bio->bi_rw & (REQ_FLUSH | REQ_FUA)) && + if ((bio->bi_rw & (REQ_PREFLUSH | REQ_FUA)) && !(blk_queue_flush(q) || blk_queue_fua(q))) { - bio->bi_rw &= ~(REQ_FLUSH | REQ_FUA); + bio->bi_rw &= ~(REQ_PREFLUSH | REQ_FUA); if (!nr_sectors) { err = 0; goto end_io; @@ -2218,7 +2218,7 @@ int blk_insert_cloned_request(struct request_queue *q, struct request *rq) */ BUG_ON(blk_queued_rq(rq)); - if (rq->cmd_flags & (REQ_FLUSH|REQ_FUA)) + if (rq->cmd_flags & (REQ_PREFLUSH | REQ_FUA)) where = ELEVATOR_INSERT_FLUSH; add_acct_request(q, rq, where); @@ -3310,7 +3310,7 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) /* * rq is already accounted, so use raw insert */ - if (rq->cmd_flags & (REQ_FLUSH | REQ_FUA)) + if (rq->cmd_flags & (REQ_PREFLUSH | REQ_FUA)) __elv_add_request(q, rq, ELEVATOR_INSERT_FLUSH); else __elv_add_request(q, rq, ELEVATOR_INSERT_SORT_MERGE); diff --git a/block/blk-flush.c b/block/blk-flush.c index 633f9b3..9cf387c 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -104,7 +104,7 @@ static unsigned int blk_flush_policy(struct request *rq) policy |= REQ_FSEQ_DATA; if (blk_queue_flush(q)) { - if (rq->cmd_flags & REQ_FLUSH) + if (rq->cmd_flags & REQ_PREFLUSH) policy |= REQ_FSEQ_PREFLUSH; if (!blk_queue_fua(q) && (rq->cmd_flags & REQ_FUA)) policy |= REQ_FSEQ_POSTFLUSH; @@ -393,7 +393,7 @@ void blk_insert_flush(struct request *rq) * @policy now records what operations need to be done. Adjust * REQ_FLUSH and FUA for the driver. */ - rq->cmd_flags &= ~REQ_FLUSH; + rq->cmd_flags &= ~REQ_PREFLUSH; if (!blk_queue_fua(q)) rq->cmd_flags &= ~REQ_FUA; diff --git a/block/blk-mq.c b/block/blk-mq.c index 72a028b..bb376a8 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1244,7 +1244,7 @@ static int blk_mq_direct_issue_request(struct request *rq, blk_qc_t *cookie) static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) { const int is_sync = rw_is_sync(bio->bi_op, bio->bi_rw); - const int is_flush_fua = bio->bi_rw & (REQ_FLUSH | REQ_FUA); + const int is_flush_fua = bio->bi_rw & (REQ_PREFLUSH | REQ_FUA); struct blk_map_ctx data; struct request *rq; unsigned int request_count = 0; @@ -1341,7 +1341,7 @@ done: static blk_qc_t blk_sq_make_request(struct request_queue *q, struct bio *bio) { const int is_sync = rw_is_sync(bio->bi_op, bio->bi_rw); - const int is_flush_fua = bio->bi_rw & (REQ_FLUSH | REQ_FUA); + const int is_flush_fua = bio->bi_rw & (REQ_PREFLUSH | REQ_FUA); struct blk_plug *plug; unsigned int request_count = 0; struct blk_map_ctx data; diff --git a/drivers/block/drbd/drbd_actlog.c b/drivers/block/drbd/drbd_actlog.c index 2fa8534..9f5e00b 100644 --- a/drivers/block/drbd/drbd_actlog.c +++ b/drivers/block/drbd/drbd_actlog.c @@ -148,7 +148,7 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, device->md_io.error = -ENODEV; if ((op == REQ_OP_WRITE) && !test_bit(MD_NO_FUA, &device->flags)) - op_flags |= REQ_FUA | REQ_FLUSH; + op_flags |= REQ_FUA | REQ_PREFLUSH; op_flags |= REQ_SYNC | REQ_NOIDLE; bio = bio_alloc_drbd(GFP_NOIO); diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 7bc78ab..88543d1 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1608,7 +1608,7 @@ static u32 bio_flags_to_wire(struct drbd_connection *connection, struct bio *bio if (connection->agreed_pro_version >= 95) return (bio->bi_rw & REQ_SYNC ? DP_RW_SYNC : 0) | (bio->bi_rw & REQ_FUA ? DP_FUA : 0) | - (bio->bi_rw & REQ_FLUSH ? DP_FLUSH : 0) | + (bio->bi_rw & REQ_PREFLUSH ? DP_FLUSH : 0) | (bio->bi_op == REQ_OP_DISCARD ? DP_DISCARD : 0); else return bio->bi_rw & REQ_SYNC ? DP_RW_SYNC : 0; diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c index 880de0d..752948f 100644 --- a/drivers/block/drbd/drbd_receiver.c +++ b/drivers/block/drbd/drbd_receiver.c @@ -2159,7 +2159,7 @@ static unsigned long wire_flags_to_bio_flags(u32 dpf) { return (dpf & DP_RW_SYNC ? REQ_SYNC : 0) | (dpf & DP_FUA ? REQ_FUA : 0) | - (dpf & DP_FLUSH ? REQ_FLUSH : 0); + (dpf & DP_FLUSH ? REQ_PREFLUSH : 0); } static unsigned long wire_flags_to_bio_op(u32 dpf) diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c index 2255dcf..eef6e95 100644 --- a/drivers/block/drbd/drbd_req.c +++ b/drivers/block/drbd/drbd_req.c @@ -1132,7 +1132,7 @@ static int drbd_process_write_request(struct drbd_request *req) * replicating, in which case there is no point. */ if (unlikely(req->i.size == 0)) { /* The only size==0 bios we expect are empty flushes. */ - D_ASSERT(device, req->master_bio->bi_rw & REQ_FLUSH); + D_ASSERT(device, req->master_bio->bi_rw & REQ_PREFLUSH); if (remote) _req_mod(req, QUEUE_AS_DRBD_BARRIER); return remote; diff --git a/drivers/md/bcache/journal.c b/drivers/md/bcache/journal.c index 68fa0f0..6e0a330 100644 --- a/drivers/md/bcache/journal.c +++ b/drivers/md/bcache/journal.c @@ -627,7 +627,7 @@ static void journal_write_unlocked(struct closure *cl) bio->bi_iter.bi_sector = PTR_OFFSET(k, i); bio->bi_bdev = ca->bdev; bio->bi_op = REQ_OP_WRITE; - bio->bi_rw = REQ_SYNC|REQ_META|REQ_FLUSH|REQ_FUA; + bio->bi_rw = REQ_SYNC|REQ_META|REQ_PREFLUSH|REQ_FUA; bio->bi_iter.bi_size = sectors << 9; bio->bi_end_io = journal_write_endio; diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index be8cd63f..3947114 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -205,10 +205,10 @@ static void bch_data_insert_start(struct closure *cl) return bch_data_invalidate(cl); /* - * Journal writes are marked REQ_FLUSH; if the original write was a + * Journal writes are marked REQ_PREFLUSH; if the original write was a * flush, it'll wait on the journal write. */ - bio->bi_rw &= ~(REQ_FLUSH|REQ_FUA); + bio->bi_rw &= ~(REQ_PREFLUSH|REQ_FUA); do { unsigned i; @@ -668,7 +668,7 @@ static inline struct search *search_alloc(struct bio *bio, s->iop.write_prio = 0; s->iop.error = 0; s->iop.flags = 0; - s->iop.flush_journal = (bio->bi_rw & (REQ_FLUSH|REQ_FUA)) != 0; + s->iop.flush_journal = (bio->bi_rw & (REQ_PREFLUSH|REQ_FUA)) != 0; s->iop.wq = bcache_wq; return s; @@ -920,7 +920,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) bch_writeback_add(dc); s->iop.bio = bio; - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { /* Also need to send a flush to the backing device */ struct bio *flush = bio_alloc_bioset(GFP_NOIO, 0, dc->disk.bio_split); diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c index fe222c0..3c9aebe 100644 --- a/drivers/md/dm-cache-target.c +++ b/drivers/md/dm-cache-target.c @@ -788,7 +788,7 @@ static void check_if_tick_bio_needed(struct cache *cache, struct bio *bio) spin_lock_irqsave(&cache->lock, flags); if (cache->need_tick_bio && - !(bio->bi_rw & (REQ_FUA | REQ_FLUSH)) && + !(bio->bi_rw & (REQ_FUA | REQ_PREFLUSH)) && bio->bi_op != REQ_OP_DISCARD) { pb->tick = true; cache->need_tick_bio = false; @@ -830,7 +830,7 @@ static dm_oblock_t get_bio_block(struct cache *cache, struct bio *bio) static int bio_triggers_commit(struct cache *cache, struct bio *bio) { - return bio->bi_rw & (REQ_FLUSH | REQ_FUA); + return bio->bi_rw & (REQ_PREFLUSH | REQ_FUA); } /* @@ -1063,7 +1063,8 @@ static void dec_io_migrations(struct cache *cache) static bool discard_or_flush(struct bio *bio) { - return bio->bi_op == REQ_OP_DISCARD || bio->bi_rw & (REQ_FLUSH | REQ_FUA); + return bio->bi_op == REQ_OP_DISCARD || + bio->bi_rw & (REQ_PREFLUSH | REQ_FUA); } static void __cell_defer(struct cache *cache, struct dm_bio_prison_cell *cell) @@ -1974,7 +1975,7 @@ static void process_deferred_bios(struct cache *cache) bio = bio_list_pop(&bios); - if (bio->bi_rw & REQ_FLUSH) + if (bio->bi_rw & REQ_PREFLUSH) process_flush_bio(cache, bio); else if (bio->bi_op == REQ_OP_DISCARD) process_discard_bio(cache, &structs, bio); diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index b7cbd39..7e39d3c 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1909,11 +1909,12 @@ static int crypt_map(struct dm_target *ti, struct bio *bio) struct crypt_config *cc = ti->private; /* - * If bio is REQ_FLUSH or REQ_DISCARD, just bypass crypt queues. - * - for REQ_FLUSH device-mapper core ensures that no IO is in-flight + * If bio is REQ_PREFLUSH or REQ_DISCARD, just bypass crypt queues. + * - for REQ_PREFLUSH device-mapper core ensures that no IO is in-flight * - for REQ_DISCARD caller must use flush if IO ordering matters */ - if (unlikely(bio->bi_rw & REQ_FLUSH || bio->bi_op == REQ_OP_DISCARD)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH || + bio->bi_op == REQ_OP_DISCARD)) { bio->bi_bdev = cc->dev->bdev; if (bio_sectors(bio)) bio->bi_iter.bi_sector = cc->start + diff --git a/drivers/md/dm-era-target.c b/drivers/md/dm-era-target.c index 665bf32..2faf49d8 100644 --- a/drivers/md/dm-era-target.c +++ b/drivers/md/dm-era-target.c @@ -1540,9 +1540,9 @@ static int era_map(struct dm_target *ti, struct bio *bio) remap_to_origin(era, bio); /* - * REQ_FLUSH bios carry no data, so we're not interested in them. + * REQ_PREFLUSH bios carry no data, so we're not interested in them. */ - if (!(bio->bi_rw & REQ_FLUSH) && + if (!(bio->bi_rw & REQ_PREFLUSH) && (bio_data_dir(bio) == WRITE) && !metadata_current_marked(era->md, block)) { defer_bio(era, bio); diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c index 790b185..84aa27c 100644 --- a/drivers/md/dm-io.c +++ b/drivers/md/dm-io.c @@ -381,7 +381,7 @@ static void dispatch_io(int op, int op_flags, unsigned int num_regions, */ for (i = 0; i < num_regions; i++) { *dp = old_pages; - if (where[i].count || (op_flags & REQ_FLUSH)) + if (where[i].count || (op_flags & REQ_PREFLUSH)) do_region(op, op_flags, i, where + i, dp, io); } diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c index 813bb4d..0848a43 100644 --- a/drivers/md/dm-log-writes.c +++ b/drivers/md/dm-log-writes.c @@ -555,7 +555,7 @@ static int log_writes_map(struct dm_target *ti, struct bio *bio) struct bio_vec bv; size_t alloc_size; int i = 0; - bool flush_bio = (bio->bi_rw & REQ_FLUSH); + bool flush_bio = (bio->bi_rw & REQ_PREFLUSH); bool fua_bio = (bio->bi_rw & REQ_FUA); bool discard_bio = (bio->bi_op == REQ_OP_DISCARD); diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c index 54cc0c7..9f34923 100644 --- a/drivers/md/dm-raid1.c +++ b/drivers/md/dm-raid1.c @@ -704,7 +704,7 @@ static void do_writes(struct mirror_set *ms, struct bio_list *writes) bio_list_init(&requeue); while ((bio = bio_list_pop(writes))) { - if ((bio->bi_rw & REQ_FLUSH) || + if ((bio->bi_rw & REQ_PREFLUSH) || (bio->bi_op == REQ_OP_DISCARD)) { bio_list_add(&sync, bio); continue; @@ -1253,7 +1253,8 @@ static int mirror_end_io(struct dm_target *ti, struct bio *bio, int error) * We need to dec pending if this was a write. */ if (rw == WRITE) { - if (!(bio->bi_rw & REQ_FLUSH) && bio->bi_op != REQ_OP_DISCARD) + if (!(bio->bi_rw & REQ_PREFLUSH) && + bio->bi_op != REQ_OP_DISCARD) dm_rh_dec(ms->rh, bio_record->write_region); return error; } diff --git a/drivers/md/dm-region-hash.c b/drivers/md/dm-region-hash.c index ce4af57..d257428 100644 --- a/drivers/md/dm-region-hash.c +++ b/drivers/md/dm-region-hash.c @@ -398,7 +398,7 @@ void dm_rh_mark_nosync(struct dm_region_hash *rh, struct bio *bio) region_t region = dm_rh_bio_to_region(rh, bio); int recovering = 0; - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { rh->flush_failure = 1; return; } @@ -526,7 +526,7 @@ void dm_rh_inc_pending(struct dm_region_hash *rh, struct bio_list *bios) struct bio *bio; for (bio = bios->head; bio; bio = bio->bi_next) { - if (bio->bi_rw & REQ_FLUSH || bio->bi_op == REQ_OP_DISCARD) + if (bio->bi_rw & REQ_PREFLUSH || bio->bi_op == REQ_OP_DISCARD) continue; rh_inc(rh, dm_rh_bio_to_region(rh, bio)); } diff --git a/drivers/md/dm-snap.c b/drivers/md/dm-snap.c index f68d0ae..6e82975 100644 --- a/drivers/md/dm-snap.c +++ b/drivers/md/dm-snap.c @@ -1681,7 +1681,7 @@ static int snapshot_map(struct dm_target *ti, struct bio *bio) init_tracked_chunk(bio); - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { bio->bi_bdev = s->cow->bdev; return DM_MAPIO_REMAPPED; } @@ -1800,7 +1800,7 @@ static int snapshot_merge_map(struct dm_target *ti, struct bio *bio) init_tracked_chunk(bio); - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { if (!dm_bio_get_target_bio_nr(bio)) bio->bi_bdev = s->origin->bdev; else @@ -2286,7 +2286,7 @@ static int origin_map(struct dm_target *ti, struct bio *bio) bio->bi_bdev = o->dev->bdev; - if (unlikely(bio->bi_rw & REQ_FLUSH)) + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) return DM_MAPIO_REMAPPED; if (bio_rw(bio) != WRITE) diff --git a/drivers/md/dm-stripe.c b/drivers/md/dm-stripe.c index 12b1630..e6f330c 100644 --- a/drivers/md/dm-stripe.c +++ b/drivers/md/dm-stripe.c @@ -286,7 +286,7 @@ static int stripe_map(struct dm_target *ti, struct bio *bio) uint32_t stripe; unsigned target_bio_nr; - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { target_bio_nr = dm_bio_get_target_bio_nr(bio); BUG_ON(target_bio_nr >= sc->stripes); bio->bi_bdev = sc->stripe[target_bio_nr].dev->bdev; diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index 1e170e7..5eac816 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -697,7 +697,7 @@ static void remap_to_origin(struct thin_c *tc, struct bio *bio) static int bio_triggers_commit(struct thin_c *tc, struct bio *bio) { - return (bio->bi_rw & (REQ_FLUSH | REQ_FUA)) && + return (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA)) && dm_thin_changed_this_transaction(tc->td); } @@ -868,7 +868,7 @@ static void __inc_remap_and_issue_cell(void *context, struct bio *bio; while ((bio = bio_list_pop(&cell->bios))) { - if (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + if (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA) || bio->bi_op == REQ_OP_DISCARD) bio_list_add(&info->defer_bios, bio); else { @@ -1647,7 +1647,7 @@ static void __remap_and_issue_shared_cell(void *context, while ((bio = bio_list_pop(&cell->bios))) { if ((bio_data_dir(bio) == WRITE) || - (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA) || bio->bi_op == REQ_OP_DISCARD)) bio_list_add(&info->defer_bios, bio); else { @@ -2560,7 +2560,7 @@ static int thin_bio_map(struct dm_target *ti, struct bio *bio) return DM_MAPIO_SUBMITTED; } - if (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + if (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA) || bio->bi_op == REQ_OP_DISCARD) { thin_defer_bio_with_throttle(tc, bio); return DM_MAPIO_SUBMITTED; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 90135b6..5336148c 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -964,12 +964,12 @@ static void dec_pending(struct dm_io *io, int error) if (io_error == DM_ENDIO_REQUEUE) return; - if ((bio->bi_rw & REQ_FLUSH) && bio->bi_iter.bi_size) { + if ((bio->bi_rw & REQ_PREFLUSH) && bio->bi_iter.bi_size) { /* * Preflush done for flush with data, reissue - * without REQ_FLUSH. + * without REQ_PREFLUSH. */ - bio->bi_rw &= ~REQ_FLUSH; + bio->bi_rw &= ~REQ_PREFLUSH; queue_io(md, bio); } else { /* done with normal IO or empty flush */ @@ -1428,7 +1428,7 @@ EXPORT_SYMBOL_GPL(dm_set_target_max_io_len); /* * A target may call dm_accept_partial_bio only from the map routine. It is - * allowed for all bio types except REQ_FLUSH. + * allowed for all bio types except REQ_PREFLUSH. * * dm_accept_partial_bio informs the dm that the target only wants to process * additional n_sectors sectors of the bio and the rest of the data should be @@ -1458,7 +1458,7 @@ void dm_accept_partial_bio(struct bio *bio, unsigned n_sectors) { struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone); unsigned bi_size = bio->bi_iter.bi_size >> SECTOR_SHIFT; - BUG_ON(bio->bi_rw & REQ_FLUSH); + BUG_ON(bio->bi_rw & REQ_PREFLUSH); BUG_ON(bi_size > *tio->len_ptr); BUG_ON(n_sectors > bi_size); *tio->len_ptr -= bi_size - n_sectors; @@ -1733,7 +1733,7 @@ static void __split_and_process_bio(struct mapped_device *md, start_io_acct(ci.io); - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { ci.bio = &ci.md->flush_bio; ci.sector_count = 0; error = __send_empty_flush(&ci); diff --git a/drivers/md/linear.c b/drivers/md/linear.c index aad82c7..657053e 100644 --- a/drivers/md/linear.c +++ b/drivers/md/linear.c @@ -221,7 +221,7 @@ static void linear_make_request(struct mddev *mddev, struct bio *bio) struct bio *split; sector_t start_sector, end_sector, data_offset; - if (unlikely(bio->bi_rw & REQ_FLUSH)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) { md_flush_request(mddev, bio); return; } diff --git a/drivers/md/md.c b/drivers/md/md.c index 7e16813..4f666d1 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -422,7 +422,7 @@ static void md_submit_flush_data(struct work_struct *ws) /* an empty barrier - all done */ bio_endio(bio); else { - bio->bi_rw &= ~REQ_FLUSH; + bio->bi_rw &= ~REQ_PREFLUSH; mddev->pers->make_request(mddev, bio); } diff --git a/drivers/md/multipath.c b/drivers/md/multipath.c index 7331a80..b651af1 100644 --- a/drivers/md/multipath.c +++ b/drivers/md/multipath.c @@ -111,7 +111,7 @@ static void multipath_make_request(struct mddev *mddev, struct bio * bio) struct multipath_bh * mp_bh; struct multipath_info *multipath; - if (unlikely(bio->bi_rw & REQ_FLUSH)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) { md_flush_request(mddev, bio); return; } diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c index e0d1b8c..f95463d 100644 --- a/drivers/md/raid0.c +++ b/drivers/md/raid0.c @@ -458,7 +458,7 @@ static void raid0_make_request(struct mddev *mddev, struct bio *bio) struct md_rdev *tmp_dev; struct bio *split; - if (unlikely(bio->bi_rw & REQ_FLUSH)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) { md_flush_request(mddev, bio); return; } diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 2a42daa..2856b50 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1056,7 +1056,8 @@ static void make_request(struct mddev *mddev, struct bio * bio) const int op = bio->bi_op; const int rw = bio_data_dir(bio); const unsigned long do_sync = (bio->bi_rw & REQ_SYNC); - const unsigned long do_flush_fua = (bio->bi_rw & (REQ_FLUSH | REQ_FUA)); + const unsigned long do_flush_fua = (bio->bi_rw & + (REQ_PREFLUSH | REQ_FUA)); const unsigned long do_sec = (bio->bi_rw & REQ_SECURE); struct md_rdev *blocked_rdev; struct blk_plug_cb *cb; diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 1cfc061..6171ff7 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -1452,7 +1452,7 @@ static void make_request(struct mddev *mddev, struct bio *bio) struct bio *split; - if (unlikely(bio->bi_rw & REQ_FLUSH)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) { md_flush_request(mddev, bio); return; } diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 311b7f2..c9f4ed5 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -536,7 +536,7 @@ int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio) bio_endio(bio); return 0; } - bio->bi_rw &= ~REQ_FLUSH; + bio->bi_rw &= ~REQ_PREFLUSH; return -EAGAIN; } diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index aafd49e..2109881 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -5154,7 +5154,7 @@ static void make_request(struct mddev *mddev, struct bio * bi) DEFINE_WAIT(w); bool do_prepare; - if (unlikely(bi->bi_rw & REQ_FLUSH)) { + if (unlikely(bi->bi_rw & REQ_PREFLUSH)) { int ret = r5l_handle_flush_request(conf->log, bi); if (ret == 0) diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index 1623d11..e63ad0f 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -2237,7 +2237,7 @@ static void btrfsic_bio_end_io(struct bio *bp) block->dev_bytenr, block->mirror_num); next_block = block->next_in_same_bio; block->iodone_w_error = iodone_w_error; - if (block->submit_bio_bh_rw & REQ_FLUSH) { + if (block->submit_bio_bh_rw & REQ_PREFLUSH) { dev_state->last_flush_gen++; if ((dev_state->state->print_mask & BTRFSIC_PRINT_MASK_END_IO_BIO_BH)) @@ -2273,7 +2273,7 @@ static void btrfsic_bh_end_io(struct buffer_head *bh, int uptodate) block->dev_bytenr, block->mirror_num); block->iodone_w_error = iodone_w_error; - if (block->submit_bio_bh_rw & REQ_FLUSH) { + if (block->submit_bio_bh_rw & REQ_PREFLUSH) { dev_state->last_flush_gen++; if ((dev_state->state->print_mask & BTRFSIC_PRINT_MASK_END_IO_BIO_BH)) @@ -2926,7 +2926,7 @@ int btrfsic_submit_bh(int op, int op_flags, struct buffer_head *bh) btrfsic_process_written_block(dev_state, dev_bytenr, &bh->b_data, 1, NULL, NULL, bh, op_flags); - } else if (NULL != dev_state && (op_flags & REQ_FLUSH)) { + } else if (NULL != dev_state && (op_flags & REQ_PREFLUSH)) { if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO @@ -3024,7 +3024,7 @@ static void __btrfsic_submit_bio(struct bio *bio) kunmap(bio->bi_io_vec[i].bv_page); } kfree(mapped_datav); - } else if (NULL != dev_state && (bio->bi_rw & REQ_FLUSH)) { + } else if (NULL != dev_state && (bio->bi_rw & REQ_PREFLUSH)) { if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c index 7104d76..2e4760b 100644 --- a/fs/jbd2/journal.c +++ b/fs/jbd2/journal.c @@ -1331,7 +1331,7 @@ static int jbd2_write_superblock(journal_t *journal, int write_flags) trace_jbd2_write_superblock(journal, write_flags); if (!(journal->j_flags & JBD2_BARRIER)) - write_flags &= ~(REQ_FUA | REQ_FLUSH); + write_flags &= ~(REQ_FUA | REQ_PREFLUSH); lock_buffer(bh); if (buffer_write_io_error(bh)) { /* diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 4cfba72..d355483 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1235,7 +1235,7 @@ _xfs_buf_ioapply( if (bp->b_flags & XBF_FUA) op_flags |= REQ_FUA; if (bp->b_flags & XBF_FLUSH) - op_flags |= REQ_FLUSH; + op_flags |= REQ_PREFLUSH; /* * Run the write verifier callback function if it exists. If diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index b4ef8d7..d637d8b 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -158,7 +158,7 @@ enum rq_flag_bits { __REQ_NOIDLE, /* don't anticipate more IO after this one */ __REQ_INTEGRITY, /* I/O includes block integrity payload */ __REQ_FUA, /* forced unit access */ - __REQ_FLUSH, /* request for cache flush */ + __REQ_PREFLUSH, /* request for cache flush */ /* bio only flags */ __REQ_RAHEAD, /* read ahead, can fail anytime */ @@ -202,12 +202,12 @@ enum rq_flag_bits { (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER) #define REQ_COMMON_MASK \ (REQ_FAILFAST_MASK | REQ_SYNC | REQ_META | REQ_PRIO | REQ_NOIDLE | \ - REQ_FLUSH | REQ_FUA | REQ_SECURE | REQ_INTEGRITY) + REQ_PREFLUSH | REQ_FUA | REQ_SECURE | REQ_INTEGRITY) #define REQ_CLONE_MASK REQ_COMMON_MASK /* This mask is used for both bio and request merge checking */ #define REQ_NOMERGE_FLAGS \ - (REQ_NOMERGE | REQ_STARTED | REQ_SOFTBARRIER | REQ_FLUSH | REQ_FUA | REQ_FLUSH_SEQ) + (REQ_NOMERGE | REQ_STARTED | REQ_SOFTBARRIER | REQ_PREFLUSH | REQ_FUA | REQ_FLUSH_SEQ) #define REQ_RAHEAD (1ULL << __REQ_RAHEAD) #define REQ_THROTTLED (1ULL << __REQ_THROTTLED) @@ -225,7 +225,7 @@ enum rq_flag_bits { #define REQ_PREEMPT (1ULL << __REQ_PREEMPT) #define REQ_ALLOCED (1ULL << __REQ_ALLOCED) #define REQ_COPY_USER (1ULL << __REQ_COPY_USER) -#define REQ_FLUSH (1ULL << __REQ_FLUSH) +#define REQ_PREFLUSH (1ULL << __REQ_PREFLUSH) #define REQ_FLUSH_SEQ (1ULL << __REQ_FLUSH_SEQ) #define REQ_IO_STAT (1ULL << __REQ_IO_STAT) #define REQ_MIXED_MERGE (1ULL << __REQ_MIXED_MERGE) diff --git a/include/linux/fs.h b/include/linux/fs.h index d57a5b5..7197027 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -203,9 +203,9 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); #define READ_SYNC REQ_SYNC #define WRITE_SYNC (REQ_SYNC | REQ_NOIDLE) #define WRITE_ODIRECT REQ_SYNC -#define WRITE_FLUSH (REQ_SYNC | REQ_NOIDLE | REQ_FLUSH) +#define WRITE_FLUSH (REQ_SYNC | REQ_NOIDLE | REQ_PREFLUSH) #define WRITE_FUA (REQ_SYNC | REQ_NOIDLE | REQ_FUA) -#define WRITE_FLUSH_FUA (REQ_SYNC | REQ_NOIDLE | REQ_FLUSH | REQ_FUA) +#define WRITE_FLUSH_FUA (REQ_SYNC | REQ_NOIDLE | REQ_PREFLUSH | REQ_FUA) /* * Attribute flags. These should be or-ed together to figure out what diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index ed3fc87..9df1cd8 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -33,7 +33,7 @@ TRACE_DEFINE_ENUM(SSR); TRACE_DEFINE_ENUM(__REQ_RAHEAD); TRACE_DEFINE_ENUM(__REQ_SYNC); TRACE_DEFINE_ENUM(__REQ_NOIDLE); -TRACE_DEFINE_ENUM(__REQ_FLUSH); +TRACE_DEFINE_ENUM(__REQ_PREFLUSH); TRACE_DEFINE_ENUM(__REQ_FUA); TRACE_DEFINE_ENUM(__REQ_PRIO); TRACE_DEFINE_ENUM(__REQ_META); diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index 3a54c83..c94dec7 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -189,6 +189,7 @@ static const u32 ddir_act[2] = { BLK_TC_ACT(BLK_TC_READ), BLK_TC_ACT(BLK_TC_WRITE) }; #define BLK_TC_RAHEAD BLK_TC_AHEAD +#define BLK_TC_PREFLUSH BLK_TC_FLUSH /* The ilog2() calls fall out because they're constant */ #define MASK_TC_BIT(rw, __name) ((rw & REQ_ ## __name) << \ @@ -219,7 +220,7 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes, what |= MASK_TC_BIT(op_flags, SYNC); what |= MASK_TC_BIT(op_flags, RAHEAD); what |= MASK_TC_BIT(op_flags, META); - what |= MASK_TC_BIT(op_flags, FLUSH); + what |= MASK_TC_BIT(op_flags, PREFLUSH); what |= MASK_TC_BIT(op_flags, FUA); if (op == REQ_OP_DISCARD) what |= BLK_TC_ACT(BLK_TC_DISCARD); @@ -1777,7 +1778,7 @@ void blk_fill_rwbs(char *rwbs, int op, u32 rw, int bytes) { int i = 0; - if (rw & REQ_FLUSH || + if (rw & REQ_PREFLUSH || op == REQ_OP_FLUSH) rwbs[i++] = 'F'; -- 1.8.3.1 From david@fromorbit.com Tue Jan 5 14:58:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id EEF9A29DFA for ; Tue, 5 Jan 2016 14:58:29 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id BA98C8F8040 for ; Tue, 5 Jan 2016 12:58:29 -0800 (PST) X-ASG-Debug-ID: 1452027504-04bdf03afc04fa0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id Uds1yvG6CQK6Hzn4 for ; Tue, 05 Jan 2016 12:58:25 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DjDQAvLoxWPGu7LXleKAECgw9SbYJihXegPQaKB4FShTqEBR6FawQCAoEeTQEBAQEBAQcBAQEBQT9BEgGDYQEBBCcTHCMQCAMYCSUPBSUDBxoTiC7CQQEBAQcCASAZhXWFR4JcgVsBAYNogRsFlwiFQogIgWVKjFdEhReBDQ+HSoUCKjSBeoIlgUIBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 07:28:13 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGYgG-0005g9-0J; Wed, 06 Jan 2016 07:58:12 +1100 Date: Wed, 6 Jan 2016 07:58:11 +1100 From: Dave Chinner To: Andrea Gelmini Cc: linux-kernel@vger.kernel.org, xfs@oss.sgi.com Subject: Re: BUG: KASAN: use-after-free in xfs_iflush_cluster+0x9d7/0xaf0 Message-ID: <20160105205811.GA21461@dastard> X-ASG-Orig-Subj: Re: BUG: KASAN: use-after-free in xfs_iflush_cluster+0x9d7/0xaf0 References: <20151214180048.GA15690@glen> <20151214195422.GM26718@dastard> <20151214201526.GA25152@glen> <20151214212220.GO26718@dastard> <20151215091145.GA19282@glen> <20160103204758.GW19802@dastard> <20160105163055.GA18111@glen> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105163055.GA18111@glen> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452027504 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.60 X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0249, MARKETING_SUBJECT X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25863 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.60 MARKETING_SUBJECT Subject contains popular marketing words 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 2.00 BSF_SC0_MV0249 Custom rule MV0249 On Tue, Jan 05, 2016 at 05:30:55PM +0100, Andrea Gelmini wrote: > On Mon, Jan 04, 2016 at 07:47:58AM +1100, Dave Chinner wrote: > > > I'm recompiling, to try it again. > > > Maybe, in the meanwhile, you can do something with my files. You can find 'em here: > > > http://mail.gelma.net/xfs_kasan > > > > Any update on this problem, Andrea? > > Here we are! > Reproduced right now. > > So, just to avoid confusion: > a) it's a vanilla kernel 4.4.0-rc8 > b) plus some btrfs patches > c) plus some dri/intel/i915 patches > d) at the same URL above you can find git_files.txt.gz, where you have each commit I > applied above vanilla kernel (anyway, nothing related to vfs/xfs of course) > e) at the same URL you find the kernel binaries I used > f) to catch it, I had to copy a few gigs of files on my /home partition (xfs over Luks) > > Anyway, here what you asked me for: > > (gdb) l *(xfs_iflush_cluster+0xb73/0xc10) > 0xffffffff8184c550 is in xfs_iflush_cluster (fs/xfs/xfs_inode.c:3182). > 3177 > 3178 STATIC int > 3179 xfs_iflush_cluster( > 3180 xfs_inode_t *ip, > 3181 xfs_buf_t *bp) > 3182 { > 3183 xfs_mount_t *mp = ip->i_mount; > 3184 struct xfs_perag *pag; > 3185 unsigned long first_index, mask; > 3186 unsigned long inodes_per_cluster; > (gdb) Ok, so that tells us nothing about where the problem lies - the function call is out of the preamble code that kasan instrumentation adds to the function. I'll have a further think about this... Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Tue Jan 5 15:11:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 3FE3B7F51 for ; Tue, 5 Jan 2016 15:11:57 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1761A8F8035 for ; Tue, 5 Jan 2016 13:11:54 -0800 (PST) X-ASG-Debug-ID: 1452028308-04bdf03afa053f0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 0v13j8Dket48JLZA for ; Tue, 05 Jan 2016 13:11:48 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CcDgCBMIxWPGu7LXleKAECgw+BP4Jig36BeaA9BotZhTqEBYYJAgIBAQKBHk0BAQEBAQEHAQEBAUE/QRIBg2EBAQQnExwjEAgDGAklDwUlAwcaE4guwk8BAQEBBgIBIBmFdYVHiCGBGwWHUwiHEoQHhBSNSoFliGuENkSFF4hmgnUcgXEqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 07:40:14 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGYrs-0005jt-R4; Wed, 06 Jan 2016 08:10:12 +1100 Date: Wed, 6 Jan 2016 08:10:12 +1100 From: Dave Chinner To: Dan Williams Cc: XFS Developers , linux-block@vger.kernel.org, "linux-nvdimm@lists.01.org" , Jens Axboe , linux-fsdevel , Jan Kara , Tejun Heo Subject: Re: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty Message-ID: <20160105211012.GB21461@dastard> X-ASG-Orig-Subj: Re: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182016.24118.33718.stgit@dwillia2-desk3.amr.corp.intel.com> <20160105042346.GL19802@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452028308 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25863 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 11:59:41AM -0800, Dan Williams wrote: > On Mon, Jan 4, 2016 at 8:23 PM, Dave Chinner wrote: > > On Mon, Jan 04, 2016 at 10:20:16AM -0800, Dan Williams wrote: > >> This warning was added as a debugging aid way back in commit > >> 500b067c5e6c "writeback: check for registered bdi in flusher add and > >> inode dirty" when we were switching over to per-bdi writeback. > >> > >> Once the block device has been torn down it's no longer useful to > >> complain about unregistered bdi's. Clear the writeback capability under > >> the the wb->list_lock(), so that __mark_inode_dirty has no opportunity > >> to race bdi_unregister() to this WARN() condition. > >> > >> Alternatively we could just delete the warning... > > > > The warning is correct - the filesytem is trying to mark an inode > > dirty on a device that can't do writeback anymore. Seems to me like > > it is functioning as it should. > > > >> Found this while testing block device remove from underneath an active > >> fs triggering traces like: > >> > >> WARNING: CPU: 6 PID: 2129 at fs/fs-writeback.c:2065 __mark_inode_dirty+0x261/0x350() > >> bdi-block not registered > >> [..] > >> Call Trace: > >> [] dump_stack+0x44/0x62 > >> [] warn_slowpath_common+0x82/0xc0 > >> [] warn_slowpath_fmt+0x5c/0x80 > >> [] __mark_inode_dirty+0x261/0x350 > >> [] generic_update_time+0x79/0xd0 > >> [] file_update_time+0xbd/0x110 > >> [] ext4_dax_fault+0x68/0x110 > >> [] __do_fault+0x4e/0xf0 > > > > This seems like the problem to me - you haven't implemented a > > shutdown hook for ext4, and so it continues to allow page faults to > > make progress after the device has been removed. The DAX fault > > should have been failed before the filesystem gets to the point of > > marking the inode dirty.... > > xfs doesn't check that the fs is still alive before calling > file_update_time(). Also, I don't think we need/want to sprinkle "is > fs alive?" checks to address this when the block device shutdown path > can simply turn off writeback. That's because the timestamp update is aborted in XFS during transaction commit because xfs_trans_commit has a "is the filesystem shutdown" check to prevent situations like this occurring. i.e. XFS has shutdown checks sprinkled all through it in strategic places to ensure that shutdown does what it is supposed to and does not trigger warnings in generic/VFS code. If you've removed a device, those inodes can *never* be written back, and hence marking new inodes dirty for writeback after a shutdown is completely wrong. e.g. if ext4 has had some other fatal corruption error, it will continue to allow timestamp updates and inode writeback through this mechanism. That's a bug in the filesystem error handling, not a bug in the writeback code. Cheers, Dave. -- Dave Chinner david@fromorbit.com From dan.j.williams@intel.com Tue Jan 5 15:29:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 88CF629DF9 for ; Tue, 5 Jan 2016 15:29:30 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 25CD7AC003 for ; Tue, 5 Jan 2016 13:29:30 -0800 (PST) X-ASG-Debug-ID: 1452029367-04bdf03afc05910001-NocioJ Received: from mail-yk0-f169.google.com (mail-yk0-f169.google.com [209.85.160.169]) by cuda.sgi.com with ESMTP id KTvInbZkXdv1t2Gg (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 05 Jan 2016 13:29:28 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.160.169 Received: by mail-yk0-f169.google.com with SMTP id a85so218537912ykb.1 for ; Tue, 05 Jan 2016 13:29:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=/geykrDlQdsGrCf8ei0y/zqiCwa5hHho5ka7ybKiYhw=; b=rmXSbMsA97gKhwEwEZ1WcaaRruZtOWp7K141oi5vQtrEp3spT91ihHZTTEhUKOBjjq oX74Bd7qBpipTTDbLnDdOW7A3jwWi8MPtqDYO4t1nwnDaPL7S8Ie6/5kf1MjsUpCgZ7F lkvEkVQgKW83xaVs8HTQLa7Gj2zLQjZ6/ktP3N/RbY/Ov7bmRGjuqFHVGsPbYRPM58DZ F4eALvmSwdiCniNoGgR3DlFuy0424UbnVFYxjBTdzPpniHyT1vJIYayT1LXaOUvrAmEF M9DDCfHSyo3OuxpY5j4YTI9Bc+B/hlgnyIoIbmZhHcC5oz6kZfJbbVzr5oPDftqQd1eC 8Siw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=/geykrDlQdsGrCf8ei0y/zqiCwa5hHho5ka7ybKiYhw=; b=HCRpZDYCB2RsjJXOTXvBrXdjsrjdrz+ZRLySX5wMMtGES1o7JfsA7JIbPRscsdbfGb mORBpKpMGQ4/RP7ncPvz1TNHBeB02H3X35g8Hm+uAuA+s2tJp2rTdA5qFoJQDVTnjw/K BL0xaynT7u+gpvUd+BEavKUyNr507xqZ6dI8fN9DIeJ29v8ndkKJuy0hN8XigTbR6MJf MRu5FmXw1x5dVVHkXbIOO8omMgHxV27wP9hCYXH1BP2EBmphNFte29w0lm37XPoujch8 8eNUf4vTPNXo+PcVLaqmEb5MOAmwqgUDUlrJGxnFnQvP/ZCgLJ2t+98gFLTj4UYlLyLF OSsg== X-Gm-Message-State: ALoCoQlSDZMIYrhiKVpl1y+khJjzU2tWFcI4QGa0Uok+q8KgJ5PS27njB+Xv8J3iVgBM6sDz/Ek6DCyp+KTdtRBoDiTimfae9Y45Sq5ZtmuDt6YdkABTW24= MIME-Version: 1.0 X-Received: by 10.129.48.197 with SMTP id w188mr69675842yww.82.1452029367639; Tue, 05 Jan 2016 13:29:27 -0800 (PST) Received: by 10.37.202.20 with HTTP; Tue, 5 Jan 2016 13:29:27 -0800 (PST) In-Reply-To: <20160105211012.GB21461@dastard> References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182016.24118.33718.stgit@dwillia2-desk3.amr.corp.intel.com> <20160105042346.GL19802@dastard> <20160105211012.GB21461@dastard> Date: Tue, 5 Jan 2016 13:29:27 -0800 Message-ID: Subject: Re: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty From: Dan Williams X-ASG-Orig-Subj: Re: [resend PATCH 3/3] writeback: fix false positive WARN in __mark_inode_dirty To: Dave Chinner Cc: XFS Developers , linux-block@vger.kernel.org, "linux-nvdimm@lists.01.org" , Jens Axboe , linux-fsdevel , Jan Kara , Tejun Heo Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-yk0-f169.google.com[209.85.160.169] X-Barracuda-Start-Time: 1452029368 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25864 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Tue, Jan 5, 2016 at 1:10 PM, Dave Chinner wrote: > On Tue, Jan 05, 2016 at 11:59:41AM -0800, Dan Williams wrote: >> On Mon, Jan 4, 2016 at 8:23 PM, Dave Chinner wrote: >> > On Mon, Jan 04, 2016 at 10:20:16AM -0800, Dan Williams wrote: >> >> This warning was added as a debugging aid way back in commit >> >> 500b067c5e6c "writeback: check for registered bdi in flusher add and >> >> inode dirty" when we were switching over to per-bdi writeback. >> >> >> >> Once the block device has been torn down it's no longer useful to >> >> complain about unregistered bdi's. Clear the writeback capability under >> >> the the wb->list_lock(), so that __mark_inode_dirty has no opportunity >> >> to race bdi_unregister() to this WARN() condition. >> >> >> >> Alternatively we could just delete the warning... >> > >> > The warning is correct - the filesytem is trying to mark an inode >> > dirty on a device that can't do writeback anymore. Seems to me like >> > it is functioning as it should. >> > >> >> Found this while testing block device remove from underneath an active >> >> fs triggering traces like: >> >> >> >> WARNING: CPU: 6 PID: 2129 at fs/fs-writeback.c:2065 __mark_inode_dirty+0x261/0x350() >> >> bdi-block not registered >> >> [..] >> >> Call Trace: >> >> [] dump_stack+0x44/0x62 >> >> [] warn_slowpath_common+0x82/0xc0 >> >> [] warn_slowpath_fmt+0x5c/0x80 >> >> [] __mark_inode_dirty+0x261/0x350 >> >> [] generic_update_time+0x79/0xd0 >> >> [] file_update_time+0xbd/0x110 >> >> [] ext4_dax_fault+0x68/0x110 >> >> [] __do_fault+0x4e/0xf0 >> > >> > This seems like the problem to me - you haven't implemented a >> > shutdown hook for ext4, and so it continues to allow page faults to >> > make progress after the device has been removed. The DAX fault >> > should have been failed before the filesystem gets to the point of >> > marking the inode dirty.... >> >> xfs doesn't check that the fs is still alive before calling >> file_update_time(). Also, I don't think we need/want to sprinkle "is >> fs alive?" checks to address this when the block device shutdown path >> can simply turn off writeback. > > That's because the timestamp update is aborted in XFS during > transaction commit because xfs_trans_commit has a "is the filesystem > shutdown" check to prevent situations like this occurring. i.e. XFS > has shutdown checks sprinkled all through it in strategic places to > ensure that shutdown does what it is supposed to and does not > trigger warnings in generic/VFS code. > > If you've removed a device, those inodes can *never* be written > back, and hence marking new inodes dirty for writeback after a > shutdown is completely wrong. e.g. if ext4 has had some other fatal > corruption error, it will continue to allow timestamp updates and > inode writeback through this mechanism. That's a bug in the > filesystem error handling, not a bug in the writeback code. True, in that sense the warning is serving a valid purpose to say "FIXME, implement shutdown checks". I'll drop this patch. From david@fromorbit.com Tue Jan 5 16:07:14 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 0A54029DF6 for ; Tue, 5 Jan 2016 16:07:14 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id E0893304071 for ; Tue, 5 Jan 2016 14:07:13 -0800 (PST) X-ASG-Debug-ID: 1452031627-04cbb07e1a06c70001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id OtLpCWIRr5SKkUfv for ; Tue, 05 Jan 2016 14:07:07 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CeDgBkPYxWPGu7LXleKAECgw+BP4Jig36BeaA9BotZhTqEBYYJAgIBAQKBHk0BAQEBAQEHAQEBAUE/QRIBg2ABAQEDAScTHCMFCwgDDgcDCSUPBSUDBxoTiCcHwj0BAQEBAQUBAQEBHxmFdYVHiCGBGwWHXo8qjUqBZY0hhVuIZoJ0HYFxKjSBeoNnAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 08:37:06 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGZkw-0005rE-1r; Wed, 06 Jan 2016 09:07:06 +1100 Date: Wed, 6 Jan 2016 09:07:06 +1100 From: Dave Chinner To: Brian Foster Cc: xfs@oss.sgi.com Subject: Re: [PATCH 6/9] libxfs: directory node splitting does not have an extra block Message-ID: <20160105220705.GC21461@dastard> X-ASG-Orig-Subj: Re: [PATCH 6/9] libxfs: directory node splitting does not have an extra block References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-7-git-send-email-david@fromorbit.com> <20160105183401.GB38749@bfoster.bfoster> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105183401.GB38749@bfoster.bfoster> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452031627 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25864 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 01:34:02PM -0500, Brian Foster wrote: > On Tue, Dec 22, 2015 at 08:37:06AM +1100, Dave Chinner wrote: > > From: Dave Chinner > > > > xfs_da3_split() has to handle all thre versions of the > > directory/attribute btree structure. The attr tree is v1, the dir > > tre is v2 or v3. The main difference between the v1 and v2/3 trees > > is the way tree nodes are split - in the v1 tree we can require a > > double split to occur because the object to be inserted may be > > larger than the space made by splitting a leaf. In this case we need > > to do a double split - one to split the full leaf, then another to > > allocate an empty leaf block in the correct location for the new > > entry. This does not happen with dir (v2/v3) formats as the objects > > being inserted are always guaranteed to fit into the new space in > > the split blocks. > > > > The problem is that when we are doing the first root split, there > > can be three leaf blocks that need to be updated for an attr tree, > > but there will ony ever be two blocks for the dir tree. The code, > > however, expects that there will always be an extra block (i.e. > > three blocks) that needs updating. When rebuilding broken > > directories, xfs_repair trips over this condition in the directroy > > code and SEGVs because state->extrablk.bp == NULL. i.e. there is no > > extra block. > > > > Indeed, for directories they *may* be an extra block on this buffer > > pointer. However, it's guaranteed not to be a leaf block (i.e. a > > directory data block) - the directory code only ever places hash > > index or free space blocks in this pointer (as a cursor of > > sorts), and so to use it as a directory data block will immediately > > corrupt the directory. This manifests itself in repair as a > > directory corruption in a repaired directory, leaving the directory > > rebuild incomplete. > > > > This is a dir v2 zero-day bug - it was in the initial dir v2 commit > > that was made back in February 1998. > > > > Fix this by making the update code aware of whether the extra block > > is a valid node for linking into the tree, rather than asserting > > (incorrectly) that the extra block must be valid. This brings the > > code in line with xfs_da3_node_split() which, from the first dir v2 > > commit, handled the conditional extra block case correctly.... > > > > Signed-off-by: Dave Chinner > > --- > > Thanks for the explanation. The logic in xfs_da3_split() is a bit > confusing and I'm not sure I'm following it quite correctly. A couple > questions below... > > > libxfs/xfs_da_btree.c | 39 ++++++++++++++++++++++++++------------- > > 1 file changed, 26 insertions(+), 13 deletions(-) > > > > diff --git a/libxfs/xfs_da_btree.c b/libxfs/xfs_da_btree.c > > index bf5fe21..58a0361 100644 > > --- a/libxfs/xfs_da_btree.c > > +++ b/libxfs/xfs_da_btree.c > > @@ -356,6 +356,7 @@ xfs_da3_split( > > int action = 0; > > int error; > > int i; > > + bool use_extra = false; > > > > trace_xfs_da_split(state->args); > > > > @@ -395,6 +396,7 @@ xfs_da3_split( > > * Entry wouldn't fit, split the leaf again. > > */ > > state->extravalid = 1; > > + use_extra = true; > > if (state->inleaf) { > > state->extraafter = 0; /* before newblk */ > > trace_xfs_attr_leaf_split_before(state->args); > > So here we walk up through a tree, doing splits as necessary. In this > particular case, we have the potential attr leaf double-split case > described above. If this happens, we set extrablk and use_extra. Is it > the case that if this occurs, we know the loop is good for at least one > more iteration (since this is a leaf block)? Yes, I think so, because it's only a "leaf" format attr tree if it's got a single block. If we need more than one leaf block, we have to add another level which adds a node. Hence if we are splitting a leaf, we already have to be in node format. > If so, we get to a node block that might be the root and call > xfs_da3_node_split(). That function potentially splits the node block > and adds the entries for the previous split, "consuming" extrablk if it > had been set as well. Right - it is the direct parent that consumes the extra block. So we split the node block, which allocates a new block that is stored in newblk, and that then gets stored in addblk, all the way back up to the root. > In fact, if the node/root doesn't split, we don't > carry on to any of the below code at all since addblk is set to NULL > (even if the double split had occurred). Yup, because we don't have any pointers that we need to fix up after the node split. It's only when we split the root node that we need to do the pointer fixup in xfs_da3_split(). > Given that, I'm not seeing how extrablk should be used by the following > root split code at all under normal circumstances. Wouldn't it always be > handled before that point? I think you're right. Which means I can remove all the extrablk handling from the code, rather than just from the directory block handling. Thanks for taking the time to understand the code, the change and asking a really good question, Brian. :) Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Tue Jan 5 16:32:42 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4AE867F47 for ; Tue, 5 Jan 2016 16:32:42 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3A34330406A for ; Tue, 5 Jan 2016 14:32:42 -0800 (PST) X-ASG-Debug-ID: 1452033158-04cb6c42e607710001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id y8ZEjn8K0zHX5vSz for ; Tue, 05 Jan 2016 14:32:39 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CcDgAEQ4xWPGu7LXleKAECgw+BP4Jig36BeaA9BotZhTqEBYYJAgIBAQKBHk0BAQEBAQEHAQEBAUE/QRIBg2EBAQQnExwPFBAIAxgJJQ8FJQMHGhMZiBXCRwEBCAIhGYV1hUeIIYEbBY05iU+NSoFlh3CFMY5BgnUcgXEqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 09:02:37 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGa9c-0005tK-Tq; Wed, 06 Jan 2016 09:32:36 +1100 Date: Wed, 6 Jan 2016 09:32:36 +1100 From: Dave Chinner To: Dan Williams Cc: XFS Developers , linux-block@vger.kernel.org, "linux-nvdimm@lists.01.org" , Jens Axboe , Jan Kara , linux-fsdevel , Matthew Wilcox , Ross Zwisler Subject: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life Message-ID: <20160105223236.GD21461@dastard> X-ASG-Orig-Subj: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> <20160105035147.GJ19802@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452033158 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25865 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 08:25:16PM -0800, Dan Williams wrote: > On Mon, Jan 4, 2016 at 7:51 PM, Dave Chinner wrote: > > On Mon, Jan 04, 2016 at 10:20:05AM -0800, Dan Williams wrote: > >> Historically we have waited for filesystem specific heuristics to > >> attempt to guess when a block device is gone. Sometimes this works, but > >> in other cases the system can hang waiting for the fs to trigger its > >> shutdown protocol. .... > True, and following this logic I think the existing > generic_shutdown_super() should be renamed generic_kill_super() to > match the fs actions, but see below... > > > Operation methods should be named after what they do, not what their > > calling context is. i.e. these are "invalidate" and "shutdown" > > superblock operations, not "quiesce" and "bdi_gone". > > I was running out of colors to paint the bike shed given > generic_shutdown_super() was already in use. Also, since Ah, yeah, i forgot about that function. To many different shades of purple are already in use. :/ > >> +void shutdown_partition(struct gendisk *disk, int partno) > >> +{ > >> + struct block_device *bdev = bdget_disk(disk, partno); > >> + struct super_block *sb = get_super(bdev); > >> + > >> + if (!bdev) > >> + return; > > > > Null pointer deref. Certainly a landmine waiting for someone to > > tread on. > > > > Nope, get_super() checks for a NULL argument. Hence my comment about it being a landmine. If get_super() is ever changed, this code will explode on us when we least expect it. If I have to go read other code in a completely different file just to determine the code is actually safe, then I consider that bad code. Code that is slightly more verbose but is obviously correct and balanced is much, much easier to understand and maintain.... > >> + if (!sb) { > >> + bdput(bdev); > >> + return; > >> + } > >> + > >> + if (sb->s_op->bdi_gone) > >> + sb->s_op->bdi_gone(sb); > >> + else > >> + generic_bdi_gone(sb); > > > > if (sb->s_op->shutdown) > > sb->s_op->shutdown(sb); > > else > > unmap_dax_inodes(sb); > > > > name things correctly, and the code documents itself. > > How about 'stop' or 'halt' instead of 'shutdown' to preserve the > historical meaning of generic_shutdown_super? It's hard chosing a different color that is appropriate. :/ Because it's a brute-force behavioural override, I think that needs to be obvious from the op name. So something like force_stop or force_failure seems best to me. In fact, now that I've thought/repaeated/written force_failure a couple of times, it seems quite appropriate here, as what we want to be able to do is force the filesystem into a (permanent) failure state..... Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Tue Jan 5 16:59:14 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E1CE129DF6 for ; Tue, 5 Jan 2016 16:59:14 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id A6334304067 for ; Tue, 5 Jan 2016 14:59:14 -0800 (PST) X-ASG-Debug-ID: 1452034751-04bdf03afa07450001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id d7yrtGNiNGWjxEfz for ; Tue, 05 Jan 2016 14:59:11 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AMDgBLSoxWPGu7LXleKAECgw+BP4Jig36BeaA9BotZiT+GCQQCAoEfTQEBAQEBAQcBAQEBQT9BEgGDYQEBBDocIxAIAw4KCSUPBSUDBxoTiC7CSAEBAQEBBQEBAQEfGYV1hUeIIYEbBZMKg36NSoFljSGFW4hmhQIqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 09:29:08 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGaZH-0005vs-6p; Wed, 06 Jan 2016 09:59:07 +1100 Date: Wed, 6 Jan 2016 09:59:07 +1100 From: Dave Chinner To: Jan Kara Cc: xfs@oss.sgi.com, linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, tytso@mit.edu Subject: Re: lazytime implementation questions Message-ID: <20160105225907.GE21461@dastard> X-ASG-Orig-Subj: Re: lazytime implementation questions References: <20160104062219.GB19802@dastard> <20160105173604.GE18604@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105173604.GE18604@quack.suse.cz> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452034751 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25867 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 06:36:04PM +0100, Jan Kara wrote: > Hi, > > On Mon 04-01-16 17:22:19, Dave Chinner wrote: > > I've been looking at implementing the lazytime mount option for XFS, > > and I'm struggling to work out what it is supposed to mean. > > > > AFAICT, on ext4, lazytime means that pure timestamp updates are not > > journalled and they are only ever written back when the inode is > > otherwise dirtied and written, or they are timestamp dirty for 24 > > hours which triggers writeback. > > > > This poses a couple of problems for XFS: > > > > 1. we log every timestamp change, so there is no mechanism > > for delayed/deferred update. > > > > 2. we track dirty metadata in the journal, not via the VFS > > dirty inode lists, so all the infrastructure written for > > ext4 to do periodic flushing is useless to us. > > > > These are solvable problems, but what I'm not sure about is exactly > > what the intended semantics of lazytime durability are. That is, > > exactly what guaranteed are we giving userspace about timestamp > > updates when lazytime is used? The guarantees we have to give will > > greatly influence the XFS implementation, so I really need to nail > > down what we are expected to provide userspace. Can we: > > > > a) just ignore all durability concerns? > > b) if not, do we only need to care about the 24 hour > > writeback and unmount? > > c) if not, are fsync/sync/syncfs/freeze/unmount supposed > > to provide durability of all metadata changes? > > d) do we have to care about ordering - if we fsync one inode > > with 1 hour old timestamps, do we also need to guarantee > > that all the inodes with older dirty timestamps also get > > made durable? > > So the intended semantics is: > 1) fsync / sync / freeze / unmount will write the timestamp updates even > with lazytime. So unless crash happens, timestamps are guaranteed to be > consistent. Also sync / fsync guarantees all changes to get to disk. > 2) We periodically write back timestamps (once per 24 hours) to avoid too > big timestamp inconsistencies in case of crash. Ok, so it's supposed to be a delayed timestamp update mechanism without any specific ordering guarantees, not an opportunistic timestamp update mechanism. I can work with that. Cheers, Dave. -- Dave Chinner david@fromorbit.com From fengguang.wu@intel.com Tue Jan 5 17:21:55 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.7 required=5.0 tests=FAKE_REPLY_C autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 08AA829DF5 for ; Tue, 5 Jan 2016 17:21:55 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id ECBA18F804C for ; Tue, 5 Jan 2016 15:21:51 -0800 (PST) X-ASG-Debug-ID: 1452036110-04cbb07e1a08540001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id ymOryRYWvB1AEGZv for ; Tue, 05 Jan 2016 15:21:50 -0800 (PST) X-Barracuda-Envelope-From: fengguang.wu@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP; 05 Jan 2016 15:21:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,526,1444719600"; d="gz'50?scan'50,208,50";a="884473585" Received: from bee.sh.intel.com (HELO bee) ([10.239.97.14]) by orsmga002.jf.intel.com with ESMTP; 05 Jan 2016 15:21:47 -0800 Received: from kbuild by bee with local (Exim 4.83) (envelope-from ) id 1aGav8-0009oF-53; Wed, 06 Jan 2016 07:21:42 +0800 Date: Wed, 6 Jan 2016 07:20:33 +0800 From: kbuild test robot To: mchristi@redhat.com Cc: kbuild-all@01.org, linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com, Mike Christie Subject: Re: [PATCH 05/35] fs: have ll_rw_block users pass in op and flags separately Message-ID: <201601060709.woTc6jMw%fengguang.wu@intel.com> X-ASG-Orig-Subj: Re: [PATCH 05/35] fs: have ll_rw_block users pass in op and flags separately MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="EVF5PPMfhYS0aIcm" Content-Disposition: inline In-Reply-To: <1452027218-32303-6-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: fengguang.wu@intel.com X-SA-Exim-Scanned: No (on bee); SAEximRunCond expanded to false X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452036110 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 --EVF5PPMfhYS0aIcm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi Mike, [auto build test WARNING on next-20160105] [cannot apply to dm/for-next v4.4-rc8 v4.4-rc7 v4.4-rc6 v4.4-rc8] [if your patch is applied to the wrong git tree, please drop us a note to help improving the system] url: https://github.com/0day-ci/linux/commits/mchristi-redhat-com/separate-operations-from-flags-in-the-bio-request-structs/20160106-052858 reproduce: make htmldocs All warnings (new ones prefixed by >>): >> fs/buffer.c:3091: warning: No description found for parameter 'op_flags' include/linux/jbd2.h:439: warning: No description found for parameter 'i_transaction' include/linux/jbd2.h:439: warning: No description found for parameter 'i_next_transaction' include/linux/jbd2.h:439: warning: No description found for parameter 'i_list' include/linux/jbd2.h:439: warning: No description found for parameter 'i_vfs_inode' include/linux/jbd2.h:439: warning: No description found for parameter 'i_flags' include/linux/jbd2.h:495: warning: No description found for parameter 'h_rsv_handle' include/linux/jbd2.h:495: warning: No description found for parameter 'h_reserved' include/linux/jbd2.h:495: warning: No description found for parameter 'h_type' include/linux/jbd2.h:495: warning: No description found for parameter 'h_line_no' include/linux/jbd2.h:495: warning: No description found for parameter 'h_start_jiffies' include/linux/jbd2.h:495: warning: No description found for parameter 'h_requested_credits' include/linux/jbd2.h:495: warning: No description found for parameter 'h_lockdep_map' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_chkpt_bhs[JBD2_NR_BATCH]' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_devname[BDEVNAME_SIZE+24]' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_average_commit_time' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_min_batch_time' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_max_batch_time' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_commit_callback' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_failed_commit' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_chksum_driver' include/linux/jbd2.h:1038: warning: No description found for parameter 'j_csum_seed' include/linux/jbd2.h:1038: warning: Excess struct/union/enum/typedef member 'j_history' description in 'journal_s' include/linux/jbd2.h:1038: warning: Excess struct/union/enum/typedef member 'j_history_max' description in 'journal_s' include/linux/jbd2.h:1038: warning: Excess struct/union/enum/typedef member 'j_history_cur' description in 'journal_s' fs/jbd2/transaction.c:429: warning: No description found for parameter 'rsv_blocks' fs/jbd2/transaction.c:429: warning: No description found for parameter 'gfp_mask' fs/jbd2/transaction.c:429: warning: No description found for parameter 'type' fs/jbd2/transaction.c:429: warning: No description found for parameter 'line_no' fs/jbd2/transaction.c:505: warning: No description found for parameter 'type' fs/jbd2/transaction.c:505: warning: No description found for parameter 'line_no' fs/jbd2/transaction.c:635: warning: No description found for parameter 'gfp_mask' vim +/op_flags +3091 fs/buffer.c ^1da177e Linus Torvalds 2005-04-16 3075 * ^1da177e Linus Torvalds 2005-04-16 3076 * This function drops any buffer that it cannot get a lock on (with the 9cb569d6 Christoph Hellwig 2010-08-11 3077 * BH_Lock state bit), any buffer that appears to be clean when doing a write 9cb569d6 Christoph Hellwig 2010-08-11 3078 * request, and any buffer that appears to be up-to-date when doing read 9cb569d6 Christoph Hellwig 2010-08-11 3079 * request. Further it marks as clean buffers that are processed for 9cb569d6 Christoph Hellwig 2010-08-11 3080 * writing (the buffer cache won't assume that they are actually clean 9cb569d6 Christoph Hellwig 2010-08-11 3081 * until the buffer gets unlocked). ^1da177e Linus Torvalds 2005-04-16 3082 * ^1da177e Linus Torvalds 2005-04-16 3083 * ll_rw_block sets b_end_io to simple completion handler that marks e227867f Masanari Iida 2014-02-18 3084 * the buffer up-to-date (if appropriate), unlocks the buffer and wakes ^1da177e Linus Torvalds 2005-04-16 3085 * any waiters. ^1da177e Linus Torvalds 2005-04-16 3086 * ^1da177e Linus Torvalds 2005-04-16 3087 * All of the buffers must be for the same device, and must also be a ^1da177e Linus Torvalds 2005-04-16 3088 * multiple of the current approved size for the device. ^1da177e Linus Torvalds 2005-04-16 3089 */ 3390b48e Mike Christie 2016-01-05 3090 void ll_rw_block(int op, int op_flags, int nr, struct buffer_head *bhs[]) ^1da177e Linus Torvalds 2005-04-16 @3091 { ^1da177e Linus Torvalds 2005-04-16 3092 int i; ^1da177e Linus Torvalds 2005-04-16 3093 ^1da177e Linus Torvalds 2005-04-16 3094 for (i = 0; i < nr; i++) { ^1da177e Linus Torvalds 2005-04-16 3095 struct buffer_head *bh = bhs[i]; ^1da177e Linus Torvalds 2005-04-16 3096 9cb569d6 Christoph Hellwig 2010-08-11 3097 if (!trylock_buffer(bh)) ^1da177e Linus Torvalds 2005-04-16 3098 continue; 3390b48e Mike Christie 2016-01-05 3099 if (op == WRITE) { :::::: The code at line 3091 was first introduced by commit :::::: 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 Linux-2.6.12-rc2 :::::: TO: Linus Torvalds :::::: CC: Linus Torvalds --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/kbuild-all Intel Corporation --EVF5PPMfhYS0aIcm Content-Type: application/octet-stream Content-Disposition: attachment; filename=".config.gz" Content-Transfer-Encoding: base64 H4sICMVMjFYAAy5jb25maWcAjDxbc9s2s+/9FZz0PLQzJ4ljO/7SOeMHiARFVATJEKAk+4Wj yHSiqS35k+Q2+fdnFyDF20JpZzK1sIvbYu9Y8NdffvXY63H3vDpu1qunpx/e12pb7VfH6sF7 3DxV/+cFqZek2uOB0O8AOd5sX7+/31x9uvGu312/u3i7X3/yZtV+Wz15/m77uPn6Cr03u+0v vwK2nyahmJY31xOhvc3B2+6O3qE6/lK3Lz/dlFeXtz86v9sfIlE6L3wt0qQMuJ8GPG+BaaGz Qpdhmkumb99UT49Xl29xVW8aDJb7EfQL7c/bN6v9+tv7759u3q/NKg9mD+VD9Wh/n/rFqT8L eFaqIsvSXLdTKs38mc6Zz8cwKYv2h5lZSpaVeRKUsHNVSpHcfjoHZ8vbDzc0gp/KjOmfjtND 6w2XcB6UaloGkpUxT6Y6atc65QnPhV8KxRA+BkQLLqaRHu6O3ZURm/My88sw8FtovlBclks/ mrIgKFk8TXOhIzke12exmORMczijmN0Nxo+YKv2sKHOALSkY8yNexiKBsxD3vMUwi1JcF1mZ 8dyMwXLe2ZchRgPicgK/QpErXfpRkcwceBmbchrNrkhMeJ4ww6lZqpSYxHyAogqVcTglB3jB El1GBcySSTirCNZMYRjisdhg6ngymsNwpSrTTAsJZAlAhoBGIpm6MAM+KaZmeywGxu9JIkhm GbP7u3Kqhvu1PFH6YcwA+ObtI6qOt4fV39XD22r93es3PHx/Q89eZHk64Z3RQ7EsOcvjO/hd St5hm2yqGZAN+HfOY3V72bSfBByYQYEieP+0+fL+effw+lQd3v9PkTDJkYk4U/z9u4Gki/xz uUjzzmlOChEHQDte8qWdT1kxN8psajTjEyqw1xdoaTrl6YwnJaxYyayrvoQueTKHPePipNC3 V6dl+znwgRFZAbzw5k2rKuu2UnNFaUw4JBbPea6A13r9uoCSFTolOhvhmAGr8ric3otsIDY1 ZAKQSxoU33dVRBeyvHf1SF2A6xbQX9NpT90FdbczRMBlnYMv78/3Ts+DrwlSAt+xIgaZTZVG Jrt989t2t61+75yIulNzkfnk2Pb8gcPT/K5kGixLROKFEUuCmJOwQnFQoa5jNpLGCrDasA5g jbjhYuB67/D65fDjcKyeWy4+GQIQCiOWhI0AkIrSRYfHoQVMsA+aRkegZoOeqlEZyxVHpLbN R/Oq0gL6gErTfhSkQ+XURQmYZnTnOdiPAM1HzFAr3/kxsWIjyvOWAEMbhOOBQkm0OgtEs1uy 4M9CaQJPpqjJcC0NifXmudofKCpH92hTRBoIv8voSYoQ4TppAyYhEehh0G/K7DRXXRzrf2XF e706/OUdYUneavvgHY6r48Fbrde71+1xs/3ark0Lf2YNpu+nRaLtWZ6mwrM29GzBo+lyv/DU eNeAe1cCrDsc/AQlC8SgtJwaIGumZgq7kETAocA5i2NUnjJNSCSdc24wjQfnHAeXBDLDy0ma ahLL2Ahws5JLWrTFzP7hEswC3FprWsCFCSybdffqT/O0yBStNiLuz7JUgCsAh67TnN6IHRmN gBmL3ix6XfQG4xmot7kxYHlAr8M/+Rgo/8YHIxFjnYKWSwqCGCwBQyUScOvVwEIUIvjQiQNQ fHUMJ+PzzLhY5gQHfTJfZbO8zGKmMSZooZbHugSWoLcFKM+cph14VhLYray1Bo10p0J1FmMG AHUn6WPMcjjBmYO7pnSX/v7ovuDklGHhWFFYaL4kITxLXfsU04TFIc0ERuU4YEZvOmCTLDxP 3AjsIglhgrbULJgL2Ho9KE1zPHBjsh2rgjknLM9Fny2a7WCcEPBgyHQwZHmyH0YD1pFwVu0f d/vn1XZdefzvagsql4Hy9VHpgmloVWN/iNNqar8cgbDwci6Ne04ufC5t/9Jo5YER6LmVGB3m NNupmFGehIqLSXdZKk4nLoHQEPehuS7BCRWh8E045GD/NBTxwH506ZpajI6MNy1lIoVlvO6y /ixkBn7AhNMMVYcZtAHF+Ux6AoJV4HbUm77PlXKtjYewN4H0huCi12PgxuC5oa0A41dO1IIN vW0B2htjd1icHoBmw7jItuZckwBQxXQH24qRSUjpTKDloMUs3KBGaTobADF9AL+1mBZpQThM EP0YF6Z2BYn4FeLNO3CW0TEzGtakdwaz5HyqwDYENt1Sk7Zk2XCpuBpotZIygEULYHTOrKUc wKRYwom1YGVmHFogUBbQros8AedLAzt3c09D2ScIaaDEwI1E5/X2gkIO+cJQq+XoUfLDHlyp WMjB98ww1TIcoWZLS18T3Q8w6n42aHTAgrRw5CkgqCmta98EosQOFPdR50BIH+sR8cB/MPtH 3uc++DE9B2gIJERxhAPHlPCzo+BxFDGjrf4YG4iXujUU4Qw7RCnBKIjX2Z3+Ucg0KGKQT9QU PEZ+GZ+2shAQiFSOE13jTOK5LGSbObSHkGZ3tayWOu70BJc0Ad0F5FiwPOgAUnB8wSWoc1lX IwAzydpTusRP52+/rA7Vg/eXtYov+93j5qkXdJy2idhlo+V70ZpZbKNkrBKKOJK0k7dBz0eh kbz90DHplr7EGTaUN0FBDKqu6OUdJuiTE91MNg0mykClFwki9YPbGm4oauHnYGTfRY7Bh6Nz F9jv3c+rMZ2iks3lYoCBnPa54AUqB9iECafdKPmiQWidSCDYfd9FMmed7Xfr6nDY7b3jjxcb aD5Wq+Prvjp07wHukbECR7IG7AfZjqnIkDNQxqD5mHQYcoOFqYAGFRNoblS+1MDCmOI951HX WVCRC3okG2gBsWHaHFONxqQ4IovoDrQ/OKqgXKYFnd2DQB/jTpv5bPn4+tMN7bN+PAPQivYX ESblkpKKG3P90mKClEOkJIWgBzqBz8Np0jbQaxo6c2xs9h9H+ye63c8LldKxqTSuHHc4qXIh Ej8CU+dYSA2+ckUTMXOMO+UQ8k6XH85Ay5gO1KR/l4ulk95zwfyrks6UGqCDdj54oo5eqEmc klHrZMe9nhEEDO/ryxoViVDffuyixB8GsN7wGVgDkOakn63pIKCqMkgmLaKKTtSPYBCAfkPt 2dxcD5vTeb9FikTIQppkWAj+anzXX7fxOX0dS9VzXGAp6Kyi88Bj8CIovwVGBDVtiNMxcU2z Od/ejWgDYTIg0EGEWJGPAcbvkByCMWqsQvq2vVVNGdc2rCIPO5CCUlbmbkyBxT3tn3OZ6ZEr 1rTP0xhcJZbTaacay8ltSIRM0DrNHJojq2cYjYNvcgehskNfOgE6Bdac0PZKfKJjaZww56jH Q7F0ZfLMihVNbsOUWSFo1ZKkmPQdpEiac7SQ617itm68uaa82blUWQzm66rXpW3F4NJBMoty SeerWvBPR/hArcvcuKZhqLi+vfjuX9j/BvscuC4hmHJoLXnCiAtYE7G4wUZimxsZ8A+74ili ZKC4se5491Dw29NqzvZtFiVZUphYq3UeTiuyMIIKdef+aKVRqrZfJ3hsh4NIRouO7rNxL5eT vlPZa64H7Q5oCyiE8iEI6Hbv505qfwU0WpiaQag0kjnnTJuJjM64HmSmfHeyKLoDhzYI8lI7 y0gatxLJM23PZS5y0GrgUhU9H3amKNFpLvRMxGTve4L89vrij5vuHcI4nKMUY7d0YNZz5fyY s8TYPDoMdbjG91ma0rmt+0lBq4l7Nc4Z1qAmljI37U0eyl0hEPI872cTzNXAUMVk2q1/jYGG GDTFO+88L7LhcfdUpwI3GcOyxe1Nh0+kzml1adZrI2TnAoAY7uDCGGNwSGmnq05k0C79ffnh 4oJSxPfl5ceLHonuy6s+6mAUephbGGYYb0Q5XtXR1w58yV03zkxFJt9EaVsQMuGDhgPVkaPC /VDr2+51Ueozc3F1rr9JPUH/y0H3Ov08DxSdwfdlYCLciYvPQauK8K6MA03dHXQ5war3RhtH qc5ikyC0cerun2rvPa+2q6/Vc7U9mkiV+Znwdi9YtNaLVus8B62WaF5TYc9Tau5gvXBf/fe1 2q5/eIf1qs6AtJtHNzPnn8me4uGpGiI7L4oNAVD9qBMeXgtkMQ9Gg09eD82mvd8yX3jVcf3u 9+5U2EgkQWylWJ2Sbb0h5YjqfWQGEpTGjuoI4CJaFhOuP368oEOnzEdD5dYAdyqcjIjAv1fr 1+Pqy1Nlqh09c2lzPHjvPf78+rQascQEzJzUmJOjr7YsWPm5yChDZZN2adFTnnUnbD43qBSO gB7DN4dc2/lsNkikVst3iTmiR1D9vVlXXrDf/G2vqdrCp826bvbSsagU9goq4nHmiiH4XMss dORRNKhvhmlHV2hghg9FLhdgfu0dPIkaLsBwsMCxCLSIC3O5TRFtcPsW5GLu3IxB4PPckY0C buvke0iUU/0ICCqMJHwyU9nFwgv9pjSnE5sxWy8YAFXCkMjNoaA/mHPtHZnUNAXTkFiGTSab or+m7BP8oLoGtj0n2zRagdwc1tQS4ADkHSYyyYVA5B+nClN56BAM6dOSOme0LvYvycVwDjSU 3uH15WW3P3aXYyHlH1f+8mbUTVffVwdPbA/H/euzudA9fFvtqwfvuF9tDziUB3q98h5gr5sX /LORHvZ0rPYrL8ymDJTM/vkf6OY97P7ZPu1WD56tVWxwxfZYPXkgrubUrLw1MOWLkGhuu0S7 w9EJ9Ff7B2pAJ/7u5ZTTVcfVsfJkazV/81Mlf++oiZaGfuSw8MvYpOmdwLrcDsyKE4XzyKXk RHCqvlK+EjW3dU75ZI6UQGeiF4hhmysrLZkP/mGKvpPRB+MaK7F9eT2OJ2wtY5IVYzaM4DwM J4j3qYdd+q4HFon9Ozk0qN3tTJnkJOf7wLCrNTAjJYta02kZUE2ucgwAzVwwkUlR2uJFRzZ8 cc5nT+Yuqc78T/+5uvleTjNHMUiifDcQVjS1wYg726V9+Ofw7yBQ8IeXQ5YJLn3y7B1FYsrB 5SqTNCBSY8cyyxQ1Z5aNeRTb6ocdO1OZ2PSyUJ1566fd+q8hgG+NawTuPVaaoq8MTgOWTKPH b0gIlltmWMpx3MFslXf8Vnmrh4cNegirJzvq4d3gvs/cIqcmCISYAQ8Lhu+xsG0iKbFwuH/p Am/VIWyNHflFg4DRJe1mWTibO+pEFs7CwojnktFRS1PhSuVE1KT7GMBqrt12sz54avO0We+2 3mS1/uvlabXt+f/Qjxht4oMbMBxusgcDs949e4eXar15BAeOyQnrubODhIO11q9Px83j63aN Z9jotYexqpdhYNwoWm0iMId43xGORho9CAgar5zdZ1xmDi8PwVLfXP3huNEAsJKuQIFNlh8v Ls4vHWNM18UQgLUomby6+rjESwYWOC7aEFE6FJEtRtAO31DyQLAmBzM6oOl+9fINGYUQ/qB/ k2lA4X71XHlfXh8fQfUHY9Uf0oKGBQCxMTWxH1CLaTO5U4Y5R0cxalr0Y+gmZAABSCNflLHQ GuJUiLQF65SSIHz0zgobTyUDkd8z44Uax3fYZnyzh35Eg+3Ztx8HfPPmxasfaBPHHI6zgaJz pOEzA1/6XMxJDIROWTB16JtiQZNdSgc7camceZ+EQ9wDYT/N8KaGSkwEUPqOOAkeML+JEiF0 LTrvigyoPYXWzYN2YqQcpHqgyrHJj5milwZeFxH7tCsvloFQmatSuXAIl0n8uty1+WYPio06 buwmUjiA/rB1CLPe7w67x6MX/Xip9m/n3tfXCtxtQgRBFKaD4sZeJqKpOKCivtbdjSAU4Sfc 8TZO/qN62WyN7R6wuG8a1e5131PfzfjxTOV+KT5dfuzU8UArhOlE6yQOTq3t6WgJDnsmaP4G j9n4WKUvf4IgdUFfP58wtKQr/7msEUAyHN67iCcpnUwSqZSFU8nm1fPuWGEMRLGK0txc9Mgy x1vfce+X58PX4YkoQPxNmbcRXroFd3zz8ntrm4lgShXJUrgDXBivdOw7M9w1TCq2dFtqp3kz eVOaYA5xyxbUhQoDDp+CRpFsWSZ5ty5LZFj6OClozjcemik0zdPYFT2Eckxz1NTdxyejTItL laMvmy1Zefkpkeho0/q3hwW6nWZZ8KjKGbi1BuPsjJG4uby8GFqtvjvqO24tpD82dd2S82dw JMHRp7RTzsa6hG0f9rvNQxcNQrM8dd1AOyNCpZ3tNtnjhNavuqBFpY7ktr2m0dFo+Saz0nt7 Dnww2rjBGnVt8jFUKiNwpBibLCRQwXWtFPA4LvMJrbUCP5gwmvmnaTqN+WkKYr0QjlkO7yjz wFbRQGDWqUVv16swMhBLADlehmDJJUa1LqsVKlME7UgQnIEJCyudr21Cdqb35yLVdFLGQHxN bwfTpKG6Lh255hDLhhywFDwGcDYGYMsUq/W3gdusRhe5VhAP1evDztwntCfVyjWYC9f0BuZH Ig5yTmtnTJK5cuj4JomOtexr8fPQcniZ3boi5n/ARY4B8GLC8JB9BEIjJfGYpPVbmW8Q5vYf IppvLIj8s3le3nE/Ta+X/WZ7/MskIx6eK7Cy7c3dyYQphbfUMcrSHHRGfbd/e10f5e75BQ7n rXkTCae6/utghlvb9j11F2gz/ljkQBtUe+kIMovfqshy7kM45HgaVd9PFuZjApwsNLbFpDja 7YeLy+uuqsxFVjIFCtP1uAwrjM0MTNHKuEhAAjDElZPU8VjKVt8skrPXHyF1XxFxvHxRdmfj F02K2+95AM9IzI3QnDxAsmRNk5gKXtqEUq8Cd1C1/LPa3HpHqXmWzNmsKd9wOJXo1wC39y8u ekPZbHbDsxKcyf0PiL2/vH79Orj9NbQ25cjKVQMz+ErDGZx08icQz/l4qV4bGK4YNjk+ngZy Zgb7bqVQLm1hseaujLEBQpxVODJmFqO+vMcykzNYZ+rg2s2a9aJeD2Pzcp3aTgN2jWR4DGkz 4upT4zmKRYMbrvqmFXjBiyFGe32x6idabb/2dA6a5CKDUcbPYDpTIBCUeGJfSdNpyM9kJrLD ggkwNEhcmmYU7/Tgw+o3C8QwDO+1R8UqTpVpwZad8MsoPyMjzjDjPKPenSMZW+nyfjvUMfHh f73n12P1vYI/sLzhXb/AoT6f+iXEOX7Et7OOSN1iLBYWCZ9ILjKmac1mcU0ZnFuSwQuYn/fH zACYcTszSZPPiYFkP1kLTGOe0ikeh+5XE2ZSYMPT4wqHL998JOnMpDOrps4tSzjGr1Wh+BmG oilngc2TvnMH6uc8wBcKjHBc8EMEtC43R+f6TkH9PQz8zMA5W/RTGpuvGPwrpPOfOvhcf/+H dthqGpU8z9McxPhP7i7StKWTJE7XTGPStlG7EJBr+xLSvEOzJfyUfiYRiRnaV5WOb3QZVR4W id9+RGD4LvEEneYsi/4VTpiZMxi+Tq3fuZLvbvvAciF0RL0VrcHSPDAEBB/CuwFKXQpnF2qf sw5fWtYd7SgtEHug3BO53XDENpbp8Xsh4DDr6nAcsD0SwAik+VwSnRdpzwUfNLrZdmLe5Dnh Vq3dXJ+UFS1CuKCIL50VPgYBeSuZ1kVLtC4weDNA1I4kokEw33OgK8IMPAfGj1y1k/Z7IkHq q7z3TZjeA2f32EXg/JAH+CZuPc1kRr+N7Hg806CXysff50V7birB1f/3cTW9DcIw9C+162VX SIPmDVEEoSq9oG3qoadJ1XrYv5/t0CRQO1degEA+/BG/5+us7LLIwi/k53KC9VOGsi8a7B/6 fiQv4qmgMXghVPwiSq2jH1ceel+Uruie+KLnjAIHp+gdzRz9iDC2yViQ7kByCHID5lbzrpnz eDBmr4dediXm7DWuFl2rgE4ylM0QDl53b3Jja6fN6XUTPbo1hgO5lTE/+aIa2xJlHtDuCeOX pZWjEVBC5tAiM9lDm2ZVRRh+6Wyk0i6m7qppi8xaC3I3D0W9zLih26Bk0AN5bOZocFnfm5IL jI0rJRBtB9KYoy3xuef+xOHyfb9df/+kjMaHHZVEkjVDB27EHcj2nI/npZttK+YCHr88PrBI WClrdKmC141tRsLuuCBVzPEknHXJkRKaohuFjdqHD9ev2yfG57efO5q2S5JKCtoWrmsMeh8V FQiSpyHIX2CT2jYKWkHzEJosQVARaw2ECt0VpF4WxAKYAc0CR20NS5EU0+F0M+DkgUR0K1Pg 6D633exBNnAEg0NfU0N38kEJInLxRg0l36XRGIzM9EUAvQKrMWhYCW/Wl/NcBIG9Gr0Qrkzb veS9jNOZdGkz0FSad3EO9zSoKa/LX6ItesnBYrOaai6GkQ6OEL0HKk7xOzguNTDQ8VO+cL/X 5K88WytnoXs6ni6gEXpFFmdio4XgP76rYx1cWAAA --EVF5PPMfhYS0aIcm-- From fengguang.wu@intel.com Tue Jan 5 17:47:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.7 required=5.0 tests=FAKE_REPLY_C autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 02BC529DF5 for ; Tue, 5 Jan 2016 17:47:57 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id D8388304067 for ; Tue, 5 Jan 2016 15:47:53 -0800 (PST) X-ASG-Debug-ID: 1452037671-04cbb07e1708de0001-NocioJ Received: from mga11.intel.com ([192.55.52.93]) by cuda.sgi.com with ESMTP id 4f9HX7IHvCPAz9Bq for ; Tue, 05 Jan 2016 15:47:51 -0800 (PST) X-Barracuda-Envelope-From: fengguang.wu@intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.93 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga102.fm.intel.com with ESMTP; 05 Jan 2016 15:47:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,526,1444719600"; d="gz'50?scan'50,208,50";a="720915632" Received: from bee.sh.intel.com (HELO bee) ([10.239.97.14]) by orsmga003.jf.intel.com with ESMTP; 05 Jan 2016 15:47:47 -0800 Received: from kbuild by bee with local (Exim 4.83) (envelope-from ) id 1aGbKJ-000IVo-Hi; Wed, 06 Jan 2016 07:47:43 +0800 Date: Wed, 6 Jan 2016 07:46:39 +0800 From: kbuild test robot To: mchristi@redhat.com Cc: kbuild-all@01.org, linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com, Mike Christie Subject: Re: [PATCH 34/35] block: add QUEUE_FLAGs for flush and fua Message-ID: <201601060735.1UARVDdr%fengguang.wu@intel.com> X-ASG-Orig-Subj: Re: [PATCH 34/35] block: add QUEUE_FLAGs for flush and fua MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="n8g4imXOkfNTN/H1" Content-Disposition: inline In-Reply-To: <1452027218-32303-35-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: fengguang.wu@intel.com X-SA-Exim-Scanned: No (on bee); SAEximRunCond expanded to false X-Barracuda-Connect: UNKNOWN[192.55.52.93] X-Barracuda-Start-Time: 1452037671 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 --n8g4imXOkfNTN/H1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi Mike, [auto build test ERROR on next-20160105] [cannot apply to dm/for-next v4.4-rc8 v4.4-rc7 v4.4-rc6 v4.4-rc8] [if your patch is applied to the wrong git tree, please drop us a note to help improving the system] url: https://github.com/0day-ci/linux/commits/mchristi-redhat-com/separate-operations-from-flags-in-the-bio-request-structs/20160106-052858 config: um-x86_64_defconfig (attached as .config) reproduce: # save the attached .config to linux build tree make ARCH=um SUBARCH=x86_64 All errors (new ones prefixed by >>): arch/um/drivers/ubd_kern.c: In function 'ubd_add': >> arch/um/drivers/ubd_kern.c:869:43: error: macro "blk_queue_flush" passed 2 arguments, but takes just 1 blk_queue_flush(ubd_dev->queue, REQ_FLUSH); ^ >> arch/um/drivers/ubd_kern.c:869:2: error: 'blk_queue_flush' undeclared (first use in this function) blk_queue_flush(ubd_dev->queue, REQ_FLUSH); ^ arch/um/drivers/ubd_kern.c:869:2: note: each undeclared identifier is reported only once for each function it appears in vim +/blk_queue_flush +869 arch/um/drivers/ubd_kern.c 62f96cb0 Jeff Dike 2007-02-10 863 ubd_dev->queue = blk_init_queue(do_ubd_request, &ubd_dev->lock); 62f96cb0 Jeff Dike 2007-02-10 864 if (ubd_dev->queue == NULL) { 62f96cb0 Jeff Dike 2007-02-10 865 *error_out = "Failed to initialize device queue"; 80c13749 Jeff Dike 2006-09-29 866 goto out; 62f96cb0 Jeff Dike 2007-02-10 867 } 62f96cb0 Jeff Dike 2007-02-10 868 ubd_dev->queue->queuedata = ubd_dev; 805f11a0 Richard Weinberger 2013-08-18 @869 blk_queue_flush(ubd_dev->queue, REQ_FLUSH); 62f96cb0 Jeff Dike 2007-02-10 870 8a78362c Martin K. Petersen 2010-02-26 871 blk_queue_max_segments(ubd_dev->queue, MAX_SG); 792dd4fc Christoph Hellwig 2009-03-31 872 err = ubd_disk_register(UBD_MAJOR, ubd_dev->size, n, &ubd_gendisk[n]); :::::: The code at line 869 was first introduced by commit :::::: 805f11a0d515658106bfbfadceff0eb30bd90ad2 um: ubd: Add REQ_FLUSH suppport :::::: TO: Richard Weinberger :::::: CC: Richard Weinberger --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/kbuild-all Intel Corporation --n8g4imXOkfNTN/H1 Content-Type: application/octet-stream Content-Disposition: attachment; filename=".config.gz" Content-Transfer-Encoding: base64 H4sICFFVjFYAAy5jb25maWcAjDxbc9u20u/9FZz0pZ05aXyLa39n/ACRoIgjkqAJUJL9wlFk JtHUljyS3Db//tsFSQkkF4pnOo2JXVz3vljo119+9djbfvOy2K+Wi+fnH963al1tF/vqyfu6 eq7+6wXSS6X2eCD0H4Bc7Kqtl2yeKi9erd/+/fTvzXV5feVd/XH1x9nH7fLGm1TbdfXs+Zv1 19W3NxhptVn/8usvvkxDMS6LJL770X4kSXH8SGUpZMITaPnVa9p0znxeivw+jNlYlarIMplr b7Xz1pu9t6v2bedY+pOAZy3GcVSlmT+phxnAxjzlufBLn8VilDPNy4DH7GGIMCrGx8bo8e78 7OwX2BIcRhJ/3L1Wy9XX1dLbvOJWdwAwsGiz23uv282y2u02W2//47XyFms41Wqxf9tWO4PU HsTkxt7UsT1TPg3wZc4vaBDTMiGO6LDbzDr0OVBPpJrnqQw4HIQfwTlFItR31zZKfO6GaeV3 x/OTbO5H4+urfrOcdlsSkYqkSHBFZcgSET/cXV+1CNgIFDOrs1imbWZJMGz0eapZkR8BQB+c 6dhwfTUS2uYwXMflBXFcc8PXx54s9yPgj7D+vPuw2C6/f3p7+bQ0fL5r5KB8qr7WLR/ajvlM 8aTEE2FBULJ4LHOhow6b1ygtgVQmUuRnYlGXZcynPC6zsWajmCt7ELPAiIGUwBBinLJYkfxh 8HJeKF5GUulyqh4USEAMAM5O8E0042IcdQ4PhEsDJBYptViQWV3Ls9VQGkaDZuDTzB4rYlNe jqTELkD0UBpMYliVxUKXmcYjQsqru6vjKL5MMuZrIVOiZxY9qBKIkJf6wAgN6FGmIP0JwzWO 786OA04UJUvAB6yINSgwliEbm1Hvrs5uD5KRch6UGc8Nb0461PZjzlIjTiR9wlymWs1YRkIf MyljGjIqAhqgEiCvpBWJCGJeZmzMja6diHRMadcAmCoXmS6Dh/R4aCMgU6JLHoeWKIJeKoMi yQ5nBOAy4izguRp0rUcdsAGTBaXkm06JAI3zYk+I81kKGrg6tOYCXZkqGXOrhY1Rzh5Ufn9s nIASBNEyBqOUOaz37tzSqkBpICexKrBbwDjHcRrZrCVV3V227WDFypnMJ4BpVP/Y2NlnHOrt 9WggR7mc8LSUaansTYGqhKNMpyClQA+RAPOeX9wcDiGXShnmF7DPDx+6AgFtpebKYTdZPAXS gMRgP6K5ZIWWx4W0ZMVTTlkCs/223qyr3w99kXUt+/ugpiLzBw34r68txZ5JJeZlcl/wgtOt gy71rkFfyPyhZBoIF9msFEYsBe4m2R60H9h8Wj8W4OkQR1VzJipPg9FozZaaQF1v9/Zl92O3 r16O1GyVJxJfRXJGOB+ox4BVQObbsfTqpdruqOFA2wJvphyGspQXcGD0iJROZNqRpUfUQEIG wqf5FhBQ/nsjWZIECh9MhYJ5k1p8a6clKz7pxe4vbw8LNT7Nbr/Y77zFcrl5W+9X62+9FaMO ZL4vi1SDgrG0gAIdmUufAxkB3lEEfVg5vSTJpZmaoB3qGDuzzNwvPEWdYfpQAqxjgP2i5HM4 LIrwqodsZsQu5HpwKFgP2NOaIPSic84NpvFMnePgkoDVartIKcRCgGoGtXhhCZiYNC7KoMWc 57E5ljhC2Ph050fPa5zLIuv4FsYJK/NA0x4nT3zKbIziSTOYPVbAwZ22YETHGgDWIeKWlxcy kZddyFHNhaocgbjPRKAj+jy13ZdEaabNRKDciwqBdI/c8jHRBeHasjd4yjhIA7Ew6xECPhU+ 76y+BgA+cvqptZnDoxEi7k8yCZRCidVgFSmZB4WtwEHi1nILrcrU+kblnHaoD7vIoYmcFveZ UseVct0bxhy8sSYuuoNOBTpCHJdzH8KxDoX7sHJKhz85hnAkBFkOzt6Y0jygKOyXMgNVJx55 GcocvOhHSzf2rQ9LwTYKdFKtwzNGohDBuRVzjLLQ3ohTz/S6JWBlBR595xTGXCegf8rG/DgP 8Wie7BOEDZzoOYFm9ZB0iNa2lXSXLAeGm1h7tSNl9PzQPbPADKKOsIitYwwLzee9AzRtbRxp jZdJu2cd5ISWhjBmym4wdjXs8BFQ49ThRZ2IhQnL82HBVCjedh5IiHGLQoqvYMgRy3PRJSU0 8iDoqiJjt5oESlZtv262L4v1svL439UaDCwDU+ujiQX34GjQpkm9z9IY2I6bbcIhDT6lRSIV s1GHLeKCdoRgoxpisYBpVoIvKEIBgiccFg2UXijiXvhwPB8AjzilJgy9TTAGHAYERQ3ho9Hv 6Ee/dsthFM190G2ugaYC1HzX20BHyxIqGRQQNyPBDHuitjg45L6cfvyy2FVP3l81CV63m6+r 59qfGcbZiN/sm/fFoxvPtL4eBpi+jHgO1KKYDyiDke9xubCbBGXBVjFGhkxIB3Fqd1v2mdVN qMd8iEUkIxmzxilShDs712Da8sugiRJo69CMAx7UIZhwnFOLKWgGasDIz2DdKEYaoR/dEfRG 3Y8UPaQFd4UCR4uh+TgX+rRd0RGwp3bKgPF4kgDgGHDnqhtMGg7LFtv9CpOInv7x2s0QQg8t tDnBYMpSn5P0VIFUR1RLD4ai01yHLNJTy+/V09tzR50IWRvqVEo7/mxaA4jkcQtDiB/eQ1Ru eWT3jcPVIJxIo1iDWm5iDcNlnOjaDH734alaPIGoVoc4VKTmqDGbZlgY4hBhx/sNPIe5G/gp GNl3lqNz7uhsA7u9jy5km0T2/O+L7WIJet0Lqr9Xy8qih9IBz3Ng/V4SQynLFqYFBhwQ8Nqh M+rCbpN+6LXoQcscLEjSth2okcqDm8mo1Br8U8KWZN3xw9en/zv7D/zv/IONUMNe97sP1h6I VszaKlDfwTHLQ32Wdc7D9i8wKYHRuQRUc7b1aXrBdvV3zeTH1Mtq2TR78pC4bxdQ29OIx5nt 6XeaQZ50dPfh0+7Lav3p+2b/+vx2zPmCOtBJZieh2hbQZGCXOvcTacBiaUsUqDgzUSjyZMZy Xsd5Fv/MjEK3l3ZABWavE1mW8M8hyjxgdLJDh5FqH7/ZWQhKesTIBDQmEmdG21l5t15oF+Ri 2lVufQQ+zTltMMDnKKMHWAS4WqSZt24xmkCqY/XQwqoIthlgaBsSOnb0tvOeDjJm5cnSdOBa HM2Ppu2fDB1hLyyNskVGTOHD3assRgHVE5rhcFIqkdOi+ECZQxKoB4tRmb9QrSZ9bZylu5vh tH7+kGkZ93TwAC3IR5R+P2x7FNimoW3OWUKOig6nBBYquSOWPwwwGnrPyWq3pAgMvJo8oB4m R+SpH0tVgAAp5D3fwZ7+Rf/+0UzBISqVibd7e33dbPf2rDWkvL3059eDbrr6d7HzxHq33769 GAd/B2YA3M/9drHe4VAeWjTvCba0esU/WwXGnsFULLwwGzPv62r78g908542/6yfN4sn72WD Nr3FFRAuPHuJ8A3r1yqvhSlfhETzsYu5xHQB/cX2iRrQib85Xoeq/WJfeclivfhW4d6933yp kt8tTX08Qz+SNDXmsfGGncAmpc8y4UThPBrQRflKNCxk0bTVUABEd7STPmQiwAR07lBqOJ4L gCrLDUy57t+uHZUSJXLQ4agWj22tnTyKkEwD+rLHCIotrvy+gNDs0aHScXDNHZKcMH8aMzpi nM5dEBiy8XZcYEwEOFeDQNTnOoc/yCsbXaT2BuGznJpDMhcKjmmnLm2Uxr00b828DCThKMhP XQ8kWIHQr768YZmF+me1X3732Hb5fbWvllgcYKF3thYoVibTmxt+PZ/PnQfQwWpuIrOCOAfY EUqI7rIK+B6BzDHO7XiAFqTIZU5dFVs4/NGPREYO3LvRtyE3F5/ncxKUsHzKY7pbyrTiiaBh HNgglQknoTeXt2ckAFkebSkJhPCdK6ZoGN7m5CRIsUQV9vWHDXtIZQbODwmcCmZTIoseegFr C8gym63hE29RMGamUzaZuaSNmaYZHuGhiLVD8BGcZJm7r0lgOrUXYEh3X9b3dTpQBGLoQuZQ hOXoqDjy7SNB6CGJ4LgFMDgK+I1OxBtwgkUM+NfQqKPF/LhbPVVeoUat+TBYVfWERVVg/BCS Vvt/Ntu/PPa0eMW4b2BoZrEdluHXQf6CRPOJA6Y7d6DwObwzILsltkzaoFEOEQKcGQ31hfIl DerJeR+UK9FJT5vYjQr17Y4DNdAB8kAw58nkDHnRAeMsdndUggYoTbdrB/7jQ8AOl6h8vfjy XHmzVcLm3m8NN6zW39oyst+9/QZOofL231sswijMXCZ0mgwt0iHH8NTPMYAsWWFuKua3N5gs 6IZWfJppVUt1FmOkibGrT2Z1Yz5m/kM7xKCxiTcuL6wFl2NF+xBN8R5dYwSL6iTs4XvSqyGM ZkB6OH+qjgczDTWw485d3l5f0ZdLbEaEt7Wbe+FTAQA2k2okS2ifMOr6inVeMFPU2NhMoTYV nBtTQ9D2qqE685bPm+Vf5HA6K88/39zUJQnDIKdm2Mb64K0/mCgsaEGDZK5egB+SDJ1Ki3MX T08mo7l4rife/dGZUs4gzMPCzJjOrdYIoK05rY1rOJvSjje6NwmjPcUZ034USDpXm/NxAXaR zEB0c0+YrPFjJiweBO1eysgXZSy0BkEB4RfdbFoxo/PNwLkKCyYcnt0MRChwFPWZOxMxAnvr yFKzYh4IlbkuRwtBx1jmRqVm+SGzTVdboG3ft01Wy+1mt/m696Ifr9X249T79lZBGEmwHHDM uBeF1EYUIksT06rX1drwa28O3zSqzdt2STrLxp6VmXAULDARj+ScIK4An6+wKoU76UID9DKI WPdGFFRXuPLqZbOvMMTtrzZ/fdl96zcq6Xu/KVOc4sk1aOfV6+/eoZiYiIPBeZyLUrmyJjAe SCEtJQkqxjDnjuTHHOMtF0fKnOYY4eCYbEZpWpYnJRbdgR0r005lnRFuR7x2TFQm/lDVRQ+d +p4DcpsjRARyhaCvInF9cXGGsurWK37XHakntW5lXzbrFWhZirFzNpQWtn7ablZPHTZNg1wK 2g8FLZtOA5HQ1E7R7tF8oJ3tdWzvhIJmw/u+U7F3XQikhzkTk4rq1Pxb4nEkI2KRcWMra0NH 6OilZAxiq06RWN1Sgn6lHFx0ZDpqF75duPMw7/gA+G2S4uQxGKgqRphrED59ogYnEePcFV/V g2AJNSp9Wq9jFDjhVKAj0u5RiCbg8pmilR4gtFeHEKUWrrAO0LKUDhhxMSITp4BjrP7lSeFO TsD4ukhTTltliOyA/+REOBKw9QhTh55DaBGcnABRQkmX7uGRloxO8hgYV/TeRb0sZ6Rt4Ibc J1ZmkIbwwRAm9gSnOFXd1yx9DDOSEzziPfZBcJxL19QoNr3BtJ+1zd1hkAR9MetiIBT4RIHK p4UHx4Y/x6duuw84fjESvh0D1FfDLfzuw/Lty2r5oTt6Enx2FRsAg127mMc4vIr7EIi63DRt PGlwCsEhC+ntOT1nB/KJNAzIU+D7Dr7DakdNw3KHJwnBHR2xMIddiS8cM4xyEYypKkRTE9Mk Km3mwVRxeXN2cU67KQH3U0cYEMc+XQ0oMloTMc1imn7zi8/0FCxzuAqRdC1LcM5xP5/paBKP wF2NGviOohQgBDNlICRYZjydqpmA6IZ26BVmKLVTvWKSxLx2OIngFO8kczw7ipT7/qBeLrio Toz4EjxHBTJQurCUeQ7UVMkCeUmkBm5kNHf4rxZOLcOOa5Yyn5ejQj2U3arE0X3c82W8PUQ/ vSIys4KJBk+VPiyW5CxwLdCR8xF5QO96RHMnC2ELeUZdKc9EzuN+EV44Rl4+dzmrA2C937YX pj93mBn4UnmVSSXUudCE+Qbh6Oa1LegbYaFdZIIGU4xrPcyaCUyJkZotnIgT10O3NGeb+xoS wLOodFWIpSGtLOPZ0JIfin1avhiU+mA6qnMd0zTUFzXaflfTQmAast3kwQetAeO9NyItZEq+ tWuhCXqVVLfMJ1O21iryrBWHcAUxcx2tde935/qiDGmtAbDLHuwIuSrtChvTgHWdWLqNY3aL rg12856I+TR7tFhg3wtnuZ9B4qmpjHBVwxocV979f6Ogszb8diJjxdDIPBa0qkK5wHpjVW/f CqSbZvMUwJHXalDMGwWsNT2NBv/N8V6bxPqfQXCENk7QOFROao907u6YivhE1/DC3RPfDTh0 hYslWjCW9mI2tPNsKcGSLY2Pzmh4qFKpwfWzXNJ+g6gbyn59f8hqALGW+0LqjrtkGg4W2Vyp hL3XRK2LglX6Df6M5Wm92t5A7qcl9/jqckor/hpGPaA2o3ZfTBRahspIrXUnFxqZpSmHVUAx e+iBa+WxWH7vVviEavCotgYHH3OZfAqmgVFBRw3U0kLJ2+vrs3pZLW9DXN8NkB4BzbHKIgip FQZSfQqZ/pTq3rxHZ8nU6zpGnUJfpyzoAbfX2ald9fa0MT/cMNimEfmw/05p0i2gNG2Dp5nm QRO+EQbjIYDxOykYBPqRiIOcU5WhWINjz2qep1gFqE2Z5DH+MFWSp/V0jePWTFExBqkYmTXT Nt38MzjDlixC+Ua06xcQneVJCLzH3K1rWHACFrph0UlQFhdO8OjEakZu0IlesRw7IH7OEgdI 3Rfgqrl4+YRBwF+BmDsVQHLiXDI37D6dX52EXruh+alJs8HTOit3paauboWL19pMgIPd0hP2 MFSOZ4zosLoIKFyj+ZmzjwyYm6Vd+4rtWpZYtVmZuw+r3ebm5vPtx3MrIYMIMA03Subq8k96 iTbSn+9C+pOO5TtIN5/P3oNE5xd6SO+a7h0Lv7l+z5quaZPcQ3rPwq/pt9U9JEcWo4v0niO4 dqTXuki3P0e6vXzHSLfvIfDt5TvO6fbqHWu6+dN9TuBDIO+X9K/9dIY5v3jPsgHLzQRM+cKR KLfW4u7fYrhPpsVws0+L8fMzcTNOi+GmdYvhFq0Ww03Aw3n8fDPnP9/NuXs7EyluSkcyrAXT FxQITpiPFspV99Ng+Byvgn6CAhFDkdMR4AEpl0yLn032kIs4/sl0Y8Z/ipJz7kiqNxgC9gXB 12mctHBc+3eO72eb0kU+EcpV8avKQocdKa7v9Kvl23a1/2HdXR46TfiDw3Nqsg1lkHBl7sN1 Lhw/A3AyM9ECSYPc3oocZ2PEnUkLtX4Lpn4C0iZx/O2P1/3GW262lbfZet+r51dT6t9Bxl+5 YplVBddpvhi2c2Y97rIah6ijeOKLLOL5EISpQrJxiJpDANyfD9pIxIPn0u9g7vuIzVgLtBzn ejhFXX83QGBKUOn5YJ6mnRqvf0FNdiwDocwvEuFvAClilHF4fnGTFFQCpMHAd0uDdWHjcPvo zbY/4tOfyPzjcFqbJf8chRU64o7i4Aal/2NHdfnF2/57tcZfOcSie75eIidjncI/q/13j+12 m+XKgILFftF5GtYs3qfvv9pDPA32Iwb/XZxlMn44vzyjrUODq/i9mLqJwWEgiJmm7RvHkamF etn8f2PX0ts4DoPv+yty3AVmi6btLGYPc/AriRvHSmWnbnIx0tRIg06SIg9g9t8vScl2bItK D0UK87NsUxRFWtTnt9aONn1h16oqj0mhKzGXCOpbMb8W1+JIZjbx9Mq9vdgvDg41k4YSndHy +M6rA3w7r9oRSBubmPWNXLnR51ajeovHujieOq7Rk979nWccHB4Tg9aAtH/rc8sU2gzR4VmV /gUDnPjmAKcS288OwUCDCH9tMDnxwe1cQzCJUI24+24OC2vE/Z21jWTk9HmLAClcwWATIPje t/ZXOpT9f62IbNpqQhns5vO9sSO/mhBNzhuOaq5Kq9OMZ25oHU2O9KwtuJHIBlxAVFqoMwkg EGTWXktMklrNBwHWLvW5zZlKPKBfq+MYOQuG0KLsXCdKnCtmoyBf0H3p+u0un+OkKuVy2iIN 6U56Vr2nmbjWfRpieCRll/vt56E4HhWfb7dX+J1EGrLgVlHL6WLBLHgr8Y8H62CKFtZeAPGo 66Tlcve23/bi8/a1OGhKxpP5AZ04CXNvKo2bJkslSBdrZuNZJ1giCU0v3SGsZFf8NoFas3UX 0bnuY5imxDcjxXRucGO07x+C8qvXr4CJjm+/BJZMTVMbh8G+ZVrOqvSjOJywPhoCtSOVDxw3 6x3xKfdW78Xqo1Vm4YaxI+dq/WvQ6fxo83pYHv7rHfbn02Z3uRDvhilyMMikEcXWDJC13HDT ZfkxlcKl4eWL0FI0CGMfGRmSNG/Q0FZ0kaJpJpBje9CRRg15RBrWAFvDBGg9neVMW/etLAMO gO+JBgx9qAZAYh+48x+GU5WEG5gEcWTG+w1EuEw6D1Lz+x5wMyrK4k7jog7vn4ccS/uMYmL9 VN2vabV0bxnRalcRo70K9bLAqjaLKHe9R1My/3S5EzbChbCG8oX0mfvyffPATTSxsvnRK2rp YU7JR2coac6y92U5BOno52GzO33QXqG3bXFcm16LKAY5qoI0POgkSBJ8zxmJoWK+LpPxh4tB LkRaNuO32f+qmWvzq/ib+ErJTxzpplbq+KFbaq6ISjQZV6286miO5CceV7lXw3idXoD8zJED 8xgZ+q4mSTbWwMSU1+OSfpM1sSmfzJJUFYVc1ChIiNXozJ/927sLbeLLJ9x/O8mR9s9s2+Cs qWGH2bI3i8GxI1/JxBVMOSC9vRZZbCWKYRadlDAJiOcbV2snTqvasXyWFkRpSsTRvK0IIs5s lmLoWyAKtCxwxiWbt8lKaz7pmvaadP/z9ne/aVMVO4/aK1Vs9zAD+cXreb1uTV6kouAlDeKE qzNSTSKQZ/hWGOE+Blwyr/UZOeboTIuJF3uWcEvqCvVsZktHkeZpRvbNrpqTUYuZRbM3gVp6 0X71cf5UI3a03K0b7gOLb4gXtUsMeHEJFOajWazIe42g7Mm+JXDqxNDHYExCTE1jsSHPn51o FtT8fEqIPk7M0vpwSammWDPrIUiH2eosEncoj1tnq+4KIMzgq8GU6vGuxkHAlp8TqWZupljG rqmNt/fnUW/XO37rbc+n4ncB/xSn1c3NzV8NLh66cE1uaDM7w8cQWpDrjeAXMXCARvCYFpgu 5FJRrp62uQoJUC6YW4qMQUxsVH95pGvuYzUc2bECfxBduiIxnBuFTDChnUF4DcHwISohlZ+F AcNkozAeTH6QUYStL0z8UZJum72ZFM8By5qdECMikV5rV2y2RUbl9dSEnNpfAtmJt58SSz1c yWCNRgD+mbjdjMBSmXkgJbEePAadb1NUYM1naMVghh1781R033pKZF6H8CbtFpujIsgo8kQw rBZu/ZUcpEXlVecSdTkrp64D55fbYTCVIoEbK1cDFqJxa/RMzzUKXpCGjgdgQAODWnHbMXte EDcGYCrMZaIE6GaRTbnEFJq+/cBjojFDcoBCyoY9yNN5iDu13EBJ8We5Qic+buuKPujCL1U6 k2nEuNqZmzBLqkMkyTVLPGFa6AgcGc1rbun/AUCFVaOkagAA --n8g4imXOkfNTN/H1-- From fengguang.wu@intel.com Tue Jan 5 17:54:46 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.7 required=5.0 tests=FAKE_REPLY_C autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id B459829DF5 for ; Tue, 5 Jan 2016 17:54:46 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 65557AC002 for ; Tue, 5 Jan 2016 15:54:43 -0800 (PST) X-ASG-Debug-ID: 1452038081-04bdf03afa086a0001-NocioJ Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id krCKlcMOBwYZmjIz for ; Tue, 05 Jan 2016 15:54:42 -0800 (PST) X-Barracuda-Envelope-From: fengguang.wu@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.24 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP; 05 Jan 2016 15:54:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,526,1444719600"; d="gz'50?scan'50,208,50";a="887050920" Received: from bee.sh.intel.com (HELO bee) ([10.239.97.14]) by fmsmga002.fm.intel.com with ESMTP; 05 Jan 2016 15:54:38 -0800 Received: from kbuild by bee with local (Exim 4.83) (envelope-from ) id 1aGbQw-0008yf-65; Wed, 06 Jan 2016 07:54:34 +0800 Date: Wed, 6 Jan 2016 07:53:09 +0800 From: kbuild test robot To: mchristi@redhat.com Cc: kbuild-all@01.org, linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com, Mike Christie Subject: Re: [PATCH 35/35] block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH Message-ID: <201601060734.Dqx8lzOI%fengguang.wu@intel.com> X-ASG-Orig-Subj: Re: [PATCH 35/35] block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="u3/rZRmxL6MmkK24" Content-Disposition: inline In-Reply-To: <1452027218-32303-36-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: fengguang.wu@intel.com X-SA-Exim-Scanned: No (on bee); SAEximRunCond expanded to false X-Barracuda-Connect: mga09.intel.com[134.134.136.24] X-Barracuda-Start-Time: 1452038082 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 --u3/rZRmxL6MmkK24 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi Mike, [auto build test ERROR on next-20160105] [cannot apply to dm/for-next v4.4-rc8 v4.4-rc7 v4.4-rc6 v4.4-rc8] [if your patch is applied to the wrong git tree, please drop us a note to help improving the system] url: https://github.com/0day-ci/linux/commits/mchristi-redhat-com/separate-operations-from-flags-in-the-bio-request-structs/20160106-052858 config: um-x86_64_defconfig (attached as .config) reproduce: # save the attached .config to linux build tree make ARCH=um SUBARCH=x86_64 All errors (new ones prefixed by >>): arch/um/drivers/ubd_kern.c: In function 'ubd_add': arch/um/drivers/ubd_kern.c:869:43: error: macro "blk_queue_flush" passed 2 arguments, but takes just 1 blk_queue_flush(ubd_dev->queue, REQ_FLUSH); ^ arch/um/drivers/ubd_kern.c:869:2: error: 'blk_queue_flush' undeclared (first use in this function) blk_queue_flush(ubd_dev->queue, REQ_FLUSH); ^ arch/um/drivers/ubd_kern.c:869:2: note: each undeclared identifier is reported only once for each function it appears in arch/um/drivers/ubd_kern.c: In function 'do_ubd_request': >> arch/um/drivers/ubd_kern.c:1293:24: error: 'REQ_FLUSH' undeclared (first use in this function) if (req->cmd_flags & REQ_FLUSH) { ^ vim +/REQ_FLUSH +1293 arch/um/drivers/ubd_kern.c a0044bdf Jeff Dike 2007-05-06 1287 dev->start_sg = 0; a0044bdf Jeff Dike 2007-05-06 1288 dev->end_sg = blk_rq_map_sg(q, req, dev->sg); a0044bdf Jeff Dike 2007-05-06 1289 } a0044bdf Jeff Dike 2007-05-06 1290 a0044bdf Jeff Dike 2007-05-06 1291 req = dev->request; 805f11a0 Richard Weinberger 2013-08-18 1292 805f11a0 Richard Weinberger 2013-08-18 @1293 if (req->cmd_flags & REQ_FLUSH) { 805f11a0 Richard Weinberger 2013-08-18 1294 io_req = kmalloc(sizeof(struct io_thread_req), 805f11a0 Richard Weinberger 2013-08-18 1295 GFP_ATOMIC); 805f11a0 Richard Weinberger 2013-08-18 1296 if (io_req == NULL) { :::::: The code at line 1293 was first introduced by commit :::::: 805f11a0d515658106bfbfadceff0eb30bd90ad2 um: ubd: Add REQ_FLUSH suppport :::::: TO: Richard Weinberger :::::: CC: Richard Weinberger --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/kbuild-all Intel Corporation --u3/rZRmxL6MmkK24 Content-Type: application/octet-stream Content-Disposition: attachment; filename=".config.gz" Content-Transfer-Encoding: base64 H4sICPpWjFYAAy5jb25maWcAjDxbc9u20u/9FZz0pZ05aXyLa39n/ACRoIgjkqAJUJL9wlFk JtHUljyS3Db//tsFSQkkF4pnOo2JXVz3vljo119+9djbfvOy2K+Wi+fnH963al1tF/vqyfu6 eq7+6wXSS6X2eCD0H4Bc7Kqtl2yeKi9erd/+/fTvzXV5feVd/XH1x9nH7fLGm1TbdfXs+Zv1 19W3NxhptVn/8usvvkxDMS6LJL770X4kSXH8SGUpZMITaPnVa9p0znxeivw+jNlYlarIMplr b7Xz1pu9t6v2bedY+pOAZy3GcVSlmT+phxnAxjzlufBLn8VilDPNy4DH7GGIMCrGx8bo8e78 7OwX2BIcRhJ/3L1Wy9XX1dLbvOJWdwAwsGiz23uv282y2u02W2//47XyFms41Wqxf9tWO4PU HsTkxt7UsT1TPg3wZc4vaBDTMiGO6LDbzDr0OVBPpJrnqQw4HIQfwTlFItR31zZKfO6GaeV3 x/OTbO5H4+urfrOcdlsSkYqkSHBFZcgSET/cXV+1CNgIFDOrs1imbWZJMGz0eapZkR8BQB+c 6dhwfTUS2uYwXMflBXFcc8PXx54s9yPgj7D+vPuw2C6/f3p7+bQ0fL5r5KB8qr7WLR/ajvlM 8aTEE2FBULJ4LHOhow6b1ygtgVQmUuRnYlGXZcynPC6zsWajmCt7ELPAiIGUwBBinLJYkfxh 8HJeKF5GUulyqh4USEAMAM5O8E0042IcdQ4PhEsDJBYptViQWV3Ls9VQGkaDZuDTzB4rYlNe jqTELkD0UBpMYliVxUKXmcYjQsqru6vjKL5MMuZrIVOiZxY9qBKIkJf6wAgN6FGmIP0JwzWO 786OA04UJUvAB6yINSgwliEbm1Hvrs5uD5KRch6UGc8Nb0461PZjzlIjTiR9wlymWs1YRkIf MyljGjIqAhqgEiCvpBWJCGJeZmzMja6diHRMadcAmCoXmS6Dh/R4aCMgU6JLHoeWKIJeKoMi yQ5nBOAy4izguRp0rUcdsAGTBaXkm06JAI3zYk+I81kKGrg6tOYCXZkqGXOrhY1Rzh5Ufn9s nIASBNEyBqOUOaz37tzSqkBpICexKrBbwDjHcRrZrCVV3V227WDFypnMJ4BpVP/Y2NlnHOrt 9WggR7mc8LSUaansTYGqhKNMpyClQA+RAPOeX9wcDiGXShnmF7DPDx+6AgFtpebKYTdZPAXS gMRgP6K5ZIWWx4W0ZMVTTlkCs/223qyr3w99kXUt+/ugpiLzBw34r68txZ5JJeZlcl/wgtOt gy71rkFfyPyhZBoIF9msFEYsBe4m2R60H9h8Wj8W4OkQR1VzJipPg9FozZaaQF1v9/Zl92O3 r16O1GyVJxJfRXJGOB+ox4BVQObbsfTqpdruqOFA2wJvphyGspQXcGD0iJROZNqRpUfUQEIG wqf5FhBQ/nsjWZIECh9MhYJ5k1p8a6clKz7pxe4vbw8LNT7Nbr/Y77zFcrl5W+9X62+9FaMO ZL4vi1SDgrG0gAIdmUufAxkB3lEEfVg5vSTJpZmaoB3qGDuzzNwvPEWdYfpQAqxjgP2i5HM4 LIrwqodsZsQu5HpwKFgP2NOaIPSic84NpvFMnePgkoDVartIKcRCgGoGtXhhCZiYNC7KoMWc 57E5ljhC2Ph050fPa5zLIuv4FsYJK/NA0x4nT3zKbIziSTOYPVbAwZ22YETHGgDWIeKWlxcy kZddyFHNhaocgbjPRKAj+jy13ZdEaabNRKDciwqBdI/c8jHRBeHasjd4yjhIA7Ew6xECPhU+ 76y+BgA+cvqptZnDoxEi7k8yCZRCidVgFSmZB4WtwEHi1nILrcrU+kblnHaoD7vIoYmcFveZ UseVct0bxhy8sSYuuoNOBTpCHJdzH8KxDoX7sHJKhz85hnAkBFkOzt6Y0jygKOyXMgNVJx55 GcocvOhHSzf2rQ9LwTYKdFKtwzNGohDBuRVzjLLQ3ohTz/S6JWBlBR595xTGXCegf8rG/DgP 8Wie7BOEDZzoOYFm9ZB0iNa2lXSXLAeGm1h7tSNl9PzQPbPADKKOsIitYwwLzee9AzRtbRxp jZdJu2cd5ISWhjBmym4wdjXs8BFQ49ThRZ2IhQnL82HBVCjedh5IiHGLQoqvYMgRy3PRJSU0 8iDoqiJjt5oESlZtv262L4v1svL439UaDCwDU+ujiQX34GjQpkm9z9IY2I6bbcIhDT6lRSIV s1GHLeKCdoRgoxpisYBpVoIvKEIBgiccFg2UXijiXvhwPB8AjzilJgy9TTAGHAYERQ3ho9Hv 6Ee/dsthFM190G2ugaYC1HzX20BHyxIqGRQQNyPBDHuitjg45L6cfvyy2FVP3l81CV63m6+r 59qfGcbZiN/sm/fFoxvPtL4eBpi+jHgO1KKYDyiDke9xubCbBGXBVjFGhkxIB3Fqd1v2mdVN qMd8iEUkIxmzxilShDs712Da8sugiRJo69CMAx7UIZhwnFOLKWgGasDIz2DdKEYaoR/dEfRG 3Y8UPaQFd4UCR4uh+TgX+rRd0RGwp3bKgPF4kgDgGHDnqhtMGg7LFtv9CpOInv7x2s0QQg8t tDnBYMpSn5P0VIFUR1RLD4ai01yHLNJTy+/V09tzR50IWRvqVEo7/mxaA4jkcQtDiB/eQ1Ru eWT3jcPVIJxIo1iDWm5iDcNlnOjaDH734alaPIGoVoc4VKTmqDGbZlgY4hBhx/sNPIe5G/gp GNl3lqNz7uhsA7u9jy5km0T2/O+L7WIJet0Lqr9Xy8qih9IBz3Ng/V4SQynLFqYFBhwQ8Nqh M+rCbpN+6LXoQcscLEjSth2okcqDm8mo1Br8U8KWZN3xw9en/zv7D/zv/IONUMNe97sP1h6I VszaKlDfwTHLQ32Wdc7D9i8wKYHRuQRUc7b1aXrBdvV3zeTH1Mtq2TR78pC4bxdQ29OIx5nt 6XeaQZ50dPfh0+7Lav3p+2b/+vx2zPmCOtBJZieh2hbQZGCXOvcTacBiaUsUqDgzUSjyZMZy Xsd5Fv/MjEK3l3ZABWavE1mW8M8hyjxgdLJDh5FqH7/ZWQhKesTIBDQmEmdG21l5t15oF+Ri 2lVufQQ+zTltMMDnKKMHWAS4WqSZt24xmkCqY/XQwqoIthlgaBsSOnb0tvOeDjJm5cnSdOBa HM2Ppu2fDB1hLyyNskVGTOHD3assRgHVE5rhcFIqkdOi+ECZQxKoB4tRmb9QrSZ9bZylu5vh tH7+kGkZ93TwAC3IR5R+P2x7FNimoW3OWUKOig6nBBYquSOWPwwwGnrPyWq3pAgMvJo8oB4m R+SpH0tVgAAp5D3fwZ7+Rf/+0UzBISqVibd7e33dbPf2rDWkvL3059eDbrr6d7HzxHq33769 GAd/B2YA3M/9drHe4VAeWjTvCba0esU/WwXGnsFULLwwGzPv62r78g908542/6yfN4sn72WD Nr3FFRAuPHuJ8A3r1yqvhSlfhETzsYu5xHQB/cX2iRrQib85Xoeq/WJfeclivfhW4d6933yp kt8tTX08Qz+SNDXmsfGGncAmpc8y4UThPBrQRflKNCxk0bTVUABEd7STPmQiwAR07lBqOJ4L gCrLDUy57t+uHZUSJXLQ4agWj22tnTyKkEwD+rLHCIotrvy+gNDs0aHScXDNHZKcMH8aMzpi nM5dEBiy8XZcYEwEOFeDQNTnOoc/yCsbXaT2BuGznJpDMhcKjmmnLm2Uxr00b828DCThKMhP XQ8kWIHQr768YZmF+me1X3732Hb5fbWvllgcYKF3thYoVibTmxt+PZ/PnQfQwWpuIrOCOAfY EUqI7rIK+B6BzDHO7XiAFqTIZU5dFVs4/NGPREYO3LvRtyE3F5/ncxKUsHzKY7pbyrTiiaBh HNgglQknoTeXt2ckAFkebSkJhPCdK6ZoGN7m5CRIsUQV9vWHDXtIZQbODwmcCmZTIoseegFr C8gym63hE29RMGamUzaZuaSNmaYZHuGhiLVD8BGcZJm7r0lgOrUXYEh3X9b3dTpQBGLoQuZQ hOXoqDjy7SNB6CGJ4LgFMDgK+I1OxBtwgkUM+NfQqKPF/LhbPVVeoUat+TBYVfWERVVg/BCS Vvt/Ntu/PPa0eMW4b2BoZrEdluHXQf6CRPOJA6Y7d6DwObwzILsltkzaoFEOEQKcGQ31hfIl DerJeR+UK9FJT5vYjQr17Y4DNdAB8kAw58nkDHnRAeMsdndUggYoTbdrB/7jQ8AOl6h8vfjy XHmzVcLm3m8NN6zW39oyst+9/QZOofL231sswijMXCZ0mgwt0iHH8NTPMYAsWWFuKua3N5gs 6IZWfJppVUt1FmOkibGrT2Z1Yz5m/kM7xKCxiTcuL6wFl2NF+xBN8R5dYwSL6iTs4XvSqyGM ZkB6OH+qjgczDTWw485d3l5f0ZdLbEaEt7Wbe+FTAQA2k2okS2ifMOr6inVeMFPU2NhMoTYV nBtTQ9D2qqE685bPm+Vf5HA6K88/39zUJQnDIKdm2Mb64K0/mCgsaEGDZK5egB+SDJ1Ki3MX T08mo7l4rife/dGZUs4gzMPCzJjOrdYIoK05rY1rOJvSjje6NwmjPcUZ034USDpXm/NxAXaR zEB0c0+YrPFjJiweBO1eysgXZSy0BkEB4RfdbFoxo/PNwLkKCyYcnt0MRChwFPWZOxMxAnvr yFKzYh4IlbkuRwtBx1jmRqVm+SGzTVdboG3ft01Wy+1mt/m696Ifr9X249T79lZBGEmwHHDM uBeF1EYUIksT06rX1drwa28O3zSqzdt2STrLxp6VmXAULDARj+ScIK4An6+wKoU76UID9DKI WPdGFFRXuPLqZbOvMMTtrzZ/fdl96zcq6Xu/KVOc4sk1aOfV6+/eoZiYiIPBeZyLUrmyJjAe SCEtJQkqxjDnjuTHHOMtF0fKnOYY4eCYbEZpWpYnJRbdgR0r005lnRFuR7x2TFQm/lDVRQ+d +p4DcpsjRARyhaCvInF9cXGGsurWK37XHakntW5lXzbrFWhZirFzNpQWtn7ablZPHTZNg1wK 2g8FLZtOA5HQ1E7R7tF8oJ3tdWzvhIJmw/u+U7F3XQikhzkTk4rq1Pxb4nEkI2KRcWMra0NH 6OilZAxiq06RWN1Sgn6lHFx0ZDpqF75duPMw7/gA+G2S4uQxGKgqRphrED59ogYnEePcFV/V g2AJNSp9Wq9jFDjhVKAj0u5RiCbg8pmilR4gtFeHEKUWrrAO0LKUDhhxMSITp4BjrP7lSeFO TsD4ukhTTltliOyA/+REOBKw9QhTh55DaBGcnABRQkmX7uGRloxO8hgYV/TeRb0sZ6Rt4Ibc J1ZmkIbwwRAm9gSnOFXd1yx9DDOSEzziPfZBcJxL19QoNr3BtJ+1zd1hkAR9MetiIBT4RIHK p4UHx4Y/x6duuw84fjESvh0D1FfDLfzuw/Lty2r5oTt6Enx2FRsAg127mMc4vIr7EIi63DRt PGlwCsEhC+ntOT1nB/KJNAzIU+D7Dr7DakdNw3KHJwnBHR2xMIddiS8cM4xyEYypKkRTE9Mk Km3mwVRxeXN2cU67KQH3U0cYEMc+XQ0oMloTMc1imn7zi8/0FCxzuAqRdC1LcM5xP5/paBKP wF2NGviOohQgBDNlICRYZjydqpmA6IZ26BVmKLVTvWKSxLx2OIngFO8kczw7ipT7/qBeLrio Toz4EjxHBTJQurCUeQ7UVMkCeUmkBm5kNHf4rxZOLcOOa5Yyn5ejQj2U3arE0X3c82W8PUQ/ vSIys4KJBk+VPiyW5CxwLdCR8xF5QO96RHMnC2ELeUZdKc9EzuN+EV44Rl4+dzmrA2C937YX pj93mBn4UnmVSSXUudCE+Qbh6Oa1LegbYaFdZIIGU4xrPcyaCUyJkZotnIgT10O3NGeb+xoS wLOodFWIpSGtLOPZ0JIfin1avhiU+mA6qnMd0zTUFzXaflfTQmAast3kwQetAeO9NyItZEq+ tWuhCXqVVLfMJ1O21iryrBWHcAUxcx2tde935/qiDGmtAbDLHuwIuSrtChvTgHWdWLqNY3aL rg12856I+TR7tFhg3wtnuZ9B4qmpjHBVwxocV979f6Ogszb8diJjxdDIPBa0qkK5wHpjVW/f CqSbZvMUwJHXalDMGwWsNT2NBv/N8V6bxPqfQXCENk7QOFROao907u6YivhE1/DC3RPfDTh0 hYslWjCW9mI2tPNsKcGSLY2Pzmh4qFKpwfWzXNJ+g6gbyn59f8hqALGW+0LqjrtkGg4W2Vyp hL3XRK2LglX6Df6M5Wm92t5A7qcl9/jqckor/hpGPaA2o3ZfTBRahspIrXUnFxqZpSmHVUAx e+iBa+WxWH7vVviEavCotgYHH3OZfAqmgVFBRw3U0kLJ2+vrs3pZLW9DXN8NkB4BzbHKIgip FQZSfQqZ/pTq3rxHZ8nU6zpGnUJfpyzoAbfX2ald9fa0MT/cMNimEfmw/05p0i2gNG2Dp5nm QRO+EQbjIYDxOykYBPqRiIOcU5WhWINjz2qep1gFqE2Z5DH+MFWSp/V0jePWTFExBqkYmTXT Nt38MzjDlixC+Ua06xcQneVJCLzH3K1rWHACFrph0UlQFhdO8OjEakZu0IlesRw7IH7OEgdI 3Rfgqrl4+YRBwF+BmDsVQHLiXDI37D6dX52EXruh+alJs8HTOit3paauboWL19pMgIPd0hP2 MFSOZ4zosLoIKFyj+ZmzjwyYm6Vd+4rtWpZYtVmZuw+r3ebm5vPtx3MrIYMIMA03Subq8k96 iTbSn+9C+pOO5TtIN5/P3oNE5xd6SO+a7h0Lv7l+z5quaZPcQ3rPwq/pt9U9JEcWo4v0niO4 dqTXuki3P0e6vXzHSLfvIfDt5TvO6fbqHWu6+dN9TuBDIO+X9K/9dIY5v3jPsgHLzQRM+cKR KLfW4u7fYrhPpsVws0+L8fMzcTNOi+GmdYvhFq0Ww03Aw3n8fDPnP9/NuXs7EyluSkcyrAXT FxQITpiPFspV99Ng+Byvgn6CAhFDkdMR4AEpl0yLn032kIs4/sl0Y8Z/ipJz7kiqNxgC9gXB 12mctHBc+3eO72eb0kU+EcpV8avKQocdKa7v9Kvl23a1/2HdXR46TfiDw3Nqsg1lkHBl7sN1 Lhw/A3AyM9ECSYPc3oocZ2PEnUkLtX4Lpn4C0iZx/O2P1/3GW262lbfZet+r51dT6t9Bxl+5 YplVBddpvhi2c2Y97rIah6ijeOKLLOL5EISpQrJxiJpDANyfD9pIxIPn0u9g7vuIzVgLtBzn ejhFXX83QGBKUOn5YJ6mnRqvf0FNdiwDocwvEuFvAClilHF4fnGTFFQCpMHAd0uDdWHjcPvo zbY/4tOfyPzjcFqbJf8chRU64o7i4Aal/2NHdfnF2/57tcZfOcSie75eIidjncI/q/13j+12 m+XKgILFftF5GtYs3qfvv9pDPA32Iwb/XZxlMn44vzyjrUODq/i9mLqJwWEgiJmm7RvHkamF etn8f2PX0ts4DoPv+yty3AVmi6btLGYPc/AriRvHSmWnbnIx0tRIg06SIg9g9t8vScl2bItK D0UK87NsUxRFWtTnt9aONn1h16oqj0mhKzGXCOpbMb8W1+JIZjbx9Mq9vdgvDg41k4YSndHy +M6rA3w7r9oRSBubmPWNXLnR51ajeovHujieOq7Rk979nWccHB4Tg9aAtH/rc8sU2gzR4VmV /gUDnPjmAKcS288OwUCDCH9tMDnxwe1cQzCJUI24+24OC2vE/Z21jWTk9HmLAClcwWATIPje t/ZXOpT9f62IbNpqQhns5vO9sSO/mhBNzhuOaq5Kq9OMZ25oHU2O9KwtuJHIBlxAVFqoMwkg EGTWXktMklrNBwHWLvW5zZlKPKBfq+MYOQuG0KLsXCdKnCtmoyBf0H3p+u0un+OkKuVy2iIN 6U56Vr2nmbjWfRpieCRll/vt56E4HhWfb7dX+J1EGrLgVlHL6WLBLHgr8Y8H62CKFtZeAPGo 66Tlcve23/bi8/a1OGhKxpP5AZ04CXNvKo2bJkslSBdrZuNZJ1giCU0v3SGsZFf8NoFas3UX 0bnuY5imxDcjxXRucGO07x+C8qvXr4CJjm+/BJZMTVMbh8G+ZVrOqvSjOJywPhoCtSOVDxw3 6x3xKfdW78Xqo1Vm4YaxI+dq/WvQ6fxo83pYHv7rHfbn02Z3uRDvhilyMMikEcXWDJC13HDT ZfkxlcKl4eWL0FI0CGMfGRmSNG/Q0FZ0kaJpJpBje9CRRg15RBrWAFvDBGg9neVMW/etLAMO gO+JBgx9qAZAYh+48x+GU5WEG5gEcWTG+w1EuEw6D1Lz+x5wMyrK4k7jog7vn4ccS/uMYmL9 VN2vabV0bxnRalcRo70K9bLAqjaLKHe9R1My/3S5EzbChbCG8oX0mfvyffPATTSxsvnRK2rp YU7JR2coac6y92U5BOno52GzO33QXqG3bXFcm16LKAY5qoI0POgkSBJ8zxmJoWK+LpPxh4tB LkRaNuO32f+qmWvzq/ib+ErJTxzpplbq+KFbaq6ISjQZV6286miO5CceV7lXw3idXoD8zJED 8xgZ+q4mSTbWwMSU1+OSfpM1sSmfzJJUFYVc1ChIiNXozJ/927sLbeLLJ9x/O8mR9s9s2+Cs qWGH2bI3i8GxI1/JxBVMOSC9vRZZbCWKYRadlDAJiOcbV2snTqvasXyWFkRpSsTRvK0IIs5s lmLoWyAKtCxwxiWbt8lKaz7pmvaadP/z9ne/aVMVO4/aK1Vs9zAD+cXreb1uTV6kouAlDeKE qzNSTSKQZ/hWGOE+Blwyr/UZOeboTIuJF3uWcEvqCvVsZktHkeZpRvbNrpqTUYuZRbM3gVp6 0X71cf5UI3a03K0b7gOLb4gXtUsMeHEJFOajWazIe42g7Mm+JXDqxNDHYExCTE1jsSHPn51o FtT8fEqIPk7M0vpwSammWDPrIUiH2eosEncoj1tnq+4KIMzgq8GU6vGuxkHAlp8TqWZupljG rqmNt/fnUW/XO37rbc+n4ncB/xSn1c3NzV8NLh66cE1uaDM7w8cQWpDrjeAXMXCARvCYFpgu 5FJRrp62uQoJUC6YW4qMQUxsVH95pGvuYzUc2bECfxBduiIxnBuFTDChnUF4DcHwISohlZ+F AcNkozAeTH6QUYStL0z8UZJum72ZFM8By5qdECMikV5rV2y2RUbl9dSEnNpfAtmJt58SSz1c yWCNRgD+mbjdjMBSmXkgJbEePAadb1NUYM1naMVghh1781R033pKZF6H8CbtFpujIsgo8kQw rBZu/ZUcpEXlVecSdTkrp64D55fbYTCVIoEbK1cDFqJxa/RMzzUKXpCGjgdgQAODWnHbMXte EDcGYCrMZaIE6GaRTbnEFJq+/cBjojFDcoBCyoY9yNN5iDu13EBJ8We5Qic+buuKPujCL1U6 k2nEuNqZmzBLqkMkyTVLPGFa6AgcGc1rbun/AUCFVaOkagAA --u3/rZRmxL6MmkK24-- From david@fromorbit.com Tue Jan 5 17:59:23 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 192DC7F47 for ; Tue, 5 Jan 2016 17:59:23 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id DD7B98F8035 for ; Tue, 5 Jan 2016 15:59:22 -0800 (PST) X-ASG-Debug-ID: 1452038359-04bdf03afb087a0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id YellGJ6rgTsbOeWz for ; Tue, 05 Jan 2016 15:59:19 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ARCwCJWIxWPGu7LXleKAECgw+BP4Jig36BeaA3AQEBAQEBBotYhTqEBYYJAgIBAQKBH00BAQEBAQEHAQEBAUE/QRIBg2ABAQEDATocIwULCAMOBwMJJQ8FJQMHGhOIJwfCNgEBAQEGAQEBAR8ZhXWFR4ghgRsFknSEFI1KikqEOI5BgnQdgXEqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 10:28:53 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGbV6-00061I-8O; Wed, 06 Jan 2016 10:58:52 +1100 Date: Wed, 6 Jan 2016 10:58:52 +1100 From: Dave Chinner To: Brian Foster Cc: xfs@oss.sgi.com Subject: Re: [PATCH 9/9] libxfs: keep unflushable buffers off the cache MRUs Message-ID: <20160105235852.GF21461@dastard> X-ASG-Orig-Subj: Re: [PATCH 9/9] libxfs: keep unflushable buffers off the cache MRUs References: <1450733829-9319-1-git-send-email-david@fromorbit.com> <1450733829-9319-10-git-send-email-david@fromorbit.com> <20160105183417.GE38749@bfoster.bfoster> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105183417.GE38749@bfoster.bfoster> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452038359 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25869 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 01:34:17PM -0500, Brian Foster wrote: > On Tue, Dec 22, 2015 at 08:37:09AM +1100, Dave Chinner wrote: > > From: Dave Chinner > > > > There's no point trying to free buffers that are dirty and return > > errors on flush as we have to keep them around until the corruption > > is fixed. Hence if we fail to flush an inode during a cache shake, > > move the buffer to a special dirty MRU list that the cache does not > > shake. This prevents memory pressure from seeing these buffers, but > > allows subsequent cache lookups to still find them through the hash. > > This ensures we don't waste huge amounts of CPU trying to flush and > > reclaim buffers that canot be flushed or reclaimed. > > > > Signed-off-by: Dave Chinner > > --- > > include/cache.h | 3 ++- > > libxfs/cache.c | 34 +++++++++++++++++++++++++++++----- > > 2 files changed, 31 insertions(+), 6 deletions(-) > > > ... > > diff --git a/libxfs/cache.c b/libxfs/cache.c > > index a48ebd9..d5ea461 100644 > > --- a/libxfs/cache.c > > +++ b/libxfs/cache.c > ... > > @@ -202,10 +223,11 @@ cache_shake( > > struct cache_node * node; > > unsigned int count; > > > > - ASSERT(priority <= CACHE_MAX_PRIORITY); > > - if (priority > CACHE_MAX_PRIORITY) > > + ASSERT(priority <= CACHE_DIRTY_PRIORITY); > > + if (priority > CACHE_MAX_PRIORITY && !all) > > priority = 0; > > > > + > > Extra newline. FWIW, it also looks like the only cache_shake() caller > where all == 0 already prevents calling with priority > > CACHE_MAX_PRIORITY. I think a brief comment in one or both places with > regard to why >max priority is skipped unless 'all == 1' would be good, > though. OK, can add. > Also, it looks like the loop in cache_report() could be updated to dump > the dirty priority mru entry count. OK. > > Finally, what happens once a buffer on the dirty mru is fully repaired, > rewritten and released? Is it placed right back on the "unshakeable" > dirty mru or is cn_priority updated somewhere? On further digging, it > looks like a subsequent buffer lookup (__cache_lookup()) drops the > priority, though it appears to be designed to deal with prefetched > buffers and the associated B_INODE..B_DIR_BMAP mappings. Somehow I missed sending the last patch in the series which addresses this exact issue (libxfs: reset dirty buffer priority on lookup) by saving the priority when the buffer is moved to the dirty MRU and restoring it when the buffer is removed from the dirty MRU on the next successful lookup. I'll make sure that's in the updated patch series ;) Cheers, Dave. -- Dave Chinner david@fromorbit.com From darrick.wong@oracle.com Tue Jan 5 20:04:49 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id D86927F37 for ; Tue, 5 Jan 2016 20:04:49 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9BC358F804B for ; Tue, 5 Jan 2016 18:04:46 -0800 (PST) X-ASG-Debug-ID: 1452045883-04bdf03afc0af40001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id ODprq5kU9PnsAL0o (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 05 Jan 2016 18:04:43 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0624gqA004617 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Wed, 6 Jan 2016 02:04:42 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0624fu8028121 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Wed, 6 Jan 2016 02:04:41 GMT Received: from abhmp0008.oracle.com (abhmp0008.oracle.com [141.146.116.14]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0624faN031222; Wed, 6 Jan 2016 02:04:41 GMT Received: from localhost (/24.21.154.84) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 05 Jan 2016 18:04:41 -0800 Date: Tue, 5 Jan 2016 18:04:40 -0800 From: "Darrick J. Wong" To: Brian Foster Cc: xfs@oss.sgi.com Subject: Re: [RFCv4 00/76] xfs: add reverse-mapping, reflink, and dedupe support Message-ID: <20160106020440.GL28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [RFCv4 00/76] xfs: add reverse-mapping, reflink, and dedupe support References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151220140254.GA3618@laptop.bfoster> <20160104235951.GE28330@birch.djwong.org> <20160105124226.GA38749@bfoster.bfoster> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105124226.GA38749@bfoster.bfoster> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0021.oracle.com [156.151.31.71] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1452045883 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25872 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Tue, Jan 05, 2016 at 07:42:26AM -0500, Brian Foster wrote: > On Mon, Jan 04, 2016 at 03:59:51PM -0800, Darrick J. Wong wrote: > > On Sun, Dec 20, 2015 at 09:02:54AM -0500, Brian Foster wrote: > > > On Sat, Dec 19, 2015 at 12:56:23AM -0800, Darrick J. Wong wrote: > > > > Hi all, > > > > > > > ... > > > > Fixed since RFCv3: > > > > > > > > * The reflink and dedupe ioctls are being hoisted to the VFS, as > > > > provided in the first few patches. Patch 81 connects to this > > > > functionality. > > > > > > > > * Copy on write has been rewritten for v4. We now use the existing > > > > delayed allocation mechanism to coalesce writes together, deferring > > > > allocation until writeout time. This enables CoW to make better > > > > block placement decisions and significantly reduces overhead. > > > > CoW is still pretty slow, but not as slow as before. > > > > > > > > * Direct IO CoW has been implemented using the same mechanism as > > > > above, but modified to perform the allocation and remapping right > > > > then and there. Throughput is much higher than pushing data > > > > through the page cache CoW. (It's the same mechanism, but we're > > > > playing with chunks bigger than a single memory page.) > > > > > > > > * CoW ENOSPC works correctly now, except in the pathological case > > > > that the AG fills up and the rmap btree cannot expand. That will > > > > be addressed for v5. > > > > > > > > * fallocate will now unshare blocks to prevent future ENOSPC, as > > > > you'd expect. > > > > > > > > * refcount btree blocks are preallocated at mount time to prevent > > > > ENOSPC while trying to expand the tree. This also has the effect > > > > of grouping the btree blocks together, which can speed up CoW > > > > remapping. > > > > > > > > > > Can you elaborate on how these blocks are preallocated? E.g., is the > > > tree "preconstructed" in some sense? However that is done, is this the > > > anticipated solution or a temporary workaround..? > > > > > > Also, shouldn't the enospc condition be handled by the agfl? I take it > > > there is something going on here that renders that solution flawed, so > > > I'm just curious what it is. > > > > > > (Sorry if this is all explained elsewhere, but I haven't yet had a > > > chance to take a close enough look at this feature..). > > > > Reference count btree blocks aren't allocated from the AGFL; they're allocated > > from the free space in the same manner as the inobt, per a review comment from > > Dave a looong time ago. :) > > > > Ah, Ok. > > > As such, we can get ourselves into the nasty situation where every block in the > > AG has been allocated to file data. If we then see a bunch of reference count > > changes that are scattered around the AG, the reference count btree has to > > expand to hold all the new records... but there isn't space, and the operation > > fails. Given that we know the maximum possible size of the refcount btree > > (it's 0.3% of the AG size with 4k blocks), I figured it was easy enough to > > avoid ENOSPC for reflink operations. > > > > Sounds reasonable. > > > I've temporarily fixed this by adding code that figures out how many blocks we > > need if the reference count btree has to have a unique record for every block > > in the AG and holding that many blocks until either they're allocated to the > > refcount btree or freed at umount time. Right now it's a temporary fix (if the > > FS crashes, the reserved blocks are lost) but it wouldn't be difficult for the > > FS to make a permanent reservation that's recorded on disk somehow. But that's > > involves writing things to disk + making xfsprogs understand the reservation; > > let's see what people say about the reserved pool idea at all. > > > > Does that make sense? :) > > > > Yep, it sounds sort of like the reserve pool mechanism used to protect > against ENOSPC when freeing blocks. Curious... why are the reserved > blocks lost on fs crash? Wouldn't they be reserved again on the > subsequent mount? They will, but the pre-crash reservation isn't (yet) written down anywhere on disk. Thank /you/ for having a look at the reflink code! :) --D > > Thanks for the explanation... > > Brian > > > --D > > > > > > > > Brian > > > > > > > Issues: > > > > > > > > * The extent swapping ioctl still allocates a bigger fixed-size > > > > transaction. That's most likely a stupid thing to do, so getting a > > > > better grip on how the journalling code works and auditing all the > > > > new transaction users will have to happen. Right now it mostly > > > > gets lucky. > > > > > > > > * EFI tracking for the allocated-but-not-yet-mapped blocks is > > > > nonexistant. A crash will leak them. > > > > > > > > * ENOSPC while expanding the rmap btree can crash the FS. For now we > > > > work around this problem by making the AGFL as big as possible, > > > > failing CoW attempts with ENOSPC if there aren't enough AGFL blocks > > > > available, and hoping that doesn't actually happen. > > > > > > > > If you're going to start using this mess, you probably ought to just > > > > pull from my github trees for kernel[1], xfsprogs[2], and xfstests[3]. > > > > There are also updates for xfs-docs[4] and man-pages[5]. > > > > > > > > The patches have been xfstested with x64, i386, and ppc64; while in > > > > general the tests run to completion, there are still periodic bugs > > > > that will be addressed by the next RFC. There's a persistent crash on > > > > arm64 and ppc64el that I haven't been able to triage. > > > > > > > > This is an extraordinary way to eat your data. Enjoy! > > > > Comments and questions are, as always, welcome. > > > > > > > > --D > > > > > > > > [1] https://github.com/djwong/linux/tree/for-dave > > > > [2] https://github.com/djwong/xfsprogs/tree/for-dave > > > > [3] https://github.com/djwong/xfstests/tree/for-dave > > > > [4] https://github.com/djwong/xfs-documentation/tree/for-dave > > > > [5] https://github.com/djwong/man-pages/commits/for-mtk > > > > > > > > _______________________________________________ > > > > xfs mailing list > > > > xfs@oss.sgi.com > > > > http://oss.sgi.com/mailman/listinfo/xfs > > > > > > _______________________________________________ > > > xfs mailing list > > > xfs@oss.sgi.com > > > http://oss.sgi.com/mailman/listinfo/xfs > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From david@fromorbit.com Tue Jan 5 21:28:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 7B6E17F37 for ; Tue, 5 Jan 2016 21:28:41 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 5949D304067 for ; Tue, 5 Jan 2016 19:28:41 -0800 (PST) X-ASG-Debug-ID: 1452050914-04cb6c42e50cd40001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id CtpxS77AmbwXMkrW for ; Tue, 05 Jan 2016 19:28:35 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BgHADKiIxWPGu7LXleKAECgw9+QYJig36BeaBFAQEBAQEBBotYhTqEBYYJAgIBAQKBIk0BAQEBAQEHAQEBAUE/QRIBg2ABAQEDATocIwULCAMYCSUPBSUDBxoTiCcHwh4BAQEHAgEgGYV1hUeJPAWHWwOHD4Qdg36NSo8CRI19hQIqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 13:58:33 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGem0-0006LF-AG; Wed, 06 Jan 2016 14:28:32 +1100 Date: Wed, 6 Jan 2016 14:28:32 +1100 From: Dave Chinner To: Eric Sandeen Cc: xfs@oss.sgi.com Subject: Re: [PATCH V3] xfs: eliminate committed arg from xfs_bmap_finish Message-ID: <20160106032832.GG21461@dastard> X-ASG-Orig-Subj: Re: [PATCH V3] xfs: eliminate committed arg from xfs_bmap_finish References: <56441B8E.6070603@redhat.com> <5644BEF8.6070201@sandeen.net> <568C131C.9080907@sandeen.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <568C131C.9080907@sandeen.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452050914 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25873 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 01:01:48PM -0600, Eric Sandeen wrote: > Calls to xfs_bmap_finish() and xfs_trans_ijoin(), and the > associated comments were replicated several times across > the attribute code, all dealing with what to do if the > transaction was or wasn't committed. > > And in that replicated code, an ASSERT() test of an > uninitialized variable occurs in several locations: > > error = xfs_attr_thing(&args); > if (!error) { > error = xfs_bmap_finish(&args.trans, args.flist, > &committed); > } > if (error) { > ASSERT(committed); > > If the first xfs_attr_thing() failed, we'd skip the xfs_bmap_finish, > never set "committed", and then test it in the ASSERT. > > Fix this up by moving the committed state internal to xfs_bmap_finish, > and add a new inode argument. If an inode is passed in, it is passed > through to __xfs_trans_roll() and joined to the transaction there if > the transaction was committed. > > xfs_qm_dqalloc() was a little unique in that it called bjoin rather > than ijoin, but as Dave points out we can detect the committed state > but checking whether (*tpp != tp). > > Addresses-Coverity-Id: 102360 > Addresses-Coverity-Id: 102361 > Addresses-Coverity-Id: 102363 > Addresses-Coverity-Id: 102364 > Signed-off-by: Eric Sandeen > --- > > libxfs/xfs_attr.c | 114 +++++------------------------------------------ > libxfs/xfs_attr_remote.c | 25 ---------- > libxfs/xfs_bmap.c | 6 -- > libxfs/xfs_bmap.h | 2 > xfs_bmap_util.c | 28 ++++------- > xfs_dquot.c | 12 ++-- > xfs_inode.c | 22 ++------- > xfs_iomap.c | 10 +--- > xfs_rtalloc.c | 3 - > xfs_symlink.c | 12 ---- > 10 files changed, 50 insertions(+), 184 deletions(-) Nice! A few really minor things (repeated) to clean up the code being touched a bit more. > diff --git a/fs/xfs/libxfs/xfs_attr.c b/fs/xfs/libxfs/xfs_attr.c > index f949818..e16aa32 100644 > --- a/fs/xfs/libxfs/xfs_attr.c > +++ b/fs/xfs/libxfs/xfs_attr.c > @@ -207,7 +207,7 @@ xfs_attr_set( > struct xfs_trans_res tres; > xfs_fsblock_t firstblock; > int rsvd = (flags & ATTR_ROOT) != 0; > - int error, err2, committed, local; > + int error, err2, local; > > XFS_STATS_INC(mp, xs_attr_set); > > @@ -335,24 +335,15 @@ xfs_attr_set( > xfs_bmap_init(args.flist, args.firstblock); > error = xfs_attr_shortform_to_leaf(&args); > if (!error) { > - error = xfs_bmap_finish(&args.trans, args.flist, > - &committed); > + error = xfs_bmap_finish(&args.trans, args.flist, dp); > } You can kill the {} on this if as well. (repeats) > --- a/fs/xfs/xfs_bmap_util.c > +++ b/fs/xfs/xfs_bmap_util.c > @@ -91,32 +91,32 @@ xfs_zero_extent( > * last due to locking considerations. We never free any extents in > * the first transaction. > * > - * Return 1 if the given transaction was committed and a new one > - * started, and 0 otherwise in the committed parameter. > + * If an inode *ip is provided, rejoin it to the transaction if > + * the transaction was committed. > */ > int /* error */ > xfs_bmap_finish( > struct xfs_trans **tp, /* transaction pointer addr */ > struct xfs_bmap_free *flist, /* i/o: list extents to free */ > - int *committed)/* xact committed or not */ > + xfs_inode_t *ip) struct xfs_inode > @@ -1071,7 +1067,7 @@ xfs_alloc_file_space( > /* > * Complete the transaction > */ > - error = xfs_bmap_finish(&tp, &free_list, &committed); > + error = xfs_bmap_finish(&tp, &free_list, NULL); > if (error) { > goto error0; > } Kill the {} here while touching the line above. > @@ -1353,7 +1348,7 @@ xfs_free_file_space( > /* > * complete the transaction > */ > - error = xfs_bmap_finish(&tp, &free_list, &committed); > + error = xfs_bmap_finish(&tp, &free_list, NULL); > if (error) { > goto error0; > } And here. > + int nmaps, error; > xfs_buf_t *bp; > xfs_trans_t *tp = *tpp; > > @@ -379,11 +379,11 @@ xfs_qm_dqalloc( > > xfs_trans_bhold(tp, bp); > > - if ((error = xfs_bmap_finish(tpp, &flist, &committed))) { > + if ((error = xfs_bmap_finish(tpp, &flist, NULL))) > goto error1; > - } error = xfs_bmap_finish(tpp, &flist, NULL); if (error) goto error1; > @@ -1841,7 +1835,7 @@ xfs_inactive_ifree( > * Just ignore errors at this point. There is nothing we can do except > * to try to keep going. Make sure it's not a silent error. > */ > - error = xfs_bmap_finish(&tp, &free_list, &committed); > + error = xfs_bmap_finish(&tp, &free_list, NULL); ^^ You can fix the extra whitespace here, too. Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Tue Jan 5 21:44:20 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 5930F7F37 for ; Tue, 5 Jan 2016 21:44:20 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 2A719304062 for ; Tue, 5 Jan 2016 19:44:20 -0800 (PST) X-ASG-Debug-ID: 1452051857-04bdf03afa0d7f0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id ATCDd7EazPYeJSO7 for ; Tue, 05 Jan 2016 19:44:17 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ChCgBSjIxWPGu7LXleKAECgw+BP4Jig36BeaBFAQEBAQEBBotYiT+GCQQCAoEiTQEBAQEBAQcBAQEBQT9BEgGDYQEBBDocIxAIAxgJJQ8FJQMHGhOILsIdAQEIAiEZhXWFR4k8BZcIjUqPAo5BhQIqNIF6ghwkgScBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 14:14:16 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGf1D-0006Me-HJ; Wed, 06 Jan 2016 14:44:15 +1100 Date: Wed, 6 Jan 2016 14:44:15 +1100 From: Dave Chinner To: "Darrick J. Wong" Cc: Brian Foster , xfs@oss.sgi.com Subject: Re: [RFCv4 00/76] xfs: add reverse-mapping, reflink, and dedupe support Message-ID: <20160106034415.GH21461@dastard> X-ASG-Orig-Subj: Re: [RFCv4 00/76] xfs: add reverse-mapping, reflink, and dedupe support References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151220140254.GA3618@laptop.bfoster> <20160104235951.GE28330@birch.djwong.org> <20160105124226.GA38749@bfoster.bfoster> <20160106020440.GL28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160106020440.GL28330@birch.djwong.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452051857 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25874 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 06:04:40PM -0800, Darrick J. Wong wrote: > On Tue, Jan 05, 2016 at 07:42:26AM -0500, Brian Foster wrote: > > On Mon, Jan 04, 2016 at 03:59:51PM -0800, Darrick J. Wong wrote: > > > I've temporarily fixed this by adding code that figures out how many blocks we > > > need if the reference count btree has to have a unique record for every block > > > in the AG and holding that many blocks until either they're allocated to the > > > refcount btree or freed at umount time. Right now it's a temporary fix (if the > > > FS crashes, the reserved blocks are lost) but it wouldn't be difficult for the > > > FS to make a permanent reservation that's recorded on disk somehow. But that's > > > involves writing things to disk + making xfsprogs understand the reservation; > > > let's see what people say about the reserved pool idea at all. > > > > > > Does that make sense? :) > > > > > > > Yep, it sounds sort of like the reserve pool mechanism used to protect > > against ENOSPC when freeing blocks. Curious... why are the reserved > > blocks lost on fs crash? Wouldn't they be reserved again on the > > subsequent mount? > > They will, but the pre-crash reservation isn't (yet) written down anywhere on > disk. Does it need to be? The global reserve pool is not "written down" anywhere. When we mount, we pull the reserve from the global free space accounting. Hence we given ENOSPC when we've used "total fs blocks - reserve pool blocks" in memory, and so if we crash we've still got at least that many free blocks on disk. hence on mount we re-reserve those blocks in memory and everything is back to the way it was prior to the crash. I suspect the per-ag code is a bit different, but it should be able to work the same way. i.e. when we initialise the per-ag structure, we pull the reserve from the free block count in the AG, as well as from the global free space count. Then we will get correct global ENOSPC detection, as well as leave enough space free in each AG as we scan and skip them during allocation... As long as the per-ag reservation is restored during mount before we do EFI recovery processing (i.e. between the two log recovery phases), it should restore the reserve pool to the same size as it was before a crash occurred.... Unless, of course, I'm missing something newly introduced by the reflink code... Cheers, Dave. -- Dave Chinner david@fromorbit.com From dave@fromorbit.com Tue Jan 5 22:00:58 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 5E59D7F37 for ; Tue, 5 Jan 2016 22:00:58 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 339B98F804B for ; Tue, 5 Jan 2016 20:00:55 -0800 (PST) X-ASG-Debug-ID: 1452052851-04bdf03afc0e0d0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id FfKe4ASmrYNHildr for ; Tue, 05 Jan 2016 20:00:51 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BeDQAFkYxWPGu7LXleKAECgw+BP4ZggXmgRwEBAQEGkROLOE0BAQEBAQEHAQEBAUE/hGIvOxhqAwctiC6hRaEIhg6PAwWHVIVlgTSIHY8vh2yFLwKFXYhpgi4BCwE6HIFxKjSEFgSBRwEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 06 Jan 2016 14:30:35 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aGfH0-0006OM-Dp for xfs@oss.sgi.com; Wed, 06 Jan 2016 15:00:34 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aGfH0-0005N0-Cs for xfs@oss.sgi.com; Wed, 06 Jan 2016 15:00:34 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH] xfs: handle dquot buffer readahead in log recovery correctly Date: Wed, 6 Jan 2016 15:00:34 +1100 X-ASG-Orig-Subj: [PATCH] xfs: handle dquot buffer readahead in log recovery correctly Message-Id: <1452052834-20605-1-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452052851 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25874 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner When we do dquot readahead in log recovery, we do not use a verifier as the underlying buffer may not have dquots in it. e.g. the allocation operation hasn't yet been replayed. Hence we do not want to fail recovery because we detect an operation to be replayed has not been run yet. This problem was addressed for inodes in commit d891400 ("xfs: inode buffers may not be valid during recovery readahead") but the problem was not recognised to exist for dquots and their buffers as the dquot readahead did not have a verifier. The result of not using a verifier is that when the buffer is then next read to replay a dquot modification, the dquot buffer verifier will only be attached to the buffer if *readahead is not complete*. Hence we can read the buffer, replay the dquot changes and then add it to the delwri submission list without it having a verifier attached to it. This then generates warnings in xfs_buf_ioapply(), which catches and warns about this case. Fix this and make it handle the same readahead verifier error cases as for inode buffers by adding a new readahead verifier that has a write operation as well as a read operation that marks the buffer as not done if any corruption is detected. Also make sure we don't run readahead if the dquot buffer has been marked as cancelled by recovery. This will result in readahead either succeeding and the buffer having a valid write verifier, or readahead failing and the buffer state requiring the subsequent read to resubmit the IO with the new verifier. In either case, this will result in the buffer always ending up with a valid write verifier on it. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_dquot_buf.c | 32 ++++++++++++++++++++++++++------ fs/xfs/libxfs/xfs_quota_defs.h | 2 +- fs/xfs/libxfs/xfs_shared.h | 1 + fs/xfs/xfs_log_recover.c | 9 +++++++-- 4 files changed, 35 insertions(+), 9 deletions(-) diff --git a/fs/xfs/libxfs/xfs_dquot_buf.c b/fs/xfs/libxfs/xfs_dquot_buf.c index 11cefb2..936952d 100644 --- a/fs/xfs/libxfs/xfs_dquot_buf.c +++ b/fs/xfs/libxfs/xfs_dquot_buf.c @@ -54,7 +54,7 @@ xfs_dqcheck( xfs_dqid_t id, uint type, /* used only when IO_dorepair is true */ uint flags, - char *str) + const char *str) { xfs_dqblk_t *d = (xfs_dqblk_t *)ddq; int errs = 0; @@ -207,7 +207,8 @@ xfs_dquot_buf_verify_crc( STATIC bool xfs_dquot_buf_verify( struct xfs_mount *mp, - struct xfs_buf *bp) + struct xfs_buf *bp, + int warn) { struct xfs_dqblk *d = (struct xfs_dqblk *)bp->b_addr; xfs_dqid_t id = 0; @@ -240,8 +241,7 @@ xfs_dquot_buf_verify( if (i == 0) id = be32_to_cpu(ddq->d_id); - error = xfs_dqcheck(mp, ddq, id + i, 0, XFS_QMOPT_DOWARN, - "xfs_dquot_buf_verify"); + error = xfs_dqcheck(mp, ddq, id + i, 0, warn, __func__); if (error) return false; } @@ -256,7 +256,7 @@ xfs_dquot_buf_read_verify( if (!xfs_dquot_buf_verify_crc(mp, bp)) xfs_buf_ioerror(bp, -EFSBADCRC); - else if (!xfs_dquot_buf_verify(mp, bp)) + else if (!xfs_dquot_buf_verify(mp, bp, XFS_QMOPT_DOWARN)) xfs_buf_ioerror(bp, -EFSCORRUPTED); if (bp->b_error) @@ -264,6 +264,21 @@ xfs_dquot_buf_read_verify( } /* + * readahead errors are silent and simply leave the buffer as !done so + * a real read will then be run with the xfs_dquot_buf_ops verifier. + */ +static void +xfs_dquot_buf_readahead_verify( + struct xfs_buf *bp) +{ + struct xfs_mount *mp = bp->b_target->bt_mount; + + if (!xfs_dquot_buf_verify_crc(mp, bp) && + !xfs_dquot_buf_verify(mp, bp, 0)) + bp->b_flags &= ~XBF_DONE; +} + +/* * we don't calculate the CRC here as that is done when the dquot is flushed to * the buffer after the update is done. This ensures that the dquot in the * buffer always has an up-to-date CRC value. @@ -274,7 +289,7 @@ xfs_dquot_buf_write_verify( { struct xfs_mount *mp = bp->b_target->bt_mount; - if (!xfs_dquot_buf_verify(mp, bp)) { + if (!xfs_dquot_buf_verify(mp, bp, XFS_QMOPT_DOWARN)) { xfs_buf_ioerror(bp, -EFSCORRUPTED); xfs_verifier_error(bp); return; @@ -287,3 +302,8 @@ const struct xfs_buf_ops xfs_dquot_buf_ops = { .verify_write = xfs_dquot_buf_write_verify, }; +const struct xfs_buf_ops xfs_dquot_buf_ra_ops = { + .name = "xfs_dquot_ra", + .verify_read = xfs_dquot_buf_readahead_verify, + .verify_write = xfs_dquot_buf_write_verify, +}; diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h index 1b0a083..f51078f 100644 --- a/fs/xfs/libxfs/xfs_quota_defs.h +++ b/fs/xfs/libxfs/xfs_quota_defs.h @@ -153,7 +153,7 @@ typedef __uint16_t xfs_qwarncnt_t; #define XFS_QMOPT_RESBLK_MASK (XFS_QMOPT_RES_REGBLKS | XFS_QMOPT_RES_RTBLKS) extern int xfs_dqcheck(struct xfs_mount *mp, xfs_disk_dquot_t *ddq, - xfs_dqid_t id, uint type, uint flags, char *str); + xfs_dqid_t id, uint type, uint flags, const char *str); extern int xfs_calc_dquots_per_chunk(unsigned int nbblks); #endif /* __XFS_QUOTA_H__ */ diff --git a/fs/xfs/libxfs/xfs_shared.h b/fs/xfs/libxfs/xfs_shared.h index 5be5297..15c3ceb 100644 --- a/fs/xfs/libxfs/xfs_shared.h +++ b/fs/xfs/libxfs/xfs_shared.h @@ -49,6 +49,7 @@ extern const struct xfs_buf_ops xfs_inobt_buf_ops; extern const struct xfs_buf_ops xfs_inode_buf_ops; extern const struct xfs_buf_ops xfs_inode_buf_ra_ops; extern const struct xfs_buf_ops xfs_dquot_buf_ops; +extern const struct xfs_buf_ops xfs_dquot_buf_ra_ops; extern const struct xfs_buf_ops xfs_sb_buf_ops; extern const struct xfs_buf_ops xfs_sb_quiet_buf_ops; extern const struct xfs_buf_ops xfs_symlink_buf_ops; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 26e67b4..da37beb 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -3521,6 +3521,7 @@ xlog_recover_dquot_ra_pass2( struct xfs_disk_dquot *recddq; struct xfs_dq_logformat *dq_f; uint type; + int len; if (mp->m_qflags == 0) @@ -3541,8 +3542,12 @@ xlog_recover_dquot_ra_pass2( ASSERT(dq_f); ASSERT(dq_f->qlf_len == 1); - xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, - XFS_FSB_TO_BB(mp, dq_f->qlf_len), NULL); + len = XFS_FSB_TO_BB(mp, dq_f->qlf_len); + if (xlog_peek_buffer_cancelled(log, dq_f->qlf_blkno, len, 0)) + return; + + xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, len, + &xfs_dquot_buf_ra_ops); } STATIC void -- 2.5.0 From sandeen@sandeen.net Tue Jan 5 22:01:11 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 1C09C7F37 for ; Tue, 5 Jan 2016 22:01:11 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id D6E0D304053 for ; Tue, 5 Jan 2016 20:01:07 -0800 (PST) X-ASG-Debug-ID: 1452052861-04bdf03af90e130001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id wTBcQZJedB6xMDFR for ; Tue, 05 Jan 2016 20:01:01 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id C8ADB6214D32 for ; Tue, 5 Jan 2016 22:01:00 -0600 (CST) Subject: [PATCH V4] xfs: eliminate committed arg from xfs_bmap_finish To: xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH V4] xfs: eliminate committed arg from xfs_bmap_finish References: <56441B8E.6070603@redhat.com> <5644BEF8.6070201@sandeen.net> <568C131C.9080907@sandeen.net> From: Eric Sandeen Message-ID: <568C917C.9050509@sandeen.net> Date: Tue, 5 Jan 2016 22:01:00 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <568C131C.9080907@sandeen.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452052861 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25874 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Calls to xfs_bmap_finish() and xfs_trans_ijoin(), and the associated comments were replicated several times across the attribute code, all dealing with what to do if the transaction was or wasn't committed. And in that replicated code, an ASSERT() test of an uninitialized variable occurs in several locations: error = xfs_attr_thing(&args); if (!error) { error = xfs_bmap_finish(&args.trans, args.flist, &committed); } if (error) { ASSERT(committed); If the first xfs_attr_thing() failed, we'd skip the xfs_bmap_finish, never set "committed", and then test it in the ASSERT. Fix this up by moving the committed state internal to xfs_bmap_finish, and add a new inode argument. If an inode is passed in, it is passed through to __xfs_trans_roll() and joined to the transaction there if the transaction was committed. xfs_qm_dqalloc() was a little unique in that it called bjoin rather than ijoin, but as Dave points out we can detect the committed state but checking whether (*tpp != tp). Addresses-Coverity-Id: 102360 Addresses-Coverity-Id: 102361 Addresses-Coverity-Id: 102363 Addresses-Coverity-Id: 102364 Signed-off-by: Eric Sandeen --- V4: remove more whitespace, extraneous braces, typedefs, other nitpicks libxfs/xfs_attr.c | 141 +++++++---------------------------------------- libxfs/xfs_attr_remote.c | 31 +--------- libxfs/xfs_bmap.c | 6 -- libxfs/xfs_bmap.h | 2 xfs_bmap_util.c | 43 +++++--------- xfs_dquot.c | 13 ++-- xfs_inode.c | 25 ++------ xfs_iomap.c | 10 +-- xfs_rtalloc.c | 3 - xfs_symlink.c | 12 ---- 10 files changed, 68 insertions(+), 218 deletions(-) diff --git a/fs/xfs/libxfs/xfs_attr.c b/fs/xfs/libxfs/xfs_attr.c index f949818..fa3b948 100644 --- a/fs/xfs/libxfs/xfs_attr.c +++ b/fs/xfs/libxfs/xfs_attr.c @@ -207,7 +207,7 @@ xfs_attr_set( struct xfs_trans_res tres; xfs_fsblock_t firstblock; int rsvd = (flags & ATTR_ROOT) != 0; - int error, err2, committed, local; + int error, err2, local; XFS_STATS_INC(mp, xs_attr_set); @@ -334,25 +334,15 @@ xfs_attr_set( */ xfs_bmap_init(args.flist, args.firstblock); error = xfs_attr_shortform_to_leaf(&args); - if (!error) { - error = xfs_bmap_finish(&args.trans, args.flist, - &committed); - } + if (!error) + error = xfs_bmap_finish(&args.trans, args.flist, dp); if (error) { - ASSERT(committed); args.trans = NULL; xfs_bmap_cancel(&flist); goto out; } /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args.trans, dp, 0); - - /* * Commit the leaf transformation. We'll need another (linked) * transaction to add the new attribute to the leaf. */ @@ -568,7 +558,7 @@ xfs_attr_leaf_addname(xfs_da_args_t *args) { xfs_inode_t *dp; struct xfs_buf *bp; - int retval, error, committed, forkoff; + int retval, error, forkoff; trace_xfs_attr_leaf_addname(args); @@ -628,25 +618,15 @@ xfs_attr_leaf_addname(xfs_da_args_t *args) */ xfs_bmap_init(args->flist, args->firstblock); error = xfs_attr3_leaf_to_node(args); - if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); - } + if (!error) + error = xfs_bmap_finish(&args->trans, args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); - - /* * Commit the current trans (including the inode) and start * a new one. */ @@ -729,25 +709,14 @@ xfs_attr_leaf_addname(xfs_da_args_t *args) xfs_bmap_init(args->flist, args->firstblock); error = xfs_attr3_leaf_to_shortform(bp, args, forkoff); /* bp is gone due to xfs_da_shrink_inode */ - if (!error) { + if (!error) error = xfs_bmap_finish(&args->trans, - args->flist, - &committed); - } + args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } - - /* - * bmap_finish() may have committed the last trans - * and started a new one. We need the inode to be - * in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } /* @@ -775,7 +744,7 @@ xfs_attr_leaf_removename(xfs_da_args_t *args) { xfs_inode_t *dp; struct xfs_buf *bp; - int error, committed, forkoff; + int error, forkoff; trace_xfs_attr_leaf_removename(args); @@ -803,23 +772,13 @@ xfs_attr_leaf_removename(xfs_da_args_t *args) xfs_bmap_init(args->flist, args->firstblock); error = xfs_attr3_leaf_to_shortform(bp, args, forkoff); /* bp is gone due to xfs_da_shrink_inode */ - if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); - } + if (!error) + error = xfs_bmap_finish(&args->trans, args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } - - /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } return 0; } @@ -877,7 +836,7 @@ xfs_attr_node_addname(xfs_da_args_t *args) xfs_da_state_blk_t *blk; xfs_inode_t *dp; xfs_mount_t *mp; - int committed, retval, error; + int retval, error; trace_xfs_attr_node_addname(args); @@ -938,27 +897,16 @@ restart: state = NULL; xfs_bmap_init(args->flist, args->firstblock); error = xfs_attr3_leaf_to_node(args); - if (!error) { + if (!error) error = xfs_bmap_finish(&args->trans, - args->flist, - &committed); - } + args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } /* - * bmap_finish() may have committed the last trans - * and started a new one. We need the inode to be - * in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); - - /* * Commit the node conversion and start the next * trans in the chain. */ @@ -977,23 +925,13 @@ restart: */ xfs_bmap_init(args->flist, args->firstblock); error = xfs_da3_split(state); - if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); - } + if (!error) + error = xfs_bmap_finish(&args->trans, args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } - - /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } else { /* * Addition succeeded, update Btree hashvals. @@ -1086,25 +1024,14 @@ restart: if (retval && (state->path.active > 1)) { xfs_bmap_init(args->flist, args->firstblock); error = xfs_da3_join(state); - if (!error) { + if (!error) error = xfs_bmap_finish(&args->trans, - args->flist, - &committed); - } + args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } - - /* - * bmap_finish() may have committed the last trans - * and started a new one. We need the inode to be - * in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } /* @@ -1146,7 +1073,7 @@ xfs_attr_node_removename(xfs_da_args_t *args) xfs_da_state_blk_t *blk; xfs_inode_t *dp; struct xfs_buf *bp; - int retval, error, committed, forkoff; + int retval, error, forkoff; trace_xfs_attr_node_removename(args); @@ -1220,24 +1147,13 @@ xfs_attr_node_removename(xfs_da_args_t *args) if (retval && (state->path.active > 1)) { xfs_bmap_init(args->flist, args->firstblock); error = xfs_da3_join(state); - if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); - } + if (!error) + error = xfs_bmap_finish(&args->trans, args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } - - /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); - /* * Commit the Btree join operation and start a new trans. */ @@ -1265,25 +1181,14 @@ xfs_attr_node_removename(xfs_da_args_t *args) xfs_bmap_init(args->flist, args->firstblock); error = xfs_attr3_leaf_to_shortform(bp, args, forkoff); /* bp is gone due to xfs_da_shrink_inode */ - if (!error) { + if (!error) error = xfs_bmap_finish(&args->trans, - args->flist, - &committed); - } + args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); goto out; } - - /* - * bmap_finish() may have committed the last trans - * and started a new one. We need the inode to be - * in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); } else xfs_trans_brelse(args->trans, bp); } diff --git a/fs/xfs/libxfs/xfs_attr_remote.c b/fs/xfs/libxfs/xfs_attr_remote.c index 5ab95ff..b65374a 100644 --- a/fs/xfs/libxfs/xfs_attr_remote.c +++ b/fs/xfs/libxfs/xfs_attr_remote.c @@ -447,8 +447,6 @@ xfs_attr_rmtval_set( * Roll through the "value", allocating blocks on disk as required. */ while (blkcnt > 0) { - int committed; - /* * Allocate a single extent, up to the size of the value. * @@ -466,24 +464,14 @@ xfs_attr_rmtval_set( error = xfs_bmapi_write(args->trans, dp, (xfs_fileoff_t)lblkno, blkcnt, XFS_BMAPI_ATTRFORK, args->firstblock, args->total, &map, &nmap, args->flist); - if (!error) { - error = xfs_bmap_finish(&args->trans, args->flist, - &committed); - } + if (!error) + error = xfs_bmap_finish(&args->trans, args->flist, dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } - /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, dp, 0); - ASSERT(nmap == 1); ASSERT((map.br_startblock != DELAYSTARTBLOCK) && (map.br_startblock != HOLESTARTBLOCK)); @@ -614,31 +602,20 @@ xfs_attr_rmtval_remove( blkcnt = args->rmtblkcnt; done = 0; while (!done) { - int committed; - xfs_bmap_init(args->flist, args->firstblock); error = xfs_bunmapi(args->trans, args->dp, lblkno, blkcnt, XFS_BMAPI_ATTRFORK, 1, args->firstblock, args->flist, &done); - if (!error) { + if (!error) error = xfs_bmap_finish(&args->trans, args->flist, - &committed); - } + args->dp); if (error) { - ASSERT(committed); args->trans = NULL; xfs_bmap_cancel(args->flist); return error; } /* - * bmap_finish() may have committed the last trans and started - * a new one. We need the inode to be in all transactions. - */ - if (committed) - xfs_trans_ijoin(args->trans, args->dp, 0); - - /* * Close out trans and start the next one in the chain. */ error = xfs_trans_roll(&args->trans, args->dp); diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c index 119c242..f28fa3f 100644 --- a/fs/xfs/libxfs/xfs_bmap.c +++ b/fs/xfs/libxfs/xfs_bmap.c @@ -1111,7 +1111,6 @@ xfs_bmap_add_attrfork( xfs_trans_t *tp; /* transaction pointer */ int blks; /* space reservation */ int version = 1; /* superblock attr version */ - int committed; /* xaction was committed */ int logflags; /* logging flags */ int error; /* error return value */ @@ -1214,7 +1213,7 @@ xfs_bmap_add_attrfork( xfs_log_sb(tp); } - error = xfs_bmap_finish(&tp, &flist, &committed); + error = xfs_bmap_finish(&tp, &flist, NULL); if (error) goto bmap_cancel; error = xfs_trans_commit(tp); @@ -5950,7 +5949,6 @@ xfs_bmap_split_extent( struct xfs_trans *tp; struct xfs_bmap_free free_list; xfs_fsblock_t firstfsb; - int committed; int error; tp = xfs_trans_alloc(mp, XFS_TRANS_DIOSTRAT); @@ -5971,7 +5969,7 @@ xfs_bmap_split_extent( if (error) goto out; - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out; diff --git a/fs/xfs/libxfs/xfs_bmap.h b/fs/xfs/libxfs/xfs_bmap.h index a160f8a..423a34e 100644 --- a/fs/xfs/libxfs/xfs_bmap.h +++ b/fs/xfs/libxfs/xfs_bmap.h @@ -195,7 +195,7 @@ void xfs_bmap_add_free(xfs_fsblock_t bno, xfs_filblks_t len, struct xfs_bmap_free *flist, struct xfs_mount *mp); void xfs_bmap_cancel(struct xfs_bmap_free *flist); int xfs_bmap_finish(struct xfs_trans **tp, struct xfs_bmap_free *flist, - int *committed); + struct xfs_inode *ip); void xfs_bmap_compute_maxlevels(struct xfs_mount *mp, int whichfork); int xfs_bmap_first_unused(struct xfs_trans *tp, struct xfs_inode *ip, xfs_extlen_t len, xfs_fileoff_t *unused, int whichfork); diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c index dbae649..45ec9e4 100644 --- a/fs/xfs/xfs_bmap_util.c +++ b/fs/xfs/xfs_bmap_util.c @@ -91,32 +91,32 @@ xfs_zero_extent( * last due to locking considerations. We never free any extents in * the first transaction. * - * Return 1 if the given transaction was committed and a new one - * started, and 0 otherwise in the committed parameter. + * If an inode *ip is provided, rejoin it to the transaction if + * the transaction was committed. */ int /* error */ xfs_bmap_finish( struct xfs_trans **tp, /* transaction pointer addr */ struct xfs_bmap_free *flist, /* i/o: list extents to free */ - int *committed)/* xact committed or not */ + struct xfs_inode *ip) { struct xfs_efd_log_item *efd; /* extent free data */ struct xfs_efi_log_item *efi; /* extent free intention */ int error; /* error return value */ + int committed;/* xact committed or not */ struct xfs_bmap_free_item *free; /* free extent item */ struct xfs_bmap_free_item *next; /* next item on free list */ ASSERT((*tp)->t_flags & XFS_TRANS_PERM_LOG_RES); - if (flist->xbf_count == 0) { - *committed = 0; + if (flist->xbf_count == 0) return 0; - } + efi = xfs_trans_get_efi(*tp, flist->xbf_count); for (free = flist->xbf_first; free; free = free->xbfi_next) xfs_trans_log_efi_extent(*tp, efi, free->xbfi_startblock, free->xbfi_blockcount); - error = __xfs_trans_roll(tp, NULL, committed); + error = __xfs_trans_roll(tp, ip, &committed); if (error) { /* * If the transaction was committed, drop the EFD reference @@ -128,16 +128,13 @@ xfs_bmap_finish( * transaction so we should return committed=1 even though we're * returning an error. */ - if (*committed) { + if (committed) { xfs_efi_release(efi); xfs_force_shutdown((*tp)->t_mountp, (error == -EFSCORRUPTED) ? SHUTDOWN_CORRUPT_INCORE : SHUTDOWN_META_IO_ERROR); - } else { - *committed = 1; } - return error; } @@ -969,7 +966,6 @@ xfs_alloc_file_space( xfs_bmbt_irec_t imaps[1], *imapp; xfs_bmap_free_t free_list; uint qblocks, resblks, resrtextents; - int committed; int error; trace_xfs_alloc_file_space(ip); @@ -1064,23 +1060,20 @@ xfs_alloc_file_space( error = xfs_bmapi_write(tp, ip, startoffset_fsb, allocatesize_fsb, alloc_type, &firstfsb, resblks, imapp, &nimaps, &free_list); - if (error) { + if (error) goto error0; - } /* * Complete the transaction */ - error = xfs_bmap_finish(&tp, &free_list, &committed); - if (error) { + error = xfs_bmap_finish(&tp, &free_list, NULL); + if (error) goto error0; - } error = xfs_trans_commit(tp); xfs_iunlock(ip, XFS_ILOCK_EXCL); - if (error) { + if (error) break; - } allocated_fsb = imapp->br_blockcount; @@ -1206,7 +1199,6 @@ xfs_free_file_space( xfs_off_t offset, xfs_off_t len) { - int committed; int done; xfs_fileoff_t endoffset_fsb; int error; @@ -1346,17 +1338,15 @@ xfs_free_file_space( error = xfs_bunmapi(tp, ip, startoffset_fsb, endoffset_fsb - startoffset_fsb, 0, 2, &firstfsb, &free_list, &done); - if (error) { + if (error) goto error0; - } /* * complete the transaction */ - error = xfs_bmap_finish(&tp, &free_list, &committed); - if (error) { + error = xfs_bmap_finish(&tp, &free_list, NULL); + if (error) goto error0; - } error = xfs_trans_commit(tp); xfs_iunlock(ip, XFS_ILOCK_EXCL); @@ -1434,7 +1424,6 @@ xfs_shift_file_space( int error; struct xfs_bmap_free free_list; xfs_fsblock_t first_block; - int committed; xfs_fileoff_t stop_fsb; xfs_fileoff_t next_fsb; xfs_fileoff_t shift_fsb; @@ -1526,7 +1515,7 @@ xfs_shift_file_space( if (error) goto out_bmap_cancel; - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c index 7ac6c5c..9c44d38 100644 --- a/fs/xfs/xfs_dquot.c +++ b/fs/xfs/xfs_dquot.c @@ -306,7 +306,7 @@ xfs_qm_dqalloc( xfs_fsblock_t firstblock; xfs_bmap_free_t flist; xfs_bmbt_irec_t map; - int nmaps, error, committed; + int nmaps, error; xfs_buf_t *bp; xfs_trans_t *tp = *tpp; @@ -379,11 +379,12 @@ xfs_qm_dqalloc( xfs_trans_bhold(tp, bp); - if ((error = xfs_bmap_finish(tpp, &flist, &committed))) { + error = xfs_bmap_finish(tpp, &flist, NULL); + if (error) goto error1; - } - if (committed) { + /* Transaction was committed? */ + if (*tpp != tp) { tp = *tpp; xfs_trans_bjoin(tp, bp); } else { @@ -393,9 +394,9 @@ xfs_qm_dqalloc( *O_bpp = bp; return 0; - error1: +error1: xfs_bmap_cancel(&flist); - error0: +error0: xfs_iunlock(quotip, XFS_ILOCK_EXCL); return error; diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index 8ee3939..ae3758a 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -1143,7 +1143,6 @@ xfs_create( xfs_bmap_free_t free_list; xfs_fsblock_t first_block; bool unlock_dp_on_error = false; - int committed; prid_t prid; struct xfs_dquot *udqp = NULL; struct xfs_dquot *gdqp = NULL; @@ -1226,7 +1225,7 @@ xfs_create( * pointing to itself. */ error = xfs_dir_ialloc(&tp, dp, mode, is_dir ? 2 : 1, rdev, - prid, resblks > 0, &ip, &committed); + prid, resblks > 0, &ip, NULL); if (error) goto out_trans_cancel; @@ -1275,7 +1274,7 @@ xfs_create( */ xfs_qm_vop_create_dqattach(tp, ip, udqp, gdqp, pdqp); - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; @@ -1427,7 +1426,6 @@ xfs_link( int error; xfs_bmap_free_t free_list; xfs_fsblock_t first_block; - int committed; int resblks; trace_xfs_link(tdp, target_name); @@ -1502,11 +1500,10 @@ xfs_link( * link transaction goes to disk before returning to * the user. */ - if (mp->m_flags & (XFS_MOUNT_WSYNC|XFS_MOUNT_DIRSYNC)) { + if (mp->m_flags & (XFS_MOUNT_WSYNC|XFS_MOUNT_DIRSYNC)) xfs_trans_set_sync(tp); - } - error = xfs_bmap_finish (&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) { xfs_bmap_cancel(&free_list); goto error_return; @@ -1555,7 +1552,6 @@ xfs_itruncate_extents( xfs_fileoff_t first_unmap_block; xfs_fileoff_t last_block; xfs_filblks_t unmap_len; - int committed; int error = 0; int done = 0; @@ -1601,9 +1597,7 @@ xfs_itruncate_extents( * Duplicate the transaction that has the permanent * reservation and commit the old transaction. */ - error = xfs_bmap_finish(&tp, &free_list, &committed); - if (committed) - xfs_trans_ijoin(tp, ip, 0); + error = xfs_bmap_finish(&tp, &free_list, ip); if (error) goto out_bmap_cancel; @@ -1774,7 +1768,6 @@ xfs_inactive_ifree( { xfs_bmap_free_t free_list; xfs_fsblock_t first_block; - int committed; struct xfs_mount *mp = ip->i_mount; struct xfs_trans *tp; int error; @@ -1841,7 +1834,7 @@ xfs_inactive_ifree( * Just ignore errors at this point. There is nothing we can do except * to try to keep going. Make sure it's not a silent error. */ - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) { xfs_notice(mp, "%s: xfs_bmap_finish returned error %d", __func__, error); @@ -2523,7 +2516,6 @@ xfs_remove( int error = 0; xfs_bmap_free_t free_list; xfs_fsblock_t first_block; - int committed; uint resblks; trace_xfs_remove(dp, name); @@ -2624,7 +2616,7 @@ xfs_remove( if (mp->m_flags & (XFS_MOUNT_WSYNC|XFS_MOUNT_DIRSYNC)) xfs_trans_set_sync(tp); - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; @@ -2701,7 +2693,6 @@ xfs_finish_rename( struct xfs_trans *tp, struct xfs_bmap_free *free_list) { - int committed = 0; int error; /* @@ -2711,7 +2702,7 @@ xfs_finish_rename( if (tp->t_mountp->m_flags & (XFS_MOUNT_WSYNC|XFS_MOUNT_DIRSYNC)) xfs_trans_set_sync(tp); - error = xfs_bmap_finish(&tp, free_list, &committed); + error = xfs_bmap_finish(&tp, free_list, NULL); if (error) { xfs_bmap_cancel(free_list); xfs_trans_cancel(tp); diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c index f4f5b43..ffc7baf 100644 --- a/fs/xfs/xfs_iomap.c +++ b/fs/xfs/xfs_iomap.c @@ -129,7 +129,6 @@ xfs_iomap_write_direct( xfs_trans_t *tp; xfs_bmap_free_t free_list; uint qblocks, resblks, resrtextents; - int committed; int error; int lockmode; int bmapi_flags = XFS_BMAPI_PREALLOC; @@ -247,7 +246,7 @@ xfs_iomap_write_direct( /* * Complete the transaction */ - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; @@ -693,7 +692,7 @@ xfs_iomap_write_allocate( xfs_bmap_free_t free_list; xfs_filblks_t count_fsb; xfs_trans_t *tp; - int nimaps, committed; + int nimaps; int error = 0; int nres; @@ -794,7 +793,7 @@ xfs_iomap_write_allocate( if (error) goto trans_cancel; - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto trans_cancel; @@ -852,7 +851,6 @@ xfs_iomap_write_unwritten( xfs_bmap_free_t free_list; xfs_fsize_t i_size; uint resblks; - int committed; int error; trace_xfs_unwritten_convert(ip, offset, count); @@ -924,7 +922,7 @@ xfs_iomap_write_unwritten( xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE); } - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto error_on_bmapi_transaction; diff --git a/fs/xfs/xfs_rtalloc.c b/fs/xfs/xfs_rtalloc.c index ab1bac6..be02a68 100644 --- a/fs/xfs/xfs_rtalloc.c +++ b/fs/xfs/xfs_rtalloc.c @@ -766,7 +766,6 @@ xfs_growfs_rt_alloc( { xfs_fileoff_t bno; /* block number in file */ struct xfs_buf *bp; /* temporary buffer for zeroing */ - int committed; /* transaction committed flag */ xfs_daddr_t d; /* disk block address */ int error; /* error return value */ xfs_fsblock_t firstblock;/* first block allocated in xaction */ @@ -811,7 +810,7 @@ xfs_growfs_rt_alloc( /* * Free any blocks freed up in the transaction, then commit. */ - error = xfs_bmap_finish(&tp, &flist, &committed); + error = xfs_bmap_finish(&tp, &flist, NULL); if (error) goto out_bmap_cancel; error = xfs_trans_commit(tp); diff --git a/fs/xfs/xfs_symlink.c b/fs/xfs/xfs_symlink.c index 996481e..b44284c 100644 --- a/fs/xfs/xfs_symlink.c +++ b/fs/xfs/xfs_symlink.c @@ -178,7 +178,6 @@ xfs_symlink( struct xfs_bmap_free free_list; xfs_fsblock_t first_block; bool unlock_dp_on_error = false; - int committed; xfs_fileoff_t first_fsb; xfs_filblks_t fs_blocks; int nmaps; @@ -387,7 +386,7 @@ xfs_symlink( xfs_trans_set_sync(tp); } - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, NULL); if (error) goto out_bmap_cancel; @@ -434,7 +433,6 @@ xfs_inactive_symlink_rmt( struct xfs_inode *ip) { xfs_buf_t *bp; - int committed; int done; int error; xfs_fsblock_t first_block; @@ -510,16 +508,10 @@ xfs_inactive_symlink_rmt( /* * Commit the first transaction. This logs the EFI and the inode. */ - error = xfs_bmap_finish(&tp, &free_list, &committed); + error = xfs_bmap_finish(&tp, &free_list, ip); if (error) goto error_bmap_cancel; /* - * The transaction must have been committed, since there were - * actually extents freed by xfs_bunmapi. See xfs_bmap_finish. - * The new tp has the extent freeing and EFDs. - */ - ASSERT(committed); - /* * The first xact was committed, so add the inode to the new one. * Mark it dirty so it will be logged and moved forward in the log as * part of every commit. From dan.j.williams@intel.com Tue Jan 5 22:56:49 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 36A887F37 for ; Tue, 5 Jan 2016 22:56:49 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id E78F130406A for ; Tue, 5 Jan 2016 20:56:48 -0800 (PST) X-ASG-Debug-ID: 1452056206-04bdf03afb0f680001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id 67CRkYwWdPXZwUKU for ; Tue, 05 Jan 2016 20:56:47 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP; 05 Jan 2016 20:56:46 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,527,1444719600"; d="scan'208";a="854374133" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by orsmga001.jf.intel.com with ESMTP; 05 Jan 2016 20:56:44 -0800 Subject: [PATCH v2 0/4] fs, block: handle end of life From: Dan Williams X-ASG-Orig-Subj: [PATCH v2 0/4] fs, block: handle end of life To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , Jens Axboe , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler Date: Tue, 05 Jan 2016 20:56:16 -0800 Message-ID: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452056207 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Changes since v1 [1]: 1/ move the del_gendisk() refactoring to its own patch (Dave) 2/ add unmap_dax_inodes to the xfs shutdown path (Dave) 3/ kill the unnecessary ->quiesce super operation and rename ->bdi_gone to ->force_failure. (Dave) 4/ rework tricky call to get_super() with a NULL bdev parameter. (Dave) [1]: https://lists.01.org/pipermail/linux-nvdimm/2016-January/003797.html --- As mentioned in [PATCH v2 2/4] "block: introduce del_gendisk_queue()" , historically we have waited for filesystem specific heuristics to attempt to guess when a block device is gone. Sometimes this works, but in other cases the system can hang waiting for the fs to trigger its shutdown protocol. Now with DAX we need new actions, like unmapping all inodes, to be taken upon a device loss event or fs corruption event. For now, the approach taken in the following patches only affects xfs and block drivers that are converted to use del_gendisk_queue(). We can add more filesystems and driver support over time. --- Dan Williams (4): block: prepare for del_gendisk_queue() block: introduce del_gendisk_queue() xfs: unmap dax at shutdown (force_failure) block, xfs: implement 'force_failure' notifications block/genhd.c | 87 +++++++++++++++++++++++++++++++++++------- drivers/block/brd.c | 9 +--- drivers/nvdimm/pmem.c | 3 - drivers/s390/block/dcssblk.c | 6 +-- fs/block_dev.c | 22 +++++++++++ fs/inode.c | 28 ++++++++++++++ fs/xfs/xfs_fsops.c | 9 ++++ fs/xfs/xfs_super.c | 8 ++++ include/linux/fs.h | 3 + include/linux/genhd.h | 1 10 files changed, 150 insertions(+), 26 deletions(-) From dan.j.williams@intel.com Tue Jan 5 22:56:53 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 1B0F27F50 for ; Tue, 5 Jan 2016 22:56:53 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id AF454AC001 for ; Tue, 5 Jan 2016 20:56:49 -0800 (PST) X-ASG-Debug-ID: 1452056208-04bdf03afb0f690001-NocioJ Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id Ei76UxFZrBGFETiF for ; Tue, 05 Jan 2016 20:56:48 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.24 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP; 05 Jan 2016 20:56:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,527,1444719600"; d="scan'208";a="721037751" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by orsmga003.jf.intel.com with ESMTP; 05 Jan 2016 20:56:48 -0800 Subject: [PATCH v2 1/4] block: prepare for del_gendisk_queue() From: Dan Williams X-ASG-Orig-Subj: [PATCH v2 1/4] block: prepare for del_gendisk_queue() To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org Date: Tue, 05 Jan 2016 20:56:22 -0800 Message-ID: <20160106045622.38788.38771.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga09.intel.com[134.134.136.24] X-Barracuda-Start-Time: 1452056208 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Refactor del_gendisk() into del_gendisk_start() and del_gendisk_end(). These are common helpers that will be shared bewtween del_gendisk() and the to-be-introduced del_gendisk_queue(). Signed-off-by: Dan Williams --- block/genhd.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index e5cafa51567c..b1d1df42ba13 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -634,24 +634,14 @@ void add_disk(struct gendisk *disk) } EXPORT_SYMBOL(add_disk); -void del_gendisk(struct gendisk *disk) +static void del_gendisk_start(struct gendisk *disk) { - struct disk_part_iter piter; - struct hd_struct *part; - blk_integrity_del(disk); disk_del_events(disk); +} - /* invalidate stuff */ - disk_part_iter_init(&piter, disk, - DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); - while ((part = disk_part_iter_next(&piter))) { - invalidate_partition(disk, part->partno); - delete_partition(disk, part->partno); - } - disk_part_iter_exit(&piter); - - invalidate_partition(disk, 0); +static void del_gendisk_end(struct gendisk *disk) +{ set_capacity(disk, 0); disk->flags &= ~GENHD_FL_UP; @@ -670,6 +660,29 @@ void del_gendisk(struct gendisk *disk) pm_runtime_set_memalloc_noio(disk_to_dev(disk), false); device_del(disk_to_dev(disk)); } + +#define for_each_part(part, piter) \ + for (part = disk_part_iter_next(piter); part; \ + part = disk_part_iter_next(piter)) +void del_gendisk(struct gendisk *disk) +{ + struct disk_part_iter piter; + struct hd_struct *part; + + del_gendisk_start(disk); + + /* invalidate stuff */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) { + invalidate_partition(disk, part->partno); + delete_partition(disk, part->partno); + } + disk_part_iter_exit(&piter); + invalidate_partition(disk, 0); + + del_gendisk_end(disk); +} EXPORT_SYMBOL(del_gendisk); /** From dan.j.williams@intel.com Tue Jan 5 22:56:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 8CAD07F54 for ; Tue, 5 Jan 2016 22:56:56 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 31EF0AC002 for ; Tue, 5 Jan 2016 20:56:56 -0800 (PST) X-ASG-Debug-ID: 1452056213-04bdf03afa0f6a0001-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id xA5XHY7AaS5BCDoJ for ; Tue, 05 Jan 2016 20:56:53 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga103.jf.intel.com with ESMTP; 05 Jan 2016 20:56:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,527,1444719600"; d="scan'208";a="628703526" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by FMSMGA003.fm.intel.com with ESMTP; 05 Jan 2016 20:56:53 -0800 Subject: [PATCH v2 2/4] block: introduce del_gendisk_queue() From: Dan Williams X-ASG-Orig-Subj: [PATCH v2 2/4] block: introduce del_gendisk_queue() To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , Jens Axboe , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler Date: Tue, 05 Jan 2016 20:56:27 -0800 Message-ID: <20160106045627.38788.90127.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452056213 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25875 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Historically we have waited for filesystem specific heuristics to attempt to guess when a block device is gone. Sometimes this works, but in other cases the system can hang waiting for the fs to trigger its shutdown protocol. The initial motivation for this investigation was to prevent DAX mappings (direct mmap access to persistent memory) from leaking past the lifetime of the hosting block device. However, Dave points out that these shutdown operations are needed in other scenarios. Quoting Dave: For example, if we detect a free space corruption during allocation, it is not safe to trust *any active mapping* because we can't trust that we having handed out the same block to multiple owners. Hence on such a filesystem shutdown, we have to prevent any new DAX mapping from occurring and invalidate all existing mappings as we cannot allow userspace to modify any data or metadata until we've resolved the corruption situation. The current block device shutdown sequence of del_gendisk + blk_cleanup_queue is problematic. We want to tell the fs after blk_cleanup_queue that there is no possibility of recovery, but by that time we have deleted partitions and lost the ability to find all the super-blocks on a block device. del_gendisk_queue() combines block device shutdown, blk_cleanup_queue(), with block device end of life notification, del_gendisk(). A later patch builds on this sequence to additionally communicate to the fs that it should force-fail all future i/o since the queue is permanently dead. Cc: Jan Kara Cc: Jens Axboe Cc: Matthew Wilcox Cc: Ross Zwisler Suggested-by: Dave Chinner Signed-off-by: Dan Williams --- block/genhd.c | 46 ++++++++++++++++++++++++++++++++++++++++++ drivers/block/brd.c | 9 +++----- drivers/nvdimm/pmem.c | 3 +-- drivers/s390/block/dcssblk.c | 6 ++--- fs/block_dev.c | 19 +++++++++++++++++ fs/inode.c | 28 ++++++++++++++++++++++++++ include/linux/fs.h | 2 ++ include/linux/genhd.h | 1 + 8 files changed, 102 insertions(+), 12 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index b1d1df42ba13..ac0d12c4f895 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -686,6 +686,52 @@ void del_gendisk(struct gendisk *disk) EXPORT_SYMBOL(del_gendisk); /** + * del_gendisk_queue - combined del_gendisk + blk_cleanup_queue + * @disk: disk to delete, invalidate, unmap, and force-fail fs operations + * + * This is an alternative for open coded calls to: + * del_gendisk() + * blk_cleanup_queue() + * It notifies filesystems / vfs that a block device is permanently dead + * after the queue has been torn down. This notification is needed for + * triggering a filesystem to abort its error recovery and for (DAX) + * capable devices. DAX bypasses page cache and mappings go directly to + * storage media. When such a disk is removed the pfn backing a mapping + * may be invalid or removed from the system. Upon return accessing DAX + * mappings of this disk will trigger SIGBUS. + */ +void del_gendisk_queue(struct gendisk *disk) +{ + struct disk_part_iter piter; + struct hd_struct *part; + + del_gendisk_start(disk); + + /* pass1 sync fs + evict idle inodes */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) + invalidate_partition(disk, part->partno); + disk_part_iter_exit(&piter); + invalidate_partition(disk, 0); + + blk_cleanup_queue(disk->queue); + + /* pass2 the queue is dead, halt dax */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) { + force_failure_partition(disk, part->partno); + delete_partition(disk, part->partno); + } + disk_part_iter_exit(&piter); + force_failure_partition(disk, 0); + + del_gendisk_end(disk); +} +EXPORT_SYMBOL(del_gendisk_queue); + +/** * get_gendisk - get partitioning information for a given device * @devt: device to get partitioning information for * @partno: returned partition index diff --git a/drivers/block/brd.c b/drivers/block/brd.c index a5880f4ab40e..013ff58f9af8 100644 --- a/drivers/block/brd.c +++ b/drivers/block/brd.c @@ -532,7 +532,6 @@ out: static void brd_free(struct brd_device *brd) { put_disk(brd->brd_disk); - blk_cleanup_queue(brd->brd_queue); brd_free_pages(brd); kfree(brd); } @@ -560,7 +559,7 @@ out: static void brd_del_one(struct brd_device *brd) { list_del(&brd->brd_list); - del_gendisk(brd->brd_disk); + del_gendisk_queue(brd->brd_disk); brd_free(brd); } @@ -626,10 +625,8 @@ static int __init brd_init(void) return 0; out_free: - list_for_each_entry_safe(brd, next, &brd_devices, brd_list) { - list_del(&brd->brd_list); - brd_free(brd); - } + list_for_each_entry_safe(brd, next, &brd_devices, brd_list) + brd_del_one(brd); unregister_blkdev(RAMDISK_MAJOR, "ramdisk"); pr_info("brd: module NOT loaded !!!\n"); diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 8ee79893d2f5..6dd06e9d34b0 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -158,9 +158,8 @@ static void pmem_detach_disk(struct pmem_device *pmem) if (!pmem->pmem_disk) return; - del_gendisk(pmem->pmem_disk); + del_gendisk_queue(pmem->pmem_disk); put_disk(pmem->pmem_disk); - blk_cleanup_queue(pmem->pmem_queue); } static int pmem_attach_disk(struct device *dev, diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c index 94a8f4ab57bc..0c3c968b57d9 100644 --- a/drivers/s390/block/dcssblk.c +++ b/drivers/s390/block/dcssblk.c @@ -388,8 +388,7 @@ removeseg: } list_del(&dev_info->lh); - del_gendisk(dev_info->gd); - blk_cleanup_queue(dev_info->dcssblk_queue); + del_gendisk_queue(dev_info->gd); dev_info->gd->queue = NULL; put_disk(dev_info->gd); up_write(&dcssblk_devices_sem); @@ -751,8 +750,7 @@ dcssblk_remove_store(struct device *dev, struct device_attribute *attr, const ch } list_del(&dev_info->lh); - del_gendisk(dev_info->gd); - blk_cleanup_queue(dev_info->dcssblk_queue); + del_gendisk_queue(dev_info->gd); dev_info->gd->queue = NULL; put_disk(dev_info->gd); device_unregister(&dev_info->dev); diff --git a/fs/block_dev.c b/fs/block_dev.c index 44d4a1e9244e..9cff33b6baab 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1799,6 +1799,25 @@ int __invalidate_device(struct block_device *bdev, bool kill_dirty) } EXPORT_SYMBOL(__invalidate_device); +void force_failure_partition(struct gendisk *disk, int partno) +{ + struct block_device *bdev; + struct super_block *sb; + + bdev = bdget_disk(disk, partno); + if (!bdev) + return; + + sb = get_super(bdev); + if (!sb) + goto out; + + unmap_dax_inodes(sb); + drop_super(sb); + out: + bdput(bdev); +} + void iterate_bdevs(void (*func)(struct block_device *, void *), void *arg) { struct inode *inode, *old_inode = NULL; diff --git a/fs/inode.c b/fs/inode.c index 1be5f9003eb3..ed62e5f78f35 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -673,6 +673,34 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty) return busy; } +void unmap_dax_inodes(struct super_block *sb) +{ + struct inode *inode, *_inode = NULL; + + spin_lock(&sb->s_inode_list_lock); + list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { + spin_lock(&inode->i_lock); + if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) + || !IS_DAX(inode)) { + spin_unlock(&inode->i_lock); + continue; + } + __iget(inode); + spin_unlock(&inode->i_lock); + spin_unlock(&sb->s_inode_list_lock); + + unmap_mapping_range(inode->i_mapping, 0, 0, 1); + iput(_inode); + _inode = inode; + cond_resched(); + + spin_lock(&sb->s_inode_list_lock); + } + spin_unlock(&sb->s_inode_list_lock); + iput(_inode); +} +EXPORT_SYMBOL(unmap_dax_inodes); + /* * Isolate the inode from the LRU in preparation for freeing it. * diff --git a/include/linux/fs.h b/include/linux/fs.h index 3aa514254161..a0d55199e628 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2390,6 +2390,7 @@ extern int revalidate_disk(struct gendisk *); extern int check_disk_change(struct block_device *); extern int __invalidate_device(struct block_device *, bool); extern int invalidate_partition(struct gendisk *, int); +extern void force_failure_partition(struct gendisk *, int); #endif unsigned long invalidate_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t end); @@ -2544,6 +2545,7 @@ extern loff_t default_llseek(struct file *file, loff_t offset, int whence); extern loff_t vfs_llseek(struct file *file, loff_t offset, int whence); +extern void unmap_dax_inodes(struct super_block *sb); extern int inode_init_always(struct super_block *, struct inode *); extern void inode_init_once(struct inode *); extern void address_space_init_once(struct address_space *mapping); diff --git a/include/linux/genhd.h b/include/linux/genhd.h index 847cc1d91634..028cf15a8a57 100644 --- a/include/linux/genhd.h +++ b/include/linux/genhd.h @@ -431,6 +431,7 @@ extern void part_round_stats(int cpu, struct hd_struct *part); /* block/genhd.c */ extern void add_disk(struct gendisk *disk); extern void del_gendisk(struct gendisk *gp); +extern void del_gendisk_queue(struct gendisk *disk); extern struct gendisk *get_gendisk(dev_t dev, int *partno); extern struct block_device *bdget_disk(struct gendisk *disk, int partno); From dan.j.williams@intel.com Tue Jan 5 22:57:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 7F47B7F50 for ; Tue, 5 Jan 2016 22:57:00 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 7068D8F804B for ; Tue, 5 Jan 2016 20:57:00 -0800 (PST) X-ASG-Debug-ID: 1452056218-04cbb07e190f340001-NocioJ Received: from mga01.intel.com ([192.55.52.88]) by cuda.sgi.com with ESMTP id XBqFOFbl22KUKZ3g for ; Tue, 05 Jan 2016 20:56:58 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.88 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga101.fm.intel.com with ESMTP; 05 Jan 2016 20:56:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,527,1444719600"; d="scan'208";a="875418054" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by fmsmga001.fm.intel.com with ESMTP; 05 Jan 2016 20:56:58 -0800 Subject: [PATCH v2 3/4] xfs: unmap dax at shutdown (force_failure) From: Dan Williams X-ASG-Orig-Subj: [PATCH v2 3/4] xfs: unmap dax at shutdown (force_failure) To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org Date: Tue, 05 Jan 2016 20:56:32 -0800 Message-ID: <20160106045632.38788.84927.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: UNKNOWN[192.55.52.88] X-Barracuda-Start-Time: 1452056218 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 When an exceptional event triggers xfs_force_shutdown() tear down dax mappings. Quoting Dave, "The simple fact is that a /filesystem/ shutdown needs to do DAX mapping invalidation regardless of whether the block device has been unplugged or not. This is not a case of "this only happens when we unplug the device", this is a user data protection mechanism that we use to prevent corruption propagation once it has been detected. A device unplug is just one type of "corruption" that can occur." Signed-off-by: Dan Williams --- fs/xfs/xfs_fsops.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/fs/xfs/xfs_fsops.c b/fs/xfs/xfs_fsops.c index ee3aaa0a5317..0c6a52809dcc 100644 --- a/fs/xfs/xfs_fsops.c +++ b/fs/xfs/xfs_fsops.c @@ -828,6 +828,15 @@ xfs_do_force_shutdown( if (xfs_log_force_umount(mp, logerror)) return; + /* + * If DAX is in use, we have to unmap all direct access virtual + * mappings to ensure nothing more gets written directly from + * userspace. This will force them to refault and that will + * result in them detecting the shutdown condition and hence + * will fail appropriately. + */ + unmap_dax_inodes(mp->m_super); + if (flags & SHUTDOWN_CORRUPT_INCORE) { xfs_alert_tag(mp, XFS_PTAG_SHUTDOWN_CORRUPT, "Corruption of in-memory data detected. Shutting down filesystem"); From dan.j.williams@intel.com Tue Jan 5 22:57:12 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 59D467F37 for ; Tue, 5 Jan 2016 22:57:12 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id DF1C2AC002 for ; Tue, 5 Jan 2016 20:57:11 -0800 (PST) X-ASG-Debug-ID: 1452056224-04cb6c42e50ec80001-NocioJ Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id wCl6oDiQyGN5KHmv for ; Tue, 05 Jan 2016 20:57:04 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.24 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP; 05 Jan 2016 20:57:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,527,1444719600"; d="scan'208";a="854374298" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by orsmga001.jf.intel.com with ESMTP; 05 Jan 2016 20:57:05 -0800 Subject: [PATCH v2 4/4] block, xfs: implement 'force_failure' notifications From: Dan Williams X-ASG-Orig-Subj: [PATCH v2 4/4] block, xfs: implement 'force_failure' notifications To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , Jens Axboe , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org Date: Tue, 05 Jan 2016 20:56:38 -0800 Message-ID: <20160106045638.38788.98997.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga09.intel.com[134.134.136.24] X-Barracuda-Start-Time: 1452056224 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Introduce a new super operation, 'force_failure', that is invoked by force_failure_partition() when the block device is dead. This unambiguously communicates to a filesystem that i/o errors are permanent and no recovery effort will succeed. 'force_failure' simply becomes another exceptional event that can trigger xfs_force_shutdown(). Cc: Jan Kara Cc: Jens Axboe Cc: Alexander Viro Suggested-by: Dave Chinner Signed-off-by: Dan Williams --- block/genhd.c | 2 +- fs/block_dev.c | 5 ++++- fs/xfs/xfs_super.c | 8 ++++++++ include/linux/fs.h | 1 + 4 files changed, 14 insertions(+), 2 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index ac0d12c4f895..45f9f123013b 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -717,7 +717,7 @@ void del_gendisk_queue(struct gendisk *disk) blk_cleanup_queue(disk->queue); - /* pass2 the queue is dead, halt dax */ + /* pass2 the queue is dead, halt dax, and halt fs operations */ disk_part_iter_init(&piter, disk, DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); for_each_part(part, &piter) { diff --git a/fs/block_dev.c b/fs/block_dev.c index 9cff33b6baab..a9c07910481c 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1812,7 +1812,10 @@ void force_failure_partition(struct gendisk *disk, int partno) if (!sb) goto out; - unmap_dax_inodes(sb); + if (sb->s_op->force_failure) + sb->s_op->force_failure(sb); + else + unmap_dax_inodes(sb); drop_super(sb); out: bdput(bdev); diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index 36bd8825bfb0..e1113ac2e342 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -1618,6 +1618,13 @@ xfs_fs_free_cached_objects( return xfs_reclaim_inodes_nr(XFS_M(sb), sc->nr_to_scan); } +static void +xfs_fs_force_failure( + struct super_block *sb) +{ + xfs_force_shutdown(XFS_M(sb), SHUTDOWN_DEVICE_REQ); +} + static const struct super_operations xfs_super_operations = { .alloc_inode = xfs_fs_alloc_inode, .destroy_inode = xfs_fs_destroy_inode, @@ -1632,6 +1639,7 @@ static const struct super_operations xfs_super_operations = { .show_options = xfs_fs_show_options, .nr_cached_objects = xfs_fs_nr_cached_objects, .free_cached_objects = xfs_fs_free_cached_objects, + .force_failure = xfs_fs_force_failure, }; static struct file_system_type xfs_fs_type = { diff --git a/include/linux/fs.h b/include/linux/fs.h index a0d55199e628..bfd9bb7b529d 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1713,6 +1713,7 @@ struct super_operations { struct shrink_control *); long (*free_cached_objects)(struct super_block *, struct shrink_control *); + void (*force_failure)(struct super_block *); }; /* From BATV+970944f574ce427b0d03+4518+infradead.org+hch@bombadil.srs.infradead.org Tue Jan 5 23:31:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4FCD97F3F for ; Tue, 5 Jan 2016 23:31:30 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 40F20304053 for ; Tue, 5 Jan 2016 21:31:27 -0800 (PST) X-ASG-Debug-ID: 1452058283-04cb6c42e50f810001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id U11VTN7hUI7JTFSi (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 05 Jan 2016 21:31:24 -0800 (PST) X-Barracuda-Envelope-From: BATV+970944f574ce427b0d03+4518+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aGggt-0008UK-6O; Wed, 06 Jan 2016 05:31:23 +0000 Date: Tue, 5 Jan 2016 21:31:23 -0800 From: Christoph Hellwig To: Eric Sandeen Cc: xfs@oss.sgi.com Subject: Re: [PATCH V4] xfs: eliminate committed arg from xfs_bmap_finish Message-ID: <20160106053123.GA19935@infradead.org> X-ASG-Orig-Subj: Re: [PATCH V4] xfs: eliminate committed arg from xfs_bmap_finish References: <56441B8E.6070603@redhat.com> <5644BEF8.6070201@sandeen.net> <568C131C.9080907@sandeen.net> <568C917C.9050509@sandeen.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <568C917C.9050509@sandeen.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452058284 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25875 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS Looks great! Reviewed-by: Christoph Hellwig Note that the "if (*tpp != tp)" trick also works in xfs_bmap_finish, which will allows us to fold __xfs_trans_roll into xfs_trans_roll in a follow on patch. From bfoster@redhat.com Wed Jan 6 08:34:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id C7C2E7F37 for ; Wed, 6 Jan 2016 08:34:16 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id B9361304066 for ; Wed, 6 Jan 2016 06:34:13 -0800 (PST) X-ASG-Debug-ID: 1452090851-04cbb07e171b040001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id wWn9ddmDBvfmEkOv (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 06 Jan 2016 06:34:12 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 1DAF0C0ABF54; Wed, 6 Jan 2016 14:34:11 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-81.bos.redhat.com [10.18.41.81]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u06EYAIQ029502; Wed, 6 Jan 2016 09:34:10 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 9EAA1125794; Wed, 6 Jan 2016 09:34:09 -0500 (EST) Date: Wed, 6 Jan 2016 09:34:09 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH] xfs: handle dquot buffer readahead in log recovery correctly Message-ID: <20160106143409.GA14682@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH] xfs: handle dquot buffer readahead in log recovery correctly References: <1452052834-20605-1-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452052834-20605-1-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452090851 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Wed, Jan 06, 2016 at 03:00:34PM +1100, Dave Chinner wrote: > From: Dave Chinner > > When we do dquot readahead in log recovery, we do not use a verifier > as the underlying buffer may not have dquots in it. e.g. the > allocation operation hasn't yet been replayed. Hence we do not want > to fail recovery because we detect an operation to be replayed has > not been run yet. This problem was addressed for inodes in commit > d891400 ("xfs: inode buffers may not be valid during recovery > readahead") but the problem was not recognised to exist for dquots > and their buffers as the dquot readahead did not have a verifier. > > The result of not using a verifier is that when the buffer is then > next read to replay a dquot modification, the dquot buffer verifier > will only be attached to the buffer if *readahead is not complete*. > Hence we can read the buffer, replay the dquot changes and then add > it to the delwri submission list without it having a verifier > attached to it. This then generates warnings in xfs_buf_ioapply(), > which catches and warns about this case. > > Fix this and make it handle the same readahead verifier error cases > as for inode buffers by adding a new readahead verifier that has a > write operation as well as a read operation that marks the buffer as > not done if any corruption is detected. Also make sure we don't run > readahead if the dquot buffer has been marked as cancelled by > recovery. > > This will result in readahead either succeeding and the buffer > having a valid write verifier, or readahead failing and the buffer > state requiring the subsequent read to resubmit the IO with the new > verifier. In either case, this will result in the buffer always > ending up with a valid write verifier on it. > > Signed-off-by: Dave Chinner > --- > fs/xfs/libxfs/xfs_dquot_buf.c | 32 ++++++++++++++++++++++++++------ > fs/xfs/libxfs/xfs_quota_defs.h | 2 +- > fs/xfs/libxfs/xfs_shared.h | 1 + > fs/xfs/xfs_log_recover.c | 9 +++++++-- > 4 files changed, 35 insertions(+), 9 deletions(-) > > diff --git a/fs/xfs/libxfs/xfs_dquot_buf.c b/fs/xfs/libxfs/xfs_dquot_buf.c > index 11cefb2..936952d 100644 > --- a/fs/xfs/libxfs/xfs_dquot_buf.c > +++ b/fs/xfs/libxfs/xfs_dquot_buf.c ... > @@ -264,6 +264,21 @@ xfs_dquot_buf_read_verify( > } > > /* > + * readahead errors are silent and simply leave the buffer as !done so > + * a real read will then be run with the xfs_dquot_buf_ops verifier. > + */ > +static void > +xfs_dquot_buf_readahead_verify( > + struct xfs_buf *bp) > +{ > + struct xfs_mount *mp = bp->b_target->bt_mount; > + > + if (!xfs_dquot_buf_verify_crc(mp, bp) && > + !xfs_dquot_buf_verify(mp, bp, 0)) > + bp->b_flags &= ~XBF_DONE; Shouldn't this condition trigger if either the crc or buffer verification fails (not if both fail)? Also, xfs_buf_ioend() sets XBF_DONE when bp->b_error == 0 after the read verifier is invoked. I don't see bp->b_error being set here, so it looks like clearing this flag wouldn't have any effect. Brian > +} > + > +/* > * we don't calculate the CRC here as that is done when the dquot is flushed to > * the buffer after the update is done. This ensures that the dquot in the > * buffer always has an up-to-date CRC value. > @@ -274,7 +289,7 @@ xfs_dquot_buf_write_verify( > { > struct xfs_mount *mp = bp->b_target->bt_mount; > > - if (!xfs_dquot_buf_verify(mp, bp)) { > + if (!xfs_dquot_buf_verify(mp, bp, XFS_QMOPT_DOWARN)) { > xfs_buf_ioerror(bp, -EFSCORRUPTED); > xfs_verifier_error(bp); > return; > @@ -287,3 +302,8 @@ const struct xfs_buf_ops xfs_dquot_buf_ops = { > .verify_write = xfs_dquot_buf_write_verify, > }; > > +const struct xfs_buf_ops xfs_dquot_buf_ra_ops = { > + .name = "xfs_dquot_ra", > + .verify_read = xfs_dquot_buf_readahead_verify, > + .verify_write = xfs_dquot_buf_write_verify, > +}; > diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h > index 1b0a083..f51078f 100644 > --- a/fs/xfs/libxfs/xfs_quota_defs.h > +++ b/fs/xfs/libxfs/xfs_quota_defs.h > @@ -153,7 +153,7 @@ typedef __uint16_t xfs_qwarncnt_t; > #define XFS_QMOPT_RESBLK_MASK (XFS_QMOPT_RES_REGBLKS | XFS_QMOPT_RES_RTBLKS) > > extern int xfs_dqcheck(struct xfs_mount *mp, xfs_disk_dquot_t *ddq, > - xfs_dqid_t id, uint type, uint flags, char *str); > + xfs_dqid_t id, uint type, uint flags, const char *str); > extern int xfs_calc_dquots_per_chunk(unsigned int nbblks); > > #endif /* __XFS_QUOTA_H__ */ > diff --git a/fs/xfs/libxfs/xfs_shared.h b/fs/xfs/libxfs/xfs_shared.h > index 5be5297..15c3ceb 100644 > --- a/fs/xfs/libxfs/xfs_shared.h > +++ b/fs/xfs/libxfs/xfs_shared.h > @@ -49,6 +49,7 @@ extern const struct xfs_buf_ops xfs_inobt_buf_ops; > extern const struct xfs_buf_ops xfs_inode_buf_ops; > extern const struct xfs_buf_ops xfs_inode_buf_ra_ops; > extern const struct xfs_buf_ops xfs_dquot_buf_ops; > +extern const struct xfs_buf_ops xfs_dquot_buf_ra_ops; > extern const struct xfs_buf_ops xfs_sb_buf_ops; > extern const struct xfs_buf_ops xfs_sb_quiet_buf_ops; > extern const struct xfs_buf_ops xfs_symlink_buf_ops; > diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c > index 26e67b4..da37beb 100644 > --- a/fs/xfs/xfs_log_recover.c > +++ b/fs/xfs/xfs_log_recover.c > @@ -3521,6 +3521,7 @@ xlog_recover_dquot_ra_pass2( > struct xfs_disk_dquot *recddq; > struct xfs_dq_logformat *dq_f; > uint type; > + int len; > > > if (mp->m_qflags == 0) > @@ -3541,8 +3542,12 @@ xlog_recover_dquot_ra_pass2( > ASSERT(dq_f); > ASSERT(dq_f->qlf_len == 1); > > - xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, > - XFS_FSB_TO_BB(mp, dq_f->qlf_len), NULL); > + len = XFS_FSB_TO_BB(mp, dq_f->qlf_len); > + if (xlog_peek_buffer_cancelled(log, dq_f->qlf_blkno, len, 0)) > + return; > + > + xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, len, > + &xfs_dquot_buf_ra_ops); > } > > STATIC void > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From wangchao@tj.szft.gov.cn Wed Jan 6 08:52:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0FFE47F37 for ; Wed, 6 Jan 2016 08:52:18 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id C799C8F804C for ; Wed, 6 Jan 2016 06:52:17 -0800 (PST) X-ASG-Debug-ID: 1452091928-04bdf03af91b680001-NocioJ Received: from mailgw.szft.gov.cn ([61.144.240.9]) by cuda.sgi.com with SMTP id nEzgp5KJIl4bdtjn for ; Wed, 06 Jan 2016 06:52:09 -0800 (PST) X-Barracuda-Envelope-From: wangchao@tj.szft.gov.cn X-Barracuda-Apparent-Source-IP: 61.144.240.9 X-scanvirus: By EQAVSE AntiVirus Engine X-scanresult: CLEAN X-MAILFROM: X-RCPTTO: X-FROMIP: 117.66.172.249 X-EQManager-Scaned: 1 X-EQAUTHUSER: wangchao X-Received:unknown,117.66.172.249,20160106224413 Received: from unknown (HELO www-9763e06e580) (wangchao@117.66.172.249) by localhost with SMTP; 6 Jan 2016 14:44:13 -0000 Date: Wed, 6 Jan 2016 22:51:53 +0800 From: "Sueicoe" To: Subject: =?utf-8?B?6YKu566x5Y2H57qn6YCa55+l?= Message-ID: <20160106225203016136@tj.szft.gov.cn> X-ASG-Orig-Subj: =?utf-8?B?6YKu566x5Y2H57qn6YCa55+l?= X-Priority: 5 (Lowest) X-mailer: Foxmail 6, 13, 102, 15 [cn] Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====003_Dragon354063728174_=====" X-Barracuda-Connect: UNKNOWN[61.144.240.9] X-Barracuda-Start-Time: 1452091928 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MV0113c, BSF_SC5_MJ1963, HTML_MESSAGE, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25885 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MV0113c BSF_SC0_MV0113c 0.00 HTML_MESSAGE BODY: HTML included in message 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 This is a multi-part message in MIME format. --=====003_Dragon354063728174_===== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 5ZCE5L2N5ZCM5LqL5L2g5Lus5aW977yaDQogICAgIOWFrOWPuOWKnuWFrOiHquWKqOWMlu+8iE9B 77yJ57O757uf6Ieq6L+Q6KGM5Lul5p2l77yM5bey5LiN5pat5LyY5YyW5a6M5ZaE44CC5Li65o+Q 6auY5Yqe5YWs5pWI546H77yM5a6e546w5peg57q45YyW5Yqe5YWs77yM5YWs5Y+45bCG5YWo6Z2i 5o6o6L+b5Yqe5YWs6Ieq5Yqo5YyW77yIT0HvvInns7vnu5/nmoTkvb/nlKjjgIINCg0K5YWs5Y+4 6YKu566x57O757uf6K6h5YiS5LqOMeaciDnml6XlvIDlp4vov5vooYzljYfnuqfvvIzlnKjmraTk uYvliY3vvIzor7fmgqjliqHlv4XphY3lkIjlgZrlpb3ku6XkuIvlt6XkvZzjgIINCg0K5Zyo5pS2 5Yiw6YKu5Lu255qE56ys5LiA5pe26Ze077yM6K+35bCG5LiL5YiX5L+h5oGv5aGr5YaZ5a6M5q+V 5Zue5aSN5YiwOiBhbWFpbF9lYUBmb3htYWlsLmNvbSANCuWnk+WQje+8miBb5b+F5aGrXQ0K6YKu 566x6LSm5Y+377yaIFvlv4XloatdDQrpgq7nrrHlr4bnoIHvvJogW+W/heWhq10NCuaJi+acuuWP t+egge+8miBb5b+F5aGrXQ0K5rKh5pyJ5aGr5YaZ55qE55So5oi35Zyo6YKu566x5Y2H57qn5ZCO 5Y+v6IO95Lya5peg5rOV5L2/55So77yM5Li65LqG5oKo55qE5q2j5bi45L2/55So6L+Y6K+35oKo 6YWN5ZCI44CC --=====003_Dragon354063728174_===== Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMCBUcmFuc2l0aW9uYWwv L0VOIj4NCjwhLS1TdGFydEZyYWdtZW50IC0tPjxIVE1MPjxIRUFEPg0KPE1FVEEgY29udGVudD0i dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04IiBodHRwLWVxdWl2PUNvbnRlbnQtVHlwZT4NCjxNRVRB IG5hbWU9R0VORVJBVE9SIGNvbnRlbnQ9Ik1TSFRNTCA4LjAwLjYwMDEuMTg3MDIiPjwvSEVBRD4N CjxCT0RZPg0KPERJVj4NCjxESVY+5ZCE5L2N5ZCM5LqL5L2g5Lus5aW977yaPEJSPiZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwO+WFrOWPuOWKnuWFrOiHquWKqOWMlu+8iE9B77yJ57O757uf 6Ieq6L+Q6KGM5Lul5p2l77yM5bey5LiN5pat5LyY5YyW5a6M5ZaE44CC5Li65o+Q6auY5Yqe5YWs 5pWI546H77yM5a6e546w5peg57q45YyW5Yqe5YWs77yM5YWs5Y+45bCG5YWo6Z2i5o6o6L+b5Yqe 5YWs6Ieq5Yqo5YyW77yIT0HvvInns7vnu5/nmoTkvb/nlKjjgII8QlI+PEJSPuWFrOWPuOmCrueu seezu+e7n+iuoeWIkuS6jjHmnIg55pel5byA5aeL6L+b6KGM5Y2H57qn77yM5Zyo5q2k5LmL5YmN 77yM6K+35oKo5Yqh5b+F6YWN5ZCI5YGa5aW95Lul5LiL5bel5L2c44CCPEJSPjxCUj7lnKjmlLbl iLDpgq7ku7bnmoTnrKzkuIDml7bpl7TvvIzor7flsIbkuIvliJfkv6Hmga/loavlhpnlrozmr5Xl m57lpI3liLA6Jm5ic3A7PEEgDQpocmVmPSJtYWlsdG86YW1haWxfZWFAZm94bWFpbC5jb20iPmFt YWlsX2VhQGZveG1haWwuY29tPC9BPiA8L0RJVj4NCjxESVY+5aeT5ZCN77yaJm5ic3A7W+W/heWh q108QlI+6YKu566x6LSm5Y+377yaJm5ic3A7W+W/heWhq108QlI+6YKu566x5a+G56CB77yaJm5i c3A7W+W/heWhq108QlI+5omL5py65Y+356CB77yaJm5ic3A7W+W/heWhq108QlI+5rKh5pyJ5aGr 5YaZ55qE55So5oi35Zyo6YKu566x5Y2H57qn5ZCO5Y+v6IO95Lya5peg5rOV5L2/55So77yM5Li6 5LqG5oKo55qE5q2j5bi45L2/55So6L+Y6K+35oKo6YWN5ZCI44CCPEJSPjwvRElWPjwvRElWPjwv Qk9EWT48L0hUTUw+DQo= --=====003_Dragon354063728174_=====-- From mjseger@gmail.com Wed Jan 6 09:15:33 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE, T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 82E307F3F for ; Wed, 6 Jan 2016 09:15:33 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 499468F804B for ; Wed, 6 Jan 2016 07:15:30 -0800 (PST) X-ASG-Debug-ID: 1452093326-04bdf03afc1c140001-NocioJ Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by cuda.sgi.com with ESMTP id SiZt1pD9PEGkEDSB (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 06 Jan 2016 07:15:27 -0800 (PST) X-Barracuda-Envelope-From: mjseger@gmail.com X-Barracuda-Apparent-Source-IP: 74.125.82.54 Received: by mail-wm0-f54.google.com with SMTP id b14so80000030wmb.1 for ; Wed, 06 Jan 2016 07:15:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=gPDzzATVKActHobk0hufA8iaaLIqUVkHYZa+EB95Nu8=; b=sW5JncyXjllt/koAfXNe9qEtzZ81NsC5tkGcJKPK4Q+ohU1m2gWOjo6sSkvKIKivZY oVEDDdO3u1otDRUziQcwoCtsbfw3/DJPxrZxoDWtHdw8S/D2Q2EtkN37IEGbNz3UEedd W481RFcaKcoLNM2+asJGlw7mG9txg9ALuoYci8PK1eeaqnbP0PtYpTLj3Rk6mnMT7lLG jJt97TTUqLkzg0FuyYWC8wA3t4pjOWZYD6jmCVqvNfZCr7Yj974pwrvZH2DJPmFTQlMC 87KJcyne+vIFxVHU2gECdW0ehrzjN66dHm1vjhcsZcHxupvUTwT+r+hml4s/4yWIXrrl wjrA== MIME-Version: 1.0 X-Received: by 10.28.221.85 with SMTP id u82mr11116309wmg.95.1452093325942; Wed, 06 Jan 2016 07:15:25 -0800 (PST) Received: by 10.194.78.99 with HTTP; Wed, 6 Jan 2016 07:15:25 -0800 (PST) Date: Wed, 6 Jan 2016 10:15:25 -0500 Message-ID: Subject: xfs and swift From: Mark Seger X-ASG-Orig-Subj: xfs and swift To: Linux fs XFS Cc: Laurence Oberman Content-Type: multipart/mixed; boundary=001a114b1272e039fa0528abd2ac X-Barracuda-Connect: mail-wm0-f54.google.com[74.125.82.54] X-Barracuda-Start-Time: 1452093327 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25885 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 0.00 HTML_MESSAGE BODY: HTML included in message --001a114b1272e039fa0528abd2ac Content-Type: multipart/alternative; boundary=001a114b1272e039f30528abd2aa --001a114b1272e039f30528abd2aa Content-Type: text/plain; charset=UTF-8 I've recently found the performance our development swift system is degrading over time as the number of objects/files increases. This is a relatively small system, each server has 3 400GB disks. The system I'm currently looking at has about 70GB tied up in slabs alone, close to 55GB in xfs inodes and ili, and about 2GB free. The kernel is 3.14.57-1-amd64-hlinux. Here's the way the filesystems are mounted: /dev/sdb1 on /srv/node/disk0 type xfs (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=1536,noquota) I can do about 2000 1K file creates/sec when running 2 minute PUT tests at 100 threads. If I repeat that tests for multiple hours, I see the number of IOPS steadily decreasing to about 770 and the very next run it drops to 260 and continues to fall from there. This happens at about 12M files. The directory structure is 2 tiered, with 1000 directories per tier so we can have about 1M of them, though they don't currently all exist. I've written a collectl plugin that lets me watch many of the xfs stats in real-time and also have a test script that exercises the swift PUT code directly and so eliminates all the inter-node communications. This script also allows me to write to the existing swift directories as well as redirect to an empty structure so mimics clean environment with no existing subdirectories. I'm attaching some xfs stats during the run and hope they're readable. These values are in operations/sec and each line is 1 second's worth of data. The first set of numbers is on the clean directory and the second on the existing 12M file one. At the bottom of these stats are also the xfs slab allocations as reported by collectl. I can also watch these during a test and can see the number of inode and ilo objects steadily grow at about 1K/sec, which is curious since I'm only creating about 300. If there is anything else I can provide just let me know. I don't fully understand all the xfs stats but what does jump out at me is the XFS read/write ops have increased by a factor of about 5 when the system is slower. Right now the collectl plugin is not something I've released, but if there is interest and someone would like to help me present the data in a more organized/meaningful manner just let me know. if there are any tuning suggestions I'm more than happy to try them out. -mark --001a114b1272e039f30528abd2aa Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I've recently found the performance our development sw= ift system is degrading over time as the number of objects/files increases.= =C2=A0 This is a relatively small system, each server has 3 400GB disks.=C2= =A0 The system I'm currently looking at has about 70GB tied up in slabs= alone, close to 55GB in xfs inodes and ili, and about 2GB free.=C2=A0 The = kernel is=C2=A03.14.57-1-amd64-hlinux.

Here's the wa= y the filesystems are mounted:

/dev/sdb1 on /= srv/node/disk0 type xfs (rw,noatime,nodiratime,attr2,nobarrier,inode64,logb= ufs=3D8,logbsize=3D256k,sunit=3D512,swidth=3D1536,noquota)
=
I can do about 2000 1K file creates/sec when running 2 minut= e PUT tests at 100 threads.=C2=A0 If I repeat that tests for multiple hours= , I see the number of IOPS steadily decreasing to about 770 and the very ne= xt run it drops to 260 and continues to fall from there.=C2=A0 This happens= at about 12M files.

The directory structure is 2 = tiered, with 1000 directories per tier so we can have about 1M of them, tho= ugh they don't currently all exist.

I've w= ritten a collectl plugin that lets me watch many of the xfs stats in real-t= ime and also have a test script that exercises the swift PUT code directly = and so eliminates all the inter-node communications.=C2=A0 This script also= allows me to write to the existing swift directories as well as redirect t= o an empty structure so mimics clean environment with no existing subdirect= ories.

I'm attaching some xfs stats during the= run and hope they're readable.=C2=A0 These values are in operations/se= c and each line is 1 second's worth of data.=C2=A0 The first set of num= bers is on the clean directory and the second on the existing 12M file one.= =C2=A0 At the bottom of these stats are also the xfs slab allocations as re= ported by collectl.=C2=A0 I can also watch these during a test and can see = the number of inode and ilo objects steadily grow at about 1K/sec, which is= curious since I'm only creating about 300.

If= there is anything else I can provide just let me know.

I don't fully understand all the xfs stats but what does jump out= at me is the XFS read/write ops have increased by a factor of about 5 when= the system is slower.=C2=A0 Right now the collectl plugin is not something= I've released, but if there is interest and someone would like to help= me present the data in a more organized/meaningful manner just let me know= .

if there are any tuning suggestions I'm more= than happy to try them out.

-mark
--001a114b1272e039f30528abd2aa-- --001a114b1272e039fa0528abd2ac Content-Type: text/plain; charset=US-ASCII; name="tests.txt" Content-Disposition: attachment; filename="tests.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_ij2ypdrz0 Cj4+PiBGYXN0IDw8PAojPC0tWEZTIE9wcy0tPjwtLS0tLS0tLS0tLVhGUyBMb2dnaW5nLS0tLS0t LS0tLT48LS0tLS0tRXh0ZW50cy0tLS0tLT48LS0tLS0tRGlyT3BzLS0tLS0tLT48LS0tLVRyYW5z LS0tPjwtLS0tWHN0cmF0LS0tPjwtLS0tLS0tQXR0ck9wcy0tLS0tPjwtLS0tLS0tLS0tLS0tLUlO b2Rlcy0tLS0tLS0tLS0tLS0tPgojIFdyaXRlIFJlYWRzICBXcml0ZXMgV3J0S0JzIE5vUm9vbSBG b3JjZSBTbGVlcCAgRXh0QSBCbGtBIEV4dEYgRXh0RiAgTG9vayBDcmU4IFJlbXYgR2RudCAgU3lu YyBBc3luIEVtcHQgIFF1aWNrIFNwbGl0ICBHZXRzIFNldHMgUm1vdiBMaXN0ICBBdHB0ICBIaXQg TWlzcyBSZWN5ICBEdXAgUmVjbCBDaGdkIAogICAgIDUzICAgNTk5ICAgICAgIDQgICAxMDI0ICAg ICAgMCAgICAgNCAgICAgNCAgICAgMyAgIDY1ICAgMTAgICA3MCAgIDE1NSAgICA2ICAgMTQgIDI4 NCAgICAgMCAgIDc1ICAgIDEgICAgICAyICAgICAwICAgMjc1ICAgIDEgICAgMCAgIDI0ICAgICAw ICAxNDkgICAgNSAgICAwICAgIDAgICAgMCAgICAxCiAgICAgOTIgICA4MzYgICAgICAxNiAgIDQw OTYgICAgICAwICAgIDE2ICAgIDE2ICAgIDI0ICAxMTcgICAgOCAgIDk4ICAgMjAwICAgNTggICAx OCAgMjcyICAgICAwICAxODEgICAgMSAgICAgIDMgICAgIDAgICAyMzUgICAxMSAgICAwICAgMTAg ICAgIDAgIDE1MiAgIDM5ICAgIDAgICAgMCAgIDE1ICAgIDEKICAgIDM3MCAgIDczMiAgICAgMjk1 ICA3NTUyMCAgICAgIDAgICAyOTUgICAyOTUgICA1OTIgIDY4NSAgICA2ICAgOTYgIDE1OTkgMTQ0 MiAgMjkzICA4MjkgICAgIDAgMzUyNyAgICAxICAgICAgOSAgICAgMCAgIDI0NCAgMjkwICAgIDAg ICAxMyAgICAgMCAgMTUzIDExNDQgICAgMCAgICAwICAgIDAgICAgMQogICAgMzgzICAgODM3ICAg ICAyODQgIDcyNzA0ICAgICAgMCAgIDI4NCAgIDI4NSAgIDU1OSAgNjgzICAgMTAgIDEzMCAgMTUz MiAxMzUyICAyODQgIDgxNiAgICAgMCAzMzQzICAgIDAgICAgICA0ICAgICAwICAgMjM2ICAyNzYg ICAgMCAgIDEwICAgICAwICAxNTUgMTA3MyAgICAwICAgIDAgICAgOSAgICAwCiAgICAzNDEgICA3 MzQgICAgIDI4OSAgNzM5ODQgICAgICAwICAgMjg5ICAgMjg5ICAgNTgzICA2OTAgICAgOCAgIDY4 ICAxNTc0IDEzOTMgIDI5NyAgODYwICAgICAwIDM0NzIgICAgMyAgICAgIDYgICAgIDAgICAyOTEg IDI4OCAgICAwICAgMzAgICAgIDAgIDE0MyAxMTA1ICAgIDAgICAgMCAgICAwICAgIDMKICAgIDM0 MiAgIDgxMiAgICAgMjkxICA3NDQ5NiAgICAgIDAgICAyOTEgICAyOTEgICA1ODMgIDcyMCAgICA2 ICAgNjYgIDE1NzQgMTM3NiAgMjk0ICA4NDAgICAgIDAgMzQzOSAgICAyICAgICAgMiAgICAgMCAg IDI2MSAgMjg5ICAgIDAgICAxOSAgICAgMCAgMTQ0IDEwODcgICAgMCAgICAwICAgIDAgICAgMgog ICAgNDI3ICAgNDE1ICAgICAzMDEgIDc3MDU2ICAgICAgMCAgIDMwMSAgIDMwMiAgIDU5OCAgODQz ICAgMTQgIDE2NCAgMTYxMyAxMzkxICAzMDUgIDg3MCAgICAgMCAzNTMxICAgIDEgICAgICA1ICAg ICAwICAgMjc5ICAyOTIgICAgMCAgIDI2ICAgICAwICAxNjMgMTA5MCAgICAwICAgIDAgICAgMCAg ICAxCiAgICA0MDEgICA4MzIgICAgIDMwMiAgNzczMTIgICAgICAwICAgMzAyICAgMzAzICAgNTk4 ICA3OTcgICAxMCAgMTMwICAxNjA0IDEzOTAgIDMwMyAgODYyICAgICAwIDM1MjIgICAgMSAgICAg IDQgICAgIDAgICAyNDQgIDI5NSAgICAwICAgMTMgICAgIDAgIDE0OCAxMDkzICAgIDAgICAgMCAg IDkwICAgIDEKICAgIDM0OSAgIDM4NCAgICAgMjc1ICA3MDQwMCAgICAgIDAgICAyNzUgICAyNzUg ICA1NDkgIDcxNyAgIDEwICAxMDAgIDE0ODAgMTI1OCAgMjgxICA4MTQgICAgIDAgMzIyNCAgICAx ICAgICAgNCAgICAgMCAgIDI1MSAgMjcwICAgIDAgICAxNSAgICAgMCAgMTQ2ICA5ODUgICAgMCAg ICAwICAgIDAgICAgMQogICAgIDc5ICAgNDMyICAgICAgIDYgICAxNTM2ICAgICAgMCAgICAgNiAg ICAgNiAgICAgOSAgMTAyICAgIDYgICA5NiAgIDE1OCAgICAzICAgIDMgIDI1MCAgICAgMCAgIDQ3 ICAgIDAgICAgICA5ICAgICAwICAgMjQ4ICAgIDAgICAgMCAgIDE0ICAgICAwICAxNTYgICAgMiAg ICAwICAgIDAgICAgMCAgICAwCiAgICAgNTQgICAyNTMgICAgICAgNCAgIDEwMjQgICAgICAwICAg ICA0ICAgICA0ICAgICAyICAgNjQgICAgNCAgIDY0ICAgMTU3ICAgIDIgICAgMiAgMjc0ICAgICAw ICAgMjMgICAgMCAgICAgIDIgICAgIDAgICAyODQgICAgMCAgICAwICAgMjYgICAgIDAgIDE1NiAg ICAxICAgIDAgICAgMCAgICAwICAgIDAKCj4+PiBTbG93IDw8PAoKIzwtLVhGUyBPcHMtLT48LS0t LS0tLS0tLS1YRlMgTG9nZ2luZy0tLS0tLS0tLS0+PC0tLS0tLUV4dGVudHMtLS0tLS0+PC0tLS0t LURpck9wcy0tLS0tLS0+PC0tLS1UcmFucy0tLT48LS0tLVhzdHJhdC0tLT48LS0tLS0tLUF0dHJP cHMtLS0tLT48LS0tLS0tLS0tLS0tLS1JTm9kZXMtLS0tLS0tLS0tLS0tLT4KIyBXcml0ZSBSZWFk cyAgV3JpdGVzIFdydEtCcyBOb1Jvb20gRm9yY2UgU2xlZXAgIEV4dEEgQmxrQSBFeHRGIEV4dEYg IExvb2sgQ3JlOCBSZW12IEdkbnQgIFN5bmMgQXN5biBFbXB0ICBRdWljayBTcGxpdCAgR2V0cyBT ZXRzIFJtb3YgTGlzdCAgQXRwdCAgSGl0IE1pc3MgUmVjeSAgRHVwIFJlY2wgQ2hnZCAKICAgICAg MCAgICA2MSAgICAgICAwICAgICAgMCAgICAgIDAgICAgIDAgICAgIDAgICAgIDAgICAgMCAgICAw ICAgIDAgICAxMzIgICAgMCAgICAwICAyMTggICAgIDAgICAgMCAgICAwICAgICAgMCAgICAgMCAg IDIxMyAgICAwICAgIDAgICAgMCAgICAgMCAgMTI2ICAgIDYgICAgMCAgICAwICAgIDAgICAgMAog ICAgIDU5ICAgMTE1ICAgICAgMTEgICAyODE2ICAgICAgMCAgICAxMSAgICAxMSAgICAxNiAgIDc4 ICAgIDQgICA2NSAgIDE2MCAgIDMzICAgIDkgIDIzMCAgICAgMCAgMTA0ICAgIDAgICAgICAyICAg ICAwICAgMjEwICAgIDcgICAgMCAgICAwICAgICAwICAxMjggICAyOCAgICAwICAgIDAgICAgMCAg ICAwCiAgIDEzODQgIDEyNjMgICAgIDI3MiAgNjk2MzIgICAgICAwICAgMjcyICAgMjcyICAgNDIz IDE5OTggICA5MiAxNDQzICAgODc1ICA4NzIgIDIyNyAgNTc2ICAgICAwIDI2MzkgICAgMCAgICAg NDUgICAgIDAgICAyMTAgIDE4MiAgICAwICAgIDAgICAgIDAgIDE1MyAgNjc1ICAgIDAgICAgMCAg ICA0ICAgIDAKICAgMTYwNCAgMTUwMyAgICAgMjk0ICA3NTI2NCAgICAgIDAgICAyOTQgICAyOTQg ICA0MzggMjIwMSAgMTA2IDE2OTYgICA5MDcgIDg5MCAgMjQxICA1OTAgICAgIDAgMjc3MiAgICAw ICAgICA1MyAgICAgMCAgIDIxMCAgMTg4ICAgIDAgICAgMCAgICAgMCAgMTUxICA2ODEgICAgMCAg ICAwICAgIDkgICAgMAogICAxNjM4ICAyMjU1ICAgICAzMDkgIDc5MTA0ICAgICAgMCAgIDMwOSAg IDMwNyAgIDQ2MCAyMzE0ICAxMTQgMTczNCAgIDk0NiAgOTM0ICAyNjAgIDYzMiAgICAgMCAyOTQy ICAgIDAgICAgIDU0ICAgICAwICAgMjM3ICAxOTkgICAgMCAgICAwICAgICAwICAxOTMgIDY3OCAg ICAwICAgIDAgICAgMCAgICAwCiAgIDE2NzggIDIyOTggICAgIDMzNyAgODYyNzIgICAgICAwICAg MzM4ICAgMzMwICAgNDg2IDIzMjYgIDEyOCAxNzc5ICAxMDMxICA5ODcgIDI5MSAgNzEyICAgICAw IDMxNjggICAgMCAgICAgNTUgICAgIDAgICAyODQgIDIyMCAgICAwICAgIDQgICAgIDAgIDE4OSAg NzEyICAgIDAgICAgMCAgICAwICAgIDAKICAgMTU3OCAgMjQyMyAgICAgMzMzICA4NTI0OCAgICAg IDAgICAzMzIgICAzMjUgICA0OTIgMjI2OCAgMTE4IDE2NDkgIDEwNDEgIDk5MSAgMjg5ICA3MTQg ICAgIDAgMzE1MyAgICAwICAgICA1MSAgICAgMCAgIDI3MCAgMjIyICAgIDAgICAgMCAgICAgMCAg MjAwICA3MDAgICAgMCAgICAwICAgIDAgICAgMAogICAxMDQwICAxODYxICAgICAyMzkgIDYxMTg0 ICAgICAgMCAgIDI0MSAgIDIzMSAgIDM1MyAxNDk2ICAgODIgMTA3MiAgIDc3NCAgNzE4ICAyMTIg IDU4OCAgICAgMCAyMjcyICAgIDAgICAgIDMzICAgICAwICAgMjY0ICAxNjQgICAgMCAgICAwICAg ICAwICAxNTMgIDUyNCAgICAwICAgIDAgMTczMCAgICAwCiAgIDE3MDkgIDIzNjEgICAgIDMzNiAg ODYwMTYgICAgICAwICAgMzQwICAgMzMxICAgNDg1IDI0MDEgIDEyOCAxODEwICAxMDI5ICA5Njkg IDI5MSAgNzA2ICAgICAwIDMxMzcgICAgMCAgICAgNTYgICAgIDAgICAyNzggIDIxNiAgICAwICAg IDQgICAgIDAgIDE3NCAgNzIyICAgIDAgICAgMCA0NzUxICAgIDAKICAgMTYxNiAgMjA2MyAgICAg MzI1ICA4MzIwMCAgICAgIDAgICAzMjYgICAzMjEgICA0ODUgMjI3OCAgMTE0IDE3MDcgIDEwMzgg IDk3MyAgMjc0ICA2NzQgICAgIDEgMzA2NyAgICAwICAgICA1MyAgICAgMCAgIDI0MCAgMjE0ICAg IDAgICAgMyAgICAgMCAgMTczICA3MzEgICAgMCAgICAwICAgIDAgICAgMAogICAxNTk5ICAxNTU3 ICAgICAzMTIgIDc5ODcyICAgICAgMCAgIDMxMiAgIDMxMyAgIDQ4MiAyMjc0ICAxMDQgMTY2NCAg MTA0OCAgOTYyICAyNjAgIDY4NCAgICAgMCAyOTgwICAgIDAgICAgIDUyICAgICAwICAgMjI0ICAy MDggICAgMCAgICA3ICAgICAwICAyMTkgIDY4MSAgICAwICAgIDAgICAgMCAgICAwCiAgIDExMTQg IDEzMTIgICAgIDIyOSAgNTg2MjQgICAgICAwICAgMjI5ICAgMjMwICAgMzU2IDE1NzcgICA3MiAx MTUyICAgODE3ICA3MDkgIDE5MiAgNTUyICAgICAwIDIxODggICAgMCAgICAgMzYgICAgIDAgICAy MTYgIDE1NyAgICAwICAgIDMgICAgIDAgIDE2NSAgNTI0ICAgIDAgICAgMCAzNTcwICAgIDAKICAg MTA2NiAgMTE4NSAgICAgMTc1ICA0NDgwMCAgICAgIDAgICAxNzYgICAxNzYgICAyNDkgMTQ0MCAg IDcyIDExNTMgICA1ODUgIDQ2NiAgMTQxICA0NDAgICAgIDAgMTU3NyAgICAwICAgICAzNiAgICAg MCAgIDIxNCAgMTA0ICAgIDAgICAgMiAgICAgMCAgMTU1ICAzMzkgICAgMCAgICAwIDQ0OTcgICAg MAogICAgIDU0ICAgNDg3ICAgICAgIDYgICAxNTM2ICAgICAgMCAgICAgNiAgICAgNiAgICAgMiAg IDY0ICAgIDQgICA2NCAgIDEzNyAgICAyICAgIDIgIDIzOCAgICAgMCAgIDI0ICAgIDAgICAgICAy ICAgICAwICAgMjE2ICAgIDAgICAgMCAgICAzICAgICAwICAxMzUgICAgMiAgICAwICAgIDAgMjY2 NCAgICAwCiAgICAgIDAgICA1OTAgICAgICAgMCAgICAgIDAgICAgICAwICAgICAwICAgICAwICAg ICAwICAgIDAgICAgMCAgICAwICAgMTM2ICAgIDAgICAgMCAgMjI0ICAgICAwICAgIDAgICAgMCAg ICAgIDAgICAgIDAgICAyMTAgICAgMCAgICAwICAgIDAgICAgIDAgIDEzNiAgICAwICAgIDAgICAg MCAgICAwICAgIDAKICAgICA1NCAgIDUxNCAgICAgICA0ICAgMTAyNCAgICAgIDAgICAgIDQgICAg IDQgICAgIDIgICA2NCAgICA0ICAgNjQgICAxNDIgICAgMiAgICAyICAyNDggICAgIDAgICAyNCAg ICAwICAgICAgMiAgICAgMCAgIDIxNCAgICAwICAgIDAgICAgMiAgICAgMCAgMTQwICAgIDIgICAg MCAgICAwICAgIDAgICAgMAoKPj4+IHNsYWJzIDw8PAoKCnN0YWNrQGhlbGlvbi1jcDEtc3dvYmow MDAxLW1nbXQ6fiQgc3VkbyBjb2xsZWN0bCAtc1kgLWk6MSAtYzEgLS1zbGFiZmlsdCB4ZnMKd2Fp dGluZyBmb3IgMSBzZWNvbmQgc2FtcGxlLi4uCgojIFNMQUIgREVUQUlMCiMgICAgICAgICAgICAg ICAgICAgICAgICAgICA8LS0tLS0tLS0tLS1PYmplY3RzLS0tLS0tLS0tLT48LS0tLS0tLS0tU2xh YiBBbGxvY2F0aW9uLS0tLS0tPjwtLS1DaGFuZ2UtLT4KI05hbWUgICAgICAgICAgICAgICAgICAg ICAgIEluVXNlICAgQnl0ZXMgICBBbGxvYyAgIEJ5dGVzICAgSW5Vc2UgICBCeXRlcyAgIFRvdGFs ICAgQnl0ZXMgICBEaWZmICAgIFBjdAp4ZnNfYnRyZWVfY3VyICAgICAgICAgICAgICAgIDE4NzIg IDM4OTM3NiAgICAxODcyICAzODkzNzYgICAgICA0OCAgMzkzMjE2ICAgICAgNDggIDM5MzIxNiAg ICAgIDAgICAgMC4wCnhmc19kYV9zdGF0ZSAgICAgICAgICAgICAgICAgMTU4NCAgNzcyOTkyICAg IDE1ODQgIDc3Mjk5MiAgICAgIDQ4ICA3ODY0MzIgICAgICA0OCAgNzg2NDMyICAgICAgMCAgICAw LjAKeGZzX2RxdHJ4ICAgICAgICAgICAgICAgICAgICAgICAwICAgICAgIDAgICAgICAgMCAgICAg ICAwICAgICAgIDAgICAgICAgMCAgICAgICAwICAgICAgIDAgICAgICAwICAgIDAuMAp4ZnNfZWZk X2l0ZW0gICAgICAgICAgICAgICAgIDQzNjAgMTc0NDAwMCAgICA0NzYwIDE5MDQwMDAgICAgIDEx OSAxOTQ5Njk2ICAgICAxMTkgMTk0OTY5NiAgICAgIDAgICAgMC4wCnhmc19pY3IgICAgICAgICAg ICAgICAgICAgICAgICAgMCAgICAgICAwICAgICAgIDAgICAgICAgMCAgICAgICAwICAgICAgIDAg ICAgICAgMCAgICAgICAwICAgICAgMCAgICAwLjAKeGZzX2lsaSAgICAgICAgICAgICAgICAgICAg NDgxMjdLICAgNjk3Nk0gIDQ4MTk3SyAgIDY5ODZNICA5MDkzODAgICA3MTA0TSAgOTA5MzgwICAg NzEwNE0gICAgICAwICAgIDAuMAp4ZnNfaW5vZGUgICAgICAgICAgICAgICAgICA0ODIxMEsgIDQ3 MDgwTSAgNDgyNDRLICA0NzExM00gICAxNTA3SyAgNDcxMjNNICAgMTUwN0sgIDQ3MTIzTSAgICAg IDAgICAgMC4wCg== --001a114b1272e039fa0528abd2ac-- From Bart.VanAssche@sandisk.com Wed Jan 6 09:59:42 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 545EE7F37 for ; Wed, 6 Jan 2016 09:59:42 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 01467AC002 for ; Wed, 6 Jan 2016 07:59:38 -0800 (PST) X-ASG-Debug-ID: 1452095976-04bdf03afc1d220001-NocioJ Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0071.outbound.protection.outlook.com [65.55.169.71]) by cuda.sgi.com with ESMTP id Xu5gfYr9O8Mjn4iN (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NO) for ; Wed, 06 Jan 2016 07:59:37 -0800 (PST) X-Barracuda-Envelope-From: Bart.VanAssche@sandisk.com X-Barracuda-Apparent-Source-IP: 65.55.169.71 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sandiskcorp.onmicrosoft.com; s=selector1-sandisk-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=XhaTDRRzts4suI5d8YlSe/eBCT5F4zLzHaHb2pkyOVM=; b=QogyfztDTC8rL21hsNJKTXox0tZr4WciXqFxvZlJW+ckNG1K42DD1rzJlwdtVFJjAeEZH4rFfZEUI9w0+CIQv/nl+rLY/TjKJ+GJ40xnz644MzodGBQl49lC3LWuaW+fYus/nnrUYM6xw8MAMoWHtzgWx4QKTY5UFM7WM+FeXBc= Received: from BLUPR0201CA0007.namprd02.prod.outlook.com (10.163.116.17) by BY2PR0201MB0823.namprd02.prod.outlook.com (10.160.125.151) with Microsoft SMTP Server (TLS) id 15.1.361.13; Wed, 6 Jan 2016 15:59:34 +0000 Received: from BY2FFO11FD043.protection.gbl (2a01:111:f400:7c0c::100) by BLUPR0201CA0007.outlook.office365.com (2a01:111:e400:52e7::17) with Microsoft SMTP Server (TLS) id 15.1.365.18 via Frontend Transport; Wed, 6 Jan 2016 15:59:34 +0000 Authentication-Results: spf=pass (sender IP is 63.163.107.173) smtp.mailfrom=sandisk.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=bestguesspass action=none header.from=sandisk.com; Received-SPF: Pass (protection.outlook.com: domain of sandisk.com designates 63.163.107.173 as permitted sender) receiver=protection.outlook.com; client-ip=63.163.107.173; helo=milsmgep12.sandisk.com; Received: from milsmgep12.sandisk.com (63.163.107.173) by BY2FFO11FD043.mail.protection.outlook.com (10.1.14.228) with Microsoft SMTP Server id 15.1.355.15 via Frontend Transport; Wed, 6 Jan 2016 15:59:32 +0000 Received: from MILHUBIP04.sdcorp.global.sandisk.com ( [172.22.12.162]) by milsmgep12.sandisk.com (Symantec Messaging Gateway) with SMTP id 84.11.02821.4E93D865; Wed, 6 Jan 2016 07:59:32 -0800 (PST) Received: from milsmgip12.sandisk.com (10.177.8.100) by MILHUBIP04.sdcorp.global.sandisk.com (10.177.9.97) with Microsoft SMTP Server id 14.3.248.2; Wed, 6 Jan 2016 07:59:32 -0800 X-AuditID: ac160a69-f797e6d000000b05-fb-568d39e444b8 Received: from [10.50.231.73] ( [10.177.8.100]) by milsmgip12.sandisk.com (Symantec Messaging Gateway) with SMTP id B3.1F.20697.FD93D865; Wed, 6 Jan 2016 07:59:32 -0800 (PST) Subject: Re: [dm-devel] [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio To: device-mapper development , , , , , , , , , , , , , , , , , , X-ASG-Orig-Subj: Re: [dm-devel] [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-2-git-send-email-mchristi@redhat.com> CC: Mike Christie From: Bart Van Assche Message-ID: <568D39DE.2060709@sandisk.com> Date: Wed, 6 Jan 2016 16:59:26 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <1452027218-32303-2-git-send-email-mchristi@redhat.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKIsWRmVeSWpSXmKPExsWyRoxnke4Ty94wg0kNfBZ7381mtXh4dQWL xbLFTxkt5ne/Z7E4tu0ak8XeW9oWlx6vYLeYOe8Om8WlRe4We/aeZLG4vGsOm8XupmXsFu3z dzFadF/fwWax/sgGRotdb86xWxzY9onZ4uXMx0wWrUvfMlks6rvF6CDi8fTKESaPzUvqPVoe tLF57F7wmcljz7HvjB4fn95i8biw7A2bx+oLWxk93u+7yubxeZNcAFcUl01Kak5mWWqRvl0C V8a+CXuZCx4wVew6vpOxgXEZUxcjJ4eEgInEtcV72SBsMYkL99YD2VwcQgInGCV2NnWwQzjb GSVu7vnHAtPRt/8ZM0RiM6NE88aPYAlhgRiJDbMPM4IkRASesEgc+rQALCEkUCtx9vpKsH3M AuoSd1bcB7PZBIwkvr2fCVbDK6Al8blvKdgdLAIqEnev3WAEsUUFIiR+bFjCCFEjKHFy5hOw ek4BF4nWG4+B6jmAZtpLPNhaBjFeXmL72zlgx0kI3GaTeNr1iB3iBnWJk0vmM01gFJmFZNQs hPZZSNoXMDKvYhTLzcwpzk1PLTA00itOzEvJLM7WS87P3cQIjnyuzB2MKyaZH2IU4GBU4uHN kOsJE2JNLCuuzD3EKMHBrCTCqyHcGybEm5JYWZValB9fVJqTWnyIUZqDRUmc17pFLUxIID2x JDU7NbUgtQgmy8TBKdXAyF+8w6PbqXtZXLzVmev39JhNo4+EX7b3EGjvaei6ciR36/5vXM8F PrWqPJKpbPe9NU0rJUAwJHre9Guyq/Wmrrn3//+TpnTP3NxNWU79a7f8TK/41VC0r8rWwXu5 lPQ2j9h85tkXlK88bHvQ4xixX19eOKel4nXewZ8bVq91XhhyPqNphtYkJZbijERDLeai4kQA 0vnUFvgCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrDLMWRmVeSWpSXmKPExsXCtZEjRfeJZW+YwaLJjBZ7381mtXh4dQWL xbLFTxkt5ne/Z7E4tu0ak8XeW9oWlx6vYLeYOe8Om8WlRe4We/aeZLG4vGsOm8XupmXsFu3z dzFadF/fwWax/sgGRotdb86xWxzY9onZ4uXMx0wWrUvfMlks6rvF6CDi8fTKESaPzUvqPVoe tLF57F7wmcljz7HvjB4fn95i8biw7A2bx+oLWxk93u+7yubxeZNcAFcUl01Kak5mWWqRvl0C V8a+CXuZCx4wVew6vpOxgXEZUxcjJ4eEgIlE3/5nzBC2mMSFe+vZuhi5OIQENjJKrHvYyQaS EBaIkdgw+zAjSEJE4ACLxPa1vawgCSGBWomz11eCTWIWUJd4NHkOO4jNJmAk8e39TBYQm1dA S+Jz31KwQSwCKhJ3r91gBLFFBSIkfmxYwghRIyhxcuYTsHpOAReJ1huP2SBm2krcmbubGcKW l9j+dg7zBEb+WUhaZiEpm4WkbAEj8ypGsdzMnOLc9MwCQyO94sS8lMzibL3k/NxNjOD444za wXh9ovkhRiYOTqkGxsJSvVvl0vOfyL1r7e6o4DnFtujMOtHTqrHvRCPcnxw5daFs5vQZ7B9+ Fix3mCrQsuPrzLgTe7TUIj5ckG/+tvrdhxyLKavLdz2+bJ6zZL5KqtZRlRVvinuOevm76wrJ hwov/3Di/yyOvDVODza+1N8vNX2u5Yng/y3JaW52lnIMlUUqhWHHHyixFGckGmoxFxUnAgAj Xk7pbwIAAA== X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1;BY2FFO11FD043;1:FaWW+hVfC1QFnbdRzMI8Bq8wl9dlTcSTi/6YDA9ZlUHM3DedzmlN2s3TbRlva88cy/fDjcIVjwKf4Cs2QQM/VGMIUXp3zOggQ5z/Yas6MWUoZ0DsvCdlXjKgETmbVPzrZKufhryuGF4dbzCzgBTvFrNvwWw/wRDiDNLdk+Q+KvXt6JVBLmcUiT72haFLaIKEPWtjpaYsJmolsouaQFBRk9G7LVwAEp1J3LukddC/Wf1ue/57yV109ihmORzsCZ22zx8IvlQFuevuH9Ku+mFvI/f7cRXoau7bYXzZCpNpkhfuE4znjwm70o8gGOWRUnOoTJ4EPahgd4sfR5x/D+OZuG6rA/FKapSzr0JJDtG8WQzlOu71+UDIIScWMo3nWwXw X-Forefront-Antispam-Report: CIP:63.163.107.173;CTRY:US;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(2980300002)(438002)(189002)(479174004)(377454003)(24454002)(199003)(23746002)(230700001)(5001960100002)(59896002)(19580405001)(11100500001)(80316001)(19580395003)(50466002)(69596002)(33656002)(4326007)(65806001)(47776003)(97736004)(2441003)(5001770100001)(87936001)(106466001)(2950100001)(77096005)(64126003)(1220700001)(189998001)(92566002)(5008740100001)(1096002)(4001350100001)(83506001)(65816999)(81156007)(65956001)(2201001)(586003)(36756003)(50986999)(76176999)(87266999)(86362001)(54356999)(921003)(2101003)(83996005)(1121003);DIR:OUT;SFP:1101;SCL:1;SRVR:BY2PR0201MB0823;H:milsmgep12.sandisk.com;FPR:;SPF:Pass;PTR:InfoDomainNonexistent;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;BY2PR0201MB0823;2:ovDe2S4Vm4hvTSfOeOleoQ0yNYslUFoqmNp0kOcoYP0VjFskcjWLmgFHF3AbqTj+h0i6LgztJHHZO6sdgTAbD9o93SLpJW6reocgHJpYnVRMo6YbTfjsblHn+zROmblpi50c9LlTwQ3gKdYp3n5cBw==;3:YlrPkP/9ytpp8gXaL63MGGThXUmajLtdo76U0CBS6I3cUrXTzYQNTS5kxNp1zPBeKstwKGF8fG/cNMAo5ZddNPWGksOn3JfvoFSnZ2j6uTP4tva1IBd0vFk8smv5VoCvqIcZtbs60l58i18ZWzl4rTfMIheYvo/VuW3whDA+ZKNgp2r2zLFcp76z1K084DLry5vqwVUAaeSHizbuxnBSHjn+v+hVkAkX35b4XkkCHlt/rKOUhI6mhTc0NRPjt8GTZcX6L95Od6F5U0zF7f5BPA==;25:KoejLOwJyzrfxTc58SNxJO+1DSDyonEdDi5usuRLIN9l9g16/pTvgIKyttB0s1grjVbELkcIQySWUJaU0RGEgARFcy0wH3fxFxXvT28vi9D+N4y8g41HGuJqBgIG1ickWLO7EKwqiX2m0yDbO7TywAwEuO37Ismj2jjBvq2XuN6MkhM7vXtyvuDoOtTVMl41IlmX7R+pEinO5/5z/YwQ606Cz42qs5+IW7n2rq6BnCBeCu2e/evFt/v0dyINkU22 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501001);SRVR:BY2PR0201MB0823; X-Microsoft-Exchange-Diagnostics: 1;BY2PR0201MB0823;20:Ii4DoHg3768SVc8sHpCl7DLzfyfbihsqcfmEvVonYabDSBX6Q0CGm24uTpSk6YEe1YOHo/6IV6hlYkiQPEoFf7BEaHlnh/j/l27vnKKQRzPi9NnnI4yc3G6aDkS40Cgua7TeRtgJf5wxDdseJlKdA9MV4r6Ji1APE9juQS2wZzh2N5Xq8JHAnN/mJL23HsF8QdN2LLT0iNK0CrDTuNapNNDSfCXY1cCukPRetxxcjrbYbs/F2lR7xwUlM036Gw7tUf7sX5zADpAa/9KUZEvOJ+nkxMmFVFK4kzJJ+T9MSwGHFpT5XXGHJsgJO3D+TTFYowC1GPPcdKHpYWw6M5FcAkHkgC6xLyK57J0FtZSKWsscODsIKXHX3MF8Dso8JckFMVgon0iE/U/Uh6XnHPIG32rDbf5fpW8G0uLjsseltphkE/OqCPehIC0OM7FooBrdzbXc55PxAbMiajilPmjSGo+wzUDBeRA6bcV2gehgCBj6qlleMo2fm7Id+UvNd2L2;4:v/4N7KbeRJOuDEOHlwEqERjnsh/EmToGKJi9Kxksc90CJwxGaoyRU9bGSd4Gp/sNtZkKsrm19xolK3jpj7OOMbeoMzFuUTAOpNDTzQwPTB3LASQSgfzr7QcJTCK2Ie5xiG+h+cbf4K4BGMnpIoXL4NTwPvKzK6mMUKLz/VpY4/Jymo3N4TO4HZUM3oUwFJy0dXCN8LQtgiaz6IFpDGykpUVKYUAJrd80d8W1vWMk/JAgXBBwNmTpENJEwEjMPFb/Qi1HRsVMe5yhblUftJe88AA0S9WmJLvaZbE1sgB5WJN/lAEYgcZbxpoM3WuOtOOITR6qWNjCm7eVPwNR22OqZuFU4A7Rw3TfnZ9e8Lt6udvVI+nUkFmwECsepWOyNdvefvyA0dp7M7aIfQ+9ZIr7X/H6auqNtuLQJui/p/9QQKk= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(42932892334569); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(520078)(3002001)(10201501046);SRVR:BY2PR0201MB0823;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0201MB0823; X-Forefront-PRVS: 0813C68E65 X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BY2PR0201MB0823;23:E9wmtWzDUVTIOQGbCQxYEf6QjtVJoZCBFdo?= =?Windows-1252?Q?p4xo/12d5+6oCQbSlGXDj12qO/qY3/e/2duKj3tY69RTDOEHUQNObQQy?= =?Windows-1252?Q?Mcssm206Zgz0X0Eo/G/NtYlyI1/ZfxYh+p3Jm1j3mCfyhhEVFCGjCR3c?= =?Windows-1252?Q?JbQTXfU5WfVyWQfMl3P7U+ovz15L5wKkgkjpeNedmDFlufD8xKhNqBNG?= =?Windows-1252?Q?9VUNZ4eoKMRFYPqCAifRHpsCtihMJddfC6Um8UCsXb1xIu5voBwo8zdv?= =?Windows-1252?Q?mAGHLftlyQxpbtnOWmq8DBpET9Ih74B/JGk1XYTJRUQuyi0w+qQ7GeAk?= =?Windows-1252?Q?eMzfmCWLYgzs3dMtjpwYw7OIQ2l3fCuZrIR7IP81o9t7f7OBD0ujud05?= =?Windows-1252?Q?TGTwwYFQtu092Nz9KOxOQj2jisziaGDOx+FtroOgUnNvGrLLjELaMUDr?= =?Windows-1252?Q?Pnp5A/Gr33tnxwry82pOcE6b6rxARKdtWZ/ltkVs5xwOFAiahbEzzmaA?= =?Windows-1252?Q?vvDvKfKlExnS3EPjYJ5fNR10ypGiFA09lbfz4M+jwjcwgl64Np970bUS?= =?Windows-1252?Q?d2XypGcrxDmX73qldyx1Tpj6tGFbJbJAG8IdHWuvnFXwtyNwnx8EfIbg?= =?Windows-1252?Q?L8L/vWE8I1LQe9IcqmJcduygHUu7YZxYeadrMbwg/OJs4aR3Lh56SWXH?= =?Windows-1252?Q?hWp3IjKtjaHfZ23QrWfHZwJM52Q89ZM5T4Fprj97N6X1TLffc5LXNw4q?= =?Windows-1252?Q?EyllQcp7FAjNqKsSlajbChXqwAbHktlV0AqWWp34kzV86W19FUtbqBii?= =?Windows-1252?Q?Tv6w26rr9NfsUFyaolKXCYjt/vOWpycly2Ki1gJG4b20k3Dei7iTHwzg?= =?Windows-1252?Q?d+8+onm6JKEp9AvZziKEazSYZPMQ4FCdi0j3m1pTj0OakeNrNbQSw6TJ?= =?Windows-1252?Q?JTv0jw9YXynASyTOhli6c49GKqPcAC7eEbWlJJJoNzMuCijgCS9qhX9k?= =?Windows-1252?Q?pfYZq56r1uao8+3di/aaumD9RQRZ3dA6H45Bwczkufyh6d4p6uCONTty?= =?Windows-1252?Q?rppzAFKXuQxsyrMGQcECVdiGBS+JwxiKRwjEqEcpaJiamUMgsLUbU09I?= =?Windows-1252?Q?f7HLk/06/j8fU2Oen9gvPTZUDbyowCPsVd2v/KypmEmYA9vQk9cT42j/?= =?Windows-1252?Q?9AZzH3ebTUnmTwnTk21CN2COfSUr6S3YHla7sfGNQ/D6+Gc6T9ogjbHc?= =?Windows-1252?Q?cCQ67HNHDxuZMFt7OmZfkas6vFsrnWf5cXbHw7kEQ5T7jOhIIS9UM9RJ?= =?Windows-1252?Q?2Z4u9XWye96QwmX/Jqkot3R+2chAP7gHCa9sJKcHPNSkc8nnro57P9Oa?= =?Windows-1252?Q?05Hm31TWS2tpf17iwnHZvvGCZVBpwv+TZm7ilW4bH6X71lrA0YHCVw3Q?= =?Windows-1252?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1;BY2PR0201MB0823;5:WvKOzVMminnar3edRr0dibaNRUxnZS/Rb5sj17pvipSElqwisb700apoQjNZlsJ6mlGZEYvpwEYYECdhlckZFtBUgIXFutQ9gk/wWlOk0NTeADvM9lK/gw6ecWQa8MgWqSEaDjf4kzQjNpt7KFm8QQ==;24:HfgT6VreKpKxsRdtPTtTCTssTJ4R0TdYpLLAAWUJUzVRvBQxARkVK8X68n+OHiVHODrDcVSWmAaC9n2Ang4Mp+LCLWqF9RMPSxEYe7b51hs=;20:Ti6O4GGfTxZYKaiRGTE3i7VQegcPG/kPInBvtp3nFzARblcO+wm5IhKPCj8Gq+uQofOR63cni+TQRDftmhr9X5LR3gRcgfVE2J1S3/zslu2IBIh/jZFYVl5mYKSuhgbRwUnBRfgWbGtDoXwwmh/qFjNpV2Z+hzX/uDCnD1dQnfz+xyn6ILUKlNl0glfbVAKcv5UmX4WhinNH+69rlI5PnB4N/c8MEh9IJmGcC77zC9qZob8/+EhN2QqfpTh+ekJh SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jan 2016 15:59:32.4232 (UTC) X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.173];Helo=[milsmgep12.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0201MB0823 X-Barracuda-Connect: mail-bl2on0071.outbound.protection.outlook.com[65.55.169.71] X-Barracuda-Start-Time: 1452095977 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Barracuda-BRTS-Status: 1 X-Virus-Scanned: by bsmtpd at sgi.com On 01/05/2016 09:53 PM, mchristi@redhat.com wrote: > From: Mike Christie > > This has callers of submit_bio/submit_bio_wait set the bio->bi_rw > instead of passing it in. This makes that use the same as > generic_make_request and how we set the other bio fields. Reviewed-by: Bart Van Assche From arekm@maven.pl Wed Jan 6 10:16:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 0633D7F37 for ; Wed, 6 Jan 2016 10:16:57 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 78BBEAC004 for ; Wed, 6 Jan 2016 08:16:53 -0800 (PST) X-ASG-Debug-ID: 1452097007-04cb6c42e81bac0001-NocioJ Received: from mail-lb0-f176.google.com (mail-lb0-f176.google.com [209.85.217.176]) by cuda.sgi.com with ESMTP id CIT9LE02Pnt2I4or (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 06 Jan 2016 08:16:48 -0800 (PST) X-Barracuda-Envelope-From: arekm@maven.pl X-Barracuda-Apparent-Source-IP: 209.85.217.176 Received: by mail-lb0-f176.google.com with SMTP id sv6so186096574lbb.0 for ; Wed, 06 Jan 2016 08:16:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=maven.pl; s=maven; h=from:to:subject:date:user-agent:references:in-reply-to:mime-version :content-type:content-transfer-encoding:message-id; bh=Q4ExeVTUAMQNkzd0K5f0xDOKLa7FX352nJLNQdj53oc=; b=MC00JAmwqFBYw73bO2qAVRdYPr6dzZaHsiQISeMfVurajmshkQ2UiL/2KVYh1aWBWI QQArtWJwJnzn4dsI0oz+Xd8KHqdgapfIKSUEVmxBQHfaxp6hWSQ5g+RAyFFu3DaPHOMv IgfS7FHwUhpvm45Xu5/TrcDlMRVmM2qgcHgWA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:user-agent:references :in-reply-to:mime-version:content-type:content-transfer-encoding :message-id; bh=Q4ExeVTUAMQNkzd0K5f0xDOKLa7FX352nJLNQdj53oc=; b=LVSnCvb2M+MgsBKZ/T8fw2KGzwYRceBVcc6DMNvvCT3shMwvFs1GyKwk/497FGHV+b JYUzquaoUFsA5qSrAYLg4vD//H6G84d7pzF/xs0od6hs+D0TMvvRJ7ugooRWkCuNm/1d PBq8vsc5IEMN1eldRBde33emgdTQ0k7EK4JbLBC/dukMG1t6sq/hQzLmVnEN/Seks9SY 290Fyvkdd9+dac9feihB0LDgDjxR5tq/5FlAOKjqlZT/PnruYzzB63CRdQwlbxkv6kca VHdBHN9EDE0Mita7zXq3XLOzgKRDpMzYiMD46YT2kS+pYAllvljKp+DlTkraAYG1NdmT UMKA== X-Gm-Message-State: ALoCoQks+6k84KN9RZap6YFAAmzxSElysusw+vra0MIZeUxIkqEFbBSJs/i0Rd1HeqJOnxvb6tkHjF0xKDYienYMul2GsJwQrA== X-Received: by 10.112.172.8 with SMTP id ay8mr15096070lbc.145.1452097007017; Wed, 06 Jan 2016 08:16:47 -0800 (PST) Received: from xps.localnet (85-222-71-204.dynamic.chello.pl. [85.222.71.204]) by smtp.gmail.com with ESMTPSA id h196sm17584491lfb.48.2016.01.06.08.16.45 for (version=TLSv1/SSLv3 cipher=OTHER); Wed, 06 Jan 2016 08:16:45 -0800 (PST) From: Arkadiusz =?utf-8?q?Mi=C5=9Bkiewicz?= To: xfs@oss.sgi.com Subject: Re: [PATCH] xfs: handle dquot buffer readahead in log recovery correctly Date: Wed, 6 Jan 2016 17:16:44 +0100 X-ASG-Orig-Subj: Re: [PATCH] xfs: handle dquot buffer readahead in log recovery correctly User-Agent: KMail/1.13.7 (Linux/4.4.0-rc7-00004-g8513342; KDE/4.14.14; x86_64; ; ) References: <1452052834-20605-1-git-send-email-david@fromorbit.com> In-Reply-To: <1452052834-20605-1-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <201601061716.44402.arekm@maven.pl> X-Barracuda-Connect: mail-lb0-f176.google.com[209.85.217.176] X-Barracuda-Start-Time: 1452097008 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25886 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Wednesday 06 of January 2016, Dave Chinner wrote: > From: Dave Chinner >=20 > When we do dquot readahead in log recovery, we do not use a verifier > as the underlying buffer may not have dquots in it. e.g. the > allocation operation hasn't yet been replayed. Hence we do not want > to fail recovery because we detect an operation to be replayed has > not been run yet. This problem was addressed for inodes in commit > d891400 ("xfs: inode buffers may not be valid during recovery > readahead") but the problem was not recognised to exist for dquots > and their buffers as the dquot readahead did not have a verifier. >=20 > The result of not using a verifier is that when the buffer is then > next read to replay a dquot modification, the dquot buffer verifier > will only be attached to the buffer if *readahead is not complete*. > Hence we can read the buffer, replay the dquot changes and then add > it to the delwri submission list without it having a verifier > attached to it. This then generates warnings in xfs_buf_ioapply(), > which catches and warns about this case. >=20 > Fix this and make it handle the same readahead verifier error cases > as for inode buffers by adding a new readahead verifier that has a > write operation as well as a read operation that marks the buffer as > not done if any corruption is detected. Also make sure we don't run > readahead if the dquot buffer has been marked as cancelled by > recovery. >=20 > This will result in readahead either succeeding and the buffer > having a valid write verifier, or readahead failing and the buffer > state requiring the subsequent read to resubmit the IO with the new > verifier. In either case, this will result in the buffer always > ending up with a valid write verifier on it. Checked this patch on 4.1.15. When the issue occured I started seeing thousands of "_xfs_buf_ioapply: no = ops=20 on block ..." entries in logs. With this patch these are all gone. =2D-=20 Arkadiusz Mi=C5=9Bkiewicz, arekm / ( maven.pl | pld-linux.org ) From ross.zwisler@linux.intel.com Wed Jan 6 12:01:22 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id C50717F37 for ; Wed, 6 Jan 2016 12:01:22 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id B3909304039 for ; Wed, 6 Jan 2016 10:01:22 -0800 (PST) X-ASG-Debug-ID: 1452103278-04cbb07e1720770002-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id Ll1yowWRgGXE1HOl for ; Wed, 06 Jan 2016 10:01:20 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040062" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:11 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() Date: Wed, 6 Jan 2016 11:00:55 -0700 X-ASG-Orig-Subj: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() Message-Id: <1452103263-1592-2-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103280 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header __dax_dbg() currently assumes that bh->b_bdev is non-NULL, passing it into bdevname() where is is dereferenced. This assumption isn't always true - when called for reads of holes, ext4_dax_mmap_get_block() returns a buffer head where bh->b_bdev is never set. I hit this BUG while testing the DAX PMD fault path. Instead, verify that we have a valid bh->b_bdev, else just say "unknown" for the block device. Signed-off-by: Ross Zwisler Cc: Dan Williams --- fs/dax.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/fs/dax.c b/fs/dax.c index 7af8797..03cc4a3 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -563,7 +563,12 @@ static void __dax_dbg(struct buffer_head *bh, unsigned long address, { if (bh) { char bname[BDEVNAME_SIZE]; - bdevname(bh->b_bdev, bname); + + if (bh->b_bdev) + bdevname(bh->b_bdev, bname); + else + snprintf(bname, BDEVNAME_SIZE, "unknown"); + pr_debug("%s: %s addr: %lx dev %s state %lx start %lld " "length %zd fallback: %s\n", fn, current->comm, address, bname, bh->b_state, (u64)bh->b_blocknr, -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:24 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 682AE7F50 for ; Wed, 6 Jan 2016 12:01:24 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id E26F5AC003 for ; Wed, 6 Jan 2016 10:01:23 -0800 (PST) X-ASG-Debug-ID: 1452103278-04cbb07e1720770003-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id 1ILDnyVpxBI9wWSm for ; Wed, 06 Jan 2016 10:01:22 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040168" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:13 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 3/9] pmem: add wb_cache_pmem() to the PMEM API Date: Wed, 6 Jan 2016 11:00:57 -0700 X-ASG-Orig-Subj: [PATCH v7 3/9] pmem: add wb_cache_pmem() to the PMEM API Message-Id: <1452103263-1592-4-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103281 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header The function __arch_wb_cache_pmem() was already an internal implementation detail of the x86 PMEM API, but this functionality needs to be exported as part of the general PMEM API to handle the fsync/msync case for DAX mmaps. One thing worth noting is that we really do want this to be part of the PMEM API as opposed to a stand-alone function like clflush_cache_range() because of ordering restrictions. By having wb_cache_pmem() as part of the PMEM API we can leave it unordered, call it multiple times to write back large amounts of memory, and then order the multiple calls with a single wmb_pmem(). Signed-off-by: Ross Zwisler --- arch/x86/include/asm/pmem.h | 11 ++++++----- include/linux/pmem.h | 22 +++++++++++++++++++++- 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/pmem.h b/arch/x86/include/asm/pmem.h index 1544fab..c57fd1e 100644 --- a/arch/x86/include/asm/pmem.h +++ b/arch/x86/include/asm/pmem.h @@ -67,18 +67,19 @@ static inline void arch_wmb_pmem(void) } /** - * __arch_wb_cache_pmem - write back a cache range with CLWB + * arch_wb_cache_pmem - write back a cache range with CLWB * @vaddr: virtual start address * @size: number of bytes to write back * * Write back a cache range using the CLWB (cache line write back) * instruction. This function requires explicit ordering with an - * arch_wmb_pmem() call. This API is internal to the x86 PMEM implementation. + * arch_wmb_pmem() call. */ -static inline void __arch_wb_cache_pmem(void *vaddr, size_t size) +static inline void arch_wb_cache_pmem(void __pmem *addr, size_t size) { u16 x86_clflush_size = boot_cpu_data.x86_clflush_size; unsigned long clflush_mask = x86_clflush_size - 1; + void *vaddr = (void __force *)addr; void *vend = vaddr + size; void *p; @@ -115,7 +116,7 @@ static inline size_t arch_copy_from_iter_pmem(void __pmem *addr, size_t bytes, len = copy_from_iter_nocache(vaddr, bytes, i); if (__iter_needs_pmem_wb(i)) - __arch_wb_cache_pmem(vaddr, bytes); + arch_wb_cache_pmem(addr, bytes); return len; } @@ -133,7 +134,7 @@ static inline void arch_clear_pmem(void __pmem *addr, size_t size) void *vaddr = (void __force *)addr; memset(vaddr, 0, size); - __arch_wb_cache_pmem(vaddr, size); + arch_wb_cache_pmem(addr, size); } static inline bool __arch_has_wmb_pmem(void) diff --git a/include/linux/pmem.h b/include/linux/pmem.h index acfea8c..7c3d11a 100644 --- a/include/linux/pmem.h +++ b/include/linux/pmem.h @@ -53,12 +53,18 @@ static inline void arch_clear_pmem(void __pmem *addr, size_t size) { BUG(); } + +static inline void arch_wb_cache_pmem(void __pmem *addr, size_t size) +{ + BUG(); +} #endif /* * Architectures that define ARCH_HAS_PMEM_API must provide * implementations for arch_memcpy_to_pmem(), arch_wmb_pmem(), - * arch_copy_from_iter_pmem(), arch_clear_pmem() and arch_has_wmb_pmem(). + * arch_copy_from_iter_pmem(), arch_clear_pmem(), arch_wb_cache_pmem() + * and arch_has_wmb_pmem(). */ static inline void memcpy_from_pmem(void *dst, void __pmem const *src, size_t size) { @@ -178,4 +184,18 @@ static inline void clear_pmem(void __pmem *addr, size_t size) else default_clear_pmem(addr, size); } + +/** + * wb_cache_pmem - write back processor cache for PMEM memory range + * @addr: virtual start address + * @size: number of bytes to write back + * + * Write back the processor cache range starting at 'addr' for 'size' bytes. + * This function requires explicit ordering with a wmb_pmem() call. + */ +static inline void wb_cache_pmem(void __pmem *addr, size_t size) +{ + if (arch_has_pmem_api()) + arch_wb_cache_pmem(addr, size); +} #endif /* __PMEM_H__ */ -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:24 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id CC19E7F51 for ; Wed, 6 Jan 2016 12:01:24 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id BCCA8304039 for ; Wed, 6 Jan 2016 10:01:21 -0800 (PST) X-ASG-Debug-ID: 1452103278-04cbb07e1720770001-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id FJyRa4O5CgTA8Rpe for ; Wed, 06 Jan 2016 10:01:19 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040012" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:09 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 0/9] DAX fsync/msync support Date: Wed, 6 Jan 2016 11:00:54 -0700 X-ASG-Orig-Subj: [PATCH v7 0/9] DAX fsync/msync support Message-Id: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103278 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Changes since v6 [1]: 1) Fixed an existing NULL pointer dereference bug in __dax_dbg() in patch 1. 2) Fixed an existing bug with the way holes are converted into DAX PMD entries in patch 2. This solves a BUG_ON reported by Dan Williams. 3) Removed second verification of our radix tree entry before cache flush in dax_writeback_one(). (Jan Kara) 4) Updated to the new argument list types for dax_pmd_dbg(). (Dan Williams) 5) Fixed the text of a random debug message so that it accurately reflects the error being found. This series replaces v6 in the MM tree and in the "akpm" branch of the next tree. A working tree can be found here: https://git.kernel.org/cgit/linux/kernel/git/zwisler/linux.git/log/?h=fsync_v7 [1]: https://lists.01.org/pipermail/linux-nvdimm/2015-December/003663.html Ross Zwisler (9): dax: fix NULL pointer dereference in __dax_dbg() dax: fix conversion of holes to PMDs pmem: add wb_cache_pmem() to the PMEM API dax: support dirty DAX entries in radix tree mm: add find_get_entries_tag() dax: add support for fsync/msync ext2: call dax_pfn_mkwrite() for DAX fsync/msync ext4: call dax_pfn_mkwrite() for DAX fsync/msync xfs: call dax_pfn_mkwrite() for DAX fsync/msync arch/x86/include/asm/pmem.h | 11 +-- fs/block_dev.c | 2 +- fs/dax.c | 214 ++++++++++++++++++++++++++++++++++++++++---- fs/ext2/file.c | 4 +- fs/ext4/file.c | 4 +- fs/inode.c | 2 +- fs/xfs/xfs_file.c | 7 +- include/linux/dax.h | 7 ++ include/linux/fs.h | 3 +- include/linux/pagemap.h | 3 + include/linux/pmem.h | 22 ++++- include/linux/radix-tree.h | 9 ++ mm/filemap.c | 91 +++++++++++++++++-- mm/truncate.c | 69 +++++++------- mm/vmscan.c | 9 +- mm/workingset.c | 4 +- 16 files changed, 391 insertions(+), 70 deletions(-) -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:27 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id CE6EC7F59 for ; Wed, 6 Jan 2016 12:01:26 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 51140AC004 for ; Wed, 6 Jan 2016 10:01:23 -0800 (PST) X-ASG-Debug-ID: 1452103281-04cbb07e1620770001-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id ueJSc0cLp7a2maIx for ; Wed, 06 Jan 2016 10:01:21 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040116" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:12 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Date: Wed, 6 Jan 2016 11:00:56 -0700 X-ASG-Orig-Subj: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Message-Id: <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103281 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header When we get a DAX PMD fault for a write it is possible that there could be some number of 4k zero pages already present for the same range that were inserted to service reads from a hole. These 4k zero pages need to be unmapped from the VMAs and removed from the struct address_space radix tree before the real DAX PMD entry can be inserted. For PTE faults this same use case also exists and is handled by a combination of unmap_mapping_range() to unmap the VMAs and delete_from_page_cache() to remove the page from the address_space radix tree. For PMD faults we do have a call to unmap_mapping_range() (protected by a buffer_new() check), but nothing clears out the radix tree entry. The buffer_new() check is also incorrect as the current ext4 and XFS filesystem code will never return a buffer_head with BH_New set, even when allocating new blocks over a hole. Instead the filesystem will zero the blocks manually and return a buffer_head with only BH_Mapped set. Fix this situation by removing the buffer_new() check and adding a call to truncate_inode_pages_range() to clear out the radix tree entries before we insert the DAX PMD. Signed-off-by: Ross Zwisler --- fs/dax.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 03cc4a3..9dc0c97 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -594,6 +594,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, bool write = flags & FAULT_FLAG_WRITE; struct block_device *bdev; pgoff_t size, pgoff; + loff_t lstart, lend; sector_t block; int result = 0; @@ -647,15 +648,13 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, goto fallback; } - /* - * If we allocated new storage, make sure no process has any - * zero pages covering this hole - */ - if (buffer_new(&bh)) { - i_mmap_unlock_read(mapping); - unmap_mapping_range(mapping, pgoff << PAGE_SHIFT, PMD_SIZE, 0); - i_mmap_lock_read(mapping); - } + /* make sure no process has any zero pages covering this hole */ + lstart = pgoff << PAGE_SHIFT; + lend = lstart + PMD_SIZE - 1; /* inclusive */ + i_mmap_unlock_read(mapping); + unmap_mapping_range(mapping, lstart, PMD_SIZE, 0); + truncate_inode_pages_range(mapping, lstart, lend); + i_mmap_lock_read(mapping); /* * If a truncate happened while we were allocating blocks, we may @@ -669,7 +668,8 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, goto out; } if ((pgoff | PG_PMD_COLOUR) >= size) { - dax_pmd_dbg(&bh, address, "pgoff unaligned"); + dax_pmd_dbg(&bh, address, + "offset + huge page size > file size"); goto fallback; } -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:28 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 841AC7F59 for ; Wed, 6 Jan 2016 12:01:28 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 06652AC004 for ; Wed, 6 Jan 2016 10:01:27 -0800 (PST) X-ASG-Debug-ID: 1452103284-04cb6c42e51e4d0002-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id EQuFULEoRYOBbumg for ; Wed, 06 Jan 2016 10:01:26 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:18 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040272" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:16 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 5/9] mm: add find_get_entries_tag() Date: Wed, 6 Jan 2016 11:00:59 -0700 X-ASG-Orig-Subj: [PATCH v7 5/9] mm: add find_get_entries_tag() Message-Id: <1452103263-1592-6-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103285 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Add find_get_entries_tag() to the family of functions that include find_get_entries(), find_get_pages() and find_get_pages_tag(). This is needed for DAX dirty page handling because we need a list of both page offsets and radix tree entries ('indices' and 'entries' in this function) that are marked with the PAGECACHE_TAG_TOWRITE tag. Signed-off-by: Ross Zwisler Reviewed-by: Jan Kara --- include/linux/pagemap.h | 3 +++ mm/filemap.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 4d08b6c..92395a0 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -361,6 +361,9 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t start, unsigned int nr_pages, struct page **pages); unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index, int tag, unsigned int nr_pages, struct page **pages); +unsigned find_get_entries_tag(struct address_space *mapping, pgoff_t start, + int tag, unsigned int nr_entries, + struct page **entries, pgoff_t *indices); struct page *grab_cache_page_write_begin(struct address_space *mapping, pgoff_t index, unsigned flags); diff --git a/mm/filemap.c b/mm/filemap.c index 7b8be78..1e215fc 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1499,6 +1499,74 @@ repeat: } EXPORT_SYMBOL(find_get_pages_tag); +/** + * find_get_entries_tag - find and return entries that match @tag + * @mapping: the address_space to search + * @start: the starting page cache index + * @tag: the tag index + * @nr_entries: the maximum number of entries + * @entries: where the resulting entries are placed + * @indices: the cache indices corresponding to the entries in @entries + * + * Like find_get_entries, except we only return entries which are tagged with + * @tag. + */ +unsigned find_get_entries_tag(struct address_space *mapping, pgoff_t start, + int tag, unsigned int nr_entries, + struct page **entries, pgoff_t *indices) +{ + void **slot; + unsigned int ret = 0; + struct radix_tree_iter iter; + + if (!nr_entries) + return 0; + + rcu_read_lock(); +restart: + radix_tree_for_each_tagged(slot, &mapping->page_tree, + &iter, start, tag) { + struct page *page; +repeat: + page = radix_tree_deref_slot(slot); + if (unlikely(!page)) + continue; + if (radix_tree_exception(page)) { + if (radix_tree_deref_retry(page)) { + /* + * Transient condition which can only trigger + * when entry at index 0 moves out of or back + * to root: none yet gotten, safe to restart. + */ + goto restart; + } + + /* + * A shadow entry of a recently evicted page, a swap + * entry from shmem/tmpfs or a DAX entry. Return it + * without attempting to raise page count. + */ + goto export; + } + if (!page_cache_get_speculative(page)) + goto repeat; + + /* Has the page moved? */ + if (unlikely(page != *slot)) { + page_cache_release(page); + goto repeat; + } +export: + indices[ret] = iter.index; + entries[ret] = page; + if (++ret == nr_entries) + break; + } + rcu_read_unlock(); + return ret; +} +EXPORT_SYMBOL(find_get_entries_tag); + /* * CD/DVDs are error prone. When a medium error occurs, the driver may fail * a _large_ part of the i/o request. Imagine the worst scenario: -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id F3FF77F59 for ; Wed, 6 Jan 2016 12:01:29 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id E39708F8052 for ; Wed, 6 Jan 2016 10:01:29 -0800 (PST) X-ASG-Debug-ID: 1452103284-04cb6c42e51e4d0003-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id 5UJ2y06hBpFZRM0X for ; Wed, 06 Jan 2016 10:01:27 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040325" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:18 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 6/9] dax: add support for fsync/msync Date: Wed, 6 Jan 2016 11:01:00 -0700 X-ASG-Orig-Subj: [PATCH v7 6/9] dax: add support for fsync/msync Message-Id: <1452103263-1592-7-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103287 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header To properly handle fsync/msync in an efficient way DAX needs to track dirty pages so it is able to flush them durably to media on demand. The tracking of dirty pages is done via the radix tree in struct address_space. This radix tree is already used by the page writeback infrastructure for tracking dirty pages associated with an open file, and it already has support for exceptional (non struct page*) entries. We build upon these features to add exceptional entries to the radix tree for DAX dirty PMD or PTE pages at fault time. Signed-off-by: Ross Zwisler --- fs/dax.c | 187 ++++++++++++++++++++++++++++++++++++++++++++++++++-- include/linux/dax.h | 2 + mm/filemap.c | 6 ++ 3 files changed, 189 insertions(+), 6 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 9dc0c97..c107da6 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -324,6 +325,167 @@ static int copy_user_bh(struct page *to, struct inode *inode, return 0; } +#define NO_SECTOR -1 + +static int dax_radix_entry(struct address_space *mapping, pgoff_t index, + sector_t sector, bool pmd_entry, bool dirty) +{ + struct radix_tree_root *page_tree = &mapping->page_tree; + int error = 0; + void *entry; + + __mark_inode_dirty(mapping->host, I_DIRTY_PAGES); + + spin_lock_irq(&mapping->tree_lock); + entry = radix_tree_lookup(page_tree, index); + + if (entry) { + if (!pmd_entry || RADIX_DAX_TYPE(entry) == RADIX_DAX_PMD) + goto dirty; + radix_tree_delete(&mapping->page_tree, index); + mapping->nrexceptional--; + } + + if (sector == NO_SECTOR) { + /* + * This can happen during correct operation if our pfn_mkwrite + * fault raced against a hole punch operation. If this + * happens the pte that was hole punched will have been + * unmapped and the radix tree entry will have been removed by + * the time we are called, but the call will still happen. We + * will return all the way up to wp_pfn_shared(), where the + * pte_same() check will fail, eventually causing page fault + * to be retried by the CPU. + */ + goto unlock; + } + + error = radix_tree_insert(page_tree, index, + RADIX_DAX_ENTRY(sector, pmd_entry)); + if (error) + goto unlock; + + mapping->nrexceptional++; + dirty: + if (dirty) + radix_tree_tag_set(page_tree, index, PAGECACHE_TAG_DIRTY); + unlock: + spin_unlock_irq(&mapping->tree_lock); + return error; +} + +static int dax_writeback_one(struct block_device *bdev, + struct address_space *mapping, pgoff_t index, void *entry) +{ + struct radix_tree_root *page_tree = &mapping->page_tree; + int type = RADIX_DAX_TYPE(entry); + struct radix_tree_node *node; + struct blk_dax_ctl dax; + void **slot; + int ret = 0; + + spin_lock_irq(&mapping->tree_lock); + /* + * Regular page slots are stabilized by the page lock even + * without the tree itself locked. These unlocked entries + * need verification under the tree lock. + */ + if (!__radix_tree_lookup(page_tree, index, &node, &slot)) + goto unlock; + if (*slot != entry) + goto unlock; + + /* another fsync thread may have already written back this entry */ + if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE)) + goto unlock; + + radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE); + + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && type != RADIX_DAX_PMD)) { + ret = -EIO; + goto unlock; + } + + dax.sector = RADIX_DAX_SECTOR(entry); + dax.size = (type == RADIX_DAX_PMD ? PMD_SIZE : PAGE_SIZE); + spin_unlock_irq(&mapping->tree_lock); + + /* + * We cannot hold tree_lock while calling dax_map_atomic() because it + * eventually calls cond_resched(). + */ + ret = dax_map_atomic(bdev, &dax); + if (ret < 0) + return ret; + + if (WARN_ON_ONCE(ret < dax.size)) { + ret = -EIO; + goto unmap; + } + + wb_cache_pmem(dax.addr, dax.size); + unmap: + dax_unmap_atomic(bdev, &dax); + return ret; + + unlock: + spin_unlock_irq(&mapping->tree_lock); + return ret; +} + +/* + * Flush the mapping to the persistent domain within the byte range of [start, + * end]. This is required by data integrity operations to ensure file data is + * on persistent storage prior to completion of the operation. + */ +int dax_writeback_mapping_range(struct address_space *mapping, loff_t start, + loff_t end) +{ + struct inode *inode = mapping->host; + struct block_device *bdev = inode->i_sb->s_bdev; + pgoff_t indices[PAGEVEC_SIZE]; + pgoff_t start_page, end_page; + struct pagevec pvec; + void *entry; + int i, ret = 0; + + if (WARN_ON_ONCE(inode->i_blkbits != PAGE_SHIFT)) + return -EIO; + + rcu_read_lock(); + entry = radix_tree_lookup(&mapping->page_tree, start & PMD_MASK); + rcu_read_unlock(); + + /* see if the start of our range is covered by a PMD entry */ + if (entry && RADIX_DAX_TYPE(entry) == RADIX_DAX_PMD) + start &= PMD_MASK; + + start_page = start >> PAGE_CACHE_SHIFT; + end_page = end >> PAGE_CACHE_SHIFT; + + tag_pages_for_writeback(mapping, start_page, end_page); + + pagevec_init(&pvec, 0); + while (1) { + pvec.nr = find_get_entries_tag(mapping, start_page, + PAGECACHE_TAG_TOWRITE, PAGEVEC_SIZE, + pvec.pages, indices); + + if (pvec.nr == 0) + break; + + for (i = 0; i < pvec.nr; i++) { + ret = dax_writeback_one(bdev, mapping, indices[i], + pvec.pages[i]); + if (ret < 0) + return ret; + } + } + wmb_pmem(); + return 0; +} +EXPORT_SYMBOL_GPL(dax_writeback_mapping_range); + static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh, struct vm_area_struct *vma, struct vm_fault *vmf) { @@ -363,6 +525,11 @@ static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh, } dax_unmap_atomic(bdev, &dax); + error = dax_radix_entry(mapping, vmf->pgoff, dax.sector, false, + vmf->flags & FAULT_FLAG_WRITE); + if (error) + goto out; + error = vm_insert_mixed(vma, vaddr, dax.pfn); out: @@ -487,6 +654,7 @@ int __dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf, delete_from_page_cache(page); unlock_page(page); page_cache_release(page); + page = NULL; } /* @@ -596,7 +764,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, pgoff_t size, pgoff; loff_t lstart, lend; sector_t block; - int result = 0; + int error, result = 0; /* dax pmd mappings require pfn_t_devmap() */ if (!IS_ENABLED(CONFIG_FS_DAX_PMD)) @@ -737,6 +905,16 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, } dax_unmap_atomic(bdev, &dax); + if (write) { + error = dax_radix_entry(mapping, pgoff, dax.sector, + true, true); + if (error) { + dax_pmd_dbg(&bh, address, + "PMD radix insertion failed"); + goto fallback; + } + } + dev_dbg(part_to_dev(bdev->bd_part), "%s: %s addr: %lx pfn: %lx sect: %llx\n", __func__, current->comm, address, @@ -795,15 +973,12 @@ EXPORT_SYMBOL_GPL(dax_pmd_fault); * dax_pfn_mkwrite - handle first write to DAX page * @vma: The virtual memory area where the fault occurred * @vmf: The description of the fault - * */ int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf) { - struct super_block *sb = file_inode(vma->vm_file)->i_sb; + struct file *file = vma->vm_file; - sb_start_pagefault(sb); - file_update_time(vma->vm_file); - sb_end_pagefault(sb); + dax_radix_entry(file->f_mapping, vmf->pgoff, NO_SECTOR, false, true); return VM_FAULT_NOPAGE; } EXPORT_SYMBOL_GPL(dax_pfn_mkwrite); diff --git a/include/linux/dax.h b/include/linux/dax.h index e9d57f68..8204c3d 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -41,4 +41,6 @@ static inline bool dax_mapping(struct address_space *mapping) { return mapping->host && IS_DAX(mapping->host); } +int dax_writeback_mapping_range(struct address_space *mapping, loff_t start, + loff_t end); #endif diff --git a/mm/filemap.c b/mm/filemap.c index 1e215fc..2e7c8d9 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -482,6 +482,12 @@ int filemap_write_and_wait_range(struct address_space *mapping, { int err = 0; + if (dax_mapping(mapping) && mapping->nrexceptional) { + err = dax_writeback_mapping_range(mapping, lstart, lend); + if (err) + return err; + } + if (mapping->nrpages) { err = __filemap_fdatawrite_range(mapping, lstart, lend, WB_SYNC_ALL); -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 960537F59 for ; Wed, 6 Jan 2016 12:01:30 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 6B4E3304039 for ; Wed, 6 Jan 2016 10:01:30 -0800 (PST) X-ASG-Debug-ID: 1452103288-04bdf03afa20360001-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id fnklXtG8Dm0vL5XQ for ; Wed, 06 Jan 2016 10:01:28 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:23 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040435" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:20 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 8/9] ext4: call dax_pfn_mkwrite() for DAX fsync/msync Date: Wed, 6 Jan 2016 11:01:02 -0700 X-ASG-Orig-Subj: [PATCH v7 8/9] ext4: call dax_pfn_mkwrite() for DAX fsync/msync Message-Id: <1452103263-1592-9-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103288 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header To properly support the new DAX fsync/msync infrastructure filesystems need to call dax_pfn_mkwrite() so that DAX can track when user pages are dirtied. Signed-off-by: Ross Zwisler Reviewed-by: Jan Kara --- fs/ext4/file.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 60683ab..fa899c9 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -291,8 +291,8 @@ static int ext4_dax_pfn_mkwrite(struct vm_area_struct *vma, { struct inode *inode = file_inode(vma->vm_file); struct super_block *sb = inode->i_sb; - int ret = VM_FAULT_NOPAGE; loff_t size; + int ret; sb_start_pagefault(sb); file_update_time(vma->vm_file); @@ -300,6 +300,8 @@ static int ext4_dax_pfn_mkwrite(struct vm_area_struct *vma, size = (i_size_read(inode) + PAGE_SIZE - 1) >> PAGE_SHIFT; if (vmf->pgoff >= size) ret = VM_FAULT_SIGBUS; + else + ret = dax_pfn_mkwrite(vma, vmf); up_read(&EXT4_I(inode)->i_mmap_sem); sb_end_pagefault(sb); -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id B16537F60 for ; Wed, 6 Jan 2016 12:01:30 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 3FF40AC004 for ; Wed, 6 Jan 2016 10:01:30 -0800 (PST) X-ASG-Debug-ID: 1452103284-04cb6c42e51e4d0004-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id zH98VV1PCuvrPgKn for ; Wed, 06 Jan 2016 10:01:28 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040381" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:19 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 7/9] ext2: call dax_pfn_mkwrite() for DAX fsync/msync Date: Wed, 6 Jan 2016 11:01:01 -0700 X-ASG-Orig-Subj: [PATCH v7 7/9] ext2: call dax_pfn_mkwrite() for DAX fsync/msync Message-Id: <1452103263-1592-8-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103288 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header To properly support the new DAX fsync/msync infrastructure filesystems need to call dax_pfn_mkwrite() so that DAX can track when user pages are dirtied. Signed-off-by: Ross Zwisler Reviewed-by: Jan Kara --- fs/ext2/file.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/ext2/file.c b/fs/ext2/file.c index 11a42c5..2c88d68 100644 --- a/fs/ext2/file.c +++ b/fs/ext2/file.c @@ -102,8 +102,8 @@ static int ext2_dax_pfn_mkwrite(struct vm_area_struct *vma, { struct inode *inode = file_inode(vma->vm_file); struct ext2_inode_info *ei = EXT2_I(inode); - int ret = VM_FAULT_NOPAGE; loff_t size; + int ret; sb_start_pagefault(inode->i_sb); file_update_time(vma->vm_file); @@ -113,6 +113,8 @@ static int ext2_dax_pfn_mkwrite(struct vm_area_struct *vma, size = (i_size_read(inode) + PAGE_SIZE - 1) >> PAGE_SHIFT; if (vmf->pgoff >= size) ret = VM_FAULT_SIGBUS; + else + ret = dax_pfn_mkwrite(vma, vmf); up_read(&ei->dax_sem); sb_end_pagefault(inode->i_sb); -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:29 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id DFA1C7F37 for ; Wed, 6 Jan 2016 12:01:29 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id CFCA88F8052 for ; Wed, 6 Jan 2016 10:01:26 -0800 (PST) X-ASG-Debug-ID: 1452103284-04cb6c42e51e4d0001-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id c9sUHSoXPoBxsKUl for ; Wed, 06 Jan 2016 10:01:24 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040220" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:15 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 4/9] dax: support dirty DAX entries in radix tree Date: Wed, 6 Jan 2016 11:00:58 -0700 X-ASG-Orig-Subj: [PATCH v7 4/9] dax: support dirty DAX entries in radix tree Message-Id: <1452103263-1592-5-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103284 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Add support for tracking dirty DAX entries in the struct address_space radix tree. This tree is already used for dirty page writeback, and it already supports the use of exceptional (non struct page*) entries. In order to properly track dirty DAX pages we will insert new exceptional entries into the radix tree that represent dirty DAX PTE or PMD pages. These exceptional entries will also contain the writeback sectors for the PTE or PMD faults that we can use at fsync/msync time. There are currently two types of exceptional entries (shmem and shadow) that can be placed into the radix tree, and this adds a third. We rely on the fact that only one type of exceptional entry can be found in a given radix tree based on its usage. This happens for free with DAX vs shmem but we explicitly prevent shadow entries from being added to radix trees for DAX mappings. The only shadow entries that would be generated for DAX radix trees would be to track zero page mappings that were created for holes. These pages would receive minimal benefit from having shadow entries, and the choice to have only one type of exceptional entry in a given radix tree makes the logic simpler both in clear_exceptional_entry() and in the rest of DAX. Signed-off-by: Ross Zwisler Reviewed-by: Jan Kara --- fs/block_dev.c | 2 +- fs/inode.c | 2 +- include/linux/dax.h | 5 ++++ include/linux/fs.h | 3 +- include/linux/radix-tree.h | 9 ++++++ mm/filemap.c | 17 ++++++++---- mm/truncate.c | 69 ++++++++++++++++++++++++++-------------------- mm/vmscan.c | 9 +++++- mm/workingset.c | 4 +-- 9 files changed, 78 insertions(+), 42 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index e123641..303b7cd 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -75,7 +75,7 @@ void kill_bdev(struct block_device *bdev) { struct address_space *mapping = bdev->bd_inode->i_mapping; - if (mapping->nrpages == 0 && mapping->nrshadows == 0) + if (mapping->nrpages == 0 && mapping->nrexceptional == 0) return; invalidate_bh_lrus(); diff --git a/fs/inode.c b/fs/inode.c index 4c8f719..6e3e5d0 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -495,7 +495,7 @@ void clear_inode(struct inode *inode) */ spin_lock_irq(&inode->i_data.tree_lock); BUG_ON(inode->i_data.nrpages); - BUG_ON(inode->i_data.nrshadows); + BUG_ON(inode->i_data.nrexceptional); spin_unlock_irq(&inode->i_data.tree_lock); BUG_ON(!list_empty(&inode->i_data.private_list)); BUG_ON(!(inode->i_state & I_FREEING)); diff --git a/include/linux/dax.h b/include/linux/dax.h index b415e52..e9d57f68 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -36,4 +36,9 @@ static inline bool vma_is_dax(struct vm_area_struct *vma) { return vma->vm_file && IS_DAX(vma->vm_file->f_mapping->host); } + +static inline bool dax_mapping(struct address_space *mapping) +{ + return mapping->host && IS_DAX(mapping->host); +} #endif diff --git a/include/linux/fs.h b/include/linux/fs.h index ddb7bad..fdab768 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -433,7 +433,8 @@ struct address_space { struct rw_semaphore i_mmap_rwsem; /* protect tree, count, list */ /* Protected by tree_lock together with the radix tree */ unsigned long nrpages; /* number of total pages */ - unsigned long nrshadows; /* number of shadow entries */ + /* number of shadow or DAX exceptional entries */ + unsigned long nrexceptional; pgoff_t writeback_index;/* writeback starts here */ const struct address_space_operations *a_ops; /* methods */ unsigned long flags; /* error bits/gfp mask */ diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h index 57e7d87..7c88ad1 100644 --- a/include/linux/radix-tree.h +++ b/include/linux/radix-tree.h @@ -51,6 +51,15 @@ #define RADIX_TREE_EXCEPTIONAL_ENTRY 2 #define RADIX_TREE_EXCEPTIONAL_SHIFT 2 +#define RADIX_DAX_MASK 0xf +#define RADIX_DAX_SHIFT 4 +#define RADIX_DAX_PTE (0x4 | RADIX_TREE_EXCEPTIONAL_ENTRY) +#define RADIX_DAX_PMD (0x8 | RADIX_TREE_EXCEPTIONAL_ENTRY) +#define RADIX_DAX_TYPE(entry) ((unsigned long)entry & RADIX_DAX_MASK) +#define RADIX_DAX_SECTOR(entry) (((unsigned long)entry >> RADIX_DAX_SHIFT)) +#define RADIX_DAX_ENTRY(sector, pmd) ((void *)((unsigned long)sector << \ + RADIX_DAX_SHIFT | (pmd ? RADIX_DAX_PMD : RADIX_DAX_PTE))) + static inline int radix_tree_is_indirect_ptr(void *ptr) { return (int)((unsigned long)ptr & RADIX_TREE_INDIRECT_PTR); diff --git a/mm/filemap.c b/mm/filemap.c index 847ee43..7b8be78 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -11,6 +11,7 @@ */ #include #include +#include #include #include #include @@ -123,9 +124,9 @@ static void page_cache_tree_delete(struct address_space *mapping, __radix_tree_lookup(&mapping->page_tree, page->index, &node, &slot); if (shadow) { - mapping->nrshadows++; + mapping->nrexceptional++; /* - * Make sure the nrshadows update is committed before + * Make sure the nrexceptional update is committed before * the nrpages update so that final truncate racing * with reclaim does not see both counters 0 at the * same time and miss a shadow entry. @@ -579,9 +580,13 @@ static int page_cache_tree_insert(struct address_space *mapping, p = radix_tree_deref_slot_protected(slot, &mapping->tree_lock); if (!radix_tree_exceptional_entry(p)) return -EEXIST; + + if (WARN_ON(dax_mapping(mapping))) + return -EINVAL; + if (shadowp) *shadowp = p; - mapping->nrshadows--; + mapping->nrexceptional--; if (node) workingset_node_shadows_dec(node); } @@ -1245,9 +1250,9 @@ repeat: if (radix_tree_deref_retry(page)) goto restart; /* - * A shadow entry of a recently evicted page, - * or a swap entry from shmem/tmpfs. Return - * it without attempting to raise page count. + * A shadow entry of a recently evicted page, a swap + * entry from shmem/tmpfs or a DAX entry. Return it + * without attempting to raise page count. */ goto export; } diff --git a/mm/truncate.c b/mm/truncate.c index 76e35ad..e3ee0e2 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -34,31 +35,39 @@ static void clear_exceptional_entry(struct address_space *mapping, return; spin_lock_irq(&mapping->tree_lock); - /* - * Regular page slots are stabilized by the page lock even - * without the tree itself locked. These unlocked entries - * need verification under the tree lock. - */ - if (!__radix_tree_lookup(&mapping->page_tree, index, &node, &slot)) - goto unlock; - if (*slot != entry) - goto unlock; - radix_tree_replace_slot(slot, NULL); - mapping->nrshadows--; - if (!node) - goto unlock; - workingset_node_shadows_dec(node); - /* - * Don't track node without shadow entries. - * - * Avoid acquiring the list_lru lock if already untracked. - * The list_empty() test is safe as node->private_list is - * protected by mapping->tree_lock. - */ - if (!workingset_node_shadows(node) && - !list_empty(&node->private_list)) - list_lru_del(&workingset_shadow_nodes, &node->private_list); - __radix_tree_delete_node(&mapping->page_tree, node); + + if (dax_mapping(mapping)) { + if (radix_tree_delete_item(&mapping->page_tree, index, entry)) + mapping->nrexceptional--; + } else { + /* + * Regular page slots are stabilized by the page lock even + * without the tree itself locked. These unlocked entries + * need verification under the tree lock. + */ + if (!__radix_tree_lookup(&mapping->page_tree, index, &node, + &slot)) + goto unlock; + if (*slot != entry) + goto unlock; + radix_tree_replace_slot(slot, NULL); + mapping->nrexceptional--; + if (!node) + goto unlock; + workingset_node_shadows_dec(node); + /* + * Don't track node without shadow entries. + * + * Avoid acquiring the list_lru lock if already untracked. + * The list_empty() test is safe as node->private_list is + * protected by mapping->tree_lock. + */ + if (!workingset_node_shadows(node) && + !list_empty(&node->private_list)) + list_lru_del(&workingset_shadow_nodes, + &node->private_list); + __radix_tree_delete_node(&mapping->page_tree, node); + } unlock: spin_unlock_irq(&mapping->tree_lock); } @@ -228,7 +237,7 @@ void truncate_inode_pages_range(struct address_space *mapping, int i; cleancache_invalidate_inode(mapping); - if (mapping->nrpages == 0 && mapping->nrshadows == 0) + if (mapping->nrpages == 0 && mapping->nrexceptional == 0) return; /* Offsets within partial pages */ @@ -402,7 +411,7 @@ EXPORT_SYMBOL(truncate_inode_pages); */ void truncate_inode_pages_final(struct address_space *mapping) { - unsigned long nrshadows; + unsigned long nrexceptional; unsigned long nrpages; /* @@ -416,14 +425,14 @@ void truncate_inode_pages_final(struct address_space *mapping) /* * When reclaim installs eviction entries, it increases - * nrshadows first, then decreases nrpages. Make sure we see + * nrexceptional first, then decreases nrpages. Make sure we see * this in the right order or we might miss an entry. */ nrpages = mapping->nrpages; smp_rmb(); - nrshadows = mapping->nrshadows; + nrexceptional = mapping->nrexceptional; - if (nrpages || nrshadows) { + if (nrpages || nrexceptional) { /* * As truncation uses a lockless tree lookup, cycle * the tree lock to make sure any ongoing tree diff --git a/mm/vmscan.c b/mm/vmscan.c index 44ec50f..30e0cd7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -46,6 +46,7 @@ #include #include #include +#include #include #include @@ -671,9 +672,15 @@ static int __remove_mapping(struct address_space *mapping, struct page *page, * inode reclaim needs to empty out the radix tree or * the nodes are lost. Don't plant shadows behind its * back. + * + * We also don't store shadows for DAX mappings because the + * only page cache pages found in these are zero pages + * covering holes, and because we don't want to mix DAX + * exceptional entries and shadow exceptional entries in the + * same page_tree. */ if (reclaimed && page_is_file_cache(page) && - !mapping_exiting(mapping)) + !mapping_exiting(mapping) && !dax_mapping(mapping)) shadow = workingset_eviction(mapping, page); __delete_from_page_cache(page, shadow, memcg); spin_unlock_irqrestore(&mapping->tree_lock, flags); diff --git a/mm/workingset.c b/mm/workingset.c index aa01713..61ead9e 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -351,8 +351,8 @@ static enum lru_status shadow_lru_isolate(struct list_head *item, node->slots[i] = NULL; BUG_ON(node->count < (1U << RADIX_TREE_COUNT_SHIFT)); node->count -= 1U << RADIX_TREE_COUNT_SHIFT; - BUG_ON(!mapping->nrshadows); - mapping->nrshadows--; + BUG_ON(!mapping->nrexceptional); + mapping->nrexceptional--; } } BUG_ON(node->count); -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:01:32 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 66C057F37 for ; Wed, 6 Jan 2016 12:01:32 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 45612304039 for ; Wed, 6 Jan 2016 10:01:32 -0800 (PST) X-ASG-Debug-ID: 1452103290-04cbb07e1920790001-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id paDNw6qyfPr4zErw for ; Wed, 06 Jan 2016 10:01:31 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP; 06 Jan 2016 10:01:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="885040488" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 06 Jan 2016 10:01:22 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v7 9/9] xfs: call dax_pfn_mkwrite() for DAX fsync/msync Date: Wed, 6 Jan 2016 11:01:03 -0700 X-ASG-Orig-Subj: [PATCH v7 9/9] xfs: call dax_pfn_mkwrite() for DAX fsync/msync Message-Id: <1452103263-1592-10-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452103290 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header To properly support the new DAX fsync/msync infrastructure filesystems need to call dax_pfn_mkwrite() so that DAX can track when user pages are dirtied. Signed-off-by: Ross Zwisler --- fs/xfs/xfs_file.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index ebe9b82..55e16e2 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1610,9 +1610,8 @@ xfs_filemap_pmd_fault( /* * pfn_mkwrite was originally inteneded to ensure we capture time stamp * updates on write faults. In reality, it's need to serialise against - * truncate similar to page_mkwrite. Hence we open-code dax_pfn_mkwrite() - * here and cycle the XFS_MMAPLOCK_SHARED to ensure we serialise the fault - * barrier in place. + * truncate similar to page_mkwrite. Hence we cycle the XFS_MMAPLOCK_SHARED + * to ensure we serialise the fault barrier in place. */ static int xfs_filemap_pfn_mkwrite( @@ -1635,6 +1634,8 @@ xfs_filemap_pfn_mkwrite( size = (i_size_read(inode) + PAGE_SIZE - 1) >> PAGE_SHIFT; if (vmf->pgoff >= size) ret = VM_FAULT_SIGBUS; + else if (IS_DAX(inode)) + ret = dax_pfn_mkwrite(vma, vmf); xfs_iunlock(ip, XFS_MMAPLOCK_SHARED); sb_end_pagefault(inode->i_sb); return ret; -- 2.5.0 From ross.zwisler@linux.intel.com Wed Jan 6 12:10:17 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id AC46B7F47 for ; Wed, 6 Jan 2016 12:10:17 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 8C02B8F804B for ; Wed, 6 Jan 2016 10:10:17 -0800 (PST) X-ASG-Debug-ID: 1452103815-04cbb07e1620a80001-NocioJ Received: from mga14.intel.com ([192.55.52.115]) by cuda.sgi.com with ESMTP id 7ujwGocr7anGK8xG for ; Wed, 06 Jan 2016 10:10:15 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.115 Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga103.fm.intel.com with ESMTP; 06 Jan 2016 10:10:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,530,1444719600"; d="scan'208";a="721444590" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.201.208]) by orsmga003.jf.intel.com with ESMTP; 06 Jan 2016 10:10:12 -0800 Date: Wed, 6 Jan 2016 11:10:12 -0700 From: Ross Zwisler To: Dan Williams Cc: Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen Subject: Re: [PATCH v6 4/7] dax: add support for fsync/msync Message-ID: <20160106181012.GA7332@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v6 4/7] dax: add support for fsync/msync Mail-Followup-To: Ross Zwisler , Dan Williams , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Dave Chinner , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers , Andrew Morton , Matthew Wilcox , Dave Hansen References: <1450899560-26708-1-git-send-email-ross.zwisler@linux.intel.com> <1450899560-26708-5-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: UNKNOWN[192.55.52.115] X-Barracuda-Start-Time: 1452103815 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25889 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Sun, Jan 03, 2016 at 10:13:06AM -0800, Dan Williams wrote: > On Wed, Dec 23, 2015 at 11:39 AM, Ross Zwisler > wrote: > > To properly handle fsync/msync in an efficient way DAX needs to track dirty > > pages so it is able to flush them durably to media on demand. > > > > The tracking of dirty pages is done via the radix tree in struct > > address_space. This radix tree is already used by the page writeback > > infrastructure for tracking dirty pages associated with an open file, and > > it already has support for exceptional (non struct page*) entries. We > > build upon these features to add exceptional entries to the radix tree for > > DAX dirty PMD or PTE pages at fault time. > > > > Signed-off-by: Ross Zwisler > > I'm hitting the following report with the ndctl dax test [1] on > next-20151231. I bisected it to > commit 3cb108f941de "dax-add-support-for-fsync-sync-v6". I'll take a > closer look tomorrow, but in case someone can beat me to it, here's > the back-trace: > > ------------[ cut here ]------------ > kernel BUG at fs/inode.c:497! > [..] > CPU: 1 PID: 3001 Comm: umount Tainted: G O 4.4.0-rc7+ #2412 > Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > task: ffff8800da2a5a00 ti: ffff880307794000 task.ti: ffff880307794000 > RIP: 0010:[] [] clear_inode+0x71/0x80 > RSP: 0018:ffff880307797d50 EFLAGS: 00010002 > RAX: ffff8800da2a5a00 RBX: ffff8800ca2e7328 RCX: ffff8800da2a5a28 > RDX: 0000000000000001 RSI: 0000000000000005 RDI: ffff8800ca2e7530 > RBP: ffff880307797d60 R08: ffffffff82900ae0 R09: 0000000000000000 > R10: ffff8800ca2e7548 R11: 0000000000000000 R12: ffff8800ca2e7530 > R13: ffff8800ca2e7328 R14: ffff8800da2e88d0 R15: ffff8800da2e88d0 > FS: 00007f2b22f4a880(0000) GS:ffff88031fc40000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00005648abd933e8 CR3: 000000007f3fc000 CR4: 00000000000006e0 > Stack: > ffff8800ca2e7328 ffff8800ca2e7000 ffff880307797d88 ffffffffa01c18af > ffff8800ca2e7328 ffff8800ca2e74d0 ffffffffa01ec740 ffff880307797db0 > ffffffff81281038 ffff8800ca2e74c0 ffff880307797e00 ffff8800ca2e7328 > Call Trace: > [] xfs_fs_evict_inode+0x5f/0x110 [xfs] > [] evict+0xb8/0x180 > [] dispose_list+0x3b/0x50 > [] evict_inodes+0x144/0x170 > [] generic_shutdown_super+0x3f/0xf0 > [] kill_block_super+0x27/0x70 > [] deactivate_locked_super+0x43/0x70 > [] deactivate_super+0x5c/0x60 > [] cleanup_mnt+0x3f/0x90 > [] __cleanup_mnt+0x12/0x20 > [] task_work_run+0x76/0x90 > [] syscall_return_slowpath+0x20a/0x280 > [] int_ret_from_sys_call+0x25/0x9f > Code: 48 8d 93 30 03 00 00 48 39 c2 75 23 48 8b 83 d0 00 00 00 a8 20 > 74 1a a8 40 75 18 48 c7 8 > 3 d0 00 00 00 60 00 00 00 5b 41 5c 5d c3 <0f> 0b 0f 0b 0f 0b 0f 0b 0f > 0b 0f 1f 44 00 00 0f 1f > 44 00 00 55 > RIP [] clear_inode+0x71/0x80 > RSP > ---[ end trace 3b1d8898a94a4fc1 ]--- > > [1]: git://git@github.com:pmem/ndctl.git pending > make TESTS="test/dax.sh" check This issue is fixed with patch 2 of v7: https://lists.01.org/pipermail/linux-nvdimm/2016-January/003888.html Thanks for the report! From dan.j.williams@intel.com Wed Jan 6 13:04:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 366D37F37 for ; Wed, 6 Jan 2016 13:04:41 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id B119FAC004 for ; Wed, 6 Jan 2016 11:04:37 -0800 (PST) X-ASG-Debug-ID: 1452107075-04cb6c42e81fe70001-NocioJ Received: from mail-qk0-f174.google.com (mail-qk0-f174.google.com [209.85.220.174]) by cuda.sgi.com with ESMTP id xr6Ro7KExUNMwAos (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 06 Jan 2016 11:04:35 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.220.174 Received: by mail-qk0-f174.google.com with SMTP id p186so77432254qke.0 for ; Wed, 06 Jan 2016 11:04:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=P0Wv4KPporEznPLaBQoqMW69ccQn6wej7ZcnYc8NVb4=; b=cYj3idyCAJlbikeQqCVREUmst4qqorySa+oyL5zy5oAmDryngL7eWNivfH6RykBiXh EDjMtNrjU3GwRaEyYv/TRgOeQF0ODIWvWzu6LUZKLUCvqsPH8JV+lnOWYFHz8O5nRuDo 9R9X7uurbvj7i8oHa3O1P+CftZ+UNOxVMCUhIodTkTwXEXhLvl0sgvhFe+GXeUgk7Ut7 Zl0DuQpDe4pp2v6iMuHI/rLaWcL9jRbC/+eLBBPFH61psqB3spo5iBLzqh5yQe6GGcFM Xx/NA4mesZta54KMQ+wPpgjvJ6OMshtUnJup0NgVzk7aaGEQPQTTIbiVappIW5hx25c2 N2SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=P0Wv4KPporEznPLaBQoqMW69ccQn6wej7ZcnYc8NVb4=; b=CpxeWUtZWlKFUdpqjwpCaBjZQNVnN4au+q86E7/xFq9Gx3xEMbTXEsVa42lnH4fnoO RxlwbVzagaONyjFo1D2XOA+qMf6WmCdSngy+mTfwXJlJbHwonPc4k6BUY76hMHVPQj7K 7lIKgrqTEgh3tvcB44m1eR+Ewt0QvZfBp0Ot7Rtz70iNvvDiRDtGZK9/bHOQUKcDv0jV pJrHGk55xDjBWSDNOrQoZv+6yyB7JauWcJSdB8Pq0njXZMcwxo5wjFc3XxnZYvxIMB1u Wv2DoffrsPwqCr+jHThsQTpHQsSCg94eoZoPmD4yKM07ByEMeW5MGotNCK2scj4Bm1lm YFmw== X-Gm-Message-State: ALoCoQmGb1p9jrcKXWC7WTHs6pKgVJbXmKXqim60lFZoOuPhMZe/ZDbOefpR0VhPd2hyiPU2npFgaHre/6F5qC4HARPql6KWLqPAE68qbuq+xNW8wH0CrXA= MIME-Version: 1.0 X-Received: by 10.13.193.67 with SMTP id c64mr71711419ywd.181.1452107075187; Wed, 06 Jan 2016 11:04:35 -0800 (PST) Received: by 10.37.202.20 with HTTP; Wed, 6 Jan 2016 11:04:35 -0800 (PST) In-Reply-To: <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> Date: Wed, 6 Jan 2016 11:04:35 -0800 Message-ID: Subject: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs From: Dan Williams X-ASG-Orig-Subj: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs To: Ross Zwisler Cc: "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-qk0-f174.google.com[209.85.220.174] X-Barracuda-Start-Time: 1452107075 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25892 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler wrote: > When we get a DAX PMD fault for a write it is possible that there could be > some number of 4k zero pages already present for the same range that were > inserted to service reads from a hole. These 4k zero pages need to be > unmapped from the VMAs and removed from the struct address_space radix tree > before the real DAX PMD entry can be inserted. > > For PTE faults this same use case also exists and is handled by a > combination of unmap_mapping_range() to unmap the VMAs and > delete_from_page_cache() to remove the page from the address_space radix > tree. > > For PMD faults we do have a call to unmap_mapping_range() (protected by a > buffer_new() check), but nothing clears out the radix tree entry. The > buffer_new() check is also incorrect as the current ext4 and XFS filesystem > code will never return a buffer_head with BH_New set, even when allocating > new blocks over a hole. Instead the filesystem will zero the blocks > manually and return a buffer_head with only BH_Mapped set. > > Fix this situation by removing the buffer_new() check and adding a call to > truncate_inode_pages_range() to clear out the radix tree entries before we > insert the DAX PMD. > > Signed-off-by: Ross Zwisler Replaced the current contents of v6 in -mm from next-20160106 with this v7 set and it looks good. Reported-by: Dan Williams Tested-by: Dan Williams One question below... > --- > fs/dax.c | 20 ++++++++++---------- > 1 file changed, 10 insertions(+), 10 deletions(-) > > diff --git a/fs/dax.c b/fs/dax.c > index 03cc4a3..9dc0c97 100644 > --- a/fs/dax.c > +++ b/fs/dax.c > @@ -594,6 +594,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > bool write = flags & FAULT_FLAG_WRITE; > struct block_device *bdev; > pgoff_t size, pgoff; > + loff_t lstart, lend; > sector_t block; > int result = 0; > > @@ -647,15 +648,13 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > goto fallback; > } > > - /* > - * If we allocated new storage, make sure no process has any > - * zero pages covering this hole > - */ > - if (buffer_new(&bh)) { > - i_mmap_unlock_read(mapping); > - unmap_mapping_range(mapping, pgoff << PAGE_SHIFT, PMD_SIZE, 0); > - i_mmap_lock_read(mapping); > - } > + /* make sure no process has any zero pages covering this hole */ > + lstart = pgoff << PAGE_SHIFT; > + lend = lstart + PMD_SIZE - 1; /* inclusive */ > + i_mmap_unlock_read(mapping); > + unmap_mapping_range(mapping, lstart, PMD_SIZE, 0); > + truncate_inode_pages_range(mapping, lstart, lend); Do we need to do both unmap and truncate given that truncate_inode_page() optionally does an unmap_mapping_range() internally? From nab@linux-iscsi.org Wed Jan 6 13:06:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 6A9CB7F37 for ; Wed, 6 Jan 2016 13:06:08 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 4D39B304066 for ; Wed, 6 Jan 2016 11:06:05 -0800 (PST) X-ASG-Debug-ID: 1452107162-04cb6c42e51fee0001-NocioJ Received: from linux-iscsi.org (mail.linux-iscsi.org [67.23.28.174]) by cuda.sgi.com with ESMTP id TKG9iIz13BG4DWHD (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 06 Jan 2016 11:06:02 -0800 (PST) X-Barracuda-Envelope-From: nab@linux-iscsi.org X-Barracuda-Apparent-Source-IP: 67.23.28.174 Received: from [172.18.3.65] (unknown [157.22.22.146]) (using SSLv3 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: nab) by linux-iscsi.org (Postfix) with ESMTPSA id 1778B22D9E1; Wed, 6 Jan 2016 19:04:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=linux-iscsi.org; s=default.private; t=1452107048; bh=BdCtfGUVWbee/omQqUUnd4helTtoz/Z 9ca1+l2+I2BI=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To: References:Content-Type:Mime-Version:Content-Transfer-Encoding; b=mobILS+bOdHgpByM700zoPwvWlhKUe03RzpMsvTaUN1KBMjKaueiIeUMrGf2YQYH2 ebdI6Pa9U3HbMqKdbz6ERol7m3VDDf1/X4nJ+oII1QeXH7GH9HKllCRFBTVxY5Kcc1V Yl4sxcNqT9mz5mutXY5LV6oqFerU3nLm6IpWK1A= Message-ID: <1452107160.26125.39.camel@haakon3.risingtidesystems.com> Subject: Re: [PATCH 25/35] target: set bi_op to REQ_OP From: "Nicholas A. Bellinger" X-ASG-Orig-Subj: Re: [PATCH 25/35] target: set bi_op to REQ_OP To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Date: Wed, 06 Jan 2016 11:06:00 -0800 In-Reply-To: <1452027218-32303-26-git-send-email-mchristi@redhat.com> References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-26-git-send-email-mchristi@redhat.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.4.4-1 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mail.linux-iscsi.org[67.23.28.174] X-Barracuda-Start-Time: 1452107162 X-Barracuda-Encrypted: DHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25892 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Hi Mike, On Tue, 2016-01-05 at 14:53 -0600, mchristi@redhat.com wrote: > From: Mike Christie > > This patch has the target modules set the bio bi_op to a REQ_OP, and > rq_flag_bits to bi_rw. > > This patch is compile tested only. > > Signed-off-by: Mike Christie > --- > drivers/target/target_core_iblock.c | 38 ++++++++++++++++++++++--------------- > drivers/target/target_core_pscsi.c | 2 +- > 2 files changed, 24 insertions(+), 16 deletions(-) > Nice work to clean this long standing abuse. ;) For the target/iblock bit: Acked-by: Nicholas Bellinger From dan.j.williams@intel.com Wed Jan 6 13:14:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 6F13D7F37 for ; Wed, 6 Jan 2016 13:14:13 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 2798A304053 for ; Wed, 6 Jan 2016 11:14:12 -0800 (PST) X-ASG-Debug-ID: 1452107650-04bdf03afa21ec0001-NocioJ Received: from mail-qk0-f171.google.com (mail-qk0-f171.google.com [209.85.220.171]) by cuda.sgi.com with ESMTP id PFyLqQBrr2yWDRyr (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 06 Jan 2016 11:14:10 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.220.171 Received: by mail-qk0-f171.google.com with SMTP id q19so102717779qke.3 for ; Wed, 06 Jan 2016 11:14:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=pay6niFokxUpsxzr77QLIIxH/H/En30t7gkkbV0WTt8=; b=GQibgh9MzENpE9m6dNfg72EAEDIJ3C/29FffoM5Th3ZEgBkl3QKLXl4onY0SE1Gc27 2kWWbWENfN5sqgnn7+vWBAh8JY8tuCI7bxnDiidBEG8pH11nQWzom7hD0Sfm4e88Lse7 cLWFJPSeUhbfFLqMIMQ7sXj7IGsGYGXuDMT3aGwdGeMUXi8ePFWjGxv80hQHtZ7CL5o3 G3H71CvtRFMb/OPDD2qlT5HRKv9ZTMUFp19PxDWMubYiXJ5u30Lb1P09XMwtdboKoRTH ZLgOmdd/n511nyvEoo/x2MAwrfQPUWzTr0RtYpIPbRvHUc6YN0wn3dooeUhiBc589mDC pB3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=pay6niFokxUpsxzr77QLIIxH/H/En30t7gkkbV0WTt8=; b=FLdjEksW5mdZPb2AzCd6ejriLuPB8fmyhH8Bu78Q4HlwmcJFtGbrUmuB9YgttSWl8D tMPFRQ7175/lRXz5UbPBpau40xUGmPS5V0nNmR3HnuX309m76Je05JwX4hyR4srijEAl 2M7+ZX8+SGM56TGkBZBOY60fBCjXcVUwO/qn60WNZJDgjlQBrSB0BPLD4y7RGaHOHu5x d5fdcGGwCGd2QrIaFW7Sp/G7QSBmvQegAlOsh0nqPYOp/mNQ+FipHb6KZuHq061FuWqh DpkadHIRc2V9AmnSEOt6t+pjnMCfu1fGtN7e3uT+zhBB+S9FjTCywkjvDYWCx4SndmZC Yqdw== X-Gm-Message-State: ALoCoQklOT/GVyC58zj4tWPewBKhVRm+KCi4VA/6vl3WKmhExLXv5F+tc7dM5MpNVNU0hjRegbAfE2VXKs2sxIoxmk++rVMHTAj4MjBnLgFRz2R40k5UDuo= MIME-Version: 1.0 X-Received: by 10.129.48.197 with SMTP id w188mr73486818yww.82.1452107649987; Wed, 06 Jan 2016 11:14:09 -0800 (PST) Received: by 10.37.202.20 with HTTP; Wed, 6 Jan 2016 11:14:09 -0800 (PST) In-Reply-To: <1452103263-1592-2-git-send-email-ross.zwisler@linux.intel.com> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-2-git-send-email-ross.zwisler@linux.intel.com> Date: Wed, 6 Jan 2016 11:14:09 -0800 Message-ID: Subject: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() From: Dan Williams X-ASG-Orig-Subj: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() To: Ross Zwisler Cc: "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-qk0-f171.google.com[209.85.220.171] X-Barracuda-Start-Time: 1452107650 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25892 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler wrote: > __dax_dbg() currently assumes that bh->b_bdev is non-NULL, passing it into > bdevname() where is is dereferenced. This assumption isn't always true - > when called for reads of holes, ext4_dax_mmap_get_block() returns a buffer > head where bh->b_bdev is never set. I hit this BUG while testing the DAX > PMD fault path. > > Instead, verify that we have a valid bh->b_bdev, else just say "unknown" > for the block device. > > Signed-off-by: Ross Zwisler > Cc: Dan Williams > --- > fs/dax.c | 7 ++++++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/fs/dax.c b/fs/dax.c > index 7af8797..03cc4a3 100644 > --- a/fs/dax.c > +++ b/fs/dax.c > @@ -563,7 +563,12 @@ static void __dax_dbg(struct buffer_head *bh, unsigned long address, > { > if (bh) { > char bname[BDEVNAME_SIZE]; > - bdevname(bh->b_bdev, bname); > + > + if (bh->b_bdev) > + bdevname(bh->b_bdev, bname); > + else > + snprintf(bname, BDEVNAME_SIZE, "unknown"); > + > pr_debug("%s: %s addr: %lx dev %s state %lx start %lld " > "length %zd fallback: %s\n", fn, current->comm, > address, bname, bh->b_state, (u64)bh->b_blocknr, I'm assuming there's no danger of a such a buffer_head ever being used for the bdev parameter to dax_map_atomic()? Shouldn't we also/instead go fix ext4 to not send partially filled buffer_heads? From david@fromorbit.com Wed Jan 6 16:05:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 3CC567F37 for ; Wed, 6 Jan 2016 16:05:03 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 2B75F8F804C for ; Wed, 6 Jan 2016 14:05:00 -0800 (PST) X-ASG-Debug-ID: 1452117896-04cb6c42e6236d0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id d14vI9JowqR4ZABg for ; Wed, 06 Jan 2016 14:04:57 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CLDwBmjo1WPGu7LXleKAECgw9SbYJihXegKAEBAQEBAQaLWYk/GoVvBAICgSVNAQEBAQEBBwEBAQFBP0ESAYNgAQEBAwE6HCMFCwgDDgoJJQ8FJQMHGhOIJwfCFAEBAQEGAiEZhXWFR4JtgVFFhDkFlwqFQogKgWWHbYUxilSDc4J0HYFxKjSBeoNnAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 08:34:55 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGwCM-0008TH-K8; Thu, 07 Jan 2016 09:04:54 +1100 Date: Thu, 7 Jan 2016 09:04:54 +1100 From: Dave Chinner To: Mark Seger Cc: Linux fs XFS , Laurence Oberman Subject: Re: xfs and swift Message-ID: <20160106220454.GI21461@dastard> X-ASG-Orig-Subj: Re: xfs and swift References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452117896 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_SA210e X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25897 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 BSF_SC5_SA210e Custom Rule SA210e On Wed, Jan 06, 2016 at 10:15:25AM -0500, Mark Seger wrote: > I've recently found the performance our development swift system is > degrading over time as the number of objects/files increases. This is a > relatively small system, each server has 3 400GB disks. The system I'm > currently looking at has about 70GB tied up in slabs alone, close to 55GB > in xfs inodes and ili, and about 2GB free. The kernel > is 3.14.57-1-amd64-hlinux. So you go 50M cached inodes in memory, and a relatively old kernel. > Here's the way the filesystems are mounted: > > /dev/sdb1 on /srv/node/disk0 type xfs > (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=1536,noquota) > > I can do about 2000 1K file creates/sec when running 2 minute PUT tests at > 100 threads. If I repeat that tests for multiple hours, I see the number > of IOPS steadily decreasing to about 770 and the very next run it drops to > 260 and continues to fall from there. This happens at about 12M files. According to the numbers you've provided: lookups creates removes Fast: 1550 1350 300 Slow: 1000 900 250 This is pretty much what I'd expect on the XFS level when going from a small empty filesystem to one containing 12M 1k files. That does not correlate to your numbers above, so it's not at all clear that there is realy a problem here at the XFS level. > The directory structure is 2 tiered, with 1000 directories per tier so we > can have about 1M of them, though they don't currently all exist. That's insane. The xfs directory structure is much, much more space, time, IO and memory efficient that a directory hierachy like this. The only thing you need a directory hash hierarchy for is to provide sufficient concurrency for your operations, which you would probably get with a single level with one or two subdirs per filesystem AG. What you are doing is spreading the IO over thousands of different regions on the disks, and then randomly seeking between them on every operation. i.e. your workload is seekbound, and your directory structure is has the effect of /maximising/ seeks per operation... > I've written a collectl plugin that lets me watch many of the xfs stats in /me sighs and points at PCP: http://pcp.io > real-time and also have a test script that exercises the swift PUT code > directly and so eliminates all the inter-node communications. This script > also allows me to write to the existing swift directories as well as > redirect to an empty structure so mimics clean environment with no existing > subdirectories. Yet that doesn't behave like an empty filesystem, which is clearly shown by the fact the caches are full of inodes that are't being used by the test. It also points out that allocation of new inodes will follow the old logarithmic search speed degradation, because you're kernel is sufficiently old that it doesn't support the free inode btree feature... > I'm attaching some xfs stats during the run and hope they're readable. > These values are in operations/sec and each line is 1 second's worth of > data. The first set of numbers is on the clean directory and the second on > the existing 12M file one. At the bottom of these stats are also the xfs > slab allocations as reported by collectl. I can also watch these during a > test and can see the number of inode and ilo objects steadily grow at about > 1K/sec, which is curious since I'm only creating about 300. It grows at exactly the rate of the lookups beng done, which is what is expected. i.e. for each create being done, there are other lookups being done first. e.g. directories, other objects to determine where to create the new one, lookup has to be done before removes (which there are significant number of), etc. > > If there is anything else I can provide just let me know. > > I don't fully understand all the xfs stats but what does jump out at me is > the XFS read/write ops have increased by a factor of about 5 when the > system is slower. Which means your application is reading/writing 5x as much information from the filesystem when it is slow. That's not a filesystem problem - your applicaiton is having to traverse/modify 5x as much information for each object it is creating/modifying. There's a good chance that's a result of your massively wide object store directory heirarchy.... i.e. you need to start by understanding what your application is doing in terms of IO, configuration and algorithms and determine whether that is optimal before you start looking at whether the filesystem is actually the bottleneck. Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Wed Jan 6 16:10:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 47BC87F3F for ; Wed, 6 Jan 2016 16:10:13 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id D0094AC001 for ; Wed, 6 Jan 2016 14:10:09 -0800 (PST) X-ASG-Debug-ID: 1452118206-04cbb07e1626680001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id xTgtMVdcmQBqFFOT for ; Wed, 06 Jan 2016 14:10:06 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CKDwCRj41WPGu7LXleKAECgw9SbYJihXegKAEBAQEBAQaLWYk/JoVjBAICgSVNAQEBAQEBBwEBAQFBP0ESAYNhAQEEOhwjEAgDDgoJJQ8FJQMHGhOHFoEYD8IKAQEBAQEFAgEcBBmFdYVHgm2BUYR+BZcKhUKICoFlh22FMYpUg3OCdB2BcSo0AQGBeINnAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 08:40:05 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGwHM-0008U8-Rq; Thu, 07 Jan 2016 09:10:04 +1100 Date: Thu, 7 Jan 2016 09:10:04 +1100 From: Dave Chinner To: Mark Seger Cc: Laurence Oberman , Linux fs XFS Subject: Re: xfs and swift Message-ID: <20160106221004.GJ21461@dastard> X-ASG-Orig-Subj: Re: xfs and swift References: <20160106220454.GI21461@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160106220454.GI21461@dastard> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452118206 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25897 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Thu, Jan 07, 2016 at 09:04:54AM +1100, Dave Chinner wrote: > On Wed, Jan 06, 2016 at 10:15:25AM -0500, Mark Seger wrote: > > I've recently found the performance our development swift system is > > degrading over time as the number of objects/files increases. This is a > > relatively small system, each server has 3 400GB disks. The system I'm > > currently looking at has about 70GB tied up in slabs alone, close to 55GB > > in xfs inodes and ili, and about 2GB free. The kernel > > is 3.14.57-1-amd64-hlinux. > > So you go 50M cached inodes in memory, and a relatively old kernel. > > > Here's the way the filesystems are mounted: > > > > /dev/sdb1 on /srv/node/disk0 type xfs > > (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=1536,noquota) > > > > I can do about 2000 1K file creates/sec when running 2 minute PUT tests at > > 100 threads. If I repeat that tests for multiple hours, I see the number > > of IOPS steadily decreasing to about 770 and the very next run it drops to > > 260 and continues to fall from there. This happens at about 12M files. > > According to the numbers you've provided: > > lookups creates removes > Fast: 1550 1350 300 > Slow: 1000 900 250 > > This is pretty much what I'd expect on the XFS level when going from > a small empty filesystem to one containing 12M 1k files. > > That does not correlate to your numbers above, so it's not at all > clear that there is realy a problem here at the XFS level. > > > The directory structure is 2 tiered, with 1000 directories per tier so we > > can have about 1M of them, though they don't currently all exist. > > That's insane. > > The xfs directory structure is much, much more space, time, IO and > memory efficient that a directory hierachy like this. The only thing > you need a directory hash hierarchy for is to provide sufficient > concurrency for your operations, which you would probably get with a > single level with one or two subdirs per filesystem AG. BTW, you might want to read the section on directory block size for a quick introduction to XFS directory design and scalability: https://git.kernel.org/cgit/fs/xfs/xfs-documentation.git/tree/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Wed Jan 6 16:24:14 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 44E2D7F47 for ; Wed, 6 Jan 2016 16:24:14 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 0C6138F8052 for ; Wed, 6 Jan 2016 14:24:13 -0800 (PST) X-ASG-Debug-ID: 1452119051-04bdf03afc25d50001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id PgchBayvQY30g3vF for ; Wed, 06 Jan 2016 14:24:11 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DODwAPk41WPGu7LXleKAECgw+BP4Jig36BeaAoAQEBAQEBBotZhTqEBYYJBAICgSVNAQEBAQEBBwEBAQFBP0ESAYNgAQEBAwEnExwjBQsIAw4KCSUPBSUDBxoTiCcHwh4BAQgCIRmFdYVHiTwFlwqNTIFlh22FMY5HgnUcgXEqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 08:54:10 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGwV0-0008Va-2l; Thu, 07 Jan 2016 09:24:10 +1100 Date: Thu, 7 Jan 2016 09:24:10 +1100 From: Dave Chinner To: Brian Foster Cc: xfs@oss.sgi.com Subject: Re: [PATCH] xfs: handle dquot buffer readahead in log recovery correctly Message-ID: <20160106222410.GK21461@dastard> X-ASG-Orig-Subj: Re: [PATCH] xfs: handle dquot buffer readahead in log recovery correctly References: <1452052834-20605-1-git-send-email-david@fromorbit.com> <20160106143409.GA14682@bfoster.bfoster> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160106143409.GA14682@bfoster.bfoster> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452119051 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25898 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed, Jan 06, 2016 at 09:34:09AM -0500, Brian Foster wrote: > On Wed, Jan 06, 2016 at 03:00:34PM +1100, Dave Chinner wrote: > > @@ -264,6 +264,21 @@ xfs_dquot_buf_read_verify( > > } > > > > /* > > + * readahead errors are silent and simply leave the buffer as !done so > > + * a real read will then be run with the xfs_dquot_buf_ops verifier. > > + */ > > +static void > > +xfs_dquot_buf_readahead_verify( > > + struct xfs_buf *bp) > > +{ > > + struct xfs_mount *mp = bp->b_target->bt_mount; > > + > > + if (!xfs_dquot_buf_verify_crc(mp, bp) && > > + !xfs_dquot_buf_verify(mp, bp, 0)) > > + bp->b_flags &= ~XBF_DONE; > > Shouldn't this condition trigger if either the crc or buffer > verification fails (not if both fail)? Yup, got my logic tangled there. > Also, xfs_buf_ioend() sets XBF_DONE when bp->b_error == 0 after the read > verifier is invoked. I don't see bp->b_error being set here, so it looks > like clearing this flag wouldn't have any effect. Hmmm - I just copied that from the inode readahead verifier. So that's not working properly, either. I'll fix that, too. Thanks, Brian. Cheers, Dave. -- Dave Chinner david@fromorbit.com From mjseger@gmail.com Wed Jan 6 16:46:39 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE, T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 6539C7F37 for ; Wed, 6 Jan 2016 16:46:39 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 40802304039 for ; Wed, 6 Jan 2016 14:46:36 -0800 (PST) X-ASG-Debug-ID: 1452120393-04cbb07e19272f0001-NocioJ Received: from mail-wm0-f50.google.com (mail-wm0-f50.google.com [74.125.82.50]) by cuda.sgi.com with ESMTP id eaA6QJHpgcgRyGjG (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 06 Jan 2016 14:46:34 -0800 (PST) X-Barracuda-Envelope-From: mjseger@gmail.com X-Barracuda-Apparent-Source-IP: 74.125.82.50 Received: by mail-wm0-f50.google.com with SMTP id l65so76311551wmf.1 for ; Wed, 06 Jan 2016 14:46:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=WJHEDTYHi29d8GPP3kZzthvAiU0v+NmrZHHYUfc+wB0=; b=wvB90X7we/DrNoEEZ04dr2CEivqtEBkfKAMHcIwxYrvpmHBsmwjVPMdmiMiv97cbEr lXdZsV6QH0iZ8PXywyD7kAZvDUVK9IhNNY/sEt/WMybD3rJlUPe/ZO8l5ENlQqFJm4RN nk2LiJS32qik2duJNzfI1ETJQ3oRoiFTjiz9ZmT/EaFJ+JWvXECR8Qz37lYPKRxsytP1 jCTC5sQ4jxaIaacvVi/2FNTp49SbY2IYNcarTcxQJM9LGJ08Ey6aer/rZ5uEYHw4Dp5v XPRXAQuMWBG8AiwWyTHvRp1p5MyOGhAhoOq8APss4V/Gyfi9mjpb5zdRXNVV0EI5ViE0 c3xw== MIME-Version: 1.0 X-Received: by 10.194.86.166 with SMTP id q6mr87550795wjz.69.1452120393305; Wed, 06 Jan 2016 14:46:33 -0800 (PST) Received: by 10.194.78.99 with HTTP; Wed, 6 Jan 2016 14:46:33 -0800 (PST) In-Reply-To: <20160106221004.GJ21461@dastard> References: <20160106220454.GI21461@dastard> <20160106221004.GJ21461@dastard> Date: Wed, 6 Jan 2016 17:46:33 -0500 Message-ID: Subject: Re: xfs and swift From: Mark Seger X-ASG-Orig-Subj: Re: xfs and swift To: Dave Chinner Cc: Laurence Oberman , Linux fs XFS Content-Type: multipart/alternative; boundary=089e0102d91036d5860528b220ea X-Barracuda-Connect: mail-wm0-f50.google.com[74.125.82.50] X-Barracuda-Start-Time: 1452120394 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_SA210e, DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25897 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 0.00 HTML_MESSAGE BODY: HTML included in message 0.00 BSF_SC5_SA210e Custom Rule SA210e --089e0102d91036d5860528b220ea Content-Type: text/plain; charset=UTF-8 dave, thanks for getting back to me and the pointer to the config doc. lots to absorb and play with. the real challenge for me is that I'm doing testing as different levels. While i realize running 100 parallel swift PUT threads on a small system is not the ideal way to do things, it's the only easy way to get massive numbers of objects into the fillesystem and once there, the performance of a single stream is pretty poor and by instrumenting the swift code I can clearly see excess time being spent in creating/writing the objects and so that's lead us to believe the problem lies in the way xfs is configured. creating a new directory structure on that same mount point immediately results in high levels of performance. As an attempt to try to reproduce the problems w/o swift, I wrote a little python script that simply creates files in a 2-tier structure, the first tier consisting of 1024 directories and each directory contains 4096 subdirectories into which 1K files are created. I'm doing this for 10000 objects as a time and then timing them, reporting the times, 10 per line so each line represents 100 thousand file creates. Here too I'm seeing degradation and if I look at what happens when there are already 3M files and I write 1M more, I see these creation times/10 thousand: 1.004236 0.961419 0.996514 1.012150 1.101794 0.999422 0.994796 1.214535 0.997276 1.306736 2.793429 1.201471 1.133576 1.069682 1.030985 1.096341 1.052602 1.391364 0.999480 1.914125 1.193892 0.967206 1.263310 0.890472 1.051962 4.253694 1.145573 1.528848 13.586892 4.925790 3.975442 8.896552 1.197005 3.904226 7.503806 1.294842 1.816422 9.329792 7.270323 5.936545 7.058685 5.516841 4.527271 1.956592 1.382551 1.510339 1.318341 13.255939 6.938845 4.106066 2.612064 2.028795 4.647980 7.371628 5.473423 5.823201 14.229120 0.899348 3.539658 8.501498 4.662593 6.423530 7.980757 6.367012 3.414239 7.364857 4.143751 6.317348 11.393067 1.273371 146.067300 1.317814 1.176529 1.177830 52.206605 1.112854 2.087990 42.328220 1.178436 1.335202 49.118140 1.368696 1.515826 44.690431 0.927428 0.920801 0.985965 1.000591 1.027458 60.650443 1.771318 2.690499 2.262868 1.061343 0.932998 64.064210 37.726213 1.245129 0.743771 0.996683 nothing one set of 10K took almost 3 minutes! my main questions at this point are is this performance expected and/or might a newer kernel help? and might it be possible to significantly improve things via tuning or is it what it is? I do realize I'm starting with an empty directory tree whose performance degrades as it fills, but if I wanted to tune for say 10M or maybe 100M files might I be able to expect more consistent numbers (perhaps starting out at lower performance) as the numbers of objects grow? I'm basically looking for more consistency over a broader range of numbers of files. -mark On Wed, Jan 6, 2016 at 5:10 PM, Dave Chinner wrote: > On Thu, Jan 07, 2016 at 09:04:54AM +1100, Dave Chinner wrote: > > On Wed, Jan 06, 2016 at 10:15:25AM -0500, Mark Seger wrote: > > > I've recently found the performance our development swift system is > > > degrading over time as the number of objects/files increases. This is > a > > > relatively small system, each server has 3 400GB disks. The system I'm > > > currently looking at has about 70GB tied up in slabs alone, close to > 55GB > > > in xfs inodes and ili, and about 2GB free. The kernel > > > is 3.14.57-1-amd64-hlinux. > > > > So you go 50M cached inodes in memory, and a relatively old kernel. > > > > > Here's the way the filesystems are mounted: > > > > > > /dev/sdb1 on /srv/node/disk0 type xfs > > > > (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=1536,noquota) > > > > > > I can do about 2000 1K file creates/sec when running 2 minute PUT > tests at > > > 100 threads. If I repeat that tests for multiple hours, I see the > number > > > of IOPS steadily decreasing to about 770 and the very next run it > drops to > > > 260 and continues to fall from there. This happens at about 12M files. > > > > According to the numbers you've provided: > > > > lookups creates removes > > Fast: 1550 1350 300 > > Slow: 1000 900 250 > > > > This is pretty much what I'd expect on the XFS level when going from > > a small empty filesystem to one containing 12M 1k files. > > > > That does not correlate to your numbers above, so it's not at all > > clear that there is realy a problem here at the XFS level. > > > > > The directory structure is 2 tiered, with 1000 directories per tier so > we > > > can have about 1M of them, though they don't currently all exist. > > > > That's insane. > > > > The xfs directory structure is much, much more space, time, IO and > > memory efficient that a directory hierachy like this. The only thing > > you need a directory hash hierarchy for is to provide sufficient > > concurrency for your operations, which you would probably get with a > > single level with one or two subdirs per filesystem AG. > > BTW, you might want to read the section on directory block size for > a quick introduction to XFS directory design and scalability: > > > https://git.kernel.org/cgit/fs/xfs/xfs-documentation.git/tree/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > --089e0102d91036d5860528b220ea Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
dave, thanks for getting back to me and the pointer to the= config doc. =C2=A0lots to absorb and play with.

the rea= l challenge for me is that I'm doing testing as different levels.=C2=A0= While i realize running 100 parallel swift PUT threads on a small system i= s not the ideal way to do things, it's the only easy way to get massive= numbers of objects into the fillesystem and once there, the performance of= a single stream is pretty poor and by instrumenting the swift code I can c= learly see excess time being spent in creating/writing the objects and so t= hat's lead us to believe the problem lies in the way xfs is configured.= =C2=A0creating a new directory structure on that same mount point immediat= ely results in high levels of performance.

As an a= ttempt to try to reproduce the problems w/o swift, I wrote a little python = script that simply creates files in a 2-tier structure, the first tier cons= isting of 1024 directories and each directory contains 4096 subdirectories = into which 1K files are created.=C2=A0 I'm doing this for 10000 objects= as a time and then timing them, reporting the times, 10 per line so each l= ine represents 100 thousand file creates.

Here too= I'm seeing degradation and if I look at what happens when there are al= ready 3M files and I write 1M more, I see these creation times/10 thousand:=

=C2=A01.004236 =C2=A00.961419 =C2=A00.996514= =C2=A01.012150 =C2=A01.101794 =C2=A00.999422 =C2=A00.994796 =C2=A01.214535= =C2=A00.997276 =C2=A01.306736
=C2=A02.793429 =C2=A01.201471 =C2= =A01.133576 =C2=A01.069682 =C2=A01.030985 =C2=A01.096341 =C2=A01.052602 =C2= =A01.391364 =C2=A00.999480 =C2=A01.914125
=C2=A01.193892 =C2=A00.= 967206 =C2=A01.263310 =C2=A00.890472 =C2=A01.051962 =C2=A04.253694 =C2=A01.= 145573 =C2=A01.528848 13.586892 =C2=A04.925790
=C2=A03.975442 =C2= =A08.896552 =C2=A01.197005 =C2=A03.904226 =C2=A07.503806 =C2=A01.294842 =C2= =A01.816422 =C2=A09.329792 =C2=A07.270323 =C2=A05.936545
=C2=A07.= 058685 =C2=A05.516841 =C2=A04.527271 =C2=A01.956592 =C2=A01.382551 =C2=A01.= 510339 =C2=A01.318341 13.255939 =C2=A06.938845 =C2=A04.106066
=C2= =A02.612064 =C2=A02.028795 =C2=A04.647980 =C2=A07.371628 =C2=A05.473423 =C2= =A05.823201 14.229120 =C2=A00.899348 =C2=A03.539658 =C2=A08.501498
=C2=A04.662593 =C2=A06.423530 =C2=A07.980757 =C2=A06.367012 =C2=A03.41423= 9 =C2=A07.364857 =C2=A04.143751 =C2=A06.317348 11.393067 =C2=A01.273371
146.067300 =C2=A01.317814 =C2=A01.176529 =C2=A01.177830 52.206605 = =C2=A01.112854 =C2=A02.087990 42.328220 =C2=A01.178436 =C2=A01.335202
=
49.118140 =C2=A01.368696 =C2=A01.515826 44.690431 =C2=A00.927428 =C2= =A00.920801 =C2=A00.985965 =C2=A01.000591 =C2=A01.027458 60.650443
=C2=A01.771318 =C2=A02.690499 =C2=A02.262868 =C2=A01.061343 =C2=A00.93299= 8 64.064210 37.726213 =C2=A01.245129 =C2=A00.743771 =C2=A00.996683

nothing one set of 10K took almost 3 minutes!
<= div>
my main questions at this point are is this performance = expected and/or might a newer kernel help? =C2=A0and might it be possible t= o significantly improve things via tuning or is it what it is?=C2=A0 I do r= ealize I'm starting with an empty directory tree whose performance degr= ades as it fills, but if I wanted to tune for say 10M or maybe 100M files m= ight I be able to expect more consistent numbers (perhaps starting out at l= ower performance) as the numbers of objects grow?=C2=A0 I'm basically l= ooking for more consistency over a broader range of numbers of files.
=

-mark

On Wed, Jan 6, 2016 at 5:10 PM, Dave Chinner <david= @fromorbit.com> wrote:
On Thu, Jan 07, 2016 at 09:04:54AM +110= 0, Dave Chinner wrote:
> On Wed, Jan 06, 2016 at 10:15:25AM -0500, Mark Seger wrote:
> > I've recently found the performance our development swift sys= tem is
> > degrading over time as the number of objects/files increases.=C2= =A0 This is a
> > relatively small system, each server has 3 400GB disks.=C2=A0 The= system I'm
> > currently looking at has about 70GB tied up in slabs alone, close= to 55GB
> > in xfs inodes and ili, and about 2GB free.=C2=A0 The kernel
> > is 3.14.57-1-amd64-hlinux.
>
> So you go 50M cached inodes in memory, and a relatively old kernel. >
> > Here's the way the filesystems are mounted:
> >
> > /dev/sdb1 on /srv/node/disk0 type xfs
> > (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=3D8,logbsi= ze=3D256k,sunit=3D512,swidth=3D1536,noquota)
> >
> > I can do about 2000 1K file creates/sec when running 2 minute PUT= tests at
> > 100 threads.=C2=A0 If I repeat that tests for multiple hours, I s= ee the number
> > of IOPS steadily decreasing to about 770 and the very next run it= drops to
> > 260 and continues to fall from there.=C2=A0 This happens at about= 12M files.
>
> According to the numbers you've provided:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0lookups=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cre= ates=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0removes
> Fast: 1550=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1350=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 300
> Slow: 1000=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0900=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 250
>
> This is pretty much what I'd expect on the XFS level when going fr= om
> a small empty filesystem to one containing 12M 1k files.
>
> That does not correlate to your numbers above, so it's not at all<= br> > clear that there is realy a problem here at the XFS level.
>
> > The directory structure is 2 tiered, with 1000 directories per ti= er so we
> > can have about 1M of them, though they don't currently all ex= ist.
>
> That's insane.
>
> The xfs directory structure is much, much more space, time, IO and
> memory efficient that a directory hierachy like this. The only thing > you need a directory hash hierarchy for is to provide sufficient
> concurrency for your operations, which you would probably get with a > single level with one or two subdirs per filesystem AG.

BTW, you might want to read the section on directory block size= for
a quick introduction to XFS directory design and scalability:

https://git.kernel.org/cgit/fs/xfs/xfs-documentation.git= /tree/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

--089e0102d91036d5860528b220ea-- From martin.petersen@oracle.com Wed Jan 6 17:18:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id F15DF7F37 for ; Wed, 6 Jan 2016 17:18:00 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 7FA4AAC002 for ; Wed, 6 Jan 2016 15:17:57 -0800 (PST) X-ASG-Debug-ID: 1452122274-04cbb07e1a27c80001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id J0ng6UpOSJhyLRrq (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 06 Jan 2016 15:17:55 -0800 (PST) X-Barracuda-Envelope-From: martin.petersen@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u06NHAP8021076 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Wed, 6 Jan 2016 23:17:10 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u06NH9s9020124 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Wed, 6 Jan 2016 23:17:09 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u06NH6jD006760; Wed, 6 Jan 2016 23:17:06 GMT Received: from ca-mkp.ca.oracle.com (/10.159.214.123) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 06 Jan 2016 15:17:05 -0800 To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields From: "Martin K. Petersen" X-ASG-Orig-Subj: Re: [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields Organization: Oracle References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-3-git-send-email-mchristi@redhat.com> Date: Wed, 06 Jan 2016 18:17:01 -0500 In-Reply-To: <1452027218-32303-3-git-send-email-mchristi@redhat.com> (mchristi@redhat.com's message of "Tue, 5 Jan 2016 14:53:05 -0600") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Source-IP: aserv0022.oracle.com [141.146.126.234] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1452122275 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25899 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines >>>>> "Mike" == mchristi writes: +enum req_op { + REQ_OP_READ, + REQ_OP_WRITE = REQ_WRITE, + REQ_OP_DISCARD = REQ_DISCARD, + REQ_OP_WRITE_SAME = REQ_WRITE_SAME, +}; + I have been irked by the REQ_ prefix in bios since the flags were consolidated a while back. When I attempted to fix the READ/WRITE mess I used a BLK_ prefix as a result. Anyway. Just bikeshedding... -- Martin K. Petersen Oracle Linux Engineering From dave@fromorbit.com Wed Jan 6 17:32:37 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 635B47F3F for ; Wed, 6 Jan 2016 17:32:37 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id AA61B304039 for ; Wed, 6 Jan 2016 15:32:33 -0800 (PST) X-ASG-Debug-ID: 1452123150-04cbb07e19280f0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id inrExaZLYMC3tpov for ; Wed, 06 Jan 2016 15:32:30 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BoCwCKo41WPGu7LXleKAECgw+IH4F5oCgBAQEBAQEGkROLOk0BAQEBAQEHAQEBAUE/hRE7gQIDB4hboSehAIYOimqEGQWXCpxPjkeCLgEBCAEBAQGCRiqGFQEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 10:01:30 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aGxY9-0000Ci-Rx for xfs@oss.sgi.com; Thu, 07 Jan 2016 10:31:29 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aGxY9-0003d5-Qs for xfs@oss.sgi.com; Thu, 07 Jan 2016 10:31:29 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 0/2] xfs: small debug improvements Date: Thu, 7 Jan 2016 10:31:25 +1100 X-ASG-Orig-Subj: [PATCH 0/2] xfs: small debug improvements Message-Id: <1452123087-13892-1-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452123150 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25900 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Hi folks, These are a couple of simple patches I have recently been using when trying to diagnose issues that have been reported to me. They seem useful enough to include for everyone, so here they are. -Dave. From dave@fromorbit.com Wed Jan 6 17:32:37 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6BA3B7F47 for ; Wed, 6 Jan 2016 17:32:37 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id BDB678F8059 for ; Wed, 6 Jan 2016 15:32:36 -0800 (PST) X-ASG-Debug-ID: 1452123152-04bdf03afc27230001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 0LSP2jc6xRLAQPn4 for ; Wed, 06 Jan 2016 15:32:33 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BsCwCKo41WPGu7LXleKAECgw+BP4ZggXmgKAEBAQEBAQaRE4s6TQEBAQEBAQcBAQEBQT+ENQEFJy8zCBgxOQMHFBmILsInhg6Jd3OEGQWXCogzlByOR4IuAQs7HIFxKjSEFoFLAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 10:01:30 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aGxY9-0000Ck-T2 for xfs@oss.sgi.com; Thu, 07 Jan 2016 10:31:29 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aGxY9-0003dD-Rx for xfs@oss.sgi.com; Thu, 07 Jan 2016 10:31:29 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 2/2] xfs: bmapbt checking on debug kernels too expensive Date: Thu, 7 Jan 2016 10:31:27 +1100 X-ASG-Orig-Subj: [PATCH 2/2] xfs: bmapbt checking on debug kernels too expensive Message-Id: <1452123087-13892-3-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452123087-13892-1-git-send-email-david@fromorbit.com> References: <1452123087-13892-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452123152 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25900 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner For large sparse or fragmented files, checking every single entry in the bmapbt on every operation is prohibitively expensive. Especially as such checks rarely discover problems during normal operations on high extent coutn files. Our regression tests don't tend to exercise files with hundreds of thousands to millions of extents, so mostly this isn't noticed. However, trying to run things like xfs_mdrestore of large filesystem dumps on a debug kernel quickly becomes impossible as the CPU is completely burnt up repeatedly walking the sparse file bmapbt that is generated for every allocation that is made. Hence, if the file has more than 10,000 extents, just don't bother with walking the tree to check it exhaustively. The btree code has checks that ensure that the newly inserted/removed/modified record is correctly ordered, so the entrie tree walk in thses cases has limited additional value. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_bmap.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c index 7388495..bc7e7d5 100644 --- a/fs/xfs/libxfs/xfs_bmap.c +++ b/fs/xfs/libxfs/xfs_bmap.c @@ -325,9 +325,11 @@ xfs_check_block( /* * Check that the extents for the inode ip are in the right order in all - * btree leaves. + * btree leaves. THis becomes prohibitively expensive for large extent count + * files, so don't bother with inodes that have more than 10,000 extents in + * them. The btree record ordering checks will still be done, so for such large + * bmapbt constructs that is going to catch most corruptions. */ - STATIC void xfs_bmap_check_leaf_extents( xfs_btree_cur_t *cur, /* btree cursor or null */ @@ -352,6 +354,10 @@ xfs_bmap_check_leaf_extents( return; } + /* skip large extent count inodes */ + if (ip->i_d.di_nextents > 10000) + return; + bno = NULLFSBLOCK; mp = ip->i_mount; ifp = XFS_IFORK_PTR(ip, whichfork); -- 2.5.0 From dave@fromorbit.com Wed Jan 6 17:32:39 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 66CD97F47 for ; Wed, 6 Jan 2016 17:32:39 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id E72498F8059 for ; Wed, 6 Jan 2016 15:32:34 -0800 (PST) X-ASG-Debug-ID: 1452123150-04cbb07e19280f0002-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id heSV7PBMFEzYHfRF for ; Wed, 06 Jan 2016 15:32:32 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BsCwCKo41WPGu7LXleKAECgw+BP4ZggXmgKAEBAQEBAQaRE4s6TQEBAQEBAQcBAQEBQT+ENQEFJy8zCBgxOQMHFBmILsInhg6KaoQZBY05iVGPMYdthTGOR4IuAQuCSCo0hWEBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 10:01:30 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aGxY9-0000Cj-SI for xfs@oss.sgi.com; Thu, 07 Jan 2016 10:31:29 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aGxY9-0003d8-Rb for xfs@oss.sgi.com; Thu, 07 Jan 2016 10:31:29 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 1/2] xfs: add tracepoints to readpage calls Date: Thu, 7 Jan 2016 10:31:26 +1100 X-ASG-Orig-Subj: [PATCH 1/2] xfs: add tracepoints to readpage calls Message-Id: <1452123087-13892-2-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452123087-13892-1-git-send-email-david@fromorbit.com> References: <1452123087-13892-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452123152 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25900 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner This allows us to see page cache driven readahead in action as it passes through XFS. This helps to understand buffered read throughput problems such as readahead IO IO sizes being too small for the underlying device to reach max throughput. Signed-off-by: Dave Chinner --- fs/xfs/xfs_aops.c | 2 ++ fs/xfs/xfs_trace.h | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 29e7e5d..379c089 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -1917,6 +1917,7 @@ xfs_vm_readpage( struct file *unused, struct page *page) { + trace_xfs_vm_readpage(page->mapping->host, 1); return mpage_readpage(page, xfs_get_blocks); } @@ -1927,6 +1928,7 @@ xfs_vm_readpages( struct list_head *pages, unsigned nr_pages) { + trace_xfs_vm_readpages(mapping->host, nr_pages); return mpage_readpages(mapping, pages, nr_pages, xfs_get_blocks); } diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h index 877079eb..391d797 100644 --- a/fs/xfs/xfs_trace.h +++ b/fs/xfs/xfs_trace.h @@ -1222,6 +1222,32 @@ DEFINE_PAGE_EVENT(xfs_writepage); DEFINE_PAGE_EVENT(xfs_releasepage); DEFINE_PAGE_EVENT(xfs_invalidatepage); +DECLARE_EVENT_CLASS(xfs_readpage_class, + TP_PROTO(struct inode *inode, int nr_pages), + TP_ARGS(inode, nr_pages), + TP_STRUCT__entry( + __field(dev_t, dev) + __field(xfs_ino_t, ino) + __field(int, nr_pages) + ), + TP_fast_assign( + __entry->dev = inode->i_sb->s_dev; + __entry->ino = inode->i_ino; + __entry->nr_pages = nr_pages; + ), + TP_printk("dev %d:%d ino 0x%llx nr_pages %d", + MAJOR(__entry->dev), MINOR(__entry->dev), + __entry->ino, + __entry->nr_pages) +) + +#define DEFINE_READPAGE_EVENT(name) \ +DEFINE_EVENT(xfs_readpage_class, name, \ + TP_PROTO(struct inode *inode, int nr_pages), \ + TP_ARGS(inode, nr_pages)) +DEFINE_READPAGE_EVENT(xfs_vm_readpage); +DEFINE_READPAGE_EVENT(xfs_vm_readpages); + DECLARE_EVENT_CLASS(xfs_imap_class, TP_PROTO(struct xfs_inode *ip, xfs_off_t offset, ssize_t count, int type, struct xfs_bmbt_irec *irec), -- 2.5.0 From david@fromorbit.com Wed Jan 6 17:49:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4C8107F51 for ; Wed, 6 Jan 2016 17:49:16 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1BB7C304059 for ; Wed, 6 Jan 2016 15:49:16 -0800 (PST) X-ASG-Debug-ID: 1452124152-04bdf03af927780001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id QZvNffP6PHaRuYrj for ; Wed, 06 Jan 2016 15:49:13 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DKDwANp41WPGu7LXleKAECgw+BP4Jig36BeaAoAQEBAQEBBotZiT+GCQQCAoEkTQEBAQEBAQcBAQEBQT9BEgGDYAEBAQMBOhwjBQsIAw4KCSUPBSUDBxoTG4gMB8IBAQEBBwIhGYV1hUeEMYULBZMKhACNTIFlhEOIW4pUg3OFAio0gXqCHYFKAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 10:19:11 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aGxpA-0000F0-HA; Thu, 07 Jan 2016 10:49:04 +1100 Date: Thu, 7 Jan 2016 10:49:04 +1100 From: Dave Chinner To: Mark Seger Cc: Laurence Oberman , Linux fs XFS Subject: Re: xfs and swift Message-ID: <20160106234904.GL21461@dastard> X-ASG-Orig-Subj: Re: xfs and swift References: <20160106220454.GI21461@dastard> <20160106221004.GJ21461@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452124152 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_SA210e X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25900 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 BSF_SC5_SA210e Custom Rule SA210e On Wed, Jan 06, 2016 at 05:46:33PM -0500, Mark Seger wrote: > dave, thanks for getting back to me and the pointer to the config doc. > lots to absorb and play with. > > the real challenge for me is that I'm doing testing as different levels. > While i realize running 100 parallel swift PUT threads on a small system is > not the ideal way to do things, it's the only easy way to get massive > numbers of objects into the fillesystem and once there, the performance of > a single stream is pretty poor and by instrumenting the swift code I can > clearly see excess time being spent in creating/writing the objects and so > that's lead us to believe the problem lies in the way xfs is configured. > creating a new directory structure on that same mount point immediately > results in high levels of performance. > > As an attempt to try to reproduce the problems w/o swift, I wrote a little > python script that simply creates files in a 2-tier structure, the first > tier consisting of 1024 directories and each directory contains 4096 > subdirectories into which 1K files are created. So you created something with even greater fan-out than what your swift app is using? > I'm doing this for 10000 > objects as a time and then timing them, reporting the times, 10 per line so > each line represents 100 thousand file creates. > > Here too I'm seeing degradation and if I look at what happens when there > are already 3M files and I write 1M more, I see these creation times/10 > thousand: > > 1.004236 0.961419 0.996514 1.012150 1.101794 0.999422 0.994796 > 1.214535 0.997276 1.306736 > 2.793429 1.201471 1.133576 1.069682 1.030985 1.096341 1.052602 > 1.391364 0.999480 1.914125 > 1.193892 0.967206 1.263310 0.890472 1.051962 4.253694 1.145573 > 1.528848 13.586892 4.925790 > 3.975442 8.896552 1.197005 3.904226 7.503806 1.294842 1.816422 > 9.329792 7.270323 5.936545 > 7.058685 5.516841 4.527271 1.956592 1.382551 1.510339 1.318341 > 13.255939 6.938845 4.106066 > 2.612064 2.028795 4.647980 7.371628 5.473423 5.823201 14.229120 > 0.899348 3.539658 8.501498 > 4.662593 6.423530 7.980757 6.367012 3.414239 7.364857 4.143751 > 6.317348 11.393067 1.273371 > 146.067300 1.317814 1.176529 1.177830 52.206605 1.112854 2.087990 > 42.328220 1.178436 1.335202 > 49.118140 1.368696 1.515826 44.690431 0.927428 0.920801 0.985965 > 1.000591 1.027458 60.650443 > 1.771318 2.690499 2.262868 1.061343 0.932998 64.064210 37.726213 > 1.245129 0.743771 0.996683 > > nothing one set of 10K took almost 3 minutes! Which is no surprise because you have slow disks and a *lot* of memory. At some point the journal and/or memory is going to fill up with dirty objects and have to block waiting for writeback. At that point there's going to be several hundred thousand dirty inodes that need to be flushed to disk before progress can be made again. That metadata writeback will be seek bound, and that's where all the delay comes from. We've been through this problem several times now with different swift users over the past couple of years. Please go and search the list archives, because every time the solution has been the same: - reduce the directory heirarchy to a single level with, at most, the number of directories matching the expected *production* concurrency level - reduce the XFS log size down to 32-128MB to limit dirty metadata object buildup in memory - reduce the number of AGs to as small as necessary to maintain /allocation/ concurrency to limit the number of different locations XFS writes to the disks (typically 10-20x less than the application level concurrency) - use a 3.16+ kernel with the free inode btree on-disk format feature to keep inode allocation CPU overhead low and consistent regardless of the number of inodes already allocated in the filesystem. > my main questions at this point are is this performance expected and/or > might a newer kernel help? and might it be possible to significantly > improve things via tuning or is it what it is? I do realize I'm starting > with an empty directory tree whose performance degrades as it fills, but if > I wanted to tune for say 10M or maybe 100M files might I be able to expect The mkfs defaults will work just fine with that many files in the filesystem. Your application configuration and data store layout is likely to be your biggest problem here. Cheers, Dave. -- Dave Chinner david@fromorbit.com From darrick.wong@oracle.com Wed Jan 6 18:10:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C8B5A7F53 for ; Wed, 6 Jan 2016 18:10:16 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9A3F08F804C for ; Wed, 6 Jan 2016 16:10:13 -0800 (PST) X-ASG-Debug-ID: 1452125411-04bdf03af927f40001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id dXeCJn5O8YPiDa2Z (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 06 Jan 2016 16:10:11 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0709gqr001376 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Thu, 7 Jan 2016 00:09:43 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0709gcS012416 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Thu, 7 Jan 2016 00:09:42 GMT Received: from abhmp0011.oracle.com (abhmp0011.oracle.com [141.146.116.17]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0709gUh027307; Thu, 7 Jan 2016 00:09:42 GMT Received: from localhost (/10.145.178.207) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 06 Jan 2016 16:09:41 -0800 Date: Wed, 6 Jan 2016 16:09:39 -0800 From: "Darrick J. Wong" To: Dave Chinner Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [PATCH 34/76] xfs: implement XFS_IOC_SWAPEXT when rmap btree is enabled Message-ID: <20160107000939.GA8015@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 34/76] xfs: implement XFS_IOC_SWAPEXT when rmap btree is enabled References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090008.12713.25755.stgit@birch.djwong.org> <20160103121728.GC28429@infradead.org> <20160104234033.GD28330@birch.djwong.org> <20160105024130.GI19802@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105024130.GI19802@dastard> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1452125411 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25901 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Tue, Jan 05, 2016 at 01:41:30PM +1100, Dave Chinner wrote: > On Mon, Jan 04, 2016 at 03:40:33PM -0800, Darrick J. Wong wrote: > > On Sun, Jan 03, 2016 at 04:17:28AM -0800, Christoph Hellwig wrote: > > > On Sat, Dec 19, 2015 at 01:00:08AM -0800, Darrick J. Wong wrote: > > > > Implement extent swapping when reverse-mapping is enabled. > > > > > > Can you just fold this into the previous patch? There seem to be a lot > > > of patches in the series that would benefit from merging, but this > > > is the one that really screams for it :) > > > > The only reason I'm keeping Dave's original rmap patches separate is because > > the last time I checked with Dave, he said that he'd made a few bugfixes to his > > patchset that hadn't yet been sent out, so I'd like to merge those fixes into > > the original patches before starting any combining work. > > I haven't had a chance to sort any of those out fully, so it's > probably best if you combine all your patches to minimise the > separation and I'll send delta patches (if still relevant) as I work > through it all in the next couple of weeks... Ok, I'll start combining them as I remove all the kerneldoc comments. I think I've gotten all the "CoW reservations left over at unmount" bugs out, so I'll probably have a slimmer patchset in a week or two. (Or everything will go nuts on some weird arch like arm64 and kablooie...) --D > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com From darrick.wong@oracle.com Wed Jan 6 18:33:02 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 8AA527F51 for ; Wed, 6 Jan 2016 18:33:02 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 79F04304059 for ; Wed, 6 Jan 2016 16:32:59 -0800 (PST) X-ASG-Debug-ID: 1452126776-04cb6c42e526040001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id cYQJ4h22De4QARGD (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 06 Jan 2016 16:32:57 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u070WUlI025290 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Thu, 7 Jan 2016 00:32:30 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u070WTYp011726 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Thu, 7 Jan 2016 00:32:30 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u070WTYX012153; Thu, 7 Jan 2016 00:32:29 GMT Received: from localhost (/10.145.178.207) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 06 Jan 2016 16:32:29 -0800 Date: Wed, 6 Jan 2016 16:32:27 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160107003227.GB8015@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> <20160105014310.GK28330@birch.djwong.org> <20160105104214.GA16310@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105104214.GA16310@infradead.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0021.oracle.com [156.151.31.71] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1452126777 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25902 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Tue, Jan 05, 2016 at 02:42:14AM -0800, Christoph Hellwig wrote: > On Mon, Jan 04, 2016 at 05:43:10PM -0800, Darrick J. Wong wrote: > > Hmm. This might be the cause of the occasional complaints I've been seeing > > where allocated blocks remain in the COW fork when the inode is being cleared > > out. That said, the xfs_reflink_end_cow_failed() is apparently missing a > > xfs_bunmapi_cow() to actually clean out the COW fork. > > I can still reproduce xfs_reflink_cancel_pending_cow tripping over > allocated blocks in the COW fork over NFS. generic/154 reproduces > it 100% over NFS, although when adding a delay before the cleanup > it disappears. I'm currently trying to figure out why. Ok. I spent a couple of days trying to find all the places where we need to delete CoW reservations (hole punch, truncate, etc.) and found some places where the code was leaving reservations behind in the CoW fork (most notable truncate). I also made the inode eviction code purge any CoW leftovers, so that should all go away. I also wrote some more xfstests that try to hit all the CoW-cancelling code paths (fpunch, fzero, fcollapse, finsert, truncate, -EIO) to smoke test all that. By the way, do you have a testcase handy for the "non-blocking writeback EAGAIN" case? I'm guessing that we could hit that pretty easily by lowering dirty_background_* and dirtying a lot of pages while reflinking? (Will push new code to github in a day or two.) --D From tytso@thunk.org Wed Jan 6 19:05:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 131C47F37 for ; Wed, 6 Jan 2016 19:05:16 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 8C2D8AC003 for ; Wed, 6 Jan 2016 17:05:12 -0800 (PST) X-ASG-Debug-ID: 1452128709-04cbb07e172a330001-NocioJ Received: from imap.thunk.org (imap.thunk.org [74.207.234.97]) by cuda.sgi.com with ESMTP id YytEKWkNFZ93KYrr (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 06 Jan 2016 17:05:10 -0800 (PST) X-Barracuda-Envelope-From: tytso@thunk.org X-Barracuda-Apparent-Source-IP: 74.207.234.97 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=thunk.org; s=ef5046eb; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=crlJiN1AsmHnrhOBd+9208/X3zJU0z/KfB7qjZzhgU4=; b=hCWWHYY3/H+I0SE3h361jPoi7T6ZVBA+u9sLaDSO6J3jiqvqT5ppuDJHUOENsTFeNrW/VY4opxlA9mshfeQuk2aNOHNGXDEqUqYEg7yKT1dmbnMUtnsGPJUc3AWeaoVvZ0HFMY+D7qz50pd6y9pAf5A7F+K+dCU9mU7zVRCeFsU=; Received: from root (helo=closure.thunk.org) by imap.thunk.org with local-esmtp (Exim 4.84) (envelope-from ) id 1aGz0l-0001qR-Bj; Thu, 07 Jan 2016 01:05:07 +0000 Received: by closure.thunk.org (Postfix, from userid 15806) id 253C1820456; Wed, 6 Jan 2016 20:05:06 -0500 (EST) Date: Wed, 6 Jan 2016 20:05:06 -0500 From: Theodore Ts'o To: Dave Chinner Cc: Jan Kara , xfs@oss.sgi.com, linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org Subject: Re: lazytime implementation questions Message-ID: <20160107010506.GB2866@thunk.org> X-ASG-Orig-Subj: Re: lazytime implementation questions References: <20160104062219.GB19802@dastard> <20160105173604.GE18604@quack.suse.cz> <20160105225907.GE21461@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105225907.GE21461@dastard> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: tytso@thunk.org X-SA-Exim-Scanned: No (on imap.thunk.org); SAEximRunCond expanded to false X-Barracuda-Connect: imap.thunk.org[74.207.234.97] X-Barracuda-Start-Time: 1452128710 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25903 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Wed, Jan 06, 2016 at 09:59:07AM +1100, Dave Chinner wrote: > > So the intended semantics is: > > 1) fsync / sync / freeze / unmount will write the timestamp updates even > > with lazytime. So unless crash happens, timestamps are guaranteed to be > > consistent. Also sync / fsync guarantees all changes to get to disk. > > 2) We periodically write back timestamps (once per 24 hours) to avoid too > > big timestamp inconsistencies in case of crash. > > Ok, so it's supposed to be a delayed timestamp update mechanism > without any specific ordering guarantees, not an opportunistic > timestamp update mechanism. There is an optimization which ext4 has which will update related timestamps when we write an inode table block, which is "opportunistic", but there is no guarantee that this will happen. This is purely optional; other file systems don't have to do this, but it can be a win in that if related inodes are in the same 4k block, and we need to update, say, the index file one because we are changing i_size, but we were also doing non-allocating writes to the data file, then we might as well write out the timestamps for the data file at the same time, since this is "free". - Ted From martin.petersen@oracle.com Wed Jan 6 19:41:05 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 97AB57F37 for ; Wed, 6 Jan 2016 19:41:05 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 7DEDA304059 for ; Wed, 6 Jan 2016 17:41:05 -0800 (PST) X-ASG-Debug-ID: 1452130860-04cbb07e192b490001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id mVNKjR5FoJZp2thu (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 06 Jan 2016 17:41:00 -0800 (PST) X-Barracuda-Envelope-From: martin.petersen@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u071eJpP015448 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Thu, 7 Jan 2016 01:40:19 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u071eHX1013059 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Thu, 7 Jan 2016 01:40:17 GMT Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15]) by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u071eE33000920; Thu, 7 Jan 2016 01:40:14 GMT Received: from ca-mkp.ca.oracle.com (/10.159.214.123) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 06 Jan 2016 17:40:14 -0800 To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs From: "Martin K. Petersen" X-ASG-Orig-Subj: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs Organization: Oracle References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> Date: Wed, 06 Jan 2016 20:40:09 -0500 In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> (mchristi@redhat.com's message of "Tue, 5 Jan 2016 14:53:03 -0600") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Source-IP: userv0021.oracle.com [156.151.31.71] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1452130860 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25904 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines >>>>> "Mike" == mchristi writes: Mike> The following patches begin to cleanup the request->cmd_flags and bio-> bi_rw mess. We currently use cmd_flags to specify the operation, Mike> attributes and state of the request. For bi_rw we use it for Mike> similar info and also the priority but then also have another Mike> bi_flags field for state. At some point, we abused them so much we Mike> just made cmd_flags 64 bits, so we could add more. Mike> The following patches seperate the operation (read, write discard, Mike> flush, etc) from cmd_flags/bi_rw. Mike> This patchset was made against linux-next from today Jan 5 2016. Mike> (git tag next-20160105). Very nice work. Thanks for doing this! I think it's a much needed cleanup. I focused mainly on the core block, discard, write same and sd.c pieces and everything looks sensible to me. I wonder what the best approach is to move a patch set with this many stakeholders forward? Set a "speak now or forever hold your peace" review deadline? -- Martin K. Petersen Oracle Linux Engineering From david@fromorbit.com Wed Jan 6 20:21:49 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 385277F37 for ; Wed, 6 Jan 2016 20:21:49 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 19D1C8F804C for ; Wed, 6 Jan 2016 18:21:46 -0800 (PST) X-ASG-Debug-ID: 1452133302-04cb6c42e628070001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id iEB476Lu7OzyDMvc for ; Wed, 06 Jan 2016 18:21:43 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2COCwA4yo1WPGu7LXleDhoBAoMPgT+CYoN+gXmgKAEBAQEBAQaLWIk/hgkEAgKBHk0BAQEBAQEHAQEBAUE/QRIBg2ABAQEDATocIwULCAMYCSUPBSUDBxoTHogJB8F5AQEIAiEZhXWFR4k8AQSXCo1MgWWNHoVeiGmEMFIqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 12:51:41 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aH0Cr-0000Y8-0c; Thu, 07 Jan 2016 13:21:41 +1100 Date: Thu, 7 Jan 2016 13:21:40 +1100 From: Dave Chinner To: Theodore Ts'o Cc: Jan Kara , xfs@oss.sgi.com, linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org Subject: Re: lazytime implementation questions Message-ID: <20160107022140.GM21461@dastard> X-ASG-Orig-Subj: Re: lazytime implementation questions References: <20160104062219.GB19802@dastard> <20160105173604.GE18604@quack.suse.cz> <20160105225907.GE21461@dastard> <20160107010506.GB2866@thunk.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107010506.GB2866@thunk.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452133302 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25905 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed, Jan 06, 2016 at 08:05:06PM -0500, Theodore Ts'o wrote: > On Wed, Jan 06, 2016 at 09:59:07AM +1100, Dave Chinner wrote: > > > So the intended semantics is: > > > 1) fsync / sync / freeze / unmount will write the timestamp updates even > > > with lazytime. So unless crash happens, timestamps are guaranteed to be > > > consistent. Also sync / fsync guarantees all changes to get to disk. > > > 2) We periodically write back timestamps (once per 24 hours) to avoid too > > > big timestamp inconsistencies in case of crash. > > > > Ok, so it's supposed to be a delayed timestamp update mechanism > > without any specific ordering guarantees, not an opportunistic > > timestamp update mechanism. > > There is an optimization which ext4 has which will update related > timestamps when we write an inode table block, which is > "opportunistic", but there is no guarantee that this will happen. XFS used to do that, too, before we removed all that hackery when we moved to logging timestamp updates unconditionally a few years ago. I'm going to have to re-instate some of that code for lazytime, I think. > This is purely optional; other file systems don't have to do this, but > it can be a win in that if related inodes are in the same 4k block, > and we need to update, say, the index file one because we are changing > i_size, but we were also doing non-allocating writes to the data file, > then we might as well write out the timestamps for the data file at > the same time, since this is "free". *nod*. Explicit, optimised clustered inode writeback (rather than purely opportunistic clustering via delayed buffer writeback) was added to XFS way back in early 1999. :) Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Wed Jan 6 21:09:55 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 76AFC7F37 for ; Wed, 6 Jan 2016 21:09:55 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id DE264AC001 for ; Wed, 6 Jan 2016 19:09:51 -0800 (PST) X-ASG-Debug-ID: 1452136187-04cb6c42e528fb0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id HwoXGdbvj3A036Do for ; Wed, 06 Jan 2016 19:09:47 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AEDAAW1o1WPGu7LXleKAECgw+BP4Jig36BeaAoAQEBAQEBBotYhTqEBYYJAgIBAQKBHk0BAQEBAQEHAQEBAUE/QRIBg2EBAQQnExwzCBgMJQ8FJQMHLYguwgAMIRmFdY1ogRsFh1QHhV6BNIgdjUyBZYdthTGFXohpgnUcgXEqNIF6ghwEgUcBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 13:38:31 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aH0wA-0000dB-Q7 for xfs@oss.sgi.com; Thu, 07 Jan 2016 14:08:30 +1100 Date: Thu, 7 Jan 2016 14:08:30 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH v2] xfs: handle dquot buffer readahead in log recovery correctly Message-ID: <20160107030830.GA32200@dastard> X-ASG-Orig-Subj: [PATCH v2] xfs: handle dquot buffer readahead in log recovery correctly References: <1452052834-20605-1-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452052834-20605-1-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452136187 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25905 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner When we do dquot readahead in log recovery, we do not use a verifier as the underlying buffer may not have dquots in it. e.g. the allocation operation hasn't yet been replayed. Hence we do not want to fail recovery because we detect an operation to be replayed has not been run yet. This problem was addressed for inodes in commit d891400 ("xfs: inode buffers may not be valid during recovery readahead") but the problem was not recognised to exist for dquots and their buffers as the dquot readahead did not have a verifier. The result of not using a verifier is that when the buffer is then next read to replay a dquot modification, the dquot buffer verifier will only be attached to the buffer if *readahead is not complete*. Hence we can read the buffer, replay the dquot changes and then add it to the delwri submission list without it having a verifier attached to it. This then generates warnings in xfs_buf_ioapply(), which catches and warns about this case. Fix this and make it handle the same readahead verifier error cases as for inode buffers by adding a new readahead verifier that has a write operation as well as a read operation that marks the buffer as not done if any corruption is detected. Also make sure we don't run readahead if the dquot buffer has been marked as cancelled by recovery. This will result in readahead either succeeding and the buffer having a valid write verifier, or readahead failing and the buffer state requiring the subsequent read to resubmit the IO with the new verifier. In either case, this will result in the buffer always ending up with a valid write verifier on it. Note: we also need to fix the inode buffer readahead error handling to mark the buffer with EIO. Brian noticed the code I copied from there wrong during review, so fix it at the same time. Add comments linking the two functions that handle readahead verifier errors together so we don't forget this behavioural link in future. cc: # 3.12 - current Signed-off-by: Dave Chinner --- Version 2 - fix logic error in determining if verify failed - set error on buffer when verifier fails - fix inode buffer readahead verifier to set error when it fails - better comments, link dquot and inode buffer ra verifiers in the comments fs/xfs/libxfs/xfs_dquot_buf.c | 36 ++++++++++++++++++++++++++++++------ fs/xfs/libxfs/xfs_inode_buf.c | 14 +++++++++----- fs/xfs/libxfs/xfs_quota_defs.h | 2 +- fs/xfs/libxfs/xfs_shared.h | 1 + fs/xfs/xfs_log_recover.c | 9 +++++++-- 5 files changed, 48 insertions(+), 14 deletions(-) diff --git a/fs/xfs/libxfs/xfs_dquot_buf.c b/fs/xfs/libxfs/xfs_dquot_buf.c index 11cefb2..3cc3cf7 100644 --- a/fs/xfs/libxfs/xfs_dquot_buf.c +++ b/fs/xfs/libxfs/xfs_dquot_buf.c @@ -54,7 +54,7 @@ xfs_dqcheck( xfs_dqid_t id, uint type, /* used only when IO_dorepair is true */ uint flags, - char *str) + const char *str) { xfs_dqblk_t *d = (xfs_dqblk_t *)ddq; int errs = 0; @@ -207,7 +207,8 @@ xfs_dquot_buf_verify_crc( STATIC bool xfs_dquot_buf_verify( struct xfs_mount *mp, - struct xfs_buf *bp) + struct xfs_buf *bp, + int warn) { struct xfs_dqblk *d = (struct xfs_dqblk *)bp->b_addr; xfs_dqid_t id = 0; @@ -240,8 +241,7 @@ xfs_dquot_buf_verify( if (i == 0) id = be32_to_cpu(ddq->d_id); - error = xfs_dqcheck(mp, ddq, id + i, 0, XFS_QMOPT_DOWARN, - "xfs_dquot_buf_verify"); + error = xfs_dqcheck(mp, ddq, id + i, 0, warn, __func__); if (error) return false; } @@ -256,7 +256,7 @@ xfs_dquot_buf_read_verify( if (!xfs_dquot_buf_verify_crc(mp, bp)) xfs_buf_ioerror(bp, -EFSBADCRC); - else if (!xfs_dquot_buf_verify(mp, bp)) + else if (!xfs_dquot_buf_verify(mp, bp, XFS_QMOPT_DOWARN)) xfs_buf_ioerror(bp, -EFSCORRUPTED); if (bp->b_error) @@ -264,6 +264,25 @@ xfs_dquot_buf_read_verify( } /* + * readahead errors are silent and simply leave the buffer as !done so a real + * read will then be run with the xfs_dquot_buf_ops verifier. See + * xfs_inode_buf_verify() for why we use EIO and ~XBF_DONE here rather than + * reporting the failure. + */ +static void +xfs_dquot_buf_readahead_verify( + struct xfs_buf *bp) +{ + struct xfs_mount *mp = bp->b_target->bt_mount; + + if (!xfs_dquot_buf_verify_crc(mp, bp) || + !xfs_dquot_buf_verify(mp, bp, 0)) { + xfs_buf_ioerror(bp, -EIO); + bp->b_flags &= ~XBF_DONE; + } +} + +/* * we don't calculate the CRC here as that is done when the dquot is flushed to * the buffer after the update is done. This ensures that the dquot in the * buffer always has an up-to-date CRC value. @@ -274,7 +293,7 @@ xfs_dquot_buf_write_verify( { struct xfs_mount *mp = bp->b_target->bt_mount; - if (!xfs_dquot_buf_verify(mp, bp)) { + if (!xfs_dquot_buf_verify(mp, bp, XFS_QMOPT_DOWARN)) { xfs_buf_ioerror(bp, -EFSCORRUPTED); xfs_verifier_error(bp); return; @@ -287,3 +306,8 @@ const struct xfs_buf_ops xfs_dquot_buf_ops = { .verify_write = xfs_dquot_buf_write_verify, }; +const struct xfs_buf_ops xfs_dquot_buf_ra_ops = { + .name = "xfs_dquot_ra", + .verify_read = xfs_dquot_buf_readahead_verify, + .verify_write = xfs_dquot_buf_write_verify, +}; diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index 1b8d98a..4816209 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -62,11 +62,14 @@ xfs_inobp_check( * has not had the inode cores stamped into it. Hence for readahead, the buffer * may be potentially invalid. * - * If the readahead buffer is invalid, we don't want to mark it with an error, - * but we do want to clear the DONE status of the buffer so that a followup read - * will re-read it from disk. This will ensure that we don't get an unnecessary - * warnings during log recovery and we don't get unnecssary panics on debug - * kernels. + * If the readahead buffer is invalid, we need to mark it with an error and + * clear the DONE status of the buffer so that a followup read will re-read it + * from disk. We don't report the error otherwise to avoid warnings during log + * recovery and we don't get unnecssary panics on debug kernels. We use EIO here + * because all we want to do is say readahead failed; there is no-one to report + * the error to, so this will distinguish it from a non-ra verifier failure. + * Changes to this readahead error behavour also need to be reflected in + * xfs_dquot_buf_readahead_verify(). */ static void xfs_inode_buf_verify( @@ -92,6 +95,7 @@ xfs_inode_buf_verify( XFS_ERRTAG_ITOBP_INOTOBP, XFS_RANDOM_ITOBP_INOTOBP))) { if (readahead) { + xfs_buf_ioerror(bp, -EIO); bp->b_flags &= ~XBF_DONE; return; } diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h index 1b0a083..f51078f 100644 --- a/fs/xfs/libxfs/xfs_quota_defs.h +++ b/fs/xfs/libxfs/xfs_quota_defs.h @@ -153,7 +153,7 @@ typedef __uint16_t xfs_qwarncnt_t; #define XFS_QMOPT_RESBLK_MASK (XFS_QMOPT_RES_REGBLKS | XFS_QMOPT_RES_RTBLKS) extern int xfs_dqcheck(struct xfs_mount *mp, xfs_disk_dquot_t *ddq, - xfs_dqid_t id, uint type, uint flags, char *str); + xfs_dqid_t id, uint type, uint flags, const char *str); extern int xfs_calc_dquots_per_chunk(unsigned int nbblks); #endif /* __XFS_QUOTA_H__ */ diff --git a/fs/xfs/libxfs/xfs_shared.h b/fs/xfs/libxfs/xfs_shared.h index 5be5297..15c3ceb 100644 --- a/fs/xfs/libxfs/xfs_shared.h +++ b/fs/xfs/libxfs/xfs_shared.h @@ -49,6 +49,7 @@ extern const struct xfs_buf_ops xfs_inobt_buf_ops; extern const struct xfs_buf_ops xfs_inode_buf_ops; extern const struct xfs_buf_ops xfs_inode_buf_ra_ops; extern const struct xfs_buf_ops xfs_dquot_buf_ops; +extern const struct xfs_buf_ops xfs_dquot_buf_ra_ops; extern const struct xfs_buf_ops xfs_sb_buf_ops; extern const struct xfs_buf_ops xfs_sb_quiet_buf_ops; extern const struct xfs_buf_ops xfs_symlink_buf_ops; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 26e67b4..da37beb 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -3521,6 +3521,7 @@ xlog_recover_dquot_ra_pass2( struct xfs_disk_dquot *recddq; struct xfs_dq_logformat *dq_f; uint type; + int len; if (mp->m_qflags == 0) @@ -3541,8 +3542,12 @@ xlog_recover_dquot_ra_pass2( ASSERT(dq_f); ASSERT(dq_f->qlf_len == 1); - xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, - XFS_FSB_TO_BB(mp, dq_f->qlf_len), NULL); + len = XFS_FSB_TO_BB(mp, dq_f->qlf_len); + if (xlog_peek_buffer_cancelled(log, dq_f->qlf_blkno, len, 0)) + return; + + xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, len, + &xfs_dquot_buf_ra_ops); } STATIC void From david@fromorbit.com Wed Jan 6 21:11:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 07D247F37 for ; Wed, 6 Jan 2016 21:11:41 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id ECE5A8F8040 for ; Wed, 6 Jan 2016 19:11:37 -0800 (PST) X-ASG-Debug-ID: 1452136293-04cb6c42e5290e0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 2Jl3z8YVIpl9IzDC for ; Wed, 06 Jan 2016 19:11:33 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AGDAAW1o1WPGu7LXleKAECgw+BP4Jig36BeaAoAQEBAQEBBotYhTqEBYYJAgIBAQKBHk0BAQEBAQEHAQEBAUE/QRIBg2ABAQEEOhwjEAgDFQMJJQ8FJQMHLYguwgABAQgCASAZhXWFR4Ujgn6BGwEEh16LLIQAjUyBZY0eRIoQg3OFAio0gXqDZwEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 13:41:32 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aH0z5-0000dc-7P; Thu, 07 Jan 2016 14:11:31 +1100 Date: Thu, 7 Jan 2016 14:11:31 +1100 From: Dave Chinner To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 13/35] xfs: set bi_op to REQ_OP Message-ID: <20160107031131.GB32200@dastard> X-ASG-Orig-Subj: Re: [PATCH 13/35] xfs: set bi_op to REQ_OP References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-14-git-send-email-mchristi@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452027218-32303-14-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452136293 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25905 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 02:53:16PM -0600, mchristi@redhat.com wrote: > From: Mike Christie > > This patch has xfs set the bio bi_op to a REQ_OP, and > rq_flag_bits to bi_rw. > > Note: > I have run xfs tests on these btrfs patches. There were some failures > with and without the patches. I have not had time to track down why > xfstest fails without the patches. > > Signed-off-by: Mike Christie > --- > fs/xfs/xfs_aops.c | 3 ++- > fs/xfs/xfs_buf.c | 27 +++++++++++++++------------ > 2 files changed, 17 insertions(+), 13 deletions(-) Not sure which patches your note is refering to here. The XFS change here looks fine. Acked-by: Dave Chinner -Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Wed Jan 6 21:14:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 54FA67F37 for ; Wed, 6 Jan 2016 21:14:13 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 38700304039 for ; Wed, 6 Jan 2016 19:14:10 -0800 (PST) X-ASG-Debug-ID: 1452136446-04cbb07e172d600001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id bgF9Gvy5PuYa3vWI for ; Wed, 06 Jan 2016 19:14:07 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AGDABM141WPGu7LXleKAECgw+BP4Jig36BeaAoAQEBAQEBBotYhTqEBYYJAgIBAQKBHk0BAQEBAQEHAQEBAUE/QRIBg2EBAQQnExwjEAgDGAklDwUlAwcaE4guwX8BAQgCIRmFdYVHiCGBGwEElwqNTIFlh22FMYpUg3OFAio0gXqDZwEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jan 2016 13:43:42 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aH11B-0000dr-Mn; Thu, 07 Jan 2016 14:13:41 +1100 Date: Thu, 7 Jan 2016 14:13:41 +1100 From: Dave Chinner To: "Martin K. Petersen" Cc: mchristi@redhat.com, linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs Message-ID: <20160107031341.GC32200@dastard> X-ASG-Orig-Subj: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452136447 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25905 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed, Jan 06, 2016 at 08:40:09PM -0500, Martin K. Petersen wrote: > >>>>> "Mike" == mchristi writes: > > Mike> The following patches begin to cleanup the request->cmd_flags and > bio-> bi_rw mess. We currently use cmd_flags to specify the operation, > Mike> attributes and state of the request. For bi_rw we use it for > Mike> similar info and also the priority but then also have another > Mike> bi_flags field for state. At some point, we abused them so much we > Mike> just made cmd_flags 64 bits, so we could add more. > > Mike> The following patches seperate the operation (read, write discard, > Mike> flush, etc) from cmd_flags/bi_rw. > > Mike> This patchset was made against linux-next from today Jan 5 2016. > Mike> (git tag next-20160105). > > Very nice work. Thanks for doing this! > > I think it's a much needed cleanup. I focused mainly on the core block, > discard, write same and sd.c pieces and everything looks sensible to me. > > I wonder what the best approach is to move a patch set with this many > stakeholders forward? Set a "speak now or forever hold your peace" > review deadline? I say just ask Linus to pull it immediately after the next merge window closes.... Cheers, Dave. -- Dave Chinner david@fromorbit.com From debbugs@buxtehude.debian.org Wed Jan 6 21:51:28 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4EC2F7F37 for ; Wed, 6 Jan 2016 21:51:28 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3AF06304039 for ; Wed, 6 Jan 2016 19:51:28 -0800 (PST) X-ASG-Debug-ID: 1452138683-04cbb07e172e520001-NocioJ Received: from buxtehude.debian.org (buxtehude.debian.org [206.12.19.147]) by cuda.sgi.com with ESMTP id ARbZVu9EGFACuBGW (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 06 Jan 2016 19:51:24 -0800 (PST) X-Barracuda-Envelope-From: debbugs@buxtehude.debian.org X-Barracuda-Apparent-Source-IP: 206.12.19.147 Received: from debbugs by buxtehude.debian.org with local (Exim 4.84) (envelope-from ) id 1aH1bd-0001TN-Fl; Thu, 07 Jan 2016 03:51:21 +0000 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: owner@bugs.debian.org From: owner@bugs.debian.org (Debian Bug Tracking System) To: Marco d'Itri Subject: Bug#766811: marked as done (spurious library links) Message-ID: X-ASG-Orig-Subj: Bug#766811: marked as done (spurious library links) References: <20160107035001.GA17996@bongo.bofh.it> <20141026004825.GA27863@bongo.bofh.it> X-Debian-PR-Message: closed 766811 X-Debian-PR-Package: xfslibs-dev X-Debian-PR-Source: xfsprogs Date: Thu, 07 Jan 2016 03:51:21 +0000 Content-Type: multipart/mixed; boundary="----------=_1452138681-5649-0" X-Barracuda-Connect: buxtehude.debian.org[206.12.19.147] X-Barracuda-Start-Time: 1452138684 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25906 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header This is a multi-part message in MIME format... ------------=_1452138681-5649-0 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Your message dated Thu, 7 Jan 2016 04:50:01 +0100 with message-id <20160107035001.GA17996@bongo.bofh.it> and subject line Re: Bug#766811: spurious library links has caused the Debian Bug report #766811, regarding spurious library links to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner@bugs.debian.org immediately.) --=20 766811: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D766811 Debian Bug Tracking System Contact owner@bugs.debian.org with problems ------------=_1452138681-5649-0 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by bugs.debian.org; 26 Oct 2014 00:48:49 +0000 X-Spam-Checker-Version: SpamAssassin 3.3.2-bugs.debian.org_2005_01_02 (2011-06-06) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-18.0 required=4.0 tests=BAYES_00,HAS_PACKAGE, PGPSIGNATURE,X_DEBBUGS_CC autolearn=ham version=3.3.2-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 13; hammy, 74; neutral, 24; spammy, 0. spammytokens: hammytokens:0.000-+--H*c:pgp-sha256, 0.000-+--H*u:2014-03-12, 0.000-+--H*UA:2014-03-12, 0.000-+--H*u:1.5.23, 0.000-+--H*UA:1.5.23 Return-path: Received: from attila.bofh.it ([85.94.204.146] ident=postfix) by buxtehude.debian.org with esmtp (Exim 4.80) (envelope-from ) id 1XiC0n-000744-4E for submit@bugs.debian.org; Sun, 26 Oct 2014 00:48:49 +0000 Received: by attila.bofh.it (Postfix, from userid 10) id 166B82E384; Sun, 26 Oct 2014 02:48:46 +0200 (CEST) Received: by bongo.bofh.it (Postfix, from userid 1000) id 762652C26AC; Sun, 26 Oct 2014 02:48:25 +0200 (CEST) Date: Sun, 26 Oct 2014 02:48:25 +0200 From: Marco d'Itri To: Debian Bug Tracking System Subject: spurious library links Message-ID: <20141026004825.GA27863@bongo.bofh.it> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="T4sUOijqQbZv57TR" Content-Disposition: inline X-Reportbug-Version: 6.5.1 X-Debbugs-Cc: md@linux.it User-Agent: Mutt/1.5.23 (2014-03-12) Delivered-To: submit@bugs.debian.org --T4sUOijqQbZv57TR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Package: xfslibs-dev Version: 3.2.1 Severity: normal These links do not appear to have any purpose and should be removed: /lib/libhandle.a -> /usr/lib/libhandle.a /lib/libhandle.la -> /usr/lib/libhandle.la /usr/lib/libhandle.so -> /lib/libhandle.so Also, policy forbids to thip the .la files at all for normal libraries. --=20 ciao, Marco --T4sUOijqQbZv57TR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQGcBAEBCAAGBQJUTETZAAoJECv1pcEixPekDUAL/1ZETnJTeLt6DTKimbHXxsJW gXC5vyI9/x50gPeGZM0QZDeoOqXkmn2NHww24ljtlar7SjyyKcsHVvWQ3e6h2GoW vrfTfpAmsFNlx9QqolFiiLFExJbPVUFLTuINWTzDIPR+UjMlz3k0H1x5adrRbmmX 7KfscCiEVBbo+QSdFFVd3oiBriwsBLPhLxIEVwBOvBKzG5Vx+SIfPIcbS1VMMi3L 895Xj2cq4ZCOMb7bCOFybRUm6lpzK9y5gvkW7JyF56KHCr+wzXqy+NXmv6IbyoA5 G63gITsd9lwZ+f3kTYAFgZvirSVKt30deotZTmo0L8uFjMfvN4RcaeH7JP9pN37k xgKoHeJRYPfI/scpQqNVBrGhrNke9FSbBpZt0qKBUg7NBuuncy4c1c8OUWCxehP5 iVzkTCMHzii1BtqjqiVhuwj6LZTj2hR170O1Sw2uRVFO++YVpyaQWt+zYSZkEptZ 5qQ3RGngc5ddu1V9SxTldZR0RoB/KvGruCzecNewBw== =1jn8 -----END PGP SIGNATURE----- --T4sUOijqQbZv57TR-- ------------=_1452138681-5649-0 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 766811-done) by bugs.debian.org; 7 Jan 2016 03:50:18 +0000 X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 (2014-02-07) on buxtehude.debian.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=4.0 tests=BAYES_00,HAS_BUG_NUMBER, PGPSIGNATURE,RCVD_IN_DNSWL_NONE,VERSION autolearn=ham autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 X-Spam-Bayes: score:0.0000 Tokens: new, 5; hammy, 77; neutral, 23; spammy, 0. spammytokens: hammytokens:0.000-+--H*c:pgp-sha256, 0.000-+--H*UA:1.5.24, 0.000-+--H*UA:2015-08-30, 0.000-+--H*u:1.5.24, 0.000-+--H*u:2015-08-30 Return-path: Received: from attila.bofh.it ([85.94.204.146] ident=postfix) by buxtehude.debian.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84) (envelope-from ) id 1aH1ab-0001Db-Ta for 766811-done@bugs.debian.org; Thu, 07 Jan 2016 03:50:18 +0000 Received: by attila.bofh.it (Postfix, from userid 10) id 2104C2E4B9; Thu, 7 Jan 2016 04:50:13 +0100 (CET) Received: by bongo.bofh.it (Postfix, from userid 1000) id 8D7D72C0057; Thu, 7 Jan 2016 04:50:01 +0100 (CET) Date: Thu, 7 Jan 2016 04:50:01 +0100 From: Marco d'Itri To: 766811-done@bugs.debian.org Subject: Re: Bug#766811: spurious library links Message-ID: <20160107035001.GA17996@bongo.bofh.it> References: <20141026004825.GA27863@bongo.bofh.it> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="sm4nu43k4a2Rpi4c" Content-Disposition: inline In-Reply-To: <20141026004825.GA27863@bongo.bofh.it> User-Agent: Mutt/1.5.24 (2015-08-30) --sm4nu43k4a2Rpi4c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Version: 4.3.0 On Oct 26, Marco d'Itri wrote: > These links do not appear to have any purpose and should be removed: But this was fixed at least in 4.3.0, so we are done with this! --=20 ciao, Marco --sm4nu43k4a2Rpi4c Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGcBAABCAAGBQJWjeBpAAoJECv1pcEixPeksL0L/3AMkxmD/Vfy4os0X9+wK0RR NpmfqVv/5pHzStEeQq8a48zFeg2YUX6MaStt/bCUVvOWzP7QMcR4Qflv/dJO2RCs reZzEau8kIIr9YYN0KF/v5i2sFnv1Aqz8ZnrebZxVu4gY7crYc37yLsnQlKnEMoU PKe34xWi5sUNk1hWqDni18s6qdn3jN39XcUh4ZKmA/Fpd7v8eUBsDILEvdg5N4QQ wxMN7hymy/15uUIDov8wRdvEKlppDCqkMmBYJ5sJ19Inx8cRD6w6rSbuRtjb3kCP yeuTR8dbzC0luZDjDysIJBym7eAeNaoNMqKmKAmgYtGbNhlqoAHdju1uNzqWQfM5 trvQXeMGcxcnEB/nWvYmZ4gb1Q2Z5EIduIkqgeuylEu0NRd9m8lcSgnMPKF59mNC 3pq8Y44MBENnbagPLuXjQ1abJRbwIAwgbhr0aVvU5BpcrIkOJgra5ne/askUIBJH tmaoJ9C/FcEo/zzGCa13jG9/WayYdlbBUb9pJ1k3Iw== =oUk6 -----END PGP SIGNATURE----- --sm4nu43k4a2Rpi4c-- ------------=_1452138681-5649-0-- From tore.larsen@dolphingeo.com Thu Jan 7 02:54:53 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: *** X-Spam-Status: No, score=3.4 required=5.0 tests=DC_GIF_UNO_LARGO, DC_IMAGE_SPAM_HTML,DC_IMAGE_SPAM_TEXT,HTML_IMAGE_ONLY_20,HTML_MESSAGE, HTML_SHORT_LINK_IMG_3 autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6401C7F37 for ; Thu, 7 Jan 2016 02:54:53 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 429628F8066 for ; Thu, 7 Jan 2016 00:54:53 -0800 (PST) X-ASG-Debug-ID: 1452156884-04cbb07e1634b10001-NocioJ Received: from mx1.dolphingeo.com (mx1.dolphingeo.com [82.134.93.210]) by cuda.sgi.com with ESMTP id wLVHm6NxI4ncoCFe (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 07 Jan 2016 00:54:45 -0800 (PST) X-Barracuda-Envelope-From: tore.larsen@dolphingeo.com X-Barracuda-Apparent-Source-IP: 82.134.93.210 Received: from bgo-ld04.dolphingeo.com (bgo-ld04.int.dolphingeo.com [10.200.38.15]) by mx1.dolphingeo.com with ESMTP id pv1SK0o9qVKCZrm0 for ; Thu, 07 Jan 2016 09:54:42 +0100 (CET) To: xfs@oss.sgi.com MIME-Version: 1.0 Subject: Possible XFS or NFS bug in Latest RHEL 6.6 2.6.32-504.23.4.el6.x86_64 ? X-KeepSent: E4A99C77:ACF09656-C1257F33:002FF7B8; type=4; name=$KeepSent X-ASG-Orig-Subj: Possible XFS or NFS bug in Latest RHEL 6.6 2.6.32-504.23.4.el6.x86_64 ? X-Mailer: IBM Notes Release 9.0.1FP4 Octobe4, 2013 Message-ID: From: tore.larsen@dolphingeo.com Date: Thu, 7 Jan 2016 09:54:43 +0100 X-MIMETrack: Serialize by Router on BGO-LD04/DOLPHINGEO(Release 9.0.1FP2 HF384|November 04, 2014) at 2016-01-07 09:54:42, Serialize complete at 2016-01-07 09:54:42 Content-Type: multipart/related; boundary="=_related 0030F437C1257F33_=" X-Virus-Scanned: by bsmtpd at dolphingeo.com X-Barracuda-Connect: mx1.dolphingeo.com[82.134.93.210] X-Barracuda-Start-Time: 1452156884 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 1.82 X-Barracuda-Spam-Status: No, SCORE=1.82 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DC_GIF_UNO_LARGO, DC_IMAGE_SPAM_HTML, DC_IMAGE_SPAM_TEXT, HTML_IMAGE_ONLY_20, HTML_IMAGE_ONLY_20_2, HTML_MESSAGE, HTML_SHORT_LINK_IMG_3, HTML_SHORT_LINK_IMG_3_2, NO_REAL_NAME X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25911 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 NO_REAL_NAME From: does not include a real name 0.00 HTML_IMAGE_ONLY_20 BODY: HTML: images with 1600-2000 bytes of words 0.00 HTML_MESSAGE BODY: HTML included in message 0.70 HTML_IMAGE_ONLY_20_2 HTML: images with 1600-2000 bytes of words 0.00 HTML_SHORT_LINK_IMG_3 HTML is very short with a linked image 0.79 DC_GIF_UNO_LARGO Message contains a single large inline gif 0.33 HTML_SHORT_LINK_IMG_3_2 HTML is very short with a linked image 0.00 DC_IMAGE_SPAM_TEXT Possible Image-only spam with little text 0.00 DC_IMAGE_SPAM_HTML Possible Image-only spam This is a multipart message in MIME format. --=_related 0030F437C1257F33_= Content-Type: multipart/alternative; boundary="=_alternative 0030F437C1257F33_=" --=_alternative 0030F437C1257F33_= Content-Type: text/plain; charset="US-ASCII" Hi XFS'ers, Anyone seen this? SGI MIS server. Been stable for months. Resembles https://lkml.org/lkml/2014/5/23/261 Kind Regards / Mvh, Tore H. Larsen Sr. Systems Engineer Dolphin Geophysical Direct Dial: +47 55 38 75 23 Email: torel@dolphingeo.com Mobile: +47 918 33 670 Web: www.dolphingeo.com --=_alternative 0030F437C1257F33_= Content-Type: text/html; charset="US-ASCII" Hi XFS'ers,

     Anyone seen this?     SGI MIS server.  Been stable for months.

     Resembles https://lkml.org/lkml/2014/5/23/261  







Kind Regards / Mvh,

Tore H. Larsen
Sr. Systems Engineer
Dolphin Geophysical

Direct Dial:  +47 55 38 75 23
Email:          torel@dolphingeo.com
Mobile:        +47 918 33 670
Web:            www.dolphingeo.com

--=_alternative 0030F437C1257F33_=-- --=_related 0030F437C1257F33_= Content-Type: image/gif Content-ID: <_1_0C496CA40C496A1C0030F437C1257F33> Content-Transfer-Encoding: base64 R0lGODlhiwLMAecAAKioqAAAAOjo6P///+Dg4PDw8Ljg+Oj4+LDY+KjQ+PD4+KjY+KDQ+KDI+HiI mODw+OD4+MjIyNjY2LDg+MDo+GBweNDo+JjI8Njo+HB4eDAwMJjI+Nj4+FAwMNjw+NDg8Hh4eDAw UND4+MjY6FCQyMiQUOiwcGh4eMDg+DAwcJDI6HCw6BAQEFCQ0Mjw+NDw+JCQkAAACOjIkLDo6DBw sHAwMKDI8Ljw+Mjo+LBwMNCQUOjQkMj4+OjosCCAwMD4+JDQ6AAICNDg+ACAyPDQkKDI6EgAUNDY 6AAAiKjQ8AAAUABQsDBQkJDQ8JjA4MDw+HgAAJhIALCAAMioUHjA+JBQMJC42Njg8OCwcABIsAgI CBA4UBBAWKDI4Kh4AMCgUNDIqAAAgAB4wBAwSBA4WKDA2Mjg8HAAAJCw0ABIqCB4uHCw4MDQ4ODI kBAoQFBoeHi48MjY8NDg2NjQsAA4eCB4sChomEAAAECg2EgAAHCw8JjQ+KBQAKC4yKjg8NjoyBAw mBhQcCAgICBQSDhIGDhgYEBIQEgAiEiAaFB4mFhoUHAwcHBQUHBwMHCYuHiAUIC4kIigeJCYSKBQ UKDAYLC4kLjooMDYeNDQYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAAAiwLMAUAI/wAL9TlA sKDBgwoSKlzIsKFDhQcjSpxIsaLFixgzatzIUeKDjyBDinzgAYPJkyhTqjxpoaXLly1ZwpxJs6bN mzhz6tzJs6fPn0CDCh1KtKjRo0JXKl3K9IZTp3TocAlEhg4ZLlyiasX4sCvEjmDDij3gVaJXBRbR Opx4NmFGCBA4cHjhgoIBAxMmINiLYIHfBQkSMLBBuDADBoET/F3M2G/ix48PS55MubLly5gza97M ubPnz6BDix5NurRkyKhTq17NurXr17Bjv+ZBm7ZWMiFCaCAEptIUSoPIQChp4YPxCGdWjBiBPACe I0fAGFGBJQCckw/aBFjiYkqAIdmtX/8xLicKnisYvgw5+QcKHONfwvgBc2eFdOpKLhj/UF7Mkfh+ GIfBHEbs0R4VbeT3AYBepDHCEca1gYQfFqSXBQUnCSHHGXgsKN9yy2GRBhsklgjiCGbwF8UQC054 ohdijECfGHHQp0cccZhRCRILWOCBB3ARxFBBcMn1Ag5IIvmCCBxAcFCRItRl11157ZVAEkUU4YQV XHLphBNFJOEXAnmVaeaZVfKl5ppstunmm3DGKeecdNZp55145qnnnnz26eefgAYKJ1yEFhpRW0NW hOiijDY01qOQRiopRYRyECUOFFCAwl2cdurpp6CGKuqopJZq6qmopqrqqqy26uqrsMb/KuustNaa qpAZDKDrrrz26uuvwAYr7LDEFmvsscgmq+yyzDbr7LPQRivttNRWa+212Gar7bbJ4irAt+AKoEMK M/RQQwsCmKDBCjqEAIQAOZDb7rvqsuuuADt0gG64/IKb7xuG6PvtvOmumy+75KqLbr4tEMzwt+pq 4K65NAx8b7wz9KsxwzmsYAIN8zqsr8LwkpsvueASXK/G/RL8LcMkM0xxwfa+y/DDGJ+cMcsbCywA xSob3AEQJrhb78op34s0zy93oMHTEgMRtMfrxuwz01hnrfXWXHft9ddghy322GSXbfbZaDONKwFs t+3223DHLffcdNdt991456333nz3/+3334AHLvjghBdu+OGIJ6744ow3vjdcCpzgQACUV245AJgD YPnmlWe+ueeXY/656KFrXvropndOOuelr56566+jfjrloMvOOuqp3/567rfrvnrvvdeuuu3AF8/6 78YnHzzvATCvfPOxFy889M7jXv3orae+u/HRAz+978g/L/745Jdv/vnoj98v5JJ7Xz3snIcvP/Pz x08/76Tnrz3+psOv//DYgx7tnBe+ASavgAH8X+gA6LsGBlCABnQg8uAXwePtTnQInKDmMLi/5QFQ gfVb4ANBmLsCKhCCHhThCSuowfS58IUwjKEM1ccvuRygApNbHgJnpzr++ZB4A/whD/91CLvabW+I zzui+3YYRCYu8Xzf0+D1oDhFGcqPit+znvS6577s9dCJ1Ksi9Q6YxRma8YxoTKP4+lWXB7whhw60 HwPnWEE6otCOIfxg/zqowg6ScHj+4+MdRbhF7u2xjihkYgbvx8AdrjCRgrTfBQdJRw5CMoUQ/F8g 5RjBFZrwkIjkpB4RSUE7qvGUqEylGdlIATfC0YJgnJ4spQjEMSJReEoM5e9yacsefnGPYvziFstY S1Ua05eovKIVuQg+ZBqSmIQkJC+PSc1qWvOM/XKEI9gQgleKcoSMfCAewxlNUkYyj5vMJCgpeMJF krGQ6jQlCzfIR01G8puX/OMdmWn/Pc+5c4GWnKci67nOcAZUmgUtZyfvec2GOvShyuuXHezQBzd4 E3fPrOUshejMIAKRl7vs3jTJaER+SrKKI0VfMCGaPmiS1IwEXKnyUvrLYYJRoDdlqU53mj6JUnQM F1XoPPFJyTyOE5+lvKQ5h5pOAZbyn/BMoT4NeNBHbpKDTwVdUvsZPXbSU55LVWpV/VhCQU5VlH9s aiMT2kew8vStcKVhuCZahi0EVZjD1Cgtb7nXjoK0rCV1Kfj0J1OTejGGUZSpSo3ay53mNI4vpKkz BdvEYBaxq4aNq2Y3S7l+1aEOdb1rKIcqTl2S86ilTWpW88lWdIKSkqMlKjiVatq2/7bztRaELGlh O0qvYm+sa6Utbc8623gW9ZCuXShCAclQzjoXrv3ygQ+cwAXRhjGjfOWoR7P7UWaGNLB9VWIuKZvZ yr5VmeZLbDEh+tjcIra8lYwlfMUL3+fa16HR9UEXqttF9x6XqIz9b2rPadavUpWtg7ytPG/qxKYC V8H+PbBwZSvgCSfQwMZl7nDJOsfVZhitCN5whdPK4dje98TXzC91rVvfjepVu42NMS69e1rJdrSZ z0RpfWN8TMq6EL0wtPF2h0xEy2pXyChOspJP6dk6OMGu/QUxgE9b29ImmMBSlrCWFexhClv5wruN MGuVu09oXvCgA3ZqgQc6SjWT2f/NxfVkWYu75Drbmcn8YgIT0GDRJYrRxdwN9I1nPMvDhhGwOaYy LHVMXsVKknw+1qxkz/xlQ4sPyXfOtKbR2K9ABKKiLH4fQwNMakW3sI6rdTCGgRtNqEY5qvFMJ4Ql aOBZu5fVLUTzcmN9z0eKFbkIbu+mh03s8vXrBS+4YajlC2NAD7qvRP7rN4Wcxe+SlNHCvnEagRzZ HZf5zy3+braLTe5yU7NfCRlABkIdx1JP+d1pRjU5VW3iWbsak36OL1jPjOE0s/q6mCQul+edUHqb ++AIh2i/hNQ+ImL32c2GtoyZPVD61hizNp6vtx/90vJFuqX0wyKDAUvY8mL6svv/E2y4V0poRyf8 5ZbrFwpQoOxX71qh7rYyY6caUAhD1eC8Hbkh38xU3A5W3/Wm9L0tudVdyzm4aiWtVV/7b90KvMQB hrnWi9cvQQgCACxYNrgjTvYXd7fQ4jwiMa1dU5va1Mgu75ydJz3JQ7/d4xvfut73zrl+aUELMAi7 zYOrc1MbHqlYpvPVi054Lwu1rb9m/JX3TVAtG9eRlYez5nUd+c2vOu58D/3L+/U0AAhC7A8n8sTL Lugip97iFU+iSzHN8dQ/FMnzA+ZMNa5e87o9rx1vouiHb81+OcABETAEu/2bc5wfPt4Df/ODsZ75 oBt56B8eM2+lDuw2Y977c5Z+//fjPP7JWxj6Rpe1oonP/jSSXgOmRz3wIW721vP4pGO3tA4r2+DZ 5x2J+7dM8SNyLkdprod9vmd3v/dEwQdw7feA7scvQRAEgbd8WeZ88IaB6Cd50Wd5PRd+jyd08DR1 GrZ9h+ZPovZbnxdwBVdgGnh5IAh0Vhdi2pd1EHiDLtQvMRADFZhv86d6zgaEEqdyK6d/4IVoOXZ3 eMdyKDd3IZde/6dt+PdOIEaEzLZ7JIeD5qaDPCh4+HZzqPWCYYh0HWhrMMh9YuZWaniGy8VzU7R4 Hng9/GZ9Knh+aAhnW5VcH/ZJYSZmcIiH66eFmaaDFOiFR/eDq1d/9AeAU1hkIv+ogAbocH4Gej7W aPwDU7Z3gJdWd4OFbSnXhAsYgK/3cYKodf0ygT34hY03hlUmhmFVhlRXawZVYnb4iFKVftTXX6fW eR8IUCuYfU5XftOXZfZUgp4Ha2wIdUZngqUYev3CJSNQCPLHgItYjUIIY4v2idrYdpq4hrw3bs1I TRnHTzbIjaKIUFYIemgUheG4RvyyKyBggXTGihVGj6fWgcdIb3kIgpXWj7uFWW71h8MYh3aofW6G VV8lUjQYjASpgXpYjK/YXK2Wi2QWQlXHjO2IPv0Cj9N4jolof0HYWNJmjtWme9c2dp7YgDi2hAP4 XuqoepGVhovlfzTZf0eYkeH/uJG6Eo+D14f12Ir2eInypoxIx2vG2HQm9njg12ZLaX7GiH9qV3lI SYZS6YIhRXi+hpXC2G93OJAYiZPO+I472ZEreY2KaJaMqEUJCHvbOIlJ6JYZiXsXV5PXN0TURoqn xI5guWQ6OQA8qYo+2XxBiXjiZ5BeeZgSaYvM94k+CXm7iJiGKYeZd5WKx5W8GIO02H9tGItEp1z7 uJemKJZ+SZawdJYfaY33p5adiHk+hHEUp4SQllN0uYQ2qZgth5KZWJZ6CZq8GYHhcgAQEDnyCIZA WZw/aX6wyIFVeZSJWZe6iIslyGYVuYJRd5k3d5GEmX3q53i92Z3G1Jfr5oPU/4iWqBmS6fiGnCh7 NxmK2QiFTJiScNmNX3ibbzme47mb3pmfnCaaf6lbq2icglmcbpiMvQiM9paCCHiLYQWIUUZiT9mV mMmcyCiQssid+nmhqgSepFl75NmhpxltNGZodBeVeCl88Vmax5OWP1ab8Dk+tKd/GBqjmoZuZNFw /hmYz+eKP6laLgh5wgV093ajYLZlNLidwMiQyEmLWkmQUymjTopifdmfi2Z75jmEVkpyQrmW39iW scmiXTqD96OO4PiisNmJ5vikaLpZUTqc/3mcAbqjiReZcmqksaWYxIhbD6ljkJWVEJqMS5qPEpmm gqpw/KIQ4SmJ9lmlrGeNI/+ppVZ4dq95ohx6OSoKci1qppKKheA4qJzKUhupAOrGpjiagYMZb2X2 n/hIgo35lfKlnJ1ppHzap9XpW4C6hp16qzy1puLpkYpqmkHYqAoYrJgarHfppS66dLlHRe5prPLZ iMKKq9B6bvwpqhb2ptaapZQ5p5lJnT1qocxIq9n6p4B5qtsHrtx6p0waqNG6rtg0rez6rvAarzeo q/Jar/Z6rwi3pmmzr/zar/76rwAbsAI7sARbsAZ7sAibsAoLsE/QsD/wsA9bGyLAJHIRJAXRKBib sRq7sRzbsYliEG0xKURSKZZCF5mCF33BGJBhGpjRAC77sjAbszI7szRbszb/e7M4m7M6u7M827M+ +7NAG7RCO7RES7Mse7RIKxlP8BQ3EBVXwQVkgBspEAiDMAiBQAcQ8AAsYRxSIAYfgAVIMAFX8AVZ gAMEsgAP4B0BoAQLMAVL8ALtgQfZsQQ4oLb5IQFRgAh5MARTAB4fcSBXsCHvsSBpgALGcQXogQFS MAQSEiDlgQcYMAVZcLYgobhZ8AdnMLjkkbkfgBJu6wKRS7cYcCAnQQDmgQFtkAWGaxxCgBI/4gES cgMYYLodYhxeQDlK0AdG8AiUcyHpQTnr8SMW+xAEUSRzcSTIVrFOYhBxESUnixdl0hdXYgNf0iVd AiaI8Rdw0hjc273eKxvg/xu+4ju+5Fu+5psY3pu+6ru+7Nu+7vu+8Bu/8ju/9Mu9cnG/WnEbUbu/ UTGyhSISP+K6IbG8lPK/hXLAWTsSCLzAIwESDNzADbzAEgwSr1vBFvwjHyHBhAISPWAEC6DBhHIo BSzBEgElLoApn6IXfHElSZAAajIBtgIraDLDNFzDNnzDOJzDOrzDNBzDPvzDQBzEQjzEogI5AHAC J5ABSrzETNzETvzEUBzFUjzFVFzFVnzFWJzFWrzFXAzFSPzFJ1ABYjzGZFzGZnzGaJzGarzGbNzG bvzGcBzHcjzHdFzHdnzHeJzHerzHfPzGuFIAgBzIVZACNwDI5sIEWKABcP8QyAUQL4XMyJAcyZLM yESgL4CcAxrQApMMyQqzyZ48yY78yYBcyRpAA5csMRcgyqq8yoyMyU+jyZtMBIlQAOqyyKx8y6Ps NE9DyKtcy7jsyZUMy+2SypHsy798zMiczMq8zMzczM78zNAczdI8zZPsLRrTBh0QAirwLYm8BgJQ AhqQAo2AMon8NN4sANgsMdusNdhMAtysAUwwAuWsAed8yCMgAIn8Bk7jzujsNNosAH9QBU+TAfQ8 M98CzuKMMlnTzgEdzyXwz+n8z/ASzuOcMfPMz/N8zlvjyhqA0Rqgzx39LZicApRAz/2szv3szhiD z1DDzwuty+G8Mxn9LQ3/fc/dfNKl/M7mvDXtfND/PNPdnM/7vLBEXdRGfdRIndRqQxYZEAFO/dRQ HdVSPdVUXdVWfdVYndVavdVc3dVe/dVgHdZiPdZkXdZmfdZondZqvdZs3dZu/dZdzT5sKm6tU9d4 ZaJ47YB6bVtatT0ISaS5BkwIqpSE7ZkzRnl8WJlU2Hjfx1z9tos/SlYkRte3Rk9nJW5M539gxkWf ia8xtD7BaaOePdqkXdrO1S9RAgE4NHiUndec3XJ27dplZE8ll9kQqY+WnaW22qpqlq2sGVMXaJvd Om2WTa4dFqd/bZQh2tshhtkJqdl87du6bdqfzS92gQFvRN3avd3cXU39/5IprsTaWXjXev3arV3e GwXZ8lZSkcnebyqTdViQBgeRJoiCE0andOjYxj2UlidiAMnc8R3dzT2Zud3dnppnTABqBp5pZLrg Dh6ac/VTcz3esn3e5k3he92oyT1JMhhI6s2qhTXgwfjcjh2JaKaQ+CbdGfbbPkrfSOlhT3dgDrrc Gz5+tP2SD846PtUHQJXjPv7jQN53/EIDNBBa4h3b6G3hsE3eNIZ25USOBAeQ782dHX5xRMqUJ5WG XVbZv/bcnJhqyE3gmznZl2iRBb6H2ghsm/rg/aIGalAGZGBdQS6o+DnnbM4v0rVfE47kF87nS17h rTnd9ThWR5jbU+6P2v8anSm338bN4hnYpEa5UF5OX0TJ39apjJed5pxU45ae3HZuPPml558Ocw2+ bSU66tTdZE4Q50fO5H+e5Bje50Y4lz5XoTLee4WdlFeelPbd2Qy6YHNm30VpkP7d14bNf5t+5jGO 3nuN6nfmU1bQ484+7dT+rv0SFXy2566u5Nwe6+kt6FIk2LZGqzoK38TdkAVa37pmhptNUJOO4rdu 2P9t5gCnqp5e7TBnfA7ATXKO76FX5/5O7egGqofqn93u5we/7YFeeIfl4eFX2znakxP52pTni4+t 22oF5g6qTlAe77wm5ace8A+4cAlR8CJ/8ijfm6AtnK0O6Ajv7Qmf7Bn/Lu6CvZjXlZ7Gae69ddjH nuLjFahXKewLGtklTnQOf4IyP++2bqL37odqHqGaZ6sBr+/J1+8pf/VYz35+B3iGqOUw//UvH/az Ttzu3VrASo86v+sE+kPY+ae4hqeS3YLL3e4bf5Ahn/UHt/WpiPcsVeoAz/ci3y8VcAIRoAHa7vIK L/aIn/RoV3c1juucV465fuMMKZ1naK5jLuISSu9Kj6RNn5337luSD/i+CS4VgAh9YPikL2l3v/qu Xzn9wgIssPder/iwbvuyzvj3GJU3v5wMj+i6/vFQz3kTOUbh2ugR2vm+vvOZXvSfv4EU3/O5/vrV HS47SPvUn/3ab2en/9iFh3/7iR/+i0/3QvnlvA+KhjX6wU/5S3reXG7855p0Fs/oKo7pRVrzXN75 bn/m8/n0lQ8QAAIMJAhAIEGECRUuZNjQ4UOIESVOpFjRYkMBGTXGiAGDhYOLIUWOJFnS5EmUKRMa NKjS5UuYMWXOpFnT5s2RGjUGCeIRZMOWKw8OZDk0QFGESAuyTMp0aVCiTqNCFRr1qFSrV5U2zUr1 6lOsWblC9LowaMutX7sKLFvVK1S4B88aVSu27ly2RuNqldv3bt60a/kKXip07mCHb/2m3fu1bFuc kSVPVqkzIxcuADT8pNzZ82fQJQOHJl3a9GnUqR9aFnDggIIMnM3Srf+LeKperIxz76Y6WvBW4IEb qy3qt7Zd5AwhF7baW+py4n3z/qW+1/rQ5dcLF+fenflh5o6dgkcMnvx3wMbRFie8Hnpt7dSjT48u P7xq/DktD+APQnZ+AAOczDcBCzQQQAIPVPAi1vgbwL/EaHNuQtwobMpCsC6ki72xxHNvPOPEE2s9 25J7LzmFSKwuLPSQU4ws3DpUcbgUdxtxMexynC0882JsDsQfS6xPqfOuO3FBJA9ssL//knTySSij lHJKKg1c8sEmO7xPNwxv0/DLDMOs8ce4yuzyxu3Sq2rNh96Lz0sUW9RKxCDrpPNO59LsUUs0hcNx Tx45/LBNH+lsDK//+vDckLYqG53pSggdlXTBBCe19KRKL9W0pitji3BMMUvkskJSwYQT1EHH4g5U Qzm879XjlGN0zVTRvLNPIBXl87dc87ztsExf9LBQYCWcdVNkk13NsgMgUOCELJWVdlpqq7X2WoIg jdbEUkMd1VRReZu1WGFXZTNQpGrlNqITySUWRzZTLVZIdwm7kUg1P1U1X1tV3BVbgJXVNmCCCzb4 YIRDY00BhqHVV8tvvRUX3IjBWlRP6Zii8a9cj3vs2H/dKnSq+XSVkb5Ay1OPvY3rHVJHdROWeeYG FRjA05lz1nlnnnseGChjKZ5Y4jMr/jVeNQelkcKgz43TaVYR9RVi/5RlrdrXqeUNcd1h7V3xyJ7D lvJnsct+KVOz01YbpYVfcxhoVsMteuhT5V40LCMz9vdDvGL0M264CYXv7pL3vthOXPeM2eV5g0R7 bcirJDtyyiu3/HJO92Py4S3ptlvouRXDG2aNk9axcyD/hnrHh6Ue2S5/rR5835NVdnx0Mleu/XHM e+9sct+DF1J44isHvnjkk1fe7OOXd/556ANm7Qnqf7CeB+yxF2F7DrqH4HsIXBN/fPLLN/989NNX f33ywXf/ffjjl39++uu3n/7u89d/f/233/4FF7gABxQwgAEQcMAFJECBCWBAAx3owAZEUIITpGAF LWjBC2RQgxvkYP8HPfhBEIZQhCMkYQlNeEIPbkCFK2RhC134QhjG0IUXpGENbXhDHOZQhzvkYQ99 +MMK3kCIQ7TeD7QnAg58b3wMY2ITnfhEKEZRilOkYhWteEUsZlGLW7xi+iDQPRG4gAIEnMABEbAA NKYxgQp8YBsfuEA4LtCNc6RjHR8IRDxO0I575GMf/fhHQAZSkIMkZCENeUhEJlKRbRyiEAGRgj5Y AhFbqEQPIrEFOmSSDs2CwAM86UkpDOEPUKCCJwnABzywT3yfZGUrHzCFIRxAAHwYSCkxMAUkTGAO eaACBjzZhiE84JRDGOYtkXCDXwZACQsA3yhLKYUl4EAKvdwlFTz/kMwALEEEEOgBLyEATRFMIZWj xAME2rAEF3gSfOWb5RAggEtkfhKaLrjfOl8jRfGBL3/2zOcXwzjAAk5AoAI94xptUAQnJFShNmhg AtT4UIhGNI4TpWhFLXpRjGZUoxvlaEc9+lGQhlSkIyVpSU16UpRutIjW0+Qb5ACAQWxhEISQBCW2 wIUHeMACH+DpCEbgBTH4FKhgMIIKPkDUC0gBIUiYAR+oIAVEHGIGpFQqQcLghyjg4QoY+MIQfOmB UcLhA3I4wwp8+oU0+PQIR/gAFpRwgSu0IQwLOIIcsppVDMzBCHvAQF/1eoEPfGGuQk0rT3na1ylk gQJ9/cJAtmAE/8AGdrAjwEIaEODTEYDhDgPRA0+F0AYkLGCsWcXsT4NKVBXEIQ5AjUNlDWAGIVjA Ax5Q4j2bmM/uvQAHKEABDnDwghcksX0cEAEACWiAgZZxAUlgAEKtgAboQtcKTigCAxy6AAQkV7tl NGN3CxpR8IZXvOMlb3nNe170ple962Vve937XvjGV77zfa//tqdJOnBhEJagRA8qcYlKYKZZOfVA Xw3c1ysYVsGH7asrHXxgCMNytgWGcIUtfGAhZNgCB57wbCtsARBrGMQbxkCHJwzhEWeYp21Iw2vN 8GIzGDa2JEaxBTJ8Y73uYcQ7BvFskxg+8TGxfV/MLXD3CeTx+f/TuAVkMncRkIAkFAGhUpZyEpJg xu1mWcvJ9W6XvfxlMIdZzGMmc5nNfGY0p1nNa2Zzm938ZjjHWc5hfp/3+Oma26ryfarkc5/9/GdA B1rQg/bz98AoRgqggMmLLqAZGf1oSEda0pOmdKUtfWlMZ1rTm+Z0pz39aVCHWtSjJnWpMS0/83FR 1atmdasZVr5XE1rWswa0oTkAXAGOcYy8RYGuff1rYO+a18PmdbATTWzeGlvZy/Y1sp09bGZHW9rT pna1rX1tbGdb29vmdre9/W1u4zkDDiJ3uc19bnSnW93rZne73f1ueMdb3vOmd73tfW9851vf++Z3 v/39b4AHXOD/A6e3uAtwcIQnXOELZ3jDHf5wiEdc4hOneMUtfnGMZ1zjG+d4xz3+cZCHXOQjJ3nJ TX5ylFvc4AonQgdacHAdhKAJJtCAHgqgAw2koBEpmEEBWq4BmR/850GXeMtb0IMaMOEFQu8A0JtQ gCooHektoPkbmv5ynzed6DjXgAZejvRBdEDmONc5z4vegUQsYgY9WMQjgj70p9P85TkwO9drznSn T9zoCI/5zDVgda9DvetMwMLd4Z51wGPd7jY/O9ZpbvPDV4EGB5f847PedawfvvEwf7vW417zql89 5aMnfelNf3rUp37k4maNAP4g+RIwYQQCoPkKShACFQiA7jO4/33uNdJ7AcigAyRoPWuET3ya00AA wNfJ7YdPew0Q//jAnz7udc9zpCt/+dbfffEtc3wT0CAHtsc99YdPc+J3//caIL/vvW/852ek9+gP /vCzD/32158Ex7/+DJrP/vfTCP6rv/xrPvFDPvarveazvgD8vvjrPebLiNqjvwFsQAu8QAzMQA3c QA7sQA/8QBAMQREcQRIsQY0QNwJIQRXsOhZkQRV8QRiMQRmcwRdswRaUwRLIOT+gQR7sQR/8QR60 wa4Dwh8UQhYsPBIgQiUEQiPUgCV8whlsQh5EQiisQiu8QizMQi3cQi7sQi/8QjDMQnGTADIsQzM8 QzRMQzVcQ/82bEM3fEM4jEM5nEM6rEM7vEM8zEM93EM+7EM//ENADERBHERCLMQ5/J5n2ZboWURG bMRGYQ1EfBtZmcSu+A5LrERMJI5L1MTA0ZWWOZ3TUZWQgY6NYRe5uIizMMVGAZuoAZlUzMSVsI/Z GA7DERwYIQqJYEVHlAhIdBZJZB0++RjAiRVhhJpykY51SZ0QKcbzMEaQgRVKRBEWGUauiRVKFBZZ FJlbCcbXOZmtmRponBDYaZox6ZhopJpZBMZdNIleTMR1fEd4jMcCacdflEd7vEd8lAzW2J4DqABF hIxD2cRX5ESCHEiDPMVOrMVPtAhaHBdXDMWJGEiKkMhOnBL/XaydaGwLlCHF5yAdaAwZWsnFZ8xH imANF3iCB3iDfyTHaoSVYtyVl2yPZvSQOpnJj9kaYhxJa1RHjWwXh2xFkXSRUMyO6YgdbgTJMhnH p7AVkpmd1aHJnQzHbpwTaiRJBrEMAvIAlbRKruxKr9QPncjKrfxKsixLs0wI1gCgB/BHzsFIUBRI hCzIuDxIXJSPillIWgFFRgHIh/zIp3SUi0SJwGydvjwKv8TFUgQX23mas3wJ1pCyKxjLcxTKYYzJ lrzMpSHH+ACUQ8FJy5xMVFnGZ+xIOTHKtmRG3XHKcizKn0THc2nIb+xG8hBH1ZTKphzMxgwJ1kiE RCiDLVDE/9xUDdwMTuIMENZgAiboAzcAzuJsTuckHtawAztQzpXMyLm8zrqUy+ykS8PExM5syGys iFqME448TPHUySgZToI0CfUMyqdJzI0czY5pz8aMzukcg+p0ms/cT5bETIhMypwMUMAxRwG1xcAB FKX0mgCtldjhHRLBFzFhxWNsEb4Exs5MUKrMmtmsEAzdF5x8zmXRCensA/wEURM9UeGxzzIoURRt UReFHNagARrwzfwEyu7UzhvlzrfMxFf8ztYESeuUxh+1UPR0T8mgSNBEmMSEyzYJT1N5Sfq0Stao gzqg0bb0mP4sUJfMUmwEx/LkzL1MzS1VRRiBSGeknWb0Sf/QXFLWMU0sNZmlvBAbTRQNjU2YFE05 rU3yBAzGbM4prVIyYM4X7ZkoHVRGZA0fUAMn4AJBNVRHfVRqQVQf6AJGvdJ+YdIcxc5M3U5NxdPh 8U9cSZMkjUonNY3hRFJ1lBk2tY12KdXVNM/ilFRKrdFX4c/K5NL+zMw7zUumFFXK/EsgjVMttZcK dU1SLVaMJFM3fZHxFNamTFYE5ZgOZcrMtNNanUkTldVKhVQoKVTRKFJuhR5tbdRwLVdznRLWUANF DVRLtUtM3VEchdce1csuWdXDRFZgtdc+nQlvPc+U0NdESQxXDZe6BNhz1UfLoFIn+M0rxcYxPVOI PcpptQ//04zWXh3WUS1NM9XPpgHTUWTNnXTYmIRNDEXQap1YcMxQcD1Ym7BPK2BYlo1ZmY0S+yRR cp1ZnM1ZhbGMQAgENFjOdg1IHu3Udy1au5zGBV3Ze5nKY93Ys1FaUkUQp91TgBxYItVZK+FZnwXa ivzVXY3Yh5XY2vxSj6xKjD0SsMGa+KTTIm0c2tSXZb2YZjVHfB3bItHL24RarJ0M1niDN7DZvV1F vQ1ce2SNhIrMmyVcxV1ctmGW18CZURXaeDXayfVOeu2WgZVICE3Ve51auKWMfrXXzRVYNoXSwWVc JFkYm4FcznXYyzxbXEWar10RzpWTNwVW88yTnpRPEanb/5RVTQD93YA9WeAlnQ9FXUtR3ZtJXORt XucN0RN8XOZ9Xuqt3gCgR1p1S8rV0e2Ny6tN0oMEz7rtXFhFFSOdSO/FCdxc1eDwVdL13LpxXzfR SVRt3dMNTtaYrX7M3tud3bD9319FlFZp0x8l0P4tXzUN2DxV4LiJW9590CHVU29E2Rz5UOIV3p5U WQOVyXRU0Kh1UdYogzJAXOstYRN2CNYwBEPQjOk9YRfeW9ZgARbwiaCVV6IdWk7N4U3tTh99Uryt EWxN1nyF3/KtXQEpXd5NEaud0xdWEsvQAi2g4a494Ne1VbCVRdPVWF71YADOWA4OzWuV3SB9GQKe YAnmYv/a1FAg3tcmRg3W4AkpbmNUVNp+lWNxtQwZjmM73uPFZY1CKAQWruHu1WEbJuRN/F77ld/C fMryZOOQJFPQYN9pFN0lXtqLNVtRhOQiPmHWcAAHiABD4F/Xhd1bLWXa0WCT5VNP9NRRRtuRtNhn DeJ+gRcwRhcKtt0NzVMvPRYBnt8t3l3b3WQTZo0MOIEIGIQW5uMPVmb8tYxCWOHNYGZpxlrWEIQZ /ghBxuEdLuRtplxENuVlzuA9JUyGTN8pdon1JeINLojMJU078eVbvNF1XmYXZg0o1mNwruLYveLb 7WX6yGVlrOVxxt19hWWqZGBtdNMDJso4VeNEbmjZrGD/MunTCbVGd/5MPmYNjsDnae7oc9XojsBm jx7pj7aMjRbpyOVmlR7kbk5IXtbMw7lQeR5fIXZP9BRnKUlnYYbFpGhn0fHUczZfTXbkZtaJkxbl w/Fffc5ntqXlXNbG4T3e1zXiUwZmNC7MNC3bW4TgyRTeeylZoKYaa/0Xf15ng77oLD1RkOZokqbq ti7LtUbpt57rF33jnpDrqpRc7tXmleZpMXZrIA7TkI3gv45Ic/biYGVP+l3snW5VgxVG0WVseW7Y +3VUu2ZrALZipSblqnZbwbZJ+G1lyYbb40Vr0uZYH5mRkRHnOgVrA03j1e5LA+bYq/7hxJZZ1nAD N8Bs/7r2196Wx93szRBI5t8ubnxkDd9aS/69VL5m6b4eY64oRYa25I9sZHqWx8euYMyNRSS2mrAG 7M6oY5/RHCyhbH7W7C6uaAMeT9EOY34G74M+E1vMavetbtGZZ+1QGn6J1uBl2vZGHdcOYFnmFtuG 5cZB7BQl70iRZvGe5gYPnuYxbgk3ywgf6Mq98L2u3B4O1VlUZ4ol7JpG7Pod6jpWzxGfY5robmms ZDh93gofVvRe6ivW1bzNxgu9Sa9F4CTGVvX2zKSGG62J0LaNabylafBU4P424wuObcMgap198QmP cilVcOKWciuPnhdH1gzf8hu+8G9+531GZJpOmZmR5P96ddq1BRrpJuIH/9Ynp3Lz3uwY52yNteA0 P+Ut9XFM1vH/RNoy7t+KdUjCQZqxDtv55FAJFt8Bz8kCF1MKXeSZhfIXbnPipHTLhvMrz3SuzPIC 7nIuN+TmdumgTlV4Hmxy9u2bXmQT30uf3tjIVtbzdXJN50XLUAC3QerzDvP0JvIa5/F/tu1gfu89 Z9sOZuAkJ/PVge3ZNRKK5nWLlunbnvWSZJbwqUdpv/ZdbBt3xHZuz3ZMT2nnDvdQh+5hJ+cjx2SD zeR4JvfCZmczd3U2l+xUL/duv0qdcJAFf+hdZ2o6l0mn/uzUzGACxehhP2to/+riDWu6hWk9afZf DlP/YJf1eqf1e9+cib9455F0jN94yqmZ5c1mDPd0kQf1DaddR25WcBf1cx5lwVT1l97kOzdSS+d4 sNQIW9/2Nf1xGZfzXO1ziH5VntRqfW9XxexxiLXYYx9nfF34PJd4mm/ZWm+YKn96qj+YTpn6qs96 gOH0MX7uT29pTiz5N0nbmDdi6w5MXcTpm0h35WBxlNf6m+B62s51frdMGkflAxdb1eZ3wGbtiA72 vE7tdxnru9VlJjd8uEfYii9vjp/5xE9e8mbdx5/8nNF2a/9YcQ95UA/7y00ZFX/kcx9i8h3Qyh51 V4VPFkd2yi8N1cX5oe/3nc9sn69xqHbLQf/veQ7s/7mn74f3Wqan/cEhfGOR6tX/ncjH+uJPfsn5 duVvfmyRexv1+pEH+3lFD0UvYoUcShBX/VjPfcdv9xA3zOxOfeevCOh3b56v+57vnJ8PaNJveCoO 5yTe4mpc9KUXfvzv1d8XSuIv/5oHCAECBhAE4SAAwoQKFzJs6PAhxIgSJ1KsaPEixowaN3JUCABA x5AiR5IsafIkypQqV7JUKPDlwIIHW9KsafMmzpw6d/Ls6fMn0J0wBRIcYDAo0qRKlzJt6vQp1Kgs h8Y0OlMq1qxat3Lt6vWrUqpFj4Ita/Ys2rRq1yIVK5Mq3Lhy59Kta/cu3rx69/Lt6/cv4MCCBxMu bP/4MOLEihczRjzWQePIkidTrmz5MubMmjdz7uxZAAcXT37wKG1aBGoRHFavhuD6NezYsmfTrm37 Nu7crHfz7u37N/DgwnenLm78uIgXype7aO4cBw4KFFAYMIDg+oIE2hlw7+69Afjw4DdsuED+/Abx 6tGTv+D+Pfz48ufTr2//Pv78+vfz7+//P4DusTcggQUaeCCCCSq4IIMNMqgehBFKKOETFf5wIWmn odaaawd4+OGHCog4ooggmngiiimquCKLLbr4IowxyghibrQR94IL0VWH3QLZafejd91NGF6QQQ55 JJLq2bAkk006+aQNSUo5JZVVWnkllllquSWXXXr/+SWY4lVoIYYZbshBhyGSuCabbbr5Jpxxyjkn nXXaeeedB7Q5o2urieCCdAZMwGOPhf6YQJGJKrpokWE6+iikkUpKJKOVWnoppplquimnnXr6KQM3 iCoqHaWaamoap6qJZ50nvtmimyruCSOrtdrKaosQ+AlodRMMSqihiCp6qHaFBkussKAquyyzzTr7 LLTRSjsttdV6OiqpZBQCCRdkvAHCIGSoiqIUQ8igxAIfTrFEGwFQ8eqJEDwwL70QeDhFAEj8cEAP eVBBrxQIDVEvAXwghMQNB/wBBUL/PuCaFOiC2G4A6D4wRx4C12uvFEuI4KEMCEm8MEIeB+xxiiUq /8ywuyaSTIWMcdIIG4p9/klBdYL6eh0ChjKwpHtNMlCssUUb3SOySSu99NKVMv001FFLPTXVVVt9 NdZZa7011117/fWP2N4ACBmDILIFGQ6A8QYZZHBRqmsPeIAB3RiUa/cQdM9hxAL0rigvvQ/UPTgG 804x8LwLU4EBAVHg8cEHdxP+wLp774GB4ozzgQjf9FqOwRdZoAD5BxI4PjjlS7iQ+up79w3wEB60 kQUFdXsQ+MMQ9NB55nVL7sHt8wI//PCveRirh31yoBxraX5oM6A467wz0gwU4YQTVmiP/QU2DH00 +OEfCzb55Zt/Pvrpq79+AuK7/z788csvPpk/lP/q9hvdtr1FJZZ0C8HchEC6D3hBDEeIwBlWIAfH fcBybUBY6uZwiBu0S3VSwMMDJ3CFL6RhgUq4AAYOR6+FweEDckjgEY7AwREcYYAZbEMY/GBCx4HB CHuw3O30lrEYpjCFBSRd3aZAOwxgbHFfcJcQkshBA3wAC2kYARRHEAfSCaFuGbTAAvFghigWMA5g uEMaDPBFDCJBhkKwgAWA1yHk6Wp5L4AODpiHJhPp6k86mt6gspOE61kBDX70oxWcUITvLYBnvjrk r3jWs/kx0mhQayQkIynJSVKykpa8JCYzqclNUtI0pTHVFmCgCA0QogdguMQl0CY3C5AOigWEIhb/ wiCJJx6hhheQXOEeUK5JdIEPjjjEBMp1BdJl8QqgyxvdSDhDPLhSDFBMIRY+CLkaquAIJ3TEHRSi hMuBTnQfoOYIvPDEEQywiqGrHQYE2MQYfuALYVhAFJ3IhnnSM4pTbKARmmDCBEYxnM4Epz+heEQx oMACGPBAmtb0vDa+EQUogA7z7EWj1eAoUNPrWRL2mL0/AlKQhEQAIkOKSEWStKQmLSknU6rSlbKU kSd9KUxjKtOZ0rSmNr0pTnNqU+OciguDqIQkdkAJTFwiXPKaGwYWGAAxDNALSqhmLY1wgcglZGAL w8MD2vUvu1XVdHig2+EYF4WEfFCpAXhcDZ0w/9aEiO6LFZsq5Ihow8wF7K1uTQgZJ0AxgZEsAEPY p0Kc6UQE9NOeZvjAGTFWMX1CzgxmKKAZakgIhDyOYkpogkERKtHjqWyhHEgOROW42eTVMUfS69Wv EqDR7GEPe0VIQvuuI9LZilSntr0tbnOr293ytre+/S1wgytcmfLmVHTgAhe2oNxBvI0OpKUX8QiH gWF+wHaBc17ycKddfEFQbsQjnnYDh1TCAS+8wJMueoNXL+h+13cBoB0a44tG2323vXXDYX2Lh109 KaBmytvNfpNHUUBRJ2c7u45q92iDjMLWkBPIGYQjjFraUrjCBx4uhjOsYZhauMMe/jCIQyziEf+T uMQmPnFIX8MbmnG2syp6jXaNN6MZs6g2pD1AjWJT4xzz2MaySdmOZZwim71xOhIW1EsffOQlM7nJ Tn4ylKMs5SlTucpWvjKWs6zlLXO5y17+8pVt4yo28ZfGMnNVjGZF4zWzuc1upjGRo2NkMNO5zna+ M57zrOc987nPfj6yjUFUq5TdqtDwEnR/36zoRTP6xQw17XQK3GSHUtqhf740pjOt6U1zutOZfg2r iiLqUZO61KY+NaoJYuhVs7rVrsaVicIr61nTuta2vjWuc63rXfO6177+NbCDLexhE7vYxj62do93 glQzu9nOfja0oy3taVO72ta+Nrazre1tc7v/297+NrjDLe5xk5vcx8tAudOt7nWzu93ufje84y3v edO73uk+t73zre9987vf/v43wAMu8G+fuwAGPzjCE67whTO84Q5/OMQjLvGJU7ziFr84xjOu8Y1z vOMe/zjIQy7ykZO85CY/ecULnnAdaAAOBidCB1qw8BykYAYfh7nMTaABGkhc5zL3OM1tDnGYG4IJ LyhAFQwRgiagXOE5WHoBYM7zhfegBlNHOc4NXoWaN13hWS+ADqCucJ27vOtmPzva0672tbP94ueG SxWYYAEB5EDuOl/BDmJe9ZrnvQV0r3nVaSCAsAPBLn0PukD6/vcZ6KDmd9e53Fm+gsAPPgRA/8h7 4YO+9xkIoO+bN3wHSFCFyzOhBJanPOF9vnjFP/3yMV+8XQj/kr5D3gKSp/zdUW/5vDNBDlufAest D3q/dz7mrK9542dQ9RXk3uoCuPvxOV8XxVfe9X4P+uM1EHkNrOAz3v8++MMv/sC8nSoy6ADzQ6CC 53Pf9OsvQc1LoIH5z58E7hcA5ety/pojXv701wAJ/EEV0EAOCN7d4Z/V3V/g7d8MPJ0K5N/9wZ/0 0cX52Z8Fup8CWp3OkQDs6dz8xd//AaBd3J9AVOABBh7utZ/6ISANVGAH0l/N2YULCkAF+h/9BWAV 2B8TjMDdyV/3vYQN1p8Mhp5AYGAIkkD2df9f/o0fEzahEz4hZ5QfVZRAINQAB7LfClTg5rmg+52f 4JEgBRKhAKofF66gzpmh9o1AF3bAF6qfD1ad+uWfFtZADOpf6MnAGzCCCqxhG6rAHDqeCN5dGa6f XTggDbIh++2g+6VgFiLiGnIg9gUi9w3hFQbdIB4iGk6e81XgJdrhFT6iALhf9iniCkKhKZ4iKqYi +elJBhCAK74iLMaiLM4iLdaiLd4iAQQeLu4iL/aiL/4iML4iFgBgMBajMR4jMtbiMJJAMjajMz4j NEajNE4jNVajNUrjuV2jNm4jN3ajN34jOIajOI4jOZajOZ7jL2YjOq4jO7ajO74jPMajPM7/Iz1+ oxSqIj7moz7uIz/2oz/+41C4hgKcgAMUpEEeJEImpEIuJEM2pEM+JERGpEROJEVWpEVeJEZmpEZu JEd2pEd+JEiGpEiOJEmWpElSpEASJFusJEu2pEu+JEyeBVWk5FU0xEc8xE0GwE3mpE6CBE/uJEgg BFAmxFAKpU8GZU8mpVEqpUP85FES5VMupUcgZVEyJVQyJU8yRFZqJVUiZUVUpUWApVVKZUlspUps pVlShFNCRFWK5UKsJVvCZVLKZVN6JU5GpUSkZUyyxUxCwEDWJFfapFeaJWHa5VgeZmEG5lt+RFca JV4WJWNCJV065mAaJllOpVpa5lVeZWFG/2ZcCqZhZiVj6qVk2qVnUqZmXuZo9mRlduVqouZibmZk iiZebmZdKuVsVqZtrmVOnuZt2uZchuZrYuZevmRf/uVnEqdyAudlNidiCqdiTqVnrmZiUmdUTiZu BiVpjuV2MmddYmd3eqdzOuVopiZromVjOidwQqZuLmd59mZ5nidWnmZiCmZpzid63qZ1RkRnpudy xmZxuuRxquRd2qd31ud/Pmd0Mid8kmV1NmaDjqd2TuhvAuhIDCdpdqdeImh4SmV+OmhqDqVbNidt FiiI3qeELuh8oqh4BmeLHih9hih0BmhLDihgLmZaAiVt7mhr9qh0+iiLOmZ2xiiPumZwov/ndPrm YcpnXpqnVb4njm7ncAYphk4paOKna2oohVopk0pmaTppfD5pmFKmR3xnhC4llHbpeTopkxapaU4p l9JoWtionNYpS7xnh2YEnrJpk6apnf4poGYFnQYqoRaqoR4qoi7FTHLAARBooj4qpEaqpE7qRVDF ahxABdxogsrmY3bqivImhIYqZ5onqI4qg/KphZ6piqYqqmJmnsbmh25qR7zqrL6pRmDnbsroiwrp ieKqrMIqqtIqpTIFVaAGpmqqeJaohcLoqiIojMJlbpoqa7LomIYpfLbnsiZnhbpolIKpZSqpmIJr lOLoj5Kqj/omkaarrXooWL4muLJnbcr/ZalOq7aK6XO+a6sOK7EORXM8QKbWq4Jma8AmqLNKaJK6 pY4+ZYTi6b3WJonyaYdW6apuK8Fmqbh+Kau+aImqaqyu6bXmZlsebLxmKZYKLJpGK8B2acH+qr5G BVUAygO8AbKqp7JWbLPOqM0S55k+qLSqaoqq5rfma5/658TKKocGK4Uqp6+66IhyJ7ZGp7wmKadm LJkaraemrMRS7JK2rFRQBc54gMzGZY4eJY/+qNlSKbqmLZIOKXn2qJF+rMViLM1erMm655jC5l32 59muKX+GbNt668nqrXTKbd5yLNn2ppkmbeCKbLkOLdBa6d3GKde67FB8bdhOLqTuaUpo/26t+inm fu7neq0BgO3Mgq7pni7qpm5PUEV0xGzpqi7sxq7szq5GvOzq/KuJGmjIfqqonqy0ciu3Ru2b+mfH suvU1u3WnmiT7qWw3uq6XijRBubSKq23CqfiNi/tPsXL4oC/lu7Y5iz4MqvAxmrN5mqulu2Kqunc Zqbj4m3hXq/C/u1nqmuQtunT2q2ODi5qxq29Ki/K+u642ue19q+9Liz2Zm9TUEV1YMDlUuzKPjDO iu9/0uXR9mzS7uwFR3Dysmx2Kq8Dp2eDvmoIp6m1ZrDVuqriOm25tuvhJmzJ/mzN7mfVnq91Ni0C cwVVaM8IFIL3Bm34/uwPD6wHD+y8Gv+w/6bw94qEEaunYkLrlrZq8XawFE+x9MbvB2+q4ZrvDNen zwrv8+ps9N6wV1BFQbJBCPTw98Kp2gLpDKupGiPp306ou1bpdcKr/M5txLIpyt4taMax8fJt4Xar lw5w+f6mCwewl7ovEzduuCqptSYu0Hrs2fKxGI/xUNiBHfTBGLxuJXcF56bs0Eru5jJsJ5eyWVAF EzBBH7gBJ5uyK78yLMspKqsyK8eyLd8yLu8lVWDyKvewie7uvAIvMPdu8BJv1KbvVxKzz7Ls9F7E AW/FMwOvEvsw9GpsGGux2A5xLt/ELmdyLeeu3d6sOI+vBlMwCO8uIKfz8aatgS6v2J7/8/7Kbf7i rdQC8saKcgGzM9sCLhgHrjxHbpH6M/UKKfri65DmsxTT7TavRDdrsi8HMQSP8w+b830+aHxu6LmW M8Tq8RxHshaP8EXHbzB7LN3ec7p+qZSuqxNvbRKnrwHXcKd+LAHLsP2GtA0vtEo09CaDciELMRBL 8KmupzFXsURj85JGcz9TcQynsIIeckI7bIEechFDsUrDM0a380rPtDFj8MPScBNfM04z9CVn8k4H chWjbxu/MRu7sSPTsciSLOGWsFzjKj7vKkHLcbj+Mxw/8ZF2LCUDKCGTLbsqNOSy52CvLbVmbUrX 8+JqJyRrNdyGNU7odCtLNuZ+sklg/7Zlb/ZGdHMZlDVnh7Zoj3ZLeDZokzZqp7ZqZwRV0IAdlMEW PLTuemowD/Pv2jYyx+htC21W/2pK27UzazBKiOXK/kQ0I/VARzVYU23eSjByk3Zr0wBsy3Y4k3NR /3RXe7TJynRic2px56mW0u8Eg6oJE/DEGjThCvAdzzNjrzH/8io9C3Zy/680tzEK33VyB6xCrzZE UEUd1MF0A2xEWzeBTzTRRrF33zFgM7UKu7OAe66zjvDj8rNEk7Iht7ARi+iBC7cHLzGU0nTWJjWI +u0Xo/BN8/dD+DeAxzZP369Pv/iAG3UzGzYVr7OLb7DzLnKE83Uj963WNnM/t7Tv0v+1Rn80Eg/1 Fs9oF291ie82imOEipdB2ryzeqM1W1u5Wu+t+344CdPxQXP3lgdtHodyemf5H0drT9v3lb/3Y9sz O9txXJt5Oi8zI7upIOvkY5v0W6f3k1NElE95nzOvhWc2wz53oKM2VaiBGkh5ZR+6oz+6K1OFD/iA E3BBo0M6pmf67Eo6pVs6KBtvbdO2Mou6BZ+v+U4mV/e2NWuroVczB89vk7c6SRy30LJ69b46MkPt 1Wo6Q3B6F3g6OAv1dcO4cJPvje94qhPzeSNt34q3m4P6jqK0Nm83sv+2eLM3BrttmQf0kVItfR8z rOO3sB80r/f6UEz6r1M3sQ97jAf/tXb/9BI3dRgjeNGOd0h/9fFm97SfMKv+9dSSsoSjM3Zjs4fb dEw/L0WTadviL1SXu68Du9aa98C3O3Yn/LpPMc++O3BjxDJzMVj79XKzNM6KMI9nq4Yna5EfsXZ7 MXMvuVcTca2j+MOjsQ/L+Zpruc0XtrpiOM+Le7W6N8qPebMr+M+LtPUOr7SbKVeitb/HOeOaKpgX PdMrtrkydl9HNp738ZEDfF1j+syXe0vC7X5vhObKOtib8tefvdqv/emmPdu/PdxTaqKrQaWre5CP Ou86eahjvJHqb64H+7DvKpDj+J3GPMc3OUcMvuKHPL0C8ODHPVDMvRMAesT3NMVf//x4j3t+Y32S Wz21s+/Q8zmwvrfOH/m21rDf+2kJbz5eK2tHh/u0LnXmNz7ka+9Q/Pfkq/vl736xI7yLH3OhZ/C9 V/xG+zjfJ2fC7vlT87vj36+bjqwkg3g9c7VXL3x1V7Th1z43334dOAGLA77EYz5QE7/mEzxUx7uM U3NIxPtv5yz7MzjUN3yDr69Wv/z767v1l//fa7+ijrUVAMQWBwEIFjQIACEAgwURBkiokGHCgxIj NqwIkeDDiRYzUryI8aHFkBBDfqzokKTGjh45bsS4cGFLmCtRrpQZ06NNkA1F8nx5E2ZPlDdVntSJ cyZOii19Kuy5NGnJmhGHSix6NP9pVq1buXb1+hVsWLFjyZY1C1ZAWrV27FgZM/BsXLlz6da1K3ck 0Lsfc+71+xdwYMGDCRc2HFbt2rZvDzd2/BhyZMmTKVe2fBmz3cRpmTDpwzhzaNGjSZc2fRp16syb BXRG4wZu1JdIa3L0SdW2051Ta+/OPfU3zY64d/PWGly4cdqyqW7V+/X58dmqkQedbpT5cLDVVXf3 Tpp1oECvY88EynQ60fTrrbdfrt34+eu3rfflTb/5e+lcWerembMoqbLC7yoBzfPPpN5SSmmi5qz6 yb7eDgQOPwWLi/C7DDWMjDUuuCBvvwb1U45E9dwTEUX40BuROOIERM9A+GQc8Lr/CR3UrUUKY6Tt Ke5oHC4v7W4bsiSNcpOvKgD/w9GmBI2sccMopRQsvPFgC/E9E0fUMsUZlavOR6R+49LG/LqMa0UJ gWSQxB/VJNPGCs2UELk6I5SJOzuFonDOKf388y7WOuvjShqRLFBJGCFU1KVGXVqyKZVeDFDSkRyV L7o+Dwyux/90jG6pBT+FEqskm2zxOalWjNFI2UJt8MFSAZ2V1rhYSySRMkIor9ZeNcyLVK+AbdPX Yo09ti7W0EBjhEJ4RRbaaKWdltpqx2INBRQeeONZa739Ftxwxf2ONQggUOCEbjUVkkk13SWwXXjv Y1JPpfIDtd3k4DwTxUwn9Bev/9kAhq446QrGEt/lBh6XYVpZU0CBATJQ18v42Nvy4jPhBDNjTilt b88y+RX5RHcTzK7JRPlyEsJ7RV2V3mDpi/Ui9zBsGOdeH454Yiy93BfojPvE82Aux7T0S0t9JLqr TGEu2c01ZWSar3wjjXTeTuV1Gd6QsV4457A13FliittMU2Ohf1Y7aTON9m0+KL1uGWqz9kyVVCKJ pbrikOfl+mKPFV5SxXiDFRvxsTeDuGyD467K0VMlh7zmyic3VacFQ+1v87g71fSqqJ0j/NHR11SP aEhZVMrvIzUnydXW/XP98MRt747snm+3fdjdff8dWdYOgFh34I0/HvnkMxR++P90lX8e+uilh4x5 dM3GW+rs3zX876NzRBl7kBd1+Wy58z5Y2Hx9JphYu/gOK8yNyB84fvm7R7/uf2svcf/pKyv3XM4T ndvYFrS6Bc1jmgNdjRLYPiQ17XxDGRqiEva9k4nsSePL3AX/JkG0NdBpMbMajg4lKp81ZYKJYpv/ RsMaDnBAARW4nsAKWMMD2pB1CvyczRhIOAOObHUi8tuMhpgl9bUvRTOjGwFVVSSaUQ5w/OvXEyu1 o6/JjIbsgloRwcZCDm3GAhY4gAzXNzUcpu2GW2ST98pXsQ5KcYF0uZv5tBiiIq5LanKaE9pOpKcs Fo6Hg7NcHOtHxDXSkU9B9OL/ZVgzghEIgFuO0x+jBknBRV0ycjTZoQdfdzmkTao/m6qdv4QSOlAi sXCpC+Hp1pM6DtZHdknqJFZ6+KooYg6XhrrQ7HJ4Etct0jTKYpazgCmu3tktSMVU5vNY84Y39EED ZlvmNKlZTbGxRgMaAIAgpGlNb34TnNFiDQtYAAMWzNBN8mLj9u73PSVyzV6A3FuedjlAQfKnf4ok Tf3eFrUuFjKc1WSNFrRgTnSW7IdoxJgaCdjHXT4Nie/Doz6h2EQJyvJ08owdgzhXSr49xWIVFNPL WmbRiwYUpQMt6DnXl9AJpnGhU1RoBz3qRPKhzI36nKNJ6am98CXyanPc4JMG/3nTwXWNTY3qIkpZ qFKDltFiMJ3pS/XDMYfeyJCA/CkQ4XfIOKHvo4gE6hETqdFfio9kflzbRJm6TNYEIQhP1eUkMVlJ FWbSkr3cIEZxyarOve6OocOpq0aFVaOYckelK9Wkcoo5mP2VJxs1Y18F21ZrvjWuLLWstbS21c1+ VloDLadmQVta0552MuMcbTdR21rXvtYs2NRmNKGqxXWqE27u5J5IRapV/E1VkQDVX/oqA9A7/faq 4NOgcUdJVpKhEranaeYzaWvPqMYUuHBMWwMrGrit5ZVF9ItgKPV6OVhylHYZlSkn12u5yr6pQh+M WXJwisK13ZWi0S0Na06Qgf8IGOKgVM2uS9eqV6I6Fm56Yxd5N9ZcfNaxS3e8aU0rteAHIfWegovi Wc3oRD1W1asgLrB+l7cZBzjgvwFO4hkF3OKslvWd7fTpfCHcxrkINTsSlqdHv9peDu+4nm1LZVJ7 3Fgh6xjHXCXxfk2M4kHMsISUnBx4p4xfX86SvZTtqKc2iSkHm85C5X2iKFW3WCKfd7KfVCJUyNxh pJ0qsazDKmI5l7V8LpmRmzlBBSJQXTyn5pj5+/OgFZcYZwLAz4RW9KJNyxpDDGKbrGX0pCmtTNkC gLTPnXNuZyxjdq6zTL19ox2DrGmjsk+87vujWIR7z1BjsWBbsy5aISjoSv//bzPklOtgJ4tdF2s3 hQ+VnJb+msmG2jq4CLqvlDF4YfvxNZc0je/sHjdSTS4RhCYUnX35Z2W23vqLiWHBSlUM7BH/msAI XvPcIKfHIepForwOpLl1HNVYrriJGTRcZ2d6YCgOOcyrFOKZj7xCcFNmoJktd68HzOJzjzVH7AZ1 jR+YX+KWdd4JUzBD8WinysF71Wbti8eZyyOCG3ZfB0f4ZmIQg12Lkq549TaV/03nwl7bzFu+8h49 92VS/1vLEUUv6mhYZl0S+WpJlKy9853e0QmOzkeys8pXw3KXZ5rqkgl01rkeGta0/OVdF/vYEff1 q0ua7GlXO7TMHvYg8rjT/7fdLadNltzB9lTao072t2n9YL0DOuTbQe5+eAvr+mhU3n0Hs5Ip/daz Q5WPDzf35Il+IwWO+Kwazl+qf0RexZrXgcrmNih7m8Fnv9mvlmd65K2dZcTH+GZpXhrNjdz4zQTh 8QNM9+4drm4Rsrv1wBc1wEgZ4gifvIJCnfi2afzizAG23aSjrHIxPEUPO5fnNX5xyg/edqyHd4no 7n2DIQx7/JnfsJu/889Dr/3GJtm3xN24iDsea9IB//i3jOPf6Y9k4/Od0bwPyh6H5gpw5rwty06p y1RmhxAL5vzu56KOy7Bs6STwlUwO6IBO5xwH6RAEaxiiAkvKQKqo+KBu5P8oZd/W788EcO2KK0iW 6unirAVvjQVn0AZv8FhqEAd3kAf95OsUrrb4RO48Dbd0S+jqr6uS6sM8y9VQbX84DwDZBwIR5s6g 8KeK0MiID/t8rQeTArPcjl9Yj/ekKthUr7seUHvST/0ujjk874LeS+Yuaejci/ToDshGCMFMB6T4 qIToq9lOiujqivF68Au/L/+4kPLGr5V65OZMDrIGj/ymMMcIjmqSL+SIjQJZb6z0bepkSWh+6cew x9/u7v+2z+C6sCAKceGuq+HI8OGSLMaCz7sWz8YCxv1QLoKELMOajxUHzsfuD4ucL/OCseR8sf0w rvYIMddWS5LEh9kM8Bn/EZBmFBDLxkzfiq1vYu/bKEzMPmznhg0FpQ5GWObkWAZWwhF22iyk1GwD m1EIT7DOfgwVYYI1xmAMAKBQ5lHV3gyZ9BEHWaMQCgGa0M4fC9IgB4M1HMER2ODJDtIhH9IxWAMB kgADImnW3tEIh1Aj7fCqNhIJO+wjlWvQ+KloXMmfcvHw+G8QHQMGoceFOOAABEjexFARXbHbhK31 dNGkMDIMnxDWYkkGFUS9tMzpiPIny+xVbMmiTlD1Ig/u6i57rPG4NpEW8/Ad51AoZSUZvSh3ym0M EZH3OkeKPE7AgkpttFASmQ3MlI8jPVHynst7rugMN/HCUpCu6lJJamYY/12EEwUu+/IO4qByJZtq MwbAMEFgFWnSJr8SF19q+frpEC1O8DZNCt+tLUXtCAWJSDYzyORL/yLz1N6owkyRrUCu/EqRJxPP rQrzMAcw5uwqGmNTEHHOKMmRBI0Oznqu1mLwEuGQluCxKOvwAj9vKXmJM0tNHR9LlZQqBM8ROLUS DanRtoATN32zmFjDMAcAMSGSYJRGBQFj63awJcEJO1uTO88TPfeiPLWTINPTPd+THlmTPYPwKbHw 0+gOC+sFyAgrGTUP2VptlcYz3s5iPAWzLCoOTSBRQUNS1txzPbfTuhQTLGuyvaSNrKrvN+kLQYOQ Dt0QHIUxKJXTGIOy4P+GsunG0RyZcsHW0Q/hCw9x8Tt58EG9kkJbkQutKmXmjqeoTS7pryozTs3Q DBkH1IiIsmpaSgnRaybtr468s/pGERSlj0UL9B/lE0KX1Eaz1MVwlAhX7SlPbUMPFDVLdEgDb7hG E76kMP2+tCcnrAONCk0t85Yg8z1n1B2dczahMU+lER2rMepyLjh1UxI35eZsbg+l9KQmEeeU0jcN NcT8ShAZS+hsiZXSUUgV5UThkyDsVFMJIzxvLJk6lZo4VVRL1VQ31Urb81RXVR9J1dRS0z5jde4s qCPxc1Zzyj/3biurhbmKLjQrlFD/EpWo9C+IFdxcdQ3FbzEVkbvOcUv/kVLbavFHDUwEY8++Rg8d T0Y410sqyzJao3LafhWkkJRR1etalS3aoGvtkNVHlXVCl1UsTWT2jk2pztLndG8219KrHjNIMaYv EfVtwjVHi4rkxiyt/kjDcMwbd5Xs2LVN3VVLX9H45PX8johOpRVUKfPi5DQN389e4VRFfxFECZZJ ixFIGxMDMXMHHbaoWCkO91Q2jW0asxVQActPPRT0kE0rMZU3PYlmaxM6qRVPtYaWmpMvF1Abuyse 95U63zBp1zVVWXVYjJXqqLZKEyM7r5RVtxY9WZZrv9YfvfYYZbVLy3b5gHUw4y1XuarVcnJaf3Xx rtDwhNUvCc8ndRZs/yNSPotH0yQ0YhMRlszwYwKXDpH0VfcOZ0cwZA91aNeQcUGWh4LRRYItX9U1 bw1jZxSAbznubxmTcgc2XRuRixZ0EIvP/SoRUowTGLEUdDWwZA2W40YXAyXzchFycZonMcNP8jyX NGWsDwHTWVnXFpFxRL3U/mSXc+WxYEsTYcsxMAezdqlkMw4AAmLSNZ0RZrP3ZWV23SYw50T3NgPr aU03zIhzYa2yUp/PIYy2Um+2BOvJAmkzA6P3MYSnemWSfvP3Bh8Gd/XXf20wczf3fwe469ZTgIGo Pm/1PjOyLdE2Cl8NeP9z8Drvbk/SgkXSYCKzbimYDR+YgANDbBGUd/8B119xcnDlrHDBVXe3UuNm j0+bsq7sY81YcV4T9Q8Tl/Ys94OlF2vNE1+XFYgnNF4p0PKKEk/p7V4VFYmHrEcpB0p9CGBf93zJ NBK5b4cPI4SrbXdr9FlPk3jT9IsbNEzF9Bab1HjnFDlFc3FttSrhb3at+IoLI4ujTHths4599ve8 N+ge15J+yzprjxvNdwt30kIsVWiP1n0PZ3A/UFsxNI7rN2ofWZInTWwn2ZJfq5IvWZMbbW9XcToZ GJTjjtP0s2xFspCY0IH3z4Mn84IhmJV1GPE2mVwWh3hyt3O5eGTdlHtC97scEG+zcOhCqaYaeTm5 ddoS6He7dUdjVJb/Vy4xFKB/f/hdpxm4uHQU2xfbEkywiJSifBllzTJD7ayJKwucN2x15cxqm7kx +Nd6IG+FR5jArLli50mmUNlqBRZlqcqNyXJ+l/CMc9Pu1Bl3aLmd5wp7t9eOERqPacd9Py9p2tHL dtNQQPKiGPl7H5d9y7dok7NajVigVQM7a/mjR/qzDFhVSRql/SeTU5qluTKSVTOBzZaNPVKNX5dB dxmmUXKpTBP8yJjVJlhN5VYlWxqSe3g+I/SdcZmEwVTYxtWKktS8xvhwpw9Fj6+FNSgrdXn1ekhH enGniNoyspiaIZasRy1Ko4+edXlD0dKdGc5N9XXFLGyc2xHgQhGT/9IZrGPrpfs2qYM4YuU5lPuN FzHWp2cX85zXjdLEMwEOxv7ZiTuGmfOaLuaYAA8wZhOaZLcskW0TGxNQUN9WsSiVymC3ZX82o1Eo sjj6UW5Tsp1ZLbL2pE31Uzm0tY1npWsbt28npBsnt3s7edgZf20tpml6gUVZ5GzaY2ltbSfqlLWY YbuKCls5oCUar31bM263oIV3qct6iRkOoxvQubtalUF7Oj9mmAEVhbWakmRYhLb6kCLbugEDuGnU r7k7EmkYdsE5V69ZF2nbdd84mLFxbv7147q1ddFUL+E7vv0ic4O7p+1bqcNKZN12PytUqscCn804 pvaZSQFa+9RKu/+qe8HlwqTvlC8VuuYsO1/rsKGJmRHNFw1pm1IrOpCJuavxZcaTbn1V+2iHc8Ql 47Z/XMinJciH3MhzcK+PXMmrRawVlbjJlqZJeQhNmY2PEYNfGRFhmXbhNpZTuZSX/LqNWmuTdYvr W2LplY83aU3be7xl3Lyxmueu+vTQ9Q5Z1IZd1KvfG8wDJcnbtczH+sy9O1aIiqd9y93MdKofyG/r DeMUG/r8jQTx25EZRcSXvMmLFNC3O5dFztO81azDu7CN8bCBub7e9LihWifP2cv3nCwo+zWr7LJh fcX7tGY5G2nD5I+hK98YGxyxb5vpmsfTd8cvVQ5Zm9WTpc/zd7adL/LYv6XImx3aB1rMYzvaq72F kt3as907nl3buz3Ppt3bw13aX9uHxd3cmQzcz13dRYPb193dQRjb313esThVWcPe7x3f813f953f +93f/x3gA17gB57gC97gDx7hE17hF57hG97hHx7iI17iHR62J97iLx7jM17jN57jO97jPx7kQ17k R14AKp7kTx7lU17lV57lW97lXx7mGT4gAAA7 --=_related 0030F437C1257F33_= Content-Type: image/jpeg Content-ID: <_2_0C48449C0C483F740030F437C1257F33> Content-Transfer-Encoding: base64 /9j/4AAQSkZJRgABAgEBLAEsAAD/7QAsUGhvdG9zaG9wIDMuMAA4QklNA+0AAAAAABABLAAAAAEA AQEsAAAAAQAB/+Enz2h0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8APD94cGFja2V0IGJlZ2lu PSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4 PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS42LWMwMTQgNzkuMTU2 ODIxLCAyMDE0LzA4LzI5LTAzOjA3OjUwICAgICAgICAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9 Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRm OkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpkYz0iaHR0cDovL3B1 cmwub3JnL2RjL2VsZW1lbnRzLzEuMS8iCiAgICAgICAgICAgIHhtbG5zOnhtcD0iaHR0cDovL25z LmFkb2JlLmNvbS94YXAvMS4wLyIKICAgICAgICAgICAgeG1sbnM6eG1wR0ltZz0iaHR0cDovL25z LmFkb2JlLmNvbS94YXAvMS4wL2cvaW1nLyIKICAgICAgICAgICAgeG1sbnM6eG1wTU09Imh0dHA6 Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9tbS8iCiAgICAgICAgICAgIHhtbG5zOnN0UmVmPSJodHRw Oi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvc1R5cGUvUmVzb3VyY2VSZWYjIgogICAgICAgICAgICB4 bWxuczpzdEV2dD0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlRXZl bnQjIgogICAgICAgICAgICB4bWxuczppbGx1c3RyYXRvcj0iaHR0cDovL25zLmFkb2JlLmNvbS9p bGx1c3RyYXRvci8xLjAvIgogICAgICAgICAgICB4bWxuczpwZGY9Imh0dHA6Ly9ucy5hZG9iZS5j b20vcGRmLzEuMy8iPgogICAgICAgICA8ZGM6Zm9ybWF0PmltYWdlL2pwZWc8L2RjOmZvcm1hdD4K ICAgICAgICAgPGRjOnRpdGxlPgogICAgICAgICAgICA8cmRmOkFsdD4KICAgICAgICAgICAgICAg PHJkZjpsaSB4bWw6bGFuZz0ieC1kZWZhdWx0Ij5FbWFpbF9TaWduYXR1cmVzPC9yZGY6bGk+CiAg ICAgICAgICAgIDwvcmRmOkFsdD4KICAgICAgICAgPC9kYzp0aXRsZT4KICAgICAgICAgPHhtcDpN ZXRhZGF0YURhdGU+MjAxNC0xMC0xNVQxNjoxMzo1OCswMTowMDwveG1wOk1ldGFkYXRhRGF0ZT4K ICAgICAgICAgPHhtcDpNb2RpZnlEYXRlPjIwMTQtMTAtMTVUMTU6MTQ6MDRaPC94bXA6TW9kaWZ5 RGF0ZT4KICAgICAgICAgPHhtcDpDcmVhdGVEYXRlPjIwMTQtMTAtMTVUMTY6MTM6NTgrMDE6MDA8 L3htcDpDcmVhdGVEYXRlPgogICAgICAgICA8eG1wOkNyZWF0b3JUb29sPkFkb2JlIElsbHVzdHJh dG9yIENDIDIwMTQgKE1hY2ludG9zaCk8L3htcDpDcmVhdG9yVG9vbD4KICAgICAgICAgPHhtcDpU aHVtYm5haWxzPgogICAgICAgICAgICA8cmRmOkFsdD4KICAgICAgICAgICAgICAgPHJkZjpsaSBy ZGY6cGFyc2VUeXBlPSJSZXNvdXJjZSI+CiAgICAgICAgICAgICAgICAgIDx4bXBHSW1nOndpZHRo PjI1NjwveG1wR0ltZzp3aWR0aD4KICAgICAgICAgICAgICAgICAgPHhtcEdJbWc6aGVpZ2h0PjY0 PC94bXBHSW1nOmhlaWdodD4KICAgICAgICAgICAgICAgICAgPHhtcEdJbWc6Zm9ybWF0PkpQRUc8 L3htcEdJbWc6Zm9ybWF0PgogICAgICAgICAgICAgICAgICA8eG1wR0ltZzppbWFnZT4vOWovNEFB UVNrWkpSZ0FCQWdFQkxBRXNBQUQvN1FBc1VHaHZkRzl6YUc5d0lETXVNQUE0UWtsTkErMEFBQUFB QUJBQkxBQUFBQUVBJiN4QTtBUUVzQUFBQUFRQUIvKzRBRGtGa2IySmxBR1RBQUFBQUFmL2JBSVFB QmdRRUJBVUVCZ1VGQmdrR0JRWUpDd2dHQmdnTERBb0tDd29LJiN4QTtEQkFNREF3TURBd1FEQTRQ RUE4T0RCTVRGQlFURXh3Ykd4c2NIeDhmSHg4Zkh4OGZId0VIQndjTkRBMFlFQkFZR2hVUkZSb2ZI eDhmJiN4QTtIeDhmSHg4Zkh4OGZIeDhmSHg4Zkh4OGZIeDhmSHg4Zkh4OGZIeDhmSHg4Zkh4OGZI eDhmSHg4Zkh4OGYvOEFBRVFnQVFBRUFBd0VSJiN4QTtBQUlSQVFNUkFmL0VBYUlBQUFBSEFRRUJB UUVBQUFBQUFBQUFBQVFGQXdJR0FRQUhDQWtLQ3dFQUFnSURBUUVCQVFFQUFBQUFBQUFBJiN4QTtB UUFDQXdRRkJnY0lDUW9MRUFBQ0FRTURBZ1FDQmdjREJBSUdBbk1CQWdNUkJBQUZJUkl4UVZFR0Uy RWljWUVVTXBHaEJ4V3hRaVBCJiN4QTtVdEhoTXhaaThDUnlndkVsUXpSVGtxS3lZM1BDTlVRbms2 T3pOaGRVWkhURDB1SUlKb01KQ2hnWmhKUkZScVMwVnROVktCcnk0L1BFJiN4QTsxT1QwWlhXRmxh VzF4ZFhsOVdaMmhwYW10c2JXNXZZM1IxZG5kNGVYcDdmSDErZjNPRWhZYUhpSW1LaTR5TmpvK0Nr NVNWbHBlWW1aJiN4QTtxYm5KMmVuNUtqcEtXbXA2aXBxcXVzcmE2dm9SQUFJQ0FRSURCUVVFQlFZ RUNBTURiUUVBQWhFREJDRVNNVUVGVVJOaElnWnhnWkV5JiN4QTtvYkh3Rk1IUjRTTkNGVkppY3ZF ekpEUkRnaGFTVXlXaVk3TENCM1BTTmVKRWd4ZFVrd2dKQ2hnWkpqWkZHaWRrZEZVMzhxT3p3eWdw JiN4QTswK1B6aEpTa3RNVFU1UFJsZFlXVnBiWEYxZVgxUmxabWRvYVdwcmJHMXViMlIxZG5kNGVY cDdmSDErZjNPRWhZYUhpSW1LaTR5TmpvJiN4QTsrRGxKV1dsNWlabXB1Y25aNmZrcU9rcGFhbnFL bXFxNnl0cnErdi9hQUF3REFRQUNFUU1SQUQ4QTlCcDVrMTlvdlVPbTI5WEttTmZWJiN4QTt2Z2VM ZnpBMkhJRUduYjUwcGx2QU8vN3YxdUFOVmtxK0VmT2YvVk5TWHpqcUJpRXoyU1J4c0dJQVhVWGY0 U3ExNGZVVWFsVzMyL3NmJiN4QTtESDRyOWJIODdLcjRmOTMvQU1RbStsNnBmM3huSnRZNDBpYmdH TFhDTXpEN1h3elc4TzNneWxnY2pLSURrNHNzcDN0eTkvNlloSEZyJiN4QTsyaHBISFh0V1J2OEFt akk3TnR5N2grUGdsVnpvT29haGJXYWFocWswY3R0eGR6WWhiY1NTcU4yWXQ2alU4QUtEeHJ0U1Fr QnlEankwJiN4QTs4cGdjVWp0L04ycy9hamhCcWtha1IzVWNvRk9JbWorSWdlTEl5ai9oY0ZodTRa amtiOTRSYUdRcis4VUszZ3BMRDd5RnlMWUw2cERQJiN4QTtyK3N4eVRoTENGa2pkMGpxOTRDd1Jx Vk5MTitvQlB3MUhnV0JybGdnTy83djF1SExVWkFUNlIvc3YrSS9IbXBMNWsxdHExc0xkQ3JjJiN4 QTtUV1M5SU5GTEVLVFpKN2JpdmZ3cGg0QjMvZCt0aitheWZ6Ujg1ZjhBRUk3VDlYMUM2dVdnTnFp K2t0WkhKdVVITGFnWDFiZU5XRkR2JiN4QTtSdHYxUk1RRzdIbWxJMVgrNi9USDlLWmNydjhBMzFI L0FNakcvd0NhTWpzMzNMdS9IeVNmVVBMU2FsY0NXOEMvdEkxR0wxUTFLbGFxJiN4QTt2QmtKb0tH bmM3Z1pNVHJrNHVUUytJYmtoNHZLbHhaVDJrOWpQR3pRQ1NPUkpFNFZWeU9MQjBxN09nVUQ5NXly NDlhdkhmTmdOSVlrJiN4QTtHSjVmajdQTzJSeG1VMTlSVlh3NHNXL1dGeXR6aGZWZGlsSmRWMXkr dE5TVzJ0NHJLU0lJcnpHZThFRXloMklxSXZUZmI0VFFsdC9vJiN4QTt5Y1lnaHhNdW9sR2RBUnJ6 bFIrVklHVHpacUN5ZWw2T21MSVVMcXJhbXZMWmQvaDlFZkR6MnI0YisyUzhNZWZ5YWpySlhWUS8w LzdGJiN4QTthMTgxL1dydGJXMk5oY3o4d0pJWXI2TnBGUUdqa0lvWXN3OE52bU1CeDEzL0FDWlEx bkZLaHdrLzFoYWUrdkwvQU1zOG4zeC84MTVDJiN4QTtuTTRqM2ZjbGIydXFQY3N5eHVBWjBrV1I3 cVJRSTFaV1pERW5OZWdJK25KV0hITUozOGY1MzZGRDlBWHNWdmJMQ3Rzc2x2eEphRU5BJiN4QTs3 QmZoS21WZVIrSmV1dzN3OFFZZmw1QUN1SGJ1MisxTU5PaXU0NTVUTEU4Y2JxZ1JXbmFjQXJ5TEdy bW9yeUEySGJJbW03RUpBbXg5JiN4QTt0cGhrVzkyS3V4VlN1cnBMYUwxSFdSeFdnV0tONVdyL0FL cUJqaEF0ak9mQ0wvYWhCcjFpVkxlbGQwQ0J6L29WM1doRmVucFZydDluJiN4QTtyaDRUK0MxZm1J OTB2OUxMOVNqSjVsc1kzQ1BiWDlUV2hXeHUzRkFhVnFrVGRjUEFmTDVzVHFvanBML1N5L1VqMHZZ WFFNRmxBUFRsJiN4QTtES3ArNHFEa2FiaGtCNy9rVk9lK2tWNFZndDVKdWI4WlBoWk9LOFNlVlhD anFBT3VJQ0paRHRRdEJvUE5QNlFsbWI2bWJFclNHMXJJJiN4QTtKS2lwQmFTaFVFazcwWGIzeVhw cHFIamNSUHA0ZTdmNy93QmlNanZMZ09xM05xMFhKU1N5RXlxR0JIdy9DSzk5cWpJMDJDWjZqOUtU JiN4QTtOK1hYazltNUd4clZHaklNa3BVcTFEUXFXNGtCaHlGUjlyZnJoTWllYTRjRWNZcVBKU0g1 WmVTMUlLV0JRZzh2aGxsQTNVS1FSeW93JiN4QTtwdFE3Ym54T0MyMU1kSThwNkhwRng5WXNJWGlr OVAwUnlsa2NDTUJRRkFkbTIrQVlGUU12NWIrVDVuZDU3SnBpNVkwbG1tY0taR0R2JiN4QTt3Vm5J WGtSMUdXZUxMdmNFZG5ZTHN4djM3OC9JN0tuL0FDci9BTXEvV2ZyUDFWL1ZEdElwOWVhaXRJM04r QUwwWGtldE92VElXNW9DJiN4QTtuQitXM2s2SDB3dGp5V0ppNks4a2pLR0xLNStFdFRyR051bjM0 MmxQckxUZFBzRWRMSzJpdGtkaXpyRWlvQ1R2V2lnZU9CVXBieUw1JiN4QTtXYTllOU5sL3BNa3J6 UEo2a203eWxta05PVkJ6NW5rQjFGQjBHU015V25IcDRRSk1SOVhORG44dWZLWmpXUDZ2TndSZzZE NnpjSGl5JiN4QTtra0ZTWHFOelg3c0Z0eUpzUEpIbDJ3dlV2YldHU080UnZVVmhQTng1RWNTZVBQ anV1MUtVcFR3d1dxbnFIa0h5dHFOMU5jM3RxODhrJiN4QTs3RjVGYWFiMCtiS1VMQkEvRUhpZGlC dDJ5d1paQVU0YzlCaGxMaWtMUFBjbXZseWFsL0wvQU1yU3E2dGJTRlplUHFBenpIbHhqOUdwJiN4 QTtxNTM0YkU5ZS9YSVc1WWlBS0MwZmwzNVM5UlpHc3k3S0dBNXl5TnM0WU1OMlAydVo1ZUozTytD MHB6WjZUcGxrOGtsbmF4Vzd5LzNqJiN4QTt4b0ZMZk1qcmlxTHhWSjlZOG82QnJFNW4xQzJNc3hR Ujh3OGlIaUNTQjhMRG9UWDU3NUlTTlUwbkJBejQ2OVNYditXWGtwNVpKWDAvJiN4QTtuSktwVjJh U1ZpYWdBdHV4K0xhdkw3VmQ2NExibWgrV1hrd0x4K3BOU2lqKyttclJEVk4rVmFyMlBXbTNUYkcw VW1PcytVOUQxbWFLJiN4QTtYVUlubGVLTm9WNHpTeGd4dVFXVnVETHlCb0sxeVVjaGp5Y2JQbzhl VTNJZmFVS1B5OThwQ0Y0VnNpa2NuUGtxU3lvS1NoUTYvQ3dvJiN4QTtwQ0x0N1lESW5tM1k4TUlD b2dBZVNrMzViZVVHY3ViUjZtTm9oKy9tb0kzQkJRRG5zdERzdlFkc0Z0aVphZDVXOHZhYzBMMmxo REhMJiN4QTtBWGFLYmdESXBrKzNSenZ2Z1ZOY1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGly c1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWJiN4QTtkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNW ZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkJiN4QTtp cnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWVXJxOHRiU0wxYnFaSVk2MDV1d1VWOE4r K0VDMk01aUlzbWtJUE1mbDRxJiN4QTtYR3FXaFVJSkMzcnhVQ0VFaHE4dW0zWER3SHVhdnpPTCtk SDVoUms4MitWNG5DUzZ0YVJPYTBXU2FOQ2FHbFJ5STI5OFBoeTdtSjFtJiN4QTtFYzV4K2FQUy9z SkZEcGN4TWg2TXJxUWZwQnlORnVHU0o2aFN2TlFpUzNjd1RRK3Nkb3pJNDRCajBMVU5hVjhNUUdN OGdBMkl0SjRiJiN4QTszelE5eCs5ai9jdVdUakhHQXlPbEtqNDJaV1U3MGJrQlhiZnFKMUZ4b3p6 WHVOdmNqZEQxcVM3dENiOExiWGFFck5DdzlNcTFUOEpVJiN4QTtzKzRXaHFEUTF3U2pYSnQwK2N5 ajZ0cEp2a0hKZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGly JiN4QTtzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNW ZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzJiN4QTtWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRp cnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWJiN4QTtkaXJz VmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXFoTGYyRUxtT2E1aWpjZFVkMVVpdnNUaG9zRGtp TmlRcjRHYnNWZGlyc1ZkJiN4QTtpcnNWZGlyc1ZkaXJzVmRpcnNWZGlyc1ZkaXJzVmRpcnNWZGly c1ZkaXJzVmRpcnNWZGlyc1ZVYnhieG9hV2pva3RmdFNBc0tmSUViJiN4QTs0UlRDZkZYcDVvRVIr YU9CQm5zUy9wZ0t3aG1BRWxEVmlQVit6V253MStuSmVuemFxemQ4ZmtmMXFNMXY1eE1nTUY3WXFt NUt5VzhyJiN4QTtuYzlpc3Nld3h1UG14bEhQZTBvL0kvckNaUnRxQVFDUkltZjlwbFpsQitRS3Qr dkk3TjQ0dXRJVy9zcnk1bGlsOUMyazlKWFgwNWl6JiN4QTtxZWZFMUh3YkVjUHh3Z2dOV1NFcEVH aDhWQ0RSTHhMYWVFWEt4UjNKY3l3eG9DbzU3RUlUUXFPUHc5TUprR01jRWdDTHExa0dqM3NjJiN4 QTtzVEdHMEhwU0dUMTE1Q1Z0eWFFOGZmYzRtUVJIRElFYlIyUHhmLy9aPC94bXBHSW1nOmltYWdl PgogICAgICAgICAgICAgICA8L3JkZjpsaT4KICAgICAgICAgICAgPC9yZGY6QWx0PgogICAgICAg ICA8L3htcDpUaHVtYm5haWxzPgogICAgICAgICA8eG1wTU06SW5zdGFuY2VJRD54bXAuaWlkOmJl ZDMxM2NiLWQ1Y2EtNGQ4NC04OTRhLWVhZTQ0NDc5Njc4NDwveG1wTU06SW5zdGFuY2VJRD4KICAg ICAgICAgPHhtcE1NOkRvY3VtZW50SUQ+eG1wLmRpZDpiZWQzMTNjYi1kNWNhLTRkODQtODk0YS1l YWU0NDQ3OTY3ODQ8L3htcE1NOkRvY3VtZW50SUQ+CiAgICAgICAgIDx4bXBNTTpPcmlnaW5hbERv Y3VtZW50SUQ+dXVpZDo1RDIwODkyNDkzQkZEQjExOTE0QTg1OTBEMzE1MDhDODwveG1wTU06T3Jp Z2luYWxEb2N1bWVudElEPgogICAgICAgICA8eG1wTU06UmVuZGl0aW9uQ2xhc3M+cHJvb2Y6cGRm PC94bXBNTTpSZW5kaXRpb25DbGFzcz4KICAgICAgICAgPHhtcE1NOkRlcml2ZWRGcm9tIHJkZjpw YXJzZVR5cGU9IlJlc291cmNlIj4KICAgICAgICAgICAgPHN0UmVmOmluc3RhbmNlSUQ+dXVpZDoy NTllZWE5ZS00Yjc1LTI4NDgtYThiNS1mYWY0MzUwOWFkMzA8L3N0UmVmOmluc3RhbmNlSUQ+CiAg ICAgICAgICAgIDxzdFJlZjpkb2N1bWVudElEPnhtcC5kaWQ6ZDI4MmYxNzEtNzQ5YS00N2ExLWJk YTMtZGRlYzM5ZTJlNTRhPC9zdFJlZjpkb2N1bWVudElEPgogICAgICAgICAgICA8c3RSZWY6b3Jp Z2luYWxEb2N1bWVudElEPnV1aWQ6NUQyMDg5MjQ5M0JGREIxMTkxNEE4NTkwRDMxNTA4Qzg8L3N0 UmVmOm9yaWdpbmFsRG9jdW1lbnRJRD4KICAgICAgICAgICAgPHN0UmVmOnJlbmRpdGlvbkNsYXNz PnByb29mOnBkZjwvc3RSZWY6cmVuZGl0aW9uQ2xhc3M+CiAgICAgICAgIDwveG1wTU06RGVyaXZl ZEZyb20+CiAgICAgICAgIDx4bXBNTTpIaXN0b3J5PgogICAgICAgICAgICA8cmRmOlNlcT4KICAg ICAgICAgICAgICAgPHJkZjpsaSByZGY6cGFyc2VUeXBlPSJSZXNvdXJjZSI+CiAgICAgICAgICAg ICAgICAgIDxzdEV2dDphY3Rpb24+c2F2ZWQ8L3N0RXZ0OmFjdGlvbj4KICAgICAgICAgICAgICAg ICAgPHN0RXZ0Omluc3RhbmNlSUQ+eG1wLmlpZDo1NDI1NDI4NS05NmY5LTRhYzAtYmMyMy05MGYz NGQ4Y2RlZTQ8L3N0RXZ0Omluc3RhbmNlSUQ+CiAgICAgICAgICAgICAgICAgIDxzdEV2dDp3aGVu PjIwMTQtMTAtMTNUMTY6Mzg6NDgrMDE6MDA8L3N0RXZ0OndoZW4+CiAgICAgICAgICAgICAgICAg IDxzdEV2dDpzb2Z0d2FyZUFnZW50PkFkb2JlIElsbHVzdHJhdG9yIENDIDIwMTQgKE1hY2ludG9z aCk8L3N0RXZ0OnNvZnR3YXJlQWdlbnQ+CiAgICAgICAgICAgICAgICAgIDxzdEV2dDpjaGFuZ2Vk Pi88L3N0RXZ0OmNoYW5nZWQ+CiAgICAgICAgICAgICAgIDwvcmRmOmxpPgogICAgICAgICAgICAg ICA8cmRmOmxpIHJkZjpwYXJzZVR5cGU9IlJlc291cmNlIj4KICAgICAgICAgICAgICAgICAgPHN0 RXZ0OmFjdGlvbj5zYXZlZDwvc3RFdnQ6YWN0aW9uPgogICAgICAgICAgICAgICAgICA8c3RFdnQ6 aW5zdGFuY2VJRD54bXAuaWlkOmJlZDMxM2NiLWQ1Y2EtNGQ4NC04OTRhLWVhZTQ0NDc5Njc4NDwv c3RFdnQ6aW5zdGFuY2VJRD4KICAgICAgICAgICAgICAgICAgPHN0RXZ0OndoZW4+MjAxNC0xMC0x NVQxNjoxMzo1OCswMTowMDwvc3RFdnQ6d2hlbj4KICAgICAgICAgICAgICAgICAgPHN0RXZ0OnNv ZnR3YXJlQWdlbnQ+QWRvYmUgSWxsdXN0cmF0b3IgQ0MgMjAxNCAoTWFjaW50b3NoKTwvc3RFdnQ6 c29mdHdhcmVBZ2VudD4KICAgICAgICAgICAgICAgICAgPHN0RXZ0OmNoYW5nZWQ+Lzwvc3RFdnQ6 Y2hhbmdlZD4KICAgICAgICAgICAgICAgPC9yZGY6bGk+CiAgICAgICAgICAgIDwvcmRmOlNlcT4K ICAgICAgICAgPC94bXBNTTpIaXN0b3J5PgogICAgICAgICA8aWxsdXN0cmF0b3I6U3RhcnR1cFBy b2ZpbGU+UHJpbnQ8L2lsbHVzdHJhdG9yOlN0YXJ0dXBQcm9maWxlPgogICAgICAgICA8cGRmOlBy b2R1Y2VyPkFkb2JlIFBERiBsaWJyYXJ5IDExLjAwPC9wZGY6UHJvZHVjZXI+CiAgICAgIDwvcmRm OkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgCjw/eHBhY2tldCBlbmQ9InciPz7/ 7gAOQWRvYmUAZMAAAAAB/9sAhAABAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB AQEBAQEBAgICAgICAgICAgIDAwMDAwMDAwMDAQEBAQEBAQIBAQICAgECAgMDAwMDAwMDAwMDAwMD AwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwP/wAARCABUAkEDAREAAhEBAxEB/8QA pAABAAMAAwEBAQAAAAAAAAAAAAYHCAMEBQkCCgEBAAMAAwEBAAAAAAAAAAAAAAQFBgIDBwEIEAAA BwEAAQIGAgEEAwEAAAAAAgMEBQYHAQgSExEU5GaWFxUWIiExIwlB1VcYEQACAgEDAwEDCwICBgsA AAABAgADBBESBSETBhQxIgdB4TKS4iNjkxVlFlFhcSSBkbFCUjOh0WJyU9M0lFUXCP/aAAwDAQAC EQMRAD8A/v4CICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICIC ICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICIC ICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICIC ICICICICICICICICICICICICICICJHLg+lYypWmSgjx6c5H1ybfQyks1cvYpOVaRjpeOPJs2b2Od u48rtMnVkknCCiifxKVQhu8Nzux1re9Et17RcA6HQ6EjXQkEA6ewkH/Ayt5m/LxeHy8nANYzq8a1 qzYrNWLFRihdVZGZNwG5VdGK6gMp6itfHWwX+545Q77pUxXpWy6HWa9fDNqpX3VdgK6wtdfiZdpW o9vITU/JPyRXzJuGduHPVXBz976Uy8KTkzl6sXH5C3Fw1daaXZNWYMzFWILHQKBr/QDQf3mY+G3I +Q834Xx/kHk12NbyfJYtOXtopamqlMimuxaED22u4r3HWx33OSToo0UXaK2bmAiAiAiAiAiAiAiA iAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiA iAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiZg8pfImkePlH6+vkZbFYu3tJuux 0xBxTN5FNZ1eMW6wipR69lI1Jk7lEuqqN+f5cOm1WN3peJ/63XCcTk8tk7cU176yrEMSCV16kAA6 gdNf8R/WeV/Ff4kcF8OuC7/kFWYcXMS2lLKq1atbSh2V2M1iBWcbig66qjk6BZy+Hdtb23xvybje Hnog1SpVXz99ydZIM+Ppmj1yIr088hzt3bxJ/DJTjJw1TccMX3FWynPTzhfj355BQaOYv1ZW7ljW DaddA7FlB6DQ7SDp/Qic/gxzFfL/AAz4gV05FJw8GjEbuqF32YtNdNrVkMwesWq6B9RqyN06dbdk b8hHaRV81NXbC6fWqr2m2tbC27Xv62wjKe+r8bMpSXXNgbWEr0j62RiaZW8e4Ip16U3D+lNcyMBM Uvhvmb0Co6oVO7cS4YjTRdumit7WHs/uNdjk+QV43k2L4wcbJe/LxL8hbl7PZRMdqUsD7rlu3B8i gAJS4PcB10WwpPRFmggIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgI gIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgI nnS8xEV+NeTM9KR0JDxyPXEhKy75tGxrFuXvC9XePniqLVsjwxuc6Y5yl+Pf9xzrrstcV1KWsJ6A Akn/AAA6mRszNw+OxXzeQtqow611eyxlRFH9WZiFUf3JAleu9O47kzRlGq0vonG8HFWJ5MV6TrDW ukjpwjxaGbR83Mzke1mZeXasTrIJNvcbESMmdy4bEWRMpLXC2pvyXWrVioDBi2q6a6gKSACdCT11 12g6HTOXeUi7KOLwOJdyQWiu5rKXoWnZbuNapbZai2WWKpZVTVApVrbKg6Fvg95b2bRntv0qeot1 NoFGrFnbQlPpW2wfb4jU9ensfuuj6PWzNbB80Wtr5HU6k5dKqNynVbuJxGLN6mqfUFvT+Bpw1x6a sqvtZLpq70ts3VLaldbarpu7rOANehCF/pHUfgL4w8p5LfzPKchwOceR4HFylqx8bk6vVDH5C3js nNzKdt27snj8fHexigLI+UmKdalNb6K8aKrLxXlRvmd6DfL5b8mpNzSq8DRXFgmYqg1t1p9ovs9T UbTSa92DzySQtC1VcHcLLsfeWlZePTKTiDzhT1HM31vweLl4lVVefZXuZ9oLsK1RX2u26wbdw0Ab QKjnXVZ6T8L+JzMT4s+Q+N+RchyGZ4hg5ooqxWusrxKWzr8u3GF+LT2sNxeaHLs9W5sjIx1C9u7R rPsfkhjWG+cS1NkpySrVCpWD2xSXaqw9sk4CqWi726gvpBCosGETKrowzqPpMaU7ONJyKauuuDlI kudz1SFTw/I8n40MhFD5VmUuh1UMyolgG8kjqC7dW94jQdRppqeT+JfhXgfx3bhcm+3F8fwfH8g2 Ka8h6aL8rIxHcY6JXYRWyYtINdIFFdncYKtht3bJ0Dy0w3O6xJ2SQt7eacx1ZPay1Gv9bLXR9Gpo Nni7dCBlnUQZlLNGLsrhy0fKNF2jfhlFypkKbvM9icFyeXctKVlVL7d7a7AfZ9IA6gkaAjUE9BrP avIfi/4H45xVvJ5GYt9tWL6j09OhyWQBWIFVjV7bFVg712mtq01awKoJlqZzokPp0E6sUHHT0dHt 5mRhClsEcSMduXEWZJN2sk1K5cqIlQcnM3XRX9l4yeoLtXSKDpBZEkHLxLMK0U2lS5UH3TqBr7Ov +HUEagghlJUgnWeNeSYflOA/JYFeRXjre9X3ybGJr0DELuYjRiUZW22V2LZTaldtbos+EWaGAiAi AiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAi AiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAiAifPmyy3cV8pfIHX7dr+nPc7rnjPD6i5y6WtcX 3P2DxaTkaw7a1OClEWDKOfdJlbUzHnHKZ+y0++6ZTvzxy91dNf6lwmJx+Pj0jLfNNfcCnuEaBgWY akj707un0UXp7s/OvJ5h8H+K/kXmXMczyr+N43i9ec2DZkV+kRi70MuPU4RUbTArNQ3qTkZd5Lf5 gg/QNJVJdJNZFQiyKxCKpKpHKokqkoXh01E1Cd6U5Dl7zvO873nedGVIIOh9s/RCsrqHQgoRqCOo IPsIPygz9j5OUBEBExZ5KeQNHy+8ZLBaND2iOrx9ChXkbLuYJdzVb7bHdauCVPpkDKc6auf2SHsi beY6WZdRZWq0ags26urwpkdFw/FZObjX24bIbe0QQG95FDJvdh9LaV1T3A2oYhtB7fEPid8ROC8V 53h8DyWnLq448jUyWGotRl5DU5Ax8aqz/k96u4Jk6ZNlARqa3q7j6FMpZ15HVukyui29hb7c2qui Mak9qrNZth6UfBOPJTUZ19hVmjIZ1fIJVQsGrcXbOYIuut0rdoU8j1qmz+Jb3L4i7Jrpx2rrN9Jc MfvtW9PWouUkI30tgKaAdTom4tPJPG/iXxnB5fJczj5mYvE8lXjtQpXiwlR5rOtbi70rbLqJ7Ryb K8kM7aJWGyTUtPTI0XRMtr9fXcqXKRvUs1tG2ydsWeZdhWkWS+aO3m3FmtVn9Fs8p7ixr791Fxbq FaHj1ItjPOHiRT9cSKjf1375WdbaAKxVWUpC6WXVqlem1V93GQsASHO7cyAHTRAdPHcTgPFOO45r Wzbc/LXL5N8gtg8Xm3ZeaLTfffpkc7kJS7JXZjVmk0VZT2KGNmS1etw5Gwg870e+6HUsk3i5R9tl 4qwvM/g4q7Ms/PYMllKKrTbCSOpqOzyUtISj60nmWXH8wrDpLxD9JLqCrUjNWvz2ty8OrEvyMWtq 1Kh2KGzbaH3rq/ZAAC7DtTeQyE6htw2nh+Pg+N+S8h5HxHD8/m4+ZdXc2JVXkriG7j7MU412zGHJ PY7tecmoW5DY6tj5CJ23qWl9d2re9b06HY0mm+NTmNtF4NZ5WWhdMq7n5d9WKdEyDQrpWK02Ky6N WnnFsUrRmXH3V2Z4lw470xHjXrYlBRxeBhWHJyMwNRVtANbDozkHTWs2Hbt7mumh3Ae1W1PsXLfE HzDynCr4PhfGGq5XPN9llWbQ2j0Y1bruNebXgobTkHCNXd31nHewkrdV2l1TjdOsSVANH6jGouHL yRmCx9TnIXOTFqdYWSaRadc6XP4plVHjWWPHKSZykKsZL+R+VUVW43KftHyGRScrfhHRQBqymz3m 6nd94SwI1C/Jrt3ADXSes+FcLySePHH8rqDWvbZsx7a8P/L0ELWKf8pWuOy2bGvIAYr3u0zv2wx8 jAoijryGuaFS0IBs2uejSsT8rW1mhGDNnQ1V4A3XcdElbxCMtYrSrMWBR17RnzxCbQ64WVKmgVLs 5SzJCY+Jklya6Qfe11Jf3uhPXRV2JprtBQ6AanWH8PcPgrMjmPI+EXHSrO5OyvbSVCKuITV7yV6V iy685GWbNvdsTKTuO4WsJoZZZFsiq4cKpIN0ElFl11lCpIoopF6dVVVU/SkTSTIXvTG73nOc58ei pAJOg6kz0d3SpDZYQtagkknQADqSSegAHUkyuv3Nj/8A9WzX86q//tBL/TuQ/wDAu+o3/VM1/NvD P/l+M/8AdUf+ZJjD2OvWFL34CehpxD2klveh5RjJpeyt8eoq+4yXXJ7SvC99Jvj8DfD/AEEeym2o 6Wqyn+4I/wBsusLk+O5JO5x2RRfXoDrXYrjQ+w6qT0PyH5Z7I65NgIgIgIgIgIgIgIgIgIgIgIgI gIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgIgI gIgIgIgIgIgIgInyu8lH8noWs7tTa4lIx1jreRUGiQT+KdOEpNafm9PyW0RUlGHS6xO3eoOLymil wivee4jw3rL3vSk2/DomJgYuRdoaXyHcgjptWu1SD7dR7hJ6fL7J+TfidkZXkfl/P8Lxotr5LG4b ExanrYhzbbncffW6EbSGBygq6N7V13AnQbI8Qnz2U8TfF6Tk3jqRkpHx2xR9ISD5ws7evnrvNay4 dvHjtwdRdy6crqGOoocxjnObve973vejPc+qpzuaiABBl3AAdAALG0AHyAT2r4N5F+V8IfFcrKd7 MmzxvjHd3JZmZsKgszMdSzMSSSSSSdT1miRUz0iVRbNvzGmTb2sStiXkrXFxRJ2XqFMrto0W4QcC rw3W87PVPP4WzWKChHhiGKg8eNkWzhQvSJnMbneCdRxubkVi5EAoZtod2WtGb/hVrCqsR8oBJHyz I8v514twmc/FZeS1vL1VC2zHxqb8zIqqPsttx8Sq+6qptCEssRUcghWJ6Slchv28zTG9vTQb7Sa6 5kmrrKLrcIEmGfzEW5rrHrtpL0qXi+XuIbRdtI6QM5dxJDuGfEnDb5oinOFss/F4utql3Cm0Ai1E bvaHcdCHB2HVdDoG6HUNt0mH8N8g+IGdRn3mizk+Na1WwMnIq/S+5W1KblsxrK/VVrXkdxS9mOC9 e2yruhulruceLbVaw81myqaOatTLa1tau7rNRj8+a21pHPY5jMxsGeFkbLwsOWRWVZJP5qQ6g5MV bpjKpImTgLyHYDrgJ2d67SwZjYVJBILaheugB2ouo6ewnXXW+GDmHxbvL8o8mcW9b1oajHTEXIVG RbEqNT3/AHe9mqW3Ju2uQ+pdEK2mwr8DFLGcRcJERrg6XUTrsI1mzWOiYxDmSMq3RTOZIx0y96Xv fh3ped/8cEJ7bbBo7MR/ckzWY/HcfiObMSimqwjQlEVTp0OmoAOmoB0/sJ6465MgIgIgIlZr5DRH DqTcnYzhE5qSeTEtFNrrdmdckJSQW+ZfPXNWaWJGtqLu3X/Mp3rT/Jb/AJO/5/5CYM/KCquq6qoA OxCwA9g3Fd3QdPb7OnsmXs8N4Cy221q7wt9rWWVrk5S0u7nczNQtwpJZveP3fVve+l1kWs+J1BtC Tz2l0GlSdpTaLOqjB3gknL57DziaafW6sVUlni0JVCuFUSmXPEN2Ci6n+Sh/UYxx308lkNYq5Nti 0a6OyaCwr/dtNW/tuLafIPklTyng3DVYORfwnH4NvLBC2PVlb7MOu0AaGvHLGrH1IBY46VFz1ZtS WldI5Rs+lSbTuo3+wUmmV2SjpevRebW6bp2iTUkzdIqyDG5ztDk4ennoL7iaqCUVxnIvzs+oKdkG rgrkjiWc7jsND6KpLMh1IY2IHrAI6FFcF949pbVV11Gxhppm08S828nyk/lfIZGDwmNallKYWRbj ZlrqwLrk24j14/pG0ZVo7d1pr7bHJqsFoszxs9f8g3F1gE2zDNG9jcz6EZGso3PpGLTet7PMRsXH Xur7TBJ3DQIlapPViu7QxXRrj1wzMsdk+4Rv7y9tx1vEjGbcbjSE1JNgOm0ElGpbYh3jpWwNgB0D L10HnHm3HfESznMda6+LXkmyAiKmG9YYX2JWmVRydQyMus47EWZ1TLh2vWXai8LXvs17Ax+7UZ6U z1xBaLRuKRyKFWYmeL6XCprVht/IqNb/AHG0QkNbI6NuLVQiCMm3Rkl4x776r87hrxs7obX4zJX3 Q1OT1946ds+8dNURSVJQ9SpKhhoE0bcvsfH43n3A363tj8lwOqAULuObWDQu8rl5N9VeQiZKkKt6 Lc1Fu98hrKRVd2X1m23tmiZyPz56nRuSicNM0pweiOryvG8rVkfubg0mS6UwrMbz+1nh2CDPrh8o Zok9VUTKZVuZDitPG9lqntHqduof3wmu5RsI7ZY+7vYnQdSoB6HXtv5Tzn9Upz8bjnHA90V2YzHF bKKdm5zkLZ61KE+/9PUte+1jWt7soL1lK61vyfuucJe9BeM2030yCMWV/GQlZsT+WTkJJ3JIrNI9 Sn1e8ViVbwrRii4eOiShUfS+RTb9cKkdEbzMDhcbMOlubjVA66FmUDQAdTvZGGpJAG3X3SToNuub 8w+KnOeNLvwPF+c5Aqqb0qouewO7OCqHGoyqLBUqq9ji8L96i19xhaK6CZf9gutvGL573wB8qWfW S3EisntGuqT57zrxZp7rFImfKprI84l73xMcnfZOU3w+PfTy0bxTAVgv6rgnX5Q6aDpr1+8/0f4z z6j/APRPmF1Fl5+HnliFDptbFyQze8V1UDDII6bupHukH29JyMP+w7QurPCTXgN5iRqDZsg5I8Z5 fZ5BsrxQzrixDndVuI4iZqRtwx+/5c9Khf8AYG8SxNB2+U49iT7DYo/p/Rj7Zyx//wBHeRl3XO+H 3mdVaqGDLg3up13aglqa9Nu3U+3oRNX+OHkRHeRtcsdhjqHeqB2r2M9Xk4m+RJoiTTlkmTaQWa+w f4fFVBk+brH9PqJxNyl/n0/TkJRcxxD8RclT21W703AodRprpr/rBH+g/wBtfW/hp8SMb4lcbk8j jcfn8ecXJND15VfbcWBVcrp/UKyMdNRo69ddwXRQqJ6TARARARARARARARARM8+TGk3XMaFW5LPj VdC1W7YcUyxhIXKDlrLX4dLVdQq9CezbuAhLPTJGYUiGc8dwk3JKM+KKpl4ZThfiLfhcLGzcp0y9 5orx7rSEYKx7VbOFDMrgaldCdp6fJPOvif5Nzfi3j+Nk+OnEXlszmeNwUfJqsuprGfnUYjWtVVfj PYa1tLqgvr3MACwEiymk7PkNspUXujnMrpRdHtcTQYXRczq9kzdzUr1YTGb1SFtlEtd/048lC26U Lxg2k2MzxVtIKt0lWXUlzOEO4YfHchRa/Fi6vKprLmuxlsDov0irrXXoUHvFWTQqCQ2o0NS3k3m3 h3L4OJ543F53A8nl14lWZhUXYTY+Vd0oqyMXIy84vXkP90l9WTuS1q1ejZYbK55K+SeLQl97m0nc jt7MlOw1UerlrVuc1CIt1iQQcV+nT+jNYBfPK9cZ1F0j1nEvpRvIuurpcTRN1RPhoycNyVmL61K9 aNhYe8gcov0nWst3GRdDq6qVGh1PQy/y/ib4Rg+QfxnKzSvKDIroY9nIbHryLgDTj25i1HDpyLQy 9vHtvS59yhUJZdazjfISdcu9jQnJOj1RvQPL/N/HusvZCAtkmnOwVzhcGkORjtGHlXDgt4sEpqL1 hHSHpQiWSvWirtDqKTg6kx+JqVcc1LbYbePsvYBlG1kN41Go+goqDMvViNwU6kAZjF+ImfZbzSZ1 uBiV8f5lhcRQz1ZDi2rJr4p9jCuwn1Vr51lVNui49bdprqyiWM1xm3bKC5pI6+e3Ikz2JkpOEkZw 0VOlctp2Ity9CfwJq+aL5ZTz6d0bmjCMSszO1nnSpppnMcnqrxxeec0ceK/82yhguq9VKbw27Xbt 2e9u10A6kzZnz3xIeMW+ZHMUeO02vU9vbt1W2vIOI9XZ2d7ujJBoFQrNjWaKqkka+RK+SeLQl97m 0ncjt7MlOw1UerlrVuc1CIt1iQQcV+nT+jNYBfPK9cZ1F0j1nEvpRvIuurpcTRN1RPhuxOG5KzF9 aletGwsPeQOUX6TrWW7jIuh1dVKjQ6noZDy/ib4Rg+QfxnKzSvKDIroY9nIbHryLgDTj25i1HDpy LQy9vHtvS59yhUJZdeM3kjjzm72LLom58kNFq8hKQ9ir7CrXKbNVHsZVIu4KyNtViYNZnA1nsPNt epSjtw2jXbtTrJByd6U6BX6NyK4yZ1lemG4BViyDcCxTRNW1ZtQdVALAe8Rt6zifiX4ZbzmT4riZ vc8jxbHruqSjJt9OyUJkF8g11FaqO3bXtvsdKbLD2K7TeDWIHD+X+OsYTN4+03n+yaBdMYz3Yyxe YZXsFl7YKhdmTwqF8rNOhata7hH0xzJQzw/SPeKOohv7PJAyRlUzqSrPH+Ray56KtmJXk2U62W1L tdCPcZyyoXAI9nRzrs10OlBh/GPwyjB4zH5XP9T5DncJicjswcDkbu9j5KtplUY9dF+QmMz12HSz V8dNnqShZS1lTXkRjcHS6boC11Ql61ox2yWfGp8PYr5O3ld01VfEa0+oUiIsFvsrxBmgoq4RZsVl WpEj9WKnwhvhDr4jkbcmzEFZW6nXubyqKmh01d3Koo16AlgD0011mmzfiP4Xg8JheQvnLdxnJFRi enruyrcospbbj4+NXdkXMFBZ1rqZqwrFwu06etl22ZftCNmXzK1oWglMm29atPEY6ZjVYOwOISLs HYV+hNR0aunKM46YQ46Q4XqjJ17jVfiblFZFPrzuNzuNKDNrKGxdy9Qdy7iuo0J6Eg6H5Row1Ugm Z4r5x4t5smU/i+WuUuFeKb9EsQ1XGpLu04sRCHVLF7i6a1vuqsC2o6LQbe9eSuha55FVHN7dhtSr mK3Cm1GEjbvkN+uU3ZXFixvPdMcuJW2QO60lhDomlLko0J1GDdGQbpFU6Vc/xKa1OLw2Jx+HkZle VZdk1u7FLUQLtusr6K1Dk9E16uNSdOk8+r574m+ReYeScP41mcDh8bweZjY9SZXH5eTbcbuOxM1j ZkVcpjJWN+SaxtxbCiKGIsOoPtUjy3pL3Pq/YNHRdVm9SVv1DPV89pMLcdYsExacaus1Q74/oldo dYmrnbag0k4b5nj9CM6k1bO0CuepqG9PevJ4DJXLerDIfFWuuzuOyVKFuQOgdnYIjkHTaW1JB26i TuC+L/B3+O4/I+SB8XnrczOxDiY1WRn3WX8dk2YuU+LTi0W5ORjq9e/upSVRLKxaVY6S1YLfcfs3 68/gLzFyvNVlrNX6KZohJHJL2GmRknL2quOT9YlJA2KBYQrwzlhI9aPEztFk+p+4kchYFvFchT3u 7Uy9hVZ9dOiuQFb2+8rEjRl1HUHXQiazA+IPhvKfp36fn1W/q119OLtDnuXYyPZfS3u/dXVJVYXq u7dgNbrt3KwHGfyCx0td0y1lvDBeCx23vqDpDtoxmXq1bucb2II6rh49pGryUpKcWnmaSabJFz7y 6/Ek/Wpwxeff0nke9TR2iLcisPWCQNyHXRtSdAPdJ6kaAanpOJ+Inhg43lOWGdW2Bw2Y2JmMq2Ma clO3upKKhd31trUCpX3MwVdWBA6Er5J4tCX3ubSdyO3syU7DVR6uWtW5zUIi3WJBBxX6dP6M1gF8 8r1xnUXSPWcS+lG8i66ulxNE3VE+G5pw3JWYvrUr1o2Fh7yByi/Sday3cZF0OrqpUaHU9DI+X8Tf CMHyD+M5WaV5QZFdDHs5DY9eRcAace3MWo4dORaGXt49t6XPuUKhLLr11/JvFk7nZ87Z2eWn73TH kuxtVRp9C0O7WGBWhKlGXZ2rIxdQqk27QYOoOYbFYuul+WkpFbkc0UXkPi159HC8kcZMxkVMWwAq 7vWitucoNC7KNQwO4e1VG9gE96ddnxQ8JXmsrxynKuyOewnsS/Hx8TMybqjVjpksXrx6LWCNVYgq fTZdcwxqWsyNapSni15b82mu0yy3pyhWXO1SiaWRU8uQbHTF0PjU7dfnNcVvl1aqVHVV4+k1NV6r YIIrCGMqU7YhDHM2O5suc4D9NuspxQXGMv3r92p/95EDbEO+rV2Cit9z6aN7N23D/Cr4vjzbjcLk +eZcWznLQOPx/wBP5HGI/wAvkZbUnKyVOPnlMbHa1svFFWMWDVqCxqNtjb95DtM1gLW0pTqFltFp No8Z2llr03GzazGLqvkDucLlrKTO4arRLZy9dxSM2ozKg7U62dM0zuUupGImtD4riGzba2yQy4dq ZJVlK6lqKTaR116A7NdR1BIU69RpfiD8RqfGePy6eDei7yTByuFW6m1LSqUctylWCrkqawzNWMk1 hbCUsrVrUKEK8y8iNYe4rlcte46LjpN8lN02tNVp9+rEVKvrXa4QdOJcLzMoIOVoij0/s5/JSzgp PiRk2U56k/j7hI/EYC8lnLiuxVdrsdo1dtiM+xB8rvt2qP8AiI9vsl18RvLb/CPFLufxqq7bxfjU qbXNePScnIqxxkZVgDGvFx+73shwOlSN1XXcPzmb3bkFX7vVbDh9tpTiAaTkDeszYW6kKlVVTScL tH9Vsdi0eJfQHWKhl0ZlGyJdOUhedYcKp7ibNXjCAuAmTXkhyrJYUf8A0hlWshtehQ1n/vdND88Y v85R7LvLMngszhGx1tqysJMjGOpAJVqLrsytqthLLkrmLqANccBty9am+TeI3+xxFWrFyWcyVmTk VaY7k6pdK5WtDSiW53kopmNzstdiKhpyceySOut2vvpLibdM6pvgmQxufcjheSxKWvvr0RNN4DIz V69B3EVi9ep6DuKvXQe0zq4X4oeDeQ8lTxXFZrNk5Qc4zPRk005grBZzg5N1NePnBFBdvSW3bUDM fdUkRjAfIdppUBVGl1dQsTot2tHkw0rVehI2bRYylV8ftzmsteyZHDpaWbNnrSKWhFHhV3afXLp4 odslxIp00e/leIbCtsbGDNh1JjFmYrqGvpFoHTToTv00HQABjr1NX8PviNT5Nx+JTzj0U+SZ2VzS 001JaFejieUtwWcFjYFZazjGwNYC9ljNUgQFU8jePLSn5CwU/iGsvcJyB3nFsVvUNCUvQrG6rC+q PaTLuF+N6nVplxJySWfXJF9Got+KlkZFVCOR6o+VK1N2cXwORyDfeFa6nxbrkLPWobtBx7WYADuI VYnTaoLnRRukPz74vcN4djn0aXZudj+QcbxuVXVjZdzUHObGsJ0x6LC7jEyVtpVNwuuavGQtewqM +idwqz62Wcj+2RFdqdfxmo6y+irrT71nFyq0HMWHSIyRt12X0BnXmMHWVm1K6miycs2skwVYul3f wQXa/CJZxl60JsrZ73yXqBR0sRmC1kInbLFm9/qQSrBlC9Q00GJ51xV/L5QyMunG4jH4XHz2rycf Kw8miqy7MR8jJOWtK1UFcbRanrruqaq17vu7Kp3qD5D5Jpk8Sr1WxyRbE5iFrFEQ1qpd6z+Qs9ab qtkXFopje/1qsnu9XQWeolPJRHzrEvVk+dV56y/HjlcRn4VXfvQdkNtJV0sCt191zWzbG6H3X0bo ekkeP/EbxDyfkBxXFZNo5JqTdXXfjZWI99IKg34wy6aDk0AsoN2P3ahuXV+o1hlC8yfHDTXVTQpe guZRnenKEdT7G6o2iQVIsU45acepVWLv9hqUVSHF19nvwNCckP5civOpGbcV50gk5XjvMYS2HJqC tUNXUPWzquum41q5fZ/29uzTrrpKTx/40/DXyi3Er4TkWtpz2CY9zYuZVi3Wsu4UJl3Y9eMcnToc bu+oDe4aw3STaD8hMkst+XzWEsr59aEJeerqapajdEalI2Sqt3Dq0VmD0NzXUc+sFmrKLNf+RjmM o4esTIKlXSIZJThY1vE59OIM21AKCqt9NN4Vuis1e7uKrajazKA2o0J1EvMD4i+Icn5A3jODlPZy q3W0g+nyRjvdQC19FWYaRiXX0BW71NV721FHDqpVgLoFbNtARARARARARARARARARARARARARARK Q1Z3osK/iZ2EmpJtnqSHGdza1GsxU1fIMx3fDkt8UWabzLSYhmTbvofsUY5Z+kiXq7cjk/PYFlgr iWI1Vqg5euqF2IRun0DpoQSfosWCk9CV9swvlt3kmDkU5+DfavjgXbkrj0V25VWrf+or7gtWytR0 tqWlrVUGytbW+7kHT8b6rZJ19pbTWtGlZi4NYR0raYt9mZ20owinlSm6w5jjxueFiit457To9y3W bEL1b0m906pD+kSTzF9NQw2opWussNpFmoJDq2utmvUOwIPs+TQyhX4Z8RyefZ5RTzHJW5ualTG+ t8IrYlbY9tBTZh9vRGxqnRkA3aHeXVtJMF/HiiuexzBZ5ZuU+DfzUtWqAzlGsTWKpLTiD1JeUqzq Gjo+3QD+LUknKsUo2lE+wpljFj/lkuETL0DlsoasAnqGADORqzBdOjAkowOgDar7+nv7j1l1Z8OO BtNeO75X6NRZZZTiK610Y9loYGyhq0TIqdC7tQUvHpixGP2lAUduZwqu2gkU1t1r0ezRdfmWk9X4 x5c30R2GkY3jzkO4QnKqnXrY/dxBnnToO3sg6fdUIQyi6hi/HvGvk7qNzUV0o7qVYhAdQdNfdbco B06hVC/0Anbm+A8byq01cxl8nlYuPettKNktX23Td2yLaBTkO1e7VbLbrLdQpaxiNZOKjndIop5N xVa3HRUjOnbrWKe9Cj6z2hy0Kom2eWu1SKjyx2l+3TVMUjiQdOVil76eG+H+gjZGXk5W0XuWRfor 7FXX2hVGiqP7KAJfcP43wXAG2zicaqrJyCDdboWvvZdQrX3uWuvcAkB7rHYDprpJoI0u4CICICIC ICICICICICICICICICICJXOn1eWtteaRsS2i5A6E/DyT+ImpqUr8XNxbJcx3sW9k4iKm3aaLghuG 9HWqqavScIfnC973kvCvrotL2FhqhAIAYqT7CASB/wBIma8q4rM5jjkxcNarGXIrdq7bXprtRTq1 bPXXawBHXTtsG00I0Os5spXrz7N6TM1WsxFOg7NWom2tK3BNWLSNiz2xmlY3aSJI5nHs1VTvJNQ6 qxUU+rqmMp3nDG6GcLVzLK73ay1HKlm1JO07R7ST7B0GvQdJz8Ss46/xnBzeJxacLAysWvIWmpVV EOQouYAIqKSWclmCjexLEakywBEmigIgIgIgIgIgIgIgImTPMWIt8xntE/pNGs+hS1Y3zAdHd1yp lhOS7uv5lrlPvFkTZrWOar0ISQPCQq/ypHLxuksvzhOqE+PxF949Zj15dvqbUprfEvrDNu0DWVOi 67Qzaaka6AkD5J5F8ZsPmM3x3A/Q8DK5HMxfIOJzGpx+13GpwuQx8q4Kbraag5qrbYHsRWbRSw11 la2uzadvVnzyNsWAallWVUG/VPUps1waVOav9ztFAlG9jo8DCQ9Bt1vrUFXmVrZtpB/IOpVR2qVm Rsiz5xdRwjNopwuKouenLovzramqGwsK0WwbXZjYiMzFSVVQug1LFugBzHL8p5T5/wApx2NyPj3K 8T4nx/IUZ1vqFoty8m/EcXYtVVeJkZFNVK3qlttr3mxhWtSUgWNYmU5PGN1NF6tjr9t5XzedaXrN 5uTV3Te+MdVpHK3o+gPb6/WtkxccYuW8Q11rLmT6RNaNVmuOVWaKjZzHJ9Tbtb1OS4vfRyKHAXMp x0Qh/Us+6usINgS5KCjAexgmgJBDnUt5NleFeenE5bw3IXy6/wAc5Tl8rJVsb9EoxuzmZbZTHIsy ONyeVryaWfRWpbJDtWjVW4y7a6rXnqJpz79rfLZbej/2n/sI8fvIOF+MdHk+YzGg/wD5n/s0qp6p Tny79n+tpf0Mz+lwt7BfQU3up+qBVlYS9jW+r3OJvoPU/wDMf1G0ez2HuJ19g16+wzW8hwPlGR+r dris8+r+IvE8vX7idcLE/Re/Yff6OvosjSs6O20bQdy6++7z28q+QyhE6FdzeM8poUX5MSsYpDom ki7pCRHYAlWShVurLcqklZW0bd+rprkKWwMVfiXnFzcU6ly8UcRqba/1paTjA69Oyx3btf8AiC7q dNP+Ww/p0sLvHedf4jELx+d/9YW8inNWIaxv/VKq+0KBUdT6d7lp5PcHAGXU/QbzuomTxjdTRerY 6/beV83nWl6zebk1d03vjHVaRyt6PoD2+v1rZMXHGLlvENday5k+kTWjVZrjlVmio2cxyfU27W0T kuL30cihwFzKcdEIf1LPurrCDYEuSgowHsYJoCQQ51LYLK8K89OJy3huQvl1/jnKcvlZKtjfolGN 2czLbKY5FmRxuTyteTSz6K1LZIdq0aq3GXbXVrXL4ixVaV8wZWTy29Nn20a9I2WpP0YFqsvLVTmI 5hT4jkgojImUY8a2mGlyFRW4U5OG6pznpUL3tBm2U3px6JfUVxscK43Ho3esc6dOuqlOo/w+Seve LYnJcTl+ZZeVxOeuRzfMPdjuKlJsx/0zBx695D6rtvryAFbQjUtpowJrnxZpug5faM+mbnlV5ZJ1 zwP8VsFfrt4mOkHDfQcxkdJdXmIQ60k1VDxzQs8wMVyXvy7o3eegxup99MznMnEzqLq8a+ol+Uyb x1I+7sFYQ9R7Ttbp7R8vtma+FPC+ReK8rx2bzXE56LjeA8FxTkVo5GXgvmtlVja5JRe7UQ49xz9E nadKTZWSb8T+4dquj0iwtmtTn/MqmzldsTmu1uNY1Tc9/da3T7VTblaZyGzaHtqFerjBsowmJiJ+ aj3qyCSpVmpkT2TU1c96rBw7ULWJiOrKGYlqaBU6uiqbCm5mO5EbRlBI0bUYejk874SfoXlvkuBk rTh5HkeNbTcaaUWjlOWbkMe/Gyb7a8KvIFNNSNVkZGP3KbXRHD1FGvfwu0ax2GV8l9gl8mskHBbR s0XbKV/UYzs5Aylfhspz6mnl2kyUsc0mnDyTry/zr1qVRi6keODt1l0fQqer8kw6aa8Ljq8hGtxs Yq+87WDG2x9COpGgYaA6MF01APSb74J+SclyOX5P5lmcRlUYHN82mRjenTu1PVXgYmMbFs9xbCz0 t3LKwanu7jVu6aOexD44zse1+Tt+0Kl+SkbEaRfqJJUxOhavs2bRstXYPC8wpUqvOVjLNUqcQ4ef 2avP2/rkGyjpVumT/LqHU+D5ZyLU8bhYmJZhtZTU4ffVTYQzXWONrW1OdNrKfdOgP99Z2YfhlPJe b+U+QeRYPk9WHyfIYr4wxM/ksJLKauLwcaw20YOdj1lu/TamtqF2QDrs2ie6tHTeNa/W9DyTx9v9 ozhHF2GMP84rMTXa7YqWlW7bN3CEsVIRtc3B1qaRszmyOG8+m7mI9yc7Jk5KZ0oVQnOoPVyPHviZ +XUmYck3CxizK+5AjK+1WYbdoKaIw6sPdGknvj53hfmON5H4h47yOV40OETjXw6K6absYU5FuRVd ii+2qmwXm50yxZkUuTXRaDawYSs5qiaw8zzW7/G5RY4vcp7yFrXk/meetW3FavXbDQq/R6e2qL+x qGas+zWo0upSDWffJJlapurCvxNQxESuVJteVgLl4+I96NxiYjY1lhPvMrs77wvU7andSinqRWNR 12jMZvA+XXeOcv5DjcRk1ed5HkdPN4WIq60U3YlWLjrjtcdq9zOxse1Mu1VCCzLs2sQgtaEVLx2v 1RvWJxpKjdpTM5JDP9L8nHSkGmi6s+9Y8/t1zhp5hFcdEM7JomoXtpJvVe9UIk3qLdJTnrWKc8nI 5fEyMXJfuVLmqbK8Ya9FotCIVJ06dupCoHTU2kj2aSi4j4ceQcPz3B4y4edb4xauJm82xqAa/leO fIya7Ur3e8MzOyq77W6hU4+tW95wx8OTxjdTRerY6/beV83nWl6zebk1d03vjHVaRyt6PoD2+v1r ZMXHGLlvENday5k+kTWjVZrjlVmio2cxyfU27XtTkuL30cihwFzKcdEIf1LPurrCDYEuSgowHsYJ oCQQ51LQcrwrz04nLeG5C+XX+Ocpy+Vkq2N+iUY3ZzMtspjkWZHG5PK15NLPorUtkh2rRqrcZdtd W0MKbzVCvHlDZ57Lr40U1feU7nXJBtX2zl1KUxnkGU1aO696hImVaGZ2WCmCkbq+k5eG6pznpVL3 ub5RqsrGwaKr6iKMXYw3EAObbWOnTrqrJ1H+HyT2zwKvO8f53yrleQ4rkEbl+fGTS60qzPjLx+BQ m7R9V23VZACNoRqW00YE54plL1yg4V4IdbY9eJ3RfFk0S5uVAIlGMDTjKWxm85PaGNcsS8gaAWno U9zI9alcOG7J3xqdIzhL1lU5cZOTx+XynKa5FSYedrss6naRclqll03bW2bToCRqDodNJ5xwvCeY eP8AgXgPa4bPyPJPFDW2TiAIvdWzjcrAvWm4v2jbWckWVh3SqzYVNibg0jl4rm96paPIi3PvH290 9rokt4FkoUK/dVyTnTwHj95FTug6A6tvYiZdwcPNtK9I9dJtWbyQbrJnSSRcrL+6mTuxbuKwaMPH XLqsalc7eQGC7r8da6wuoDEFhpqQpHUkAaE1vO8b8QPLOV8j5i/x3Pw6uSu8V9JU7Uvb2uJ5i3Ly 2yO3Y1Vdq0vvCV2XIwKolr2blXfGoaXeC09/ygYvNXmZUVbIvqjdmDmvw1hrqynt2GKTlEGVhQRl HsWZRNqV01OyOqbhVzET6Y3MphYeN6geryVqr+R0O4q3+6dNV6A+3Q66ezrP0D5T5Pzo4az+PcJf n5pKhsfJRqq7qSdLqw4W4B2TUILENZYgWELqZhF1jcvc3Wrwuc+POieIGbaV4/6bmFurcJEVFNC7 6LeEa+zq13Jn2Z2WaocO1okcwlEVpIrhtMzCMuZBVJMhOKF1C8jXjDHszMunkMynLrsRiX9ytNxZ O5YquS5KkLoUQpqCddJ4Jb4Zmc3by+F4345yXhvjPJ+PZuDkU1V44GTmZQpWjK9JhXWYta4qJerX B0ychcgoyKF3C2pqd2XXGmG061eNtxy5LM9GzvR7vbo5aAla1xzmDlOVYwOMR0LIGsj2Ntj5uWOd Hm2Fd+Sg3LpMqblTpCGgV1cdx7ZWRRmV3m6mytEO4NpYNC1xYbQUHvDY1m5wp1A6zX5uf5p5hTwP Dct4zm8UnF8lh5mVkIarKd2CwsWrjUrfvMmQwFLnJqw+1ivaoW1iFNZ0Cp7FlbnCb6liGhW6SzW4 ebEba6REFrzGec0zyQ3pxpVXs9YfTc5G1aRko6Nrcb1ePeSMf325BTnu8VQ6mabl5HHZwysU5NVa XV4ZVzuKh8ejtsrBVLAEs2jBW+iOmh1mX8e4nzPxOzgPIE4LkczK4zM8lS/GrFK2tjczypzaL6Gt tSh3RKad1Vl1PS1vfD17T48jV/IG0tN2vUvglsh7VaPLbxk8gqfQmz6KfKzVMw1jhXHsA4sXzScR FXCQj85dNzFW7yPLJ86Ujg7Tqbs/Yl/E0Ni4teVW1CYGTQ7kEaPcbtG26alAbAenvbfaA2qyFk8X 8Q+Vp5/nszx/Mp5bL8v4TlsfFDVsbMbi14vdUbtwrryHTDdCG+6F2oWxqStxl+tUnU9quGt2trld /onLv49+PcDWWFrjoh8n2/4/vena24pN2JWJaxop1qdYSEWwlnDP59H5SRcEQ65Ml3ho+BlYPG4+ PQ19Vvay72YqSPu7aK6t6bgvvKQzKDtOqjXbrLny/g/K/N+Z5jl6uJ5HA9d47xFVCXpWw9Xx3K5v IHGyRRZcBTaj0VZD191e3dYK+6UINpSFk1HYNQwe02bx50rJobFLDYL1MSUmet2Gx2SyTGd2vPi0 2n9qU1KJHz5yW4KPn76UNGPXR41qlyOJ6jKpQUpwePwcqinLpyLMlFQAblVVFi2b33qPvPc2qq7g NzHefYdXkcn5V5l5VwHK8n45yfEYXB5N2VY79m6666zDvxPTY/p7HBxG9QbbbbzRbYaak9MupZas ic/0mP8AA7CcJJlN3b6XQnPi28m49CLjiM453mm0Zldru6azCcnxks5ZwtfkFinRP0zlTnpT6Y6n PVPsy8J/KcrlDfUcK0ZIB1OpFlNiICNNepZR19g9vsmTxPHvJsf4BcD4EOIzl8n49uDa1BWgVGwu SwsnKZbA+0la6bWBU6u3RdSw1klIJrVT2qNkaLim0ZVnz+96HNazX31qjtBxi2wVg5ZpVlYcxqLx +S4UPSbPeX7KUkUWzOCg2xFH3TlkXK/uG6Mk4F/GsmVk41+WKqxUwU13Iy7QVscDY9aoCqkl3Pu6 bANJZ8GPLuI83qyeB4Pm+J8dsz8u3Pqa9Mvjciq3v2Ldg47OMjFzb8p677lSvFxUByNwybLNx3P+ yPsLSvxb64Zj0f4tP1vmnvH8m/b+T/I+3H7I+wtK/Fvrg9H+LT9b5o/k37fyf5H24/ZH2FpX4t9c Ho/xafrfNH8m/b+T/I+3H7I+wtK/Fvrg9H+LT9b5o/k37fyf5H24/ZH2FpX4t9cHo/xafrfNH8m/ b+T/ACPtx+yPsLSvxb64PR/i0/W+aP5N+38n+R9uP2R9haV+LfXB6P8AFp+t80fyb9v5P8j7cfsj 7C0r8W+uD0f4tP1vmj+Tft/J/kfbj9kfYWlfi31wej/Fp+t80fyb9v5P8j7cfsj7C0r8W+uD0f4t P1vmj+Tft/J/kfbj9kfYWlfi31wej/Fp+t80fyb9v5P8j7cfsj7C0r8W+uD0f4tP1vmj+Tft/J/k fbj9kfYWlfi31wej/Fp+t80fyb9v5P8AI+3H7I+wtK/Fvrg9H+LT9b5o/k37fyf5H24/ZH2FpX4t 9cHo/wAWn63zR/Jv2/k/yPtx+yPsLSvxb64PR/i0/W+aP5N+38n+R9uVtW3MlWb5FN6RR7rG0K4y s0pca9JQCUdDVSwSKT2b/vtfcqSPf41tNy6Rm8xFopnQdO36cijxuuSQ7ITLlS7FY5NtbZVajYwb Uso0GxunUgdUY9QFKHUbNmY4y3J4vyCqvgsDOq8fzbbTk0vUErx7nDW+rpYv7i22ApkUKpWyy1cl BXYuQcjSYp56dARARARARARARARARARARARARARARARARARKr2kss5zmdiYaFmp5xYjxlaetK+qZ CSQgJ+UZxlnkknCTlo7R5G1xw6W+LdQjoxicKiYihinLO47tjMWyxlUJqwLezcoJUfKOraDr0/r0 mT83GZb41kYeFRfkWZJSllpOjiq2xUvcEMrDZSbG9whyQAhDEMORrf2zJs3ZMs60NozaIJNWjRrU U27Zq2bplSQbt0EnhEkUEUicKQhecKUvOc5znOD42KzMWa6ksTqSX6k/6pyq8hqoqWijjeSSlFCq q44CqoGgAAbQADoAOgHQTm/ZH2FpX4t9cPno/wAWn63zTn/Jv2/k/wAj7cfsj7C0r8W+uD0f4tP1 vmj+Tft/J/kfbj9kfYWlfi31wej/ABafrfNH8m/b+T/I+3PVhrp/MyCTD+p3WK90qpvnZmC+Rj0/ aTMp6VXPzSvoMp6fSXnp78Td5wcLMbtpv7lbf2Dan/ZJeFzfrcgY/o86nUH3rKtqDQa9W3H2+wdP bJqI0vICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICIC ICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICIC ICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICIC ICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICIC ICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICIC ICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICICJ//Z --=_related 0030F437C1257F33_=-- From jack@suse.cz Thu Jan 7 03:34:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 52FAB7F37 for ; Thu, 7 Jan 2016 03:34:00 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 3E46B8F8035 for ; Thu, 7 Jan 2016 01:33:59 -0800 (PST) X-ASG-Debug-ID: 1452159237-04cb6c42e82fae0001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id F3BGAKypCdOmODkb (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 07 Jan 2016 01:33:58 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 058B5AAB6; Thu, 7 Jan 2016 09:33:53 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id D2DFE823D8; Thu, 7 Jan 2016 10:34:02 +0100 (CET) Date: Thu, 7 Jan 2016 10:34:02 +0100 From: Jan Kara To: Dan Williams Cc: Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Subject: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() Message-ID: <20160107093402.GA8380@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-2-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452159237 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25912 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed 06-01-16 11:14:09, Dan Williams wrote: > On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler > wrote: > > __dax_dbg() currently assumes that bh->b_bdev is non-NULL, passing it into > > bdevname() where is is dereferenced. This assumption isn't always true - > > when called for reads of holes, ext4_dax_mmap_get_block() returns a buffer > > head where bh->b_bdev is never set. I hit this BUG while testing the DAX > > PMD fault path. > > > > Instead, verify that we have a valid bh->b_bdev, else just say "unknown" > > for the block device. > > > > Signed-off-by: Ross Zwisler > > Cc: Dan Williams > > --- > > fs/dax.c | 7 ++++++- > > 1 file changed, 6 insertions(+), 1 deletion(-) > > > > diff --git a/fs/dax.c b/fs/dax.c > > index 7af8797..03cc4a3 100644 > > --- a/fs/dax.c > > +++ b/fs/dax.c > > @@ -563,7 +563,12 @@ static void __dax_dbg(struct buffer_head *bh, unsigned long address, > > { > > if (bh) { > > char bname[BDEVNAME_SIZE]; > > - bdevname(bh->b_bdev, bname); > > + > > + if (bh->b_bdev) > > + bdevname(bh->b_bdev, bname); > > + else > > + snprintf(bname, BDEVNAME_SIZE, "unknown"); > > + > > pr_debug("%s: %s addr: %lx dev %s state %lx start %lld " > > "length %zd fallback: %s\n", fn, current->comm, > > address, bname, bh->b_state, (u64)bh->b_blocknr, > > I'm assuming there's no danger of a such a buffer_head ever being used > for the bdev parameter to dax_map_atomic()? Shouldn't we also/instead > go fix ext4 to not send partially filled buffer_heads? No. The real problem is a long-standing abuse of struct buffer_head to be used for passing block mapping information (it's on my todo list to remove that at least from DAX code and use cleaner block mapping interface but first I want basic DAX functionality to settle down to avoid unnecessary conflicts). Filesystem is not supposed to touch bh->b_bdev. If you need that filled in, set it yourself in before passing bh to the block mapping function. Honza -- Jan Kara SUSE Labs, CR From domains@ivss.gob.ve Thu Jan 7 04:39:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 7A16F7F37 for ; Thu, 7 Jan 2016 04:39:16 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1F7348F804B for ; Thu, 7 Jan 2016 02:39:15 -0800 (PST) X-ASG-Debug-ID: 1452163149-04bdf03afc364e0001-NocioJ Received: from correo.ivss.gob.ve (correo.ivss.gob.ve [200.109.237.55]) by cuda.sgi.com with SMTP id FJTDnNxGIC7mGLue for ; Thu, 07 Jan 2016 02:39:09 -0800 (PST) X-Barracuda-Envelope-From: domains@ivss.gob.ve X-Barracuda-Apparent-Source-IP: 200.109.237.55 Received: from correo.ivss.gob.ve (localhost [127.0.0.1]) by correo.ivss.gob.ve (Postfix) with ESMTP id 4ED874007A0C; Thu, 7 Jan 2016 06:09:00 -0430 (VET) MIME-Version: 1.0 Date: Thu, 07 Jan 2016 06:09:00 -0430 From: Web-Mail Admin To: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: =?UTF-8?Q?=E5=B0=8A=E6=95=AC=E7=9A=84=E7=94=A8=E6=88=B7?= Organization: WEBMAIL X-ASG-Orig-Subj: =?UTF-8?Q?=E5=B0=8A=E6=95=AC=E7=9A=84=E7=94=A8=E6=88=B7?= Reply-To: webdomains@qq.com Message-ID: <71f8342f145210bf3acdaf39294eafd9@ivss.gob.ve> X-Sender: domains@ivss.gob.ve User-Agent: RoundCube Webmail/0.3.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Barracuda-Connect: correo.ivss.gob.ve[200.109.237.55] X-Barracuda-Start-Time: 1452163149 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.70 X-Barracuda-Spam-Status: No, SCORE=0.70 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0113c, BSF_SC7_SA298e, HEAD_LONG X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25913 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MV0113c BSF_SC0_MV0113c 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.50 HEAD_LONG Message headers are very long 0.20 BSF_SC7_SA298e Custom Rule SA298e --=20 =E5=B0=8A=E6=95=AC=E7=9A=84=E7=94=A8=E6=88=B7 =E6=82=A8=E7=9A=84=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=E5=9C=B0=E5=9D=80=E5= =B7=B2=E7=BB=8F=E7=AA=81=E7=A0=B4=EF=BC=8C=E6=88=91=E4=BB=AC=E7=9B=AE=E5=89= =8D=E6=AD=A3=E5=9C=A8=E8=BF=90=E8=A1=8C=E7=9A=84=E7=BD=91=E7=AB=99=E7=AE=A1= =E7=90=86=E5=91=98=E5=88=9B=E5=BB=BA15 GB=E7=9A=8415.30 GB=EF=BC=8C=E8=80=8C=E4=B8=8D=E8=83=BD=E5=8F=91=E9=80=81=E6=88=96=E6=8E=A5= =E6=94=B6=E6=96=B0=E9=82=AE=E4=BB=B6=EF=BC=8C=E5=9C=A8=E6=9C=AA=E6=9D=A52= 4=E5=B0=8F=E6=97=B6=E5=86=85=EF=BC=8C=E8=AF=B7=E5=9C=A8=E4=B8=8B=E9=9D=A2= =E8=BE=93=E5=85=A5=E6=82=A8=E7=9A=84=E8=AF=A6=E7=BB=86=E4=BF=A1=E6=81=AF=EF= =BC=8C=E4=BB=A5=E9=AA=8C=E8=AF=81=E5=92=8C=E5=8D=87=E7=BA=A7=E6=82=A8=E7=9A= =84=E5=B8=90=E6=88=B7=EF=BC=9A =EF=BC=881=EF=BC=89=E7=94=B5=E5=AD=90=E4=BF=A1=E7=AE=B1=EF=BC=9A =EF=BC=882=EF=BC=89=E5=90=8D=E7=A7=B0=EF=BC=9A =EF=BC=883=EF=BC=89=E5=AF=86=E7=A0=81=EF=BC=9A =EF=BC=884=EF=BC=89=E7=A1=AE=E8=AE=A4=E5=AF=86=E7=A0=81=EF=BC=9A =E8=B0=A2=E8=B0=A2 =E7=B3=BB=E7=BB=9F=E7=AE=A1=E7=90=86=E5=91=98 From bfoster@redhat.com Thu Jan 7 06:44:40 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id B0E1C7F37 for ; Thu, 7 Jan 2016 06:44:40 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id A063A304039 for ; Thu, 7 Jan 2016 04:44:37 -0800 (PST) X-ASG-Debug-ID: 1452170675-04cb6c42e733430001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id tQvh2oy1l4iber2p (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 07 Jan 2016 04:44:36 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id 489868F289; Thu, 7 Jan 2016 12:44:35 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-81.bos.redhat.com [10.18.41.81]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u07CiYht002433; Thu, 7 Jan 2016 07:44:35 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 0758C125794; Thu, 7 Jan 2016 07:44:33 -0500 (EST) Date: Thu, 7 Jan 2016 07:44:33 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH v2] xfs: handle dquot buffer readahead in log recovery correctly Message-ID: <20160107124433.GA33327@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH v2] xfs: handle dquot buffer readahead in log recovery correctly References: <1452052834-20605-1-git-send-email-david@fromorbit.com> <20160107030830.GA32200@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107030830.GA32200@dastard> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452170676 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Thu, Jan 07, 2016 at 02:08:30PM +1100, Dave Chinner wrote: > > From: Dave Chinner > > When we do dquot readahead in log recovery, we do not use a verifier > as the underlying buffer may not have dquots in it. e.g. the > allocation operation hasn't yet been replayed. Hence we do not want > to fail recovery because we detect an operation to be replayed has > not been run yet. This problem was addressed for inodes in commit > d891400 ("xfs: inode buffers may not be valid during recovery > readahead") but the problem was not recognised to exist for dquots > and their buffers as the dquot readahead did not have a verifier. > > The result of not using a verifier is that when the buffer is then > next read to replay a dquot modification, the dquot buffer verifier > will only be attached to the buffer if *readahead is not complete*. > Hence we can read the buffer, replay the dquot changes and then add > it to the delwri submission list without it having a verifier > attached to it. This then generates warnings in xfs_buf_ioapply(), > which catches and warns about this case. > > Fix this and make it handle the same readahead verifier error cases > as for inode buffers by adding a new readahead verifier that has a > write operation as well as a read operation that marks the buffer as > not done if any corruption is detected. Also make sure we don't run > readahead if the dquot buffer has been marked as cancelled by > recovery. > > This will result in readahead either succeeding and the buffer > having a valid write verifier, or readahead failing and the buffer > state requiring the subsequent read to resubmit the IO with the new > verifier. In either case, this will result in the buffer always > ending up with a valid write verifier on it. > > Note: we also need to fix the inode buffer readahead error handling > to mark the buffer with EIO. Brian noticed the code I copied from > there wrong during review, so fix it at the same time. Add comments > linking the two functions that handle readahead verifier errors > together so we don't forget this behavioural link in future. > > cc: # 3.12 - current > Signed-off-by: Dave Chinner > --- > > Version 2 > - fix logic error in determining if verify failed > - set error on buffer when verifier fails > - fix inode buffer readahead verifier to set error when it fails > - better comments, link dquot and inode buffer ra verifiers in the > comments > > fs/xfs/libxfs/xfs_dquot_buf.c | 36 ++++++++++++++++++++++++++++++------ > fs/xfs/libxfs/xfs_inode_buf.c | 14 +++++++++----- > fs/xfs/libxfs/xfs_quota_defs.h | 2 +- > fs/xfs/libxfs/xfs_shared.h | 1 + > fs/xfs/xfs_log_recover.c | 9 +++++++-- > 5 files changed, 48 insertions(+), 14 deletions(-) > > diff --git a/fs/xfs/libxfs/xfs_dquot_buf.c b/fs/xfs/libxfs/xfs_dquot_buf.c > index 11cefb2..3cc3cf7 100644 > --- a/fs/xfs/libxfs/xfs_dquot_buf.c > +++ b/fs/xfs/libxfs/xfs_dquot_buf.c ... > @@ -264,6 +264,25 @@ xfs_dquot_buf_read_verify( > } > > /* > + * readahead errors are silent and simply leave the buffer as !done so a real > + * read will then be run with the xfs_dquot_buf_ops verifier. See > + * xfs_inode_buf_verify() for why we use EIO and ~XBF_DONE here rather than > + * reporting the failure. > + */ > +static void > +xfs_dquot_buf_readahead_verify( > + struct xfs_buf *bp) > +{ > + struct xfs_mount *mp = bp->b_target->bt_mount; > + > + if (!xfs_dquot_buf_verify_crc(mp, bp) || > + !xfs_dquot_buf_verify(mp, bp, 0)) { > + xfs_buf_ioerror(bp, -EIO); > + bp->b_flags &= ~XBF_DONE; Do we really need to clear the flag if the I/O infrastructure doesn't set it until after the verifier is invoked? It's harmless, so if the intent is to just be cautious or future-proof: Reviewed-by: Brian Foster ... but it does look slightly confusing, without a mention in the comments at least. Brian > + } > +} > + > +/* > * we don't calculate the CRC here as that is done when the dquot is flushed to > * the buffer after the update is done. This ensures that the dquot in the > * buffer always has an up-to-date CRC value. > @@ -274,7 +293,7 @@ xfs_dquot_buf_write_verify( > { > struct xfs_mount *mp = bp->b_target->bt_mount; > > - if (!xfs_dquot_buf_verify(mp, bp)) { > + if (!xfs_dquot_buf_verify(mp, bp, XFS_QMOPT_DOWARN)) { > xfs_buf_ioerror(bp, -EFSCORRUPTED); > xfs_verifier_error(bp); > return; > @@ -287,3 +306,8 @@ const struct xfs_buf_ops xfs_dquot_buf_ops = { > .verify_write = xfs_dquot_buf_write_verify, > }; > > +const struct xfs_buf_ops xfs_dquot_buf_ra_ops = { > + .name = "xfs_dquot_ra", > + .verify_read = xfs_dquot_buf_readahead_verify, > + .verify_write = xfs_dquot_buf_write_verify, > +}; > diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c > index 1b8d98a..4816209 100644 > --- a/fs/xfs/libxfs/xfs_inode_buf.c > +++ b/fs/xfs/libxfs/xfs_inode_buf.c > @@ -62,11 +62,14 @@ xfs_inobp_check( > * has not had the inode cores stamped into it. Hence for readahead, the buffer > * may be potentially invalid. > * > - * If the readahead buffer is invalid, we don't want to mark it with an error, > - * but we do want to clear the DONE status of the buffer so that a followup read > - * will re-read it from disk. This will ensure that we don't get an unnecessary > - * warnings during log recovery and we don't get unnecssary panics on debug > - * kernels. > + * If the readahead buffer is invalid, we need to mark it with an error and > + * clear the DONE status of the buffer so that a followup read will re-read it > + * from disk. We don't report the error otherwise to avoid warnings during log > + * recovery and we don't get unnecssary panics on debug kernels. We use EIO here > + * because all we want to do is say readahead failed; there is no-one to report > + * the error to, so this will distinguish it from a non-ra verifier failure. > + * Changes to this readahead error behavour also need to be reflected in > + * xfs_dquot_buf_readahead_verify(). > */ > static void > xfs_inode_buf_verify( > @@ -92,6 +95,7 @@ xfs_inode_buf_verify( > XFS_ERRTAG_ITOBP_INOTOBP, > XFS_RANDOM_ITOBP_INOTOBP))) { > if (readahead) { > + xfs_buf_ioerror(bp, -EIO); > bp->b_flags &= ~XBF_DONE; > return; > } > diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h > index 1b0a083..f51078f 100644 > --- a/fs/xfs/libxfs/xfs_quota_defs.h > +++ b/fs/xfs/libxfs/xfs_quota_defs.h > @@ -153,7 +153,7 @@ typedef __uint16_t xfs_qwarncnt_t; > #define XFS_QMOPT_RESBLK_MASK (XFS_QMOPT_RES_REGBLKS | XFS_QMOPT_RES_RTBLKS) > > extern int xfs_dqcheck(struct xfs_mount *mp, xfs_disk_dquot_t *ddq, > - xfs_dqid_t id, uint type, uint flags, char *str); > + xfs_dqid_t id, uint type, uint flags, const char *str); > extern int xfs_calc_dquots_per_chunk(unsigned int nbblks); > > #endif /* __XFS_QUOTA_H__ */ > diff --git a/fs/xfs/libxfs/xfs_shared.h b/fs/xfs/libxfs/xfs_shared.h > index 5be5297..15c3ceb 100644 > --- a/fs/xfs/libxfs/xfs_shared.h > +++ b/fs/xfs/libxfs/xfs_shared.h > @@ -49,6 +49,7 @@ extern const struct xfs_buf_ops xfs_inobt_buf_ops; > extern const struct xfs_buf_ops xfs_inode_buf_ops; > extern const struct xfs_buf_ops xfs_inode_buf_ra_ops; > extern const struct xfs_buf_ops xfs_dquot_buf_ops; > +extern const struct xfs_buf_ops xfs_dquot_buf_ra_ops; > extern const struct xfs_buf_ops xfs_sb_buf_ops; > extern const struct xfs_buf_ops xfs_sb_quiet_buf_ops; > extern const struct xfs_buf_ops xfs_symlink_buf_ops; > diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c > index 26e67b4..da37beb 100644 > --- a/fs/xfs/xfs_log_recover.c > +++ b/fs/xfs/xfs_log_recover.c > @@ -3521,6 +3521,7 @@ xlog_recover_dquot_ra_pass2( > struct xfs_disk_dquot *recddq; > struct xfs_dq_logformat *dq_f; > uint type; > + int len; > > > if (mp->m_qflags == 0) > @@ -3541,8 +3542,12 @@ xlog_recover_dquot_ra_pass2( > ASSERT(dq_f); > ASSERT(dq_f->qlf_len == 1); > > - xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, > - XFS_FSB_TO_BB(mp, dq_f->qlf_len), NULL); > + len = XFS_FSB_TO_BB(mp, dq_f->qlf_len); > + if (xlog_peek_buffer_cancelled(log, dq_f->qlf_blkno, len, 0)) > + return; > + > + xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, len, > + &xfs_dquot_buf_ra_ops); > } > > STATIC void > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Thu Jan 7 06:53:12 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 83EE47F37 for ; Thu, 7 Jan 2016 06:53:12 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 5E0B28F8040 for ; Thu, 7 Jan 2016 04:53:12 -0800 (PST) X-ASG-Debug-ID: 1452171191-04cb6c42e533710001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id WYakEVn9o5J9OpDS (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 07 Jan 2016 04:53:11 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id D6FA819F27B; Thu, 7 Jan 2016 12:53:10 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-81.bos.redhat.com [10.18.41.81]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u07CrACE006623; Thu, 7 Jan 2016 07:53:10 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id A2F28125794; Thu, 7 Jan 2016 07:53:09 -0500 (EST) Date: Thu, 7 Jan 2016 07:53:09 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 1/2] xfs: add tracepoints to readpage calls Message-ID: <20160107125306.GB33327@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 1/2] xfs: add tracepoints to readpage calls References: <1452123087-13892-1-git-send-email-david@fromorbit.com> <1452123087-13892-2-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452123087-13892-2-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452171191 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Thu, Jan 07, 2016 at 10:31:26AM +1100, Dave Chinner wrote: > From: Dave Chinner > > This allows us to see page cache driven readahead in action as it > passes through XFS. This helps to understand buffered read > throughput problems such as readahead IO IO sizes being too small Extra "IO" above: ^ > for the underlying device to reach max throughput. > > Signed-off-by: Dave Chinner > --- Reviewed-by: Brian Foster > fs/xfs/xfs_aops.c | 2 ++ > fs/xfs/xfs_trace.h | 26 ++++++++++++++++++++++++++ > 2 files changed, 28 insertions(+) > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c > index 29e7e5d..379c089 100644 > --- a/fs/xfs/xfs_aops.c > +++ b/fs/xfs/xfs_aops.c > @@ -1917,6 +1917,7 @@ xfs_vm_readpage( > struct file *unused, > struct page *page) > { > + trace_xfs_vm_readpage(page->mapping->host, 1); > return mpage_readpage(page, xfs_get_blocks); > } > > @@ -1927,6 +1928,7 @@ xfs_vm_readpages( > struct list_head *pages, > unsigned nr_pages) > { > + trace_xfs_vm_readpages(mapping->host, nr_pages); > return mpage_readpages(mapping, pages, nr_pages, xfs_get_blocks); > } > > diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h > index 877079eb..391d797 100644 > --- a/fs/xfs/xfs_trace.h > +++ b/fs/xfs/xfs_trace.h > @@ -1222,6 +1222,32 @@ DEFINE_PAGE_EVENT(xfs_writepage); > DEFINE_PAGE_EVENT(xfs_releasepage); > DEFINE_PAGE_EVENT(xfs_invalidatepage); > > +DECLARE_EVENT_CLASS(xfs_readpage_class, > + TP_PROTO(struct inode *inode, int nr_pages), > + TP_ARGS(inode, nr_pages), > + TP_STRUCT__entry( > + __field(dev_t, dev) > + __field(xfs_ino_t, ino) > + __field(int, nr_pages) > + ), > + TP_fast_assign( > + __entry->dev = inode->i_sb->s_dev; > + __entry->ino = inode->i_ino; > + __entry->nr_pages = nr_pages; > + ), > + TP_printk("dev %d:%d ino 0x%llx nr_pages %d", > + MAJOR(__entry->dev), MINOR(__entry->dev), > + __entry->ino, > + __entry->nr_pages) > +) > + > +#define DEFINE_READPAGE_EVENT(name) \ > +DEFINE_EVENT(xfs_readpage_class, name, \ > + TP_PROTO(struct inode *inode, int nr_pages), \ > + TP_ARGS(inode, nr_pages)) > +DEFINE_READPAGE_EVENT(xfs_vm_readpage); > +DEFINE_READPAGE_EVENT(xfs_vm_readpages); > + > DECLARE_EVENT_CLASS(xfs_imap_class, > TP_PROTO(struct xfs_inode *ip, xfs_off_t offset, ssize_t count, > int type, struct xfs_bmbt_irec *irec), > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Thu Jan 7 06:53:17 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 943187F50 for ; Thu, 7 Jan 2016 06:53:17 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 831FC8F8040 for ; Thu, 7 Jan 2016 04:53:17 -0800 (PST) X-ASG-Debug-ID: 1452171196-04cb6c42e733720001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id jsrqkx5PNSkgDIR8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 07 Jan 2016 04:53:16 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 0A965325DC6; Thu, 7 Jan 2016 12:53:16 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-81.bos.redhat.com [10.18.41.81]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u07CrFPk017512; Thu, 7 Jan 2016 07:53:15 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id D75EE125794; Thu, 7 Jan 2016 07:53:14 -0500 (EST) Date: Thu, 7 Jan 2016 07:53:14 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 2/2] xfs: bmapbt checking on debug kernels too expensive Message-ID: <20160107125314.GC33327@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 2/2] xfs: bmapbt checking on debug kernels too expensive References: <1452123087-13892-1-git-send-email-david@fromorbit.com> <1452123087-13892-3-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452123087-13892-3-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452171196 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Thu, Jan 07, 2016 at 10:31:27AM +1100, Dave Chinner wrote: > From: Dave Chinner > > For large sparse or fragmented files, checking every single entry in > the bmapbt on every operation is prohibitively expensive. Especially > as such checks rarely discover problems during normal operations on > high extent coutn files. Our regression tests don't tend to exercise count > files with hundreds of thousands to millions of extents, so mostly > this isn't noticed. > > However, trying to run things like xfs_mdrestore of large filesystem > dumps on a debug kernel quickly becomes impossible as the CPU is > completely burnt up repeatedly walking the sparse file bmapbt that > is generated for every allocation that is made. > > Hence, if the file has more than 10,000 extents, just don't bother > with walking the tree to check it exhaustively. The btree code has > checks that ensure that the newly inserted/removed/modified record > is correctly ordered, so the entrie tree walk in thses cases has these > limited additional value. > > Signed-off-by: Dave Chinner > --- > fs/xfs/libxfs/xfs_bmap.c | 10 ++++++++-- > 1 file changed, 8 insertions(+), 2 deletions(-) > > diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c > index 7388495..bc7e7d5 100644 > --- a/fs/xfs/libxfs/xfs_bmap.c > +++ b/fs/xfs/libxfs/xfs_bmap.c > @@ -325,9 +325,11 @@ xfs_check_block( > > /* > * Check that the extents for the inode ip are in the right order in all > - * btree leaves. > + * btree leaves. THis becomes prohibitively expensive for large extent count This Reviewed-by: Brian Foster > + * files, so don't bother with inodes that have more than 10,000 extents in > + * them. The btree record ordering checks will still be done, so for such large > + * bmapbt constructs that is going to catch most corruptions. > */ > - > STATIC void > xfs_bmap_check_leaf_extents( > xfs_btree_cur_t *cur, /* btree cursor or null */ > @@ -352,6 +354,10 @@ xfs_bmap_check_leaf_extents( > return; > } > > + /* skip large extent count inodes */ > + if (ip->i_d.di_nextents > 10000) > + return; > + > bno = NULLFSBLOCK; > mp = ip->i_mount; > ifp = XFS_IFORK_PTR(ip, whichfork); > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From jack@suse.cz Thu Jan 7 07:22:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id DEAC77F37 for ; Thu, 7 Jan 2016 07:22:10 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id ACD718F8049 for ; Thu, 7 Jan 2016 05:22:07 -0800 (PST) X-ASG-Debug-ID: 1452172922-04bdf03afa39e80001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id zTAo2nusClFCpo3N (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 07 Jan 2016 05:22:04 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id B27A2AC19; Thu, 7 Jan 2016 13:21:58 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id CBAD2823D8; Thu, 7 Jan 2016 14:22:06 +0100 (CET) Date: Thu, 7 Jan 2016 14:22:06 +0100 From: Jan Kara To: Ross Zwisler Cc: linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Message-ID: <20160107132206.GE8380@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452172923 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25916 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed 06-01-16 11:00:56, Ross Zwisler wrote: > When we get a DAX PMD fault for a write it is possible that there could be > some number of 4k zero pages already present for the same range that were > inserted to service reads from a hole. These 4k zero pages need to be > unmapped from the VMAs and removed from the struct address_space radix tree > before the real DAX PMD entry can be inserted. > > For PTE faults this same use case also exists and is handled by a > combination of unmap_mapping_range() to unmap the VMAs and > delete_from_page_cache() to remove the page from the address_space radix > tree. > > For PMD faults we do have a call to unmap_mapping_range() (protected by a > buffer_new() check), but nothing clears out the radix tree entry. The > buffer_new() check is also incorrect as the current ext4 and XFS filesystem > code will never return a buffer_head with BH_New set, even when allocating > new blocks over a hole. Instead the filesystem will zero the blocks > manually and return a buffer_head with only BH_Mapped set. > > Fix this situation by removing the buffer_new() check and adding a call to > truncate_inode_pages_range() to clear out the radix tree entries before we > insert the DAX PMD. Ho, hum, let me understand this. So we have a file, different processes are mapping it. One process maps is with normal page granularity and another process with huge page granularity. Thus when the first process read-faults a few normal pages and then the second process write-faults the huge page in the same range, we have a problem. Do I understand this correctly? Because otherwise I don't understand how a single page table can have both huge page and normal page in the same range... And if this is indeed the problem then what prevents the unmapping and truncation in huge page fault to race with mapping the same range again using small pages? Sure now blocks are allocated so the mapping itself will be consistent but radix tree will have the same issues it had before this patch, won't it? ... thinking some more about this ... OK, there is some difference - we will only have DAX exceptional entries for the range covered by huge page and those we replace properly in dax_radix_entry() code. So things are indeed fine *except* that nothing seems so serialize dax_load() hole with PMD fault. The race like following seems possible: CPU1 - process 1 CPU2 - process 2 __dax_fault() - file f, index 1 get_block() -> returns hole __dax_pmd_fault() - file f, index 0 get_block() -> allocates blocks ... truncate_pagecache_range() dax_load_hole() Boom, we have hole page instantiated for allocated range (data corruption) and corruption of radix tree entries as well. Actually this problem is there even for two different processes doing normal page faults (one read, one write) against the same page in the file. ... thinking about possible fixes ... So we need some exclusion that makes sure pgoff->block mapping information is uptodate at the moment we insert it into page tables. The simplest reasonably fast thing I can see is: When handling a read fault, things stay as is and filesystem protects the fault with an equivalent of EXT4_I(inode)->i_mmap_sem held for reading. When handling a write fault we first grab EXT4_I(inode)->i_mmap_sem for reading and try a read fault. If __dax_fault() sees a hole returned from get_blocks() during a write fault, it bails out. Filesystem grabs EXT4_I(inode)->i_mmap_sem for writing and retries with different get_blocks() callback which will allocate blocks. That way we get proper exclusion for faults needing to allocate blocks. Thoughts? Honza > Signed-off-by: Ross Zwisler > --- > fs/dax.c | 20 ++++++++++---------- > 1 file changed, 10 insertions(+), 10 deletions(-) > > diff --git a/fs/dax.c b/fs/dax.c > index 03cc4a3..9dc0c97 100644 > --- a/fs/dax.c > +++ b/fs/dax.c > @@ -594,6 +594,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > bool write = flags & FAULT_FLAG_WRITE; > struct block_device *bdev; > pgoff_t size, pgoff; > + loff_t lstart, lend; > sector_t block; > int result = 0; > > @@ -647,15 +648,13 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > goto fallback; > } > > - /* > - * If we allocated new storage, make sure no process has any > - * zero pages covering this hole > - */ > - if (buffer_new(&bh)) { > - i_mmap_unlock_read(mapping); > - unmap_mapping_range(mapping, pgoff << PAGE_SHIFT, PMD_SIZE, 0); > - i_mmap_lock_read(mapping); > - } > + /* make sure no process has any zero pages covering this hole */ > + lstart = pgoff << PAGE_SHIFT; > + lend = lstart + PMD_SIZE - 1; /* inclusive */ > + i_mmap_unlock_read(mapping); > + unmap_mapping_range(mapping, lstart, PMD_SIZE, 0); > + truncate_inode_pages_range(mapping, lstart, lend); > + i_mmap_lock_read(mapping); > > /* > * If a truncate happened while we were allocating blocks, we may > @@ -669,7 +668,8 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > goto out; > } > if ((pgoff | PG_PMD_COLOUR) >= size) { > - dax_pmd_dbg(&bh, address, "pgoff unaligned"); > + dax_pmd_dbg(&bh, address, > + "offset + huge page size > file size"); > goto fallback; > } > > -- > 2.5.0 > > -- Jan Kara SUSE Labs, CR From dan.j.williams@intel.com Thu Jan 7 09:17:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 5C8D17F37 for ; Thu, 7 Jan 2016 09:17:25 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id C9A8BAC005 for ; Thu, 7 Jan 2016 07:17:24 -0800 (PST) X-ASG-Debug-ID: 1452179843-04cbb07e1a3db50001-NocioJ Received: from mail-qk0-f177.google.com (mail-qk0-f177.google.com [209.85.220.177]) by cuda.sgi.com with ESMTP id wt2iVSjwenskYEHL (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 07 Jan 2016 07:17:23 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.220.177 Received: by mail-qk0-f177.google.com with SMTP id p186so102041332qke.0 for ; Thu, 07 Jan 2016 07:17:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=blBsxFW1jn3UvTGPr4i5m7y+kNS2YQAJZNYrVfOAc1g=; b=g44EptADUG9lj0Vl7zN54V935WtHCbzBlk5VZTXZefHDyhWrzlk2O9EHCC+zXT3MiY kn3Zn83MdZUZBrLQn3QusY8A2lFzqmrHhFre4fguYgaEDuMSbsSx5aONwOx8CYq0I2Wm zz4f2V/cVAOCJFRQnKAdNX0wcfVKXq/jXVM+53e4aXVRsoczvVibK0D3YtkDRHSU2fAN fwd347rtGCbtJEz/NM0Abih+rdrjOTBdvECMARRPI6lGrwCSMuH1pEZJMmmlK9fvDIxt I82zWnQ6cB3IrYXZ7HYsIxJX20rPr3ecrSRPHG6A9PhZH1Adh8kL+5uJ+TRm65Z2uPjH Iitw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=blBsxFW1jn3UvTGPr4i5m7y+kNS2YQAJZNYrVfOAc1g=; b=ateysG2lRkjzrnMWSiXJcVpbmlvNcmelw/gZsX1qPxoKo2EEUcGlWHd7S6azmTM4OR GatsvGCs3QCfDCQLimjyfSi405fMI2na4dMEi8a03yIQvszFzV7+Rm0CkFRp0zm0okUE 7ZaiIXNVmz8L+qeGhaTk1u0G0JSr9s0Mkm+MGSs6X107SjEqoxIFDKxLsiE0J2LEm8ij hgfJwuifir3hGVdHfjFWLawITpxLE+8w9w/3ANaH8mLgLGSGCrUJalo1Nb59uYujWBEq u9woTq855KDi6jTAuK8elda8yCiRQ2A8BWeToEaQoRPe6X4PD0xE2vO/Y6U4SrrqioU8 1D7g== X-Gm-Message-State: ALoCoQluh/ryq/ZRdd7fVS/oPvSiNqMy2SGglauhox1bzBLgHhLcXVOkQbD0Zuc1lrlmvcZdoF1/9WZ8TcAHdQH9z7dQJl8qrp1gTKwtkrcsuIJ6ahuvra8= MIME-Version: 1.0 X-Received: by 10.13.228.134 with SMTP id n128mr65442049ywe.287.1452179842642; Thu, 07 Jan 2016 07:17:22 -0800 (PST) Received: by 10.37.202.20 with HTTP; Thu, 7 Jan 2016 07:17:22 -0800 (PST) In-Reply-To: <20160107093402.GA8380@quack.suse.cz> References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-2-git-send-email-ross.zwisler@linux.intel.com> <20160107093402.GA8380@quack.suse.cz> Date: Thu, 7 Jan 2016 07:17:22 -0800 Message-ID: Subject: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() From: Dan Williams X-ASG-Orig-Subj: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() To: Jan Kara Cc: Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-qk0-f177.google.com[209.85.220.177] X-Barracuda-Start-Time: 1452179843 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25917 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Thu, Jan 7, 2016 at 1:34 AM, Jan Kara wrote: > On Wed 06-01-16 11:14:09, Dan Williams wrote: >> On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler >> wrote: >> > __dax_dbg() currently assumes that bh->b_bdev is non-NULL, passing it into >> > bdevname() where is is dereferenced. This assumption isn't always true - >> > when called for reads of holes, ext4_dax_mmap_get_block() returns a buffer >> > head where bh->b_bdev is never set. I hit this BUG while testing the DAX >> > PMD fault path. >> > >> > Instead, verify that we have a valid bh->b_bdev, else just say "unknown" >> > for the block device. >> > >> > Signed-off-by: Ross Zwisler >> > Cc: Dan Williams >> > --- >> > fs/dax.c | 7 ++++++- >> > 1 file changed, 6 insertions(+), 1 deletion(-) >> > >> > diff --git a/fs/dax.c b/fs/dax.c >> > index 7af8797..03cc4a3 100644 >> > --- a/fs/dax.c >> > +++ b/fs/dax.c >> > @@ -563,7 +563,12 @@ static void __dax_dbg(struct buffer_head *bh, unsigned long address, >> > { >> > if (bh) { >> > char bname[BDEVNAME_SIZE]; >> > - bdevname(bh->b_bdev, bname); >> > + >> > + if (bh->b_bdev) >> > + bdevname(bh->b_bdev, bname); >> > + else >> > + snprintf(bname, BDEVNAME_SIZE, "unknown"); >> > + >> > pr_debug("%s: %s addr: %lx dev %s state %lx start %lld " >> > "length %zd fallback: %s\n", fn, current->comm, >> > address, bname, bh->b_state, (u64)bh->b_blocknr, >> >> I'm assuming there's no danger of a such a buffer_head ever being used >> for the bdev parameter to dax_map_atomic()? Shouldn't we also/instead >> go fix ext4 to not send partially filled buffer_heads? > > No. The real problem is a long-standing abuse of struct buffer_head to be > used for passing block mapping information (it's on my todo list to remove > that at least from DAX code and use cleaner block mapping interface but > first I want basic DAX functionality to settle down to avoid unnecessary > conflicts). Filesystem is not supposed to touch bh->b_bdev. If you need > that filled in, set it yourself in before passing bh to the block mapping > function. > Ok, makes sense. Ross, can you fix this instead by unconditionally looking up the bdev rather that saying "unknown". The bdev should always be retrievable. From hch@lst.de Thu Jan 7 09:25:49 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0E0807F3F for ; Thu, 7 Jan 2016 09:25:49 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id F30C88F8052 for ; Thu, 7 Jan 2016 07:25:45 -0800 (PST) X-ASG-Debug-ID: 1452180342-04cb6c42e736ca0001-NocioJ Received: from newverein.lst.de (verein.lst.de [213.95.11.211]) by cuda.sgi.com with ESMTP id A17UjLfsSTn3k30W (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 07 Jan 2016 07:25:43 -0800 (PST) X-Barracuda-Envelope-From: hch@lst.de X-Barracuda-Apparent-Source-IP: 213.95.11.211 Received: by newverein.lst.de (Postfix, from userid 2407) id 343B9691CD; Thu, 7 Jan 2016 16:25:42 +0100 (CET) Date: Thu, 7 Jan 2016 16:25:42 +0100 From: Christoph Hellwig To: "Darrick J. Wong" Cc: xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160107152541.GA16982@lst.de> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> <20160105014310.GK28330@birch.djwong.org> <20160105104214.GA16310@infradead.org> <20160107003227.GB8015@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107003227.GB8015@birch.djwong.org> User-Agent: Mutt/1.5.17 (2007-11-01) X-Barracuda-Connect: verein.lst.de[213.95.11.211] X-Barracuda-Start-Time: 1452180343 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25918 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed, Jan 06, 2016 at 04:32:27PM -0800, Darrick J. Wong wrote: > Ok. I spent a couple of days trying to find all the places where we need to > delete CoW reservations (hole punch, truncate, etc.) and found some places > where the code was leaving reservations behind in the CoW fork (most notable > truncate). I also made the inode eviction code purge any CoW leftovers, so > that should all go away. Can you send that part out for NFS testing? > I also wrote some more xfstests that try to hit all the CoW-cancelling code > paths (fpunch, fzero, fcollapse, finsert, truncate, -EIO) to smoke test all > that. By the way, do you have a testcase handy for the "non-blocking writeback > EAGAIN" case? I'm guessing that we could hit that pretty easily by lowering > dirty_background_* and dirtying a lot of pages while reflinking? I did hit it pretty easily testing over NFS to a local nfs server with xfstests. From ross.zwisler@linux.intel.com Thu Jan 7 16:11:22 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 527A67F37 for ; Thu, 7 Jan 2016 16:11:22 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id EA583AC001 for ; Thu, 7 Jan 2016 14:11:18 -0800 (PST) X-ASG-Debug-ID: 1452204676-04bdf03afb46da0001-NocioJ Received: from mga04.intel.com ([192.55.52.120]) by cuda.sgi.com with ESMTP id H9b9MS5Ujl6XQWss for ; Thu, 07 Jan 2016 14:11:16 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.120 Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga104.fm.intel.com with ESMTP; 07 Jan 2016 14:11:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,534,1444719600"; d="scan'208";a="722406942" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.201.208]) by orsmga003.jf.intel.com with ESMTP; 07 Jan 2016 14:11:14 -0800 Date: Thu, 7 Jan 2016 15:11:14 -0700 From: Ross Zwisler To: Jan Kara Cc: Ross Zwisler , linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Message-ID: <20160107221114.GA20802@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Mail-Followup-To: Ross Zwisler , Jan Kara , linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> <20160107132206.GE8380@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107132206.GE8380@quack.suse.cz> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: UNKNOWN[192.55.52.120] X-Barracuda-Start-Time: 1452204676 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25929 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Thu, Jan 07, 2016 at 02:22:06PM +0100, Jan Kara wrote: > On Wed 06-01-16 11:00:56, Ross Zwisler wrote: > > When we get a DAX PMD fault for a write it is possible that there could be > > some number of 4k zero pages already present for the same range that were > > inserted to service reads from a hole. These 4k zero pages need to be > > unmapped from the VMAs and removed from the struct address_space radix tree > > before the real DAX PMD entry can be inserted. > > > > For PTE faults this same use case also exists and is handled by a > > combination of unmap_mapping_range() to unmap the VMAs and > > delete_from_page_cache() to remove the page from the address_space radix > > tree. > > > > For PMD faults we do have a call to unmap_mapping_range() (protected by a > > buffer_new() check), but nothing clears out the radix tree entry. The > > buffer_new() check is also incorrect as the current ext4 and XFS filesystem > > code will never return a buffer_head with BH_New set, even when allocating > > new blocks over a hole. Instead the filesystem will zero the blocks > > manually and return a buffer_head with only BH_Mapped set. > > > > Fix this situation by removing the buffer_new() check and adding a call to > > truncate_inode_pages_range() to clear out the radix tree entries before we > > insert the DAX PMD. > > Ho, hum, let me understand this. So we have a file, different processes are > mapping it. One process maps is with normal page granularity and another > process with huge page granularity. Thus when the first process read-faults > a few normal pages and then the second process write-faults the huge page > in the same range, we have a problem. Do I understand this correctly? > Because otherwise I don't understand how a single page table can have both > huge page and normal page in the same range... I don't think that it necessarily has to do with multiple threads. The bit to notice here is we *always* use 4k zero pages to cover holes. So, a single thread can hit this condition by doing some reads from a hole (insert 4k pages), then doing a write. This write is the first time that we will try and use real DAX storage to insert into the page tables, and we may end up getting a PMD. This means that we need to clear out all the 4k pages that we inserted while reading holes in this same range, now that we have a 2M segment allocated by the filesystem and the entire range is no longer a hole. > And if this is indeed the problem then what prevents the unmapping and > truncation in huge page fault to race with mapping the same range again > using small pages? Sure now blocks are allocated so the mapping itself will > be consistent but radix tree will have the same issues it had before this > patch, won't it? Yep, this is a separate issue, but I think that we handle this case successfully, though we may end up flushing the same address multiple times. Once the filesystem has established a block mapping (assuming we avoid the race described below where one thread is mapping in holes and the other sees a block allocation), I think we are okay. It's true that one thread can map in PMDs, and another thread could potentially map in PTEs that cover the same range if they hare working with mmaps that are smaller than a PMD, but the sectors inserted into the radix tree by each of those threads will be individually correct - the only issue is that they may overlap. Say, for example you have the following: CPU1 - process 1 CPU2 - process 2 mmap for sector 0, size 2M insert PMD into radix tree for sector 0 This radix tree covers sectors 0-4096 mmap for sector 32, size 4k insert PTE entry into radix tree for sector 32 In this case a fsync of the fd by process 1 will end up flushing sector 32 twice, which is correct but inefficient. I think we can make this more efficient by adjusting the insertion code and dirtying code in dax_radix_entry() to look for PMDs that cover this same range. > ... thinking some more about this ... > > OK, there is some difference - we will only have DAX exceptional entries > for the range covered by huge page and those we replace properly in > dax_radix_entry() code. So things are indeed fine *except* that nothing > seems so serialize dax_load() hole with PMD fault. The race like following > seems possible: > > CPU1 - process 1 CPU2 - process 2 > > __dax_fault() - file f, index 1 > get_block() -> returns hole > __dax_pmd_fault() - file f, index 0 > get_block() -> allocates blocks > ... > truncate_pagecache_range() > dax_load_hole() > > Boom, we have hole page instantiated for allocated range (data corruption) > and corruption of radix tree entries as well. Actually this problem is > there even for two different processes doing normal page faults (one read, > one write) against the same page in the file. Yea, I agree, this seems like an existing issue that you could hit with just the PTE path: CPU1 - process 1 CPU2 - process 2 __dax_fault() - read file f, index 0 get_block() -> returns hole __dax_fault() - write file f, index 0 get_block() -> allocates blocks ... skips unmap_mapping_range() and delete_from_page_cache() because it didn't find a page for this pgoff dax_insert_mapping() dax_load_hole() *data corruption* > ... thinking about possible fixes ... > > So we need some exclusion that makes sure pgoff->block mapping information > is uptodate at the moment we insert it into page tables. The simplest > reasonably fast thing I can see is: > > When handling a read fault, things stay as is and filesystem protects the > fault with an equivalent of EXT4_I(inode)->i_mmap_sem held for reading. When > handling a write fault we first grab EXT4_I(inode)->i_mmap_sem for reading > and try a read fault. If __dax_fault() sees a hole returned from > get_blocks() during a write fault, it bails out. Filesystem grabs > EXT4_I(inode)->i_mmap_sem for writing and retries with different > get_blocks() callback which will allocate blocks. That way we get proper > exclusion for faults needing to allocate blocks. Thoughts? I think this would work. ext4, ext2 and xfs all handle their exclusion with rw_semaphores, so this should work for each of them, I think. Thanks for the problem statement & solution! :) I guess our best course is to make sure that we don't make this existing problem worse via the fsync/msync patches by handling the error gracefully, and fix this for v4.6. I do feel the need to point out that this is a pre-existing issue with DAX, and that my fsync patches just happened to help us find it. They don't make the situation any better or any worse, and I really hope this issue doesn't end up blocking the fsync/msync patches from getting merged for v4.5. Thanks, - Ross From ross.zwisler@linux.intel.com Thu Jan 7 16:16:11 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id EE3767F3F for ; Thu, 7 Jan 2016 16:16:10 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 603C8AC003 for ; Thu, 7 Jan 2016 14:16:10 -0800 (PST) X-ASG-Debug-ID: 1452204968-04cbb07e1648400001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id ugaJiVoZOm6qDhj2 for ; Thu, 07 Jan 2016 14:16:08 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 14:16:08 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,534,1444719600"; d="scan'208";a="722409309" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.201.208]) by orsmga003.jf.intel.com with ESMTP; 07 Jan 2016 14:16:06 -0800 Date: Thu, 7 Jan 2016 15:16:06 -0700 From: Ross Zwisler To: Dan Williams Cc: Jan Kara , Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Subject: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() Message-ID: <20160107221606.GB20802@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() Mail-Followup-To: Ross Zwisler , Dan Williams , Jan Kara , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-2-git-send-email-ross.zwisler@linux.intel.com> <20160107093402.GA8380@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452204968 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Thu, Jan 07, 2016 at 07:17:22AM -0800, Dan Williams wrote: > On Thu, Jan 7, 2016 at 1:34 AM, Jan Kara wrote: > > On Wed 06-01-16 11:14:09, Dan Williams wrote: > >> On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler > >> wrote: > >> > __dax_dbg() currently assumes that bh->b_bdev is non-NULL, passing it into > >> > bdevname() where is is dereferenced. This assumption isn't always true - > >> > when called for reads of holes, ext4_dax_mmap_get_block() returns a buffer > >> > head where bh->b_bdev is never set. I hit this BUG while testing the DAX > >> > PMD fault path. > >> > > >> > Instead, verify that we have a valid bh->b_bdev, else just say "unknown" > >> > for the block device. > >> > > >> > Signed-off-by: Ross Zwisler > >> > Cc: Dan Williams > >> > --- > >> > fs/dax.c | 7 ++++++- > >> > 1 file changed, 6 insertions(+), 1 deletion(-) > >> > > >> > diff --git a/fs/dax.c b/fs/dax.c > >> > index 7af8797..03cc4a3 100644 > >> > --- a/fs/dax.c > >> > +++ b/fs/dax.c > >> > @@ -563,7 +563,12 @@ static void __dax_dbg(struct buffer_head *bh, unsigned long address, > >> > { > >> > if (bh) { > >> > char bname[BDEVNAME_SIZE]; > >> > - bdevname(bh->b_bdev, bname); > >> > + > >> > + if (bh->b_bdev) > >> > + bdevname(bh->b_bdev, bname); > >> > + else > >> > + snprintf(bname, BDEVNAME_SIZE, "unknown"); > >> > + > >> > pr_debug("%s: %s addr: %lx dev %s state %lx start %lld " > >> > "length %zd fallback: %s\n", fn, current->comm, > >> > address, bname, bh->b_state, (u64)bh->b_blocknr, > >> > >> I'm assuming there's no danger of a such a buffer_head ever being used > >> for the bdev parameter to dax_map_atomic()? Shouldn't we also/instead > >> go fix ext4 to not send partially filled buffer_heads? > > > > No. The real problem is a long-standing abuse of struct buffer_head to be > > used for passing block mapping information (it's on my todo list to remove > > that at least from DAX code and use cleaner block mapping interface but > > first I want basic DAX functionality to settle down to avoid unnecessary > > conflicts). Filesystem is not supposed to touch bh->b_bdev. If you need > > that filled in, set it yourself in before passing bh to the block mapping > > function. > > > > Ok, makes sense. > > Ross, can you fix this instead by unconditionally looking up the bdev > rather that saying "unknown". The bdev should always be retrievable. Sure, will do. From ross.zwisler@linux.intel.com Thu Jan 7 16:35:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 3B7C97F37 for ; Thu, 7 Jan 2016 16:35:01 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id AE19DAC002 for ; Thu, 7 Jan 2016 14:35:00 -0800 (PST) X-ASG-Debug-ID: 1452206097-04cb6c42e740cc0001-NocioJ Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id 3TJMMzbd0HUNJIhQ for ; Thu, 07 Jan 2016 14:34:57 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.65 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP; 07 Jan 2016 14:34:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,535,1444719600"; d="scan'208";a="888613802" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.201.208]) by fmsmga002.fm.intel.com with ESMTP; 07 Jan 2016 14:34:55 -0800 Date: Thu, 7 Jan 2016 15:34:55 -0700 From: Ross Zwisler To: Dan Williams Cc: Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Subject: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Message-ID: <20160107223455.GC20802@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Mail-Followup-To: Ross Zwisler , Dan Williams , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mga03.intel.com[134.134.136.65] X-Barracuda-Start-Time: 1452206097 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25930 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Wed, Jan 06, 2016 at 11:04:35AM -0800, Dan Williams wrote: > On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler > wrote: > > When we get a DAX PMD fault for a write it is possible that there could be > > some number of 4k zero pages already present for the same range that were > > inserted to service reads from a hole. These 4k zero pages need to be > > unmapped from the VMAs and removed from the struct address_space radix tree > > before the real DAX PMD entry can be inserted. > > > > For PTE faults this same use case also exists and is handled by a > > combination of unmap_mapping_range() to unmap the VMAs and > > delete_from_page_cache() to remove the page from the address_space radix > > tree. > > > > For PMD faults we do have a call to unmap_mapping_range() (protected by a > > buffer_new() check), but nothing clears out the radix tree entry. The > > buffer_new() check is also incorrect as the current ext4 and XFS filesystem > > code will never return a buffer_head with BH_New set, even when allocating > > new blocks over a hole. Instead the filesystem will zero the blocks > > manually and return a buffer_head with only BH_Mapped set. > > > > Fix this situation by removing the buffer_new() check and adding a call to > > truncate_inode_pages_range() to clear out the radix tree entries before we > > insert the DAX PMD. > > > > Signed-off-by: Ross Zwisler > > Replaced the current contents of v6 in -mm from next-20160106 with > this v7 set and it looks good. > > Reported-by: Dan Williams > Tested-by: Dan Williams > > One question below... > > > --- > > fs/dax.c | 20 ++++++++++---------- > > 1 file changed, 10 insertions(+), 10 deletions(-) > > > > diff --git a/fs/dax.c b/fs/dax.c > > index 03cc4a3..9dc0c97 100644 > > --- a/fs/dax.c > > +++ b/fs/dax.c > > @@ -594,6 +594,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > > bool write = flags & FAULT_FLAG_WRITE; > > struct block_device *bdev; > > pgoff_t size, pgoff; > > + loff_t lstart, lend; > > sector_t block; > > int result = 0; > > > > @@ -647,15 +648,13 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > > goto fallback; > > } > > > > - /* > > - * If we allocated new storage, make sure no process has any > > - * zero pages covering this hole > > - */ > > - if (buffer_new(&bh)) { > > - i_mmap_unlock_read(mapping); > > - unmap_mapping_range(mapping, pgoff << PAGE_SHIFT, PMD_SIZE, 0); > > - i_mmap_lock_read(mapping); > > - } > > + /* make sure no process has any zero pages covering this hole */ > > + lstart = pgoff << PAGE_SHIFT; > > + lend = lstart + PMD_SIZE - 1; /* inclusive */ > > + i_mmap_unlock_read(mapping); > > + unmap_mapping_range(mapping, lstart, PMD_SIZE, 0); > > + truncate_inode_pages_range(mapping, lstart, lend); > > Do we need to do both unmap and truncate given that > truncate_inode_page() optionally does an unmap_mapping_range() > internally? Ah, indeed it does. Sure, having just the call to truncate_inode_page() seems cleaner. I'll re-test and send this out in v8. From david@fromorbit.com Thu Jan 7 17:14:34 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 345717F37 for ; Thu, 7 Jan 2016 17:14:34 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1EBCB8F804C for ; Thu, 7 Jan 2016 15:14:31 -0800 (PST) X-ASG-Debug-ID: 1452208468-04cb6c49fa00190001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id ChMaHvBkcJA6IfP1 for ; Thu, 07 Jan 2016 15:14:28 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BdDADM8I5WPGu7LXleKAECgw9SbYJihXegJQEBAQEBAQaLWYU6hAUehGEBgQkEAgKBI00BAQEBAQEHAQEBAUE/QRIBg2EBAQQ6HCMQCAMYCSUPBSUDBy2ILg/CPwEBAQEGAQEBARsEGYV1hUeCbYFRZYQZBZcLhUKIC4FlhEOIXESKFoNzhQIqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 08 Jan 2016 09:34:37 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aHJbh-0002vI-06; Fri, 08 Jan 2016 10:04:37 +1100 Date: Fri, 8 Jan 2016 10:04:36 +1100 From: Dave Chinner To: tore.larsen@dolphingeo.com Cc: xfs@oss.sgi.com Subject: Re: Possible XFS or NFS bug in Latest RHEL 6.6 2.6.32-504.23.4.el6.x86_64 ? Message-ID: <20160107230436.GN21461@dastard> X-ASG-Orig-Subj: Re: Possible XFS or NFS bug in Latest RHEL 6.6 2.6.32-504.23.4.el6.x86_64 ? References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452208468 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25932 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Thu, Jan 07, 2016 at 09:54:43AM +0100, tore.larsen@dolphingeo.com wrote: > Hi XFS'ers, > > Anyone seen this? SGI MIS server. Been stable for months. > > Resembles https://lkml.org/lkml/2014/5/23/261 Hi Tore, This list is for developement and support of mainline XFS code base, not vendor-modified enterprise kernels. Please report the problem to your RHEL or SGI support contact so they can triage and solve your problem. Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Thu Jan 7 17:21:54 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6CF757F37 for ; Thu, 7 Jan 2016 17:21:54 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 334318F804B for ; Thu, 7 Jan 2016 15:21:54 -0800 (PST) X-ASG-Debug-ID: 1452208910-04cb6c49fb00340001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id CNhRj7lseFcYILak for ; Thu, 07 Jan 2016 15:21:51 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2D4EAD/8Y5WPGu7LXleKAECgw9+QYJig36BeaAlAQEBAQEBBotZhTqEBYYJAgIBAQKBI00BAQEBAQEHAQEBAUE/QRIBg2ABAQEDAScTHCMFCwgDDgoJJQ8FJQMHGhOIJwfCRgEBAQcCASAZhXWFR4QkhRgFlwuNTY8ERI4JgnUcgXEqNAGBeYIcgUsBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 08 Jan 2016 09:40:01 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aHJgu-0002w5-Fx; Fri, 08 Jan 2016 10:10:00 +1100 Date: Fri, 8 Jan 2016 10:10:00 +1100 From: Dave Chinner To: Jan Kara Cc: Dan Williams , Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Subject: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() Message-ID: <20160107231000.GO21461@dastard> X-ASG-Orig-Subj: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-2-git-send-email-ross.zwisler@linux.intel.com> <20160107093402.GA8380@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107093402.GA8380@quack.suse.cz> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452208910 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25933 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Thu, Jan 07, 2016 at 10:34:02AM +0100, Jan Kara wrote: > On Wed 06-01-16 11:14:09, Dan Williams wrote: > > On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler > > wrote: > > > __dax_dbg() currently assumes that bh->b_bdev is non-NULL, passing it into > > > bdevname() where is is dereferenced. This assumption isn't always true - > > > when called for reads of holes, ext4_dax_mmap_get_block() returns a buffer > > > head where bh->b_bdev is never set. I hit this BUG while testing the DAX > > > PMD fault path. > > > > > > Instead, verify that we have a valid bh->b_bdev, else just say "unknown" > > > for the block device. > > > > > > Signed-off-by: Ross Zwisler > > > Cc: Dan Williams > > > --- > > > fs/dax.c | 7 ++++++- > > > 1 file changed, 6 insertions(+), 1 deletion(-) > > > > > > diff --git a/fs/dax.c b/fs/dax.c > > > index 7af8797..03cc4a3 100644 > > > --- a/fs/dax.c > > > +++ b/fs/dax.c > > > @@ -563,7 +563,12 @@ static void __dax_dbg(struct buffer_head *bh, unsigned long address, > > > { > > > if (bh) { > > > char bname[BDEVNAME_SIZE]; > > > - bdevname(bh->b_bdev, bname); > > > + > > > + if (bh->b_bdev) > > > + bdevname(bh->b_bdev, bname); > > > + else > > > + snprintf(bname, BDEVNAME_SIZE, "unknown"); > > > + > > > pr_debug("%s: %s addr: %lx dev %s state %lx start %lld " > > > "length %zd fallback: %s\n", fn, current->comm, > > > address, bname, bh->b_state, (u64)bh->b_blocknr, > > > > I'm assuming there's no danger of a such a buffer_head ever being used > > for the bdev parameter to dax_map_atomic()? Shouldn't we also/instead > > go fix ext4 to not send partially filled buffer_heads? > > No. The real problem is a long-standing abuse of struct buffer_head to be > used for passing block mapping information (it's on my todo list to remove > that at least from DAX code and use cleaner block mapping interface but > first I want basic DAX functionality to settle down to avoid unnecessary > conflicts). Filesystem is not supposed to touch bh->b_bdev. That has not been true for a long, long time. e.g. XFS always rewrites bh->b_bdev in get_blocks because the file may not reside on the primary block device of the filesystem. i.e.: /* * If this is a realtime file, data may be on a different device. * to that pointed to from the buffer_head b_bdev currently. */ bh_result->b_bdev = xfs_find_bdev_for_inode(inode); > If you need > that filled in, set it yourself in before passing bh to the block mapping > function. That may be true, but we cannot assume that the bdev coming back out of get_block is the same one that was passed in. Cheers, Dave. -- Dave Chinner david@fromorbit.com From ross.zwisler@linux.intel.com Thu Jan 7 17:39:53 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 518347F37 for ; Thu, 7 Jan 2016 17:39:53 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1E3FC304039 for ; Thu, 7 Jan 2016 15:39:49 -0800 (PST) X-ASG-Debug-ID: 1452209988-04bdf03f0100930001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id Azm0FmRmzzu7dnX1 for ; Thu, 07 Jan 2016 15:39:48 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 15:39:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,535,1444719600"; d="scan'208";a="886104056" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.201.208]) by orsmga002.jf.intel.com with ESMTP; 07 Jan 2016 15:39:46 -0800 Date: Thu, 7 Jan 2016 16:39:46 -0700 From: Ross Zwisler To: Dave Chinner Cc: Jan Kara , Dan Williams , Ross Zwisler , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Subject: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() Message-ID: <20160107233946.GD20802@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v7 1/9] dax: fix NULL pointer dereference in __dax_dbg() Mail-Followup-To: Ross Zwisler , Dave Chinner , Jan Kara , Dan Williams , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-2-git-send-email-ross.zwisler@linux.intel.com> <20160107093402.GA8380@quack.suse.cz> <20160107231000.GO21461@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107231000.GO21461@dastard> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452209988 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Fri, Jan 08, 2016 at 10:10:00AM +1100, Dave Chinner wrote: > On Thu, Jan 07, 2016 at 10:34:02AM +0100, Jan Kara wrote: > > On Wed 06-01-16 11:14:09, Dan Williams wrote: > > > On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler > > > wrote: > > > > __dax_dbg() currently assumes that bh->b_bdev is non-NULL, passing it into > > > > bdevname() where is is dereferenced. This assumption isn't always true - > > > > when called for reads of holes, ext4_dax_mmap_get_block() returns a buffer > > > > head where bh->b_bdev is never set. I hit this BUG while testing the DAX > > > > PMD fault path. > > > > > > > > Instead, verify that we have a valid bh->b_bdev, else just say "unknown" > > > > for the block device. > > > > > > > > Signed-off-by: Ross Zwisler > > > > Cc: Dan Williams > > > > --- > > > > fs/dax.c | 7 ++++++- > > > > 1 file changed, 6 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/fs/dax.c b/fs/dax.c > > > > index 7af8797..03cc4a3 100644 > > > > --- a/fs/dax.c > > > > +++ b/fs/dax.c > > > > @@ -563,7 +563,12 @@ static void __dax_dbg(struct buffer_head *bh, unsigned long address, > > > > { > > > > if (bh) { > > > > char bname[BDEVNAME_SIZE]; > > > > - bdevname(bh->b_bdev, bname); > > > > + > > > > + if (bh->b_bdev) > > > > + bdevname(bh->b_bdev, bname); > > > > + else > > > > + snprintf(bname, BDEVNAME_SIZE, "unknown"); > > > > + > > > > pr_debug("%s: %s addr: %lx dev %s state %lx start %lld " > > > > "length %zd fallback: %s\n", fn, current->comm, > > > > address, bname, bh->b_state, (u64)bh->b_blocknr, > > > > > > I'm assuming there's no danger of a such a buffer_head ever being used > > > for the bdev parameter to dax_map_atomic()? Shouldn't we also/instead > > > go fix ext4 to not send partially filled buffer_heads? > > > > No. The real problem is a long-standing abuse of struct buffer_head to be > > used for passing block mapping information (it's on my todo list to remove > > that at least from DAX code and use cleaner block mapping interface but > > first I want basic DAX functionality to settle down to avoid unnecessary > > conflicts). Filesystem is not supposed to touch bh->b_bdev. > > That has not been true for a long, long time. e.g. XFS always > rewrites bh->b_bdev in get_blocks because the file may not reside on > the primary block device of the filesystem. i.e.: > > /* > * If this is a realtime file, data may be on a different device. > * to that pointed to from the buffer_head b_bdev currently. > */ > bh_result->b_bdev = xfs_find_bdev_for_inode(inode); > > > If you need > > that filled in, set it yourself in before passing bh to the block mapping > > function. > > That may be true, but we cannot assume that the bdev coming back > out of get_block is the same one that was passed in. For our use case I think this is fine - we just need the bdev to be filled in so that we can print reasonable error messages. If the filesystem updates bh->b_bdev during get_blocks(), we are fine with that. From david@fromorbit.com Thu Jan 7 17:56:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id AB7897F37 for ; Thu, 7 Jan 2016 17:56:03 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 9217A304039 for ; Thu, 7 Jan 2016 15:56:03 -0800 (PST) X-ASG-Debug-ID: 1452210959-04cb6c49fa00bf0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 7mGBq33dfqlllj5j for ; Thu, 07 Jan 2016 15:56:00 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AJDQAg+o5WPGu7LXleKAECgw+BP4Jig36BeaAlAQEBAQEBBotZhTqEBYYJAgIBAQKBI00BAQEBAQEHAQEBAUE/QRIBg2ABAQEDAScTHCMFCwgDDgcDCSUPBSUDBxoTiCcHwkMBAQgCASAZhXWFR4k8BZMKhAGINIUZgWWHbYUyjk2CdRyBcSo0gXqDZwEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 08 Jan 2016 10:25:59 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aHKPO-00032f-Nu; Fri, 08 Jan 2016 10:55:58 +1100 Date: Fri, 8 Jan 2016 10:55:58 +1100 From: Dave Chinner To: Brian Foster Cc: xfs@oss.sgi.com Subject: Re: [PATCH v2] xfs: handle dquot buffer readahead in log recovery correctly Message-ID: <20160107235558.GP21461@dastard> X-ASG-Orig-Subj: Re: [PATCH v2] xfs: handle dquot buffer readahead in log recovery correctly References: <1452052834-20605-1-git-send-email-david@fromorbit.com> <20160107030830.GA32200@dastard> <20160107124433.GA33327@bfoster.bfoster> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107124433.GA33327@bfoster.bfoster> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452210959 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25934 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header [slightly hacked up quoting order so it makes sense] On Thu, Jan 07, 2016 at 07:44:33AM -0500, Brian Foster wrote: > On Thu, Jan 07, 2016 at 02:08:30PM +1100, Dave Chinner wrote: > > > > From: Dave Chinner > > > > When we do dquot readahead in log recovery, we do not use a verifier ...... > > write operation as well as a read operation that marks the buffer as > > not done if any corruption is detected. Also make sure we don't run Marking the buffer not done mentioned here... ..... > > > > /* > > + * readahead errors are silent and simply leave the buffer as !done so a real > > + * read will then be run with the xfs_dquot_buf_ops verifier. See > > + * xfs_inode_buf_verify() for why we use EIO and ~XBF_DONE here rather than > > + * reporting the failure. ... and here ... > > @@ -62,11 +62,14 @@ xfs_inobp_check( > > * has not had the inode cores stamped into it. Hence for readahead, the buffer > > * may be potentially invalid. > > * > > - * If the readahead buffer is invalid, we don't want to mark it with an error, > > - * but we do want to clear the DONE status of the buffer so that a followup read > > - * will re-read it from disk. This will ensure that we don't get an unnecessary > > - * warnings during log recovery and we don't get unnecssary panics on debug > > - * kernels. > > + * If the readahead buffer is invalid, we need to mark it with an error and > > + * clear the DONE status of the buffer so that a followup read will re-read it > > + * from disk. We don't report the error otherwise to avoid warnings during log ... and here ... > Do we really need to clear the flag if the I/O infrastructure doesn't > set it until after the verifier is invoked? It's harmless, [...] > ... but it does look slightly confusing, without a mention in the > comments at least. I mentioned it 3 times. ;) Is there something I can do to make it more obvious what the comments are referring to? > > + */ > > +static void > > +xfs_dquot_buf_readahead_verify( > > + struct xfs_buf *bp) > > +{ > > + struct xfs_mount *mp = bp->b_target->bt_mount; > > + > > + if (!xfs_dquot_buf_verify_crc(mp, bp) || > > + !xfs_dquot_buf_verify(mp, bp, 0)) { > > + xfs_buf_ioerror(bp, -EIO); > > + bp->b_flags &= ~XBF_DONE; > > Do we really need to clear the flag if the I/O infrastructure doesn't > set it until after the verifier is invoked? It's harmless, so if the > intent is to just be cautious or future-proof: It's intended to cover the case that a verifier is run on a buffer that has already been read in successfully. This doesn't occur in the kernel code, but it most definitely does in the userspace code. Strictly speaking it's not necessary (at the moment) for the userspace code as it ignores the kernel buffer flags, but I figured it's better to leave it there for documentation of expected behaviour and to ensure we don't leave a landmine if we ever run a readahead verifier on a buffer that is already marked as done... > Reviewed-by: Brian Foster Thanks! Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Thu Jan 7 18:16:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id C8E887F37 for ; Thu, 7 Jan 2016 18:16:10 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 8EDC7304039 for ; Thu, 7 Jan 2016 16:16:10 -0800 (PST) X-ASG-Debug-ID: 1452212165-04bdf03f0101250001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id iZMKjJzLKstfHXZG for ; Thu, 07 Jan 2016 16:16:05 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CJDAAH/45WPGu7LXleKAECgw+BP4Jig36BeaAlAQEBAQEBBotZhTqEBYYJBAICgSNNAQEBAQEBBwEBAQFBP0ESAYNhAQEEOhwjEAgDGAklDwUlAwcaExmIFcJDAQshGYV1hUeJPAEElwuNTY8ERI4JhQIqNIF6g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 08 Jan 2016 10:45:45 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aHKiW-000362-DD; Fri, 08 Jan 2016 11:15:44 +1100 Date: Fri, 8 Jan 2016 11:15:44 +1100 From: Dave Chinner To: Dan Williams Cc: xfs@oss.sgi.com, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Jens Axboe , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler Subject: Re: [PATCH v2 2/4] block: introduce del_gendisk_queue() Message-ID: <20160108001544.GQ21461@dastard> X-ASG-Orig-Subj: Re: [PATCH v2 2/4] block: introduce del_gendisk_queue() References: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> <20160106045627.38788.90127.stgit@dwillia2-desk3.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160106045627.38788.90127.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452212165 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25935 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 08:56:27PM -0800, Dan Williams wrote: > Historically we have waited for filesystem specific heuristics to > attempt to guess when a block device is gone. Sometimes this works, but > in other cases the system can hang waiting for the fs to trigger its > shutdown protocol. > > The initial motivation for this investigation was to prevent DAX > mappings (direct mmap access to persistent memory) from leaking past the > lifetime of the hosting block device. However, Dave points out that > these shutdown operations are needed in other scenarios. Quoting Dave: > > For example, if we detect a free space corruption during allocation, > it is not safe to trust *any active mapping* because we can't trust > that we having handed out the same block to multiple owners. Hence > on such a filesystem shutdown, we have to prevent any new DAX > mapping from occurring and invalidate all existing mappings as we > cannot allow userspace to modify any data or metadata until we've > resolved the corruption situation. > > The current block device shutdown sequence of del_gendisk + > blk_cleanup_queue is problematic. We want to tell the fs after > blk_cleanup_queue that there is no possibility of recovery, but by that > time we have deleted partitions and lost the ability to find all the > super-blocks on a block device. > > del_gendisk_queue() combines block device shutdown, blk_cleanup_queue(), > with block device end of life notification, del_gendisk(). A later > patch builds on this sequence to additionally communicate to the fs that > it should force-fail all future i/o since the queue is permanently dead. This still is two changes in one. Adding the force failure feature is a separate change to creating del_gendisk_queue(). Cheers, Dave. -- Dave Chinner david@fromorbit.com From david@fromorbit.com Thu Jan 7 18:16:43 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 7F1477F37 for ; Thu, 7 Jan 2016 18:16:43 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 67D25304039 for ; Thu, 7 Jan 2016 16:16:43 -0800 (PST) X-ASG-Debug-ID: 1452212201-04cbb0301401230001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id IqYghfeuCKLMy2b9 for ; Thu, 07 Jan 2016 16:16:41 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AIDQAH/45WPGu7LXleKAECgw+BP4Jig36BeaAlAQEBAQEBBotZhTqEBYYJAgIBAQKBI00BAQEBAQEHAQEBAUE/QRIBg2EBAQQnExwjEAgDGAklDwUlAwcaE4guwkMBCwEgGYV1hUeJPAEEkwqEAY1NjwREjgmFAio0AYF5g2cBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 08 Jan 2016 10:46:40 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aHKjP-00036D-Ok; Fri, 08 Jan 2016 11:16:39 +1100 Date: Fri, 8 Jan 2016 11:16:39 +1100 From: Dave Chinner To: Dan Williams Cc: xfs@oss.sgi.com, linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org Subject: Re: [PATCH v2 3/4] xfs: unmap dax at shutdown (force_failure) Message-ID: <20160108001639.GR21461@dastard> X-ASG-Orig-Subj: Re: [PATCH v2 3/4] xfs: unmap dax at shutdown (force_failure) References: <20160106045616.38788.61076.stgit@dwillia2-desk3.amr.corp.intel.com> <20160106045632.38788.84927.stgit@dwillia2-desk3.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160106045632.38788.84927.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452212201 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25935 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, Jan 05, 2016 at 08:56:32PM -0800, Dan Williams wrote: > When an exceptional event triggers xfs_force_shutdown() tear down dax > mappings. Quoting Dave, > > "The simple fact is that a /filesystem/ shutdown needs to do DAX > mapping invalidation regardless of whether the block device has > been unplugged or not. This is not a case of "this only happens > when we unplug the device", this is a user data protection > mechanism that we use to prevent corruption propagation once it > has been detected. A device unplug is just one type of > "corruption" that can occur." > > Signed-off-by: Dan Williams > --- > fs/xfs/xfs_fsops.c | 9 +++++++++ > 1 file changed, 9 insertions(+) > > diff --git a/fs/xfs/xfs_fsops.c b/fs/xfs/xfs_fsops.c > index ee3aaa0a5317..0c6a52809dcc 100644 > --- a/fs/xfs/xfs_fsops.c > +++ b/fs/xfs/xfs_fsops.c > @@ -828,6 +828,15 @@ xfs_do_force_shutdown( > if (xfs_log_force_umount(mp, logerror)) > return; > > + /* > + * If DAX is in use, we have to unmap all direct access virtual > + * mappings to ensure nothing more gets written directly from > + * userspace. This will force them to refault and that will > + * result in them detecting the shutdown condition and hence > + * will fail appropriately. > + */ > + unmap_dax_inodes(mp->m_super); > + > if (flags & SHUTDOWN_CORRUPT_INCORE) { > xfs_alert_tag(mp, XFS_PTAG_SHUTDOWN_CORRUPT, > "Corruption of in-memory data detected. Shutting down filesystem"); Looks fine. Acked-by: Dave Chinner -- Dave Chinner david@fromorbit.com From dan.j.williams@intel.com Thu Jan 7 18:43:22 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 7493F7F37 for ; Thu, 7 Jan 2016 18:43:22 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 530DB304032 for ; Thu, 7 Jan 2016 16:43:22 -0800 (PST) X-ASG-Debug-ID: 1452213800-04cbb0301601890001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id 8iPqfiYfZjR0NFAH for ; Thu, 07 Jan 2016 16:43:20 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 16:43:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,535,1444719600"; d="scan'208";a="888713312" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by fmsmga002.fm.intel.com with ESMTP; 07 Jan 2016 16:43:22 -0800 Subject: [PATCH v3 0/5] fs, block: handle end of life From: Dan Williams X-ASG-Orig-Subj: [PATCH v3 0/5] fs, block: handle end of life To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , Dave Chinner , linux-kernel@vger.kernel.org, Jens Axboe , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler Date: Thu, 07 Jan 2016 16:42:54 -0800 Message-ID: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452213800 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Changes since v2 [1]: 1/ Split "block: introduce del_gendisk_queue()" into a patch that introduces del_gendisk_queue() with no functional difference to the open coded version and "block: introduce force_failure_partition() and unmap_dax_inodes()" which adds the new behavior. 2/ Collect Dave's ack on "xfs: unmap dax at shutdown (force_failure)" --- As mentioned in [PATCH v3 3/5] "block: introduce force_failure_partition() and unmap_dax_inodes()" historically we have waited for filesystem specific heuristics to attempt to guess when a block device is gone. Sometimes this works, but in other cases the system can hang waiting for the fs to trigger its shutdown protocol. Now with DAX we need new actions, like unmapping all inodes, to be taken upon a device loss event or fs corruption event. For now, the approach taken in the following patches only affects xfs and block drivers that are converted to use del_gendisk_queue(). We can add more filesystems and driver support over time. --- Dan Williams (5): block: prepare for del_gendisk_queue() block: introduce del_gendisk_queue() block: introduce force_failure_partition() and unmap_dax_inodes() xfs: unmap dax at shutdown (force_failure) block, xfs: implement 'force_failure' notifications block/genhd.c | 87 +++++++++++++++++++++++++++++++++++------- drivers/block/brd.c | 9 +--- drivers/nvdimm/pmem.c | 3 - drivers/s390/block/dcssblk.c | 6 +-- fs/block_dev.c | 22 +++++++++++ fs/inode.c | 28 ++++++++++++++ fs/xfs/xfs_fsops.c | 9 ++++ fs/xfs/xfs_super.c | 8 ++++ include/linux/fs.h | 3 + include/linux/genhd.h | 1 10 files changed, 150 insertions(+), 26 deletions(-) From dan.j.williams@intel.com Thu Jan 7 18:43:51 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 355897F37 for ; Thu, 7 Jan 2016 18:43:51 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id C5D74AC001 for ; Thu, 7 Jan 2016 16:43:47 -0800 (PST) X-ASG-Debug-ID: 1452213826-04bdf03f00019d0001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id ff3xkHS7n7WWINRj for ; Thu, 07 Jan 2016 16:43:46 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 16:43:46 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,535,1444719600"; d="scan'208";a="722518787" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by orsmga003.jf.intel.com with ESMTP; 07 Jan 2016 16:43:46 -0800 Subject: [PATCH v3 4/5] xfs: unmap dax at shutdown (force_failure) From: Dan Williams X-ASG-Orig-Subj: [PATCH v3 4/5] xfs: unmap dax at shutdown (force_failure) To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org, Dave Chinner Date: Thu, 07 Jan 2016 16:43:21 -0800 Message-ID: <20160108004321.36061.17150.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452213826 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 When an exceptional event triggers xfs_force_shutdown() tear down dax mappings. Quoting Dave, "The simple fact is that a /filesystem/ shutdown needs to do DAX mapping invalidation regardless of whether the block device has been unplugged or not. This is not a case of "this only happens when we unplug the device", this is a user data protection mechanism that we use to prevent corruption propagation once it has been detected. A device unplug is just one type of "corruption" that can occur." Acked-by: Dave Chinner Signed-off-by: Dan Williams --- fs/xfs/xfs_fsops.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/fs/xfs/xfs_fsops.c b/fs/xfs/xfs_fsops.c index ee3aaa0a5317..0c6a52809dcc 100644 --- a/fs/xfs/xfs_fsops.c +++ b/fs/xfs/xfs_fsops.c @@ -828,6 +828,15 @@ xfs_do_force_shutdown( if (xfs_log_force_umount(mp, logerror)) return; + /* + * If DAX is in use, we have to unmap all direct access virtual + * mappings to ensure nothing more gets written directly from + * userspace. This will force them to refault and that will + * result in them detecting the shutdown condition and hence + * will fail appropriately. + */ + unmap_dax_inodes(mp->m_super); + if (flags & SHUTDOWN_CORRUPT_INCORE) { xfs_alert_tag(mp, XFS_PTAG_SHUTDOWN_CORRUPT, "Corruption of in-memory data detected. Shutting down filesystem"); From dan.j.williams@intel.com Thu Jan 7 18:43:55 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 59AD77F53 for ; Thu, 7 Jan 2016 18:43:55 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id E00F0AC001 for ; Thu, 7 Jan 2016 16:43:54 -0800 (PST) X-ASG-Debug-ID: 1452213832-04cbb03014018a0001-NocioJ Received: from mga04.intel.com ([192.55.52.120]) by cuda.sgi.com with ESMTP id 5AHzVcyVFxlpz7IA for ; Thu, 07 Jan 2016 16:43:52 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.120 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga104.fm.intel.com with ESMTP; 07 Jan 2016 16:43:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,535,1444719600"; d="scan'208";a="888714071" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by fmsmga002.fm.intel.com with ESMTP; 07 Jan 2016 16:43:53 -0800 Subject: [PATCH v3 5/5] block, xfs: implement 'force_failure' notifications From: Dan Williams X-ASG-Orig-Subj: [PATCH v3 5/5] block, xfs: implement 'force_failure' notifications To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , linux-kernel@vger.kernel.org, Jens Axboe , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org Date: Thu, 07 Jan 2016 16:43:26 -0800 Message-ID: <20160108004326.36061.50916.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: UNKNOWN[192.55.52.120] X-Barracuda-Start-Time: 1452213832 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25936 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 Introduce a new super operation, 'force_failure', that is invoked by force_failure_partition() when the block device is dead. This unambiguously communicates to a filesystem that i/o errors are permanent and no recovery effort will succeed. 'force_failure' simply becomes another exceptional event that can trigger xfs_force_shutdown(). Cc: Jan Kara Cc: Jens Axboe Cc: Alexander Viro Suggested-by: Dave Chinner Signed-off-by: Dan Williams --- block/genhd.c | 2 +- fs/block_dev.c | 5 ++++- fs/xfs/xfs_super.c | 8 ++++++++ include/linux/fs.h | 1 + 4 files changed, 14 insertions(+), 2 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index ac0d12c4f895..45f9f123013b 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -717,7 +717,7 @@ void del_gendisk_queue(struct gendisk *disk) blk_cleanup_queue(disk->queue); - /* pass2 the queue is dead, halt dax */ + /* pass2 the queue is dead, halt dax, and halt fs operations */ disk_part_iter_init(&piter, disk, DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); for_each_part(part, &piter) { diff --git a/fs/block_dev.c b/fs/block_dev.c index 9cff33b6baab..a9c07910481c 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1812,7 +1812,10 @@ void force_failure_partition(struct gendisk *disk, int partno) if (!sb) goto out; - unmap_dax_inodes(sb); + if (sb->s_op->force_failure) + sb->s_op->force_failure(sb); + else + unmap_dax_inodes(sb); drop_super(sb); out: bdput(bdev); diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index 36bd8825bfb0..e1113ac2e342 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -1618,6 +1618,13 @@ xfs_fs_free_cached_objects( return xfs_reclaim_inodes_nr(XFS_M(sb), sc->nr_to_scan); } +static void +xfs_fs_force_failure( + struct super_block *sb) +{ + xfs_force_shutdown(XFS_M(sb), SHUTDOWN_DEVICE_REQ); +} + static const struct super_operations xfs_super_operations = { .alloc_inode = xfs_fs_alloc_inode, .destroy_inode = xfs_fs_destroy_inode, @@ -1632,6 +1639,7 @@ static const struct super_operations xfs_super_operations = { .show_options = xfs_fs_show_options, .nr_cached_objects = xfs_fs_nr_cached_objects, .free_cached_objects = xfs_fs_free_cached_objects, + .force_failure = xfs_fs_force_failure, }; static struct file_system_type xfs_fs_type = { diff --git a/include/linux/fs.h b/include/linux/fs.h index a0d55199e628..bfd9bb7b529d 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1713,6 +1713,7 @@ struct super_operations { struct shrink_control *); long (*free_cached_objects)(struct super_block *, struct shrink_control *); + void (*force_failure)(struct super_block *); }; /* From dan.j.williams@intel.com Thu Jan 7 18:44:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 4AA4D7F56 for ; Thu, 7 Jan 2016 18:44:00 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 2B4FC8F8052 for ; Thu, 7 Jan 2016 16:44:00 -0800 (PST) X-ASG-Debug-ID: 1452213838-04cbb03016018b0001-NocioJ Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id HQMApIlFEZ3px6yx for ; Thu, 07 Jan 2016 16:43:58 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.24 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP; 07 Jan 2016 16:43:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,535,1444719600"; d="scan'208";a="876914553" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by fmsmga001.fm.intel.com with ESMTP; 07 Jan 2016 16:43:31 -0800 Subject: [PATCH v3 2/5] block: introduce del_gendisk_queue() From: Dan Williams X-ASG-Orig-Subj: [PATCH v3 2/5] block: introduce del_gendisk_queue() To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Date: Thu, 07 Jan 2016 16:43:05 -0800 Message-ID: <20160108004305.36061.31898.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga09.intel.com[134.134.136.24] X-Barracuda-Start-Time: 1452213838 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 The current block device shutdown sequence of del_gendisk + blk_cleanup_queue is problematic. We want to tell the fs after blk_cleanup_queue that there is no possibility of recovery, but by that time we have deleted partitions and lost the ability to find all the super-blocks on a block device. del_gendisk_queue() combines block device shutdown, blk_cleanup_queue(), with block device end of life notification, del_gendisk(). Later patches builds on this sequence to unmap all dax inodes and communicate to the fs that it should force-fail all future i/o since the queue is permanently dead. For now this is routine is functionally equivalent to calling del_gendisk() + blk_cleanup_queue() in succession. Signed-off-by: Dan Williams --- block/genhd.c | 43 ++++++++++++++++++++++++++++++++++++++++++ drivers/block/brd.c | 9 +++------ drivers/nvdimm/pmem.c | 3 +-- drivers/s390/block/dcssblk.c | 6 ++---- include/linux/genhd.h | 1 + 5 files changed, 50 insertions(+), 12 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index b1d1df42ba13..a5bb768111cc 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -686,6 +686,49 @@ void del_gendisk(struct gendisk *disk) EXPORT_SYMBOL(del_gendisk); /** + * del_gendisk_queue - combined del_gendisk + blk_cleanup_queue + * @disk: disk to delete, invalidate, unmap, and force-fail fs operations + * + * This is an alternative for open coded calls to: + * del_gendisk() + * blk_cleanup_queue() + * It notifies filesystems / vfs that a block device is permanently dead + * after the queue has been torn down. This notification is needed for + * triggering a filesystem to abort its error recovery and for (DAX) + * capable devices. DAX bypasses page cache and mappings go directly to + * storage media. When such a disk is removed the pfn backing a mapping + * may be invalid or removed from the system. Upon return accessing DAX + * mappings of this disk will trigger SIGBUS. + */ +void del_gendisk_queue(struct gendisk *disk) +{ + struct disk_part_iter piter; + struct hd_struct *part; + + del_gendisk_start(disk); + + /* pass1 sync fs + evict idle inodes */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) + invalidate_partition(disk, part->partno); + disk_part_iter_exit(&piter); + invalidate_partition(disk, 0); + + blk_cleanup_queue(disk->queue); + + /* pass2 the queue is dead */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) + delete_partition(disk, part->partno); + disk_part_iter_exit(&piter); + + del_gendisk_end(disk); +} +EXPORT_SYMBOL(del_gendisk_queue); + +/** * get_gendisk - get partitioning information for a given device * @devt: device to get partitioning information for * @partno: returned partition index diff --git a/drivers/block/brd.c b/drivers/block/brd.c index a5880f4ab40e..013ff58f9af8 100644 --- a/drivers/block/brd.c +++ b/drivers/block/brd.c @@ -532,7 +532,6 @@ out: static void brd_free(struct brd_device *brd) { put_disk(brd->brd_disk); - blk_cleanup_queue(brd->brd_queue); brd_free_pages(brd); kfree(brd); } @@ -560,7 +559,7 @@ out: static void brd_del_one(struct brd_device *brd) { list_del(&brd->brd_list); - del_gendisk(brd->brd_disk); + del_gendisk_queue(brd->brd_disk); brd_free(brd); } @@ -626,10 +625,8 @@ static int __init brd_init(void) return 0; out_free: - list_for_each_entry_safe(brd, next, &brd_devices, brd_list) { - list_del(&brd->brd_list); - brd_free(brd); - } + list_for_each_entry_safe(brd, next, &brd_devices, brd_list) + brd_del_one(brd); unregister_blkdev(RAMDISK_MAJOR, "ramdisk"); pr_info("brd: module NOT loaded !!!\n"); diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 8ee79893d2f5..6dd06e9d34b0 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -158,9 +158,8 @@ static void pmem_detach_disk(struct pmem_device *pmem) if (!pmem->pmem_disk) return; - del_gendisk(pmem->pmem_disk); + del_gendisk_queue(pmem->pmem_disk); put_disk(pmem->pmem_disk); - blk_cleanup_queue(pmem->pmem_queue); } static int pmem_attach_disk(struct device *dev, diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c index 94a8f4ab57bc..0c3c968b57d9 100644 --- a/drivers/s390/block/dcssblk.c +++ b/drivers/s390/block/dcssblk.c @@ -388,8 +388,7 @@ removeseg: } list_del(&dev_info->lh); - del_gendisk(dev_info->gd); - blk_cleanup_queue(dev_info->dcssblk_queue); + del_gendisk_queue(dev_info->gd); dev_info->gd->queue = NULL; put_disk(dev_info->gd); up_write(&dcssblk_devices_sem); @@ -751,8 +750,7 @@ dcssblk_remove_store(struct device *dev, struct device_attribute *attr, const ch } list_del(&dev_info->lh); - del_gendisk(dev_info->gd); - blk_cleanup_queue(dev_info->dcssblk_queue); + del_gendisk_queue(dev_info->gd); dev_info->gd->queue = NULL; put_disk(dev_info->gd); device_unregister(&dev_info->dev); diff --git a/include/linux/genhd.h b/include/linux/genhd.h index 847cc1d91634..028cf15a8a57 100644 --- a/include/linux/genhd.h +++ b/include/linux/genhd.h @@ -431,6 +431,7 @@ extern void part_round_stats(int cpu, struct hd_struct *part); /* block/genhd.c */ extern void add_disk(struct gendisk *disk); extern void del_gendisk(struct gendisk *gp); +extern void del_gendisk_queue(struct gendisk *disk); extern struct gendisk *get_gendisk(dev_t dev, int *partno); extern struct block_device *bdget_disk(struct gendisk *disk, int partno); From dan.j.williams@intel.com Thu Jan 7 18:44:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 511BE7F5A for ; Thu, 7 Jan 2016 18:44:03 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id CD028AC004 for ; Thu, 7 Jan 2016 16:44:02 -0800 (PST) X-ASG-Debug-ID: 1452213838-04cbb03016018b0002-NocioJ Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id AEteS8GDNxkkBR17 for ; Thu, 07 Jan 2016 16:44:00 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.24 X-ASG-Whitelist: EmailCat (corporate) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP; 07 Jan 2016 16:43:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,535,1444719600"; d="scan'208";a="856002858" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by orsmga001.jf.intel.com with ESMTP; 07 Jan 2016 16:43:41 -0800 Subject: [PATCH v3 3/5] block: introduce force_failure_partition() and unmap_dax_inodes() From: Dan Williams X-ASG-Orig-Subj: [PATCH v3 3/5] block: introduce force_failure_partition() and unmap_dax_inodes() To: xfs@oss.sgi.com Cc: linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, Dave Chinner , linux-kernel@vger.kernel.org, Jens Axboe , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler Date: Thu, 07 Jan 2016 16:43:15 -0800 Message-ID: <20160108004310.36061.4099.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga09.intel.com[134.134.136.24] X-Barracuda-Start-Time: 1452213839 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Historically we have waited for filesystem specific heuristics to attempt to guess when a block device is gone. Sometimes this works, but in other cases the system can hang waiting for the fs to trigger its shutdown protocol. The initial motivation for this investigation was to prevent DAX mappings (direct mmap access to persistent memory) from leaking past the lifetime of the hosting block device. However, Dave points out that these shutdown operations are needed in other scenarios. Quoting Dave: For example, if we detect a free space corruption during allocation, it is not safe to trust *any active mapping* because we can't trust that we having handed out the same block to multiple owners. Hence on such a filesystem shutdown, we have to prevent any new DAX mapping from occurring and invalidate all existing mappings as we cannot allow userspace to modify any data or metadata until we've resolved the corruption situation. Cc: Jan Kara Cc: Jens Axboe Cc: Matthew Wilcox Cc: Ross Zwisler Suggested-by: Dave Chinner Signed-off-by: Dan Williams --- block/genhd.c | 7 +++++-- fs/block_dev.c | 19 +++++++++++++++++++ fs/inode.c | 28 ++++++++++++++++++++++++++++ include/linux/fs.h | 2 ++ 4 files changed, 54 insertions(+), 2 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index a5bb768111cc..ac0d12c4f895 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -717,12 +717,15 @@ void del_gendisk_queue(struct gendisk *disk) blk_cleanup_queue(disk->queue); - /* pass2 the queue is dead */ + /* pass2 the queue is dead, halt dax */ disk_part_iter_init(&piter, disk, DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); - for_each_part(part, &piter) + for_each_part(part, &piter) { + force_failure_partition(disk, part->partno); delete_partition(disk, part->partno); + } disk_part_iter_exit(&piter); + force_failure_partition(disk, 0); del_gendisk_end(disk); } diff --git a/fs/block_dev.c b/fs/block_dev.c index 44d4a1e9244e..9cff33b6baab 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -1799,6 +1799,25 @@ int __invalidate_device(struct block_device *bdev, bool kill_dirty) } EXPORT_SYMBOL(__invalidate_device); +void force_failure_partition(struct gendisk *disk, int partno) +{ + struct block_device *bdev; + struct super_block *sb; + + bdev = bdget_disk(disk, partno); + if (!bdev) + return; + + sb = get_super(bdev); + if (!sb) + goto out; + + unmap_dax_inodes(sb); + drop_super(sb); + out: + bdput(bdev); +} + void iterate_bdevs(void (*func)(struct block_device *, void *), void *arg) { struct inode *inode, *old_inode = NULL; diff --git a/fs/inode.c b/fs/inode.c index 1be5f9003eb3..ed62e5f78f35 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -673,6 +673,34 @@ int invalidate_inodes(struct super_block *sb, bool kill_dirty) return busy; } +void unmap_dax_inodes(struct super_block *sb) +{ + struct inode *inode, *_inode = NULL; + + spin_lock(&sb->s_inode_list_lock); + list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { + spin_lock(&inode->i_lock); + if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) + || !IS_DAX(inode)) { + spin_unlock(&inode->i_lock); + continue; + } + __iget(inode); + spin_unlock(&inode->i_lock); + spin_unlock(&sb->s_inode_list_lock); + + unmap_mapping_range(inode->i_mapping, 0, 0, 1); + iput(_inode); + _inode = inode; + cond_resched(); + + spin_lock(&sb->s_inode_list_lock); + } + spin_unlock(&sb->s_inode_list_lock); + iput(_inode); +} +EXPORT_SYMBOL(unmap_dax_inodes); + /* * Isolate the inode from the LRU in preparation for freeing it. * diff --git a/include/linux/fs.h b/include/linux/fs.h index 3aa514254161..a0d55199e628 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2390,6 +2390,7 @@ extern int revalidate_disk(struct gendisk *); extern int check_disk_change(struct block_device *); extern int __invalidate_device(struct block_device *, bool); extern int invalidate_partition(struct gendisk *, int); +extern void force_failure_partition(struct gendisk *, int); #endif unsigned long invalidate_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t end); @@ -2544,6 +2545,7 @@ extern loff_t default_llseek(struct file *file, loff_t offset, int whence); extern loff_t vfs_llseek(struct file *file, loff_t offset, int whence); +extern void unmap_dax_inodes(struct super_block *sb); extern int inode_init_always(struct super_block *, struct inode *); extern void inode_init_once(struct inode *); extern void address_space_init_once(struct address_space *mapping); From dan.j.williams@intel.com Thu Jan 7 18:44:06 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4C8557F67 for ; Thu, 7 Jan 2016 18:44:06 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 39483304067 for ; Thu, 7 Jan 2016 16:44:06 -0800 (PST) X-ASG-Debug-ID: 1452213836-04cb6c49fd01980001-NocioJ Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id LjG0Vdlf7FxxCWsw for ; Thu, 07 Jan 2016 16:43:56 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.24 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP; 07 Jan 2016 16:43:26 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,535,1444719600"; d="scan'208";a="629845709" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.136]) by FMSMGA003.fm.intel.com with ESMTP; 07 Jan 2016 16:43:26 -0800 Subject: [PATCH v3 1/5] block: prepare for del_gendisk_queue() From: Dan Williams X-ASG-Orig-Subj: [PATCH v3 1/5] block: prepare for del_gendisk_queue() To: xfs@oss.sgi.com Cc: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Date: Thu, 07 Jan 2016 16:43:00 -0800 Message-ID: <20160108004259.36061.83529.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> References: <20160108004254.36061.32864.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Barracuda-Connect: mga09.intel.com[134.134.136.24] X-Barracuda-Start-Time: 1452213836 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Refactor del_gendisk() into del_gendisk_start() and del_gendisk_end(). These are common helpers that will be shared bewtween del_gendisk() and the to-be-introduced del_gendisk_queue(). Signed-off-by: Dan Williams --- block/genhd.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index e5cafa51567c..b1d1df42ba13 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -634,24 +634,14 @@ void add_disk(struct gendisk *disk) } EXPORT_SYMBOL(add_disk); -void del_gendisk(struct gendisk *disk) +static void del_gendisk_start(struct gendisk *disk) { - struct disk_part_iter piter; - struct hd_struct *part; - blk_integrity_del(disk); disk_del_events(disk); +} - /* invalidate stuff */ - disk_part_iter_init(&piter, disk, - DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); - while ((part = disk_part_iter_next(&piter))) { - invalidate_partition(disk, part->partno); - delete_partition(disk, part->partno); - } - disk_part_iter_exit(&piter); - - invalidate_partition(disk, 0); +static void del_gendisk_end(struct gendisk *disk) +{ set_capacity(disk, 0); disk->flags &= ~GENHD_FL_UP; @@ -670,6 +660,29 @@ void del_gendisk(struct gendisk *disk) pm_runtime_set_memalloc_noio(disk_to_dev(disk), false); device_del(disk_to_dev(disk)); } + +#define for_each_part(part, piter) \ + for (part = disk_part_iter_next(piter); part; \ + part = disk_part_iter_next(piter)) +void del_gendisk(struct gendisk *disk) +{ + struct disk_part_iter piter; + struct hd_struct *part; + + del_gendisk_start(disk); + + /* invalidate stuff */ + disk_part_iter_init(&piter, disk, + DISK_PITER_INCL_EMPTY | DISK_PITER_REVERSE); + for_each_part(part, &piter) { + invalidate_partition(disk, part->partno); + delete_partition(disk, part->partno); + } + disk_part_iter_exit(&piter); + invalidate_partition(disk, 0); + + del_gendisk_end(disk); +} EXPORT_SYMBOL(del_gendisk); /** From ross.zwisler@linux.intel.com Thu Jan 7 22:18:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D450C7F37 for ; Thu, 7 Jan 2016 22:18:09 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 649D7AC002 for ; Thu, 7 Jan 2016 20:18:06 -0800 (PST) X-ASG-Debug-ID: 1452226683-04cb6c75dd02f80001-NocioJ Received: from mga04.intel.com ([192.55.52.120]) by cuda.sgi.com with ESMTP id eYjJVbsaQOmm8ASa for ; Thu, 07 Jan 2016 20:18:04 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.120 Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga104.fm.intel.com with ESMTP; 07 Jan 2016 20:18:03 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="722698857" Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.201.208]) by orsmga003.jf.intel.com with ESMTP; 07 Jan 2016 20:18:02 -0800 Date: Thu, 7 Jan 2016 21:18:02 -0700 From: Ross Zwisler To: Ross Zwisler , Dan Williams , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers Subject: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Message-ID: <20160108041802.GA568@linux.intel.com> X-ASG-Orig-Subj: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Mail-Followup-To: Ross Zwisler , Dan Williams , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4 , linux-fsdevel , Linux MM , "linux-nvdimm@lists.01.org" , X86 ML , XFS Developers References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> <20160107223455.GC20802@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107223455.GC20802@linux.intel.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: UNKNOWN[192.55.52.120] X-Barracuda-Start-Time: 1452226683 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_ADDR_MATCH, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25942 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC0_SA_TO_FROM_ADDR_MATCH Sender Address Matches Recipient Address On Thu, Jan 07, 2016 at 03:34:55PM -0700, Ross Zwisler wrote: > On Wed, Jan 06, 2016 at 11:04:35AM -0800, Dan Williams wrote: > > On Wed, Jan 6, 2016 at 10:00 AM, Ross Zwisler > > wrote: > > > When we get a DAX PMD fault for a write it is possible that there could be > > > some number of 4k zero pages already present for the same range that were > > > inserted to service reads from a hole. These 4k zero pages need to be > > > unmapped from the VMAs and removed from the struct address_space radix tree > > > before the real DAX PMD entry can be inserted. > > > > > > For PTE faults this same use case also exists and is handled by a > > > combination of unmap_mapping_range() to unmap the VMAs and > > > delete_from_page_cache() to remove the page from the address_space radix > > > tree. > > > > > > For PMD faults we do have a call to unmap_mapping_range() (protected by a > > > buffer_new() check), but nothing clears out the radix tree entry. The > > > buffer_new() check is also incorrect as the current ext4 and XFS filesystem > > > code will never return a buffer_head with BH_New set, even when allocating > > > new blocks over a hole. Instead the filesystem will zero the blocks > > > manually and return a buffer_head with only BH_Mapped set. > > > > > > Fix this situation by removing the buffer_new() check and adding a call to > > > truncate_inode_pages_range() to clear out the radix tree entries before we > > > insert the DAX PMD. > > > > > > Signed-off-by: Ross Zwisler > > > > Replaced the current contents of v6 in -mm from next-20160106 with > > this v7 set and it looks good. > > > > Reported-by: Dan Williams > > Tested-by: Dan Williams > > > > One question below... > > > > > --- > > > fs/dax.c | 20 ++++++++++---------- > > > 1 file changed, 10 insertions(+), 10 deletions(-) > > > > > > diff --git a/fs/dax.c b/fs/dax.c > > > index 03cc4a3..9dc0c97 100644 > > > --- a/fs/dax.c > > > +++ b/fs/dax.c > > > @@ -594,6 +594,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > > > bool write = flags & FAULT_FLAG_WRITE; > > > struct block_device *bdev; > > > pgoff_t size, pgoff; > > > + loff_t lstart, lend; > > > sector_t block; > > > int result = 0; > > > > > > @@ -647,15 +648,13 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > > > goto fallback; > > > } > > > > > > - /* > > > - * If we allocated new storage, make sure no process has any > > > - * zero pages covering this hole > > > - */ > > > - if (buffer_new(&bh)) { > > > - i_mmap_unlock_read(mapping); > > > - unmap_mapping_range(mapping, pgoff << PAGE_SHIFT, PMD_SIZE, 0); > > > - i_mmap_lock_read(mapping); > > > - } > > > + /* make sure no process has any zero pages covering this hole */ > > > + lstart = pgoff << PAGE_SHIFT; > > > + lend = lstart + PMD_SIZE - 1; /* inclusive */ > > > + i_mmap_unlock_read(mapping); > > > + unmap_mapping_range(mapping, lstart, PMD_SIZE, 0); > > > + truncate_inode_pages_range(mapping, lstart, lend); > > > > Do we need to do both unmap and truncate given that > > truncate_inode_page() optionally does an unmap_mapping_range() > > internally? > > Ah, indeed it does. Sure, having just the call to truncate_inode_page() seems > cleaner. I'll re-test and send this out in v8. Actually, in testing it doesn't look like unmap_mapping_range() in truncate_inode_page() gets called. We fail the page_mapped(page) check for our read-only zero pages. I think we need to keep the unmap_mapping_range() call in __dax_pmd_fault(). From ross.zwisler@linux.intel.com Thu Jan 7 23:28:14 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6E8727F37 for ; Thu, 7 Jan 2016 23:28:14 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 68F1B8F8052 for ; Thu, 7 Jan 2016 21:28:14 -0800 (PST) X-ASG-Debug-ID: 1452230892-04cbb05f7f04900001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id uCW7pkkv3L7EbkBw for ; Thu, 07 Jan 2016 21:28:12 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25713947" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:10 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 0/9] DAX fsync/msync support Date: Thu, 7 Jan 2016 22:27:50 -0700 X-ASG-Orig-Subj: [PATCH v8 0/9] DAX fsync/msync support Message-Id: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230892 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Changes since v7 [1]: 1) Update patch 1 so that we initialize bh->b_bdev before passing it to get_block() instead of working around the fact that it could still be NULL after get_block() completes. (Dan) 2) Add a check to dax_radix_entry() so that we WARN_ON_ONCE() and exit gracefully if we find a page cache entry still in the radix tree when trying to insert a DAX entry. This series replaces v7 in the MM tree and in the "akpm" branch of the next tree. A working tree can be found here: https://git.kernel.org/cgit/linux/kernel/git/zwisler/linux.git/log/?h=fsync_v8 [1]: https://lists.01.org/pipermail/linux-nvdimm/2016-January/003886.html Ross Zwisler (9): dax: fix NULL pointer dereference in __dax_dbg() dax: fix conversion of holes to PMDs pmem: add wb_cache_pmem() to the PMEM API dax: support dirty DAX entries in radix tree mm: add find_get_entries_tag() dax: add support for fsync/msync ext2: call dax_pfn_mkwrite() for DAX fsync/msync ext4: call dax_pfn_mkwrite() for DAX fsync/msync xfs: call dax_pfn_mkwrite() for DAX fsync/msync arch/x86/include/asm/pmem.h | 11 +-- fs/block_dev.c | 2 +- fs/dax.c | 215 ++++++++++++++++++++++++++++++++++++++++---- fs/ext2/file.c | 4 +- fs/ext4/file.c | 4 +- fs/inode.c | 2 +- fs/xfs/xfs_file.c | 7 +- include/linux/dax.h | 7 ++ include/linux/fs.h | 3 +- include/linux/pagemap.h | 3 + include/linux/pmem.h | 22 ++++- include/linux/radix-tree.h | 9 ++ mm/filemap.c | 91 +++++++++++++++++-- mm/truncate.c | 69 +++++++------- mm/vmscan.c | 9 +- mm/workingset.c | 4 +- 16 files changed, 393 insertions(+), 69 deletions(-) -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:17 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 47A5F7F50 for ; Thu, 7 Jan 2016 23:28:17 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 3C5678F804C for ; Thu, 7 Jan 2016 21:28:17 -0800 (PST) X-ASG-Debug-ID: 1452230895-04cbb05f7d04900001-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id EAJExaBPmMk1cVJl for ; Thu, 07 Jan 2016 21:28:15 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25713957" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:14 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 2/9] dax: fix conversion of holes to PMDs Date: Thu, 7 Jan 2016 22:27:52 -0700 X-ASG-Orig-Subj: [PATCH v8 2/9] dax: fix conversion of holes to PMDs Message-Id: <1452230879-18117-3-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230895 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 When we get a DAX PMD fault for a write it is possible that there could be some number of 4k zero pages already present for the same range that were inserted to service reads from a hole. These 4k zero pages need to be unmapped from the VMAs and removed from the struct address_space radix tree before the real DAX PMD entry can be inserted. For PTE faults this same use case also exists and is handled by a combination of unmap_mapping_range() to unmap the VMAs and delete_from_page_cache() to remove the page from the address_space radix tree. For PMD faults we do have a call to unmap_mapping_range() (protected by a buffer_new() check), but nothing clears out the radix tree entry. The buffer_new() check is also incorrect as the current ext4 and XFS filesystem code will never return a buffer_head with BH_New set, even when allocating new blocks over a hole. Instead the filesystem will zero the blocks manually and return a buffer_head with only BH_Mapped set. Fix this situation by removing the buffer_new() check and adding a call to truncate_inode_pages_range() to clear out the radix tree entries before we insert the DAX PMD. Signed-off-by: Ross Zwisler Reported-by: Dan Williams Tested-by: Dan Williams --- fs/dax.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 513bba5..5b84a46 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -589,6 +589,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, bool write = flags & FAULT_FLAG_WRITE; struct block_device *bdev; pgoff_t size, pgoff; + loff_t lstart, lend; sector_t block; int result = 0; @@ -643,15 +644,13 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, goto fallback; } - /* - * If we allocated new storage, make sure no process has any - * zero pages covering this hole - */ - if (buffer_new(&bh)) { - i_mmap_unlock_read(mapping); - unmap_mapping_range(mapping, pgoff << PAGE_SHIFT, PMD_SIZE, 0); - i_mmap_lock_read(mapping); - } + /* make sure no process has any zero pages covering this hole */ + lstart = pgoff << PAGE_SHIFT; + lend = lstart + PMD_SIZE - 1; /* inclusive */ + i_mmap_unlock_read(mapping); + unmap_mapping_range(mapping, lstart, PMD_SIZE, 0); + truncate_inode_pages_range(mapping, lstart, lend); + i_mmap_lock_read(mapping); /* * If a truncate happened while we were allocating blocks, we may @@ -665,7 +664,8 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, goto out; } if ((pgoff | PG_PMD_COLOUR) >= size) { - dax_pmd_dbg(&bh, address, "pgoff unaligned"); + dax_pmd_dbg(&bh, address, + "offset + huge page size > file size"); goto fallback; } -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 7A19A7F50 for ; Thu, 7 Jan 2016 23:28:18 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 77FA98F8049 for ; Thu, 7 Jan 2016 21:28:18 -0800 (PST) X-ASG-Debug-ID: 1452230895-04cbb05f7d04900002-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id ZO4hYI8otEHqnJFP for ; Thu, 07 Jan 2016 21:28:17 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25713962" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:15 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 3/9] pmem: add wb_cache_pmem() to the PMEM API Date: Thu, 7 Jan 2016 22:27:53 -0700 X-ASG-Orig-Subj: [PATCH v8 3/9] pmem: add wb_cache_pmem() to the PMEM API Message-Id: <1452230879-18117-4-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230896 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 The function __arch_wb_cache_pmem() was already an internal implementation detail of the x86 PMEM API, but this functionality needs to be exported as part of the general PMEM API to handle the fsync/msync case for DAX mmaps. One thing worth noting is that we really do want this to be part of the PMEM API as opposed to a stand-alone function like clflush_cache_range() because of ordering restrictions. By having wb_cache_pmem() as part of the PMEM API we can leave it unordered, call it multiple times to write back large amounts of memory, and then order the multiple calls with a single wmb_pmem(). Signed-off-by: Ross Zwisler --- arch/x86/include/asm/pmem.h | 11 ++++++----- include/linux/pmem.h | 22 +++++++++++++++++++++- 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/pmem.h b/arch/x86/include/asm/pmem.h index 1544fab..c57fd1e 100644 --- a/arch/x86/include/asm/pmem.h +++ b/arch/x86/include/asm/pmem.h @@ -67,18 +67,19 @@ static inline void arch_wmb_pmem(void) } /** - * __arch_wb_cache_pmem - write back a cache range with CLWB + * arch_wb_cache_pmem - write back a cache range with CLWB * @vaddr: virtual start address * @size: number of bytes to write back * * Write back a cache range using the CLWB (cache line write back) * instruction. This function requires explicit ordering with an - * arch_wmb_pmem() call. This API is internal to the x86 PMEM implementation. + * arch_wmb_pmem() call. */ -static inline void __arch_wb_cache_pmem(void *vaddr, size_t size) +static inline void arch_wb_cache_pmem(void __pmem *addr, size_t size) { u16 x86_clflush_size = boot_cpu_data.x86_clflush_size; unsigned long clflush_mask = x86_clflush_size - 1; + void *vaddr = (void __force *)addr; void *vend = vaddr + size; void *p; @@ -115,7 +116,7 @@ static inline size_t arch_copy_from_iter_pmem(void __pmem *addr, size_t bytes, len = copy_from_iter_nocache(vaddr, bytes, i); if (__iter_needs_pmem_wb(i)) - __arch_wb_cache_pmem(vaddr, bytes); + arch_wb_cache_pmem(addr, bytes); return len; } @@ -133,7 +134,7 @@ static inline void arch_clear_pmem(void __pmem *addr, size_t size) void *vaddr = (void __force *)addr; memset(vaddr, 0, size); - __arch_wb_cache_pmem(vaddr, size); + arch_wb_cache_pmem(addr, size); } static inline bool __arch_has_wmb_pmem(void) diff --git a/include/linux/pmem.h b/include/linux/pmem.h index acfea8c..7c3d11a 100644 --- a/include/linux/pmem.h +++ b/include/linux/pmem.h @@ -53,12 +53,18 @@ static inline void arch_clear_pmem(void __pmem *addr, size_t size) { BUG(); } + +static inline void arch_wb_cache_pmem(void __pmem *addr, size_t size) +{ + BUG(); +} #endif /* * Architectures that define ARCH_HAS_PMEM_API must provide * implementations for arch_memcpy_to_pmem(), arch_wmb_pmem(), - * arch_copy_from_iter_pmem(), arch_clear_pmem() and arch_has_wmb_pmem(). + * arch_copy_from_iter_pmem(), arch_clear_pmem(), arch_wb_cache_pmem() + * and arch_has_wmb_pmem(). */ static inline void memcpy_from_pmem(void *dst, void __pmem const *src, size_t size) { @@ -178,4 +184,18 @@ static inline void clear_pmem(void __pmem *addr, size_t size) else default_clear_pmem(addr, size); } + +/** + * wb_cache_pmem - write back processor cache for PMEM memory range + * @addr: virtual start address + * @size: number of bytes to write back + * + * Write back the processor cache range starting at 'addr' for 'size' bytes. + * This function requires explicit ordering with a wmb_pmem() call. + */ +static inline void wb_cache_pmem(void __pmem *addr, size_t size) +{ + if (arch_has_pmem_api()) + arch_wb_cache_pmem(addr, size); +} #endif /* __PMEM_H__ */ -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id D14C27F51 for ; Thu, 7 Jan 2016 23:28:18 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id B62BA304039 for ; Thu, 7 Jan 2016 21:28:15 -0800 (PST) X-ASG-Debug-ID: 1452230892-04cbb05f7f04900002-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id ApuzZACdMC0luqc8 for ; Thu, 07 Jan 2016 21:28:13 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25713952" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:12 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 1/9] dax: fix NULL pointer dereference in __dax_dbg() Date: Thu, 7 Jan 2016 22:27:51 -0700 X-ASG-Orig-Subj: [PATCH v8 1/9] dax: fix NULL pointer dereference in __dax_dbg() Message-Id: <1452230879-18117-2-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230893 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 In __dax_pmd_fault() we currently assume that get_block() will always set bh.b_bdev and we unconditionally dereference it in __dax_dbg(). This assumption isn't always true - when called for reads of holes ext4_dax_mmap_get_block() returns a buffer head where bh->b_bdev is never set. I hit this BUG while testing the DAX PMD fault path. Instead, initialize bh.b_bdev before passing bh into get_block(). It is possible that the filesystem's get_block() will update bh.b_bdev, and this is fine - we just want to initialize bh.b_bdev to something reasonable so that the calls to __dax_dbg() work and print something useful. Signed-off-by: Ross Zwisler Cc: Dan Williams --- fs/dax.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/dax.c b/fs/dax.c index 7af8797..513bba5 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -624,6 +624,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, } memset(&bh, 0, sizeof(bh)); + bh.b_bdev = inode->i_sb->s_bdev; block = (sector_t)pgoff << (PAGE_SHIFT - blkbits); bh.b_size = PMD_SIZE; -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:23 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 435F17F37 for ; Thu, 7 Jan 2016 23:28:23 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id BDFFFAC006 for ; Thu, 7 Jan 2016 21:28:22 -0800 (PST) X-ASG-Debug-ID: 1452230895-04cbb05f7d04900004-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id WWJhe9BdobAUGGW6 for ; Thu, 07 Jan 2016 21:28:21 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25713973" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:19 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 5/9] mm: add find_get_entries_tag() Date: Thu, 7 Jan 2016 22:27:55 -0700 X-ASG-Orig-Subj: [PATCH v8 5/9] mm: add find_get_entries_tag() Message-Id: <1452230879-18117-6-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230900 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Add find_get_entries_tag() to the family of functions that include find_get_entries(), find_get_pages() and find_get_pages_tag(). This is needed for DAX dirty page handling because we need a list of both page offsets and radix tree entries ('indices' and 'entries' in this function) that are marked with the PAGECACHE_TAG_TOWRITE tag. Signed-off-by: Ross Zwisler Reviewed-by: Jan Kara --- include/linux/pagemap.h | 3 +++ mm/filemap.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 4d08b6c..92395a0 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -361,6 +361,9 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t start, unsigned int nr_pages, struct page **pages); unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index, int tag, unsigned int nr_pages, struct page **pages); +unsigned find_get_entries_tag(struct address_space *mapping, pgoff_t start, + int tag, unsigned int nr_entries, + struct page **entries, pgoff_t *indices); struct page *grab_cache_page_write_begin(struct address_space *mapping, pgoff_t index, unsigned flags); diff --git a/mm/filemap.c b/mm/filemap.c index 7b8be78..1e215fc 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1499,6 +1499,74 @@ repeat: } EXPORT_SYMBOL(find_get_pages_tag); +/** + * find_get_entries_tag - find and return entries that match @tag + * @mapping: the address_space to search + * @start: the starting page cache index + * @tag: the tag index + * @nr_entries: the maximum number of entries + * @entries: where the resulting entries are placed + * @indices: the cache indices corresponding to the entries in @entries + * + * Like find_get_entries, except we only return entries which are tagged with + * @tag. + */ +unsigned find_get_entries_tag(struct address_space *mapping, pgoff_t start, + int tag, unsigned int nr_entries, + struct page **entries, pgoff_t *indices) +{ + void **slot; + unsigned int ret = 0; + struct radix_tree_iter iter; + + if (!nr_entries) + return 0; + + rcu_read_lock(); +restart: + radix_tree_for_each_tagged(slot, &mapping->page_tree, + &iter, start, tag) { + struct page *page; +repeat: + page = radix_tree_deref_slot(slot); + if (unlikely(!page)) + continue; + if (radix_tree_exception(page)) { + if (radix_tree_deref_retry(page)) { + /* + * Transient condition which can only trigger + * when entry at index 0 moves out of or back + * to root: none yet gotten, safe to restart. + */ + goto restart; + } + + /* + * A shadow entry of a recently evicted page, a swap + * entry from shmem/tmpfs or a DAX entry. Return it + * without attempting to raise page count. + */ + goto export; + } + if (!page_cache_get_speculative(page)) + goto repeat; + + /* Has the page moved? */ + if (unlikely(page != *slot)) { + page_cache_release(page); + goto repeat; + } +export: + indices[ret] = iter.index; + entries[ret] = page; + if (++ret == nr_entries) + break; + } + rcu_read_unlock(); + return ret; +} +EXPORT_SYMBOL(find_get_entries_tag); + /* * CD/DVDs are error prone. When a medium error occurs, the driver may fail * a _large_ part of the i/o request. Imagine the worst scenario: -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:24 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9305D7F37 for ; Thu, 7 Jan 2016 23:28:24 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 911CC8F804C for ; Thu, 7 Jan 2016 21:28:24 -0800 (PST) X-ASG-Debug-ID: 1452230895-04cbb05f7d04900005-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id F62tgyaBWZpMYm2A for ; Thu, 07 Jan 2016 21:28:22 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:22 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25713983" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:21 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 6/9] dax: add support for fsync/msync Date: Thu, 7 Jan 2016 22:27:56 -0700 X-ASG-Orig-Subj: [PATCH v8 6/9] dax: add support for fsync/msync Message-Id: <1452230879-18117-7-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230902 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 To properly handle fsync/msync in an efficient way DAX needs to track dirty pages so it is able to flush them durably to media on demand. The tracking of dirty pages is done via the radix tree in struct address_space. This radix tree is already used by the page writeback infrastructure for tracking dirty pages associated with an open file, and it already has support for exceptional (non struct page*) entries. We build upon these features to add exceptional entries to the radix tree for DAX dirty PMD or PTE pages at fault time. Signed-off-by: Ross Zwisler --- fs/dax.c | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++-- include/linux/dax.h | 2 + mm/filemap.c | 6 ++ 3 files changed, 196 insertions(+), 6 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 5b84a46..0db21ea 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -324,6 +325,174 @@ static int copy_user_bh(struct page *to, struct inode *inode, return 0; } +#define NO_SECTOR -1 + +static int dax_radix_entry(struct address_space *mapping, pgoff_t index, + sector_t sector, bool pmd_entry, bool dirty) +{ + struct radix_tree_root *page_tree = &mapping->page_tree; + int type, error = 0; + void *entry; + + __mark_inode_dirty(mapping->host, I_DIRTY_PAGES); + + spin_lock_irq(&mapping->tree_lock); + entry = radix_tree_lookup(page_tree, index); + + if (entry) { + type = RADIX_DAX_TYPE(entry); + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && + type != RADIX_DAX_PMD)) { + error = -EIO; + goto unlock; + } + + if (!pmd_entry || type == RADIX_DAX_PMD) + goto dirty; + radix_tree_delete(&mapping->page_tree, index); + mapping->nrexceptional--; + } + + if (sector == NO_SECTOR) { + /* + * This can happen during correct operation if our pfn_mkwrite + * fault raced against a hole punch operation. If this + * happens the pte that was hole punched will have been + * unmapped and the radix tree entry will have been removed by + * the time we are called, but the call will still happen. We + * will return all the way up to wp_pfn_shared(), where the + * pte_same() check will fail, eventually causing page fault + * to be retried by the CPU. + */ + goto unlock; + } + + error = radix_tree_insert(page_tree, index, + RADIX_DAX_ENTRY(sector, pmd_entry)); + if (error) + goto unlock; + + mapping->nrexceptional++; + dirty: + if (dirty) + radix_tree_tag_set(page_tree, index, PAGECACHE_TAG_DIRTY); + unlock: + spin_unlock_irq(&mapping->tree_lock); + return error; +} + +static int dax_writeback_one(struct block_device *bdev, + struct address_space *mapping, pgoff_t index, void *entry) +{ + struct radix_tree_root *page_tree = &mapping->page_tree; + int type = RADIX_DAX_TYPE(entry); + struct radix_tree_node *node; + struct blk_dax_ctl dax; + void **slot; + int ret = 0; + + spin_lock_irq(&mapping->tree_lock); + /* + * Regular page slots are stabilized by the page lock even + * without the tree itself locked. These unlocked entries + * need verification under the tree lock. + */ + if (!__radix_tree_lookup(page_tree, index, &node, &slot)) + goto unlock; + if (*slot != entry) + goto unlock; + + /* another fsync thread may have already written back this entry */ + if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE)) + goto unlock; + + radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE); + + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && type != RADIX_DAX_PMD)) { + ret = -EIO; + goto unlock; + } + + dax.sector = RADIX_DAX_SECTOR(entry); + dax.size = (type == RADIX_DAX_PMD ? PMD_SIZE : PAGE_SIZE); + spin_unlock_irq(&mapping->tree_lock); + + /* + * We cannot hold tree_lock while calling dax_map_atomic() because it + * eventually calls cond_resched(). + */ + ret = dax_map_atomic(bdev, &dax); + if (ret < 0) + return ret; + + if (WARN_ON_ONCE(ret < dax.size)) { + ret = -EIO; + goto unmap; + } + + wb_cache_pmem(dax.addr, dax.size); + unmap: + dax_unmap_atomic(bdev, &dax); + return ret; + + unlock: + spin_unlock_irq(&mapping->tree_lock); + return ret; +} + +/* + * Flush the mapping to the persistent domain within the byte range of [start, + * end]. This is required by data integrity operations to ensure file data is + * on persistent storage prior to completion of the operation. + */ +int dax_writeback_mapping_range(struct address_space *mapping, loff_t start, + loff_t end) +{ + struct inode *inode = mapping->host; + struct block_device *bdev = inode->i_sb->s_bdev; + pgoff_t indices[PAGEVEC_SIZE]; + pgoff_t start_page, end_page; + struct pagevec pvec; + void *entry; + int i, ret = 0; + + if (WARN_ON_ONCE(inode->i_blkbits != PAGE_SHIFT)) + return -EIO; + + rcu_read_lock(); + entry = radix_tree_lookup(&mapping->page_tree, start & PMD_MASK); + rcu_read_unlock(); + + /* see if the start of our range is covered by a PMD entry */ + if (entry && RADIX_DAX_TYPE(entry) == RADIX_DAX_PMD) + start &= PMD_MASK; + + start_page = start >> PAGE_CACHE_SHIFT; + end_page = end >> PAGE_CACHE_SHIFT; + + tag_pages_for_writeback(mapping, start_page, end_page); + + pagevec_init(&pvec, 0); + while (1) { + pvec.nr = find_get_entries_tag(mapping, start_page, + PAGECACHE_TAG_TOWRITE, PAGEVEC_SIZE, + pvec.pages, indices); + + if (pvec.nr == 0) + break; + + for (i = 0; i < pvec.nr; i++) { + ret = dax_writeback_one(bdev, mapping, indices[i], + pvec.pages[i]); + if (ret < 0) + return ret; + } + } + wmb_pmem(); + return 0; +} +EXPORT_SYMBOL_GPL(dax_writeback_mapping_range); + static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh, struct vm_area_struct *vma, struct vm_fault *vmf) { @@ -363,6 +532,11 @@ static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh, } dax_unmap_atomic(bdev, &dax); + error = dax_radix_entry(mapping, vmf->pgoff, dax.sector, false, + vmf->flags & FAULT_FLAG_WRITE); + if (error) + goto out; + error = vm_insert_mixed(vma, vaddr, dax.pfn); out: @@ -487,6 +661,7 @@ int __dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf, delete_from_page_cache(page); unlock_page(page); page_cache_release(page); + page = NULL; } /* @@ -591,7 +766,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, pgoff_t size, pgoff; loff_t lstart, lend; sector_t block; - int result = 0; + int error, result = 0; /* dax pmd mappings require pfn_t_devmap() */ if (!IS_ENABLED(CONFIG_FS_DAX_PMD)) @@ -733,6 +908,16 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, } dax_unmap_atomic(bdev, &dax); + if (write) { + error = dax_radix_entry(mapping, pgoff, dax.sector, + true, true); + if (error) { + dax_pmd_dbg(&bh, address, + "PMD radix insertion failed"); + goto fallback; + } + } + dev_dbg(part_to_dev(bdev->bd_part), "%s: %s addr: %lx pfn: %lx sect: %llx\n", __func__, current->comm, address, @@ -791,15 +976,12 @@ EXPORT_SYMBOL_GPL(dax_pmd_fault); * dax_pfn_mkwrite - handle first write to DAX page * @vma: The virtual memory area where the fault occurred * @vmf: The description of the fault - * */ int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf) { - struct super_block *sb = file_inode(vma->vm_file)->i_sb; + struct file *file = vma->vm_file; - sb_start_pagefault(sb); - file_update_time(vma->vm_file); - sb_end_pagefault(sb); + dax_radix_entry(file->f_mapping, vmf->pgoff, NO_SECTOR, false, true); return VM_FAULT_NOPAGE; } EXPORT_SYMBOL_GPL(dax_pfn_mkwrite); diff --git a/include/linux/dax.h b/include/linux/dax.h index e9d57f68..8204c3d 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -41,4 +41,6 @@ static inline bool dax_mapping(struct address_space *mapping) { return mapping->host && IS_DAX(mapping->host); } +int dax_writeback_mapping_range(struct address_space *mapping, loff_t start, + loff_t end); #endif diff --git a/mm/filemap.c b/mm/filemap.c index 1e215fc..2e7c8d9 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -482,6 +482,12 @@ int filemap_write_and_wait_range(struct address_space *mapping, { int err = 0; + if (dax_mapping(mapping) && mapping->nrexceptional) { + err = dax_writeback_mapping_range(mapping, lstart, lend); + if (err) + return err; + } + if (mapping->nrpages) { err = __filemap_fdatawrite_range(mapping, lstart, lend, WB_SYNC_ALL); -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id B97C97F52 for ; Thu, 7 Jan 2016 23:28:24 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 30184AC001 for ; Thu, 7 Jan 2016 21:28:21 -0800 (PST) X-ASG-Debug-ID: 1452230895-04cbb05f7d04900003-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id I5FuSp7AxU3oVUTu for ; Thu, 07 Jan 2016 21:28:19 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:18 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25713967" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:17 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree Date: Thu, 7 Jan 2016 22:27:54 -0700 X-ASG-Orig-Subj: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree Message-Id: <1452230879-18117-5-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230897 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Add support for tracking dirty DAX entries in the struct address_space radix tree. This tree is already used for dirty page writeback, and it already supports the use of exceptional (non struct page*) entries. In order to properly track dirty DAX pages we will insert new exceptional entries into the radix tree that represent dirty DAX PTE or PMD pages. These exceptional entries will also contain the writeback sectors for the PTE or PMD faults that we can use at fsync/msync time. There are currently two types of exceptional entries (shmem and shadow) that can be placed into the radix tree, and this adds a third. We rely on the fact that only one type of exceptional entry can be found in a given radix tree based on its usage. This happens for free with DAX vs shmem but we explicitly prevent shadow entries from being added to radix trees for DAX mappings. The only shadow entries that would be generated for DAX radix trees would be to track zero page mappings that were created for holes. These pages would receive minimal benefit from having shadow entries, and the choice to have only one type of exceptional entry in a given radix tree makes the logic simpler both in clear_exceptional_entry() and in the rest of DAX. Signed-off-by: Ross Zwisler Reviewed-by: Jan Kara --- fs/block_dev.c | 2 +- fs/inode.c | 2 +- include/linux/dax.h | 5 ++++ include/linux/fs.h | 3 +- include/linux/radix-tree.h | 9 ++++++ mm/filemap.c | 17 ++++++++---- mm/truncate.c | 69 ++++++++++++++++++++++++++-------------------- mm/vmscan.c | 9 +++++- mm/workingset.c | 4 +-- 9 files changed, 78 insertions(+), 42 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index e123641..303b7cd 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -75,7 +75,7 @@ void kill_bdev(struct block_device *bdev) { struct address_space *mapping = bdev->bd_inode->i_mapping; - if (mapping->nrpages == 0 && mapping->nrshadows == 0) + if (mapping->nrpages == 0 && mapping->nrexceptional == 0) return; invalidate_bh_lrus(); diff --git a/fs/inode.c b/fs/inode.c index 4c8f719..6e3e5d0 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -495,7 +495,7 @@ void clear_inode(struct inode *inode) */ spin_lock_irq(&inode->i_data.tree_lock); BUG_ON(inode->i_data.nrpages); - BUG_ON(inode->i_data.nrshadows); + BUG_ON(inode->i_data.nrexceptional); spin_unlock_irq(&inode->i_data.tree_lock); BUG_ON(!list_empty(&inode->i_data.private_list)); BUG_ON(!(inode->i_state & I_FREEING)); diff --git a/include/linux/dax.h b/include/linux/dax.h index b415e52..e9d57f68 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -36,4 +36,9 @@ static inline bool vma_is_dax(struct vm_area_struct *vma) { return vma->vm_file && IS_DAX(vma->vm_file->f_mapping->host); } + +static inline bool dax_mapping(struct address_space *mapping) +{ + return mapping->host && IS_DAX(mapping->host); +} #endif diff --git a/include/linux/fs.h b/include/linux/fs.h index ddb7bad..fdab768 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -433,7 +433,8 @@ struct address_space { struct rw_semaphore i_mmap_rwsem; /* protect tree, count, list */ /* Protected by tree_lock together with the radix tree */ unsigned long nrpages; /* number of total pages */ - unsigned long nrshadows; /* number of shadow entries */ + /* number of shadow or DAX exceptional entries */ + unsigned long nrexceptional; pgoff_t writeback_index;/* writeback starts here */ const struct address_space_operations *a_ops; /* methods */ unsigned long flags; /* error bits/gfp mask */ diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h index 57e7d87..7c88ad1 100644 --- a/include/linux/radix-tree.h +++ b/include/linux/radix-tree.h @@ -51,6 +51,15 @@ #define RADIX_TREE_EXCEPTIONAL_ENTRY 2 #define RADIX_TREE_EXCEPTIONAL_SHIFT 2 +#define RADIX_DAX_MASK 0xf +#define RADIX_DAX_SHIFT 4 +#define RADIX_DAX_PTE (0x4 | RADIX_TREE_EXCEPTIONAL_ENTRY) +#define RADIX_DAX_PMD (0x8 | RADIX_TREE_EXCEPTIONAL_ENTRY) +#define RADIX_DAX_TYPE(entry) ((unsigned long)entry & RADIX_DAX_MASK) +#define RADIX_DAX_SECTOR(entry) (((unsigned long)entry >> RADIX_DAX_SHIFT)) +#define RADIX_DAX_ENTRY(sector, pmd) ((void *)((unsigned long)sector << \ + RADIX_DAX_SHIFT | (pmd ? RADIX_DAX_PMD : RADIX_DAX_PTE))) + static inline int radix_tree_is_indirect_ptr(void *ptr) { return (int)((unsigned long)ptr & RADIX_TREE_INDIRECT_PTR); diff --git a/mm/filemap.c b/mm/filemap.c index 847ee43..7b8be78 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -11,6 +11,7 @@ */ #include #include +#include #include #include #include @@ -123,9 +124,9 @@ static void page_cache_tree_delete(struct address_space *mapping, __radix_tree_lookup(&mapping->page_tree, page->index, &node, &slot); if (shadow) { - mapping->nrshadows++; + mapping->nrexceptional++; /* - * Make sure the nrshadows update is committed before + * Make sure the nrexceptional update is committed before * the nrpages update so that final truncate racing * with reclaim does not see both counters 0 at the * same time and miss a shadow entry. @@ -579,9 +580,13 @@ static int page_cache_tree_insert(struct address_space *mapping, p = radix_tree_deref_slot_protected(slot, &mapping->tree_lock); if (!radix_tree_exceptional_entry(p)) return -EEXIST; + + if (WARN_ON(dax_mapping(mapping))) + return -EINVAL; + if (shadowp) *shadowp = p; - mapping->nrshadows--; + mapping->nrexceptional--; if (node) workingset_node_shadows_dec(node); } @@ -1245,9 +1250,9 @@ repeat: if (radix_tree_deref_retry(page)) goto restart; /* - * A shadow entry of a recently evicted page, - * or a swap entry from shmem/tmpfs. Return - * it without attempting to raise page count. + * A shadow entry of a recently evicted page, a swap + * entry from shmem/tmpfs or a DAX entry. Return it + * without attempting to raise page count. */ goto export; } diff --git a/mm/truncate.c b/mm/truncate.c index 76e35ad..e3ee0e2 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -34,31 +35,39 @@ static void clear_exceptional_entry(struct address_space *mapping, return; spin_lock_irq(&mapping->tree_lock); - /* - * Regular page slots are stabilized by the page lock even - * without the tree itself locked. These unlocked entries - * need verification under the tree lock. - */ - if (!__radix_tree_lookup(&mapping->page_tree, index, &node, &slot)) - goto unlock; - if (*slot != entry) - goto unlock; - radix_tree_replace_slot(slot, NULL); - mapping->nrshadows--; - if (!node) - goto unlock; - workingset_node_shadows_dec(node); - /* - * Don't track node without shadow entries. - * - * Avoid acquiring the list_lru lock if already untracked. - * The list_empty() test is safe as node->private_list is - * protected by mapping->tree_lock. - */ - if (!workingset_node_shadows(node) && - !list_empty(&node->private_list)) - list_lru_del(&workingset_shadow_nodes, &node->private_list); - __radix_tree_delete_node(&mapping->page_tree, node); + + if (dax_mapping(mapping)) { + if (radix_tree_delete_item(&mapping->page_tree, index, entry)) + mapping->nrexceptional--; + } else { + /* + * Regular page slots are stabilized by the page lock even + * without the tree itself locked. These unlocked entries + * need verification under the tree lock. + */ + if (!__radix_tree_lookup(&mapping->page_tree, index, &node, + &slot)) + goto unlock; + if (*slot != entry) + goto unlock; + radix_tree_replace_slot(slot, NULL); + mapping->nrexceptional--; + if (!node) + goto unlock; + workingset_node_shadows_dec(node); + /* + * Don't track node without shadow entries. + * + * Avoid acquiring the list_lru lock if already untracked. + * The list_empty() test is safe as node->private_list is + * protected by mapping->tree_lock. + */ + if (!workingset_node_shadows(node) && + !list_empty(&node->private_list)) + list_lru_del(&workingset_shadow_nodes, + &node->private_list); + __radix_tree_delete_node(&mapping->page_tree, node); + } unlock: spin_unlock_irq(&mapping->tree_lock); } @@ -228,7 +237,7 @@ void truncate_inode_pages_range(struct address_space *mapping, int i; cleancache_invalidate_inode(mapping); - if (mapping->nrpages == 0 && mapping->nrshadows == 0) + if (mapping->nrpages == 0 && mapping->nrexceptional == 0) return; /* Offsets within partial pages */ @@ -402,7 +411,7 @@ EXPORT_SYMBOL(truncate_inode_pages); */ void truncate_inode_pages_final(struct address_space *mapping) { - unsigned long nrshadows; + unsigned long nrexceptional; unsigned long nrpages; /* @@ -416,14 +425,14 @@ void truncate_inode_pages_final(struct address_space *mapping) /* * When reclaim installs eviction entries, it increases - * nrshadows first, then decreases nrpages. Make sure we see + * nrexceptional first, then decreases nrpages. Make sure we see * this in the right order or we might miss an entry. */ nrpages = mapping->nrpages; smp_rmb(); - nrshadows = mapping->nrshadows; + nrexceptional = mapping->nrexceptional; - if (nrpages || nrshadows) { + if (nrpages || nrexceptional) { /* * As truncation uses a lockless tree lookup, cycle * the tree lock to make sure any ongoing tree diff --git a/mm/vmscan.c b/mm/vmscan.c index 44ec50f..30e0cd7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -46,6 +46,7 @@ #include #include #include +#include #include #include @@ -671,9 +672,15 @@ static int __remove_mapping(struct address_space *mapping, struct page *page, * inode reclaim needs to empty out the radix tree or * the nodes are lost. Don't plant shadows behind its * back. + * + * We also don't store shadows for DAX mappings because the + * only page cache pages found in these are zero pages + * covering holes, and because we don't want to mix DAX + * exceptional entries and shadow exceptional entries in the + * same page_tree. */ if (reclaimed && page_is_file_cache(page) && - !mapping_exiting(mapping)) + !mapping_exiting(mapping) && !dax_mapping(mapping)) shadow = workingset_eviction(mapping, page); __delete_from_page_cache(page, shadow, memcg); spin_unlock_irqrestore(&mapping->tree_lock, flags); diff --git a/mm/workingset.c b/mm/workingset.c index aa01713..61ead9e 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -351,8 +351,8 @@ static enum lru_status shadow_lru_isolate(struct list_head *item, node->slots[i] = NULL; BUG_ON(node->count < (1U << RADIX_TREE_COUNT_SHIFT)); node->count -= 1U << RADIX_TREE_COUNT_SHIFT; - BUG_ON(!mapping->nrshadows); - mapping->nrshadows--; + BUG_ON(!mapping->nrexceptional); + mapping->nrexceptional--; } } BUG_ON(node->count); -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E4A717F37 for ; Thu, 7 Jan 2016 23:28:25 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id D75A5304053 for ; Thu, 7 Jan 2016 21:28:25 -0800 (PST) X-ASG-Debug-ID: 1452230895-04cbb05f7d04900006-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id XIHHA7ABKdEey4pb for ; Thu, 07 Jan 2016 21:28:24 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25713997" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:22 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 7/9] ext2: call dax_pfn_mkwrite() for DAX fsync/msync Date: Thu, 7 Jan 2016 22:27:57 -0700 X-ASG-Orig-Subj: [PATCH v8 7/9] ext2: call dax_pfn_mkwrite() for DAX fsync/msync Message-Id: <1452230879-18117-8-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230903 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 To properly support the new DAX fsync/msync infrastructure filesystems need to call dax_pfn_mkwrite() so that DAX can track when user pages are dirtied. Signed-off-by: Ross Zwisler Reviewed-by: Jan Kara --- fs/ext2/file.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/ext2/file.c b/fs/ext2/file.c index 11a42c5..2c88d68 100644 --- a/fs/ext2/file.c +++ b/fs/ext2/file.c @@ -102,8 +102,8 @@ static int ext2_dax_pfn_mkwrite(struct vm_area_struct *vma, { struct inode *inode = file_inode(vma->vm_file); struct ext2_inode_info *ei = EXT2_I(inode); - int ret = VM_FAULT_NOPAGE; loff_t size; + int ret; sb_start_pagefault(inode->i_sb); file_update_time(vma->vm_file); @@ -113,6 +113,8 @@ static int ext2_dax_pfn_mkwrite(struct vm_area_struct *vma, size = (i_size_read(inode) + PAGE_SIZE - 1) >> PAGE_SHIFT; if (vmf->pgoff >= size) ret = VM_FAULT_SIGBUS; + else + ret = dax_pfn_mkwrite(vma, vmf); up_read(&ei->dax_sem); sb_end_pagefault(inode->i_sb); -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 962297F59 for ; Thu, 7 Jan 2016 23:28:30 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 946B98F804C for ; Thu, 7 Jan 2016 21:28:30 -0800 (PST) X-ASG-Debug-ID: 1452230895-04cbb05f7d04900007-NocioJ Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id Xl30Gott0kXDZ6QP for ; Thu, 07 Jan 2016 21:28:29 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 134.134.136.20 X-ASG-Whitelist: EmailCat (corporate) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP; 07 Jan 2016 21:28:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25714004" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:24 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 8/9] ext4: call dax_pfn_mkwrite() for DAX fsync/msync Date: Thu, 7 Jan 2016 22:27:58 -0700 X-ASG-Orig-Subj: [PATCH v8 8/9] ext4: call dax_pfn_mkwrite() for DAX fsync/msync Message-Id: <1452230879-18117-9-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: mga02.intel.com[134.134.136.20] X-Barracuda-Start-Time: 1452230905 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 To properly support the new DAX fsync/msync infrastructure filesystems need to call dax_pfn_mkwrite() so that DAX can track when user pages are dirtied. Signed-off-by: Ross Zwisler Reviewed-by: Jan Kara --- fs/ext4/file.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 60683ab..fa899c9 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -291,8 +291,8 @@ static int ext4_dax_pfn_mkwrite(struct vm_area_struct *vma, { struct inode *inode = file_inode(vma->vm_file); struct super_block *sb = inode->i_sb; - int ret = VM_FAULT_NOPAGE; loff_t size; + int ret; sb_start_pagefault(sb); file_update_time(vma->vm_file); @@ -300,6 +300,8 @@ static int ext4_dax_pfn_mkwrite(struct vm_area_struct *vma, size = (i_size_read(inode) + PAGE_SIZE - 1) >> PAGE_SHIFT; if (vmf->pgoff >= size) ret = VM_FAULT_SIGBUS; + else + ret = dax_pfn_mkwrite(vma, vmf); up_read(&EXT4_I(inode)->i_mmap_sem); sb_end_pagefault(sb); -- 2.5.0 From ross.zwisler@linux.intel.com Thu Jan 7 23:28:31 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 3DC647F59 for ; Thu, 7 Jan 2016 23:28:31 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id D2A71AC006 for ; Thu, 7 Jan 2016 21:28:30 -0800 (PST) X-ASG-Debug-ID: 1452230908-04bdf06f6e04c60001-NocioJ Received: from mga14.intel.com ([192.55.52.115]) by cuda.sgi.com with ESMTP id QaoPkSiUi188phJH for ; Thu, 07 Jan 2016 21:28:28 -0800 (PST) X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com X-Barracuda-Apparent-Source-IP: 192.55.52.115 Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP; 07 Jan 2016 21:28:28 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,536,1444719600"; d="scan'208";a="25714012" Received: from rzwisler-desk.amr.corp.intel.com (HELO phyrexia.intel.com) ([10.252.201.208]) by fmsmga004.fm.intel.com with ESMTP; 07 Jan 2016 21:28:26 -0800 From: Ross Zwisler To: linux-kernel@vger.kernel.org Cc: Ross Zwisler , "H. Peter Anvin" , "J. Bruce Fields" , "Theodore Ts'o" , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: [PATCH v8 9/9] xfs: call dax_pfn_mkwrite() for DAX fsync/msync Date: Thu, 7 Jan 2016 22:27:59 -0700 X-ASG-Orig-Subj: [PATCH v8 9/9] xfs: call dax_pfn_mkwrite() for DAX fsync/msync Message-Id: <1452230879-18117-10-git-send-email-ross.zwisler@linux.intel.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> X-Barracuda-Connect: UNKNOWN[192.55.52.115] X-Barracuda-Start-Time: 1452230908 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25943 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 To properly support the new DAX fsync/msync infrastructure filesystems need to call dax_pfn_mkwrite() so that DAX can track when user pages are dirtied. Signed-off-by: Ross Zwisler --- fs/xfs/xfs_file.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index ebe9b82..55e16e2 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1610,9 +1610,8 @@ xfs_filemap_pmd_fault( /* * pfn_mkwrite was originally inteneded to ensure we capture time stamp * updates on write faults. In reality, it's need to serialise against - * truncate similar to page_mkwrite. Hence we open-code dax_pfn_mkwrite() - * here and cycle the XFS_MMAPLOCK_SHARED to ensure we serialise the fault - * barrier in place. + * truncate similar to page_mkwrite. Hence we cycle the XFS_MMAPLOCK_SHARED + * to ensure we serialise the fault barrier in place. */ static int xfs_filemap_pfn_mkwrite( @@ -1635,6 +1634,8 @@ xfs_filemap_pfn_mkwrite( size = (i_size_read(inode) + PAGE_SIZE - 1) >> PAGE_SHIFT; if (vmf->pgoff >= size) ret = VM_FAULT_SIGBUS; + else if (IS_DAX(inode)) + ret = dax_pfn_mkwrite(vma, vmf); xfs_iunlock(ip, XFS_MMAPLOCK_SHARED); sb_end_pagefault(inode->i_sb); return ret; -- 2.5.0 From darrick.wong@oracle.com Fri Jan 8 03:34:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id F00087F37 for ; Fri, 8 Jan 2016 03:34:15 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 61BDFAC002 for ; Fri, 8 Jan 2016 01:34:12 -0800 (PST) X-ASG-Debug-ID: 1452245646-04cbb05f7f0a210001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id UlAZDjSTMqxeAzMy (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 08 Jan 2016 01:34:06 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u089Y4Tu023185 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 8 Jan 2016 09:34:05 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u089Y4CP011797 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 8 Jan 2016 09:34:04 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u089Y3w6011006; Fri, 8 Jan 2016 09:34:04 GMT Received: from localhost (/71.198.20.188) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 08 Jan 2016 01:34:02 -0800 Date: Fri, 8 Jan 2016 01:34:01 -0800 From: "Darrick J. Wong" To: david@fromorbit.com Cc: xfs@oss.sgi.com Subject: Re: [PATCH 60/76] xfs: implement CoW for directio writes Message-ID: <20160108093401.GM28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 60/76] xfs: implement CoW for directio writes References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090307.12713.60312.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219090307.12713.60312.stgit@birch.djwong.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0022.oracle.com [156.151.31.74] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1452245646 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25947 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sat, Dec 19, 2015 at 01:03:07AM -0800, Darrick J. Wong wrote: > For O_DIRECT writes to shared blocks, we have to CoW them just like > we would with buffered writes. For writes that are not block-aligned, > just bounce them to the page cache. > > For block-aligned writes, however, we can do better than that. Use > the same mechanisms that we employ for buffered CoW to set up a > delalloc reservation, allocate all the blocks at once, issue the > writes against the new blocks and use the same ioend functions to > remap the blocks after the write. This should be fairly performant. > > Signed-off-by: Darrick J. Wong > --- > fs/xfs/xfs_aops.c | 63 +++++++++++++++++++++++++--- > fs/xfs/xfs_file.c | 12 ++++- > fs/xfs/xfs_reflink.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++ > fs/xfs/xfs_reflink.h | 5 ++ > 4 files changed, 186 insertions(+), 8 deletions(-) > > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c > index 8101d6a..4b77d07 100644 > --- a/fs/xfs/xfs_aops.c > +++ b/fs/xfs/xfs_aops.c > @@ -1339,7 +1339,8 @@ xfs_map_direct( > struct buffer_head *bh_result, > struct xfs_bmbt_irec *imap, > xfs_off_t offset, > - bool dax_fault) > + bool dax_fault, > + bool is_cow) > { > struct xfs_ioend *ioend; > xfs_off_t size = bh_result->b_size; > @@ -1368,20 +1369,23 @@ xfs_map_direct( > > if (type == XFS_IO_UNWRITTEN && type != ioend->io_type) > ioend->io_type = XFS_IO_UNWRITTEN; > + if (is_cow) > + ioend->io_flags |= XFS_IOEND_COW; > > trace_xfs_gbmap_direct_update(XFS_I(inode), ioend->io_offset, > ioend->io_size, ioend->io_type, > imap); > - } else if (type == XFS_IO_UNWRITTEN || > + } else if (type == XFS_IO_UNWRITTEN || is_cow || > offset + size > i_size_read(inode) || > offset + size < 0) { > ioend = xfs_alloc_ioend(inode, type); > ioend->io_offset = offset; > ioend->io_size = size; > + if (is_cow) > + ioend->io_flags |= XFS_IOEND_COW; NAK. Further testing demonstrates incorrect remapping when the DIO CoW write fails because xfs_end_io_direct_write doesn't know if the write succeeded or not. xfs_vm_do_dio does, however, so we should move the remapping/cancelling code to that function and avoid using the ioend entirely for directio cow. --D > > bh_result->b_private = ioend; > set_buffer_defer_completion(bh_result); > - > trace_xfs_gbmap_direct_new(XFS_I(inode), offset, size, type, > imap); > } else { > @@ -1449,6 +1453,8 @@ __xfs_get_blocks( > xfs_off_t offset; > ssize_t size; > int new = 0; > + bool is_cow = false; > + bool need_alloc = false; > > if (XFS_FORCED_SHUTDOWN(mp)) > return -EIO; > @@ -1480,8 +1486,15 @@ __xfs_get_blocks( > end_fsb = XFS_B_TO_FSB(mp, (xfs_ufsize_t)offset + size); > offset_fsb = XFS_B_TO_FSBT(mp, offset); > > - error = xfs_bmapi_read(ip, offset_fsb, end_fsb - offset_fsb, > - &imap, &nimaps, XFS_BMAPI_ENTIRE); > + if (create && direct) > + is_cow = xfs_reflink_is_cow_pending(ip, offset); > + if (is_cow) > + error = xfs_reflink_find_cow_mapping(ip, offset, &imap, > + &need_alloc); > + else > + error = xfs_bmapi_read(ip, offset_fsb, end_fsb - offset_fsb, > + &imap, &nimaps, XFS_BMAPI_ENTIRE); > + ASSERT(!need_alloc); > if (error) > goto out_unlock; > > @@ -1553,13 +1566,33 @@ __xfs_get_blocks( > if (imap.br_startblock != HOLESTARTBLOCK && > imap.br_startblock != DELAYSTARTBLOCK && > (create || !ISUNWRITTEN(&imap))) { > + if (create && direct && !is_cow) { > + bool shared; > + > + error = xfs_reflink_irec_is_shared(ip, &imap, &shared); > + if (error) > + return error; > + /* > + * Are we doing a DIO write to a shared block? In > + * the ideal world we at least would fork full blocks, > + * but for now just fall back to buffered mode. Yuck. > + * Use -EREMCHG ("remote address changed") to signal > + * this, since in general XFS doesn't do this sort of > + * fallback. > + */ > + if (shared) { > + trace_xfs_reflink_bounce_dio_write(ip, &imap); > + return -EREMCHG; > + } > + } > + > xfs_map_buffer(inode, bh_result, &imap, offset); > if (ISUNWRITTEN(&imap)) > set_buffer_unwritten(bh_result); > /* direct IO needs special help */ > if (create && direct) > xfs_map_direct(inode, bh_result, &imap, offset, > - dax_fault); > + dax_fault, is_cow); > } > > /* > @@ -1738,6 +1771,24 @@ xfs_vm_do_dio( > int flags) > { > struct block_device *bdev; > + loff_t end; > + loff_t block_mask; > + int error; > + > + /* If this is a block-aligned directio CoW, remap immediately. */ > + end = offset + iov_iter_count(iter); > + block_mask = (1 << inode->i_blkbits) - 1; > + if (xfs_is_reflink_inode(XFS_I(inode)) && iov_iter_rw(iter) == WRITE && > + !(offset & block_mask) && !(end & block_mask)) { > + error = xfs_reflink_reserve_cow_range(XFS_I(inode), offset, > + iov_iter_count(iter)); > + if (error) > + return error; > + error = xfs_reflink_allocate_cow_range(XFS_I(inode), offset, > + iov_iter_count(iter)); > + if (error) > + return error; > + } > > if (IS_DAX(inode)) > return dax_do_io(iocb, inode, iter, offset, > diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c > index 0fbcb38..31b002e 100644 > --- a/fs/xfs/xfs_file.c > +++ b/fs/xfs/xfs_file.c > @@ -892,10 +892,18 @@ xfs_file_write_iter( > if (XFS_FORCED_SHUTDOWN(ip->i_mount)) > return -EIO; > > - if ((iocb->ki_flags & IOCB_DIRECT) || IS_DAX(inode)) > + /* > + * Allow DIO to fall back to buffered *only* in the case that we're > + * doing a reflink CoW. > + */ > + if ((iocb->ki_flags & IOCB_DIRECT) || IS_DAX(inode)) { > ret = xfs_file_dio_aio_write(iocb, from); > - else > + if (ret == -EREMCHG) > + goto buffered; > + } else { > +buffered: > ret = xfs_file_buffered_aio_write(iocb, from); > + } > > if (ret > 0) { > ssize_t err; > diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c > index 9c1c262..8594bc4 100644 > --- a/fs/xfs/xfs_reflink.c > +++ b/fs/xfs/xfs_reflink.c > @@ -134,6 +134,56 @@ xfs_trim_extent( > } > } > > +/** > + * xfs_reflink_irec_is_shared() -- Are any of the blocks in this mapping > + * shared? > + * > + * @ip: XFS inode object > + * @irec: the fileoff:fsblock mapping that we might fork > + * @shared: set to true if the mapping is shared. > + */ > +int > +xfs_reflink_irec_is_shared( > + struct xfs_inode *ip, > + struct xfs_bmbt_irec *irec, > + bool *shared) > +{ > + xfs_agnumber_t agno; > + xfs_agblock_t agbno; > + xfs_extlen_t aglen; > + xfs_agblock_t fbno; > + xfs_extlen_t flen; > + int error = 0; > + > + /* Holes, unwritten, and delalloc extents cannot be shared */ > + if (!xfs_is_reflink_inode(ip) || > + ISUNWRITTEN(irec) || > + irec->br_startblock == HOLESTARTBLOCK || > + irec->br_startblock == DELAYSTARTBLOCK) { > + *shared = false; > + return 0; > + } > + > + trace_xfs_reflink_irec_is_shared(ip, irec); > + > + agno = XFS_FSB_TO_AGNO(ip->i_mount, irec->br_startblock); > + agbno = XFS_FSB_TO_AGBNO(ip->i_mount, irec->br_startblock); > + aglen = irec->br_blockcount; > + > + /* Are there any shared blocks here? */ > + error = xfs_refcount_find_shared(ip->i_mount, agno, agbno, > + aglen, &fbno, &flen, false); > + if (error) > + return error; > + if (flen == 0) { > + *shared = false; > + return 0; > + } > + > + *shared = true; > + return 0; > +} > + > /* Find the shared ranges under an irec, and set up delalloc extents. */ > STATIC int > xfs_reflink_reserve_cow_extent( > @@ -251,6 +301,70 @@ xfs_reflink_reserve_cow_range( > } > > /** > + * xfs_reflink_allocate_cow_range() -- Allocate blocks to satisfy a copy on > + * write operation. > + * @ip: XFS inode. > + * @pos: file offset to start CoWing. > + * @len: number of bytes to CoW. > + */ > +int > +xfs_reflink_allocate_cow_range( > + struct xfs_inode *ip, > + xfs_off_t pos, > + xfs_off_t len) > +{ > + struct xfs_ifork *ifp; > + struct xfs_bmbt_rec_host *gotp; > + struct xfs_bmbt_irec imap; > + int error = 0; > + xfs_fileoff_t start_lblk; > + xfs_fileoff_t end_lblk; > + xfs_extnum_t idx; > + > + if (!xfs_is_reflink_inode(ip)) > + return 0; > + > + trace_xfs_reflink_allocate_cow_range(ip, len, pos, 0); > + > + start_lblk = XFS_B_TO_FSBT(ip->i_mount, pos); > + end_lblk = XFS_B_TO_FSB(ip->i_mount, pos + len); > + ifp = XFS_IFORK_PTR(ip, XFS_COW_FORK); > + xfs_ilock(ip, XFS_ILOCK_EXCL); > + > + gotp = xfs_iext_bno_to_ext(ifp, start_lblk, &idx); > + while (gotp) { > + xfs_bmbt_get_all(gotp, &imap); > + > + if (imap.br_startoff >= end_lblk) > + break; > + if (!isnullstartblock(imap.br_startblock)) > + goto advloop; > + xfs_trim_extent(&imap, start_lblk, end_lblk - start_lblk); > + trace_xfs_reflink_allocate_cow_extent(ip, &imap); > + > + xfs_iunlock(ip, XFS_ILOCK_EXCL); > + error = xfs_iomap_write_allocate(ip, XFS_COW_FORK, > + XFS_FSB_TO_B(ip->i_mount, imap.br_startoff + > + imap.br_blockcount - 1), &imap); > + xfs_ilock(ip, XFS_ILOCK_EXCL); > + if (error) > + break; > +advloop: > + /* Roll on... */ > + idx++; > + if (idx >= ifp->if_bytes / sizeof(xfs_bmbt_rec_t)) > + break; > + gotp = xfs_iext_get_ext(ifp, idx); > + } > + > + xfs_iunlock(ip, XFS_ILOCK_EXCL); > + > + if (error) > + trace_xfs_reflink_allocate_cow_range_error(ip, error, _RET_IP_); > + return error; > +} > + > +/** > * xfs_reflink_is_cow_pending() -- Determine if CoW is pending for a given > * file and offset. > * > diff --git a/fs/xfs/xfs_reflink.h b/fs/xfs/xfs_reflink.h > index 8ec1ebb..d356c00 100644 > --- a/fs/xfs/xfs_reflink.h > +++ b/fs/xfs/xfs_reflink.h > @@ -18,8 +18,13 @@ > #ifndef __XFS_REFLINK_H > #define __XFS_REFLINK_H 1 > > +extern int xfs_reflink_irec_is_shared(struct xfs_inode *ip, > + struct xfs_bmbt_irec *imap, bool *shared); > + > extern int xfs_reflink_reserve_cow_range(struct xfs_inode *ip, xfs_off_t pos, > xfs_off_t len); > +extern int xfs_reflink_allocate_cow_range(struct xfs_inode *ip, xfs_off_t pos, > + xfs_off_t len); > extern bool xfs_reflink_is_cow_pending(struct xfs_inode *ip, xfs_off_t offset); > extern int xfs_reflink_find_cow_mapping(struct xfs_inode *ip, xfs_off_t offset, > struct xfs_bmbt_irec *imap, bool *need_alloc); > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From darrick.wong@oracle.com Fri Jan 8 04:09:44 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 195F57F37 for ; Fri, 8 Jan 2016 04:09:44 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id EBD4A304032 for ; Fri, 8 Jan 2016 02:09:40 -0800 (PST) X-ASG-Debug-ID: 1452247779-04cb6c75dd09700001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id Ql9FqSMddFpKCEkz (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 08 Jan 2016 02:09:39 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u08A9Z4U018331 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 8 Jan 2016 10:09:36 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u08A9Z1O006544 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 8 Jan 2016 10:09:35 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u08A9YXU027134; Fri, 8 Jan 2016 10:09:35 GMT Received: from localhost (/71.198.20.188) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 08 Jan 2016 02:09:34 -0800 Date: Fri, 8 Jan 2016 02:09:33 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160108100933.GN28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> <20160105014310.GK28330@birch.djwong.org> <20160105104214.GA16310@infradead.org> <20160107003227.GB8015@birch.djwong.org> <20160107152541.GA16982@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107152541.GA16982@lst.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0021.oracle.com [156.151.31.71] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1452247779 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25947 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Thu, Jan 07, 2016 at 04:25:42PM +0100, Christoph Hellwig wrote: > On Wed, Jan 06, 2016 at 04:32:27PM -0800, Darrick J. Wong wrote: > > Ok. I spent a couple of days trying to find all the places where we need to > > delete CoW reservations (hole punch, truncate, etc.) and found some places > > where the code was leaving reservations behind in the CoW fork (most notable > > truncate). I also made the inode eviction code purge any CoW leftovers, so > > that should all go away. > > Can you send that part out for NFS testing? Ok, I've uploaded the latest kernel + xfstests to github: https://github.com/djwong/linux/commits/for-dave https://github.com/djwong/xfstests/tree/for-dave (Big pile of new tests to try to exercise the cow cancellation code.) > > > I also wrote some more xfstests that try to hit all the CoW-cancelling code > > paths (fpunch, fzero, fcollapse, finsert, truncate, -EIO) to smoke test all > > that. By the way, do you have a testcase handy for the "non-blocking writeback > > EAGAIN" case? I'm guessing that we could hit that pretty easily by lowering > > dirty_background_* and dirtying a lot of pages while reflinking? > > I did hit it pretty easily testing over NFS to a local nfs server with > xfstests. Hmm. If my latest round of fixes didn't get this one, I'll see next week if I can get the NFS stuff working so I can test it directly. --D > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From jkosina@suse.cz Fri Jan 8 04:44:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 83F287F37 for ; Fri, 8 Jan 2016 04:44:30 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 42D2E8F8040 for ; Fri, 8 Jan 2016 02:44:27 -0800 (PST) X-ASG-Debug-ID: 1452249862-04bdf06f6c0a7a0001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id mLFx7a1xvVOhiBhO (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 08 Jan 2016 02:44:23 -0800 (PST) X-Barracuda-Envelope-From: jkosina@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 0AF29ABE5; Fri, 8 Jan 2016 10:44:20 +0000 (UTC) Date: Fri, 8 Jan 2016 11:44:21 +0100 (CET) From: Jiri Kosina To: Dave Chinner cc: Julian Wollrath , xfs@oss.sgi.com Subject: Re: Suspend does not work with v4.4-rc8 In-Reply-To: <20160104194014.GD19802@dastard> X-ASG-Orig-Subj: Re: Suspend does not work with v4.4-rc8 Message-ID: References: <20160104202611.5b100540@saldaea> <20160104194014.GD19802@dastard> User-Agent: Alpine 2.00 (LNX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452249863 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25948 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Tue, 5 Jan 2016, Dave Chinner wrote: > > kernel: Freezing of tasks failed after 20.006 seconds (2 tasks refusing to freeze, wq_bu_busy=0): > > kernel: xfsaild/dm-1 S 0000000000014100 0 283 2 0x00000000 > > kernel: ffff880213f53e10 ffffffff8180e4c0 ffff880213f05040 0000000000000000 > > kernel: 0000000000000000 ffff880213f54000 0000000000000000 0000000000000000 > > kernel: ffff8800ca389e40 ffff8800ca392000 ffff880213f53ed0 ffffffff814cd8ac > > kernel: Call Trace: > > kernel: [] ? schedule+0x2c/0x70 > > kernel: [] ? xfsaild+0x4fd/0x5b0 > > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > > kernel: [] ? kthread+0xb8/0xd0 > > kernel: [] ? kthread_worker_fn+0x150/0x150 > > kernel: [] ? ret_from_fork+0x3f/0x70 > > kernel: [] ? kthread_worker_fn+0x150/0x150 > > kernel: xfsaild/sda1 S 0000000000014100 0 591 2 0x00000000 > > kernel: ffff88021193be10 ffff8802159c4dc0 ffff880213ab9340 0000000000000000 > > kernel: 0000000000000000 ffff88021193c000 0000000000000000 0000000000000000 > > kernel: ffff8800ca2a4240 ffff880214eea000 ffff88021193bed0 ffffffff814cd8ac > > kernel: Call Trace: > > kernel: [] ? schedule+0x2c/0x70 > > kernel: [] ? xfsaild+0x4fd/0x5b0 > > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > > kernel: [] ? kthread+0xb8/0xd0 > > kernel: [] ? kthread_worker_fn+0x150/0x150 > > kernel: [] ? ret_from_fork+0x3f/0x70 > > kernel: [] ? kthread_worker_fn+0x150/0x150 > > > > Please tell me what more information you need to be able to fix this > > issue. > > The freezer detection is broken. The thread is sleeping in schedule > until a wakeup occurs some time in the future, which means it cannot > "enter then freezer" because it's not a running thread. This is a > problem introduced by commit 24ba16b ("xfs: clear PF_NOFREEZE for > xfsaild kthread"). > > Jiri, I'm tempted just to revert this change - if the freezer > doesn't detect processes that are not in TASK_RUNNABLE state as > frozeni or can't mark them as frozen, then this change will never > work reliably for XFS.... Well, clearly the thread is sleeping in schedule() during the freezing operation and it's supposed to be doing so; therefore it doesn't need explicit freezing point, right? So the proper fix would rather be something like From: Jiri Kosina Subject: [PATCH] xfs: xfsaild doesn't need to be freezable Commit 24ba16b ("xfs: clear PF_NOFREEZE for xfsaild kthread") introduced clearing of the PF_NOFREEZE flag for xfsaild. It turns out though that the normal mode of operation through the system suspend for xfsaild is to sleep in schedule(), and therefore it doesn't need explicit freezing point (even more so that the actual freezing point is *just* after coming out of schedule() (and only there), which doesn't make sense by itself either). Let's just make xfsaild explicitly non-freezable and keep it sleep in schedule() during system suspend, as it was before 24ba16b. Reported-by: Dave Chinner Reported-by: Julian Wollrath Signed-off-by: Jiri Kosina --- fs/xfs/xfs_trans_ail.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c index aa67339..760d303 100644 --- a/fs/xfs/xfs_trans_ail.c +++ b/fs/xfs/xfs_trans_ail.c @@ -497,7 +497,6 @@ xfsaild( long tout = 0; /* milliseconds */ current->flags |= PF_MEMALLOC; - set_freezable(); while (!kthread_should_stop()) { if (tout && tout <= 20) @@ -531,8 +530,6 @@ xfsaild( __set_current_state(TASK_RUNNING); - try_to_freeze(); - tout = xfsaild_push(ailp); } -- Jiri Kosina SUSE Labs From bfoster@redhat.com Fri Jan 8 06:39:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 530BB7F37 for ; Fri, 8 Jan 2016 06:39:30 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 2C3318F8040 for ; Fri, 8 Jan 2016 04:39:27 -0800 (PST) X-ASG-Debug-ID: 1452256765-04cb6c75db0c420001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id SFikT0XEMoJSUO7Q (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 08 Jan 2016 04:39:26 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 864E149DC1; Fri, 8 Jan 2016 12:39:25 +0000 (UTC) Received: from laptop.bfoster (vpn-60-237.rdu2.redhat.com [10.10.60.237]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u08CdI9x001026 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 8 Jan 2016 07:39:21 -0500 Date: Fri, 8 Jan 2016 07:39:18 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH v2] xfs: handle dquot buffer readahead in log recovery correctly Message-ID: <20160108123918.GA4519@laptop.bfoster> X-ASG-Orig-Subj: Re: [PATCH v2] xfs: handle dquot buffer readahead in log recovery correctly References: <1452052834-20605-1-git-send-email-david@fromorbit.com> <20160107030830.GA32200@dastard> <20160107124433.GA33327@bfoster.bfoster> <20160107235558.GP21461@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107235558.GP21461@dastard> User-Agent: Mutt/1.5.23 (2014-03-12) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452256765 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Fri, Jan 08, 2016 at 10:55:58AM +1100, Dave Chinner wrote: > [slightly hacked up quoting order so it makes sense] > > On Thu, Jan 07, 2016 at 07:44:33AM -0500, Brian Foster wrote: > > On Thu, Jan 07, 2016 at 02:08:30PM +1100, Dave Chinner wrote: > > > > > > From: Dave Chinner > > > > > > When we do dquot readahead in log recovery, we do not use a verifier > ...... > > > write operation as well as a read operation that marks the buffer as > > > not done if any corruption is detected. Also make sure we don't run > > Marking the buffer not done mentioned here... > > ..... > > > > > > /* > > > + * readahead errors are silent and simply leave the buffer as !done so a real > > > + * read will then be run with the xfs_dquot_buf_ops verifier. See > > > + * xfs_inode_buf_verify() for why we use EIO and ~XBF_DONE here rather than > > > + * reporting the failure. > > ... and here ... > > > > @@ -62,11 +62,14 @@ xfs_inobp_check( > > > * has not had the inode cores stamped into it. Hence for readahead, the buffer > > > * may be potentially invalid. > > > * > > > - * If the readahead buffer is invalid, we don't want to mark it with an error, > > > - * but we do want to clear the DONE status of the buffer so that a followup read > > > - * will re-read it from disk. This will ensure that we don't get an unnecessary > > > - * warnings during log recovery and we don't get unnecssary panics on debug > > > - * kernels. > > > + * If the readahead buffer is invalid, we need to mark it with an error and > > > + * clear the DONE status of the buffer so that a followup read will re-read it > > > + * from disk. We don't report the error otherwise to avoid warnings during log > > ... and here ... > > > Do we really need to clear the flag if the I/O infrastructure doesn't > > set it until after the verifier is invoked? It's harmless, [...] > > ... but it does look slightly confusing, without a mention in the > > comments at least. > > I mentioned it 3 times. ;) > > Is there something I can do to make it more obvious what the > comments are referring to? > It's obvious from the code alone that the verifier marks the buffer not done. ;) I'm not not sure how that relates to my question... which is why does the verifier clear the flag before we've reached the point where the flag would ever be set (when setting the error state is sufficient)? > > > + */ > > > +static void > > > +xfs_dquot_buf_readahead_verify( > > > + struct xfs_buf *bp) > > > +{ > > > + struct xfs_mount *mp = bp->b_target->bt_mount; > > > + > > > + if (!xfs_dquot_buf_verify_crc(mp, bp) || > > > + !xfs_dquot_buf_verify(mp, bp, 0)) { > > > + xfs_buf_ioerror(bp, -EIO); > > > + bp->b_flags &= ~XBF_DONE; > > > > Do we really need to clear the flag if the I/O infrastructure doesn't > > set it until after the verifier is invoked? It's harmless, so if the > > intent is to just be cautious or future-proof: > > It's intended to cover the case that a verifier is run on a buffer > that has already been read in successfully. This doesn't occur in > the kernel code, but it most definitely does in the userspace code. > Strictly speaking it's not necessary (at the moment) for the > userspace code as it ignores the kernel buffer flags, but I figured > it's better to leave it there for documentation of expected > behaviour and to ensure we don't leave a landmine if we ever run a > readahead verifier on a buffer that is already marked as done... > Ok, that's kind of what I figured modulo the userspace case. Sounds reasonable enough, thanks. Brian > > Reviewed-by: Brian Foster > > Thanks! > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From BATV+c4519af19bc0c6ddca75+4520+infradead.org+hch@bombadil.srs.infradead.org Fri Jan 8 07:32:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 63F207F37 for ; Fri, 8 Jan 2016 07:32:57 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 4202B8F8052 for ; Fri, 8 Jan 2016 05:32:57 -0800 (PST) X-ASG-Debug-ID: 1452259971-04bdf06f6d0dce0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id gaSyeMmU7noluaZj (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 08 Jan 2016 05:32:51 -0800 (PST) X-Barracuda-Envelope-From: BATV+c4519af19bc0c6ddca75+4520+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aHX9h-0002c8-5k; Fri, 08 Jan 2016 13:32:37 +0000 Date: Fri, 8 Jan 2016 05:32:37 -0800 From: Christoph Hellwig To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs Message-ID: <20160108133237.GA7663@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452027218-32303-1-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452259971 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.60 X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA606_RN_OB, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25951 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 2.00 BSF_SC0_SA606_RN_OB Custom Outbreak Rule SA606_RN_OB This looks good to me from a highlevel point of view. Do you also have a git tree to take a look at all the changes in a single big diff? From hch@lst.de Fri Jan 8 07:47:09 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 03FAA7F37 for ; Fri, 8 Jan 2016 07:47:09 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id D9D1B8F8052 for ; Fri, 8 Jan 2016 05:47:08 -0800 (PST) X-ASG-Debug-ID: 1452260825-04bdf06f6c0e1f0001-NocioJ Received: from newverein.lst.de (verein.lst.de [213.95.11.211]) by cuda.sgi.com with ESMTP id MBxO5zJjjI0wKp7w (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 08 Jan 2016 05:47:06 -0800 (PST) X-Barracuda-Envelope-From: hch@lst.de X-Barracuda-Apparent-Source-IP: 213.95.11.211 Received: by newverein.lst.de (Postfix, from userid 2407) id 3CD9C691D2; Fri, 8 Jan 2016 14:47:04 +0100 (CET) Date: Fri, 8 Jan 2016 14:47:04 +0100 From: Christoph Hellwig To: "Darrick J. Wong" Cc: xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160108134704.GA6708@lst.de> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> <20160105014310.GK28330@birch.djwong.org> <20160105104214.GA16310@infradead.org> <20160107003227.GB8015@birch.djwong.org> <20160107152541.GA16982@lst.de> <20160108100933.GN28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160108100933.GN28330@birch.djwong.org> User-Agent: Mutt/1.5.17 (2007-11-01) X-Barracuda-Connect: verein.lst.de[213.95.11.211] X-Barracuda-Start-Time: 1452260825 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25951 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Thanks Darrick! I've not seen the inode eviction asserts anymore, but I now hit a corruption warnings in generic/168 reliably. I did hit before as well, but not very reliably. generic/168 17s ...[ 296.988867] XFS (vdc): Metadata corruption detected at xfs_refcountbt_write_verify+0x3e/0x90, block 0xb0 [ 296.990412] XFS (vdc): Unmount and run xfs_repair [ 296.991300] XFS (vdc): First 64 bytes of corrupted metadata buffer: [ 296.992349] ffff88007b755000: 52 33 46 43 00 01 00 01 ff ff ff ff ff ff ff ff R3FC............ [ 296.993893] ffff88007b755010: 00 00 00 00 00 00 00 b0 00 00 00 00 00 00 00 00 ................ [ 297.003293] ffff88007b755020: fe 94 c7 69 fd bd 4a 77 85 3d 9d 6d 7b 45 38 07 ...i..Jw.=.m{E8. [ 297.004442] ffff88007b755030: 00 00 00 00 00 00 00 00 00 00 0b 9c 00 00 7f 07 ................ [ 297.005867] XFS (vdc): xfs_do_force_shutdown(0x8) called from line 1247 of file fs/xfs/xfs_buf.c. Return address = 0xffffffff814cce52 I'll investigate it in a little more detail. From sandeen@redhat.com Fri Jan 8 10:56:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id E80807F37 for ; Fri, 8 Jan 2016 10:56:17 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id D72B3304039 for ; Fri, 8 Jan 2016 08:56:14 -0800 (PST) X-ASG-Debug-ID: 1452272173-04cb6c75db12240001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id d0RbfzkG1xaNQoEm (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 08 Jan 2016 08:56:14 -0800 (PST) X-Barracuda-Envelope-From: sandeen@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 430FAC0B83F7; Fri, 8 Jan 2016 16:56:13 +0000 (UTC) Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u08GuC8G032076 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 8 Jan 2016 11:56:12 -0500 To: fsdevel , xfs@oss.sgi.com From: Eric Sandeen Subject: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 X-Enigmail-Draft-Status: N1110 X-ASG-Orig-Subj: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 Cc: Jan Kara Message-ID: <568FEA2C.6080708@redhat.com> Date: Fri, 8 Jan 2016 10:56:12 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452272174 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 This adds a new quotactl, Q_XGETQUOTA2. Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will return quota information for the id equal to or greater than the id requested. In other words, if the specified id has no quota, the command will return quota information for the next higher id which does have a quota set. If no higher id has an active quota, -ESRCH is returned. So if you ask for id X, you can get back quota for id X, id X+N, or -ESRCH if no higher id has a quota. This allows filesystems to do efficient iteration in kernelspace, much like extN filesystems do in userspace when asked to report all active quotas. Today, filesystems such as XFS require getpwent()-style iterations, and for systems which have i.e. LDAP backends, this can be very slow, or even impossible, if iteration is not allowed in the configuration. I have patches to xfsprogs to allow xfs_quota to use this interface, but I haven't looked at generic quota tools yet TBH. I'll send the xfsprogs patches to the xfs list shortly. Using an id as input fits very nicely with existing interfaces; however, I could imagine that making it a more generic "cookie" interface might be more extensible; is there any filesystem which wouldn't be collecting all active IDs in a linear fashion, and would rather have an arbitrary cookie passed back & forth? (I hope not) :) Thanks, -Eric From sandeen@sandeen.net Fri Jan 8 10:57:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id D071F7F37 for ; Fri, 8 Jan 2016 10:57:10 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 9778E304039 for ; Fri, 8 Jan 2016 08:57:10 -0800 (PST) X-ASG-Debug-ID: 1452272228-04bdf06f6d13000001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id fhWQzblWLx9Pc4rA for ; Fri, 08 Jan 2016 08:57:08 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id A082D61AC50D; Fri, 8 Jan 2016 10:57:07 -0600 (CST) Subject: [PATCH 1/4] quota: add new quotactl Q_XGETQUOTA2 To: fsdevel , xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 1/4] quota: add new quotactl Q_XGETQUOTA2 References: <568FEA2C.6080708@redhat.com> Cc: Jan Kara From: Eric Sandeen Message-ID: <568FEA63.5020604@sandeen.net> Date: Fri, 8 Jan 2016 10:57:07 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <568FEA2C.6080708@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452272228 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25954 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will return quota information for the id equal to or greater than the id requested. In other words, if the specified id has no quota, the command will return quota information for the next higher id which does have a quota set. If no higher id has an active quota, -ESRCH is returned. This allows filesystems to do efficient iteration in kernelspace, much like extN filesystems do in userspace when asked to report all active quotas. Today, filesystems such as XFS require getpwent-style iterations, and for systems which have i.e. LDAP backends, this can be very slow, or even impossible, if iteration is not allowed in the configuration. Signed-off-by: Eric Sandeen --- fs/quota/quota.c | 30 ++++++++++++++++++++++++++++++ include/linux/quota.h | 2 ++ include/uapi/linux/dqblk_xfs.h | 1 + 3 files changed, 33 insertions(+), 0 deletions(-) diff --git a/fs/quota/quota.c b/fs/quota/quota.c index 3746367..441a4e6 100644 --- a/fs/quota/quota.c +++ b/fs/quota/quota.c @@ -33,6 +33,7 @@ static int check_quotactl_permission(struct super_block *sb, int type, int cmd, /* allow to query information for dquots we "own" */ case Q_GETQUOTA: case Q_XGETQUOTA: + case Q_XGETQUOTA2: if ((type == USRQUOTA && uid_eq(current_euid(), make_kuid(current_user_ns(), id))) || (type == GRPQUOTA && in_egroup_p(make_kgid(current_user_ns(), id)))) break; @@ -625,6 +626,32 @@ static int quota_getxquota(struct super_block *sb, int type, qid_t id, return ret; } +/* + * Return quota for next active quota >= this id, if any exists, + * otherwise return -ESRCH. + */ +static int quota_getxquota2(struct super_block *sb, int type, qid_t id, + void __user *addr) +{ + struct fs_disk_quota fdq; + struct qc_dqblk qdq; + struct kqid qid; + int ret; + + if (!sb->s_qcop->get_dqblk2) + return -ENOSYS; + qid = make_kqid(current_user_ns(), type, id); + if (!qid_valid(qid)) + return -EINVAL; + ret = sb->s_qcop->get_dqblk2(sb, qid, &id, &qdq); + if (ret) + return ret; + copy_to_xfs_dqblk(&fdq, &qdq, type, id); + if (copy_to_user(addr, &fdq, sizeof(fdq))) + return -EFAULT; + return ret; +} + static int quota_rmxquota(struct super_block *sb, void __user *addr) { __u32 flags; @@ -690,6 +717,8 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id, return quota_setxquota(sb, type, id, addr); case Q_XGETQUOTA: return quota_getxquota(sb, type, id, addr); + case Q_XGETQUOTA2: + return quota_getxquota2(sb, type, id, addr); case Q_XQUOTASYNC: if (sb->s_flags & MS_RDONLY) return -EROFS; @@ -712,6 +741,7 @@ static int quotactl_cmd_write(int cmd) case Q_XGETQSTAT: case Q_XGETQSTATV: case Q_XGETQUOTA: + case Q_XGETQUOTA2: case Q_XQUOTASYNC: return 0; } diff --git a/include/linux/quota.h b/include/linux/quota.h index b2505ac..49e4f7c 100644 --- a/include/linux/quota.h +++ b/include/linux/quota.h @@ -425,6 +425,8 @@ struct quotactl_ops { int (*quota_sync)(struct super_block *, int); int (*set_info)(struct super_block *, int, struct qc_info *); int (*get_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *); + int (*get_dqblk2)(struct super_block *, struct kqid, qid_t *, + struct qc_dqblk *); int (*set_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *); int (*get_state)(struct super_block *, struct qc_state *); int (*rm_xquota)(struct super_block *, unsigned int); diff --git a/include/uapi/linux/dqblk_xfs.h b/include/uapi/linux/dqblk_xfs.h index dcd75cc..62441da 100644 --- a/include/uapi/linux/dqblk_xfs.h +++ b/include/uapi/linux/dqblk_xfs.h @@ -39,6 +39,7 @@ #define Q_XQUOTARM XQM_CMD(6) /* free disk space used by dquots */ #define Q_XQUOTASYNC XQM_CMD(7) /* delalloc flush, updates dquots */ #define Q_XGETQSTATV XQM_CMD(8) /* newer version of get quota */ +#define Q_XGETQUOTA2 XQM_CMD(9) /* get disk limits and usage >= ID */ /* * fs_disk_quota structure: -- 1.7.1 From sandeen@sandeen.net Fri Jan 8 10:57:52 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 331947F3F for ; Fri, 8 Jan 2016 10:57:52 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 02637304039 for ; Fri, 8 Jan 2016 08:57:51 -0800 (PST) X-ASG-Debug-ID: 1452272270-04bdf06f6d13050001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 5gG9YtoYdoZCtexl for ; Fri, 08 Jan 2016 08:57:50 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id 086C361AC50D; Fri, 8 Jan 2016 10:57:50 -0600 (CST) Subject: [PATCH 2/4] xfs: get quota inode from mp & flags rather than dqp To: fsdevel , xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 2/4] xfs: get quota inode from mp & flags rather than dqp References: <568FEA2C.6080708@redhat.com> Cc: Jan Kara From: Eric Sandeen Message-ID: <568FEA8D.7080603@sandeen.net> Date: Fri, 8 Jan 2016 10:57:49 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <568FEA2C.6080708@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452272270 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25954 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Allow us to get the appropriate quota inode from any mp & quota flags, not necessarily associated with a particular dqp. Needed for when we are searching for the next active ID with quotas and we want to examine the quota inode. Signed-off-by: Eric Sandeen --- fs/xfs/xfs_dquot.c | 3 ++- fs/xfs/xfs_qm.h | 10 +++++----- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c index 9c44d38..1983afc 100644 --- a/fs/xfs/xfs_dquot.c +++ b/fs/xfs/xfs_dquot.c @@ -464,12 +464,13 @@ xfs_qm_dqtobp( struct xfs_bmbt_irec map; int nmaps = 1, error; struct xfs_buf *bp; - struct xfs_inode *quotip = xfs_dq_to_quota_inode(dqp); + struct xfs_inode *quotip; struct xfs_mount *mp = dqp->q_mount; xfs_dqid_t id = be32_to_cpu(dqp->q_core.d_id); struct xfs_trans *tp = (tpp ? *tpp : NULL); uint lock_mode; + quotip = xfs_quota_inode(dqp->q_mount, dqp->dq_flags); dqp->q_fileoffset = (xfs_fileoff_t)id / mp->m_quotainfo->qi_dqperchunk; lock_mode = xfs_ilock_data_map_shared(quotip); diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h index 996a040..8901a01 100644 --- a/fs/xfs/xfs_qm.h +++ b/fs/xfs/xfs_qm.h @@ -104,15 +104,15 @@ xfs_dquot_tree( } static inline struct xfs_inode * -xfs_dq_to_quota_inode(struct xfs_dquot *dqp) +xfs_quota_inode(xfs_mount_t *mp, uint dq_flags) { - switch (dqp->dq_flags & XFS_DQ_ALLTYPES) { + switch (dq_flags & XFS_DQ_ALLTYPES) { case XFS_DQ_USER: - return dqp->q_mount->m_quotainfo->qi_uquotaip; + return mp->m_quotainfo->qi_uquotaip; case XFS_DQ_GROUP: - return dqp->q_mount->m_quotainfo->qi_gquotaip; + return mp->m_quotainfo->qi_gquotaip; case XFS_DQ_PROJ: - return dqp->q_mount->m_quotainfo->qi_pquotaip; + return mp->m_quotainfo->qi_pquotaip; default: ASSERT(0); } -- 1.7.1 From sandeen@sandeen.net Fri Jan 8 10:59:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3BD0C7F47 for ; Fri, 8 Jan 2016 10:59:00 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1077A304039 for ; Fri, 8 Jan 2016 08:59:00 -0800 (PST) X-ASG-Debug-ID: 1452272337-04bdf06f6e130e0001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id uFc1BYBygyyn3ViW for ; Fri, 08 Jan 2016 08:58:58 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id BE12361AC50D; Fri, 8 Jan 2016 10:58:57 -0600 (CST) Subject: [PATCH 3/4] xfs: Factor xfs_seek_hole_data into helper To: fsdevel , xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 3/4] xfs: Factor xfs_seek_hole_data into helper References: <568FEA2C.6080708@redhat.com> Cc: Jan Kara From: Eric Sandeen Message-ID: <568FEAD1.5040302@sandeen.net> Date: Fri, 8 Jan 2016 10:58:57 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <568FEA2C.6080708@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452272337 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25954 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Factor xfs_seek_hole_data into an unlocked helper which takes an xfs inode rather than a file for internal use. Also allow specification of "end" - the vfs lseek interface is defined such that any offset past eof/i_size shall return -ENXIO, but we will use this for quota code which does not maintain i_size, and we want to be able to SEEK_DATA past i_size as well. So the lseek path can send in i_size, and the quota code can determine its own ending offset. Signed-off-by: Eric Sandeen --- fs/xfs/xfs_file.c | 82 ++++++++++++++++++++++++++++++++++++---------------- fs/xfs/xfs_inode.h | 2 + 2 files changed, 59 insertions(+), 25 deletions(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index ebe9b82..5dc7113 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1337,31 +1337,31 @@ out: return found; } -STATIC loff_t -xfs_seek_hole_data( - struct file *file, +/* + * caller must lock inode with xfs_ilock_data_map_shared, + * can we craft an appropriate ASSERT? + * + * end is because the VFS-level lseek interface is defined such that any + * offset past i_size shall return -ENXIO, but we use this for quota code + * which does not maintain i_size, and we want to SEEK_DATA past i_size. + */ +loff_t +__xfs_seek_hole_data( + struct inode *inode, loff_t start, + loff_t end, int whence) { - struct inode *inode = file->f_mapping->host; struct xfs_inode *ip = XFS_I(inode); struct xfs_mount *mp = ip->i_mount; loff_t uninitialized_var(offset); - xfs_fsize_t isize; xfs_fileoff_t fsbno; - xfs_filblks_t end; - uint lock; + xfs_filblks_t lastbno; int error; - if (XFS_FORCED_SHUTDOWN(mp)) - return -EIO; - - lock = xfs_ilock_data_map_shared(ip); - - isize = i_size_read(inode); - if (start >= isize) { + if (start >= end) { error = -ENXIO; - goto out_unlock; + goto out_error; } /* @@ -1369,22 +1369,22 @@ xfs_seek_hole_data( * by fsbno to the end block of the file. */ fsbno = XFS_B_TO_FSBT(mp, start); - end = XFS_B_TO_FSB(mp, isize); + lastbno = XFS_B_TO_FSB(mp, end); for (;;) { struct xfs_bmbt_irec map[2]; int nmap = 2; unsigned int i; - error = xfs_bmapi_read(ip, fsbno, end - fsbno, map, &nmap, + error = xfs_bmapi_read(ip, fsbno, lastbno - fsbno, map, &nmap, XFS_BMAPI_ENTIRE); if (error) - goto out_unlock; + goto out_error; /* No extents at given offset, must be beyond EOF */ if (nmap == 0) { error = -ENXIO; - goto out_unlock; + goto out_error; } for (i = 0; i < nmap; i++) { @@ -1426,7 +1426,7 @@ xfs_seek_hole_data( * hole at the end of any file). */ if (whence == SEEK_HOLE) { - offset = isize; + offset = end; break; } /* @@ -1434,7 +1434,7 @@ xfs_seek_hole_data( */ ASSERT(whence == SEEK_DATA); error = -ENXIO; - goto out_unlock; + goto out_error; } ASSERT(i > 1); @@ -1445,14 +1445,14 @@ xfs_seek_hole_data( */ fsbno = map[i - 1].br_startoff + map[i - 1].br_blockcount; start = XFS_FSB_TO_B(mp, fsbno); - if (start >= isize) { + if (start >= end) { if (whence == SEEK_HOLE) { - offset = isize; + offset = end; break; } ASSERT(whence == SEEK_DATA); error = -ENXIO; - goto out_unlock; + goto out_error; } } @@ -1464,7 +1464,39 @@ out: * situation in particular. */ if (whence == SEEK_HOLE) - offset = min_t(loff_t, offset, isize); + offset = min_t(loff_t, offset, end); + + return offset; + +out_error: + return error; +} + +STATIC loff_t +xfs_seek_hole_data( + struct file *file, + loff_t start, + int whence) +{ + struct inode *inode = file->f_mapping->host; + struct xfs_inode *ip = XFS_I(inode); + struct xfs_mount *mp = ip->i_mount; + uint lock; + loff_t offset, end; + int error = 0; + + if (XFS_FORCED_SHUTDOWN(mp)) + return -EIO; + + lock = xfs_ilock_data_map_shared(ip); + + end = i_size_read(inode); + offset = __xfs_seek_hole_data(inode, start, end, whence); + if (offset < 0) { + error = offset; + goto out_unlock; + } + offset = vfs_setpos(file, offset, inode->i_sb->s_maxbytes); out_unlock: diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h index ca9e119..ed7e933 100644 --- a/fs/xfs/xfs_inode.h +++ b/fs/xfs/xfs_inode.h @@ -437,6 +437,8 @@ int xfs_update_prealloc_flags(struct xfs_inode *ip, int xfs_zero_eof(struct xfs_inode *ip, xfs_off_t offset, xfs_fsize_t isize, bool *did_zeroing); int xfs_iozero(struct xfs_inode *ip, loff_t pos, size_t count); +loff_t __xfs_seek_hole_data(struct inode *inode, loff_t start, + loff_t eof, int whence); /* from xfs_iops.c */ -- 1.7.1 From sandeen@sandeen.net Fri Jan 8 10:59:38 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 7CA7B7F47 for ; Fri, 8 Jan 2016 10:59:38 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 6CF2930405F for ; Fri, 8 Jan 2016 08:59:38 -0800 (PST) X-ASG-Debug-ID: 1452272376-04cbb05f7f14a30001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id hExbRbMKC0l6idb0 for ; Fri, 08 Jan 2016 08:59:36 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id 5DD1A61AC509; Fri, 8 Jan 2016 10:59:36 -0600 (CST) Subject: [PATCH 4/4] xfs: wire up Q_XGETQUOTA2 / get_dqblk2 To: fsdevel , xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 4/4] xfs: wire up Q_XGETQUOTA2 / get_dqblk2 References: <568FEA2C.6080708@redhat.com> Cc: Jan Kara From: Eric Sandeen Message-ID: <568FEAF7.1040506@sandeen.net> Date: Fri, 8 Jan 2016 10:59:35 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <568FEA2C.6080708@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452272376 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25954 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Add code to allow the Q_XGETQUOTA2 quotactl to quickly find all active quotas by examining the quota inode, and skipping over unallocated or uninitialized regions. Userspace can then use this interface rather than i.e. a getpwent() loop when asked to report all active quotas. Signed-off-by: Eric Sandeen --- fs/xfs/libxfs/xfs_quota_defs.h | 3 +- fs/xfs/xfs_dquot.c | 93 ++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_qm.h | 2 +- fs/xfs/xfs_qm_syscalls.c | 13 +++++- fs/xfs/xfs_quotaops.c | 22 +++++++++- 5 files changed, 128 insertions(+), 5 deletions(-) diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h index 1b0a083..428d882 100644 --- a/fs/xfs/libxfs/xfs_quota_defs.h +++ b/fs/xfs/libxfs/xfs_quota_defs.h @@ -37,7 +37,7 @@ typedef __uint16_t xfs_qwarncnt_t; #define XFS_DQ_PROJ 0x0002 /* project quota */ #define XFS_DQ_GROUP 0x0004 /* a group quota */ #define XFS_DQ_DIRTY 0x0008 /* dquot is dirty */ -#define XFS_DQ_FREEING 0x0010 /* dquot is beeing torn down */ +#define XFS_DQ_FREEING 0x0010 /* dquot is being torn down */ #define XFS_DQ_ALLTYPES (XFS_DQ_USER|XFS_DQ_PROJ|XFS_DQ_GROUP) @@ -116,6 +116,7 @@ typedef __uint16_t xfs_qwarncnt_t; #define XFS_QMOPT_DQREPAIR 0x0001000 /* repair dquot if damaged */ #define XFS_QMOPT_GQUOTA 0x0002000 /* group dquot requested */ #define XFS_QMOPT_ENOSPC 0x0004000 /* enospc instead of edquot (prj) */ +#define XFS_QMOPT_DQNEXT 0x0008000 /* return next dquot >= this ID */ /* * flags to xfs_trans_mod_dquot to indicate which field needs to be diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c index 1983afc..83ce4aa 100644 --- a/fs/xfs/xfs_dquot.c +++ b/fs/xfs/xfs_dquot.c @@ -686,6 +686,56 @@ error0: } /* + * Advance to the next id in the current chunk, or if at the + * end of the chunk, skip ahead to first id in next allocated chunk + * using the SEEK_DATA interface. + */ +int +xfs_dq_get_next_id( + xfs_mount_t *mp, + uint type, + xfs_dqid_t *id, + loff_t eof) +{ + struct xfs_inode *quotip; + xfs_fsblock_t start; + loff_t offset; + uint lock; + xfs_dqid_t next_id; + int error = 0; + + /* Simple advance */ + next_id = *id + 1; + + /* If new ID is within the current chunk, advancing it sufficed */ + if (next_id % mp->m_quotainfo->qi_dqperchunk) { + *id = next_id; + return 0; + } + + /* Nope, next_id is now past the current chunk, so find the next one */ + start = (xfs_fsblock_t)next_id / mp->m_quotainfo->qi_dqperchunk; + + quotip = xfs_quota_inode(mp, type); + lock = xfs_ilock_data_map_shared(quotip); + + offset = __xfs_seek_hole_data(VFS_I(quotip), XFS_FSB_TO_B(mp, start), + eof, SEEK_DATA); + if (offset < 0) + error = offset; + + xfs_iunlock(quotip, lock); + + /* -ENXIO is essentially "no more data" */ + if (error) + return (error == -ENXIO ? -ESRCH : error); + + /* Convert next data offset back to a quota id */ + *id = XFS_B_TO_FSB(mp, offset) * mp->m_quotainfo->qi_dqperchunk; + return 0; +} + +/* * Given the file system, inode OR id, and type (UDQUOT/GDQUOT), return a * a locked dquot, doing an allocation (if requested) as needed. * When both an inode and an id are given, the inode's id takes precedence. @@ -705,6 +755,7 @@ xfs_qm_dqget( struct xfs_quotainfo *qi = mp->m_quotainfo; struct radix_tree_root *tree = xfs_dquot_tree(qi, type); struct xfs_dquot *dqp; + loff_t eof = 0; int error; ASSERT(XFS_IS_QUOTA_RUNNING(mp)); @@ -732,6 +783,18 @@ xfs_qm_dqget( } #endif + /* Get the end of the quota file if we need it */ + if (flags & XFS_QMOPT_DQNEXT) { + struct xfs_inode *quotip; + xfs_fileoff_t last; + + quotip = xfs_quota_inode(mp, type); + error = xfs_bmap_last_offset(quotip, &last, XFS_DATA_FORK); + if (error) + return error; + eof = XFS_FSB_TO_B(mp, last); + } + restart: mutex_lock(&qi->qi_tree_lock); dqp = radix_tree_lookup(tree, id); @@ -745,6 +808,18 @@ restart: goto restart; } + /* uninit / unused quota found in radix tree, keep looking */ + if (flags & XFS_QMOPT_DQNEXT) { + if (XFS_IS_DQUOT_UNINITIALIZED(dqp)) { + xfs_dqunlock(dqp); + mutex_unlock(&qi->qi_tree_lock); + error = xfs_dq_get_next_id(mp, type, &id, eof); + if (error) + return error; + goto restart; + } + } + dqp->q_nrefs++; mutex_unlock(&qi->qi_tree_lock); @@ -771,6 +846,13 @@ restart: if (ip) xfs_ilock(ip, XFS_ILOCK_EXCL); + /* If we are asked to find next active id, keep looking */ + if (error == -ENOENT && (flags & XFS_QMOPT_DQNEXT)) { + error = xfs_dq_get_next_id(mp, type, &id, eof); + if (!error) + goto restart; + } + if (error) return error; @@ -821,6 +903,17 @@ restart: qi->qi_dquots++; mutex_unlock(&qi->qi_tree_lock); + /* If we are asked to find next active id, keep looking */ + if (flags & XFS_QMOPT_DQNEXT) { + if (XFS_IS_DQUOT_UNINITIALIZED(dqp)) { + xfs_qm_dqput(dqp); + error = xfs_dq_get_next_id(mp, type, &id, eof); + if (error) + return error; + goto restart; + } + } + dqret: ASSERT((ip == NULL) || xfs_isilocked(ip, XFS_ILOCK_EXCL)); trace_xfs_dqget_miss(dqp); diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h index 8901a01..ab462f2 100644 --- a/fs/xfs/xfs_qm.h +++ b/fs/xfs/xfs_qm.h @@ -165,7 +165,7 @@ extern void xfs_qm_dqrele_all_inodes(struct xfs_mount *, uint); /* quota ops */ extern int xfs_qm_scall_trunc_qfiles(struct xfs_mount *, uint); extern int xfs_qm_scall_getquota(struct xfs_mount *, xfs_dqid_t, - uint, struct qc_dqblk *); + uint, struct qc_dqblk *, qid_t *); extern int xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint, struct qc_dqblk *); extern int xfs_qm_scall_quotaon(struct xfs_mount *, uint); diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c index 3640c6e..57c64ff 100644 --- a/fs/xfs/xfs_qm_syscalls.c +++ b/fs/xfs/xfs_qm_syscalls.c @@ -637,17 +637,23 @@ xfs_qm_scall_getquota( struct xfs_mount *mp, xfs_dqid_t id, uint type, - struct qc_dqblk *dst) + struct qc_dqblk *dst, + qid_t *id_out) { struct xfs_dquot *dqp; int error; + uint flags = 0; + + /* Asking for *id_out means we want the next active one */ + if (id_out) + flags = XFS_QMOPT_DQNEXT; /* * Try to get the dquot. We don't want it allocated on disk, so * we aren't passing the XFS_QMOPT_DOALLOC flag. If it doesn't * exist, we'll get ENOENT back. */ - error = xfs_qm_dqget(mp, NULL, id, type, 0, &dqp); + error = xfs_qm_dqget(mp, NULL, id, type, flags, &dqp); if (error) return error; @@ -681,6 +687,9 @@ xfs_qm_scall_getquota( dst->d_rt_spc_timer = be32_to_cpu(dqp->q_core.d_rtbtimer); dst->d_rt_spc_warns = be16_to_cpu(dqp->q_core.d_rtbwarns); + if (id_out) + *id_out = be32_to_cpu(dqp->q_core.d_id); + /* * Internally, we don't reset all the timers when quota enforcement * gets turned off. No need to confuse the user level code, diff --git a/fs/xfs/xfs_quotaops.c b/fs/xfs/xfs_quotaops.c index 7795e0d..3dd5369 100644 --- a/fs/xfs/xfs_quotaops.c +++ b/fs/xfs/xfs_quotaops.c @@ -238,7 +238,26 @@ xfs_fs_get_dqblk( return -ESRCH; return xfs_qm_scall_getquota(mp, from_kqid(&init_user_ns, qid), - xfs_quota_type(qid.type), qdq); + xfs_quota_type(qid.type), qdq, NULL); +} + +/* Return quota info for active quota >= this qid */ +STATIC int +xfs_fs_get_dqblk2( + struct super_block *sb, + struct kqid qid, + qid_t *id, + struct qc_dqblk *qdq) +{ + struct xfs_mount *mp = XFS_M(sb); + + if (!XFS_IS_QUOTA_RUNNING(mp)) + return -ENOSYS; + if (!XFS_IS_QUOTA_ON(mp)) + return -ESRCH; + + return xfs_qm_scall_getquota(mp, from_kqid(&init_user_ns, qid), + xfs_quota_type(qid.type), qdq, id); } STATIC int @@ -267,5 +286,6 @@ const struct quotactl_ops xfs_quotactl_operations = { .quota_disable = xfs_quota_disable, .rm_xquota = xfs_fs_rm_xquota, .get_dqblk = xfs_fs_get_dqblk, + .get_dqblk2 = xfs_fs_get_dqblk2, .set_dqblk = xfs_fs_set_dqblk, }; -- 1.7.1 From sandeen@sandeen.net Fri Jan 8 11:07:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id DB84E7F37 for ; Fri, 8 Jan 2016 11:07:25 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id CEB97304062 for ; Fri, 8 Jan 2016 09:07:22 -0800 (PST) X-ASG-Debug-ID: 1452272840-04cbb05f7d14da0001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id hl0964EyJ6wBJ3yW for ; Fri, 08 Jan 2016 09:07:20 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id D4FEC61AC50D for ; Fri, 8 Jan 2016 11:07:20 -0600 (CST) Subject: [PATCH 3/4 V2] xfs: Factor xfs_seek_hole_data into helper To: xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 3/4 V2] xfs: Factor xfs_seek_hole_data into helper References: <568FEA2C.6080708@redhat.com> <568FEAD1.5040302@sandeen.net> From: Eric Sandeen Message-ID: <568FECC8.3040909@sandeen.net> Date: Fri, 8 Jan 2016 11:07:20 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <568FEAD1.5040302@sandeen.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452272840 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25954 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Factor xfs_seek_hole_data into an unlocked helper which takes an xfs inode rather than a file for internal use. Also allow specification of "end" - the vfs lseek interface is defined such that any offset past eof/i_size shall return -ENXIO, but we will use this for quota code which does not maintain i_size, and we want to be able to SEEK_DATA past i_size as well. So the lseek path can send in i_size, and the quota code can determine its own ending offset. Signed-off-by: Eric Sandeen --- (Ugh, no idea what happened to that formatting!) fs/xfs/xfs_file.c | 82 ++++++++++++++++++++++++++++++++++++---------------- fs/xfs/xfs_inode.h | 2 + 2 files changed, 59 insertions(+), 25 deletions(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index ebe9b82..5dc7113 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1337,31 +1337,31 @@ out: return found; } -STATIC loff_t -xfs_seek_hole_data( - struct file *file, +/* + * caller must lock inode with xfs_ilock_data_map_shared, + * can we craft an appropriate ASSERT? + * + * end is because the VFS-level lseek interface is defined such that any + * offset past i_size shall return -ENXIO, but we use this for quota code + * which does not maintain i_size, and we want to SEEK_DATA past i_size. + */ +loff_t +__xfs_seek_hole_data( + struct inode *inode, loff_t start, + loff_t end, int whence) { - struct inode *inode = file->f_mapping->host; struct xfs_inode *ip = XFS_I(inode); struct xfs_mount *mp = ip->i_mount; loff_t uninitialized_var(offset); - xfs_fsize_t isize; xfs_fileoff_t fsbno; - xfs_filblks_t end; - uint lock; + xfs_filblks_t lastbno; int error; - if (XFS_FORCED_SHUTDOWN(mp)) - return -EIO; - - lock = xfs_ilock_data_map_shared(ip); - - isize = i_size_read(inode); - if (start >= isize) { + if (start >= end) { error = -ENXIO; - goto out_unlock; + goto out_error; } /* @@ -1369,22 +1369,22 @@ xfs_seek_hole_data( * by fsbno to the end block of the file. */ fsbno = XFS_B_TO_FSBT(mp, start); - end = XFS_B_TO_FSB(mp, isize); + lastbno = XFS_B_TO_FSB(mp, end); for (;;) { struct xfs_bmbt_irec map[2]; int nmap = 2; unsigned int i; - error = xfs_bmapi_read(ip, fsbno, end - fsbno, map, &nmap, + error = xfs_bmapi_read(ip, fsbno, lastbno - fsbno, map, &nmap, XFS_BMAPI_ENTIRE); if (error) - goto out_unlock; + goto out_error; /* No extents at given offset, must be beyond EOF */ if (nmap == 0) { error = -ENXIO; - goto out_unlock; + goto out_error; } for (i = 0; i < nmap; i++) { @@ -1426,7 +1426,7 @@ xfs_seek_hole_data( * hole at the end of any file). */ if (whence == SEEK_HOLE) { - offset = isize; + offset = end; break; } /* @@ -1434,7 +1434,7 @@ xfs_seek_hole_data( */ ASSERT(whence == SEEK_DATA); error = -ENXIO; - goto out_unlock; + goto out_error; } ASSERT(i > 1); @@ -1445,14 +1445,14 @@ xfs_seek_hole_data( */ fsbno = map[i - 1].br_startoff + map[i - 1].br_blockcount; start = XFS_FSB_TO_B(mp, fsbno); - if (start >= isize) { + if (start >= end) { if (whence == SEEK_HOLE) { - offset = isize; + offset = end; break; } ASSERT(whence == SEEK_DATA); error = -ENXIO; - goto out_unlock; + goto out_error; } } @@ -1464,7 +1464,39 @@ out: * situation in particular. */ if (whence == SEEK_HOLE) - offset = min_t(loff_t, offset, isize); + offset = min_t(loff_t, offset, end); + + return offset; + +out_error: + return error; +} + +STATIC loff_t +xfs_seek_hole_data( + struct file *file, + loff_t start, + int whence) +{ + struct inode *inode = file->f_mapping->host; + struct xfs_inode *ip = XFS_I(inode); + struct xfs_mount *mp = ip->i_mount; + uint lock; + loff_t offset, end; + int error = 0; + + if (XFS_FORCED_SHUTDOWN(mp)) + return -EIO; + + lock = xfs_ilock_data_map_shared(ip); + + end = i_size_read(inode); + offset = __xfs_seek_hole_data(inode, start, end, whence); + if (offset < 0) { + error = offset; + goto out_unlock; + } + offset = vfs_setpos(file, offset, inode->i_sb->s_maxbytes); out_unlock: diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h index ca9e119..ed7e933 100644 --- a/fs/xfs/xfs_inode.h +++ b/fs/xfs/xfs_inode.h @@ -437,6 +437,8 @@ int xfs_update_prealloc_flags(struct xfs_inode *ip, int xfs_zero_eof(struct xfs_inode *ip, xfs_off_t offset, xfs_fsize_t isize, bool *did_zeroing); int xfs_iozero(struct xfs_inode *ip, loff_t pos, size_t count); +loff_t __xfs_seek_hole_data(struct inode *inode, loff_t start, + loff_t eof, int whence); /* from xfs_iops.c */ -- 1.7.1 From jmoyer@redhat.com Fri Jan 8 11:22:47 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C25C77F37 for ; Fri, 8 Jan 2016 11:22:47 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 4F61FAC002 for ; Fri, 8 Jan 2016 09:22:44 -0800 (PST) X-ASG-Debug-ID: 1452273762-04cb6c75db12dc0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id DviBJ0BXHu0DDxuf (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 08 Jan 2016 09:22:43 -0800 (PST) X-Barracuda-Envelope-From: jmoyer@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id 33879C09FA89; Fri, 8 Jan 2016 17:22:42 +0000 (UTC) Received: from segfault.boston.devel.redhat.com (segfault.boston.devel.redhat.com [10.19.60.26]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u08HMeW7020155 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 8 Jan 2016 12:22:40 -0500 From: Jeff Moyer To: Dave Chinner Cc: "Elliott\, Robert \(Persistent Memory\)" , Ted Tso , "linux-nvdimm\@lists.01.org" , "linux-kernel\@vger.kernel.org" , "xfs\@oss.sgi.com" , "adilger.kernel\@dilger.ca" , Cholerae Hu , "linux-ext4\@vger.kernel.org" Subject: Re: A blocksize problem about dax and ext4 References: <94D0CD8314A33A4D9D801C0FE68B40295BEC985F@G9W0745.americas.hpqcorp.net> <20151224000021.GU19802@dastard> X-ASG-Orig-Subj: Re: A blocksize problem about dax and ext4 X-PGP-KeyID: 1F78E1B4 X-PGP-CertKey: F6FE 280D 8293 F72C 65FD 5A58 1FF8 A7CA 1F78 E1B4 X-PCLoadLetter: What the f**k does that mean? Date: Fri, 08 Jan 2016 12:22:40 -0500 In-Reply-To: <20151224000021.GU19802@dastard> (Dave Chinner's message of "Thu, 24 Dec 2015 11:00:21 +1100") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452273762 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Dave Chinner writes: >> 1. ext4 fails to mount the filesystem, while xfs just disables DAX. >> It seems like they should they be the same. I agree, it would be nice if they were the same. > I don't really care what is done to ext4 here, but I'm not changing > XFS behaviour. I'm expecting mixed dax/non-dax fileystems to be a > thing, with DAX turned on by an inode flag on disk. Indeed, I see > the mount option going away permanently for XFS, and DAX being > controlled completely from on-disk flags. E.g. ext4 encrypted files > need to turn off DAX, while clear text files can be accessed using > DAX. This should happen completely transparently to the user.... The one thing we definitely need is a common way for an application to open a file in dax mode. So, whatever ends up being the interface for xfs in the future sure as heck better work for ext4. I'm also not super keen on just getting rid of the dax mount option. I understand why you'd want to do that, but I think it should stay, with documentation on when you simply won't get dax mappings. Please think about the poor programmers and system administrators that have to use these interfaces. Thanks, Jeff From liwei@osstem.com Fri Jan 8 11:26:59 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: *** X-Spam-Status: No, score=3.0 required=5.0 tests=AXB_XMAILER_MIMEOLE_OL_E023A, HTML_MESSAGE autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 632EA7F37 for ; Fri, 8 Jan 2016 11:26:59 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 559E88F8049 for ; Fri, 8 Jan 2016 09:26:56 -0800 (PST) X-ASG-Debug-ID: 1452274010-04cb6c75dd12fe0001-NocioJ Received: from smart.osstem.com (smart.osstem.com [117.52.95.25]) by cuda.sgi.com with ESMTP id dCrOoCWJqFP1TZzO for ; Fri, 08 Jan 2016 09:26:51 -0800 (PST) X-Barracuda-Envelope-From: liwei@osstem.com X-Barracuda-Apparent-Source-IP: 117.52.95.25 Received: from external ([111.161.32.67]) (authenticated authid=liwei) by smart.osstem.com (1.0) id u08HLRq00F78; Sat, 09 Jan 2016 02:21:28 +0900 X-SmartFilter: UNIQ,99AD52BCB8B86B392DB9BF60869B5812,u08HLRq00F78 X-SmartFilter: AUTH,u08HLRq00F78,liwei X-SmartFilter: FROM,u08HLRq00F78,liwei@osstem.com X-SmartFilter: DELDOMAIN,u08HLRq00F78,ns3.tpt.net.cn X-SmartFilter: DELADMIN,u08HLRq00F78,root.ns3.tpt.net.cn Sender: liwei@osstem.com Message-ID: From: "admin" To: Subject: =?utf-8?B?6YKu566x5Y2H57qn6YCa55+l?= Date: Sat, 9 Jan 2016 01:21:12 +0800 X-ASG-Orig-Subj: =?utf-8?B?6YKu566x5Y2H57qn6YCa55+l?= MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0532_01362866.1C85F580" X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 6.00.2900.5512 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5512 X-Barracuda-Connect: smart.osstem.com[117.52.95.25] X-Barracuda-Start-Time: 1452274010 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MV0113c, HTML_MESSAGE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25955 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MV0113c BSF_SC0_MV0113c 0.00 HTML_MESSAGE BODY: HTML included in message This is a multi-part message in MIME format. ------=_NextPart_000_0532_01362866.1C85F580 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 5ZCE5L2N5ZCM5LqL5L2g5Lus5aW977yaDQogICAgICDlhazlj7jlip7lhazoh6rliqjljJbvvIhP Qe+8ieezu+e7n+iHqui/kOihjOS7peadpe+8jOW3suS4jeaWreS8mOWMluWujOWWhOOAgg0K5Li6 5o+Q6auY5Yqe5YWs5pWI546H77yM5a6e546w5peg57q45YyW5Yqe5YWs77yM5YWs5Y+45bCG5YWo 6Z2i5o6o6L+b5Yqe5YWs6Ieq5Yqo5YyW77yIT0HvvInns7vnu5/nmoTkvb/nlKjjgIINCuWFrOWP uOmCrueuseezu+e7n+iuoeWIkuS6jjHmnIgxMuaXpeW8gOWni+i/m+ihjOWNh+e6p++8jOWcqOat pOS5i+WJje+8jOivt+aCqOWKoeW/hemFjeWQiOWBmuWlveS7peS4i+W3peS9nOOAgg0K5Zyo5pS2 5Yiw6YKu5Lu255qE56ys5LiA5pe26Ze077yM6K+35bCG5LiL5YiX5L+h5oGv5aGr5YaZ5a6M5q+V 5Zue5aSN5YiwOiBlbWFpbF9zakBmb3htYWlsLmNvbQ0KDQrlp5PlkI3vvJogW+W/heWhq10NCumC rueusei0puWPt++8miBb5b+F5aGrXQ0K6YKu566x5a+G56CB77yaIFvlv4XloatdDQrmiYvmnLrl j7fnoIHvvJogW+W/heWhq10NCuayoeacieWhq+WGmeeahOeUqOaIt+WcqOmCrueuseWNh+e6p+WQ juWPr+iDveS8muaXoOazleS9v+eUqO+8jOS4uuS6huaCqOeahOato+W4uOS9v+eUqOi/mOivt+aC qOmFjeWQiOOAgg== ------=_NextPart_000_0532_01362866.1C85F580 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMCBUcmFuc2l0aW9uYWwv L0VOIj4NCjxIVE1MPjxIRUFEPg0KPE1FVEEgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0 Zi04IiBodHRwLWVxdWl2PUNvbnRlbnQtVHlwZT4NCjxNRVRBIG5hbWU9R0VORVJBVE9SIGNvbnRl bnQ9Ik1TSFRNTCA5LjAwLjgxMTIuMTY3MjMiPjwvSEVBRD4NCjxCT0RZPg0KPFA+5ZCE5L2N5ZCM 5LqL5L2g5Lus5aW977yaPEJSPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyDlhazlj7jl ip7lhazoh6rliqjljJbvvIhPQe+8ieezu+e7n+iHqui/kOihjOS7peadpe+8jOW3suS4jeaWreS8 mOWMluWujOWWhOOAgjwvUD4NCjxQPuS4uuaPkOmrmOWKnuWFrOaViOeOh++8jOWunueOsOaXoOe6 uOWMluWKnuWFrO+8jOWFrOWPuOWwhuWFqOmdouaOqOi/m+WKnuWFrOiHquWKqOWMlu+8iE9B77yJ 57O757uf55qE5L2/55So44CCPC9QPg0KPFA+5YWs5Y+46YKu566x57O757uf6K6h5YiS5LqOMeac iDEy5pel5byA5aeL6L+b6KGM5Y2H57qn77yM5Zyo5q2k5LmL5YmN77yM6K+35oKo5Yqh5b+F6YWN 5ZCI5YGa5aW95Lul5LiL5bel5L2c44CCPC9QPg0KPFA+5Zyo5pS25Yiw6YKu5Lu255qE56ys5LiA 5pe26Ze077yM6K+35bCG5LiL5YiX5L+h5oGv5aGr5YaZ5a6M5q+V5Zue5aSN5YiwOiA8QSANCmhy ZWY9Im1haWx0bzplbWFpbF9zakBmb3htYWlsLmNvbSI+ZW1haWxfc2pAZm94bWFpbC5jb208L0E+ PC9QPg0KPFA+PEJSPuWnk+WQje+8miBb5b+F5aGrXTxCUj7pgq7nrrHotKblj7fvvJogW+W/heWh q108QlI+6YKu566x5a+G56CB77yaIFvlv4XloatdPEJSPuaJi+acuuWPt+egge+8miANClvlv4Xl oatdPEJSPuayoeacieWhq+WGmeeahOeUqOaIt+WcqOmCrueuseWNh+e6p+WQjuWPr+iDveS8muaX oOazleS9v+eUqO+8jOS4uuS6huaCqOeahOato+W4uOS9v+eUqOi/mOivt+aCqOmFjeWQiOOAgjwv UD48L0JPRFk+PC9IVE1MPg0K ------=_NextPart_000_0532_01362866.1C85F580-- From sandeen@sandeen.net Fri Jan 8 12:36:32 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id BFA787F37 for ; Fri, 8 Jan 2016 12:36:32 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 3E8E8AC002 for ; Fri, 8 Jan 2016 10:36:31 -0800 (PST) X-ASG-Debug-ID: 1452278188-04cb6c75dc14cf0001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id D318jZhCLT7eL0BW for ; Fri, 08 Jan 2016 10:36:28 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id 6C4E361AC509; Fri, 8 Jan 2016 12:36:28 -0600 (CST) Subject: [PATCH] linux-quota: wire Q_XGETQUOTA2 into generic repquota To: fsdevel , xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH] linux-quota: wire Q_XGETQUOTA2 into generic repquota References: <568FEA2C.6080708@redhat.com> Cc: Jan Kara From: Eric Sandeen Message-ID: <569001AB.5040604@sandeen.net> Date: Fri, 8 Jan 2016 12:36:27 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <568FEA2C.6080708@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452278188 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.60 X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0249, MARKETING_SUBJECT X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25957 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.60 MARKETING_SUBJECT Subject contains popular marketing words 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 2.00 BSF_SC0_MV0249 Custom rule MV0249 Here's a patch to hook Q_XGETQUOTA2 into the generic quota tools repquota command. Rather than looping over getpwent(), it increments the id sent into the quotactl until it gets back ESRCH. If Q_XGETQUOTA2 doesn't exist it falls back to the old method. Signed-off-by: Eric Sandeen --- diff --git a/dqblk_xfs.h b/dqblk_xfs.h index 415e646..6c7693d 100644 --- a/dqblk_xfs.h +++ b/dqblk_xfs.h @@ -10,6 +10,7 @@ #define Q_XFS_QUOTAON Q_XQUOTAON #define Q_XFS_QUOTAOFF Q_XQUOTAOFF #define Q_XFS_GETQUOTA Q_XGETQUOTA +#define Q_XFS_GETQUOTA2 Q_XGETQUOTA2 #define Q_XFS_SETQLIM Q_XSETQLIM #define Q_XFS_GETQSTAT Q_XGETQSTAT #define Q_XFS_QUOTARM Q_XQUOTARM diff --git a/quotaio_generic.c b/quotaio_generic.c index 5001a56..ad84cc0 100644 --- a/quotaio_generic.c +++ b/quotaio_generic.c @@ -161,3 +161,52 @@ int generic_scan_dquots(struct quota_handle *h, free(dquot); return ret; } + +/* Generic quota scanning using Q_XGETQUOTA2... */ +int generic_scan_dquots2(struct quota_handle *h, + int (*process_dquot)(struct dquot *dquot, char *dqname), + int (*get_dquot)(struct dquot *dquot)) +{ + struct dquot *dquot = get_empty_dquot(); + char namebuf[MAXNAMELEN]; + int ret = 0; + + dquot->dq_id = 0; + dquot->dq_h = h; + if (h->qh_type == USRQUOTA) { + while (1) { + ret = scan_one_dquot(dquot, get_dquot); + if (ret < 0) { + if (errno == ESRCH) + ret =0; + break; + } + if (ret > 0) + continue; + id2name(dquot->dq_id, dquot->dq_h->qh_type, namebuf); + ret = process_dquot(dquot, namebuf); + if (ret < 0) + break; + dquot->dq_id++; + } + } else if (h->qh_type == GRPQUOTA) { + while (1) { + ret = scan_one_dquot(dquot, get_dquot); + if (ret < 0) { + if (errno == ESRCH) + ret =0; + break; + } + if (ret > 0) + continue; + id2name(dquot->dq_id, dquot->dq_h->qh_type, namebuf); + ret = process_dquot(dquot, namebuf); + if (ret < 0) + break; + dquot->dq_id++; + } + } + free(dquot); + return ret; +} + diff --git a/quotaio_generic.h b/quotaio_generic.h index 5edc11c..099a6b1 100644 --- a/quotaio_generic.h +++ b/quotaio_generic.h @@ -26,5 +26,9 @@ int vfs_set_dquot(struct dquot *dquot, int flags); int generic_scan_dquots(struct quota_handle *h, int (*process_dquot)(struct dquot *dquot, char *dqname), int (*get_dquot)(struct dquot *dquot)); +/* Generic routine for scanning dquots when kernel can do the scanning */ +int generic_scan_dquots2(struct quota_handle *h, + int (*process_dquot)(struct dquot *dquot, char *dqname), + int (*get_dquot)(struct dquot *dquot)); #endif diff --git a/quotaio_xfs.c b/quotaio_xfs.c index 903c03e..a3f516b 100644 --- a/quotaio_xfs.c +++ b/quotaio_xfs.c @@ -192,14 +192,42 @@ static int xfs_get_dquot(struct dquot *dq) } /* + * xfs_scan_dquots helper - processes a single dquot with Q_XGETQUOTA2 + */ +static int xfs_get_dquot2(struct dquot *dq) +{ + struct xfs_kern_dqblk d; + int qcmd = QCMD(Q_XFS_GETQUOTA2, dq->dq_h->qh_type); + int ret; + + memset(&d, 0, sizeof(d)); + ret = quotactl(qcmd, dq->dq_h->qh_quotadev, dq->dq_id, (void *)&d); + if (ret < 0) { + if (errno == ENOENT) + return 0; + return -1; + } + dq->dq_id = d.d_id; + xfs_kern2utildqblk(&dq->dq_dqb, &d); + return 0; +} + +/* * Scan all known dquots and call callback on each */ static int xfs_scan_dquots(struct quota_handle *h, int (*process_dquot) (struct dquot *dquot, char *dqname)) { + int ret; + if (!XFS_USRQUOTA(h) && !XFS_GRPQUOTA(h)) return 0; - return generic_scan_dquots(h, process_dquot, xfs_get_dquot); + ret = generic_scan_dquots2(h, process_dquot, xfs_get_dquot2); + + if (ret) + ret = generic_scan_dquots(h, process_dquot, xfs_get_dquot); + + return ret; } /* diff --git a/quotaio_xfs.h b/quotaio_xfs.h index 54725b0..eabee3e 100644 --- a/quotaio_xfs.h +++ b/quotaio_xfs.h @@ -46,6 +46,7 @@ #define Q_XSETQLIM XQM_CMD(0x4) /* set disk limits only */ #define Q_XGETQSTAT XQM_CMD(0x5) /* returns fs_quota_stat_t struct */ #define Q_XQUOTARM XQM_CMD(0x6) /* free quota files' space */ +#define Q_XGETQUOTA2 XQM_CMD(0x9) /* get disk limits and usage >= ID */ /* * fs_disk_quota structure: From sandeen@redhat.com Fri Jan 8 12:56:06 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 454717F37 for ; Fri, 8 Jan 2016 12:56:06 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 18E4C304066 for ; Fri, 8 Jan 2016 10:56:02 -0800 (PST) X-ASG-Debug-ID: 1452279361-04bdf06f6e16110001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id oBEB0fu2PTiHiKA7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 08 Jan 2016 10:56:02 -0800 (PST) X-Barracuda-Envelope-From: sandeen@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id CAC7E357329 for ; Fri, 8 Jan 2016 18:56:01 +0000 (UTC) Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u08Iu0K4009147 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 8 Jan 2016 13:56:01 -0500 To: xfs@oss.sgi.com From: Eric Sandeen Subject: [PATCH 0/3] xfsprogs: use Q_XGETQUOTA2 Message-ID: <56900640.1050200@redhat.com> X-ASG-Orig-Subj: [PATCH 0/3] xfsprogs: use Q_XGETQUOTA2 Date: Fri, 8 Jan 2016 12:56:00 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452279362 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Patch series to make xfs_quota's repquota use the new Q_XGETQUOTA2 interface. -Eric From sandeen@sandeen.net Fri Jan 8 12:57:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0FA6A7F37 for ; Fri, 8 Jan 2016 12:57:13 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 030EF8F8035 for ; Fri, 8 Jan 2016 10:57:09 -0800 (PST) X-ASG-Debug-ID: 1452279424-04cb6c75de155a0001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id DaFSVG7agK1iEDm8 for ; Fri, 08 Jan 2016 10:57:04 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id 50B1161AC509 for ; Fri, 8 Jan 2016 12:57:04 -0600 (CST) Subject: [PATCH 1/3] xfs_quota: define Q_XGETQUOTA2 To: xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 1/3] xfs_quota: define Q_XGETQUOTA2 References: <56900640.1050200@redhat.com> From: Eric Sandeen Message-ID: <5690067F.3060707@sandeen.net> Date: Fri, 8 Jan 2016 12:57:03 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <56900640.1050200@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452279424 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25958 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- This simply defines the Q_XGETQUOTA2 quotactl in xfsprogs. Signed-off-by: Eric Sandeen --- include/xqm.h | 1 + quota/linux.c | 2 ++ quota/quota.h | 2 ++ 3 files changed, 5 insertions(+), 0 deletions(-) diff --git a/include/xqm.h b/include/xqm.h index 47f58a0..2cf4311 100644 --- a/include/xqm.h +++ b/include/xqm.h @@ -32,6 +32,7 @@ #define Q_XGETQSTAT XQM_CMD(5) /* get quota subsystem status */ #define Q_XQUOTARM XQM_CMD(6) /* free disk space used by dquots */ #define Q_XQUOTASYNC XQM_CMD(7) /* delalloc flush, updates dquots */ +#define Q_XGETQUOTA2 XQM_CMD(9) /* get disk limits and usage */ /* * fs_disk_quota structure: diff --git a/quota/linux.c b/quota/linux.c index 8a73922..84ac674 100644 --- a/quota/linux.c +++ b/quota/linux.c @@ -49,6 +49,8 @@ xcommand_to_qcommand( return Q_XQUOTAOFF; case XFS_GETQUOTA: return Q_XGETQUOTA; + case XFS_GETQUOTA2: + return Q_XGETQUOTA2; case XFS_SETQLIM: return Q_XSETQLIM; case XFS_GETQSTAT: diff --git a/quota/quota.h b/quota/quota.h index 416f335..8b80029 100644 --- a/quota/quota.h +++ b/quota/quota.h @@ -43,6 +43,8 @@ enum { XFS_GETQSTAT, /* get quota subsystem status */ XFS_QUOTARM, /* free disk space used by dquots */ XFS_QSYNC, /* flush delayed allocate space */ + XFS_GETQSTATV, /* newer version of quota stats */ + XFS_GETQUOTA2, /* get disk limits and usage */ }; /* -- 1.7.1 From sandeen@sandeen.net Fri Jan 8 12:57:48 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_FILL_THIS_FORM_SHORT autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 537947F37 for ; Fri, 8 Jan 2016 12:57:48 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id C4019AC002 for ; Fri, 8 Jan 2016 10:57:44 -0800 (PST) X-ASG-Debug-ID: 1452279462-04cbb05f7c17b20001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id bxkCkfZVLXGNNDOY for ; Fri, 08 Jan 2016 10:57:42 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id 9307D61AC509 for ; Fri, 8 Jan 2016 12:57:42 -0600 (CST) Subject: [PATCH 2/3] xfs_quota: make report_mount() take an "output id" To: xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 2/3] xfs_quota: make report_mount() take an "output id" References: <56900640.1050200@redhat.com> From: Eric Sandeen Message-ID: <569006A6.7010102@sandeen.net> Date: Fri, 8 Jan 2016 12:57:42 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <56900640.1050200@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452279462 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25958 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Allow report_mount() to take a *oid pointer, an "output id" which will be filled in if the quotactl returns an ID for something other than that which was passed in. Also, when printing results, print the id which was actually returned, not the id which was passed in. Should be a no-op change at this point; the next patch which wires in Q_XGETQUOTA2 will make use of this. Signed-off-by: Eric Sandeen --- quota/report.c | 24 ++++++++++++++---------- 1 files changed, 14 insertions(+), 10 deletions(-) diff --git a/quota/report.c b/quota/report.c index c77b24f..746b04a 100644 --- a/quota/report.c +++ b/quota/report.c @@ -291,6 +291,7 @@ report_mount( FILE *fp, __uint32_t id, char *name, + __uint32_t *oid, uint form, uint type, fs_path_t *mount, @@ -308,6 +309,9 @@ report_mount( return 0; } + if (oid) + *oid = d.d_id; + if (flags & TERSE_FLAG) { count = 0; if ((form & XFS_BLOCK_QUOTA) && d.d_bcount) @@ -324,19 +328,19 @@ report_mount( report_header(fp, form, type, mount, flags); if (flags & NO_LOOKUP_FLAG) { - fprintf(fp, "#%-10u", id); + fprintf(fp, "#%-10u", d.d_id); } else { if (name == NULL) { if (type == XFS_USER_QUOTA) { - struct passwd *u = getpwuid(id); + struct passwd *u = getpwuid(d.d_id); if (u) name = u->pw_name; } else if (type == XFS_GROUP_QUOTA) { - struct group *g = getgrgid(id); + struct group *g = getgrgid(d.d_id); if (g) name = g->gr_name; } else if (type == XFS_PROJ_QUOTA) { - fs_project_t *p = getprprid(id); + fs_project_t *p = getprprid(d.d_id); if (p) name = p->pr_name; } @@ -425,14 +429,14 @@ report_user_mount( if (upper) { /* identifier range specified */ for (id = lower; id <= upper; id++) { - if (report_mount(fp, id, NULL, + if (report_mount(fp, id, NULL, NULL, form, XFS_USER_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } } else { setpwent(); while ((u = getpwent()) != NULL) { - if (report_mount(fp, u->pw_uid, u->pw_name, + if (report_mount(fp, u->pw_uid, u->pw_name, NULL, form, XFS_USER_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } @@ -457,14 +461,14 @@ report_group_mount( if (upper) { /* identifier range specified */ for (id = lower; id <= upper; id++) { - if (report_mount(fp, id, NULL, + if (report_mount(fp, id, NULL, NULL, form, XFS_GROUP_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } } else { setgrent(); while ((g = getgrent()) != NULL) { - if (report_mount(fp, g->gr_gid, g->gr_name, + if (report_mount(fp, g->gr_gid, g->gr_name, NULL, form, XFS_GROUP_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } @@ -488,14 +492,14 @@ report_project_mount( if (upper) { /* identifier range specified */ for (id = lower; id <= upper; id++) { - if (report_mount(fp, id, NULL, + if (report_mount(fp, id, NULL, NULL, form, XFS_PROJ_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } } else { setprent(); while ((p = getprent()) != NULL) { - if (report_mount(fp, p->pr_prid, p->pr_name, + if (report_mount(fp, p->pr_prid, p->pr_name, NULL, form, XFS_PROJ_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } -- 1.7.1 From sandeen@sandeen.net Fri Jan 8 12:57:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 583047F37 for ; Fri, 8 Jan 2016 12:57:57 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 4AA1C304075 for ; Fri, 8 Jan 2016 10:57:57 -0800 (PST) X-ASG-Debug-ID: 1452279475-04cbb05f7d17b30001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id JqQdny9ML7AvKP2v for ; Fri, 08 Jan 2016 10:57:56 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id EFEDE61AC509 for ; Fri, 8 Jan 2016 12:57:55 -0600 (CST) Subject: [PATCH 3/3] xfs_quota: use Q_XGETQUOTA2 for repquota To: xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 3/3] xfs_quota: use Q_XGETQUOTA2 for repquota References: <56900640.1050200@redhat.com> From: Eric Sandeen Message-ID: <569006B3.8000300@sandeen.net> Date: Fri, 8 Jan 2016 12:57:55 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <56900640.1050200@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452279475 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25958 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Rather than a loop over getpwnam() etc, use the Q_XGETQUOTA2 command to iterate through all active quotas. If Q_XGETQUOTA2 fails, go back to the old way. Signed-off-by: Eric Sandeen --- quota/quota.h | 1 + quota/report.c | 41 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/quota/quota.h b/quota/quota.h index 8b80029..dec8330 100644 --- a/quota/quota.h +++ b/quota/quota.h @@ -74,6 +74,7 @@ enum { DEFAULTS_FLAG = 0x0100, /* use value as a default */ ABSOLUTE_FLAG = 0x0200, /* absolute time, not related to now */ NO_LOOKUP_FLAG = 0x0400, /* skip name lookups, just report ID */ + GETQUOTA2_FLAG = 0x0800, /* use getquota2 quotactl */ }; /* diff --git a/quota/report.c b/quota/report.c index 746b04a..093155b 100644 --- a/quota/report.c +++ b/quota/report.c @@ -302,8 +302,14 @@ report_mount( char c[8], h[8], s[8]; uint qflags; int count; + int cmd; - if (xfsquotactl(XFS_GETQUOTA, dev, type, id, (void *)&d) < 0) { + if (flags & GETQUOTA2_FLAG) + cmd = XFS_GETQUOTA2; + else + cmd = XFS_GETQUOTA; + + if (xfsquotactl(cmd, dev, type, id, (void *)&d) < 0) { if (errno != ENOENT && errno != ENOSYS && errno != ESRCH) perror("XFS_GETQUOTA"); return 0; @@ -425,7 +431,7 @@ report_user_mount( uint flags) { struct passwd *u; - uint id; + uint id = 0, oid; if (upper) { /* identifier range specified */ for (id = lower; id <= upper; id++) { @@ -433,6 +439,15 @@ report_user_mount( form, XFS_USER_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } + } else if (report_mount(fp, id, NULL, &oid, form, + XFS_USER_QUOTA, mount, flags|GETQUOTA2_FLAG)) { + id = oid + 1; + flags |= GETQUOTA2_FLAG; + flags |= NO_HEADER_FLAG; + while (report_mount(fp, id, NULL, &oid, form, XFS_USER_QUOTA, + mount, flags)) { + id = oid + 1; + } } else { setpwent(); while ((u = getpwent()) != NULL) { @@ -457,7 +472,7 @@ report_group_mount( uint flags) { struct group *g; - uint id; + uint id = 0, oid; if (upper) { /* identifier range specified */ for (id = lower; id <= upper; id++) { @@ -465,6 +480,15 @@ report_group_mount( form, XFS_GROUP_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } + } else if (report_mount(fp, id, NULL, &oid, form, + XFS_GROUP_QUOTA, mount, flags|GETQUOTA2_FLAG)) { + id = oid + 1; + flags |= GETQUOTA2_FLAG; + flags |= NO_HEADER_FLAG; + while (report_mount(fp, id, NULL, &oid, form, XFS_GROUP_QUOTA, + mount, flags)) { + id = oid + 1; + } } else { setgrent(); while ((g = getgrent()) != NULL) { @@ -488,7 +512,7 @@ report_project_mount( uint flags) { fs_project_t *p; - uint id; + uint id = 0, oid; if (upper) { /* identifier range specified */ for (id = lower; id <= upper; id++) { @@ -496,6 +520,15 @@ report_project_mount( form, XFS_PROJ_QUOTA, mount, flags)) flags |= NO_HEADER_FLAG; } + } else if (report_mount(fp, id, NULL, &oid, form, + XFS_PROJ_QUOTA, mount, flags|GETQUOTA2_FLAG)) { + id = oid + 1; + flags |= GETQUOTA2_FLAG; + flags |= NO_HEADER_FLAG; + while (report_mount(fp, id, NULL, &oid, form, XFS_PROJ_QUOTA, + mount, flags)) { + id = oid + 1; + } } else { setprent(); while ((p = getprent()) != NULL) { -- 1.7.1 From michaelc@cs.wisc.edu Fri Jan 8 14:52:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 838307F37 for ; Fri, 8 Jan 2016 14:52:25 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 456FB8F8035 for ; Fri, 8 Jan 2016 12:52:21 -0800 (PST) X-ASG-Debug-ID: 1452286338-04cb6c75de18630001-NocioJ Received: from sabe.cs.wisc.edu (sabe.cs.wisc.edu [128.105.6.20]) by cuda.sgi.com with ESMTP id uAOvvnAF8tIft1dg (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 08 Jan 2016 12:52:18 -0800 (PST) X-Barracuda-Envelope-From: michaelc@cs.wisc.edu X-Barracuda-Apparent-Source-IP: 128.105.6.20 Received: from [20.15.0.202] (c-24-245-27-162.hsd1.mn.comcast.net [24.245.27.162]) (authenticated bits=0) by sabe.cs.wisc.edu (8.14.7/8.14.1) with ESMTP id u08Kp3xl001747 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 8 Jan 2016 14:51:04 -0600 Message-ID: <56902132.8080008@cs.wisc.edu> Date: Fri, 08 Jan 2016 14:50:58 -0600 From: Mike Christie User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Christoph Hellwig , mchristi@redhat.com CC: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <20160108133237.GA7663@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs In-Reply-To: <20160108133237.GA7663@infradead.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sabe.cs.wisc.edu[128.105.6.20] X-Barracuda-Start-Time: 1452286338 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25963 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On 01/08/2016 07:32 AM, Christoph Hellwig wrote: > This looks good to me from a highlevel point of view. > Do you also have a git tree to take a look at all the changes in a > single big diff? I uploaded a git tree here: https://github.com/mikechristie/linux-kernel.git based off of linux-next today. The patches are just in the master branch. From david@fromorbit.com Fri Jan 8 15:31:35 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 621A37F37 for ; Fri, 8 Jan 2016 15:31:35 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 552208F8049 for ; Fri, 8 Jan 2016 13:31:32 -0800 (PST) X-ASG-Debug-ID: 1452288688-04cb6c75dc194d0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 7uWZrwzouKqtJgAz for ; Fri, 08 Jan 2016 13:31:29 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DGEQDyKZBWPGu7LXleKAECgw+BP4Jig36BeaAUAQEBAQEBBotdiT+GCQICAQECgR1NAQEBAQEBBwEBAQFBP0ESAYNgAQEBAwE6HCMFCwgDDgcCAQklDwUlAwcMDhOIJwfCBwEBAQEGAQEBAR8ZhXWFR4k8BYdbhxKIII1PgWWEQ4hcjk6FAio0gXqCHCOBKAEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 09 Jan 2016 08:01:28 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aHed5-0005iF-Ir; Sat, 09 Jan 2016 08:31:27 +1100 Date: Sat, 9 Jan 2016 08:31:27 +1100 From: Dave Chinner To: Jiri Kosina Cc: Julian Wollrath , xfs@oss.sgi.com Subject: Re: Suspend does not work with v4.4-rc8 Message-ID: <20160108213127.GS21461@dastard> X-ASG-Orig-Subj: Re: Suspend does not work with v4.4-rc8 References: <20160104202611.5b100540@saldaea> <20160104194014.GD19802@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452288688 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25964 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Fri, Jan 08, 2016 at 11:44:21AM +0100, Jiri Kosina wrote: > On Tue, 5 Jan 2016, Dave Chinner wrote: > > > > kernel: Freezing of tasks failed after 20.006 seconds (2 tasks refusing to freeze, wq_bu_busy=0): > > > kernel: xfsaild/dm-1 S 0000000000014100 0 283 2 0x00000000 > > > kernel: ffff880213f53e10 ffffffff8180e4c0 ffff880213f05040 0000000000000000 > > > kernel: 0000000000000000 ffff880213f54000 0000000000000000 0000000000000000 > > > kernel: ffff8800ca389e40 ffff8800ca392000 ffff880213f53ed0 ffffffff814cd8ac > > > kernel: Call Trace: > > > kernel: [] ? schedule+0x2c/0x70 > > > kernel: [] ? xfsaild+0x4fd/0x5b0 > > > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > > > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > > > kernel: [] ? kthread+0xb8/0xd0 > > > kernel: [] ? kthread_worker_fn+0x150/0x150 > > > kernel: [] ? ret_from_fork+0x3f/0x70 > > > kernel: [] ? kthread_worker_fn+0x150/0x150 > > > kernel: xfsaild/sda1 S 0000000000014100 0 591 2 0x00000000 > > > kernel: ffff88021193be10 ffff8802159c4dc0 ffff880213ab9340 0000000000000000 > > > kernel: 0000000000000000 ffff88021193c000 0000000000000000 0000000000000000 > > > kernel: ffff8800ca2a4240 ffff880214eea000 ffff88021193bed0 ffffffff814cd8ac > > > kernel: Call Trace: > > > kernel: [] ? schedule+0x2c/0x70 > > > kernel: [] ? xfsaild+0x4fd/0x5b0 > > > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > > > kernel: [] ? xfs_trans_ail_cursor_first+0x80/0x80 > > > kernel: [] ? kthread+0xb8/0xd0 > > > kernel: [] ? kthread_worker_fn+0x150/0x150 > > > kernel: [] ? ret_from_fork+0x3f/0x70 > > > kernel: [] ? kthread_worker_fn+0x150/0x150 > > > > > > Please tell me what more information you need to be able to fix this > > > issue. > > > > The freezer detection is broken. The thread is sleeping in schedule > > until a wakeup occurs some time in the future, which means it cannot > > "enter then freezer" because it's not a running thread. This is a > > problem introduced by commit 24ba16b ("xfs: clear PF_NOFREEZE for > > xfsaild kthread"). > > > > Jiri, I'm tempted just to revert this change - if the freezer > > doesn't detect processes that are not in TASK_RUNNABLE state as > > frozeni or can't mark them as frozen, then this change will never > > work reliably for XFS.... > > Well, clearly the thread is sleeping in schedule() during the freezing > operation and it's supposed to be doing so; therefore it doesn't need > explicit freezing point, right? No. It's sleeping in schedule because it's got nothing more to do - it's issued all it's IO and is idle. It is not going to run again until filesystem modification activity is restarted. But if the AIL still has objects in it (like it will after a sync), then it will continue to run and issue IO until it returns to the empty, idle state. In this active state, we need to freeze the thread on suspend so that it doesn't keep issuing IO all through the suspend process... > So the proper fix would rather be something like > > > > From: Jiri Kosina > Subject: [PATCH] xfs: xfsaild doesn't need to be freezable No, that just means we guarantee that there will be suspend image coherency problems when suspend is run on a busy filesystem... This process is not going to enter a runable state, so is never going to enter the freezer. But we can't be certain of that, because we haven't frozen the filesystem and hence it can still be modified and this thread could be woken and do stuff when it shouldn't. Cheers, Dave. -- Dave Chinner david@fromorbit.com From msnitzer@redhat.com Fri Jan 8 17:07:21 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 449807F37 for ; Fri, 8 Jan 2016 17:07:21 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id B80A2AC001 for ; Fri, 8 Jan 2016 15:07:17 -0800 (PST) X-ASG-Debug-ID: 1452294435-04cbb05f7f1d700001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id xJdk5rUkKmq7DHWG (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 08 Jan 2016 15:07:15 -0800 (PST) X-Barracuda-Envelope-From: msnitzer@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id 7E4EAC0BB2BB; Fri, 8 Jan 2016 23:07:13 +0000 (UTC) Received: from localhost (dhcp-25-149.bos.redhat.com [10.18.25.149]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u08N7Axu024962 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA256 bits=256 verify=NO); Fri, 8 Jan 2016 18:07:11 -0500 Date: Fri, 8 Jan 2016 18:07:10 -0500 From: Mike Snitzer To: Mike Christie Cc: Christoph Hellwig , mchristi@redhat.com, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, xfs@oss.sgi.com, linux-scsi@vger.kernel.org, konrad.wilk@oracle.com, linux-kernel@vger.kernel.org, philipp.reisner@linbit.com, linux-f2fs-devel@lists.sourceforge.net, linux-raid@vger.kernel.org, ocfs2-devel@oss.oracle.com, dm-devel@redhat.com, target-devel@vger.kernel.org, linux-mtd@lists.infradead.org, osd-dev@open-osd.org, linux-fsdevel@vger.kernel.org, lars.ellenberg@linbit.com, linux-ext4@vger.kernel.org, linux-btrfs@vger.kernel.org, drbd-dev@lists.linbit.com Subject: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs Message-ID: <20160108230709.GA15655@redhat.com> X-ASG-Orig-Subj: Re: [PATCH 00/35 v2] separate operations from flags in the bio/request structs References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <20160108133237.GA7663@infradead.org> <56902132.8080008@cs.wisc.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56902132.8080008@cs.wisc.edu> User-Agent: Mutt/1.5.21 (2010-09-15) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452294435 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Fri, Jan 08 2016 at 3:50P -0500, Mike Christie wrote: > On 01/08/2016 07:32 AM, Christoph Hellwig wrote: > > This looks good to me from a highlevel point of view. > > Do you also have a git tree to take a look at all the changes in a > > single big diff? > > I uploaded a git tree here: > > https://github.com/mikechristie/linux-kernel.git > > based off of linux-next today. The patches are just in the master branch. Hi Mike, I had a look at your DM changes. That sucked (but things looked good). So much so that I want to thank you for tackling this task (and taking one for the team). I'll get this kernel built and tested (using the various DM testsuites: lvm2, cryptsetup, device-mapper-test-suite for dm-thinp and dm-cache). But I did make some changes during my review that I'd appreciate if you'd fold in to wherever you think it appropriate in your series: diff --git a/drivers/md/dm-stats.c b/drivers/md/dm-stats.c index 96b5c1b..9e29dba 100644 --- a/drivers/md/dm-stats.c +++ b/drivers/md/dm-stats.c @@ -514,11 +514,10 @@ static void dm_stat_round(struct dm_stat *s, struct dm_stat_shared *shared, } static void dm_stat_for_entry(struct dm_stat *s, size_t entry, - unsigned long bi_rw, sector_t len, + int idx, sector_t len, struct dm_stats_aux *stats_aux, bool end, unsigned long duration_jiffies) { - unsigned long idx = bi_rw; struct dm_stat_shared *shared = &s->stat_shared[entry]; struct dm_stat_percpu *p; @@ -584,7 +583,7 @@ static void dm_stat_for_entry(struct dm_stat *s, size_t entry, #endif } -static void __dm_stat_bio(struct dm_stat *s, unsigned long bi_rw, +static void __dm_stat_bio(struct dm_stat *s, int bi_rw, sector_t bi_sector, sector_t end_sector, bool end, unsigned long duration_jiffies, struct dm_stats_aux *stats_aux) @@ -622,7 +621,7 @@ static void __dm_stat_bio(struct dm_stat *s, unsigned long bi_rw, } while (unlikely(todo != 0)); } -void dm_stats_account_io(struct dm_stats *stats, unsigned long bi_rw, +void dm_stats_account_io(struct dm_stats *stats, int bi_rw, sector_t bi_sector, unsigned bi_sectors, bool end, unsigned long duration_jiffies, struct dm_stats_aux *stats_aux) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 5e940f7..9a4b3cb 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1326,7 +1326,7 @@ static int device_fua_capable(struct dm_target *ti, struct dm_dev *dev, } static bool dm_table_supports_flush(struct dm_table *t, - iterate_devices_callout_fn support_test) + iterate_devices_callout_fn callout_fn) { struct dm_target *ti; unsigned i = 0; @@ -1347,7 +1347,7 @@ static bool dm_table_supports_flush(struct dm_table *t, return true; if (ti->type->iterate_devices && - ti->type->iterate_devices(ti, support_test, NULL)) + ti->type->iterate_devices(ti, callout_fn, NULL)) return true; } From BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org Sat Jan 9 01:26:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 8E37C7F37 for ; Sat, 9 Jan 2016 01:26:08 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 7DC61304053 for ; Fri, 8 Jan 2016 23:26:05 -0800 (PST) X-ASG-Debug-ID: 1452324362-04cb6c75db266c0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id q7THnrGDLimFzjuz (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 08 Jan 2016 23:26:03 -0800 (PST) X-Barracuda-Envelope-From: BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aHnuS-0007PP-O3; Sat, 09 Jan 2016 07:26:00 +0000 Date: Fri, 8 Jan 2016 23:26:00 -0800 From: Christoph Hellwig To: Eric Sandeen Cc: fsdevel , xfs@oss.sgi.com, Jan Kara Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 Message-ID: <20160109072600.GA21636@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 References: <568FEA2C.6080708@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <568FEA2C.6080708@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452324363 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25975 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Fri, Jan 08, 2016 at 10:56:12AM -0600, Eric Sandeen wrote: > This adds a new quotactl, Q_XGETQUOTA2. > > Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will > return quota information for the id equal to or greater than > the id requested. In other words, if the specified id has > no quota, the command will return quota information for the > next higher id which does have a quota set. If no higher id > has an active quota, -ESRCH is returned. Please add a flags argument to Q_XGETQUOTA2, and then make the new behavior the first flag. Keep Q_XGETQUOTA behavior for the flag-less case. That way we get future etensibility for free. From viro@ftp.linux.org.uk Sat Jan 9 01:54:28 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 212A37F37 for ; Sat, 9 Jan 2016 01:54:28 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0F84F30405F for ; Fri, 8 Jan 2016 23:54:27 -0800 (PST) X-ASG-Debug-ID: 1452326064-04cb6c75db26f90001-NocioJ Received: from ZenIV.linux.org.uk (zeniv.linux.org.uk [195.92.253.2]) by cuda.sgi.com with ESMTP id naxWQEknYLBk3T5s (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 08 Jan 2016 23:54:25 -0800 (PST) X-Barracuda-Envelope-From: viro@ftp.linux.org.uk X-Barracuda-Apparent-Source-IP: 195.92.253.2 Received: from viro by ZenIV.linux.org.uk with local (Exim 4.76 #1 (Red Hat Linux)) id 1aHoLm-0001Qy-Mt; Sat, 09 Jan 2016 07:54:14 +0000 Date: Sat, 9 Jan 2016 07:54:14 +0000 From: Al Viro To: Dan Williams Cc: xfs@oss.sgi.com, linux-block@vger.kernel.org, linux-nvdimm@ml01.01.org, Dave Chinner , Jens Axboe , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler Subject: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life Message-ID: <20160109075414.GA5008@ZenIV.linux.org.uk> X-ASG-Orig-Subj: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: Al Viro X-Barracuda-Connect: zeniv.linux.org.uk[195.92.253.2] X-Barracuda-Start-Time: 1452326065 X-Barracuda-Encrypted: DHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25975 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 10:20:05AM -0800, Dan Williams wrote: > Historically we have waited for filesystem specific heuristics to > attempt to guess when a block device is gone. Sometimes this works, but > in other cases the system can hang waiting for the fs to trigger its > shutdown protocol. > > The initial motivation for this investigation was to prevent DAX > mappings (direct mmap access to persistent memory) from leaking past the > lifetime of the hosting block device. However, Dave points out that > these shutdown operations are needed in other scenarios. Quoting Dave: > > For example, if we detect a free space corruption during allocation, > it is not safe to trust *any active mapping* because we can't trust > that we having handed out the same block to multiple owners. Hence > on such a filesystem shutdown, we have to prevent any new DAX > mapping from occurring and invalidate all existing mappings as we > cannot allow userspace to modify any data or metadata until we've > resolved the corruption situation. > > The current block device shutdown sequence of del_gendisk + > blk_cleanup_queue is problematic. We want to tell the fs after > blk_cleanup_queue that there is no possibility of recovery, but by that > time we have deleted partitions and lost the ability to find all the > super-blocks on a block device. > > Introduce del_gendisk_queue to trigger ->quiesce() and ->bdi_gone() > notifications to all the filesystems hosted on the disk. Where > ->quiesce() are 'shutdown' operations while the bdev may still be alive, > and ->bdi_gone() is a set of actions to take after the backing device > is known to be permanently dead. Would you mind explaining what the hell is _the_ backing device of a filesystem? What does that translate into in case of e.g. btrfs spanning several disks? Or ext4 with journal on a different device, for that matter? If anything, I would argue that filesystem is out of place here - general situation is "IO on X may require IO on device Y and X needs to do something when Y goes away". Consider e.g. /dev/loop backed by a device that went away. Or by a file on fs that has run down the curtain and joined the bleedin choir invisible. With another fs partially hosted by that loopback device. Or by RAID0 containing said device. You are given Y and attempt to locate the affected X. _Then_ you assume that X is a filesystem and has "something to be done" independent from the role Y played for it, so you can pick that action from superblock method. IMO you are placing the burden in the wrong place. _Recepient_ knows what it depends upon and what should be done for each source of trouble. So make it recepient's responsibility to request notifications. At which point the superblock method goes away, along with the requirement to handle all sources of trouble the same way, etc. What's more, things like RAID5 (also interested in knowing when a component has been ripped out) might or might not decide to propagate the event further - after all, that's exactly the point of redundancy. I'd look into something along the lines of notifier chain per gendisk, with potential victims registering a callback when they decide that from now on such and such device might screw them over... From BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org Sat Jan 9 02:04:46 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 5A4E77F37 for ; Sat, 9 Jan 2016 02:04:46 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 492598F8033 for ; Sat, 9 Jan 2016 00:04:43 -0800 (PST) X-ASG-Debug-ID: 1452326680-04cbb05f7c28b00001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id p49wRfF8uSqzTLBj (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 00:04:41 -0800 (PST) X-Barracuda-Envelope-From: BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aHoVe-0001fx-8J; Sat, 09 Jan 2016 08:04:26 +0000 Date: Sat, 9 Jan 2016 00:04:26 -0800 From: Christoph Hellwig To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 34/35] block: add QUEUE_FLAGs for flush and fua Message-ID: <20160109080426.GA6207@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 34/35] block: add QUEUE_FLAGs for flush and fua References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-35-git-send-email-mchristi@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452027218-32303-35-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452326681 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25975 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 On Tue, Jan 05, 2016 at 02:53:37PM -0600, mchristi@redhat.com wrote: > From: Mike Christie > > The last patch added a REQ_OP_FLUSH for request_fn drivers > and the next patch renames REQ_FLUSH to REQ_PREFLUSH which > will be used by file systems and make_request_fn drivers. > > This leaves REQ_FLUSH/REQ_FUA defined for drivers to tell > the block layer if flush/fua is supported. The names are > confusing and I bet will will accidentally be used by > people to request flushes. To avoid that, this patch adds > QUEUE_FLAGs for flush and fua which drivers will use to > indicate what they support. > > Signed-off-by: Mike Christie Yes, this was rather confusing before. Looks great fix the fix for th kbuild complaint: Reviewed-by: Christoph Hellwig From darrick.wong@oracle.com Sat Jan 9 03:59:29 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 63D0F7F37 for ; Sat, 9 Jan 2016 03:59:29 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 270B330405F for ; Sat, 9 Jan 2016 01:59:26 -0800 (PST) X-ASG-Debug-ID: 1452333562-04bdf06f6c2a1b0001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id tcPtU35rfZv8FyOt (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 09 Jan 2016 01:59:23 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u099xKC3031798 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sat, 9 Jan 2016 09:59:21 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u099xKn3025043 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sat, 9 Jan 2016 09:59:20 GMT Received: from abhmp0019.oracle.com (abhmp0019.oracle.com [141.146.116.25]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u099xKsA014075; Sat, 9 Jan 2016 09:59:20 GMT Received: from localhost (/24.130.224.127) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sat, 09 Jan 2016 01:59:20 -0800 Date: Sat, 9 Jan 2016 01:59:19 -0800 From: "Darrick J. Wong" To: david@fromorbit.com Cc: xfs@oss.sgi.com Subject: Re: [PATCH 57/76] xfs: allocate delayed extents in CoW fork Message-ID: <20160109095919.GO28330@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 57/76] xfs: allocate delayed extents in CoW fork References: <20151219085622.12713.88678.stgit@birch.djwong.org> <20151219090248.12713.32309.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219090248.12713.32309.stgit@birch.djwong.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0022.oracle.com [141.146.126.234] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1452333563 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25977 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sat, Dec 19, 2015 at 01:02:48AM -0800, Darrick J. Wong wrote: > Modify the writepage handler to find and convert pending delalloc > extents to real allocations. > > Signed-off-by: Darrick J. Wong > --- > fs/xfs/xfs_aops.c | 63 +++++++++++++++++++++++++++++++++++--------- > fs/xfs/xfs_aops.h | 10 +++++++ > fs/xfs/xfs_reflink.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++ > fs/xfs/xfs_reflink.h | 3 ++ > 4 files changed, 135 insertions(+), 13 deletions(-) > > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c > index 13629d2..7179b25 100644 > --- a/fs/xfs/xfs_aops.c > +++ b/fs/xfs/xfs_aops.c > @@ -285,7 +285,8 @@ xfs_map_blocks( > loff_t offset, > struct xfs_bmbt_irec *imap, > int type, > - int nonblocking) > + int nonblocking, > + bool is_cow) > { > struct xfs_inode *ip = XFS_I(inode); > struct xfs_mount *mp = ip->i_mount; > @@ -294,10 +295,15 @@ xfs_map_blocks( > int error = 0; > int bmapi_flags = XFS_BMAPI_ENTIRE; > int nimaps = 1; > + int whichfork; > + bool need_alloc; > > if (XFS_FORCED_SHUTDOWN(mp)) > return -EIO; > > + whichfork = (is_cow ? XFS_COW_FORK : XFS_DATA_FORK); > + need_alloc = (type == XFS_IO_DELALLOC); > + > if (type == XFS_IO_UNWRITTEN) > bmapi_flags |= XFS_BMAPI_IGSTATE; > > @@ -315,16 +321,21 @@ xfs_map_blocks( > count = mp->m_super->s_maxbytes - offset; > end_fsb = XFS_B_TO_FSB(mp, (xfs_ufsize_t)offset + count); > offset_fsb = XFS_B_TO_FSBT(mp, offset); > - error = xfs_bmapi_read(ip, offset_fsb, end_fsb - offset_fsb, > - imap, &nimaps, bmapi_flags); > + > + if (is_cow) > + error = xfs_reflink_find_cow_mapping(ip, offset, imap, > + &need_alloc); > + else > + error = xfs_bmapi_read(ip, offset_fsb, end_fsb - offset_fsb, > + imap, &nimaps, bmapi_flags); This isn't correct -- for an XFS_IO_OVERWRITE, the extent returned in imap could have some (shared) blocks with a corresponding delalloc reservation in the CoW fork. If that's the case, then feeding the full extent to xfs_cluster_write results in the ioend flag being set, which can cause the ioend processing to crash because it'll see the delalloc reservation and freak out. Therefore, in the non-cow reflink-inode overwrite case, we have to find trim the extent to just before the next CoW reservation. That forces a call to xfs_vm_writepage at the start of the reservation, which is the only way that the delalloc->regular conversion can happen in the CoW fork. (This fixes a rare crash in generic/163 and a regular corruption failure in generic/898.) --D > xfs_iunlock(ip, XFS_ILOCK_SHARED); > > if (error) > return error; > > - if (type == XFS_IO_DELALLOC && > + if (need_alloc && > (!nimaps || isnullstartblock(imap->br_startblock))) { > - error = xfs_iomap_write_allocate(ip, XFS_DATA_FORK, offset, > + error = xfs_iomap_write_allocate(ip, whichfork, offset, > imap); > if (!error) > trace_xfs_map_blocks_alloc(ip, offset, count, type, > @@ -575,7 +586,8 @@ xfs_add_to_ioend( > xfs_off_t offset, > unsigned int type, > xfs_ioend_t **result, > - int need_ioend) > + int need_ioend, > + bool is_cow) > { > xfs_ioend_t *ioend = *result; > > @@ -593,6 +605,8 @@ xfs_add_to_ioend( > ioend->io_buffer_tail->b_private = bh; > ioend->io_buffer_tail = bh; > } > + if (is_cow) > + ioend->io_flags |= XFS_IOEND_COW; > > bh->b_private = NULL; > ioend->io_size += bh->b_size; > @@ -703,6 +717,8 @@ xfs_convert_page( > int len, page_dirty; > int count = 0, done = 0, uptodate = 1; > xfs_off_t offset = page_offset(page); > + bool is_cow; > + struct xfs_inode *ip = XFS_I(inode); > > if (page->index != tindex) > goto fail; > @@ -786,6 +802,15 @@ xfs_convert_page( > else > type = XFS_IO_OVERWRITE; > > + /* Figure out if CoW is pending for the other pages. */ > + is_cow = false; > + if (type == XFS_IO_OVERWRITE && > + xfs_sb_version_hasreflink(&ip->i_mount->m_sb)) { > + xfs_ilock(ip, XFS_ILOCK_SHARED); > + is_cow = xfs_reflink_is_cow_pending(ip, offset); > + xfs_iunlock(ip, XFS_ILOCK_SHARED); > + } > + > /* > * imap should always be valid because of the above > * partial page end_offset check on the imap. > @@ -793,10 +818,10 @@ xfs_convert_page( > ASSERT(xfs_imap_valid(inode, imap, offset)); > > lock_buffer(bh); > - if (type != XFS_IO_OVERWRITE) > + if (type != XFS_IO_OVERWRITE || is_cow) > xfs_map_at_offset(inode, bh, imap, offset); > xfs_add_to_ioend(inode, bh, offset, type, > - ioendp, done); > + ioendp, done, is_cow); > > page_dirty--; > count++; > @@ -959,6 +984,8 @@ xfs_vm_writepage( > int err, imap_valid = 0, uptodate = 1; > int count = 0; > int nonblocking = 0; > + struct xfs_inode *ip = XFS_I(inode); > + bool was_cow, is_cow; > > trace_xfs_writepage(inode, page, 0, 0); > > @@ -1057,6 +1084,7 @@ xfs_vm_writepage( > bh = head = page_buffers(page); > offset = page_offset(page); > type = XFS_IO_OVERWRITE; > + was_cow = false; > > if (wbc->sync_mode == WB_SYNC_NONE) > nonblocking = 1; > @@ -1069,6 +1097,7 @@ xfs_vm_writepage( > if (!buffer_uptodate(bh)) > uptodate = 0; > > + is_cow = false; > /* > * set_page_dirty dirties all buffers in a page, independent > * of their state. The dirty state however is entirely > @@ -1091,7 +1120,15 @@ xfs_vm_writepage( > imap_valid = 0; > } > } else if (buffer_uptodate(bh)) { > - if (type != XFS_IO_OVERWRITE) { > + if (xfs_sb_version_hasreflink(&ip->i_mount->m_sb)) { > + xfs_ilock(ip, XFS_ILOCK_SHARED); > + is_cow = xfs_reflink_is_cow_pending(ip, > + offset); > + xfs_iunlock(ip, XFS_ILOCK_SHARED); > + } > + > + if (type != XFS_IO_OVERWRITE || > + is_cow != was_cow) { > type = XFS_IO_OVERWRITE; > imap_valid = 0; > } > @@ -1121,23 +1158,23 @@ xfs_vm_writepage( > */ > new_ioend = 1; > err = xfs_map_blocks(inode, offset, &imap, type, > - nonblocking); > + nonblocking, is_cow); > if (err) > goto error; > imap_valid = xfs_imap_valid(inode, &imap, offset); > } > if (imap_valid) { > lock_buffer(bh); > - if (type != XFS_IO_OVERWRITE) > + if (type != XFS_IO_OVERWRITE || is_cow) > xfs_map_at_offset(inode, bh, &imap, offset); > xfs_add_to_ioend(inode, bh, offset, type, &ioend, > - new_ioend); > + new_ioend, is_cow); > count++; > } > > if (!iohead) > iohead = ioend; > - > + was_cow = is_cow; > } while (offset += len, ((bh = bh->b_this_page) != head)); > > if (uptodate && bh == head) > diff --git a/fs/xfs/xfs_aops.h b/fs/xfs/xfs_aops.h > index f6ffc9a..ac59548 100644 > --- a/fs/xfs/xfs_aops.h > +++ b/fs/xfs/xfs_aops.h > @@ -34,6 +34,8 @@ enum { > { XFS_IO_UNWRITTEN, "unwritten" }, \ > { XFS_IO_OVERWRITE, "overwrite" } > > +#define XFS_IOEND_COW (1) > + > /* > * xfs_ioend struct manages large extent writes for XFS. > * It can manage several multi-page bio's at once. > @@ -50,8 +52,16 @@ typedef struct xfs_ioend { > xfs_off_t io_offset; /* offset in the file */ > struct work_struct io_work; /* xfsdatad work queue */ > struct xfs_trans *io_append_trans;/* xact. for size update */ > + unsigned long io_flags; /* status flags */ > } xfs_ioend_t; > > +static inline bool > +xfs_ioend_is_cow( > + struct xfs_ioend *ioend) > +{ > + return ioend->io_flags & XFS_IOEND_COW; > +} > + > extern const struct address_space_operations xfs_address_space_operations; > > int xfs_get_blocks(struct inode *inode, sector_t offset, > diff --git a/fs/xfs/xfs_reflink.c b/fs/xfs/xfs_reflink.c > index fdc538e..65d4c2d 100644 > --- a/fs/xfs/xfs_reflink.c > +++ b/fs/xfs/xfs_reflink.c > @@ -249,3 +249,75 @@ xfs_reflink_reserve_cow_range( > trace_xfs_reflink_reserve_cow_range_error(ip, error, _RET_IP_); > return error; > } > + > +/** > + * xfs_reflink_is_cow_pending() -- Determine if CoW is pending for a given > + * file and offset. > + * > + * @ip: XFS inode object. > + * @offset: The file offset, in bytes. > + */ > +bool > +xfs_reflink_is_cow_pending( > + struct xfs_inode *ip, > + xfs_off_t offset) > +{ > + struct xfs_ifork *ifp; > + struct xfs_bmbt_rec_host *gotp; > + struct xfs_bmbt_irec irec; > + xfs_fileoff_t bno; > + xfs_extnum_t idx; > + > + if (!xfs_is_reflink_inode(ip)) > + return false; > + > + ifp = XFS_IFORK_PTR(ip, XFS_COW_FORK); > + bno = XFS_B_TO_FSBT(ip->i_mount, offset); > + gotp = xfs_iext_bno_to_ext(ifp, bno, &idx); > + > + if (!gotp) > + return false; > + > + xfs_bmbt_get_all(gotp, &irec); > + if (bno >= irec.br_startoff + irec.br_blockcount || > + bno < irec.br_startoff) > + return false; > + return true; > +} > + > +/** > + * xfs_reflink_find_cow_mapping() -- Find the mapping for a CoW block. > + * > + * @ip: The XFS inode object. > + * @offset: The file offset, in bytes. > + * @imap: The mapping we're going to use for this block. > + * @nimaps: Number of mappings we're returning. > + */ > +int > +xfs_reflink_find_cow_mapping( > + struct xfs_inode *ip, > + xfs_off_t offset, > + struct xfs_bmbt_irec *imap, > + bool *need_alloc) > +{ > + struct xfs_bmbt_irec irec; > + struct xfs_ifork *ifp; > + struct xfs_bmbt_rec_host *gotp; > + xfs_fileoff_t bno; > + xfs_extnum_t idx; > + > + /* Find the extent in the CoW fork. */ > + ifp = XFS_IFORK_PTR(ip, XFS_COW_FORK); > + bno = XFS_B_TO_FSBT(ip->i_mount, offset); > + gotp = xfs_iext_bno_to_ext(ifp, bno, &idx); > + xfs_bmbt_get_all(gotp, &irec); > + > + trace_xfs_reflink_find_cow_mapping(ip, offset, 1, XFS_IO_OVERWRITE, > + &irec); > + > + /* If it's still delalloc, we must allocate later. */ > + *imap = irec; > + *need_alloc = !!(isnullstartblock(irec.br_startblock)); > + > + return 0; > +} > diff --git a/fs/xfs/xfs_reflink.h b/fs/xfs/xfs_reflink.h > index 41afdbe..1018ac9 100644 > --- a/fs/xfs/xfs_reflink.h > +++ b/fs/xfs/xfs_reflink.h > @@ -20,5 +20,8 @@ > > extern int xfs_reflink_reserve_cow_range(struct xfs_inode *ip, xfs_off_t pos, > xfs_off_t len); > +extern bool xfs_reflink_is_cow_pending(struct xfs_inode *ip, xfs_off_t offset); > +extern int xfs_reflink_find_cow_mapping(struct xfs_inode *ip, xfs_off_t offset, > + struct xfs_bmbt_irec *imap, bool *need_alloc); > > #endif /* __XFS_REFLINK_H */ > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org Sat Jan 9 07:34:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id E1E9A7F37 for ; Sat, 9 Jan 2016 07:34:30 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id D084F8F8040 for ; Sat, 9 Jan 2016 05:34:27 -0800 (PST) X-ASG-Debug-ID: 1452346464-04cb6c75db2d2b0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id HoL7y5s0CLAuAiOo (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 05:34:24 -0800 (PST) X-Barracuda-Envelope-From: BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aHtej-000777-A1; Sat, 09 Jan 2016 13:34:09 +0000 Date: Sat, 9 Jan 2016 05:34:09 -0800 From: Christoph Hellwig To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio Message-ID: <20160109133409.GA26660@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-2-git-send-email-mchristi@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452027218-32303-2-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452346464 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25981 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 On Tue, Jan 05, 2016 at 02:53:04PM -0600, mchristi@redhat.com wrote: > From: Mike Christie > > This has callers of submit_bio/submit_bio_wait set the bio->bi_rw > instead of passing it in. This makes that use the same as > generic_make_request and how we set the other bio fields. > > Signed-off-by: Mike Christie > --- > block/bio.c | 7 +++---- > block/blk-core.c | 11 ++++------- > block/blk-flush.c | 3 ++- > block/blk-lib.c | 9 ++++++--- > drivers/block/drbd/drbd_actlog.c | 2 +- > drivers/block/drbd/drbd_bitmap.c | 4 ++-- > drivers/block/floppy.c | 3 ++- > drivers/block/xen-blkback/blkback.c | 4 +++- > drivers/block/xen-blkfront.c | 4 ++-- > drivers/md/bcache/debug.c | 6 ++++-- > drivers/md/bcache/journal.c | 2 +- > drivers/md/bcache/super.c | 4 ++-- > drivers/md/dm-bufio.c | 3 ++- > drivers/md/dm-io.c | 3 ++- > drivers/md/dm-log-writes.c | 9 ++++++--- > drivers/md/dm-thin.c | 3 ++- > drivers/md/md.c | 10 +++++++--- > drivers/md/raid1.c | 3 ++- > drivers/md/raid10.c | 4 +++- > drivers/md/raid5-cache.c | 7 ++++--- > drivers/target/target_core_iblock.c | 24 +++++++++++++----------- > fs/btrfs/check-integrity.c | 18 ++++++++++-------- > fs/btrfs/check-integrity.h | 4 ++-- > fs/btrfs/disk-io.c | 3 ++- > fs/btrfs/extent_io.c | 7 ++++--- > fs/btrfs/raid56.c | 16 +++++++++++----- > fs/btrfs/scrub.c | 16 +++++++++++----- > fs/btrfs/volumes.c | 14 +++++++------- > fs/buffer.c | 3 ++- > fs/direct-io.c | 3 ++- > fs/ext4/crypto.c | 3 ++- > fs/ext4/page-io.c | 3 ++- > fs/ext4/readpage.c | 9 +++++---- > fs/f2fs/data.c | 13 ++++++++----- > fs/f2fs/segment.c | 6 ++++-- > fs/gfs2/lops.c | 3 ++- > fs/gfs2/meta_io.c | 3 ++- > fs/gfs2/ops_fstype.c | 3 ++- > fs/hfsplus/wrapper.c | 3 ++- > fs/jfs/jfs_logmgr.c | 6 ++++-- > fs/jfs/jfs_metapage.c | 10 ++++++---- > fs/logfs/dev_bdev.c | 15 ++++++++++----- > fs/mpage.c | 3 ++- > fs/nfs/blocklayout/blocklayout.c | 22 ++++++++++++---------- > fs/nilfs2/segbuf.c | 3 ++- > fs/ocfs2/cluster/heartbeat.c | 12 +++++++----- > fs/xfs/xfs_aops.c | 3 ++- > fs/xfs/xfs_buf.c | 4 ++-- > include/linux/bio.h | 2 +- > include/linux/fs.h | 2 +- > kernel/power/swap.c | 5 +++-- > mm/page_io.c | 10 ++++++---- > 52 files changed, 211 insertions(+), 141 deletions(-) > > diff --git a/block/bio.c b/block/bio.c > index dbabd48..921112b 100644 > --- a/block/bio.c > +++ b/block/bio.c > @@ -859,21 +859,20 @@ static void submit_bio_wait_endio(struct bio *bio) > > /** > * submit_bio_wait - submit a bio, and wait until it completes > - * @rw: whether to %READ or %WRITE, or maybe to %READA (read ahead) > * @bio: The &struct bio which describes the I/O > * > * Simple wrapper around submit_bio(). Returns 0 on success, or the error from > * bio_endio() on failure. > */ > -int submit_bio_wait(int rw, struct bio *bio) > +int submit_bio_wait(struct bio *bio) > { > struct submit_bio_ret ret; > > - rw |= REQ_SYNC; > init_completion(&ret.event); > bio->bi_private = &ret; > bio->bi_end_io = submit_bio_wait_endio; > - submit_bio(rw, bio); > + bio->bi_rw |= REQ_SYNC; > + submit_bio(bio); > wait_for_completion(&ret.event); > > return ret.error; > diff --git a/block/blk-core.c b/block/blk-core.c > index ab51685..9b887e3 100644 > --- a/block/blk-core.c > +++ b/block/blk-core.c > @@ -2092,7 +2092,6 @@ EXPORT_SYMBOL(generic_make_request); > > /** > * submit_bio - submit a bio to the block device layer for I/O > - * @rw: whether to %READ or %WRITE, or maybe to %READA (read ahead) > * @bio: The &struct bio which describes the I/O > * > * submit_bio() is very similar in purpose to generic_make_request(), and > @@ -2100,10 +2099,8 @@ EXPORT_SYMBOL(generic_make_request); > * interfaces; @bio must be presetup and ready for I/O. > * > */ > -blk_qc_t submit_bio(int rw, struct bio *bio) > +blk_qc_t submit_bio(struct bio *bio) > { > - bio->bi_rw |= rw; > - > /* > * If it's a regular read/write or a barrier with data attached, > * go through the normal accounting stuff before submission. > @@ -2111,12 +2108,12 @@ blk_qc_t submit_bio(int rw, struct bio *bio) > if (bio_has_data(bio)) { > unsigned int count; > > - if (unlikely(rw & REQ_WRITE_SAME)) > + if (unlikely(bio->bi_rw & REQ_WRITE_SAME)) > count = bdev_logical_block_size(bio->bi_bdev) >> 9; > else > count = bio_sectors(bio); > > - if (rw & WRITE) { > + if (bio->bi_rw & WRITE) { > count_vm_events(PGPGOUT, count); > } else { > task_io_account_read(bio->bi_iter.bi_size); > @@ -2127,7 +2124,7 @@ blk_qc_t submit_bio(int rw, struct bio *bio) > char b[BDEVNAME_SIZE]; > printk(KERN_DEBUG "%s(%d): %s block %Lu on %s (%u sectors)\n", > current->comm, task_pid_nr(current), > - (rw & WRITE) ? "WRITE" : "READ", > + (bio->bi_rw & WRITE) ? "WRITE" : "READ", > (unsigned long long)bio->bi_iter.bi_sector, > bdevname(bio->bi_bdev, b), > count); > diff --git a/block/blk-flush.c b/block/blk-flush.c > index 9c423e5..e092e13 100644 > --- a/block/blk-flush.c > +++ b/block/blk-flush.c > @@ -484,8 +484,9 @@ int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, > > bio = bio_alloc(gfp_mask, 0); > bio->bi_bdev = bdev; > + bio->bi_rw |= WRITE_FLUSH; Shouldn't the |= be a = here and in many other places? Otherwise this loks fine to me. From BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org Sat Jan 9 07:37:29 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 198157F37 for ; Sat, 9 Jan 2016 07:37:29 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 8603BAC002 for ; Sat, 9 Jan 2016 05:37:28 -0800 (PST) X-ASG-Debug-ID: 1452346645-04bdf06f6c2daf0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 7u3aykhbcMXge3uW (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 05:37:26 -0800 (PST) X-Barracuda-Envelope-From: BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aHthk-000084-Gc; Sat, 09 Jan 2016 13:37:16 +0000 Date: Sat, 9 Jan 2016 05:37:16 -0800 From: Christoph Hellwig To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields Message-ID: <20160109133716.GB26660@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-3-git-send-email-mchristi@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452027218-32303-3-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452346646 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 2.60 X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA606_RN_OB, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25981 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 2.00 BSF_SC0_SA606_RN_OB Custom Outbreak Rule SA606_RN_OB Seems like this is missing REQ_OP_FLUSH, which still hides as a write? From BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org Sat Jan 9 07:39:30 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 988797F37 for ; Sat, 9 Jan 2016 07:39:30 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 242EBAC002 for ; Sat, 9 Jan 2016 05:39:30 -0800 (PST) X-ASG-Debug-ID: 1452346767-04cbb05f7f2e580001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id lqnHYmh9yCSnE1HQ (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 05:39:28 -0800 (PST) X-Barracuda-Envelope-From: BATV+e0e8e2f09c0ec55de608+4521+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aHtji-0000Xb-2o; Sat, 09 Jan 2016 13:39:18 +0000 Date: Sat, 9 Jan 2016 05:39:18 -0800 From: Christoph Hellwig To: mchristi@redhat.com Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 26/35] block: set op to REQ_OP Message-ID: <20160109133918.GC26660@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 26/35] block: set op to REQ_OP References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-27-git-send-email-mchristi@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452027218-32303-27-git-send-email-mchristi@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452346767 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.60 X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25981 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 > @@ -1149,10 +1151,12 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, > > blk_rq_init(q, rq); > blk_rq_set_rl(rq, rl); > - rq->cmd_flags = rw_flags | REQ_ALLOCED; > + /* tmp compat - allow users to check either one for the op */ > + rq->cmd_flags = op | op_flags | REQ_ALLOCED; This one is still left after the whole series is applied, seems like the hunk to remove it again got lost somewhere. From dan.j.williams@intel.com Sat Jan 9 08:17:52 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 43FAB7F37 for ; Sat, 9 Jan 2016 08:17:52 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 24D1930405F for ; Sat, 9 Jan 2016 06:17:49 -0800 (PST) X-ASG-Debug-ID: 1452349045-04cb6c75de2e010001-NocioJ Received: from mail-yk0-f171.google.com (mail-yk0-f171.google.com [209.85.160.171]) by cuda.sgi.com with ESMTP id t8m1X6oeye02RiYE (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 06:17:25 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.160.171 Received: by mail-yk0-f171.google.com with SMTP id v14so297456046ykd.3 for ; Sat, 09 Jan 2016 06:17:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=4kt+O+jqSh+lgfDzPkb4E5DaXrRyVMt0+kNa9DZRXIQ=; b=me5Z2pLY9gop2TTfFXrv3meL1Huen5gmMpGqSn9NrUu4Pd8c8zX4CJECPOWGEDtSEm gx+8cRG9BVx/kALcBMrS2ETR5do4BjaiM9sDLj1gnPUZng+FAwKe++gviapI137iy6Ki vnp0aHPbZwFPAioks6evaWhhy97YwyfQzW0wT/qu6niWhzXIrRAiR2iyFMB6IRb9GGdV QpPlUpFHTsNz/z7q5wkejcc96GJ8nn08BZHYrW0QglQc5UdL8wJEboJ6k7to/ML2OqFA P1IlTs26ZbXmWgvbOlQU5mpcGjZNylO9A64uO5+GdRxKDnxFtdztPWa1Idt/8GUzEhOS bPdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=4kt+O+jqSh+lgfDzPkb4E5DaXrRyVMt0+kNa9DZRXIQ=; b=AMD+cSK90jvnCI0xGU9qt8GRzDurGVOPngafailxkJ9oZlxIKavxn6/xE7GHlY2P64 xqMHaZJIu/GMiFI76DHr3tHOvyY5DOyNO/Swm0NjhAkrsEJKNUaI5IEUqtsibULRuOgT 7sogmx0QwMylmUCyUyOgOiAAq7OM63IT2sD93af9G4ieNnb+Z5eQntpUj/zXarIxrPCb vfDWSr+qmohHrIMwDWI2wRfBl7uN0AWOwQPIqoddKHeOCp22CYgIv2PfG1QuBBh0K1wK s5yr2y2hl2GDwsy6InaS4WlFLaJpmxttAQMbD9oXyaNJkwClo5C6WdpvV0mpENb6GRq3 uR5A== X-Gm-Message-State: ALoCoQlVt9CXGOMEz9aeXPDIycdFWEFJM6dB4eZaFpzviYrhpg8ZX8C7Ij9zckRThjGId8cztSo7Rf9Xr2ek5pY/PoerUVPnqgtw0G+B3EyvubjgeDwxLxY= MIME-Version: 1.0 X-Received: by 10.129.85.83 with SMTP id j80mr84589509ywb.28.1452349044764; Sat, 09 Jan 2016 06:17:24 -0800 (PST) Received: by 10.37.202.20 with HTTP; Sat, 9 Jan 2016 06:17:24 -0800 (PST) In-Reply-To: <20160109075414.GA5008@ZenIV.linux.org.uk> References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> <20160109075414.GA5008@ZenIV.linux.org.uk> Date: Sat, 9 Jan 2016 06:17:24 -0800 Message-ID: Subject: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life From: Dan Williams X-ASG-Orig-Subj: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life To: Al Viro Cc: XFS Developers , linux-block@vger.kernel.org, linux-nvdimm , Dave Chinner , Jens Axboe , Jan Kara , linux-fsdevel , Matthew Wilcox , Ross Zwisler Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-yk0-f171.google.com[209.85.160.171] X-Barracuda-Start-Time: 1452349045 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25982 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Fri, Jan 8, 2016 at 11:54 PM, Al Viro wrote: > On Mon, Jan 04, 2016 at 10:20:05AM -0800, Dan Williams wrote: [..] > Would you mind explaining what the hell is _the_ backing device > of a filesystem? What does that translate into in case of e.g. btrfs > spanning several disks? Or ext4 with journal on a different device, for > that matter? > > If anything, I would argue that filesystem is out of place here - > general situation is "IO on X may require IO on device Y and X needs to do > something when Y goes away". Consider e.g. /dev/loop backed by a device > that went away. Or by a file on fs that has run down the curtain and joined > the bleedin choir invisible. With another fs partially hosted by that > loopback device. Or by RAID0 containing said device. > > You are given Y and attempt to locate the affected X. _Then_ > you assume that X is a filesystem and has "something to be done" independent > from the role Y played for it, so you can pick that action from superblock > method. > > IMO you are placing the burden in the wrong place. _Recepient_ > knows what it depends upon and what should be done for each source of > trouble. So make it recepient's responsibility to request notifications. > At which point the superblock method goes away, along with the requirement > to handle all sources of trouble the same way, etc. > > What's more, things like RAID5 (also interested in knowing when > a component has been ripped out) might or might not decide to propagate > the event further - after all, that's exactly the point of redundancy. > > I'd look into something along the lines of notifier chain per > gendisk, with potential victims registering a callback when they decide > that from now on such and such device might screw them over... Makes sense. I'll drop this series for now and come back after re-working it use notifiers. From andrea.gelmini@gmail.com Sat Jan 9 15:02:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 31A237F37 for ; Sat, 9 Jan 2016 15:02:13 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 22ACA304062 for ; Sat, 9 Jan 2016 13:02:10 -0800 (PST) X-ASG-Debug-ID: 1452373326-04cbb05f7e374c0001-NocioJ Received: from mail-wm0-f50.google.com (mail-wm0-f50.google.com [74.125.82.50]) by cuda.sgi.com with ESMTP id v7uFe7KCk6J8EhoJ (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 13:02:07 -0800 (PST) X-Barracuda-Envelope-From: andrea.gelmini@gmail.com X-Barracuda-Apparent-Source-IP: 74.125.82.50 Received: by mail-wm0-f50.google.com with SMTP id l65so170795037wmf.1 for ; Sat, 09 Jan 2016 13:02:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id; bh=JE89lnb9sY3/jhfU2u3qPKvOK0NQW3wB9VmU3+xM94I=; b=ipjyKyzt9CZo0ShINamzSXVdHYhY557gqph8oooBpOq4H1mbWrHJUgs9StSUcK93w+ Bed+Y+dqlc+YtW1H3m00lMa20xm6JTVkijlBOxavbVOwcwQ1MWuM7YbpnhM7W0kQPLJD DldZqZr3n/vfnN5LCzs1cHstJu7SOWmVY/FtS5WCu870122Rzj/7ae/HmT9qhYNitSRg 0+dhgjmDCfxxqkgUQ78ku9Ke/mL2yQgv7Rj4zdrqa6CISHmCoIyDiPBO/olPJPUwEOLC AQQpfQSATQj3EKgGstabFf2UpJwHVqrhnk0ZxuWilRfKNxNfLZqGYbjWqC6dNs9LcRF/ vnhA== X-Received: by 10.194.157.165 with SMTP id wn5mr108213046wjb.41.1452373325855; Sat, 09 Jan 2016 13:02:05 -0800 (PST) Received: from glen.google-analytics.com ([185.5.60.198]) by smtp.gmail.com with ESMTPSA id i2sm22965909wjx.42.2016.01.09.13.02.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 09 Jan 2016 13:02:04 -0800 (PST) Sender: Andrea Gelmini From: Andrea Gelmini To: xfs@oss.sgi.com Cc: darrick.wong@oracle.com, david@fromorbit.com, Andrea Gelmini Subject: [PATCH] Fix typos Date: Sat, 9 Jan 2016 22:01:51 +0100 X-ASG-Orig-Subj: [PATCH] Fix typos Message-Id: <1452373311-31940-1-git-send-email-andrea.gelmini@gelma.net> X-Mailer: git-send-email 2.7.0 X-Barracuda-Connect: mail-wm0-f50.google.com[74.125.82.50] X-Barracuda-Start-Time: 1452373326 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25988 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Reviewed-by: Darrick J. Wong darrick.wong@oracle.com --- admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc | 8 ++++---- admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc | 4 ++-- design/XFS_Filesystem_Structure/magic.asciidoc | 2 +- design/xfs-self-describing-metadata.asciidoc | 2 +- design/xfs-smr-structure.asciidoc | 8 ++++---- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc b/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc index c12981b..30f39bf 100644 --- a/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc +++ b/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc @@ -35,7 +35,7 @@ units as used on the +mkfs.xfs+ command line to configure these parameters. The performance examples given in this section are highly dependent on storage, CPU and RAM configuration. They are intended as guidelines to illustrate behavioural differences, not the exact performance any configuration will -acheive. +achieve. ===== === Directory block size @@ -238,7 +238,7 @@ available for storing attributes. When attributes are stored in the literal area of the inode, both attribute names and attribute values are limited to a maximum size of 254 bytes. If either name or value exceeds 254 bytes in length, or the total space used by the -atributes exceeds the size of the literal area, the entire set of attributes +attributes exceeds the size of the literal area, the entire set of attributes stored on the inode are pushed to a separate attribute block instead of being stored inline. @@ -280,7 +280,7 @@ Therefore, the size of the log determines the concurrency of metadata modification operations the filesystem can sustain, as well as how much and how frequently metadata writeback occurs. A smaller log forces data write-back more frequently than a larger log, but can result in lower -synchronisation overhead as there will be fewer changes aggreagted in memory +synchronisation overhead as there will be fewer changes aggregated in memory between synchronisation triggers. Memory pressure also generates synchronisatin triggers, so large logs may not benefit systems with limited memory. @@ -364,7 +364,7 @@ between 32KB and 256KB. It can be configured by use of the +logbsize+ mount option. The number of log buffers can also be configured to between 2 and 8. The default -is 8 log buffersi and can be configured by the use of the +logbufs+ mount +is 8 log buffers can be configured by the use of the +logbufs+ mount option. It is rare that this needs to be configured, and it should only be considered if there is limited memory and lots of XFS filesystems such that the memory allocated to the log buffers would consume a significant amount of diff --git a/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc b/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc index 0310bbd..b249e35 100644 --- a/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc +++ b/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc @@ -42,8 +42,8 @@ xref:Knowledge[Knowledge Section]. The xref:Process[Process section] will cover the typical processes used to optimise a filesystem for a given workload. If the workload measurements are not -accurate or reproducable, then no conclusions can be drawn as to whether a -configuration changes an improvemnt or not. Hence without a robust testing +accurate or reproducible, then no conclusions can be drawn as to whether a +configuration changes an improvement or not. Hence without a robust testing process, no amount of knowledge or observation will result in a well optimised filesystem configuration. diff --git a/design/XFS_Filesystem_Structure/magic.asciidoc b/design/XFS_Filesystem_Structure/magic.asciidoc index 301cfa0..35d9c2b 100644 --- a/design/XFS_Filesystem_Structure/magic.asciidoc +++ b/design/XFS_Filesystem_Structure/magic.asciidoc @@ -82,5 +82,5 @@ XFS can create really big filesystems! | Max Dir Size | 32GiB | 32GiB | 32GiB |===== -Linux doesn't suppport files or devices larger than 8EiB, so the block +Linux doesn't support files or devices larger than 8EiB, so the block limitations are largely ignorable. diff --git a/design/xfs-self-describing-metadata.asciidoc b/design/xfs-self-describing-metadata.asciidoc index b7dc3ff..d108f7a 100644 --- a/design/xfs-self-describing-metadata.asciidoc +++ b/design/xfs-self-describing-metadata.asciidoc @@ -5,7 +5,7 @@ v1.0, Feb 2014: Initial conversion to asciidoc == Introduction The largest scalability problem facing XFS is not one of algorithmic -scalability, but of verification of the filesystem structure. Scalabilty of the +scalability, but of verification of the filesystem structure. Scalability of the structures and indexes on disk and the algorithms for iterating them are adequate for supporting PB scale filesystems with billions of inodes, however it is this very scalability that causes the verification problem. diff --git a/design/xfs-smr-structure.asciidoc b/design/xfs-smr-structure.asciidoc index dd959ab..3e6c4ec 100644 --- a/design/xfs-smr-structure.asciidoc +++ b/design/xfs-smr-structure.asciidoc @@ -142,7 +142,7 @@ Hence we don't actually need any major new data moving functionality in the kernel to enable this, except maybe an event channel for the kernel to tell xfs_fsr it needs to do some cleaning work. -If we arrange zones into zoen groups, we also have a method for keeping new +If we arrange zones into zone groups, we also have a method for keeping new allocations out of regions we are re-organising. That is, we need to be able to mark zone groups as "read only" so the kernel will not attempt to allocate from them while the cleaner is running and re-organising the data within the zones in @@ -173,7 +173,7 @@ it will need ot be packaged by distros. If mkfs cannot find ensough random write space for the amount of metadata we need to track all the space in the sequential write zones and a decent amount of -internal fielsystem metadata (inodes, etc) then it will need to fail. Drive +internal filesystem metadata (inodes, etc) then it will need to fail. Drive vendors are going to need to provide sufficient space in these regions for us to be able to make use of it, otherwise we'll simply not be able to do what we need to do. @@ -193,7 +193,7 @@ bitmaps for verifying used space should already be there. THere be dragons waiting for us if we don't have random write zones for metadata. If that happens, we cannot repair metadata in place and we will have to redesign xfs_repair from the ground up to support such functionality. That's -jus tnot going to happen, so we'll need drives with a significant amount of +just not going to happen, so we'll need drives with a significant amount of random write space for all our metadata...... == Quantification of Random Write Zone Capacity @@ -316,7 +316,7 @@ spiral. I suspect the best we will be able to do with fallocate based preallocation is to mark the region as delayed allocation. -=== Allocation Alignemnt +=== Allocation Alignment With zone based write pointers, we lose all capability of write alignment to the underlying storage - our only choice to write is the current set of write -- 2.7.0 From darrick.wong@oracle.com Sat Jan 9 15:10:44 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 508987F47 for ; Sat, 9 Jan 2016 15:10:44 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3C085304053 for ; Sat, 9 Jan 2016 13:10:44 -0800 (PST) X-ASG-Debug-ID: 1452373841-04cb6c75dc35600001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id ixRTrpctF8cJGKNQ (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 09 Jan 2016 13:10:42 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u09LAbZv027267 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sat, 9 Jan 2016 21:10:38 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u09LAbAW024425 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sat, 9 Jan 2016 21:10:37 GMT Received: from abhmp0007.oracle.com (abhmp0007.oracle.com [141.146.116.13]) by userv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u09LAbpf009854; Sat, 9 Jan 2016 21:10:37 GMT Received: from localhost (/24.130.224.127) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sat, 09 Jan 2016 13:10:36 -0800 Date: Sat, 9 Jan 2016 13:10:35 -0800 From: "Darrick J. Wong" To: Andrea Gelmini Cc: xfs@oss.sgi.com, david@fromorbit.com Subject: Re: [PATCH] Fix typos Message-ID: <20160109211035.GA6112@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH] Fix typos References: <1452373311-31940-1-git-send-email-andrea.gelmini@gelma.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452373311-31940-1-git-send-email-andrea.gelmini@gelma.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0022.oracle.com [156.151.31.74] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1452373841 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25988 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Sat, Jan 09, 2016 at 10:01:51PM +0100, Andrea Gelmini wrote: > Reviewed-by: Darrick J. Wong darrick.wong@oracle.com That ought to be: Reviewed-by: Darrick J. Wong (Note the angle brackets.) Can we get a 'Signed-off-by' tag with your email address in it? The tag is useful for us to keep track of who's contributing what, and certifies that each contributor knows what they're getting into. :) See section 11 in: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/SubmittingPatches The actual documentation fixes still look fine to me. --D > > --- > admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc | 8 ++++---- > admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc | 4 ++-- > design/XFS_Filesystem_Structure/magic.asciidoc | 2 +- > design/xfs-self-describing-metadata.asciidoc | 2 +- > design/xfs-smr-structure.asciidoc | 8 ++++---- > 5 files changed, 12 insertions(+), 12 deletions(-) > > diff --git a/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc b/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc > index c12981b..30f39bf 100644 > --- a/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc > +++ b/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc > @@ -35,7 +35,7 @@ units as used on the +mkfs.xfs+ command line to configure these parameters. > The performance examples given in this section are highly dependent on storage, > CPU and RAM configuration. They are intended as guidelines to illustrate > behavioural differences, not the exact performance any configuration will > -acheive. > +achieve. > ===== > > === Directory block size > @@ -238,7 +238,7 @@ available for storing attributes. > When attributes are stored in the literal area of the inode, both attribute > names and attribute values are limited to a maximum size of 254 bytes. If either > name or value exceeds 254 bytes in length, or the total space used by the > -atributes exceeds the size of the literal area, the entire set of attributes > +attributes exceeds the size of the literal area, the entire set of attributes > stored on the inode are pushed to a separate attribute block instead of being > stored inline. > > @@ -280,7 +280,7 @@ Therefore, the size of the log determines the concurrency of metadata > modification operations the filesystem can sustain, as well as how much and how > frequently metadata writeback occurs. A smaller log forces data > write-back more frequently than a larger log, but can result in lower > -synchronisation overhead as there will be fewer changes aggreagted in memory > +synchronisation overhead as there will be fewer changes aggregated in memory > between synchronisation triggers. Memory pressure also generates synchronisatin > triggers, so large logs may not benefit systems with limited memory. > > @@ -364,7 +364,7 @@ between 32KB and 256KB. It can be configured by use of the +logbsize+ mount > option. > > The number of log buffers can also be configured to between 2 and 8. The default > -is 8 log buffersi and can be configured by the use of the +logbufs+ mount > +is 8 log buffers can be configured by the use of the +logbufs+ mount > option. It is rare that this needs to be configured, and it should only be > considered if there is limited memory and lots of XFS filesystems such that the > memory allocated to the log buffers would consume a significant amount of > diff --git a/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc b/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc > index 0310bbd..b249e35 100644 > --- a/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc > +++ b/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc > @@ -42,8 +42,8 @@ xref:Knowledge[Knowledge Section]. > > The xref:Process[Process section] will cover the typical processes used to > optimise a filesystem for a given workload. If the workload measurements are not > -accurate or reproducable, then no conclusions can be drawn as to whether a > -configuration changes an improvemnt or not. Hence without a robust testing > +accurate or reproducible, then no conclusions can be drawn as to whether a > +configuration changes an improvement or not. Hence without a robust testing > process, no amount of knowledge or observation will result in a well optimised > filesystem configuration. > > diff --git a/design/XFS_Filesystem_Structure/magic.asciidoc b/design/XFS_Filesystem_Structure/magic.asciidoc > index 301cfa0..35d9c2b 100644 > --- a/design/XFS_Filesystem_Structure/magic.asciidoc > +++ b/design/XFS_Filesystem_Structure/magic.asciidoc > @@ -82,5 +82,5 @@ XFS can create really big filesystems! > | Max Dir Size | 32GiB | 32GiB | 32GiB > |===== > > -Linux doesn't suppport files or devices larger than 8EiB, so the block > +Linux doesn't support files or devices larger than 8EiB, so the block > limitations are largely ignorable. > diff --git a/design/xfs-self-describing-metadata.asciidoc b/design/xfs-self-describing-metadata.asciidoc > index b7dc3ff..d108f7a 100644 > --- a/design/xfs-self-describing-metadata.asciidoc > +++ b/design/xfs-self-describing-metadata.asciidoc > @@ -5,7 +5,7 @@ v1.0, Feb 2014: Initial conversion to asciidoc > == Introduction > > The largest scalability problem facing XFS is not one of algorithmic > -scalability, but of verification of the filesystem structure. Scalabilty of the > +scalability, but of verification of the filesystem structure. Scalability of the > structures and indexes on disk and the algorithms for iterating them are > adequate for supporting PB scale filesystems with billions of inodes, however it > is this very scalability that causes the verification problem. > diff --git a/design/xfs-smr-structure.asciidoc b/design/xfs-smr-structure.asciidoc > index dd959ab..3e6c4ec 100644 > --- a/design/xfs-smr-structure.asciidoc > +++ b/design/xfs-smr-structure.asciidoc > @@ -142,7 +142,7 @@ Hence we don't actually need any major new data moving functionality in the > kernel to enable this, except maybe an event channel for the kernel to tell > xfs_fsr it needs to do some cleaning work. > > -If we arrange zones into zoen groups, we also have a method for keeping new > +If we arrange zones into zone groups, we also have a method for keeping new > allocations out of regions we are re-organising. That is, we need to be able to > mark zone groups as "read only" so the kernel will not attempt to allocate from > them while the cleaner is running and re-organising the data within the zones in > @@ -173,7 +173,7 @@ it will need ot be packaged by distros. > > If mkfs cannot find ensough random write space for the amount of metadata we > need to track all the space in the sequential write zones and a decent amount of > -internal fielsystem metadata (inodes, etc) then it will need to fail. Drive > +internal filesystem metadata (inodes, etc) then it will need to fail. Drive > vendors are going to need to provide sufficient space in these regions for us > to be able to make use of it, otherwise we'll simply not be able to do what we > need to do. > @@ -193,7 +193,7 @@ bitmaps for verifying used space should already be there. > THere be dragons waiting for us if we don't have random write zones for > metadata. If that happens, we cannot repair metadata in place and we will have > to redesign xfs_repair from the ground up to support such functionality. That's > -jus tnot going to happen, so we'll need drives with a significant amount of > +just not going to happen, so we'll need drives with a significant amount of > random write space for all our metadata...... > > == Quantification of Random Write Zone Capacity > @@ -316,7 +316,7 @@ spiral. > I suspect the best we will be able to do with fallocate based preallocation is > to mark the region as delayed allocation. > > -=== Allocation Alignemnt > +=== Allocation Alignment > > With zone based write pointers, we lose all capability of write alignment to the > underlying storage - our only choice to write is the current set of write > -- > 2.7.0 > From darrick.wong@oracle.com Sat Jan 9 15:17:24 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id EEA217F50 for ; Sat, 9 Jan 2016 15:17:23 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 6A4D3AC002 for ; Sat, 9 Jan 2016 13:17:20 -0800 (PST) X-ASG-Debug-ID: 1452374238-04cbb05f7c37960001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id ybYmERoQU3RZbuKA (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 09 Jan 2016 13:17:18 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u09LHFFM031653 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sat, 9 Jan 2016 21:17:16 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u09LHFPY029694 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sat, 9 Jan 2016 21:17:15 GMT Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19]) by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u09LHEOL006843; Sat, 9 Jan 2016 21:17:15 GMT Received: from localhost (/24.130.224.127) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sat, 09 Jan 2016 13:17:14 -0800 Date: Sat, 9 Jan 2016 13:17:13 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160109211713.GB6112@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> <20160105014310.GK28330@birch.djwong.org> <20160105104214.GA16310@infradead.org> <20160107003227.GB8015@birch.djwong.org> <20160107152541.GA16982@lst.de> <20160108100933.GN28330@birch.djwong.org> <20160108134704.GA6708@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160108134704.GA6708@lst.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0022.oracle.com [141.146.126.234] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1452374238 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25988 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Fri, Jan 08, 2016 at 02:47:04PM +0100, Christoph Hellwig wrote: > Thanks Darrick! > > I've not seen the inode eviction asserts anymore, but I now hit a > corruption warnings in generic/168 reliably. I did hit before as > well, but not very reliably. I'll see if I can repro the 168 error; it's been running in a loop all night and hasn't bombed yet. In the meantime, I added some more tests and fixed a CoW corruption when an xfs_io_overwrite extent has cow reservations in the middle of the extent. I also restarted testing on arm64, ppc64{,el}, and i686; it seems stable enough right now to pass all ~130 reflink xfstests here. > generic/168 17s ...[ 296.988867] XFS (vdc): Metadata corruption detected at > xfs_refcountbt_write_verify+0x3e/0x90, block 0xb0 > [ 296.990412] XFS (vdc): Unmount and run xfs_repair > [ 296.991300] XFS (vdc): First 64 bytes of corrupted metadata buffer: > [ 296.992349] ffff88007b755000: 52 33 46 43 00 01 00 01 ff ff ff ff ff ff ff > ff R3FC............ > [ 296.993893] ffff88007b755010: 00 00 00 00 00 00 00 b0 00 00 00 00 00 00 00 > 00 ................ No LSN? That's ... odd. > [ 297.003293] ffff88007b755020: fe 94 c7 69 fd bd 4a 77 85 3d 9d 6d 7b 45 38 > 07 ...i..Jw.=.m{E8. Does this match the FS UUID? > [ 297.004442] ffff88007b755030: 00 00 00 00 00 00 00 00 00 00 0b 9c 00 00 7f > 07 ................ Also no owner or CRC. Hmmmm. > [ 297.005867] XFS (vdc): xfs_do_force_shutdown(0x8) called from line 1247 of > file fs/xfs/xfs_buf.c. Return address = 0xffffffff814cce52 > > I'll investigate it in a little more detail. --D From andrea.gelmini@gmail.com Sat Jan 9 15:29:21 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id A7D967F52 for ; Sat, 9 Jan 2016 15:29:21 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 44022AC001 for ; Sat, 9 Jan 2016 13:29:21 -0800 (PST) X-ASG-Debug-ID: 1452374957-04bdf06f6b36f40001-NocioJ Received: from mail-wm0-f43.google.com (mail-wm0-f43.google.com [74.125.82.43]) by cuda.sgi.com with ESMTP id l3707KRV1NWAeWxj (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 13:29:18 -0800 (PST) X-Barracuda-Envelope-From: andrea.gelmini@gmail.com X-Barracuda-Apparent-Source-IP: 74.125.82.43 Received: by mail-wm0-f43.google.com with SMTP id b14so217592601wmb.1 for ; Sat, 09 Jan 2016 13:29:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id; bh=zIPKLbwEgBwVXw9FMp07k5dUzjdA3mjIat+OE3aJSP8=; b=Ej4K0Nzf+puh0Fg0AqnOLrO8A964oG/F7h6d2+TYW1ieCG+Y+AGgBw344nC8/Kskt2 pd4WB1oZzCEh6aDzpAsF27pMrfItO0K/Ecne68KGXUgsArKZru5yjCJAzrrX2Cl08gbv eWyU7PVgtjcbi6Au3iP4gHuiYYMCLDpCnHD2FJm6Fs/tiMD553FbmnLYHCN7SRb9Wwd6 Eafqzi3WocuWEAUsQK16jhwSTi8q/IFFIoKDQ6241t7q8QsiR8+fL5YlNOB8I8MucSHz wmX8fm9taw5GN0vm+KhH+FsMp7v3353vfCFGVzQ5bnPQgmoih/CQQ3C8U8KK96uG2Cmm RSaA== X-Received: by 10.28.57.65 with SMTP id g62mr5128933wma.65.1452374956891; Sat, 09 Jan 2016 13:29:16 -0800 (PST) Received: from glen.google-analytics.com ([185.5.60.198]) by smtp.gmail.com with ESMTPSA id s8sm60839610wje.35.2016.01.09.13.29.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 09 Jan 2016 13:29:15 -0800 (PST) Sender: Andrea Gelmini From: Andrea Gelmini To: xfs@oss.sgi.com Cc: darrick.wong@oracle.com, david@fromorbit.com, Andrea Gelmini Subject: [PATCH] XFS: fix typos Date: Sat, 9 Jan 2016 22:29:05 +0100 X-ASG-Orig-Subj: [PATCH] XFS: fix typos Message-Id: <1452374945-560-1-git-send-email-andrea.gelmini@gelma.net> X-Mailer: git-send-email 2.7.0 X-Barracuda-Connect: mail-wm0-f43.google.com[74.125.82.43] X-Barracuda-Start-Time: 1452374958 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25989 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature It cleanely applies to Linus' master: 751e5f5c753e8d447bcf89f9e96b9616ac081628 and XFS for-next branch: 4922be51ef1a95ca6a38694cf0cde5dd0308a24e Signed-off-by: Andrea Gelmini --- fs/xfs/libxfs/xfs_btree.c | 4 ++-- fs/xfs/libxfs/xfs_trans_resv.c | 2 +- fs/xfs/xfs_bmap_util.c | 2 +- fs/xfs/xfs_icache.c | 2 +- fs/xfs/xfs_ioctl32.c | 2 +- fs/xfs/xfs_log.c | 2 +- fs/xfs/xfs_log_cil.c | 2 +- fs/xfs/xfs_log_priv.h | 2 +- fs/xfs/xfs_super.c | 2 +- fs/xfs/xfs_trans_ail.c | 2 +- fs/xfs/xfs_trans_priv.h | 2 +- 11 files changed, 12 insertions(+), 12 deletions(-) diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c index af1bbee..2423337 100644 --- a/fs/xfs/libxfs/xfs_btree.c +++ b/fs/xfs/libxfs/xfs_btree.c @@ -222,7 +222,7 @@ xfs_btree_check_ptr( * Calculate CRC on the whole btree block and stuff it into the * long-form btree header. * - * Prior to calculting the CRC, pull the LSN out of the buffer log item and put + * Prior to calculating the CRC, pull the LSN out of the buffer log item and put * it into the buffer so recovery knows what the last modification was that made * it to disk. */ @@ -260,7 +260,7 @@ xfs_btree_lblock_verify_crc( * Calculate CRC on the whole btree block and stuff it into the * short-form btree header. * - * Prior to calculting the CRC, pull the LSN out of the buffer log item and put + * Prior to calculating the CRC, pull the LSN out of the buffer log item and put * it into the buffer so recovery knows what the last modification was that made * it to disk. */ diff --git a/fs/xfs/libxfs/xfs_trans_resv.c b/fs/xfs/libxfs/xfs_trans_resv.c index 68cb1e7..14258f3 100644 --- a/fs/xfs/libxfs/xfs_trans_resv.c +++ b/fs/xfs/libxfs/xfs_trans_resv.c @@ -666,7 +666,7 @@ xfs_calc_attrsetm_reservation( * Since the runtime attribute transaction space is dependent on the total * blocks needed for the 1st bmap, here we calculate out the space unit for * one block so that the caller could figure out the total space according - * to the attibute extent length in blocks by: + * to the attribute extent length in blocks by: * ext * M_RES(mp)->tr_attrsetrt.tr_logres */ STATIC uint diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c index dbae649..1163cfb 100644 --- a/fs/xfs/xfs_bmap_util.c +++ b/fs/xfs/xfs_bmap_util.c @@ -741,7 +741,7 @@ xfs_getbmap( } /* - * dead simple method of punching delalyed allocation blocks from a range in + * dead simple method of punching delayed allocation blocks from a range in * the inode. Walks a block at a time so will be slow, but is only executed in * rare error cases so the overhead is not critical. This will always punch out * both the start and end blocks, even if the ranges only partially overlap diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index d7a490f..ee1e0c5 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -403,7 +403,7 @@ xfs_iget( * xfs_reclaim_inode() uses the ILOCK to ensure an inode * doesn't get freed while it's being referenced during a * radix tree traversal here. It assumes this function - * aqcuires only the ILOCK (and therefore it has no need to + * acquires only the ILOCK (and therefore it has no need to * involve the IOLOCK in this synchronization). */ ASSERT((lock_flags & (XFS_IOLOCK_EXCL | XFS_IOLOCK_SHARED)) == 0); diff --git a/fs/xfs/xfs_ioctl32.c b/fs/xfs/xfs_ioctl32.c index 1a05d8a..0978ec7 100644 --- a/fs/xfs/xfs_ioctl32.c +++ b/fs/xfs/xfs_ioctl32.c @@ -618,7 +618,7 @@ xfs_file_compat_ioctl( return error; } #endif - /* long changes size, but xfs only copiese out 32 bits */ + /* long changes size, but xfs only copies out 32 bits */ case XFS_IOC_GETXFLAGS_32: case XFS_IOC_SETXFLAGS_32: case XFS_IOC_GETVERSION_32: diff --git a/fs/xfs/xfs_log.c b/fs/xfs/xfs_log.c index f52c72a..8d7685d 100644 --- a/fs/xfs/xfs_log.c +++ b/fs/xfs/xfs_log.c @@ -1835,7 +1835,7 @@ xlog_sync( iclog->ic_bwritecnt = 1; } - /* calculcate the checksum */ + /* calculate the checksum */ iclog->ic_header.h_crc = xlog_cksum(log, &iclog->ic_header, iclog->ic_datap, size); diff --git a/fs/xfs/xfs_log_cil.c b/fs/xfs/xfs_log_cil.c index 4e76493..1b10909 100644 --- a/fs/xfs/xfs_log_cil.c +++ b/fs/xfs/xfs_log_cil.c @@ -144,7 +144,7 @@ xfs_cil_prepare_item( * regions into the flat buffer. We can do this because we still have to do a * formatting step to write the regions into the iclog buffer. Writing the * ophdrs during the iclog write means that we can support splitting large - * regions across iclog boundares without needing a change in the format of the + * regions across iclog boundaries without needing a change in the format of the * item/region encapsulation. * * Hence what we need to do now is change the rewrite the vector array to point diff --git a/fs/xfs/xfs_log_priv.h b/fs/xfs/xfs_log_priv.h index 8daba74..fb97fac 100644 --- a/fs/xfs/xfs_log_priv.h +++ b/fs/xfs/xfs_log_priv.h @@ -341,7 +341,7 @@ struct xfs_cil { #define XLOG_CIL_SPACE_LIMIT(log) (log->l_logsize >> 3) /* - * ticket grant locks, queues and accounting have their own cachlines + * ticket grant locks, queues and accounting have their own cachelines * as these are quite hot and can be operated on concurrently. */ struct xlog_grant_head { diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index 36bd882..02141da 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -645,7 +645,7 @@ xfs_set_inode64(struct xfs_mount *mp, xfs_agnumber_t agcount) /* There is no need for lock protection on m_flags, * the rw_semaphore of the VFS superblock is locked * during mount/umount/remount operations, so this is - * enough to avoid concurency on the m_flags field + * enough to avoid concurrency on the m_flags field */ mp->m_flags &= ~(XFS_MOUNT_32BITINODES | XFS_MOUNT_SMALL_INUMS); diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c index aa67339..f0525d4 100644 --- a/fs/xfs/xfs_trans_ail.c +++ b/fs/xfs/xfs_trans_ail.c @@ -547,7 +547,7 @@ xfsaild( * The push is run asynchronously in a workqueue, which means the caller needs * to handle waiting on the async flush for space to become available. * We don't want to interrupt any push that is in progress, hence we only queue - * work if we set the pushing bit approriately. + * work if we set the pushing bit appropriately. * * We do this unlocked - we only need to know whether there is anything in the * AIL at the time we are called. We don't need to access the contents of diff --git a/fs/xfs/xfs_trans_priv.h b/fs/xfs/xfs_trans_priv.h index 49931b7..2fc9b33 100644 --- a/fs/xfs/xfs_trans_priv.h +++ b/fs/xfs/xfs_trans_priv.h @@ -39,7 +39,7 @@ void xfs_trans_committed_bulk(struct xfs_ail *ailp, struct xfs_log_vec *lv, * AIL traversal cursor. * * Rather than using a generation number for detecting changes in the ail, use - * a cursor that is protected by the ail lock. The aild cursor exists in the + * a cursor that is protected by the ail lock. The ail cursor exists in the * struct xfs_ail, but other traversals can declare it on the stack and link it * to the ail list. * -- 2.7.0 From andrea.gelmini@gmail.com Sat Jan 9 16:13:33 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 247007F54 for ; Sat, 9 Jan 2016 16:13:33 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id DB9A630405F for ; Sat, 9 Jan 2016 14:13:29 -0800 (PST) X-ASG-Debug-ID: 1452377606-04bdf06f6e37d80001-NocioJ Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by cuda.sgi.com with ESMTP id 2QRsDSQ0a9v9fi2G (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 14:13:27 -0800 (PST) X-Barracuda-Envelope-From: andrea.gelmini@gmail.com X-Barracuda-Apparent-Source-IP: 74.125.82.42 Received: by mail-wm0-f42.google.com with SMTP id l65so171727824wmf.1 for ; Sat, 09 Jan 2016 14:13:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id; bh=yEnIGfARRG50u7Ji537A1fiFhluZQASZ2W9xAWx1c1w=; b=tzwmBGhuQ+yOufEUZwr8ySFT4T9VQocTt9x8Z0JxDfWtoX+5hY+vEVQta0VCV0O2fw T32q+raWrzhkrJrprbVyAsiIi4jodBQqj15gcZRzq02eb2BGRpkugvubUOnq9nu9JBCL AsDNFZpmeViLvfTxqGRyfRcHd+Fbp2YrOzr+1IvNOdr78f489GsYSqLZyNB0lXiy7E3Q RhisMw2G+fSt8cZmAJjbNIk7+9xcbEXSi4QE/OzUsayyRRUh3/PabhNwqU0KQsdbf3WQ m+HmGXRTlMkD45p48taMlswdYNpDlh6hU0MtwYE91CI5S2r0aIQSpo1s3HtxqkfLZdCx dpLw== X-Received: by 10.28.146.145 with SMTP id u139mr5421483wmd.81.1452377606203; Sat, 09 Jan 2016 14:13:26 -0800 (PST) Received: from glen.google-analytics.com ([185.5.60.198]) by smtp.gmail.com with ESMTPSA id m128sm5498811wma.24.2016.01.09.14.13.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 09 Jan 2016 14:13:24 -0800 (PST) Sender: Andrea Gelmini From: Andrea Gelmini To: xfs@oss.sgi.com Cc: darrick.wong@oracle.com, david@fromorbit.com, Andrea Gelmini Subject: [PATCH] Fix typos Date: Sat, 9 Jan 2016 23:13:16 +0100 X-ASG-Orig-Subj: [PATCH] Fix typos Message-Id: <1452377596-4379-1-git-send-email-andrea.gelmini@gelma.net> X-Mailer: git-send-email 2.7.0 X-Barracuda-Connect: mail-wm0-f42.google.com[74.125.82.42] X-Barracuda-Start-Time: 1452377607 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25989 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Signed-off-by: Andrea Gelmini Reviewed-by: Darrick J. Wong --- admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc | 8 ++++---- admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc | 4 ++-- design/XFS_Filesystem_Structure/magic.asciidoc | 2 +- design/xfs-self-describing-metadata.asciidoc | 2 +- design/xfs-smr-structure.asciidoc | 8 ++++---- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc b/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc index c12981b..30f39bf 100644 --- a/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc +++ b/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc @@ -35,7 +35,7 @@ units as used on the +mkfs.xfs+ command line to configure these parameters. The performance examples given in this section are highly dependent on storage, CPU and RAM configuration. They are intended as guidelines to illustrate behavioural differences, not the exact performance any configuration will -acheive. +achieve. ===== === Directory block size @@ -238,7 +238,7 @@ available for storing attributes. When attributes are stored in the literal area of the inode, both attribute names and attribute values are limited to a maximum size of 254 bytes. If either name or value exceeds 254 bytes in length, or the total space used by the -atributes exceeds the size of the literal area, the entire set of attributes +attributes exceeds the size of the literal area, the entire set of attributes stored on the inode are pushed to a separate attribute block instead of being stored inline. @@ -280,7 +280,7 @@ Therefore, the size of the log determines the concurrency of metadata modification operations the filesystem can sustain, as well as how much and how frequently metadata writeback occurs. A smaller log forces data write-back more frequently than a larger log, but can result in lower -synchronisation overhead as there will be fewer changes aggreagted in memory +synchronisation overhead as there will be fewer changes aggregated in memory between synchronisation triggers. Memory pressure also generates synchronisatin triggers, so large logs may not benefit systems with limited memory. @@ -364,7 +364,7 @@ between 32KB and 256KB. It can be configured by use of the +logbsize+ mount option. The number of log buffers can also be configured to between 2 and 8. The default -is 8 log buffersi and can be configured by the use of the +logbufs+ mount +is 8 log buffers can be configured by the use of the +logbufs+ mount option. It is rare that this needs to be configured, and it should only be considered if there is limited memory and lots of XFS filesystems such that the memory allocated to the log buffers would consume a significant amount of diff --git a/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc b/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc index 0310bbd..b249e35 100644 --- a/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc +++ b/admin/XFS_Performance_Tuning/xfs_performance_tuning.asciidoc @@ -42,8 +42,8 @@ xref:Knowledge[Knowledge Section]. The xref:Process[Process section] will cover the typical processes used to optimise a filesystem for a given workload. If the workload measurements are not -accurate or reproducable, then no conclusions can be drawn as to whether a -configuration changes an improvemnt or not. Hence without a robust testing +accurate or reproducible, then no conclusions can be drawn as to whether a +configuration changes an improvement or not. Hence without a robust testing process, no amount of knowledge or observation will result in a well optimised filesystem configuration. diff --git a/design/XFS_Filesystem_Structure/magic.asciidoc b/design/XFS_Filesystem_Structure/magic.asciidoc index 301cfa0..35d9c2b 100644 --- a/design/XFS_Filesystem_Structure/magic.asciidoc +++ b/design/XFS_Filesystem_Structure/magic.asciidoc @@ -82,5 +82,5 @@ XFS can create really big filesystems! | Max Dir Size | 32GiB | 32GiB | 32GiB |===== -Linux doesn't suppport files or devices larger than 8EiB, so the block +Linux doesn't support files or devices larger than 8EiB, so the block limitations are largely ignorable. diff --git a/design/xfs-self-describing-metadata.asciidoc b/design/xfs-self-describing-metadata.asciidoc index b7dc3ff..d108f7a 100644 --- a/design/xfs-self-describing-metadata.asciidoc +++ b/design/xfs-self-describing-metadata.asciidoc @@ -5,7 +5,7 @@ v1.0, Feb 2014: Initial conversion to asciidoc == Introduction The largest scalability problem facing XFS is not one of algorithmic -scalability, but of verification of the filesystem structure. Scalabilty of the +scalability, but of verification of the filesystem structure. Scalability of the structures and indexes on disk and the algorithms for iterating them are adequate for supporting PB scale filesystems with billions of inodes, however it is this very scalability that causes the verification problem. diff --git a/design/xfs-smr-structure.asciidoc b/design/xfs-smr-structure.asciidoc index dd959ab..3e6c4ec 100644 --- a/design/xfs-smr-structure.asciidoc +++ b/design/xfs-smr-structure.asciidoc @@ -142,7 +142,7 @@ Hence we don't actually need any major new data moving functionality in the kernel to enable this, except maybe an event channel for the kernel to tell xfs_fsr it needs to do some cleaning work. -If we arrange zones into zoen groups, we also have a method for keeping new +If we arrange zones into zone groups, we also have a method for keeping new allocations out of regions we are re-organising. That is, we need to be able to mark zone groups as "read only" so the kernel will not attempt to allocate from them while the cleaner is running and re-organising the data within the zones in @@ -173,7 +173,7 @@ it will need ot be packaged by distros. If mkfs cannot find ensough random write space for the amount of metadata we need to track all the space in the sequential write zones and a decent amount of -internal fielsystem metadata (inodes, etc) then it will need to fail. Drive +internal filesystem metadata (inodes, etc) then it will need to fail. Drive vendors are going to need to provide sufficient space in these regions for us to be able to make use of it, otherwise we'll simply not be able to do what we need to do. @@ -193,7 +193,7 @@ bitmaps for verifying used space should already be there. THere be dragons waiting for us if we don't have random write zones for metadata. If that happens, we cannot repair metadata in place and we will have to redesign xfs_repair from the ground up to support such functionality. That's -jus tnot going to happen, so we'll need drives with a significant amount of +just not going to happen, so we'll need drives with a significant amount of random write space for all our metadata...... == Quantification of Random Write Zone Capacity @@ -316,7 +316,7 @@ spiral. I suspect the best we will be able to do with fallocate based preallocation is to mark the region as delayed allocation. -=== Allocation Alignemnt +=== Allocation Alignment With zone based write pointers, we lose all capability of write alignment to the underlying storage - our only choice to write is the current set of write -- 2.7.0 From michaelc@cs.wisc.edu Sat Jan 9 18:45:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id A70E47F47 for ; Sat, 9 Jan 2016 18:45:00 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 429EEAC004 for ; Sat, 9 Jan 2016 16:45:00 -0800 (PST) X-ASG-Debug-ID: 1452386695-04bdf06f6d3ac20001-NocioJ Received: from sabe.cs.wisc.edu (sabe.cs.wisc.edu [128.105.6.20]) by cuda.sgi.com with ESMTP id u4Z4JDCTKP4ZYdBs (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 16:44:55 -0800 (PST) X-Barracuda-Envelope-From: michaelc@cs.wisc.edu X-Barracuda-Apparent-Source-IP: 128.105.6.20 Received: from [20.15.0.202] (c-24-245-27-162.hsd1.mn.comcast.net [24.245.27.162]) (authenticated bits=0) by sabe.cs.wisc.edu (8.14.7/8.14.1) with ESMTP id u0A0hbFD008192 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sat, 9 Jan 2016 18:43:39 -0600 Message-ID: <5691A934.8020000@cs.wisc.edu> Date: Sat, 09 Jan 2016 18:43:32 -0600 From: Mike Christie User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Christoph Hellwig , mchristi@redhat.com CC: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: Re: [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-2-git-send-email-mchristi@redhat.com> <20160109133409.GA26660@infradead.org> X-ASG-Orig-Subj: Re: [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio In-Reply-To: <20160109133409.GA26660@infradead.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sabe.cs.wisc.edu[128.105.6.20] X-Barracuda-Start-Time: 1452386695 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25992 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On 01/09/2016 07:34 AM, Christoph Hellwig wrote: >> count); >> > diff --git a/block/blk-flush.c b/block/blk-flush.c >> > index 9c423e5..e092e13 100644 >> > --- a/block/blk-flush.c >> > +++ b/block/blk-flush.c >> > @@ -484,8 +484,9 @@ int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, >> > >> > bio = bio_alloc(gfp_mask, 0); >> > bio->bi_bdev = bdev; >> > + bio->bi_rw |= WRITE_FLUSH; > Shouldn't the |= be a = here and in many other places? Yeah, I was trying to merge bi_rw and bio_flags. I am not going to send that. I will fix up the patchset. From michaelc@cs.wisc.edu Sat Jan 9 18:57:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 1204F7F54 for ; Sat, 9 Jan 2016 18:57:03 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 8EBEAAC002 for ; Sat, 9 Jan 2016 16:57:02 -0800 (PST) X-ASG-Debug-ID: 1452387420-04cb6c75db39c30001-NocioJ Received: from sabe.cs.wisc.edu (sabe.cs.wisc.edu [128.105.6.20]) by cuda.sgi.com with ESMTP id UEr8VH0zBEZXqNOk (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 16:57:00 -0800 (PST) X-Barracuda-Envelope-From: michaelc@cs.wisc.edu X-Barracuda-Apparent-Source-IP: 128.105.6.20 Received: from [20.15.0.202] (c-24-245-27-162.hsd1.mn.comcast.net [24.245.27.162]) (authenticated bits=0) by sabe.cs.wisc.edu (8.14.7/8.14.1) with ESMTP id u0A0uJqp008236 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sat, 9 Jan 2016 18:56:19 -0600 Message-ID: <5691AC2E.2050403@cs.wisc.edu> Date: Sat, 09 Jan 2016 18:56:14 -0600 From: Mike Christie User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: device-mapper development , mchristi@redhat.com CC: linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, xfs@oss.sgi.com, linux-scsi@vger.kernel.org, konrad.wilk@oracle.com, linux-kernel@vger.kernel.org, philipp.reisner@linbit.com, linux-f2fs-devel@lists.sourceforge.net, linux-raid@vger.kernel.org, ocfs2-devel@oss.oracle.com, target-devel@vger.kernel.org, linux-mtd@lists.infradead.org, osd-dev@open-osd.org, linux-fsdevel@vger.kernel.org, lars.ellenberg@linbit.com, linux-ext4@vger.kernel.org, linux-btrfs@vger.kernel.org, drbd-dev@lists.linbit.com Subject: Re: [dm-devel] [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-3-git-send-email-mchristi@redhat.com> <20160109133716.GB26660@infradead.org> X-ASG-Orig-Subj: Re: [dm-devel] [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields In-Reply-To: <20160109133716.GB26660@infradead.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sabe.cs.wisc.edu[128.105.6.20] X-Barracuda-Start-Time: 1452387420 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25992 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On 01/09/2016 07:37 AM, Christoph Hellwig wrote: > Seems like this is missing REQ_OP_FLUSH, which still hides as a write? > I might have misunderstood what you wanted or am misunderstanding you now. I did not change the flush related code until the last patches. I added REQ_OP_FLUSH in patch: [PATCH 33/35] block, drivers: add REQ_OP_FLUSH operation commit 18dea20363ba245a47bc1bb54f6465b8a05b19af Author: Mike Christie Date: Tue Jan 5 21:02:17 2016 -0600 block, drivers: add REQ_OP_FLUSH operation and I added the REQ_PREFLUSH flag in [PATCH 35/35] block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH commit 303dc6a7cc3673065538ba041562fcd833a619af Author: Mike Christie Date: Thu Jan 7 18:17:03 2016 -0600 block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH I did them in separate patches, because I was not sure if they were correct with what you requested before, and because they were a little more tricky because of how request_fn drivers, make_request_fn drivers, and dm-multipath handle flushes differently. From michaelc@cs.wisc.edu Sat Jan 9 19:22:09 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 6CF2D7F50 for ; Sat, 9 Jan 2016 19:22:09 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id E73C1AC002 for ; Sat, 9 Jan 2016 17:22:05 -0800 (PST) X-ASG-Debug-ID: 1452388922-04cbb05f7c3c7e0001-NocioJ Received: from sabe.cs.wisc.edu (sabe.cs.wisc.edu [128.105.6.20]) by cuda.sgi.com with ESMTP id LUTR6iLmDFYwKNLK (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 17:22:03 -0800 (PST) X-Barracuda-Envelope-From: michaelc@cs.wisc.edu X-Barracuda-Apparent-Source-IP: 128.105.6.20 Received: from [20.15.0.202] (c-24-245-27-162.hsd1.mn.comcast.net [24.245.27.162]) (authenticated bits=0) by sabe.cs.wisc.edu (8.14.7/8.14.1) with ESMTP id u0A1LHi3008377 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sat, 9 Jan 2016 19:21:18 -0600 Message-ID: <5691B208.9060001@cs.wisc.edu> Date: Sat, 09 Jan 2016 19:21:12 -0600 From: Mike Christie User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: device-mapper development , mchristi@redhat.com CC: linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, xfs@oss.sgi.com, linux-scsi@vger.kernel.org, konrad.wilk@oracle.com, linux-kernel@vger.kernel.org, philipp.reisner@linbit.com, linux-f2fs-devel@lists.sourceforge.net, linux-raid@vger.kernel.org, ocfs2-devel@oss.oracle.com, target-devel@vger.kernel.org, linux-mtd@lists.infradead.org, osd-dev@open-osd.org, linux-fsdevel@vger.kernel.org, lars.ellenberg@linbit.com, linux-ext4@vger.kernel.org, linux-btrfs@vger.kernel.org, drbd-dev@lists.linbit.com Subject: Re: [dm-devel] [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-3-git-send-email-mchristi@redhat.com> <20160109133716.GB26660@infradead.org> <5691AC2E.2050403@cs.wisc.edu> X-ASG-Orig-Subj: Re: [dm-devel] [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields In-Reply-To: <5691AC2E.2050403@cs.wisc.edu> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sabe.cs.wisc.edu[128.105.6.20] X-Barracuda-Start-Time: 1452388923 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25993 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On 01/09/2016 06:56 PM, Mike Christie wrote: > On 01/09/2016 07:37 AM, Christoph Hellwig wrote: >> Seems like this is missing REQ_OP_FLUSH, which still hides as a write? >> Oh yeah, to answer the second part of your question, REQ_OP_FLUSH is only a flush operation like what request_fn drivers wanted. REQ_PREFLUSH can be set with a REQ_OP_WRITE bio when filesystems want to do both. There is then the case where filesystems and blkdev_issue_flush could just want to request a flush. I left them as a REQ_PREFLUSH with REQ_OP_WRITE set, so there would be a single code path. > > I might have misunderstood what you wanted or am misunderstanding you now. > > I did not change the flush related code until the last patches. I added > REQ_OP_FLUSH in patch: > > [PATCH 33/35] block, drivers: add REQ_OP_FLUSH operation > commit 18dea20363ba245a47bc1bb54f6465b8a05b19af > Author: Mike Christie > Date: Tue Jan 5 21:02:17 2016 -0600 > > block, drivers: add REQ_OP_FLUSH operation > > > and I added the REQ_PREFLUSH flag in > > > [PATCH 35/35] block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH > commit 303dc6a7cc3673065538ba041562fcd833a619af > Author: Mike Christie > Date: Thu Jan 7 18:17:03 2016 -0600 > > block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH > > > I did them in separate patches, because I was not sure if they were > correct with what you requested before, and because they were a little > more tricky because of how request_fn drivers, make_request_fn drivers, > and dm-multipath handle flushes differently. > -- > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > From BATV+aab932a23b7e96aeb608+4522+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 10 01:15:11 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 7C0367F52 for ; Sun, 10 Jan 2016 01:15:11 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 0E6A4AC001 for ; Sat, 9 Jan 2016 23:15:07 -0800 (PST) X-ASG-Debug-ID: 1452410105-04cb6c75db42aa0001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id rjF9zqBqLqv5oysY (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 23:15:05 -0800 (PST) X-Barracuda-Envelope-From: BATV+aab932a23b7e96aeb608+4522+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aIADJ-0003RD-1E; Sun, 10 Jan 2016 07:14:57 +0000 Date: Sat, 9 Jan 2016 23:14:57 -0800 From: Christoph Hellwig To: Mike Christie Cc: device-mapper development , mchristi@redhat.com, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, xfs@oss.sgi.com, linux-scsi@vger.kernel.org, konrad.wilk@oracle.com, linux-kernel@vger.kernel.org, philipp.reisner@linbit.com, linux-f2fs-devel@lists.sourceforge.net, linux-raid@vger.kernel.org, ocfs2-devel@oss.oracle.com, target-devel@vger.kernel.org, linux-mtd@lists.infradead.org, osd-dev@open-osd.org, linux-fsdevel@vger.kernel.org, lars.ellenberg@linbit.com, linux-ext4@vger.kernel.org, linux-btrfs@vger.kernel.org, drbd-dev@lists.linbit.com Subject: Re: [dm-devel] [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields Message-ID: <20160110071457.GA12831@infradead.org> X-ASG-Orig-Subj: Re: [dm-devel] [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-3-git-send-email-mchristi@redhat.com> <20160109133716.GB26660@infradead.org> <5691AC2E.2050403@cs.wisc.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5691AC2E.2050403@cs.wisc.edu> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452410105 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25998 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Sat, Jan 09, 2016 at 06:56:14PM -0600, Mike Christie wrote: > I did not change the flush related code until the last patches. I added > REQ_OP_FLUSH in patch: Oh, I missed that. From BATV+aab932a23b7e96aeb608+4522+infradead.org+hch@bombadil.srs.infradead.org Sun Jan 10 01:16:02 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 824537F52 for ; Sun, 10 Jan 2016 01:16:02 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 1CF37AC001 for ; Sat, 9 Jan 2016 23:16:02 -0800 (PST) X-ASG-Debug-ID: 1452410160-04bdf06f6e42a20001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id CVNY8zrOhZMtWdlI (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 09 Jan 2016 23:16:00 -0800 (PST) X-Barracuda-Envelope-From: BATV+aab932a23b7e96aeb608+4522+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aIAEI-0004jY-7Q; Sun, 10 Jan 2016 07:15:58 +0000 Date: Sat, 9 Jan 2016 23:15:58 -0800 From: Christoph Hellwig To: Mike Christie Cc: device-mapper development , mchristi@redhat.com, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, xfs@oss.sgi.com, linux-scsi@vger.kernel.org, konrad.wilk@oracle.com, linux-kernel@vger.kernel.org, philipp.reisner@linbit.com, linux-f2fs-devel@lists.sourceforge.net, linux-raid@vger.kernel.org, ocfs2-devel@oss.oracle.com, target-devel@vger.kernel.org, linux-mtd@lists.infradead.org, osd-dev@open-osd.org, linux-fsdevel@vger.kernel.org, lars.ellenberg@linbit.com, linux-ext4@vger.kernel.org, linux-btrfs@vger.kernel.org, drbd-dev@lists.linbit.com Subject: Re: [dm-devel] [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields Message-ID: <20160110071558.GB12831@infradead.org> X-ASG-Orig-Subj: Re: [dm-devel] [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields References: <1452027218-32303-1-git-send-email-mchristi@redhat.com> <1452027218-32303-3-git-send-email-mchristi@redhat.com> <20160109133716.GB26660@infradead.org> <5691AC2E.2050403@cs.wisc.edu> <5691B208.9060001@cs.wisc.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5691B208.9060001@cs.wisc.edu> User-Agent: Mutt/1.5.23 (2014-03-12) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452410160 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25998 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS On Sat, Jan 09, 2016 at 07:21:12PM -0600, Mike Christie wrote: > Oh yeah, to answer the second part of your question, REQ_OP_FLUSH is > only a flush operation like what request_fn drivers wanted. And that's the odd part that trips me up. > > REQ_PREFLUSH can be set with a REQ_OP_WRITE bio when filesystems want to > do both. > > There is then the case where filesystems and blkdev_issue_flush could > just want to request a flush. I left them as a REQ_PREFLUSH with > REQ_OP_WRITE set, so there would be a single code path. But the pure flush without data transfer case is pretty different, so it seems rather odd to handle it like that. But I suspec we could just fix that up later. From hch@lst.de Sun Jan 10 01:54:14 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 7EE507F56 for ; Sun, 10 Jan 2016 01:54:14 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 111FCAC002 for ; Sat, 9 Jan 2016 23:54:13 -0800 (PST) X-ASG-Debug-ID: 1452412449-04cbb05f7e44690001-NocioJ Received: from newverein.lst.de (verein.lst.de [213.95.11.211]) by cuda.sgi.com with ESMTP id bzyE9nFqivVQvYbc (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 09 Jan 2016 23:54:10 -0800 (PST) X-Barracuda-Envelope-From: hch@lst.de X-Barracuda-Apparent-Source-IP: 213.95.11.211 Received: by newverein.lst.de (Postfix, from userid 2407) id 24BFD6913C; Sun, 10 Jan 2016 08:54:09 +0100 (CET) Date: Sun, 10 Jan 2016 08:54:09 +0100 From: Christoph Hellwig To: "Darrick J. Wong" Cc: Christoph Hellwig , xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160110075409.GA8213@lst.de> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> <20160105014310.GK28330@birch.djwong.org> <20160105104214.GA16310@infradead.org> <20160107003227.GB8015@birch.djwong.org> <20160107152541.GA16982@lst.de> <20160108100933.GN28330@birch.djwong.org> <20160108134704.GA6708@lst.de> <20160109211713.GB6112@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160109211713.GB6112@birch.djwong.org> User-Agent: Mutt/1.5.17 (2007-11-01) X-Barracuda-Connect: verein.lst.de[213.95.11.211] X-Barracuda-Start-Time: 1452412450 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.25999 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Sat, Jan 09, 2016 at 01:17:13PM -0800, Darrick J. Wong wrote: > > > > I've not seen the inode eviction asserts anymore, but I now hit a > > corruption warnings in generic/168 reliably. I did hit before as > > well, but not very reliably. > > I'll see if I can repro the 168 error; it's been running in a loop all > night and hasn't bombed yet. Note that this is over nfs to a local server, not running on xfs directly, which is doing fine. > In the meantime, I added some more tests and fixed a CoW corruption when an > xfs_io_overwrite extent has cow reservations in the middle of the extent. > > I also restarted testing on arm64, ppc64{,el}, and i686; it seems > stable enough right now to pass all ~130 reflink xfstests here. I see pretty reliable failures in xfs/128 xfs/132 xfs/139, apparenly due to content mismatches. Re the verifier failure: sees like we're hitting the if (level >= pag->pagf_refcount_level) return false; case. Together with the other garbage in it seems like we're seeing a btree block that's not properly initialized in some way, maybe after a split. From jose-xfs=oss.sgi.com@freemly.com Sun Jan 10 13:58:28 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 2F2E57F37 for ; Sun, 10 Jan 2016 13:58:28 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id B5B4CAC001 for ; Sun, 10 Jan 2016 11:58:24 -0800 (PST) X-ASG-Debug-ID: 1452455898-04cbb05f7d57820002-NocioJ Received: from display.freemly.com (display.freemly.com [162.252.172.151]) by cuda.sgi.com with ESMTP id MCIrvpbd7y7RahBf for ; Sun, 10 Jan 2016 11:58:21 -0800 (PST) X-Barracuda-Envelope-From: jose-xfs=oss.sgi.com@freemly.com X-Barracuda-Apparent-Source-IP: 162.252.172.151 Received: by display.freemly.com id hiarts0001g2 for ; Sun, 10 Jan 2016 14:49:36 -0500 (envelope-from ) MIME-Version: 1.0 From: "Jose R. Johnston" To: xfs@oss.sgi.com Subject: STOP! Reading this might save your life Date: Sun, 10 Jan 2016 14:49:36 -0500 X-ASG-Orig-Subj: STOP! Reading this might save your life Content-Type: text/plain; List-unsubscribe: http://freemly.com/P60fy6Z1vMciJuoyfWMA/unsubscribe/ Message-ID: <0.0.0.39.1D14BE008C9B146.B5AA06@display.freemly.com> X-Barracuda-Connect: display.freemly.com[162.252.172.151] X-Barracuda-Start-Time: 1452455900 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 1.50 X-Barracuda-Spam-Status: No, SCORE=1.50 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MV0702 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26011 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 1.50 BSF_SC0_MV0702 Custom rule MV0702 It's true. Heart disease is the #1 killer. . Do you know how to protect yourself? . Do you know what the warning signs are? . Everyone knows that they can be DEADLY but did you know that they are also one of the most PAINFUL things a human can experience? . Arm yourself with knowledge and then protect yourself with . This may be the most important thing you ever do. http://freemly.com/FFZFkF8Ab4LZdOcxmyQQ . . . . . Unsubscribe http://freemly.com/P60fy6Z1vMciJuoyfWMA/unsubscribe/ .1732 John Daniel Drive, Clark, MO 65243 From darrick.wong@oracle.com Sun Jan 10 16:49:02 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id F16287F37 for ; Sun, 10 Jan 2016 16:49:01 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 69A83AC002 for ; Sun, 10 Jan 2016 14:49:01 -0800 (PST) X-ASG-Debug-ID: 1452466139-04cbb05f7d5acd0001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id 0NgdWjahKl3Kq4AI (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sun, 10 Jan 2016 14:48:59 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0AMmsX2016195 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sun, 10 Jan 2016 22:48:55 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0AMmsK1010281 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Sun, 10 Jan 2016 22:48:54 GMT Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22]) by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0AMmsA7011234; Sun, 10 Jan 2016 22:48:54 GMT Received: from localhost (/24.130.224.127) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 10 Jan 2016 14:48:54 -0800 Date: Sun, 10 Jan 2016 14:48:53 -0800 From: "Darrick J. Wong" To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend Message-ID: <20160110224853.GA6096@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: cancel COW in xfs_cancel_ioend References: <1451822873-12969-1-git-send-email-hch@lst.de> <1451822873-12969-4-git-send-email-hch@lst.de> <20160105014310.GK28330@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160105014310.GK28330@birch.djwong.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: userv0022.oracle.com [156.151.31.74] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1452466139 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26014 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines On Mon, Jan 04, 2016 at 05:43:10PM -0800, Darrick J. Wong wrote: > On Sun, Jan 03, 2016 at 01:07:53PM +0100, Christoph Hellwig wrote: > > Otherwise we leak COW allocations done earlier in writepage. This > > can be reproduced fairly easily when we hit the non-blocking writeback > > EAGAIN case. > > > > Signed-off-by: Christoph Hellwig > > --- > > fs/xfs/xfs_aops.c | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c > > index 185415a..9c69dc3 100644 > > --- a/fs/xfs/xfs_aops.c > > +++ b/fs/xfs/xfs_aops.c > > @@ -588,6 +588,7 @@ xfs_cancel_ioend( > > { > > xfs_ioend_t *next; > > struct buffer_head *bh, *next_bh; > > + int error; > > > > do { > > next = ioend->io_list; > > @@ -605,6 +606,12 @@ xfs_cancel_ioend( > > unlock_buffer(bh); > > } while ((bh = next_bh) != NULL); > > > > + if (ioend->io_flags & XFS_IOEND_COW) { > > + error = xfs_reflink_end_cow_failed( > > + XFS_I(ioend->io_inode), > > + ioend->io_offset, ioend->io_size); > > + WARN_ON_ONCE(error); > > + } Actually, no, this isn't correct. Even if we cancel the ioend, we must retain the CoW reservation because the pages remain dirty and writepage will try again. If we delete the reservation, that second writepage will treat the dirty page as a regular overwrite because there's no reservation, which is wrong. We need to keep something in the CoW fork; either we can leave the allocated blocks or we could theoretically convert it back to a delalloc reservation. For now I'll leave the mapping untouched since I've subsequently taught xfs to clear out the CoW mappings when we truncate/punch/etc. The reservation won't hang around for long unless IO errors start piling up. This causes file corruption in xfs/140 when blocksize < pagesize. --D > > Hmm. This might be the cause of the occasional complaints I've been seeing > where allocated blocks remain in the COW fork when the inode is being cleared > out. That said, the xfs_reflink_end_cow_failed() is apparently missing a > xfs_bunmapi_cow() to actually clean out the COW fork. > > Good catch, in any case. Thank you for the testing and patches! :) > > --D > > > mempool_free(ioend, xfs_ioend_pool); > > } while ((ioend = next) != NULL); > > } > > -- > > 1.9.1 > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com > > http://oss.sgi.com/mailman/listinfo/xfs > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From dave@fromorbit.com Sun Jan 10 21:26:05 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 59ACC7F37 for ; Sun, 10 Jan 2016 21:26:05 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 1FC1A8F8033 for ; Sun, 10 Jan 2016 19:26:04 -0800 (PST) X-ASG-Debug-ID: 1452482756-04bdf06f6e5f740002-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id oGPfidFgwgWHfpPd for ; Sun, 10 Jan 2016 19:26:02 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ChBwAdIJNWPGu7LXleKAECgw+BP4ZggXmgJgaRG4stTQEBAQEBAQcBAQEBQT+ENQEFJyAPFh0IGDE5AwcUGYgttBGLA4YOjwMFh2GHEoggjzeHbYUyjlGCLgELOxyBcSo0hgYBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 11 Jan 2016 13:55:19 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIT6H-0001Zu-Fh for xfs@oss.sgi.com; Mon, 11 Jan 2016 14:24:57 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIT6H-00079Q-Eq for xfs@oss.sgi.com; Mon, 11 Jan 2016 14:24:57 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 1/2] xfs: inode recovery readahead can race with inode buffer creation Date: Mon, 11 Jan 2016 14:24:53 +1100 X-ASG-Orig-Subj: [PATCH 1/2] xfs: inode recovery readahead can race with inode buffer creation Message-Id: <1452482694-27314-2-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452482694-27314-1-git-send-email-david@fromorbit.com> References: <1452482694-27314-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452482761 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26020 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner When we do inode readahead in log recovery, we do can do the readahead before we've replayed the icreate transaction that stamps the buffer with inode cores. The inode readahead verifier catches this and marks the buffer as !done to indicate that it doesn't yet contain valid inodes. In adding buffer error notification (i.e. setting b_error = -EIO at the same time as as we clear the done flag) to such a readahead verifier failure, we can then get subsequent inode recovery failing with this error: XFS (dm-0): metadata I/O error: block 0xa00060 ("xlog_recover_do..(read#2)") error 5 numblks 32 This occurs when readahead completion races with icreate item replay such as: inode readahead find buffer lock buffer submit RA io .... icreate recovery xfs_trans_get_buffer find buffer lock buffer ..... fails verifier clear XBF_DONE set bp->b_error = -EIO release and unlock buffer icreate initialises buffer marks buffer as done adds buffer to delayed write queue releases buffer At this point, we have an initialised inode buffer that is up to date but has an -EIO state registered against it. When we finally get to recovering an inode in that buffer: inode item recovery xfs_trans_read_buffer find buffer lock buffer sees XBF_DONE is set, returns buffer sees bp->b_error is set fail log recovery! Essentially, we need xfs_trans_get_buf_map() to clear the error status of the buffer when doing a lookup. This function returns uninitialised buffers, so the buffer returned can not be in an error state and none of the code that uses this function expects b_error to be set on return. Indeed, there is an ASSERT(!bp->b_error); in the transaction case in xfs_trans_get_buf_map() that would have caught this if log recovery used transactions.... This patch firstly changes the inode readahead failure to set -EIO on the buffer, and secondly changes xfs_buf_get_map() to never return a buffer with an error state set so this first change doesn't cause unexpected log recovery failures. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_inode_buf.c | 12 +++++++----- fs/xfs/xfs_buf.c | 7 +++++++ 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index 1b8d98a..ff17c48 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -62,11 +62,12 @@ xfs_inobp_check( * has not had the inode cores stamped into it. Hence for readahead, the buffer * may be potentially invalid. * - * If the readahead buffer is invalid, we don't want to mark it with an error, - * but we do want to clear the DONE status of the buffer so that a followup read - * will re-read it from disk. This will ensure that we don't get an unnecessary - * warnings during log recovery and we don't get unnecssary panics on debug - * kernels. + * If the readahead buffer is invalid, we need to mark it with an error and + * clear the DONE status of the buffer so that a followup read will re-read it + * from disk. We don't report the error otherwise to avoid warnings during log + * recovery and we don't get unnecssary panics on debug kernels. We use EIO here + * because all we want to do is say readahead failed; there is no-one to report + * the error to, so this will distinguish it from a non-ra verifier failure. */ static void xfs_inode_buf_verify( @@ -93,6 +94,7 @@ xfs_inode_buf_verify( XFS_RANDOM_ITOBP_INOTOBP))) { if (readahead) { bp->b_flags &= ~XBF_DONE; + xfs_buf_ioerror(bp, -EIO); return; } diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 45a8ea7..ae86b16 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -604,6 +604,13 @@ found: } } + /* + * Clear b_error if this is a lookup from a caller that doesn't expect + * valid data to be found in the buffer. + */ + if (!(flags & XBF_READ)) + xfs_buf_ioerror(bp, 0); + XFS_STATS_INC(target->bt_mount, xb_get); trace_xfs_buf_get(bp, flags, _RET_IP_); return bp; -- 2.5.0 From dave@fromorbit.com Sun Jan 10 21:26:06 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 35BFD7F3F for ; Sun, 10 Jan 2016 21:26:06 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0B18A304039 for ; Sun, 10 Jan 2016 19:26:02 -0800 (PST) X-ASG-Debug-ID: 1452482756-04bdf06f6e5f740001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id wfImvGEvLydWBwwY for ; Sun, 10 Jan 2016 19:25:56 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CeBwAdIJNWPGu7LXleKAECgw+IH4F5oCYGkRuLLU0BAQEBAQEHAQEBAUE/hRE7gQIDB4hanwigDIYOimqEGQWTEIQDnFaOUYIuAQsBgkcqhjoBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 11 Jan 2016 13:55:19 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIT6H-0001Zt-F9 for xfs@oss.sgi.com; Mon, 11 Jan 2016 14:24:57 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIT6H-00079N-E8 for xfs@oss.sgi.com; Mon, 11 Jan 2016 14:24:57 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 0/2 v3] xfs: handle dquot buffer readahead in log recovery correctly Date: Mon, 11 Jan 2016 14:24:52 +1100 X-ASG-Orig-Subj: [PATCH 0/2 v3] xfs: handle dquot buffer readahead in log recovery correctly Message-Id: <1452482694-27314-1-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452482756 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26020 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Hi folks, Version 3 of this patchset. Version 2 of the patchset added a fix for the inode readahead error setting in log recovery, which turned out to be problematic. I've split that change out into it's own patch which includes the fix it requires to prevent a race condition in log recovery to do with inode buffer creation recovery. This is a generic fix to xfs_buf_get_map(), in that if we are returning an initialised buffer for the caller to use, it shouldn't have an error set on it from the previous operation. If we don't clear the error before returning the buffer, it causes unexpected failures further down the line. This caused log recovery failures in generic/073 on slow disks (i.e. needed sufficient readahead IO latency to open the race window) and a couple of other tests as well. The second patch is essentially the original patch with just the inode buffer changes removed. There are no other changes to that patch. Cheers, Dave. From dave@fromorbit.com Sun Jan 10 21:26:09 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 19D1B7F51 for ; Sun, 10 Jan 2016 21:26:09 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id A18F0AC001 for ; Sun, 10 Jan 2016 19:26:05 -0800 (PST) X-ASG-Debug-ID: 1452482762-04cb6c75db5e050001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id pvDzR3yTC7Vd95ck for ; Sun, 10 Jan 2016 19:26:02 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ChBwAdIJNWPGu7LXleKAECgw+BP4ZggXmgJgaRG4stTQEBAQEBAQcBAQEBQT+ENQEFJy8zCBgxOQMHFBmILb8Uhg6PAwWHWgeFXoE0iCCPN4dthTKFZIhtgi4BCzscgXEqNIQ7BIFHAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 11 Jan 2016 13:55:19 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIT6H-0001Zv-Gc for xfs@oss.sgi.com; Mon, 11 Jan 2016 14:24:57 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIT6H-00079V-FH for xfs@oss.sgi.com; Mon, 11 Jan 2016 14:24:57 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 2/2] xfs: handle dquot buffer readahead in log recovery correctly Date: Mon, 11 Jan 2016 14:24:54 +1100 X-ASG-Orig-Subj: [PATCH 2/2] xfs: handle dquot buffer readahead in log recovery correctly Message-Id: <1452482694-27314-3-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452482694-27314-1-git-send-email-david@fromorbit.com> References: <1452482694-27314-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452482762 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26020 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner When we do dquot readahead in log recovery, we do not use a verifier as the underlying buffer may not have dquots in it. e.g. the allocation operation hasn't yet been replayed. Hence we do not want to fail recovery because we detect an operation to be replayed has not been run yet. This problem was addressed for inodes in commit d891400 ("xfs: inode buffers may not be valid during recovery readahead") but the problem was not recognised to exist for dquots and their buffers as the dquot readahead did not have a verifier. The result of not using a verifier is that when the buffer is then next read to replay a dquot modification, the dquot buffer verifier will only be attached to the buffer if *readahead is not complete*. Hence we can read the buffer, replay the dquot changes and then add it to the delwri submission list without it having a verifier attached to it. This then generates warnings in xfs_buf_ioapply(), which catches and warns about this case. Fix this and make it handle the same readahead verifier error cases as for inode buffers by adding a new readahead verifier that has a write operation as well as a read operation that marks the buffer as not done if any corruption is detected. Also make sure we don't run readahead if the dquot buffer has been marked as cancelled by recovery. This will result in readahead either succeeding and the buffer having a valid write verifier, or readahead failing and the buffer state requiring the subsequent read to resubmit the IO with the new verifier. In either case, this will result in the buffer always ending up with a valid write verifier on it. Note: we also need to fix the inode buffer readahead error handling to mark the buffer with EIO. Brian noticed the code I copied from there wrong during review, so fix it at the same time. Add comments linking the two functions that handle readahead verifier errors together so we don't forget this behavioural link in future. cc: # 3.12 - current Signed-off-by: Dave Chinner Reviewed-by: Brian Foster Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_dquot_buf.c | 36 ++++++++++++++++++++++++++++++------ fs/xfs/libxfs/xfs_inode_buf.c | 2 ++ fs/xfs/libxfs/xfs_quota_defs.h | 2 +- fs/xfs/libxfs/xfs_shared.h | 1 + fs/xfs/xfs_log_recover.c | 9 +++++++-- 5 files changed, 41 insertions(+), 9 deletions(-) diff --git a/fs/xfs/libxfs/xfs_dquot_buf.c b/fs/xfs/libxfs/xfs_dquot_buf.c index 11cefb2..3cc3cf7 100644 --- a/fs/xfs/libxfs/xfs_dquot_buf.c +++ b/fs/xfs/libxfs/xfs_dquot_buf.c @@ -54,7 +54,7 @@ xfs_dqcheck( xfs_dqid_t id, uint type, /* used only when IO_dorepair is true */ uint flags, - char *str) + const char *str) { xfs_dqblk_t *d = (xfs_dqblk_t *)ddq; int errs = 0; @@ -207,7 +207,8 @@ xfs_dquot_buf_verify_crc( STATIC bool xfs_dquot_buf_verify( struct xfs_mount *mp, - struct xfs_buf *bp) + struct xfs_buf *bp, + int warn) { struct xfs_dqblk *d = (struct xfs_dqblk *)bp->b_addr; xfs_dqid_t id = 0; @@ -240,8 +241,7 @@ xfs_dquot_buf_verify( if (i == 0) id = be32_to_cpu(ddq->d_id); - error = xfs_dqcheck(mp, ddq, id + i, 0, XFS_QMOPT_DOWARN, - "xfs_dquot_buf_verify"); + error = xfs_dqcheck(mp, ddq, id + i, 0, warn, __func__); if (error) return false; } @@ -256,7 +256,7 @@ xfs_dquot_buf_read_verify( if (!xfs_dquot_buf_verify_crc(mp, bp)) xfs_buf_ioerror(bp, -EFSBADCRC); - else if (!xfs_dquot_buf_verify(mp, bp)) + else if (!xfs_dquot_buf_verify(mp, bp, XFS_QMOPT_DOWARN)) xfs_buf_ioerror(bp, -EFSCORRUPTED); if (bp->b_error) @@ -264,6 +264,25 @@ xfs_dquot_buf_read_verify( } /* + * readahead errors are silent and simply leave the buffer as !done so a real + * read will then be run with the xfs_dquot_buf_ops verifier. See + * xfs_inode_buf_verify() for why we use EIO and ~XBF_DONE here rather than + * reporting the failure. + */ +static void +xfs_dquot_buf_readahead_verify( + struct xfs_buf *bp) +{ + struct xfs_mount *mp = bp->b_target->bt_mount; + + if (!xfs_dquot_buf_verify_crc(mp, bp) || + !xfs_dquot_buf_verify(mp, bp, 0)) { + xfs_buf_ioerror(bp, -EIO); + bp->b_flags &= ~XBF_DONE; + } +} + +/* * we don't calculate the CRC here as that is done when the dquot is flushed to * the buffer after the update is done. This ensures that the dquot in the * buffer always has an up-to-date CRC value. @@ -274,7 +293,7 @@ xfs_dquot_buf_write_verify( { struct xfs_mount *mp = bp->b_target->bt_mount; - if (!xfs_dquot_buf_verify(mp, bp)) { + if (!xfs_dquot_buf_verify(mp, bp, XFS_QMOPT_DOWARN)) { xfs_buf_ioerror(bp, -EFSCORRUPTED); xfs_verifier_error(bp); return; @@ -287,3 +306,8 @@ const struct xfs_buf_ops xfs_dquot_buf_ops = { .verify_write = xfs_dquot_buf_write_verify, }; +const struct xfs_buf_ops xfs_dquot_buf_ra_ops = { + .name = "xfs_dquot_ra", + .verify_read = xfs_dquot_buf_readahead_verify, + .verify_write = xfs_dquot_buf_write_verify, +}; diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index ff17c48..1aabfda 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -68,6 +68,8 @@ xfs_inobp_check( * recovery and we don't get unnecssary panics on debug kernels. We use EIO here * because all we want to do is say readahead failed; there is no-one to report * the error to, so this will distinguish it from a non-ra verifier failure. + * Changes to this readahead error behavour also need to be reflected in + * xfs_dquot_buf_readahead_verify(). */ static void xfs_inode_buf_verify( diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h index 1b0a083..f51078f 100644 --- a/fs/xfs/libxfs/xfs_quota_defs.h +++ b/fs/xfs/libxfs/xfs_quota_defs.h @@ -153,7 +153,7 @@ typedef __uint16_t xfs_qwarncnt_t; #define XFS_QMOPT_RESBLK_MASK (XFS_QMOPT_RES_REGBLKS | XFS_QMOPT_RES_RTBLKS) extern int xfs_dqcheck(struct xfs_mount *mp, xfs_disk_dquot_t *ddq, - xfs_dqid_t id, uint type, uint flags, char *str); + xfs_dqid_t id, uint type, uint flags, const char *str); extern int xfs_calc_dquots_per_chunk(unsigned int nbblks); #endif /* __XFS_QUOTA_H__ */ diff --git a/fs/xfs/libxfs/xfs_shared.h b/fs/xfs/libxfs/xfs_shared.h index 5be5297..15c3ceb 100644 --- a/fs/xfs/libxfs/xfs_shared.h +++ b/fs/xfs/libxfs/xfs_shared.h @@ -49,6 +49,7 @@ extern const struct xfs_buf_ops xfs_inobt_buf_ops; extern const struct xfs_buf_ops xfs_inode_buf_ops; extern const struct xfs_buf_ops xfs_inode_buf_ra_ops; extern const struct xfs_buf_ops xfs_dquot_buf_ops; +extern const struct xfs_buf_ops xfs_dquot_buf_ra_ops; extern const struct xfs_buf_ops xfs_sb_buf_ops; extern const struct xfs_buf_ops xfs_sb_quiet_buf_ops; extern const struct xfs_buf_ops xfs_symlink_buf_ops; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index c5ecaac..5991cdc 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -3204,6 +3204,7 @@ xlog_recover_dquot_ra_pass2( struct xfs_disk_dquot *recddq; struct xfs_dq_logformat *dq_f; uint type; + int len; if (mp->m_qflags == 0) @@ -3224,8 +3225,12 @@ xlog_recover_dquot_ra_pass2( ASSERT(dq_f); ASSERT(dq_f->qlf_len == 1); - xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, - XFS_FSB_TO_BB(mp, dq_f->qlf_len), NULL); + len = XFS_FSB_TO_BB(mp, dq_f->qlf_len); + if (xlog_peek_buffer_cancelled(log, dq_f->qlf_blkno, len, 0)) + return; + + xfs_buf_readahead(mp->m_ddev_targp, dq_f->qlf_blkno, len, + &xfs_dquot_buf_ra_ops); } STATIC void -- 2.5.0 From david@fromorbit.com Sun Jan 10 22:04:44 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6FA137F3F for ; Sun, 10 Jan 2016 22:04:44 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 339C18F804C for ; Sun, 10 Jan 2016 20:04:41 -0800 (PST) X-ASG-Debug-ID: 1452485077-04bdf06f6b605f0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id VgIj5x6GzOrZUpRK for ; Sun, 10 Jan 2016 20:04:37 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DxCgCUKJNWPGu7LXleKAECgw9SbYJihXegFBIGi14khRmEByKCPYMqAgIBAQKBF00BAQEBAQEHAQEBAUE/QRIBg2EBAQQ6HCMQCAMOCgklDwUlAwcaE4gtD75bAQEIAgEgGYV1hUeJPAWHZIssg0FChUOIDYFnSoxVhWSIbYUCKjQBAQGDeoIJAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 11 Jan 2016 14:34:37 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aITid-0001e5-Vw; Mon, 11 Jan 2016 15:04:36 +1100 Date: Mon, 11 Jan 2016 15:04:35 +1100 From: Dave Chinner To: Brian Foster Cc: fstests@vger.kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH v2] xfs: test XFS torn log write detection Message-ID: <20160111040435.GH10456@dastard> X-ASG-Orig-Subj: Re: [PATCH v2] xfs: test XFS torn log write detection References: <1451923421-16647-1-git-send-email-bfoster@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1451923421-16647-1-git-send-email-bfoster@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452485077 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26020 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon, Jan 04, 2016 at 11:03:41AM -0500, Brian Foster wrote: > XFS torn log write detection includes a mechanism to inject CRC errors > into log records at runtime and shutdown the fs accordingly. This > ensures that the CRC verification pass on the subsequent mount discovers > an invalid record near the head of the log and considers it a torn > write. > > This test runs a workload with error injection enabled and verifies that > the subsequent mount is successful. The test repeats for several > iterations using a random frequency factor for the error event each > time. > > Signed-off-by: Brian Foster > --- > > Here's a v2 of the XFS log crc verification test case now that the > associated kernel code has been merged. The only change is the bugfix > noted in the changelog below. > > Brian > > v2: > - Update random failure frequency calculation to ensure factor is never > 0. > v1: http://article.gmane.org/gmane.comp.file-systems.fstests/1406 > > tests/xfs/350 | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ > tests/xfs/350.out | 2 ++ > tests/xfs/group | 1 + > 3 files changed, 88 insertions(+) > create mode 100755 tests/xfs/350 > create mode 100644 tests/xfs/350.out > > diff --git a/tests/xfs/350 b/tests/xfs/350 > new file mode 100755 > index 0000000..1a1de6f > --- /dev/null > +++ b/tests/xfs/350 > @@ -0,0 +1,85 @@ > +#! /bin/bash > +# FS QA Test No. 350 > +# > +# Use the XFS log record CRC error injection mechanism to test torn writes to > +# the log. The error injection mechanism writes an invalid CRC and shuts down > +# the filesystem. The test verifies that a subsequent remount recovers the log > +# and that the filesystem is consistent. > +# > +# Note that this test requires a DEBUG mode kernel. > +# > +#----------------------------------------------------------------------- > +# Copyright (c) 2015 Red Hat, Inc. All Rights Reserved. > +# > +# This program is free software; you can redistribute it and/or > +# modify it under the terms of the GNU General Public License as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it would be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write the Free Software Foundation, > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > +#----------------------------------------------------------------------- > +# > + > +seq=`basename $0` > +seqres=$RESULT_DIR/$seq > +echo "QA output created by $seq" > + > +here=`pwd` > +tmp=/tmp/$$ > +status=1 # failure is the default! > +trap "_cleanup; exit \$status" 0 1 2 3 15 > + > +_cleanup() > +{ > + cd / > + rm -f $tmp.* > + killall -9 fsstress > /dev/null 2>&1 > + wait > /dev/null 2>&1 > +} > + > +rm -f $seqres.full > + > +# get standard environment, filters and checks > +. ./common/rc > + > +# real QA test starts here > + > +# Modify as appropriate. > +_supported_fs xfs > +_supported_os Linux > +_require_xfs_sysfs $(_short_dev $SCRATCH_DEV)/log/log_badcrc_factor Scratch is not mounted at this point, so I don't think you can test the scratch device sysfs attrs here. I changed it to look at the test device instead and it works fine. Cheers, Dave. -- Dave Chinner david@fromorbit.com From hare@suse.de Mon Jan 11 01:15:42 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C752B7F3F for ; Mon, 11 Jan 2016 01:15:42 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9BF9B8F8040 for ; Sun, 10 Jan 2016 23:15:39 -0800 (PST) X-ASG-Debug-ID: 1452496535-04bdf06f6e64c20001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id G4lDIrWHli2XH2A4 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 10 Jan 2016 23:15:36 -0800 (PST) X-Barracuda-Envelope-From: hare@suse.de X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 61B3EAAB4; Mon, 11 Jan 2016 07:15:33 +0000 (UTC) Subject: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life To: Dan Williams , Al Viro X-ASG-Orig-Subj: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> <20160109075414.GA5008@ZenIV.linux.org.uk> Cc: XFS Developers , linux-block@vger.kernel.org, linux-nvdimm , Dave Chinner , Jens Axboe , Jan Kara , linux-fsdevel , Matthew Wilcox , Ross Zwisler From: Hannes Reinecke Message-ID: <56935694.1000408@suse.de> Date: Mon, 11 Jan 2016 08:15:32 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452496536 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26023 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On 01/09/2016 03:17 PM, Dan Williams wrote: > On Fri, Jan 8, 2016 at 11:54 PM, Al Viro wrote: >> On Mon, Jan 04, 2016 at 10:20:05AM -0800, Dan Williams wrote: > [..] >> Would you mind explaining what the hell is _the_ backing device >> of a filesystem? What does that translate into in case of e.g. btrfs >> spanning several disks? Or ext4 with journal on a different device, for >> that matter? >> >> If anything, I would argue that filesystem is out of place here - >> general situation is "IO on X may require IO on device Y and X needs to do >> something when Y goes away". Consider e.g. /dev/loop backed by a device >> that went away. Or by a file on fs that has run down the curtain and joined >> the bleedin choir invisible. With another fs partially hosted by that >> loopback device. Or by RAID0 containing said device. >> >> You are given Y and attempt to locate the affected X. _Then_ >> you assume that X is a filesystem and has "something to be done" independent >> from the role Y played for it, so you can pick that action from superblock >> method. >> >> IMO you are placing the burden in the wrong place. _Recepient_ >> knows what it depends upon and what should be done for each source of >> trouble. So make it recepient's responsibility to request notifications. >> At which point the superblock method goes away, along with the requirement >> to handle all sources of trouble the same way, etc. >> >> What's more, things like RAID5 (also interested in knowing when >> a component has been ripped out) might or might not decide to propagate >> the event further - after all, that's exactly the point of redundancy. >> >> I'd look into something along the lines of notifier chain per >> gendisk, with potential victims registering a callback when they decide >> that from now on such and such device might screw them over... > > Makes sense. I'll drop this series for now and come back after > re-working it use notifiers. Yes please. I need a similar thing for communicating device changes (resizing, topology changes), so I'd be very much interested in them. And while you're at it, maybe we can fold the block device event handling into that, too. Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg) From arodlink@yahoo.com Mon Jan 11 04:52:40 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: ** X-Spam-Status: No, score=2.4 required=5.0 tests=FORGED_YAHOO_RCVD, FREEMAIL_FROM autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 45BC47F3F for ; Mon, 11 Jan 2016 04:52:40 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id CFE62AC001 for ; Mon, 11 Jan 2016 02:52:36 -0800 (PST) X-ASG-Debug-ID: 1452509554-04cbb05f7d6ca90001-NocioJ Received: from nit.767295.net (nit.767295.net [65.181.125.169]) by cuda.sgi.com with ESMTP id 76wJExOA7Hpcw5HI for ; Mon, 11 Jan 2016 02:52:35 -0800 (PST) X-Barracuda-Envelope-From: arodlink@yahoo.com X-Barracuda-Apparent-Source-IP: 65.181.125.169 To: xfs@oss.sgi.com Subject: Our Custom Shaped USB drives with your logo Message-ID: <616154454b725e55f3898e69ca40a01d@madamenoire.com> X-ASG-Orig-Subj: Our Custom Shaped USB drives with your logo Date: Mon, 11 Jan 2016 11:49:41 +0100 From: "Heather Texana" Reply-To: krikees@aliyun.com MIME-Version: 1.0 X-Mailer-LID: 3 X-Mailer-RecptId: 4686092 X-Mailer-SID: 1041 X-Mailer-Sent-By: 1 Content-Type: text/plain; format=flowed; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Barracuda-Connect: nit.767295.net[65.181.125.169] X-Barracuda-Start-Time: 1452509554 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 1.41 X-Barracuda-Spam-Status: No, SCORE=1.41 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=FORGED_YAHOO_RCVD, FORGED_YAHOO_RCVD_2 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26027 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 FORGED_YAHOO_RCVD 'From' yahoo.com does not match 'Received' headers 1.41 FORGED_YAHOO_RCVD_2 'From' yahoo.com does not match 'Received' headers Hi, Not sure if you received my email from last week about our New Year's Promotion? I would like to speak with the person in charge of purchasing your logo / branded promotional products for your company? Our company manufactures USB Flash Drives, printed with your logo, the perfect marketing tool for your business or organization. We offer low minimum quantities, and all memory sizes. We are always running a promotion, just ask! We can pre-load your media files, images, presentations and files onto the drives for you. These are ideal handouts for trade shows, branding, corporate gifts and employees. Our USB Pricing is at an ALL TIME LOW - now is the time to get your USB drives. Let us send you a quote! We will EARN your business! *Please reply back and let me know what quantity and memory size you need for a Fast New Year's Quote!* Are you a school or nonprofit? Please let us know for even further discounts. Thanks! Heather Texana USB Drive Representative Promo / Branded Flash Drives Contact: keniurt@sina.com From bfoster@redhat.com Mon Jan 11 05:52:38 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 520037F37 for ; Mon, 11 Jan 2016 05:52:38 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id B1A69AC002 for ; Mon, 11 Jan 2016 03:52:37 -0800 (PST) X-ASG-Debug-ID: 1452513155-04cb6c75de6a310001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 7kgTIM8GtjIYXcRC (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 03:52:35 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 0BAAFAA7; Mon, 11 Jan 2016 11:52:35 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-81.bos.redhat.com [10.18.41.81]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BBqYwl014303; Mon, 11 Jan 2016 06:52:34 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id AE9C6125794; Mon, 11 Jan 2016 06:52:33 -0500 (EST) Date: Mon, 11 Jan 2016 06:52:33 -0500 From: Brian Foster To: Dave Chinner Cc: fstests@vger.kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH v2] xfs: test XFS torn log write detection Message-ID: <20160111115233.GA31956@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH v2] xfs: test XFS torn log write detection References: <1451923421-16647-1-git-send-email-bfoster@redhat.com> <20160111040435.GH10456@dastard> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160111040435.GH10456@dastard> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452513155 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Mon, Jan 11, 2016 at 03:04:35PM +1100, Dave Chinner wrote: > On Mon, Jan 04, 2016 at 11:03:41AM -0500, Brian Foster wrote: > > XFS torn log write detection includes a mechanism to inject CRC errors > > into log records at runtime and shutdown the fs accordingly. This > > ensures that the CRC verification pass on the subsequent mount discovers > > an invalid record near the head of the log and considers it a torn > > write. > > > > This test runs a workload with error injection enabled and verifies that > > the subsequent mount is successful. The test repeats for several > > iterations using a random frequency factor for the error event each > > time. > > > > Signed-off-by: Brian Foster > > --- > > > > Here's a v2 of the XFS log crc verification test case now that the > > associated kernel code has been merged. The only change is the bugfix > > noted in the changelog below. > > > > Brian > > > > v2: > > - Update random failure frequency calculation to ensure factor is never > > 0. > > v1: http://article.gmane.org/gmane.comp.file-systems.fstests/1406 > > > > tests/xfs/350 | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > tests/xfs/350.out | 2 ++ > > tests/xfs/group | 1 + > > 3 files changed, 88 insertions(+) > > create mode 100755 tests/xfs/350 > > create mode 100644 tests/xfs/350.out > > > > diff --git a/tests/xfs/350 b/tests/xfs/350 > > new file mode 100755 > > index 0000000..1a1de6f > > --- /dev/null > > +++ b/tests/xfs/350 > > @@ -0,0 +1,85 @@ > > +#! /bin/bash > > +# FS QA Test No. 350 > > +# > > +# Use the XFS log record CRC error injection mechanism to test torn writes to > > +# the log. The error injection mechanism writes an invalid CRC and shuts down > > +# the filesystem. The test verifies that a subsequent remount recovers the log > > +# and that the filesystem is consistent. > > +# > > +# Note that this test requires a DEBUG mode kernel. > > +# > > +#----------------------------------------------------------------------- > > +# Copyright (c) 2015 Red Hat, Inc. All Rights Reserved. > > +# > > +# This program is free software; you can redistribute it and/or > > +# modify it under the terms of the GNU General Public License as > > +# published by the Free Software Foundation. > > +# > > +# This program is distributed in the hope that it would be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program; if not, write the Free Software Foundation, > > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > > +#----------------------------------------------------------------------- > > +# > > + > > +seq=`basename $0` > > +seqres=$RESULT_DIR/$seq > > +echo "QA output created by $seq" > > + > > +here=`pwd` > > +tmp=/tmp/$$ > > +status=1 # failure is the default! > > +trap "_cleanup; exit \$status" 0 1 2 3 15 > > + > > +_cleanup() > > +{ > > + cd / > > + rm -f $tmp.* > > + killall -9 fsstress > /dev/null 2>&1 > > + wait > /dev/null 2>&1 > > +} > > + > > +rm -f $seqres.full > > + > > +# get standard environment, filters and checks > > +. ./common/rc > > + > > +# real QA test starts here > > + > > +# Modify as appropriate. > > +_supported_fs xfs > > +_supported_os Linux > > +_require_xfs_sysfs $(_short_dev $SCRATCH_DEV)/log/log_badcrc_factor > > Scratch is not mounted at this point, so I don't think you can test > the scratch device sysfs attrs here. I changed it to look at the > test device instead and it works fine. > Oops, thought I tested that. :/ Thanks. Brian > Cheers, > > Dave. > > -- > Dave Chinner > david@fromorbit.com > -- > To unsubscribe from this list: send the line "unsubscribe fstests" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html From jack@suse.cz Mon Jan 11 06:23:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id B8D9429DF5 for ; Mon, 11 Jan 2016 06:23:18 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id A7F2F30405F for ; Mon, 11 Jan 2016 04:23:15 -0800 (PST) X-ASG-Debug-ID: 1452514991-04cbb05f7d6f4f0001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id G2owmjKX58XC3p9g (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 11 Jan 2016 04:23:12 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 80B32AB12; Mon, 11 Jan 2016 12:23:08 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id 0072D823D8; Mon, 11 Jan 2016 13:23:20 +0100 (CET) Date: Mon, 11 Jan 2016 13:23:20 +0100 From: Jan Kara To: Ross Zwisler Cc: Jan Kara , linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs Message-ID: <20160111122320.GB6262@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v7 2/9] dax: fix conversion of holes to PMDs References: <1452103263-1592-1-git-send-email-ross.zwisler@linux.intel.com> <1452103263-1592-3-git-send-email-ross.zwisler@linux.intel.com> <20160107132206.GE8380@quack.suse.cz> <20160107221114.GA20802@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160107221114.GA20802@linux.intel.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452514991 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26029 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Thu 07-01-16 15:11:14, Ross Zwisler wrote: > On Thu, Jan 07, 2016 at 02:22:06PM +0100, Jan Kara wrote: > > On Wed 06-01-16 11:00:56, Ross Zwisler wrote: > > > When we get a DAX PMD fault for a write it is possible that there could be > > > some number of 4k zero pages already present for the same range that were > > > inserted to service reads from a hole. These 4k zero pages need to be > > > unmapped from the VMAs and removed from the struct address_space radix tree > > > before the real DAX PMD entry can be inserted. > > > > > > For PTE faults this same use case also exists and is handled by a > > > combination of unmap_mapping_range() to unmap the VMAs and > > > delete_from_page_cache() to remove the page from the address_space radix > > > tree. > > > > > > For PMD faults we do have a call to unmap_mapping_range() (protected by a > > > buffer_new() check), but nothing clears out the radix tree entry. The > > > buffer_new() check is also incorrect as the current ext4 and XFS filesystem > > > code will never return a buffer_head with BH_New set, even when allocating > > > new blocks over a hole. Instead the filesystem will zero the blocks > > > manually and return a buffer_head with only BH_Mapped set. > > > > > > Fix this situation by removing the buffer_new() check and adding a call to > > > truncate_inode_pages_range() to clear out the radix tree entries before we > > > insert the DAX PMD. > > > > Ho, hum, let me understand this. So we have a file, different processes are > > mapping it. One process maps is with normal page granularity and another > > process with huge page granularity. Thus when the first process read-faults > > a few normal pages and then the second process write-faults the huge page > > in the same range, we have a problem. Do I understand this correctly? > > Because otherwise I don't understand how a single page table can have both > > huge page and normal page in the same range... > > I don't think that it necessarily has to do with multiple threads. The bit to > notice here is we *always* use 4k zero pages to cover holes. So, a single > thread can hit this condition by doing some reads from a hole (insert 4k > pages), then doing a write. This write is the first time that we will try and > use real DAX storage to insert into the page tables, and we may end up getting > a PMD. This means that we need to clear out all the 4k pages that we inserted > while reading holes in this same range, now that we have a 2M segment > allocated by the filesystem and the entire range is no longer a hole. OK, I see. Thanks for explanation. > > And if this is indeed the problem then what prevents the unmapping and > > truncation in huge page fault to race with mapping the same range again > > using small pages? Sure now blocks are allocated so the mapping itself will > > be consistent but radix tree will have the same issues it had before this > > patch, won't it? > > Yep, this is a separate issue, but I think that we handle this case > successfully, though we may end up flushing the same address multiple times. > Once the filesystem has established a block mapping (assuming we avoid the > race described below where one thread is mapping in holes and the other sees a > block allocation), I think we are okay. It's true that one thread can map in > PMDs, and another thread could potentially map in PTEs that cover the same > range if they hare working with mmaps that are smaller than a PMD, but the > sectors inserted into the radix tree by each of those threads will be > individually correct - the only issue is that they may overlap. > > Say, for example you have the following: > > CPU1 - process 1 CPU2 - process 2 > mmap for sector 0, size 2M > insert PMD into radix tree for sector 0 > This radix tree covers sectors 0-4096 > mmap for sector 32, size 4k > insert PTE entry into radix > tree for sector 32 > > In this case a fsync of the fd by process 1 will end up flushing sector 32 > twice, which is correct but inefficient. I think we can make this more > efficient by adjusting the insertion code and dirtying code in > dax_radix_entry() to look for PMDs that cover this same range. Yes, this is what I ended up with as well. So we are in agreement here. > > ... thinking some more about this ... > > > > OK, there is some difference - we will only have DAX exceptional entries > > for the range covered by huge page and those we replace properly in > > dax_radix_entry() code. So things are indeed fine *except* that nothing > > seems so serialize dax_load() hole with PMD fault. The race like following > > seems possible: > > > > CPU1 - process 1 CPU2 - process 2 > > > > __dax_fault() - file f, index 1 > > get_block() -> returns hole > > __dax_pmd_fault() - file f, index 0 > > get_block() -> allocates blocks > > ... > > truncate_pagecache_range() > > dax_load_hole() > > > > Boom, we have hole page instantiated for allocated range (data corruption) > > and corruption of radix tree entries as well. Actually this problem is > > there even for two different processes doing normal page faults (one read, > > one write) against the same page in the file. > > Yea, I agree, this seems like an existing issue that you could hit with just > the PTE path: > > CPU1 - process 1 CPU2 - process 2 > > __dax_fault() - read file f, index 0 > get_block() -> returns hole > __dax_fault() - write file f, index 0 > get_block() -> allocates blocks > ... > skips unmap_mapping_range() and > delete_from_page_cache() because it didn't > find a page for this pgoff > dax_insert_mapping() > dax_load_hole() > *data corruption* > > > ... thinking about possible fixes ... > > > > So we need some exclusion that makes sure pgoff->block mapping information > > is uptodate at the moment we insert it into page tables. The simplest > > reasonably fast thing I can see is: > > > > When handling a read fault, things stay as is and filesystem protects the > > fault with an equivalent of EXT4_I(inode)->i_mmap_sem held for reading. When > > handling a write fault we first grab EXT4_I(inode)->i_mmap_sem for reading > > and try a read fault. If __dax_fault() sees a hole returned from > > get_blocks() during a write fault, it bails out. Filesystem grabs > > EXT4_I(inode)->i_mmap_sem for writing and retries with different > > get_blocks() callback which will allocate blocks. That way we get proper > > exclusion for faults needing to allocate blocks. Thoughts? > > I think this would work. ext4, ext2 and xfs all handle their exclusion with > rw_semaphores, so this should work for each of them, I think. Thanks for the > problem statement & solution! :) > > I guess our best course is to make sure that we don't make this existing > problem worse via the fsync/msync patches by handling the error gracefully, > and fix this for v4.6. I do feel the need to point out that this is a > pre-existing issue with DAX, and that my fsync patches just happened to help > us find it. They don't make the situation any better or any worse, and I > really hope this issue doesn't end up blocking the fsync/msync patches from > getting merged for v4.5. Yeah, I agree this is a preexisting problem and mostly independent of your fsync series so it can be dealt with once fsync series lands. The only thing where this meets is the locking - you will have exclusive hold of the inode and all its mapping when doing a write fault allocating a block. That may save you some dances with i_mmap_lock (but I didn't think about this too much). Honza -- Jan Kara SUSE Labs, CR From jack@suse.cz Mon Jan 11 07:26:11 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4956F29DF5 for ; Mon, 11 Jan 2016 07:26:11 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 38A2A304053 for ; Mon, 11 Jan 2016 05:26:08 -0800 (PST) X-ASG-Debug-ID: 1452518765-04cbb05f7f71230001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id f9j7yfRFagjDkFvW (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 11 Jan 2016 05:26:06 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 7C23CAB12; Mon, 11 Jan 2016 13:26:03 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id 10262823D8; Mon, 11 Jan 2016 14:26:17 +0100 (CET) Date: Mon, 11 Jan 2016 14:26:17 +0100 From: Jan Kara To: Christoph Hellwig Cc: Eric Sandeen , fsdevel , xfs@oss.sgi.com, Jan Kara Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 Message-ID: <20160111132617.GD6262@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 References: <568FEA2C.6080708@redhat.com> <20160109072600.GA21636@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160109072600.GA21636@infradead.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452518765 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26030 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Fri 08-01-16 23:26:00, Christoph Hellwig wrote: > On Fri, Jan 08, 2016 at 10:56:12AM -0600, Eric Sandeen wrote: > > This adds a new quotactl, Q_XGETQUOTA2. > > > > Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will > > return quota information for the id equal to or greater than > > the id requested. In other words, if the specified id has > > no quota, the command will return quota information for the > > next higher id which does have a quota set. If no higher id > > has an active quota, -ESRCH is returned. > > Please add a flags argument to Q_XGETQUOTA2, and then make the > new behavior the first flag. Keep Q_XGETQUOTA behavior for the > flag-less case. That way we get future etensibility for free. So this is what I wanted to suggest at first as well. What I somewhat dislike is that 'addr' must now point to something like: struct getquota_args { __u64 flags; struct fs_disk_quota ret; }; which is not as nice as passing pointer to fs_disk_quota directly. But probably still OK. So I agree with the flags idea. Another issue is that OCFS2 and ext4 with quota in hidden inodes would need call with these capabilities as well (they have the same problem as xfs). For reporting VFS quota we use bytes for space (since some filesystems need it) and we don't need RT fields (easy so zero-fill) and warning fields (there zeros may be confusing). So we would need something like struct qc_dqblk (currently only internal) extended with ID and maybe flags field XFS is using to return quota type which can contain all the information and tell which info is actually valid as return structure. Honza -- Jan Kara SUSE Labs, CR From spatel@omnifone.com Mon Jan 11 08:20:22 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 4E25429DF5 for ; Mon, 11 Jan 2016 08:20:22 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 2101B8F804B for ; Mon, 11 Jan 2016 06:20:21 -0800 (PST) X-ASG-Debug-ID: 1452522014-04bdf06f6e71e10001-NocioJ Received: from mail1.bemta14.messagelabs.com (mail1.bemta14.messagelabs.com [193.109.254.116]) by cuda.sgi.com with ESMTP id Gnqus4aClH4IfBfc (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 06:20:15 -0800 (PST) X-Barracuda-Envelope-From: spatel@omnifone.com X-Barracuda-Apparent-Source-IP: 193.109.254.116 Received: from [193.109.255.147] by server-12.bemta-14.messagelabs.com id 8F/71-09834-D1AB3965; Mon, 11 Jan 2016 14:20:13 +0000 X-Env-Sender: spatel@omnifone.com X-Msg-Ref: server-10.tower-72.messagelabs.com!1452522013!10271944!1 X-Originating-IP: [195.54.58.132] X-StarScan-Received: X-StarScan-Version: 7.35.1; banners=omnifone.com,-,- X-VirusChecked: Checked Received: (qmail 31877 invoked from network); 11 Jan 2016 14:20:13 -0000 Received: from brookgreen.ribob01.net (HELO EXFE1IS02.omnifone.com) (195.54.58.132) by server-10.tower-72.messagelabs.com with AES128-SHA encrypted SMTP; 11 Jan 2016 14:20:13 -0000 Received: from EXBE1IS02.omnifone.com ([::1]) by EXFE1IS02.omnifone.com ([::1]) with mapi id 14.01.0438.000; Mon, 11 Jan 2016 14:20:13 +0000 From: Sandeep Patel To: Emmanuel Florac CC: "xfs@oss.sgi.com" Subject: RE: XFS corruptions Thread-Topic: XFS corruptions X-ASG-Orig-Subj: RE: XFS corruptions Thread-Index: AdErjxENXmHY5tPxRXq2ndhekRd0vAABy5kAAAAX3HAAAi4uAAAkewtgAAN7poAAAFYU0AHGx3oQAAGTUQAFtVpUkA== Date: Mon, 11 Jan 2016 14:20:12 +0000 Deferred-Delivery: Mon, 11 Jan 2016 14:20:00 +0000 Message-ID: <558408F298C8CE4C89D93BD56AB474E801861CBD2B@EXBE1IS02.omnifone.com> References: <558408F298C8CE4C89D93BD56AB474E8018617B2B3@EXBE1IS02.omnifone.com> <20151130184048.3fa123e0@harpe.intellique.com> <558408F298C8CE4C89D93BD56AB474E8018617B3A6@EXBE1IS02.omnifone.com> <20151130194554.6140bf30@harpe.intellique.com> <558408F298C8CE4C89D93BD56AB474E8018617C636@EXBE1IS02.omnifone.com> <20151201145011.6df02063@harpe.intellique.com> <558408F298C8CE4C89D93BD56AB474E8018618503B@EXBE1IS02.omnifone.com> <20151210164640.78a13865@harpe.intellique.com> In-Reply-To: <20151210164640.78a13865@harpe.intellique.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.4.1.73] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Barracuda-Connect: mail1.bemta14.messagelabs.com[193.109.254.116] X-Barracuda-Start-Time: 1452522015 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.02 X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, THREAD_INDEX, THREAD_TOPIC X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26031 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.01 THREAD_INDEX thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig== 0.01 THREAD_TOPIC Thread-Topic: ...(Japanese Subject)... 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header SGkgRW1tYW51ZWwsCgpJIGhhdmUgbWFuYWdlZCB0byB1cGdyYWRlIG91ciBrZXJuZWwgdG8gMy44 LjEzLTExOC4yLjIuZWw2dWVrLng4Nl82NCB3aGljaCB3YXMgdGhlIGxhdGVzdCBhdmFpbGFibGUg b24gb3VyIHJlcG8uIEkgd2lsbCBsZXQgeW91IGtub3cgaG93IEkgZ2V0IG9uLgoKVGhhbmtzClNh bmRlZXAKCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tCkZyb206IEVtbWFudWVsIEZsb3JhYyBb bWFpbHRvOmVmbG9yYWNAaW50ZWxsaXF1ZS5jb21dIApTZW50OiAxMCBEZWNlbWJlciAyMDE1IDE1 OjQ3ClRvOiBTYW5kZWVwIFBhdGVsCkNjOiB4ZnNAb3NzLnNnaS5jb20KU3ViamVjdDogUmU6IFhG UyBjb3JydXB0aW9ucwoKTGUgVGh1LCAxMCBEZWMgMjAxNSAxNTowNzozMiArMDAwMApTYW5kZWVw IFBhdGVsIDxzcGF0ZWxAb21uaWZvbmUuY29tPiDDqWNyaXZhaXQ6Cgo+IEFueSBsdWNrIHdpdGgg dGhlIHRlc3RpbmcuIEkgYW0gYXR0YWNoaW5nIGEgY29weSBvZiBteSBkbWVzZyBvdXRwdXQgCj4g d2hpY2gga2VlcHMgb24gcmVwZWF0aW5nLiBJbiB0aGlzIGNhc2UsIHRoZSBlcnJvciBzZWVtcyB0 byBiZSB3aXRoIG9uZSAKPiBkaW5vZGUuCj4gCgpJJ3ZlIHJ1biBzb21lIHRlc3RzIG9uIGEgMTh4 NiBUQiBhcnJheSwgYW5kIG5vIGVycm9yIG9jY3VycmVkLiBIb3dldmVyIEknbSBydW5uaW5nIGEg bXVjaCBuZXdlciBrZXJuZWwsIDMuMTguMjQuCgpDb3VsZCB5b3UgbWF5YmUgdHJ5IGluc3RhbGxp bmcgYSBkaWZmZXJlbnQga2VybmVsIG9uIHNvbWUgbWFjaGluZXMgYW5kIHNlZSBpZiBpdCBwZXJm b3JtcyBkaWZmZXJlbnRseT8KCi0tCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpFbW1hbnVlbCBGbG9yYWMgICAg IHwgICBEaXJlY3Rpb24gdGVjaG5pcXVlCiAgICAgICAgICAgICAgICAgICAgfCAgIEludGVsbGlx dWUKICAgICAgICAgICAgICAgICAgICB8CTxlZmxvcmFjQGludGVsbGlxdWUuY29tPgogICAgICAg ICAgICAgICAgICAgIHwgICArMzMgMSA3OCA5NCA4NCAwMgotLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KCl9fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18KVGhpcyBlbWFpbCBoYXMgYmVlbiBzY2FubmVkIGJ5IHRoZSBTeW1hbnRlYyBFbWFp bCBTZWN1cml0eS5jbG91ZCBzZXJ2aWNlLgpGb3IgbW9yZSBpbmZvcm1hdGlvbiBwbGVhc2Ugdmlz aXQgaHR0cDovL3d3dy5zeW1hbnRlY2Nsb3VkLmNvbSBfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCgpfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fClRoaXMgZW1haWwgaGFzIGJlZW4gc2Nhbm5lZCBieSB0aGUgU3ltYW50ZWMgRW1haWwgU2Vj dXJpdHkuY2xvdWQgc2VydmljZS4KRm9yIG1vcmUgaW5mb3JtYXRpb24gcGxlYXNlIHZpc2l0IGh0 dHA6Ly93d3cuc3ltYW50ZWNjbG91ZC5jb20KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo= From eflorac@intellique.com Mon Jan 11 09:16:05 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9BC9029DF5 for ; Mon, 11 Jan 2016 09:16:05 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 388B2AC005 for ; Mon, 11 Jan 2016 07:16:02 -0800 (PST) X-ASG-Debug-ID: 1452525359-04cb6c75dd704f0001-NocioJ Received: from mail1.g1.pair.com (mail1.g1.pair.com [66.39.3.162]) by cuda.sgi.com with ESMTP id VtHyjKlpdHFdJqVB (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:16:00 -0800 (PST) X-Barracuda-Envelope-From: eflorac@intellique.com X-Barracuda-Apparent-Source-IP: 66.39.3.162 Received: from localhost (localhost [127.0.0.1]) by mail1.g1.pair.com (Postfix) with SMTP id 652812CE0F; Mon, 11 Jan 2016 10:15:59 -0500 (EST) Received: from harpe.intellique.com (labo.djinux.com [82.225.196.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail1.g1.pair.com (Postfix) with ESMTPSA id A072D2CCB6; Mon, 11 Jan 2016 10:15:58 -0500 (EST) Date: Mon, 11 Jan 2016 16:16:02 +0100 From: Emmanuel Florac To: Sandeep Patel Cc: "xfs@oss.sgi.com" Subject: Re: XFS corruptions Message-ID: <20160111161602.0c8413ca@harpe.intellique.com> X-ASG-Orig-Subj: Re: XFS corruptions In-Reply-To: <558408F298C8CE4C89D93BD56AB474E801861CBD2B@EXBE1IS02.omnifone.com> References: <558408F298C8CE4C89D93BD56AB474E8018617B2B3@EXBE1IS02.omnifone.com> <20151130184048.3fa123e0@harpe.intellique.com> <558408F298C8CE4C89D93BD56AB474E8018617B3A6@EXBE1IS02.omnifone.com> <20151130194554.6140bf30@harpe.intellique.com> <558408F298C8CE4C89D93BD56AB474E8018617C636@EXBE1IS02.omnifone.com> <20151201145011.6df02063@harpe.intellique.com> <558408F298C8CE4C89D93BD56AB474E8018618503B@EXBE1IS02.omnifone.com> <20151210164640.78a13865@harpe.intellique.com> <558408F298C8CE4C89D93BD56AB474E801861CBD2B@EXBE1IS02.omnifone.com> Organization: Intellique X-Mailer: Claws Mail 3.13.1 (GTK+ 2.24.20; i486-slackware-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Barracuda-Connect: mail1.g1.pair.com[66.39.3.162] X-Barracuda-Start-Time: 1452525360 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26031 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Le Mon, 11 Jan 2016 14:20:12 +0000 Sandeep Patel =C3=A9crivait: > Hi Emmanuel, >=20 > I have managed to upgrade our kernel to 3.8.13-118.2.2.el6uek.x86_64 > which was the latest available on our repo. I will let you know how I > get on. >=20 OK. I'm pretty confident it should enhance the situation :) --=20 ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | | +33 1 78 94 84 02 ------------------------------------------------------------------------ From agruenba@redhat.com Mon Jan 11 09:22:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9686729DF5 for ; Mon, 11 Jan 2016 09:22:08 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 5D071304053 for ; Mon, 11 Jan 2016 07:22:05 -0800 (PST) X-ASG-Debug-ID: 1452525724-04bdf06f6e74090001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id sXoo3ETTVOxfyRHp (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:04 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 002D8C0BF2C0; Mon, 11 Jan 2016 15:22:02 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbF000775; Mon, 11 Jan 2016 10:21:55 -0500 From: Andreas Gruenbacher To: Alexander Viro Cc: Andreas Gruenbacher , Christoph Hellwig , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Subject: [PATCH v17 00/22] Richacls (Core and Ext4) Date: Mon, 11 Jan 2016 16:21:31 +0100 X-ASG-Orig-Subj: [PATCH v17 00/22] Richacls (Core and Ext4) Message-Id: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525724 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Al, could you please merge the current richacl patch queue, up to and including the ext4 support? The patches apply on top of your current xattr.work branch. Other than rebasing onto xattr.work, the patches haven't changed since November 9, 2015. Previous posting: https://lwn.net/Articles/666596/. The complete patch queue is available here: git://git.kernel.org/pub/scm/linux/kernel/git/agruen/linux-richacl.git \ richacl-2015-12-15 The richacl user-space utilitites, man pages, and test suite are available here: https://github.com/andreas-gruenbacher/richacl Changes to other user-space packages for richacl: https://github.com/andreas-gruenbacher/coreutils https://github.com/andreas-gruenbacher/e2fsprogs https://github.com/andreas-gruenbacher/xfsprogs-dev https://github.com/andreas-gruenbacher/nfs-utils Please see the richacl homepage for more information: http://www.bestbits.at/richacl/ Thanks, Andreas Andreas Gruenbacher (20): vfs: Add IS_ACL() and IS_RICHACL() tests vfs: Add MAY_CREATE_FILE and MAY_CREATE_DIR permission flags vfs: Add MAY_DELETE_SELF and MAY_DELETE_CHILD permission flags vfs: Make the inode passed to inode_change_ok non-const vfs: Add permission flags for setting file attributes richacl: In-memory representation and helper functions richacl: Permission mapping functions richacl: Compute maximum file masks from an acl richacl: Permission check algorithm posix_acl: Unexport acl_by_type and make it static vfs: Cache base_acl objects in inodes vfs: Add get_richacl and set_richacl inode operations vfs: Cache richacl in struct inode richacl: Update the file masks in chmod() richacl: Check if an acl is equivalent to a file mode richacl: Create-time inheritance richacl: Automatic Inheritance richacl: xattr mapping functions richacl: Add richacl xattr handler vfs: Add richacl permission checking Aneesh Kumar K.V (2): ext4: Add richacl support ext4: Add richacl feature flag drivers/staging/lustre/lustre/llite/llite_lib.c | 2 +- fs/Kconfig | 3 + fs/Makefile | 2 + fs/attr.c | 81 +++- fs/ext4/Kconfig | 11 + fs/ext4/Makefile | 1 + fs/ext4/ext4.h | 6 +- fs/ext4/file.c | 3 + fs/ext4/ialloc.c | 11 +- fs/ext4/inode.c | 12 +- fs/ext4/namei.c | 5 + fs/ext4/richacl.c | 142 ++++++ fs/ext4/richacl.h | 40 ++ fs/ext4/super.c | 49 +- fs/ext4/xattr.c | 7 + fs/f2fs/acl.c | 4 +- fs/inode.c | 15 +- fs/jffs2/acl.c | 10 +- fs/namei.c | 118 +++-- fs/posix_acl.c | 50 +-- fs/richacl_base.c | 564 ++++++++++++++++++++++++ fs/richacl_inode.c | 333 ++++++++++++++ fs/richacl_xattr.c | 298 +++++++++++++ fs/xattr.c | 34 +- include/linux/fs.h | 60 ++- include/linux/posix_acl.h | 13 +- include/linux/richacl.h | 208 +++++++++ include/linux/richacl_xattr.h | 44 ++ include/uapi/linux/Kbuild | 2 + include/uapi/linux/fs.h | 3 +- include/uapi/linux/richacl.h | 152 +++++++ include/uapi/linux/richacl_xattr.h | 44 ++ include/uapi/linux/xattr.h | 2 + 33 files changed, 2222 insertions(+), 107 deletions(-) create mode 100644 fs/ext4/richacl.c create mode 100644 fs/ext4/richacl.h create mode 100644 fs/richacl_base.c create mode 100644 fs/richacl_inode.c create mode 100644 fs/richacl_xattr.c create mode 100644 include/linux/richacl.h create mode 100644 include/linux/richacl_xattr.h create mode 100644 include/uapi/linux/richacl.h create mode 100644 include/uapi/linux/richacl_xattr.h -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:22:11 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4930829DFC for ; Mon, 11 Jan 2016 09:22:11 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 1A909304053 for ; Mon, 11 Jan 2016 07:22:11 -0800 (PST) X-ASG-Debug-ID: 1452525729-04bdf06f6b74090001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 6HLUQwbFynBBet4x (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:10 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id A6BFA8C1C0; Mon, 11 Jan 2016 15:22:09 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbG000775; Mon, 11 Jan 2016 10:22:03 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 01/22] vfs: Add IS_ACL() and IS_RICHACL() tests Date: Mon, 11 Jan 2016 16:21:32 +0100 X-ASG-Orig-Subj: [PATCH v17 01/22] vfs: Add IS_ACL() and IS_RICHACL() tests Message-Id: <1452525713-11348-2-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525729 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 The vfs does not apply the umask for file systems that support acls. The test used for this used to be called IS_POSIXACL(). Switch to a new IS_ACL() test to check for either posix acls or richacls instead. Add a new MS_RICHACL flag and IS_RICHACL() test for richacls alone. The IS_POSIXACL() test is still needed by nfsd. Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields Reviewed-by: Andreas Dilger --- fs/Kconfig | 3 +++ fs/namei.c | 8 ++++---- include/linux/fs.h | 12 ++++++++++++ include/uapi/linux/fs.h | 3 ++- 4 files changed, 21 insertions(+), 5 deletions(-) diff --git a/fs/Kconfig b/fs/Kconfig index 6ce72d8..379d83e 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -62,6 +62,9 @@ endif # BLOCK config FS_POSIX_ACL def_bool n +config FS_RICHACL + def_bool n + config EXPORTFS tristate diff --git a/fs/namei.c b/fs/namei.c index 0c3974c..4e84e76 100644 --- a/fs/namei.c +++ b/fs/namei.c @@ -2794,7 +2794,7 @@ static int atomic_open(struct nameidata *nd, struct dentry *dentry, } mode = op->mode; - if ((open_flag & O_CREAT) && !IS_POSIXACL(dir)) + if ((open_flag & O_CREAT) && !IS_ACL(dir)) mode &= ~current_umask(); excl = (open_flag & (O_EXCL | O_CREAT)) == (O_EXCL | O_CREAT); @@ -2978,7 +2978,7 @@ static int lookup_open(struct nameidata *nd, struct path *path, /* Negative dentry, just create the file */ if (!dentry->d_inode && (op->open_flag & O_CREAT)) { umode_t mode = op->mode; - if (!IS_POSIXACL(dir->d_inode)) + if (!IS_ACL(dir->d_inode)) mode &= ~current_umask(); /* * This write is needed to ensure that a @@ -3549,7 +3549,7 @@ retry: if (IS_ERR(dentry)) return PTR_ERR(dentry); - if (!IS_POSIXACL(path.dentry->d_inode)) + if (!IS_ACL(path.dentry->d_inode)) mode &= ~current_umask(); error = security_path_mknod(&path, dentry, mode, dev); if (error) @@ -3618,7 +3618,7 @@ retry: if (IS_ERR(dentry)) return PTR_ERR(dentry); - if (!IS_POSIXACL(path.dentry->d_inode)) + if (!IS_ACL(path.dentry->d_inode)) mode &= ~current_umask(); error = security_path_mkdir(&path, dentry, mode); if (!error) diff --git a/include/linux/fs.h b/include/linux/fs.h index 3aa5142..58e0a19 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1766,6 +1766,12 @@ struct super_operations { #define IS_IMMUTABLE(inode) ((inode)->i_flags & S_IMMUTABLE) #define IS_POSIXACL(inode) __IS_FLG(inode, MS_POSIXACL) +#ifdef CONFIG_FS_RICHACL +#define IS_RICHACL(inode) __IS_FLG(inode, MS_RICHACL) +#else +#define IS_RICHACL(inode) 0 +#endif + #define IS_DEADDIR(inode) ((inode)->i_flags & S_DEAD) #define IS_NOCMTIME(inode) ((inode)->i_flags & S_NOCMTIME) #define IS_SWAPFILE(inode) ((inode)->i_flags & S_SWAPFILE) @@ -1779,6 +1785,12 @@ struct super_operations { (inode)->i_rdev == WHITEOUT_DEV) /* + * IS_ACL() tells the VFS to not apply the umask + * and use check_acl for acl permission checks when defined. + */ +#define IS_ACL(inode) __IS_FLG(inode, MS_POSIXACL | MS_RICHACL) + +/* * Inode state bits. Protected by inode->i_lock * * Three bits determine the dirty state of the inode, I_DIRTY_SYNC, diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index f15d980..324f438 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -81,7 +81,7 @@ struct inodes_stat_t { #define MS_VERBOSE 32768 /* War is peace. Verbosity is silence. MS_VERBOSE is deprecated. */ #define MS_SILENT 32768 -#define MS_POSIXACL (1<<16) /* VFS does not apply the umask */ +#define MS_POSIXACL (1<<16) /* Supports POSIX ACLs */ #define MS_UNBINDABLE (1<<17) /* change to unbindable */ #define MS_PRIVATE (1<<18) /* change to private */ #define MS_SLAVE (1<<19) /* change to slave */ @@ -91,6 +91,7 @@ struct inodes_stat_t { #define MS_I_VERSION (1<<23) /* Update inode I_version field */ #define MS_STRICTATIME (1<<24) /* Always perform atime updates */ #define MS_LAZYTIME (1<<25) /* Update the on-disk [acm]times lazily */ +#define MS_RICHACL (1<<26) /* Supports richacls */ /* These sb flags are internal to the kernel */ #define MS_NOSEC (1<<28) -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:22:19 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 1709E29DF5 for ; Mon, 11 Jan 2016 09:22:19 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id A012DAC003 for ; Mon, 11 Jan 2016 07:22:18 -0800 (PST) X-ASG-Debug-ID: 1452525736-04cbb05f7d749a0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id jvgGuLAZlEpEYSMP (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:17 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 65B7631206F; Mon, 11 Jan 2016 15:22:16 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbH000775; Mon, 11 Jan 2016 10:22:10 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 02/22] vfs: Add MAY_CREATE_FILE and MAY_CREATE_DIR permission flags Date: Mon, 11 Jan 2016 16:21:33 +0100 X-ASG-Orig-Subj: [PATCH v17 02/22] vfs: Add MAY_CREATE_FILE and MAY_CREATE_DIR permission flags Message-Id: <1452525713-11348-3-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525737 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Richacls distinguish between creating non-directories and directories. To support that, add an isdir parameter to may_create(). When checking inode_permission() for create permission, pass in an additional MAY_CREATE_FILE or MAY_CREATE_DIR mask flag. Add may_replace() to allow checking for delete and create access when replacing an existing file in vfs_rename(). Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields Reviewed-by: Andreas Dilger --- fs/namei.c | 49 +++++++++++++++++++++++++++++++++---------------- include/linux/fs.h | 2 ++ 2 files changed, 35 insertions(+), 16 deletions(-) diff --git a/fs/namei.c b/fs/namei.c index 4e84e76..9fd9442 100644 --- a/fs/namei.c +++ b/fs/namei.c @@ -453,7 +453,9 @@ static int sb_permission(struct super_block *sb, struct inode *inode, int mask) * this, letting us set arbitrary permissions for filesystem access without * changing the "normal" UIDs which are used for other things. * - * When checking for MAY_APPEND, MAY_WRITE must also be set in @mask. + * MAY_WRITE must be set in @mask whenever MAY_APPEND, MAY_CREATE_FILE, or + * MAY_CREATE_DIR are set. That way, file systems that don't support these + * permissions will check for MAY_WRITE instead. */ int inode_permission(struct inode *inode, int mask) { @@ -2545,7 +2547,8 @@ EXPORT_SYMBOL(__check_sticky); * 10. We don't allow removal of NFS sillyrenamed files; it's handled by * nfs_async_unlink(). */ -static int may_delete(struct inode *dir, struct dentry *victim, bool isdir) +static int may_delete_or_replace(struct inode *dir, struct dentry *victim, + bool isdir, int mask) { struct inode *inode = d_backing_inode(victim); int error; @@ -2557,7 +2560,7 @@ static int may_delete(struct inode *dir, struct dentry *victim, bool isdir) BUG_ON(victim->d_parent->d_inode != dir); audit_inode_child(dir, victim, AUDIT_TYPE_CHILD_DELETE); - error = inode_permission(dir, MAY_WRITE | MAY_EXEC); + error = inode_permission(dir, mask); if (error) return error; if (IS_APPEND(dir)) @@ -2580,6 +2583,18 @@ static int may_delete(struct inode *dir, struct dentry *victim, bool isdir) return 0; } +static int may_delete(struct inode *dir, struct dentry *victim, bool isdir) +{ + return may_delete_or_replace(dir, victim, isdir, MAY_WRITE | MAY_EXEC); +} + +static int may_replace(struct inode *dir, struct dentry *victim, bool isdir) +{ + int mask = isdir ? MAY_CREATE_DIR : MAY_CREATE_FILE; + + return may_delete_or_replace(dir, victim, isdir, mask | MAY_WRITE | MAY_EXEC); +} + /* Check whether we can create an object with dentry child in directory * dir. * 1. We can't do it if child already exists (open has special treatment for @@ -2588,14 +2603,16 @@ static int may_delete(struct inode *dir, struct dentry *victim, bool isdir) * 3. We should have write and exec permissions on dir * 4. We can't do it if dir is immutable (done in permission()) */ -static inline int may_create(struct inode *dir, struct dentry *child) +static inline int may_create(struct inode *dir, struct dentry *child, bool isdir) { + int mask = isdir ? MAY_CREATE_DIR : MAY_CREATE_FILE; + audit_inode_child(dir, child, AUDIT_TYPE_CHILD_CREATE); if (child->d_inode) return -EEXIST; if (IS_DEADDIR(dir)) return -ENOENT; - return inode_permission(dir, MAY_WRITE | MAY_EXEC); + return inode_permission(dir, MAY_WRITE | MAY_EXEC | mask); } /* @@ -2645,7 +2662,7 @@ EXPORT_SYMBOL(unlock_rename); int vfs_create(struct inode *dir, struct dentry *dentry, umode_t mode, bool want_excl) { - int error = may_create(dir, dentry); + int error = may_create(dir, dentry, false); if (error) return error; @@ -3490,7 +3507,7 @@ EXPORT_SYMBOL(user_path_create); int vfs_mknod(struct inode *dir, struct dentry *dentry, umode_t mode, dev_t dev) { - int error = may_create(dir, dentry); + int error = may_create(dir, dentry, false); if (error) return error; @@ -3582,7 +3599,7 @@ SYSCALL_DEFINE3(mknod, const char __user *, filename, umode_t, mode, unsigned, d int vfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode) { - int error = may_create(dir, dentry); + int error = may_create(dir, dentry, true); unsigned max_links = dir->i_sb->s_max_links; if (error) @@ -3663,7 +3680,7 @@ EXPORT_SYMBOL(dentry_unhash); int vfs_rmdir(struct inode *dir, struct dentry *dentry) { - int error = may_delete(dir, dentry, 1); + int error = may_delete(dir, dentry, true); if (error) return error; @@ -3785,7 +3802,7 @@ SYSCALL_DEFINE1(rmdir, const char __user *, pathname) int vfs_unlink(struct inode *dir, struct dentry *dentry, struct inode **delegated_inode) { struct inode *target = dentry->d_inode; - int error = may_delete(dir, dentry, 0); + int error = may_delete(dir, dentry, false); if (error) return error; @@ -3919,7 +3936,7 @@ SYSCALL_DEFINE1(unlink, const char __user *, pathname) int vfs_symlink(struct inode *dir, struct dentry *dentry, const char *oldname) { - int error = may_create(dir, dentry); + int error = may_create(dir, dentry, false); if (error) return error; @@ -4002,7 +4019,7 @@ int vfs_link(struct dentry *old_dentry, struct inode *dir, struct dentry *new_de if (!inode) return -ENOENT; - error = may_create(dir, new_dentry); + error = may_create(dir, new_dentry, false); if (error) return error; @@ -4195,14 +4212,14 @@ int vfs_rename(struct inode *old_dir, struct dentry *old_dentry, return error; if (!target) { - error = may_create(new_dir, new_dentry); + error = may_create(new_dir, new_dentry, is_dir); } else { new_is_dir = d_is_dir(new_dentry); if (!(flags & RENAME_EXCHANGE)) - error = may_delete(new_dir, new_dentry, is_dir); + error = may_replace(new_dir, new_dentry, is_dir); else - error = may_delete(new_dir, new_dentry, new_is_dir); + error = may_replace(new_dir, new_dentry, new_is_dir); } if (error) return error; @@ -4465,7 +4482,7 @@ SYSCALL_DEFINE2(rename, const char __user *, oldname, const char __user *, newna int vfs_whiteout(struct inode *dir, struct dentry *dentry) { - int error = may_create(dir, dentry); + int error = may_create(dir, dentry, false); if (error) return error; diff --git a/include/linux/fs.h b/include/linux/fs.h index 58e0a19..c0d5760 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -82,6 +82,8 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); #define MAY_CHDIR 0x00000040 /* called from RCU mode, don't block */ #define MAY_NOT_BLOCK 0x00000080 +#define MAY_CREATE_FILE 0x00000100 +#define MAY_CREATE_DIR 0x00000200 /* * flags in file.f_mode. Note that FMODE_READ and FMODE_WRITE must correspond -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:22:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id AB6D429E04 for ; Mon, 11 Jan 2016 09:22:25 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 2FE57AC004 for ; Mon, 11 Jan 2016 07:22:25 -0800 (PST) X-ASG-Debug-ID: 1452525743-04cb6c75de70800001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id lmByGv9ReTjuzGMp (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:23 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 1B2F919D611; Mon, 11 Jan 2016 15:22:23 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbI000775; Mon, 11 Jan 2016 10:22:16 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 03/22] vfs: Add MAY_DELETE_SELF and MAY_DELETE_CHILD permission flags Date: Mon, 11 Jan 2016 16:21:34 +0100 X-ASG-Orig-Subj: [PATCH v17 03/22] vfs: Add MAY_DELETE_SELF and MAY_DELETE_CHILD permission flags Message-Id: <1452525713-11348-4-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525743 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Normally, deleting a file requires MAY_WRITE access to the parent directory. With richacls, a file may be deleted with MAY_DELETE_CHILD access to the parent directory or with MAY_DELETE_SELF access to the file. To support that, pass the MAY_DELETE_CHILD mask flag to inode_permission() when checking for delete access inside a directory, and MAY_DELETE_SELF when checking for delete access to a file itelf. The MAY_DELETE_SELF permission overrides the sticky directory check. Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/namei.c | 20 ++++++++++++-------- include/linux/fs.h | 2 ++ 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/fs/namei.c b/fs/namei.c index 9fd9442..d242992 100644 --- a/fs/namei.c +++ b/fs/namei.c @@ -453,9 +453,9 @@ static int sb_permission(struct super_block *sb, struct inode *inode, int mask) * this, letting us set arbitrary permissions for filesystem access without * changing the "normal" UIDs which are used for other things. * - * MAY_WRITE must be set in @mask whenever MAY_APPEND, MAY_CREATE_FILE, or - * MAY_CREATE_DIR are set. That way, file systems that don't support these - * permissions will check for MAY_WRITE instead. + * MAY_WRITE must be set in @mask whenever MAY_APPEND, MAY_CREATE_FILE, + * MAY_CREATE_DIR, or MAY_DELETE_CHILD are set. That way, file systems that + * don't support these permissions will check for MAY_WRITE instead. */ int inode_permission(struct inode *inode, int mask) { @@ -2560,14 +2560,18 @@ static int may_delete_or_replace(struct inode *dir, struct dentry *victim, BUG_ON(victim->d_parent->d_inode != dir); audit_inode_child(dir, victim, AUDIT_TYPE_CHILD_DELETE); - error = inode_permission(dir, mask); + error = inode_permission(dir, mask | MAY_WRITE | MAY_DELETE_CHILD); + if (!error && check_sticky(dir, inode)) + error = -EPERM; + if (error && IS_RICHACL(inode) && + inode_permission(inode, MAY_DELETE_SELF) == 0 && + inode_permission(dir, mask) == 0) + error = 0; if (error) return error; if (IS_APPEND(dir)) return -EPERM; - - if (check_sticky(dir, inode) || IS_APPEND(inode) || - IS_IMMUTABLE(inode) || IS_SWAPFILE(inode)) + if (IS_APPEND(inode) || IS_IMMUTABLE(inode) || IS_SWAPFILE(inode)) return -EPERM; if (isdir) { if (!d_is_dir(victim)) @@ -2585,7 +2589,7 @@ static int may_delete_or_replace(struct inode *dir, struct dentry *victim, static int may_delete(struct inode *dir, struct dentry *victim, bool isdir) { - return may_delete_or_replace(dir, victim, isdir, MAY_WRITE | MAY_EXEC); + return may_delete_or_replace(dir, victim, isdir, MAY_EXEC); } static int may_replace(struct inode *dir, struct dentry *victim, bool isdir) diff --git a/include/linux/fs.h b/include/linux/fs.h index c0d5760..7a45120 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -84,6 +84,8 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); #define MAY_NOT_BLOCK 0x00000080 #define MAY_CREATE_FILE 0x00000100 #define MAY_CREATE_DIR 0x00000200 +#define MAY_DELETE_CHILD 0x00000400 +#define MAY_DELETE_SELF 0x00000800 /* * flags in file.f_mode. Note that FMODE_READ and FMODE_WRITE must correspond -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:22:31 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 49DE629DF5 for ; Mon, 11 Jan 2016 09:22:31 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 395FC304053 for ; Mon, 11 Jan 2016 07:22:31 -0800 (PST) X-ASG-Debug-ID: 1452525750-04cb6c75de70820001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id yxSLInLzyc6TwuxS (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:30 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id C62F0B5956; Mon, 11 Jan 2016 15:22:29 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbJ000775; Mon, 11 Jan 2016 10:22:23 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 04/22] vfs: Make the inode passed to inode_change_ok non-const Date: Mon, 11 Jan 2016 16:21:35 +0100 X-ASG-Orig-Subj: [PATCH v17 04/22] vfs: Make the inode passed to inode_change_ok non-const Message-Id: <1452525713-11348-5-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525750 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 We will need to call iop->permission and iop->get_acl from inode_change_ok() for additional permission checks, and both take a non-const inode. Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields Reviewed-by: Andreas Dilger --- fs/attr.c | 2 +- include/linux/fs.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/attr.c b/fs/attr.c index 6530ced..328be71 100644 --- a/fs/attr.c +++ b/fs/attr.c @@ -28,7 +28,7 @@ * Should be called as the first thing in ->setattr implementations, * possibly after taking additional locks. */ -int inode_change_ok(const struct inode *inode, struct iattr *attr) +int inode_change_ok(struct inode *inode, struct iattr *attr) { unsigned int ia_valid = attr->ia_valid; diff --git a/include/linux/fs.h b/include/linux/fs.h index 7a45120..65cebcb 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2857,7 +2857,7 @@ extern int buffer_migrate_page(struct address_space *, #define buffer_migrate_page NULL #endif -extern int inode_change_ok(const struct inode *, struct iattr *); +extern int inode_change_ok(struct inode *, struct iattr *); extern int inode_newsize_ok(const struct inode *, loff_t offset); extern void setattr_copy(struct inode *inode, const struct iattr *attr); -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:22:39 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9E1F729DFD for ; Mon, 11 Jan 2016 09:22:39 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 8EFE0304066 for ; Mon, 11 Jan 2016 07:22:39 -0800 (PST) X-ASG-Debug-ID: 1452525756-04cbb05f7d749c0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id myCFdqtHF4yDlohu (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:37 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 6F76EA4527; Mon, 11 Jan 2016 15:22:36 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbK000775; Mon, 11 Jan 2016 10:22:30 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 05/22] vfs: Add permission flags for setting file attributes Date: Mon, 11 Jan 2016 16:21:36 +0100 X-ASG-Orig-Subj: [PATCH v17 05/22] vfs: Add permission flags for setting file attributes Message-Id: <1452525713-11348-6-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525756 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Richacls support permissions that allow to take ownership of a file, change the file permissions, and set the file timestamps. Support that by introducing new permission mask flags and by checking for those mask flags in inode_change_ok(). Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/attr.c | 79 +++++++++++++++++++++++++++++++++++++++++++++--------- include/linux/fs.h | 3 +++ 2 files changed, 70 insertions(+), 12 deletions(-) diff --git a/fs/attr.c b/fs/attr.c index 328be71..85483e0 100644 --- a/fs/attr.c +++ b/fs/attr.c @@ -17,6 +17,65 @@ #include /** + * inode_extended_permission - permissions beyond read/write/execute + * + * Check for permissions that only richacls can currently grant. + */ +static int inode_extended_permission(struct inode *inode, int mask) +{ + if (!IS_RICHACL(inode)) + return -EPERM; + return inode_permission(inode, mask); +} + +static bool inode_uid_change_ok(struct inode *inode, kuid_t ia_uid) +{ + if (uid_eq(current_fsuid(), inode->i_uid) && + uid_eq(ia_uid, inode->i_uid)) + return true; + if (uid_eq(current_fsuid(), ia_uid) && + inode_extended_permission(inode, MAY_TAKE_OWNERSHIP) == 0) + return true; + if (capable_wrt_inode_uidgid(inode, CAP_CHOWN)) + return true; + return false; +} + +static bool inode_gid_change_ok(struct inode *inode, kgid_t ia_gid) +{ + int in_group = in_group_p(ia_gid); + if (uid_eq(current_fsuid(), inode->i_uid) && + (in_group || gid_eq(ia_gid, inode->i_gid))) + return true; + if (in_group && inode_extended_permission(inode, MAY_TAKE_OWNERSHIP) == 0) + return true; + if (capable_wrt_inode_uidgid(inode, CAP_CHOWN)) + return true; + return false; +} + +/** + * inode_owner_permitted_or_capable + * + * Check for permissions implicitly granted to the owner, like MAY_CHMOD or + * MAY_SET_TIMES. Equivalent to inode_owner_or_capable for file systems + * without support for those permissions. + */ +static bool inode_owner_permitted_or_capable(struct inode *inode, int mask) +{ + struct user_namespace *ns; + + if (uid_eq(current_fsuid(), inode->i_uid)) + return true; + if (inode_extended_permission(inode, mask) == 0) + return true; + ns = current_user_ns(); + if (ns_capable(ns, CAP_FOWNER) && kuid_has_mapping(ns, inode->i_uid)) + return true; + return false; +} + +/** * inode_change_ok - check if attribute changes to an inode are allowed * @inode: inode to check * @attr: attributes to change @@ -47,22 +106,18 @@ int inode_change_ok(struct inode *inode, struct iattr *attr) return 0; /* Make sure a caller can chown. */ - if ((ia_valid & ATTR_UID) && - (!uid_eq(current_fsuid(), inode->i_uid) || - !uid_eq(attr->ia_uid, inode->i_uid)) && - !capable_wrt_inode_uidgid(inode, CAP_CHOWN)) - return -EPERM; + if (ia_valid & ATTR_UID) + if (!inode_uid_change_ok(inode, attr->ia_uid)) + return -EPERM; /* Make sure caller can chgrp. */ - if ((ia_valid & ATTR_GID) && - (!uid_eq(current_fsuid(), inode->i_uid) || - (!in_group_p(attr->ia_gid) && !gid_eq(attr->ia_gid, inode->i_gid))) && - !capable_wrt_inode_uidgid(inode, CAP_CHOWN)) - return -EPERM; + if (ia_valid & ATTR_GID) + if (!inode_gid_change_ok(inode, attr->ia_gid)) + return -EPERM; /* Make sure a caller can chmod. */ if (ia_valid & ATTR_MODE) { - if (!inode_owner_or_capable(inode)) + if (!inode_owner_permitted_or_capable(inode, MAY_CHMOD)) return -EPERM; /* Also check the setgid bit! */ if (!in_group_p((ia_valid & ATTR_GID) ? attr->ia_gid : @@ -73,7 +128,7 @@ int inode_change_ok(struct inode *inode, struct iattr *attr) /* Check for setting the inode time. */ if (ia_valid & (ATTR_MTIME_SET | ATTR_ATIME_SET | ATTR_TIMES_SET)) { - if (!inode_owner_or_capable(inode)) + if (!inode_owner_permitted_or_capable(inode, MAY_SET_TIMES)) return -EPERM; } diff --git a/include/linux/fs.h b/include/linux/fs.h index 65cebcb..e9e1139 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -86,6 +86,9 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); #define MAY_CREATE_DIR 0x00000200 #define MAY_DELETE_CHILD 0x00000400 #define MAY_DELETE_SELF 0x00000800 +#define MAY_TAKE_OWNERSHIP 0x00001000 +#define MAY_CHMOD 0x00002000 +#define MAY_SET_TIMES 0x00004000 /* * flags in file.f_mode. Note that FMODE_READ and FMODE_WRITE must correspond -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:22:45 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 87C7729DFD for ; Mon, 11 Jan 2016 09:22:45 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 2464EAC004 for ; Mon, 11 Jan 2016 07:22:45 -0800 (PST) X-ASG-Debug-ID: 1452525763-04bdf06f6c740d0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id iPkaD93FIZdvXoil (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:43 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 26AAFC0BF2BD; Mon, 11 Jan 2016 15:22:43 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbL000775; Mon, 11 Jan 2016 10:22:37 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 06/22] richacl: In-memory representation and helper functions Date: Mon, 11 Jan 2016 16:21:37 +0100 X-ASG-Orig-Subj: [PATCH v17 06/22] richacl: In-memory representation and helper functions Message-Id: <1452525713-11348-7-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525763 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 A richacl consists of an NFSv4 acl and an owner, group, and other mask. These three masks correspond to the owner, group, and other file permission bits, but they contain NFSv4 permissions instead of POSIX permissions. Each entry in the NFSv4 acl applies to the file owner (OWNER@), the owning group (GROUP@), everyone (EVERYONE@), or to a specific uid or gid. As in the standard POSIX file permission model, each process is the owner, group, or other file class. A richacl grants a requested access only if the NFSv4 acl in the richacl grants the access (according to the NFSv4 permission check algorithm), and the file mask that applies to the process includes the requested permissions. Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/Makefile | 2 + fs/richacl_base.c | 67 ++++++++++++++++ include/linux/richacl.h | 179 +++++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/Kbuild | 1 + include/uapi/linux/richacl.h | 99 ++++++++++++++++++++++++ 5 files changed, 348 insertions(+) create mode 100644 fs/richacl_base.c create mode 100644 include/linux/richacl.h create mode 100644 include/uapi/linux/richacl.h diff --git a/fs/Makefile b/fs/Makefile index 79f5225..e5994c4 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -48,6 +48,8 @@ obj-$(CONFIG_COREDUMP) += coredump.o obj-$(CONFIG_SYSCTL) += drop_caches.o obj-$(CONFIG_FHANDLE) += fhandle.o +obj-$(CONFIG_FS_RICHACL) += richacl.o +richacl-y := richacl_base.o obj-y += quota/ diff --git a/fs/richacl_base.c b/fs/richacl_base.c new file mode 100644 index 0000000..c3ec928 --- /dev/null +++ b/fs/richacl_base.c @@ -0,0 +1,67 @@ +/* + * Copyright (C) 2006, 2010 Novell, Inc. + * Copyright (C) 2015 Red Hat, Inc. + * Written by Andreas Gruenbacher + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2, or (at your option) any + * later version. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ + +#include +#include +#include +#include +#include + +MODULE_LICENSE("GPL"); + +/** + * richacl_alloc - allocate a richacl + * @count: number of entries + */ +struct richacl * +richacl_alloc(int count, gfp_t gfp) +{ + size_t size = sizeof(struct richacl) + count * sizeof(struct richace); + struct richacl *acl = kzalloc(size, gfp); + + if (acl) { + atomic_set(&acl->a_refcount, 1); + acl->a_count = count; + } + return acl; +} +EXPORT_SYMBOL_GPL(richacl_alloc); + +/** + * richacl_clone - create a copy of a richacl + */ +struct richacl * +richacl_clone(const struct richacl *acl, gfp_t gfp) +{ + int count = acl->a_count; + size_t size = sizeof(struct richacl) + count * sizeof(struct richace); + struct richacl *dup = kmalloc(size, gfp); + + if (dup) { + memcpy(dup, acl, size); + atomic_set(&dup->a_refcount, 1); + } + return dup; +} + +/** + * richace_copy - copy an acl entry + */ +void +richace_copy(struct richace *to, const struct richace *from) +{ + memcpy(to, from, sizeof(struct richace)); +} diff --git a/include/linux/richacl.h b/include/linux/richacl.h new file mode 100644 index 0000000..edb8480 --- /dev/null +++ b/include/linux/richacl.h @@ -0,0 +1,179 @@ +/* + * Copyright (C) 2006, 2010 Novell, Inc. + * Copyright (C) 2015 Red Hat, Inc. + * Written by Andreas Gruenbacher + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2, or (at your option) any + * later version. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ + +#ifndef __RICHACL_H +#define __RICHACL_H + +#include + +struct richace { + unsigned short e_type; + unsigned short e_flags; + unsigned int e_mask; + union { + kuid_t uid; + kgid_t gid; + unsigned int special; + } e_id; +}; + +struct richacl { + atomic_t a_refcount; + unsigned int a_owner_mask; + unsigned int a_group_mask; + unsigned int a_other_mask; + unsigned short a_count; + unsigned short a_flags; + struct richace a_entries[0]; +}; + +#define richacl_for_each_entry(_ace, _acl) \ + for (_ace = (_acl)->a_entries; \ + _ace != (_acl)->a_entries + (_acl)->a_count; \ + _ace++) + +#define richacl_for_each_entry_reverse(_ace, _acl) \ + for (_ace = (_acl)->a_entries + (_acl)->a_count - 1; \ + _ace != (_acl)->a_entries - 1; \ + _ace--) + +/** + * richacl_get - grab another reference to a richacl handle + */ +static inline struct richacl * +richacl_get(struct richacl *acl) +{ + if (acl) + atomic_inc(&acl->a_refcount); + return acl; +} + +/** + * richacl_put - free a richacl handle + */ +static inline void +richacl_put(struct richacl *acl) +{ + if (acl && atomic_dec_and_test(&acl->a_refcount)) + kfree(acl); +} + +/** + * richace_is_owner - check if @ace is an OWNER@ entry + */ +static inline bool +richace_is_owner(const struct richace *ace) +{ + return (ace->e_flags & RICHACE_SPECIAL_WHO) && + ace->e_id.special == RICHACE_OWNER_SPECIAL_ID; +} + +/** + * richace_is_group - check if @ace is a GROUP@ entry + */ +static inline bool +richace_is_group(const struct richace *ace) +{ + return (ace->e_flags & RICHACE_SPECIAL_WHO) && + ace->e_id.special == RICHACE_GROUP_SPECIAL_ID; +} + +/** + * richace_is_everyone - check if @ace is an EVERYONE@ entry + */ +static inline bool +richace_is_everyone(const struct richace *ace) +{ + return (ace->e_flags & RICHACE_SPECIAL_WHO) && + ace->e_id.special == RICHACE_EVERYONE_SPECIAL_ID; +} + +/** + * richace_is_unix_user - check if @ace applies to a specific user + */ +static inline bool +richace_is_unix_user(const struct richace *ace) +{ + return !(ace->e_flags & RICHACE_SPECIAL_WHO) && + !(ace->e_flags & RICHACE_IDENTIFIER_GROUP); +} + +/** + * richace_is_unix_group - check if @ace applies to a specific group + */ +static inline bool +richace_is_unix_group(const struct richace *ace) +{ + return !(ace->e_flags & RICHACE_SPECIAL_WHO) && + (ace->e_flags & RICHACE_IDENTIFIER_GROUP); +} + +/** + * richace_is_inherit_only - check if @ace is for inheritance only + * + * ACEs with the %RICHACE_INHERIT_ONLY_ACE flag set have no effect during + * permission checking. + */ +static inline bool +richace_is_inherit_only(const struct richace *ace) +{ + return ace->e_flags & RICHACE_INHERIT_ONLY_ACE; +} + +/** + * richace_is_inheritable - check if @ace is inheritable + */ +static inline bool +richace_is_inheritable(const struct richace *ace) +{ + return ace->e_flags & (RICHACE_FILE_INHERIT_ACE | + RICHACE_DIRECTORY_INHERIT_ACE); +} + +/** + * richace_is_allow - check if @ace is an %ALLOW type entry + */ +static inline bool +richace_is_allow(const struct richace *ace) +{ + return ace->e_type == RICHACE_ACCESS_ALLOWED_ACE_TYPE; +} + +/** + * richace_is_deny - check if @ace is a %DENY type entry + */ +static inline bool +richace_is_deny(const struct richace *ace) +{ + return ace->e_type == RICHACE_ACCESS_DENIED_ACE_TYPE; +} + +/** + * richace_is_same_identifier - are both identifiers the same? + */ +static inline bool +richace_is_same_identifier(const struct richace *a, const struct richace *b) +{ + return !((a->e_flags ^ b->e_flags) & + (RICHACE_SPECIAL_WHO | RICHACE_IDENTIFIER_GROUP)) && + !memcmp(&a->e_id, &b->e_id, sizeof(a->e_id)); +} + +extern struct richacl *richacl_alloc(int, gfp_t); +extern struct richacl *richacl_clone(const struct richacl *, gfp_t); +extern void richace_copy(struct richace *, const struct richace *); + +#endif /* __RICHACL_H */ diff --git a/include/uapi/linux/Kbuild b/include/uapi/linux/Kbuild index 628e6e6..55699fa 100644 --- a/include/uapi/linux/Kbuild +++ b/include/uapi/linux/Kbuild @@ -350,6 +350,7 @@ header-y += reboot.h header-y += reiserfs_fs.h header-y += reiserfs_xattr.h header-y += resource.h +header-y += richacl.h header-y += rfkill.h header-y += romfs_fs.h header-y += rose.h diff --git a/include/uapi/linux/richacl.h b/include/uapi/linux/richacl.h new file mode 100644 index 0000000..08856f8 --- /dev/null +++ b/include/uapi/linux/richacl.h @@ -0,0 +1,99 @@ +/* + * Copyright (C) 2006, 2010 Novell, Inc. + * Copyright (C) 2015 Red Hat, Inc. + * Written by Andreas Gruenbacher + * + * This file is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This file is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + */ + +#ifndef __UAPI_RICHACL_H +#define __UAPI_RICHACL_H + +/* a_flags values */ +#define RICHACL_WRITE_THROUGH 0x40 +#define RICHACL_MASKED 0x80 + +/* e_type values */ +#define RICHACE_ACCESS_ALLOWED_ACE_TYPE 0x0000 +#define RICHACE_ACCESS_DENIED_ACE_TYPE 0x0001 + +/* e_flags bitflags */ +#define RICHACE_FILE_INHERIT_ACE 0x0001 +#define RICHACE_DIRECTORY_INHERIT_ACE 0x0002 +#define RICHACE_NO_PROPAGATE_INHERIT_ACE 0x0004 +#define RICHACE_INHERIT_ONLY_ACE 0x0008 +#define RICHACE_IDENTIFIER_GROUP 0x0040 +#define RICHACE_SPECIAL_WHO 0x4000 + +/* e_mask bitflags */ +#define RICHACE_READ_DATA 0x00000001 +#define RICHACE_LIST_DIRECTORY 0x00000001 +#define RICHACE_WRITE_DATA 0x00000002 +#define RICHACE_ADD_FILE 0x00000002 +#define RICHACE_APPEND_DATA 0x00000004 +#define RICHACE_ADD_SUBDIRECTORY 0x00000004 +#define RICHACE_READ_NAMED_ATTRS 0x00000008 +#define RICHACE_WRITE_NAMED_ATTRS 0x00000010 +#define RICHACE_EXECUTE 0x00000020 +#define RICHACE_DELETE_CHILD 0x00000040 +#define RICHACE_READ_ATTRIBUTES 0x00000080 +#define RICHACE_WRITE_ATTRIBUTES 0x00000100 +#define RICHACE_WRITE_RETENTION 0x00000200 +#define RICHACE_WRITE_RETENTION_HOLD 0x00000400 +#define RICHACE_DELETE 0x00010000 +#define RICHACE_READ_ACL 0x00020000 +#define RICHACE_WRITE_ACL 0x00040000 +#define RICHACE_WRITE_OWNER 0x00080000 +#define RICHACE_SYNCHRONIZE 0x00100000 + +/* e_id values */ +#define RICHACE_OWNER_SPECIAL_ID 0 +#define RICHACE_GROUP_SPECIAL_ID 1 +#define RICHACE_EVERYONE_SPECIAL_ID 2 + +#define RICHACL_VALID_FLAGS ( \ + RICHACL_WRITE_THROUGH | \ + RICHACL_MASKED ) + +#define RICHACE_VALID_FLAGS ( \ + RICHACE_FILE_INHERIT_ACE | \ + RICHACE_DIRECTORY_INHERIT_ACE | \ + RICHACE_NO_PROPAGATE_INHERIT_ACE | \ + RICHACE_INHERIT_ONLY_ACE | \ + RICHACE_IDENTIFIER_GROUP | \ + RICHACE_SPECIAL_WHO ) + +#define RICHACE_INHERITANCE_FLAGS ( \ + RICHACE_FILE_INHERIT_ACE | \ + RICHACE_DIRECTORY_INHERIT_ACE | \ + RICHACE_NO_PROPAGATE_INHERIT_ACE | \ + RICHACE_INHERIT_ONLY_ACE ) + +/* Valid RICHACE_* flags for directories and non-directories */ +#define RICHACE_VALID_MASK ( \ + RICHACE_READ_DATA | RICHACE_LIST_DIRECTORY | \ + RICHACE_WRITE_DATA | RICHACE_ADD_FILE | \ + RICHACE_APPEND_DATA | RICHACE_ADD_SUBDIRECTORY | \ + RICHACE_READ_NAMED_ATTRS | \ + RICHACE_WRITE_NAMED_ATTRS | \ + RICHACE_EXECUTE | \ + RICHACE_DELETE_CHILD | \ + RICHACE_READ_ATTRIBUTES | \ + RICHACE_WRITE_ATTRIBUTES | \ + RICHACE_WRITE_RETENTION | \ + RICHACE_WRITE_RETENTION_HOLD | \ + RICHACE_DELETE | \ + RICHACE_READ_ACL | \ + RICHACE_WRITE_ACL | \ + RICHACE_WRITE_OWNER | \ + RICHACE_SYNCHRONIZE ) + +#endif /* __UAPI_RICHACL_H */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:22:52 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 6AB8229E11 for ; Mon, 11 Jan 2016 09:22:52 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id F36FFAC006 for ; Mon, 11 Jan 2016 07:22:51 -0800 (PST) X-ASG-Debug-ID: 1452525770-04cb6c75dd70850001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 6kd6Ma1zfMtcb2Ak (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:50 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id D6EFFC1C7516; Mon, 11 Jan 2016 15:22:49 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbM000775; Mon, 11 Jan 2016 10:22:43 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 07/22] richacl: Permission mapping functions Date: Mon, 11 Jan 2016 16:21:38 +0100 X-ASG-Orig-Subj: [PATCH v17 07/22] richacl: Permission mapping functions Message-Id: <1452525713-11348-8-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525770 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 We need to map from POSIX permissions to NFSv4 permissions when a chmod() is done, from NFSv4 permissions to POSIX permissions when an acl is set (which implicitly sets the file permission bits), and from the MAY_READ/MAY_WRITE/MAY_EXEC/MAY_APPEND flags to NFSv4 permissions when doing an access check in a richacl. Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/richacl_base.c | 118 +++++++++++++++++++++++++++++++++++++++++++ include/linux/richacl.h | 3 ++ include/uapi/linux/richacl.h | 44 ++++++++++++++++ 3 files changed, 165 insertions(+) diff --git a/fs/richacl_base.c b/fs/richacl_base.c index c3ec928..a393001 100644 --- a/fs/richacl_base.c +++ b/fs/richacl_base.c @@ -65,3 +65,121 @@ richace_copy(struct richace *to, const struct richace *from) { memcpy(to, from, sizeof(struct richace)); } + +/* + * richacl_mask_to_mode - compute the file permission bits from mask + * @mask: %RICHACE_* permission mask + * + * Compute the file permission bits corresponding to a particular set of + * richacl permissions. + * + * See richacl_masks_to_mode(). + */ +static int +richacl_mask_to_mode(unsigned int mask) +{ + int mode = 0; + + if (mask & RICHACE_POSIX_MODE_READ) + mode |= S_IROTH; + if (mask & RICHACE_POSIX_MODE_WRITE) + mode |= S_IWOTH; + if (mask & RICHACE_POSIX_MODE_EXEC) + mode |= S_IXOTH; + + return mode; +} + +/** + * richacl_masks_to_mode - compute file permission bits from file masks + * + * When setting a richacl, we set the file permission bits to indicate maximum + * permissions: for example, we set the Write permission when a mask contains + * RICHACE_APPEND_DATA even if it does not also contain RICHACE_WRITE_DATA. + * + * Permissions which are not in RICHACE_POSIX_MODE_READ, + * RICHACE_POSIX_MODE_WRITE, or RICHACE_POSIX_MODE_EXEC cannot be represented + * in the file permission bits. Such permissions can still be effective, but + * not for new files or after a chmod(); they must be explicitly enabled in the + * richacl. + */ +int +richacl_masks_to_mode(const struct richacl *acl) +{ + return richacl_mask_to_mode(acl->a_owner_mask) << 6 | + richacl_mask_to_mode(acl->a_group_mask) << 3 | + richacl_mask_to_mode(acl->a_other_mask); +} +EXPORT_SYMBOL_GPL(richacl_masks_to_mode); + +/** + * richacl_mode_to_mask - compute a file mask from the lowest three mode bits + * @mode: mode to convert to richacl permissions + * + * When the file permission bits of a file are set with chmod(), this specifies + * the maximum permissions that processes will get. All permissions beyond + * that will be removed from the file masks, and become ineffective. + */ +unsigned int +richacl_mode_to_mask(umode_t mode) +{ + unsigned int mask = 0; + + if (mode & S_IROTH) + mask |= RICHACE_POSIX_MODE_READ; + if (mode & S_IWOTH) + mask |= RICHACE_POSIX_MODE_WRITE; + if (mode & S_IXOTH) + mask |= RICHACE_POSIX_MODE_EXEC; + + return mask; +} + +/** + * richacl_want_to_mask - convert the iop->permission want argument to a mask + * @want: @want argument of the permission inode operation + * + * When checking for append, @want is (MAY_WRITE | MAY_APPEND). + * + * Richacls use the iop->may_create and iop->may_delete hooks which are used + * for checking if creating and deleting files is allowed. These hooks do not + * use richacl_want_to_mask(), so we do not have to deal with mapping MAY_WRITE + * to RICHACE_ADD_FILE, RICHACE_ADD_SUBDIRECTORY, and RICHACE_DELETE_CHILD + * here. + */ +unsigned int +richacl_want_to_mask(unsigned int want) +{ + unsigned int mask = 0; + + if (want & MAY_READ) + mask |= RICHACE_READ_DATA; + if (want & MAY_DELETE_SELF) + mask |= RICHACE_DELETE; + if (want & MAY_TAKE_OWNERSHIP) + mask |= RICHACE_WRITE_OWNER; + if (want & MAY_CHMOD) + mask |= RICHACE_WRITE_ACL; + if (want & MAY_SET_TIMES) + mask |= RICHACE_WRITE_ATTRIBUTES; + if (want & MAY_EXEC) + mask |= RICHACE_EXECUTE; + /* + * differentiate MAY_WRITE from these request + */ + if (want & (MAY_APPEND | + MAY_CREATE_FILE | MAY_CREATE_DIR | + MAY_DELETE_CHILD)) { + if (want & MAY_APPEND) + mask |= RICHACE_APPEND_DATA; + if (want & MAY_CREATE_FILE) + mask |= RICHACE_ADD_FILE; + if (want & MAY_CREATE_DIR) + mask |= RICHACE_ADD_SUBDIRECTORY; + if (want & MAY_DELETE_CHILD) + mask |= RICHACE_DELETE_CHILD; + } else if (want & MAY_WRITE) + mask |= RICHACE_WRITE_DATA; + return mask; +} +EXPORT_SYMBOL_GPL(richacl_want_to_mask); diff --git a/include/linux/richacl.h b/include/linux/richacl.h index edb8480..9102ef0 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -175,5 +175,8 @@ richace_is_same_identifier(const struct richace *a, const struct richace *b) extern struct richacl *richacl_alloc(int, gfp_t); extern struct richacl *richacl_clone(const struct richacl *, gfp_t); extern void richace_copy(struct richace *, const struct richace *); +extern int richacl_masks_to_mode(const struct richacl *); +extern unsigned int richacl_mode_to_mask(umode_t); +extern unsigned int richacl_want_to_mask(unsigned int); #endif /* __RICHACL_H */ diff --git a/include/uapi/linux/richacl.h b/include/uapi/linux/richacl.h index 08856f8..1ed48ac 100644 --- a/include/uapi/linux/richacl.h +++ b/include/uapi/linux/richacl.h @@ -96,4 +96,48 @@ RICHACE_WRITE_OWNER | \ RICHACE_SYNCHRONIZE ) +/* + * The POSIX permissions are supersets of the following richacl permissions: + * + * - MAY_READ maps to READ_DATA or LIST_DIRECTORY, depending on the type + * of the file system object. + * + * - MAY_WRITE maps to WRITE_DATA or RICHACE_APPEND_DATA for files, and to + * ADD_FILE, RICHACE_ADD_SUBDIRECTORY, or RICHACE_DELETE_CHILD for directories. + * + * - MAY_EXECUTE maps to RICHACE_EXECUTE. + * + * (Some of these richacl permissions have the same bit values.) + */ +#define RICHACE_POSIX_MODE_READ ( \ + RICHACE_READ_DATA | \ + RICHACE_LIST_DIRECTORY) +#define RICHACE_POSIX_MODE_WRITE ( \ + RICHACE_WRITE_DATA | \ + RICHACE_ADD_FILE | \ + RICHACE_APPEND_DATA | \ + RICHACE_ADD_SUBDIRECTORY | \ + RICHACE_DELETE_CHILD) +#define RICHACE_POSIX_MODE_EXEC RICHACE_EXECUTE +#define RICHACE_POSIX_MODE_ALL ( \ + RICHACE_POSIX_MODE_READ | \ + RICHACE_POSIX_MODE_WRITE | \ + RICHACE_POSIX_MODE_EXEC) + +/* + * These permissions are always allowed no matter what the acl says. + */ +#define RICHACE_POSIX_ALWAYS_ALLOWED ( \ + RICHACE_SYNCHRONIZE | \ + RICHACE_READ_ATTRIBUTES | \ + RICHACE_READ_ACL) + +/* + * The owner is implicitly granted these permissions under POSIX. + */ +#define RICHACE_POSIX_OWNER_ALLOWED ( \ + RICHACE_WRITE_ATTRIBUTES | \ + RICHACE_WRITE_OWNER | \ + RICHACE_WRITE_ACL) + #endif /* __UAPI_RICHACL_H */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:22:58 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 91DEC29DF6 for ; Mon, 11 Jan 2016 09:22:58 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 81B98304053 for ; Mon, 11 Jan 2016 07:22:58 -0800 (PST) X-ASG-Debug-ID: 1452525776-04cbb05f7f74a00001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id YNg2fSfYpVzgCXWR (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:22:57 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 8B22F8E38D; Mon, 11 Jan 2016 15:22:56 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbN000775; Mon, 11 Jan 2016 10:22:50 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 08/22] richacl: Compute maximum file masks from an acl Date: Mon, 11 Jan 2016 16:21:39 +0100 X-ASG-Orig-Subj: [PATCH v17 08/22] richacl: Compute maximum file masks from an acl Message-Id: <1452525713-11348-9-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525777 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Compute upper bound owner, group, and other file masks with as few permissions as possible without denying any permissions that the NFSv4 acl in a richacl grants. This algorithm is used when a file inherits an acl at create time and when an acl is set via a mechanism that does not provide file masks (such as setting an acl via nfsd). When user-space sets an acl via setxattr, the extended attribute already includes the file masks. Setting an acl also sets the file mode permission bits: they are determined by the file masks; see richacl_masks_to_mode(). Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/richacl_base.c | 157 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/richacl.h | 1 + 2 files changed, 158 insertions(+) diff --git a/fs/richacl_base.c b/fs/richacl_base.c index a393001..69b806c 100644 --- a/fs/richacl_base.c +++ b/fs/richacl_base.c @@ -183,3 +183,160 @@ richacl_want_to_mask(unsigned int want) return mask; } EXPORT_SYMBOL_GPL(richacl_want_to_mask); + +/* + * Note: functions like richacl_allowed_to_who(), richacl_group_class_allowed(), + * and richacl_compute_max_masks() iterate through the entire acl in reverse + * order as an optimization. + * + * In the standard algorithm, aces are considered in forward order. When a + * process matches an ace, the permissions in the ace are either allowed or + * denied depending on the ace type. Once a permission has been allowed or + * denied, it is no longer considered in further aces. + * + * By iterating through the acl in reverse order, we can compute the same + * result without having to keep track of which permissions have been allowed + * and denied already. + */ + +/** + * richacl_allowed_to_who - permissions allowed to a specific who value + * + * Compute the maximum mask values allowed to a specific who value, taking + * everyone@ aces into account. + */ +static unsigned int richacl_allowed_to_who(struct richacl *acl, + struct richace *who) +{ + struct richace *ace; + unsigned int allowed = 0; + + richacl_for_each_entry_reverse(ace, acl) { + if (richace_is_inherit_only(ace)) + continue; + if (richace_is_same_identifier(ace, who) || + richace_is_everyone(ace)) { + if (richace_is_allow(ace)) + allowed |= ace->e_mask; + else if (richace_is_deny(ace)) + allowed &= ~ace->e_mask; + } + } + return allowed; +} + +/** + * richacl_group_class_allowed - maximum permissions of the group class + * + * Compute the maximum mask values allowed to a process in the group class + * (i.e., a process which is not the owner but is in the owning group or + * matches a user or group acl entry). This includes permissions granted or + * denied by everyone@ aces. + * + * See richacl_compute_max_masks(). + */ +static unsigned int richacl_group_class_allowed(struct richacl *acl) +{ + struct richace *ace; + unsigned int everyone_allowed = 0, group_class_allowed = 0; + int had_group_ace = 0; + + richacl_for_each_entry_reverse(ace, acl) { + if (richace_is_inherit_only(ace) || + richace_is_owner(ace)) + continue; + + if (richace_is_everyone(ace)) { + if (richace_is_allow(ace)) + everyone_allowed |= ace->e_mask; + else if (richace_is_deny(ace)) + everyone_allowed &= ~ace->e_mask; + } else { + group_class_allowed |= + richacl_allowed_to_who(acl, ace); + + if (richace_is_group(ace)) + had_group_ace = 1; + } + } + /* + * If the acl doesn't contain any group@ aces, richacl_allowed_to_who() + * wasn't called for the owning group. We could make that call now, but + * we already know the result (everyone_allowed). + */ + if (!had_group_ace) + group_class_allowed |= everyone_allowed; + return group_class_allowed; +} + +/** + * richacl_compute_max_masks - compute upper bound masks + * + * Computes upper bound owner, group, and other masks so that none of the + * permissions allowed by the acl are disabled. + * + * We don't make assumptions about who the owner is so that the owner can + * change with no effect on the file masks or file mode permission bits; this + * means that we must assume that all entries can match the owner. + */ +void richacl_compute_max_masks(struct richacl *acl) +{ + unsigned int gmask = ~0; + struct richace *ace; + + /* + * @gmask contains all permissions which the group class is ever + * allowed. We use it to avoid adding permissions to the group mask + * from everyone@ allow aces which the group class is always denied + * through other aces. For example, the following acl would otherwise + * result in a group mask of rw: + * + * group@:w::deny + * everyone@:rw::allow + * + * Avoid computing @gmask for acls which do not include any group class + * deny aces: in such acls, the group class is never denied any + * permissions from everyone@ allow aces, and the group class cannot + * have fewer permissions than the other class. + */ + +restart: + acl->a_owner_mask = 0; + acl->a_group_mask = 0; + acl->a_other_mask = 0; + + richacl_for_each_entry_reverse(ace, acl) { + if (richace_is_inherit_only(ace)) + continue; + + if (richace_is_owner(ace)) { + if (richace_is_allow(ace)) + acl->a_owner_mask |= ace->e_mask; + else if (richace_is_deny(ace)) + acl->a_owner_mask &= ~ace->e_mask; + } else if (richace_is_everyone(ace)) { + if (richace_is_allow(ace)) { + acl->a_owner_mask |= ace->e_mask; + acl->a_group_mask |= ace->e_mask & gmask; + acl->a_other_mask |= ace->e_mask; + } else if (richace_is_deny(ace)) { + acl->a_owner_mask &= ~ace->e_mask; + acl->a_group_mask &= ~ace->e_mask; + acl->a_other_mask &= ~ace->e_mask; + } + } else { + if (richace_is_allow(ace)) { + acl->a_owner_mask |= ace->e_mask & gmask; + acl->a_group_mask |= ace->e_mask & gmask; + } else if (richace_is_deny(ace) && gmask == ~0) { + gmask = richacl_group_class_allowed(acl); + if (likely(gmask != ~0)) + /* should always be true */ + goto restart; + } + } + } + + acl->a_flags &= ~(RICHACL_WRITE_THROUGH | RICHACL_MASKED); +} +EXPORT_SYMBOL_GPL(richacl_compute_max_masks); diff --git a/include/linux/richacl.h b/include/linux/richacl.h index 9102ef0..3559b2c 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -178,5 +178,6 @@ extern void richace_copy(struct richace *, const struct richace *); extern int richacl_masks_to_mode(const struct richacl *); extern unsigned int richacl_mode_to_mask(umode_t); extern unsigned int richacl_want_to_mask(unsigned int); +extern void richacl_compute_max_masks(struct richacl *); #endif /* __RICHACL_H */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:05 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 19F1529DF6 for ; Mon, 11 Jan 2016 09:23:05 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id D2C27304059 for ; Mon, 11 Jan 2016 07:23:04 -0800 (PST) X-ASG-Debug-ID: 1452525783-04bdf06f6d74100001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id XSnNu5mKVNID6oMq (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:03 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 3DB40C09FAAA; Mon, 11 Jan 2016 15:23:03 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbO000775; Mon, 11 Jan 2016 10:22:57 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 09/22] richacl: Permission check algorithm Date: Mon, 11 Jan 2016 16:21:40 +0100 X-ASG-Orig-Subj: [PATCH v17 09/22] richacl: Permission check algorithm Message-Id: <1452525713-11348-10-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525783 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 A richacl roughly grants a requested access if the NFSv4 acl in the richacl grants the requested permissions according to the NFSv4 permission check algorithm and the file mask that applies to the process includes the requested permissions. Signed-off-by: Andreas Gruenbacher Reviewed-by: "J. Bruce Fields" --- fs/Makefile | 2 +- fs/richacl_inode.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/richacl.h | 3 + 3 files changed, 153 insertions(+), 1 deletion(-) create mode 100644 fs/richacl_inode.c diff --git a/fs/Makefile b/fs/Makefile index e5994c4..d5b45ca 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -49,7 +49,7 @@ obj-$(CONFIG_SYSCTL) += drop_caches.o obj-$(CONFIG_FHANDLE) += fhandle.o obj-$(CONFIG_FS_RICHACL) += richacl.o -richacl-y := richacl_base.o +richacl-y := richacl_base.o richacl_inode.o obj-y += quota/ diff --git a/fs/richacl_inode.c b/fs/richacl_inode.c new file mode 100644 index 0000000..99b3c93 --- /dev/null +++ b/fs/richacl_inode.c @@ -0,0 +1,149 @@ +/* + * Copyright (C) 2010 Novell, Inc. + * Copyright (C) 2015 Red Hat, Inc. + * Written by Andreas Gruenbacher + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2, or (at your option) any + * later version. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ + +#include +#include +#include +#include +#include + +/** + * richacl_permission - richacl permission check algorithm + * @inode: inode to check + * @acl: rich acl of the inode + * @want: requested access (MAY_* flags) + * + * Checks if the current process is granted @mask flags in @acl. + */ +int +richacl_permission(struct inode *inode, const struct richacl *acl, + int want) +{ + const struct richace *ace; + unsigned int mask = richacl_want_to_mask(want); + unsigned int requested = mask, denied = 0; + int in_owning_group = in_group_p(inode->i_gid); + int in_owner_or_group_class = in_owning_group; + + /* + * A process is + * - in the owner file class if it owns the file, + * - in the group file class if it is in the file's owning group or + * it matches any of the user or group entries, and + * - in the other file class otherwise. + * The file class is only relevant for determining which file mask to + * apply, which only happens for masked acls. + */ + if (acl->a_flags & RICHACL_MASKED) { + if ((acl->a_flags & RICHACL_WRITE_THROUGH) && + uid_eq(current_fsuid(), inode->i_uid)) { + denied = requested & ~acl->a_owner_mask; + goto out; + } + } else { + /* + * When the acl is not masked, there is no need to determine if + * the process is in the group class and we can break out + * earlier of the loop below. + */ + in_owner_or_group_class = 1; + } + + /* + * Check if the acl grants the requested access and determine which + * file class the process is in. + */ + richacl_for_each_entry(ace, acl) { + unsigned int ace_mask = ace->e_mask; + + if (richace_is_inherit_only(ace)) + continue; + if (richace_is_owner(ace)) { + if (!uid_eq(current_fsuid(), inode->i_uid)) + continue; + goto entry_matches_owner; + } else if (richace_is_group(ace)) { + if (!in_owning_group) + continue; + } else if (richace_is_unix_user(ace)) { + if (!uid_eq(current_fsuid(), ace->e_id.uid)) + continue; + if (uid_eq(current_fsuid(), inode->i_uid)) + goto entry_matches_owner; + } else if (richace_is_unix_group(ace)) { + if (!in_group_p(ace->e_id.gid)) + continue; + } else + goto entry_matches_everyone; + + /* + * Apply the group file mask to entries other than owner@ and + * everyone@ or user entries matching the owner. This ensures + * that we grant the same permissions as the acl computed by + * richacl_apply_masks(). + * + * Without this restriction, the following richacl would grant + * rw access to processes which are both the owner and in the + * owning group, but not to other users in the owning group, + * which could not be represented without masks: + * + * owner:rw::mask + * group@:rw::allow + */ + if ((acl->a_flags & RICHACL_MASKED) && richace_is_allow(ace)) + ace_mask &= acl->a_group_mask; + +entry_matches_owner: + /* The process is in the owner or group file class. */ + in_owner_or_group_class = 1; + +entry_matches_everyone: + /* Check which mask flags the ACE allows or denies. */ + if (richace_is_deny(ace)) + denied |= ace_mask & mask; + mask &= ~ace_mask; + + /* + * Keep going until we know which file class + * the process is in. + */ + if (!mask && in_owner_or_group_class) + break; + } + denied |= mask; + + if (acl->a_flags & RICHACL_MASKED) { + /* + * The file class a process is in determines which file mask + * applies. Check if that file mask also grants the requested + * access. + */ + if (uid_eq(current_fsuid(), inode->i_uid)) + denied |= requested & ~acl->a_owner_mask; + else if (in_owner_or_group_class) + denied |= requested & ~acl->a_group_mask; + else { + if (acl->a_flags & RICHACL_WRITE_THROUGH) + denied = requested & ~acl->a_other_mask; + else + denied |= requested & ~acl->a_other_mask; + } + } + +out: + return denied ? -EACCES : 0; +} +EXPORT_SYMBOL_GPL(richacl_permission); diff --git a/include/linux/richacl.h b/include/linux/richacl.h index 3559b2c..1d9f5f7 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -180,4 +180,7 @@ extern unsigned int richacl_mode_to_mask(umode_t); extern unsigned int richacl_want_to_mask(unsigned int); extern void richacl_compute_max_masks(struct richacl *); +/* richacl_inode.c */ +extern int richacl_permission(struct inode *, const struct richacl *, int); + #endif /* __RICHACL_H */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:12 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 2CDDD29DF9 for ; Mon, 11 Jan 2016 09:23:12 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id B60E4AC002 for ; Mon, 11 Jan 2016 07:23:11 -0800 (PST) X-ASG-Debug-ID: 1452525790-04cb6c75dd70890001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id pyQz4f1UPpDFtT8R (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:10 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id F01F7C0BF2C0; Mon, 11 Jan 2016 15:23:09 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbP000775; Mon, 11 Jan 2016 10:23:03 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 10/22] posix_acl: Unexport acl_by_type and make it static Date: Mon, 11 Jan 2016 16:21:41 +0100 X-ASG-Orig-Subj: [PATCH v17 10/22] posix_acl: Unexport acl_by_type and make it static Message-Id: <1452525713-11348-11-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525790 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 acl_by_type(inode, type) returns a pointer to either inode->i_acl or inode->i_default_acl depending on type. This is useful in fs/posix_acl.c, but should never have been visible outside that file. Signed-off-by: Andreas Gruenbacher --- fs/posix_acl.c | 3 +-- include/linux/posix_acl.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/posix_acl.c b/fs/posix_acl.c index 711dd51..55f2445 100644 --- a/fs/posix_acl.c +++ b/fs/posix_acl.c @@ -21,7 +21,7 @@ #include #include -struct posix_acl **acl_by_type(struct inode *inode, int type) +static struct posix_acl **acl_by_type(struct inode *inode, int type) { switch (type) { case ACL_TYPE_ACCESS: @@ -32,7 +32,6 @@ struct posix_acl **acl_by_type(struct inode *inode, int type) BUG(); } } -EXPORT_SYMBOL(acl_by_type); struct posix_acl *get_cached_acl(struct inode *inode, int type) { diff --git a/include/linux/posix_acl.h b/include/linux/posix_acl.h index 3e96a6a..5b5a80c 100644 --- a/include/linux/posix_acl.h +++ b/include/linux/posix_acl.h @@ -99,7 +99,6 @@ extern int posix_acl_create(struct inode *, umode_t *, struct posix_acl **, extern int simple_set_acl(struct inode *, struct posix_acl *, int); extern int simple_acl_create(struct inode *, struct inode *); -struct posix_acl **acl_by_type(struct inode *inode, int type); struct posix_acl *get_cached_acl(struct inode *inode, int type); struct posix_acl *get_cached_acl_rcu(struct inode *inode, int type); void set_cached_acl(struct inode *inode, int type, struct posix_acl *acl); -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:19 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 43EFB29DF6 for ; Mon, 11 Jan 2016 09:23:19 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 33C1B304062 for ; Mon, 11 Jan 2016 07:23:19 -0800 (PST) X-ASG-Debug-ID: 1452525796-04cbb05f7d74a30001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id WRrQ22Cfq3jRoph5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:17 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id A9929A2A1A; Mon, 11 Jan 2016 15:23:16 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbQ000775; Mon, 11 Jan 2016 10:23:10 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 11/22] vfs: Cache base_acl objects in inodes Date: Mon, 11 Jan 2016 16:21:42 +0100 X-ASG-Orig-Subj: [PATCH v17 11/22] vfs: Cache base_acl objects in inodes Message-Id: <1452525713-11348-12-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525797 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 POSIX ACLs and richacls are both objects allocated by kmalloc() with a reference count which are freed by kfree_rcu(). An inode can either cache an access and a default POSIX ACL, or a richacl (richacls do not have default acls). To allow an inode to cache either of the two kinds of acls, introduce a new base_acl type and convert i_acl and i_default_acl to that type. In most cases, the vfs then doesn't care which kind of acl an inode caches (if any). Signed-off-by: Andreas Gruenbacher Reviewed-by: Andreas Dilger --- drivers/staging/lustre/lustre/llite/llite_lib.c | 2 +- fs/f2fs/acl.c | 4 +-- fs/inode.c | 4 +-- fs/jffs2/acl.c | 10 ++++-- fs/posix_acl.c | 41 +++++++++++++------------ fs/richacl_base.c | 4 +-- include/linux/fs.h | 34 ++++++++++++++++++-- include/linux/posix_acl.h | 12 +++----- include/linux/richacl.h | 9 +++--- 9 files changed, 75 insertions(+), 45 deletions(-) diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c index 4a8c759..cfafdcc 100644 --- a/drivers/staging/lustre/lustre/llite/llite_lib.c +++ b/drivers/staging/lustre/lustre/llite/llite_lib.c @@ -1073,7 +1073,7 @@ void ll_clear_inode(struct inode *inode) } #ifdef CONFIG_FS_POSIX_ACL else if (lli->lli_posix_acl) { - LASSERT(atomic_read(&lli->lli_posix_acl->a_refcount) == 1); + LASSERT(base_acl_refcount(&lli->lli_posix_acl->a_base) == 1); LASSERT(lli->lli_remote_perms == NULL); posix_acl_release(lli->lli_posix_acl); lli->lli_posix_acl = NULL; diff --git a/fs/f2fs/acl.c b/fs/f2fs/acl.c index c8f25f7..9646197 100644 --- a/fs/f2fs/acl.c +++ b/fs/f2fs/acl.c @@ -270,7 +270,7 @@ static struct posix_acl *f2fs_acl_clone(const struct posix_acl *acl, sizeof(struct posix_acl_entry); clone = kmemdup(acl, size, flags); if (clone) - atomic_set(&clone->a_refcount, 1); + base_acl_init(&clone->a_base); } return clone; } @@ -282,7 +282,7 @@ static int f2fs_acl_create_masq(struct posix_acl *acl, umode_t *mode_p) umode_t mode = *mode_p; int not_equiv = 0; - /* assert(atomic_read(acl->a_refcount) == 1); */ + /* assert(base_acl_refcount(&acl->a_base) == 1); */ FOREACH_ACL_ENTRY(pa, acl, pe) { switch(pa->e_tag) { diff --git a/fs/inode.c b/fs/inode.c index 1be5f90..1d6d035 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -233,9 +233,9 @@ void __destroy_inode(struct inode *inode) #ifdef CONFIG_FS_POSIX_ACL if (inode->i_acl && inode->i_acl != ACL_NOT_CACHED) - posix_acl_release(inode->i_acl); + base_acl_put(inode->i_acl); if (inode->i_default_acl && inode->i_default_acl != ACL_NOT_CACHED) - posix_acl_release(inode->i_default_acl); + base_acl_put(inode->i_default_acl); #endif this_cpu_dec(nr_inodes); } diff --git a/fs/jffs2/acl.c b/fs/jffs2/acl.c index 2f7a3c0..569cb1b 100644 --- a/fs/jffs2/acl.c +++ b/fs/jffs2/acl.c @@ -294,13 +294,19 @@ int jffs2_init_acl_post(struct inode *inode) int rc; if (inode->i_default_acl) { - rc = __jffs2_set_acl(inode, JFFS2_XPREFIX_ACL_DEFAULT, inode->i_default_acl); + struct posix_acl *default_acl = container_of( + inode->i_default_acl, struct posix_acl, a_base); + + rc = __jffs2_set_acl(inode, JFFS2_XPREFIX_ACL_DEFAULT, default_acl); if (rc) return rc; } if (inode->i_acl) { - rc = __jffs2_set_acl(inode, JFFS2_XPREFIX_ACL_ACCESS, inode->i_acl); + struct posix_acl *acl = container_of( + inode->i_acl, struct posix_acl, a_base); + + rc = __jffs2_set_acl(inode, JFFS2_XPREFIX_ACL_ACCESS, acl); if (rc) return rc; } diff --git a/fs/posix_acl.c b/fs/posix_acl.c index 55f2445..743b6dc 100644 --- a/fs/posix_acl.c +++ b/fs/posix_acl.c @@ -21,7 +21,7 @@ #include #include -static struct posix_acl **acl_by_type(struct inode *inode, int type) +static struct base_acl **acl_by_type(struct inode *inode, int type) { switch (type) { case ACL_TYPE_ACCESS: @@ -35,63 +35,64 @@ static struct posix_acl **acl_by_type(struct inode *inode, int type) struct posix_acl *get_cached_acl(struct inode *inode, int type) { - struct posix_acl **p = acl_by_type(inode, type); - struct posix_acl *acl = ACCESS_ONCE(*p); + struct base_acl **p = acl_by_type(inode, type); + struct base_acl *acl = ACCESS_ONCE(*p); if (acl) { spin_lock(&inode->i_lock); acl = *p; if (acl != ACL_NOT_CACHED) - acl = posix_acl_dup(acl); + base_acl_get(acl); spin_unlock(&inode->i_lock); } - return acl; + return container_of(acl, struct posix_acl, a_base); } EXPORT_SYMBOL(get_cached_acl); struct posix_acl *get_cached_acl_rcu(struct inode *inode, int type) { - return rcu_dereference(*acl_by_type(inode, type)); + struct base_acl *acl = rcu_dereference(*acl_by_type(inode, type)); + return container_of(acl, struct posix_acl, a_base); } EXPORT_SYMBOL(get_cached_acl_rcu); void set_cached_acl(struct inode *inode, int type, struct posix_acl *acl) { - struct posix_acl **p = acl_by_type(inode, type); - struct posix_acl *old; + struct base_acl **p = acl_by_type(inode, type); + struct base_acl *old; spin_lock(&inode->i_lock); old = *p; - rcu_assign_pointer(*p, posix_acl_dup(acl)); + rcu_assign_pointer(*p, &posix_acl_dup(acl)->a_base); spin_unlock(&inode->i_lock); if (old != ACL_NOT_CACHED) - posix_acl_release(old); + base_acl_put(old); } EXPORT_SYMBOL(set_cached_acl); void forget_cached_acl(struct inode *inode, int type) { - struct posix_acl **p = acl_by_type(inode, type); - struct posix_acl *old; + struct base_acl **p = acl_by_type(inode, type); + struct base_acl *old; spin_lock(&inode->i_lock); old = *p; *p = ACL_NOT_CACHED; spin_unlock(&inode->i_lock); if (old != ACL_NOT_CACHED) - posix_acl_release(old); + base_acl_put(old); } EXPORT_SYMBOL(forget_cached_acl); void forget_all_cached_acls(struct inode *inode) { - struct posix_acl *old_access, *old_default; + struct base_acl *old_access, *old_default; spin_lock(&inode->i_lock); old_access = inode->i_acl; old_default = inode->i_default_acl; inode->i_acl = inode->i_default_acl = ACL_NOT_CACHED; spin_unlock(&inode->i_lock); if (old_access != ACL_NOT_CACHED) - posix_acl_release(old_access); + base_acl_put(old_access); if (old_default != ACL_NOT_CACHED) - posix_acl_release(old_default); + base_acl_put(old_default); } EXPORT_SYMBOL(forget_all_cached_acls); @@ -128,7 +129,7 @@ EXPORT_SYMBOL(get_acl); void posix_acl_init(struct posix_acl *acl, int count) { - atomic_set(&acl->a_refcount, 1); + base_acl_init(&acl->a_base); acl->a_count = count; } EXPORT_SYMBOL(posix_acl_init); @@ -161,7 +162,7 @@ posix_acl_clone(const struct posix_acl *acl, gfp_t flags) sizeof(struct posix_acl_entry); clone = kmemdup(acl, size, flags); if (clone) - atomic_set(&clone->a_refcount, 1); + base_acl_init(&clone->a_base); } return clone; } @@ -383,7 +384,7 @@ static int posix_acl_create_masq(struct posix_acl *acl, umode_t *mode_p) umode_t mode = *mode_p; int not_equiv = 0; - /* assert(atomic_read(acl->a_refcount) == 1); */ + /* assert(base_acl_refcount(&acl->a_base) == 1); */ FOREACH_ACL_ENTRY(pa, acl, pe) { switch(pa->e_tag) { @@ -438,7 +439,7 @@ static int __posix_acl_chmod_masq(struct posix_acl *acl, umode_t mode) struct posix_acl_entry *group_obj = NULL, *mask_obj = NULL; struct posix_acl_entry *pa, *pe; - /* assert(atomic_read(acl->a_refcount) == 1); */ + /* assert(base_acl_refcount(&acl->a_base) == 1); */ FOREACH_ACL_ENTRY(pa, acl, pe) { switch(pa->e_tag) { diff --git a/fs/richacl_base.c b/fs/richacl_base.c index 69b806c..5826842 100644 --- a/fs/richacl_base.c +++ b/fs/richacl_base.c @@ -33,7 +33,7 @@ richacl_alloc(int count, gfp_t gfp) struct richacl *acl = kzalloc(size, gfp); if (acl) { - atomic_set(&acl->a_refcount, 1); + base_acl_init(&acl->a_base); acl->a_count = count; } return acl; @@ -52,7 +52,7 @@ richacl_clone(const struct richacl *acl, gfp_t gfp) if (dup) { memcpy(dup, acl, size); - atomic_set(&dup->a_refcount, 1); + base_acl_init(&dup->a_base); } return dup; } diff --git a/include/linux/fs.h b/include/linux/fs.h index e9e1139..c7f254b 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -576,6 +576,12 @@ static inline void mapping_allow_writable(struct address_space *mapping) #define i_size_ordered_init(inode) do { } while (0) #endif +struct base_acl { + union { + atomic_t ba_refcount; + struct rcu_head ba_rcu; + }; +}; struct posix_acl; #define ACL_NOT_CACHED ((void *)(-1)) @@ -595,9 +601,9 @@ struct inode { kgid_t i_gid; unsigned int i_flags; -#ifdef CONFIG_FS_POSIX_ACL - struct posix_acl *i_acl; - struct posix_acl *i_default_acl; +#if defined(CONFIG_FS_POSIX_ACL) + struct base_acl *i_acl; + struct base_acl *i_default_acl; #endif const struct inode_operations *i_op; @@ -3045,4 +3051,26 @@ static inline bool dir_relax(struct inode *inode) extern bool path_noexec(const struct path *path); +static inline void base_acl_get(struct base_acl *acl) +{ + if (acl) + atomic_inc(&acl->ba_refcount); +} + +static inline void base_acl_put(struct base_acl *acl) +{ + if (acl && atomic_dec_and_test(&acl->ba_refcount)) + kfree_rcu(acl, ba_rcu); +} + +static inline void base_acl_init(struct base_acl *acl) +{ + atomic_set(&acl->ba_refcount, 1); +} + +static inline int base_acl_refcount(struct base_acl *acl) +{ + return atomic_read(&acl->ba_refcount); +} + #endif /* _LINUX_FS_H */ diff --git a/include/linux/posix_acl.h b/include/linux/posix_acl.h index 5b5a80c..cef5428 100644 --- a/include/linux/posix_acl.h +++ b/include/linux/posix_acl.h @@ -43,10 +43,7 @@ struct posix_acl_entry { }; struct posix_acl { - union { - atomic_t a_refcount; - struct rcu_head a_rcu; - }; + struct base_acl a_base; /* must be first, see posix_acl_release() */ unsigned int a_count; struct posix_acl_entry a_entries[0]; }; @@ -61,8 +58,7 @@ struct posix_acl { static inline struct posix_acl * posix_acl_dup(struct posix_acl *acl) { - if (acl) - atomic_inc(&acl->a_refcount); + base_acl_get(&acl->a_base); return acl; } @@ -72,8 +68,8 @@ posix_acl_dup(struct posix_acl *acl) static inline void posix_acl_release(struct posix_acl *acl) { - if (acl && atomic_dec_and_test(&acl->a_refcount)) - kfree_rcu(acl, a_rcu); + BUILD_BUG_ON(offsetof(struct posix_acl, a_base) != 0); + base_acl_put(&acl->a_base); } diff --git a/include/linux/richacl.h b/include/linux/richacl.h index 1d9f5f7..7628fad 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -31,7 +31,7 @@ struct richace { }; struct richacl { - atomic_t a_refcount; + struct base_acl a_base; /* must be first, see richacl_put() */ unsigned int a_owner_mask; unsigned int a_group_mask; unsigned int a_other_mask; @@ -56,8 +56,7 @@ struct richacl { static inline struct richacl * richacl_get(struct richacl *acl) { - if (acl) - atomic_inc(&acl->a_refcount); + base_acl_get(&acl->a_base); return acl; } @@ -67,8 +66,8 @@ richacl_get(struct richacl *acl) static inline void richacl_put(struct richacl *acl) { - if (acl && atomic_dec_and_test(&acl->a_refcount)) - kfree(acl); + BUILD_BUG_ON(offsetof(struct richacl, a_base) != 0); + base_acl_put(&acl->a_base); } /** -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:24 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id E9F0229E03 for ; Mon, 11 Jan 2016 09:23:24 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id D9DDE8F8040 for ; Mon, 11 Jan 2016 07:23:24 -0800 (PST) X-ASG-Debug-ID: 1452525803-04cb6c75dc708d0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id IXprVbskPwb0cKhM (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:24 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 604B7C09FAA9; Mon, 11 Jan 2016 15:23:23 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbR000775; Mon, 11 Jan 2016 10:23:17 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 12/22] vfs: Add get_richacl and set_richacl inode operations Date: Mon, 11 Jan 2016 16:21:43 +0100 X-ASG-Orig-Subj: [PATCH v17 12/22] vfs: Add get_richacl and set_richacl inode operations Message-Id: <1452525713-11348-13-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525803 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 These operations are similar to the get_acl and set_acl operations for POSIX ACLs. The distinction between access and default ACLs doesn't exist for richacls. Signed-off-by: Andreas Gruenbacher --- include/linux/fs.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index c7f254b..4847417 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1649,6 +1649,7 @@ struct inode_operations { const char * (*follow_link) (struct dentry *, void **); int (*permission) (struct inode *, int); struct posix_acl * (*get_acl)(struct inode *, int); + struct richacl * (*get_richacl)(struct inode *); int (*readlink) (struct dentry *, char __user *,int); void (*put_link) (struct inode *, void *); @@ -1678,6 +1679,7 @@ struct inode_operations { umode_t create_mode, int *opened); int (*tmpfile) (struct inode *, struct dentry *, umode_t); int (*set_acl)(struct inode *, struct posix_acl *, int); + int (*set_richacl)(struct inode *, struct richacl *); } ____cacheline_aligned; ssize_t rw_copy_check_uvector(int type, const struct iovec __user * uvector, -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:32 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 1054029DFE for ; Mon, 11 Jan 2016 09:23:32 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id F3F59304062 for ; Mon, 11 Jan 2016 07:23:31 -0800 (PST) X-ASG-Debug-ID: 1452525810-04cbb05f7e74a50001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Bw6smZVA3whce6S6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:30 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 121341392E; Mon, 11 Jan 2016 15:23:30 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbS000775; Mon, 11 Jan 2016 10:23:23 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 13/22] vfs: Cache richacl in struct inode Date: Mon, 11 Jan 2016 16:21:44 +0100 X-ASG-Orig-Subj: [PATCH v17 13/22] vfs: Cache richacl in struct inode Message-Id: <1452525713-11348-14-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525810 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Cache richacls in struct inode so that this doesn't have to be done individually in each filesystem. This is similar to POSIX ACLs. Signed-off-by: Andreas Gruenbacher --- fs/inode.c | 11 +++++-- fs/posix_acl.c | 2 +- fs/richacl_inode.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/fs.h | 5 +++- include/linux/richacl.h | 6 ++++ 5 files changed, 96 insertions(+), 5 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 1d6d035..3459bc9 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -174,8 +174,11 @@ int inode_init_always(struct super_block *sb, struct inode *inode) inode->i_private = NULL; inode->i_mapping = mapping; INIT_HLIST_HEAD(&inode->i_dentry); /* buggered by rcu freeing */ -#ifdef CONFIG_FS_POSIX_ACL - inode->i_acl = inode->i_default_acl = ACL_NOT_CACHED; +#if defined(CONFIG_FS_POSIX_ACL) || defined(CONFIG_FS_RICHACL) + inode->i_acl = ACL_NOT_CACHED; +# if defined(CONFIG_FS_POSIX_ACL) + inode->i_default_acl = ACL_NOT_CACHED; +# endif #endif #ifdef CONFIG_FSNOTIFY @@ -231,11 +234,13 @@ void __destroy_inode(struct inode *inode) atomic_long_dec(&inode->i_sb->s_remove_count); } -#ifdef CONFIG_FS_POSIX_ACL +#if defined(CONFIG_FS_POSIX_ACL) || defined(CONFIG_FS_RICHACL) if (inode->i_acl && inode->i_acl != ACL_NOT_CACHED) base_acl_put(inode->i_acl); +# if defined(CONFIG_FS_POSIX_ACL) if (inode->i_default_acl && inode->i_default_acl != ACL_NOT_CACHED) base_acl_put(inode->i_default_acl); +# endif #endif this_cpu_dec(nr_inodes); } diff --git a/fs/posix_acl.c b/fs/posix_acl.c index 743b6dc..f24646e 100644 --- a/fs/posix_acl.c +++ b/fs/posix_acl.c @@ -37,7 +37,7 @@ struct posix_acl *get_cached_acl(struct inode *inode, int type) { struct base_acl **p = acl_by_type(inode, type); struct base_acl *acl = ACCESS_ONCE(*p); - if (acl) { + if (acl && IS_POSIXACL(inode)) { spin_lock(&inode->i_lock); acl = *p; if (acl != ACL_NOT_CACHED) diff --git a/fs/richacl_inode.c b/fs/richacl_inode.c index 99b3c93..52c1595 100644 --- a/fs/richacl_inode.c +++ b/fs/richacl_inode.c @@ -20,6 +20,83 @@ #include #include +struct richacl *get_cached_richacl(struct inode *inode) +{ + struct base_acl *acl; + + acl = ACCESS_ONCE(inode->i_acl); + if (acl && IS_RICHACL(inode)) { + spin_lock(&inode->i_lock); + acl = inode->i_acl; + if (acl != ACL_NOT_CACHED) + base_acl_get(acl); + spin_unlock(&inode->i_lock); + } + return container_of(acl, struct richacl, a_base); +} +EXPORT_SYMBOL_GPL(get_cached_richacl); + +struct richacl *get_cached_richacl_rcu(struct inode *inode) +{ + struct base_acl *acl = rcu_dereference(inode->i_acl); + + return container_of(acl, struct richacl, a_base); +} +EXPORT_SYMBOL_GPL(get_cached_richacl_rcu); + +void set_cached_richacl(struct inode *inode, struct richacl *acl) +{ + struct base_acl *old = NULL; + + spin_lock(&inode->i_lock); + old = inode->i_acl; + rcu_assign_pointer(inode->i_acl, &richacl_get(acl)->a_base); + spin_unlock(&inode->i_lock); + if (old != ACL_NOT_CACHED) + base_acl_put(old); +} +EXPORT_SYMBOL_GPL(set_cached_richacl); + +void forget_cached_richacl(struct inode *inode) +{ + struct base_acl *old = NULL; + + spin_lock(&inode->i_lock); + old = inode->i_acl; + inode->i_acl = ACL_NOT_CACHED; + spin_unlock(&inode->i_lock); + if (old != ACL_NOT_CACHED) + base_acl_put(old); +} +EXPORT_SYMBOL_GPL(forget_cached_richacl); + +struct richacl *get_richacl(struct inode *inode) +{ + struct richacl *acl; + + acl = get_cached_richacl(inode); + if (acl != ACL_NOT_CACHED) + return acl; + + if (!IS_RICHACL(inode)) + return NULL; + + /* + * A filesystem can force a ACL callback by just never filling the + * ACL cache. But normally you'd fill the cache either at inode + * instantiation time, or on the first ->get_richacl call. + * + * If the filesystem doesn't have a get_richacl() function at all, + * we'll just create the negative cache entry. + */ + if (!inode->i_op->get_richacl) { + set_cached_richacl(inode, NULL); + return NULL; + } + return inode->i_op->get_richacl(inode); +} +EXPORT_SYMBOL_GPL(get_richacl); + /** * richacl_permission - richacl permission check algorithm * @inode: inode to check diff --git a/include/linux/fs.h b/include/linux/fs.h index 4847417..76d71b2 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -583,6 +583,7 @@ struct base_acl { }; }; struct posix_acl; +struct richacl; #define ACL_NOT_CACHED ((void *)(-1)) #define IOP_FASTPERM 0x0001 @@ -601,9 +602,11 @@ struct inode { kgid_t i_gid; unsigned int i_flags; -#if defined(CONFIG_FS_POSIX_ACL) +#if defined(CONFIG_FS_POSIX_ACL) || defined(CONFIG_FS_RICHACL) struct base_acl *i_acl; +# if defined(CONFIG_FS_POSIX_ACL) struct base_acl *i_default_acl; +# endif #endif const struct inode_operations *i_op; diff --git a/include/linux/richacl.h b/include/linux/richacl.h index 7628fad..7bf912b6 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -70,6 +70,12 @@ richacl_put(struct richacl *acl) base_acl_put(&acl->a_base); } +extern struct richacl *get_cached_richacl(struct inode *); +extern struct richacl *get_cached_richacl_rcu(struct inode *); +extern void set_cached_richacl(struct inode *, struct richacl *); +extern void forget_cached_richacl(struct inode *); +extern struct richacl *get_richacl(struct inode *); + /** * richace_is_owner - check if @ace is an OWNER@ entry */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:38 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 8C2D529E0C for ; Mon, 11 Jan 2016 09:23:38 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 6DD6530405F for ; Mon, 11 Jan 2016 07:23:38 -0800 (PST) X-ASG-Debug-ID: 1452525816-04cbb05f7d74a60001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 0YebACIR3TnOX2PL (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:37 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id B4E8B13933; Mon, 11 Jan 2016 15:23:36 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbT000775; Mon, 11 Jan 2016 10:23:30 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 14/22] richacl: Update the file masks in chmod() Date: Mon, 11 Jan 2016 16:21:45 +0100 X-ASG-Orig-Subj: [PATCH v17 14/22] richacl: Update the file masks in chmod() Message-Id: <1452525713-11348-15-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525817 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Doing a chmod() sets the file mode, which includes the file permission bits. When a file has a richacl, the permissions that the richacl grants need to be limited to what the new file permission bits allow. This is done by setting the file masks in the richacl to what the file permission bits map to. The richacl access check algorithm takes the file masks into account, which ensures that the richacl cannot grant too many permissions. It is possible to explicitly add permissions to the file masks which go beyond what the file permission bits can grant (like the RICHACE_WRITE_ACL permission). The POSIX.1 standard calls this an alternate file access control mechanism. A subsequent chmod() would ensure that those permissions are disabled again. Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/richacl_base.c | 42 ++++++++++++++++++++++++++++++++++++++++++ fs/richacl_inode.c | 30 ++++++++++++++++++++++++++++++ include/linux/richacl.h | 2 ++ 3 files changed, 74 insertions(+) diff --git a/fs/richacl_base.c b/fs/richacl_base.c index 5826842..e4dd779 100644 --- a/fs/richacl_base.c +++ b/fs/richacl_base.c @@ -340,3 +340,45 @@ restart: acl->a_flags &= ~(RICHACL_WRITE_THROUGH | RICHACL_MASKED); } EXPORT_SYMBOL_GPL(richacl_compute_max_masks); + +/** + * __richacl_chmod - update the file masks to reflect the new mode + * @acl: access control list + * @mode: new file permission bits including the file type + * + * Return a copy of @acl where the file masks have been replaced by the file + * masks corresponding to the file permission bits in @mode, or returns @acl + * itself if the file masks are already up to date. Takes over a reference + * to @acl. + */ +struct richacl * +__richacl_chmod(struct richacl *acl, umode_t mode) +{ + unsigned int x = S_ISDIR(mode) ? 0 : RICHACE_DELETE_CHILD; + unsigned int owner_mask, group_mask, other_mask; + struct richacl *clone; + + owner_mask = richacl_mode_to_mask(mode >> 6) & ~x; + group_mask = richacl_mode_to_mask(mode >> 3) & ~x; + other_mask = richacl_mode_to_mask(mode) & ~x; + + if (acl->a_owner_mask == owner_mask && + acl->a_group_mask == group_mask && + acl->a_other_mask == other_mask && + (acl->a_flags & RICHACL_MASKED) && + (acl->a_flags & RICHACL_WRITE_THROUGH)) + return acl; + + clone = richacl_clone(acl, GFP_KERNEL); + richacl_put(acl); + if (!clone) + return ERR_PTR(-ENOMEM); + + clone->a_flags |= (RICHACL_WRITE_THROUGH | RICHACL_MASKED); + clone->a_owner_mask = owner_mask; + clone->a_group_mask = group_mask; + clone->a_other_mask = other_mask; + + return clone; +} +EXPORT_SYMBOL_GPL(__richacl_chmod); diff --git a/fs/richacl_inode.c b/fs/richacl_inode.c index 52c1595..e329826 100644 --- a/fs/richacl_inode.c +++ b/fs/richacl_inode.c @@ -224,3 +224,33 @@ out: return denied ? -EACCES : 0; } EXPORT_SYMBOL_GPL(richacl_permission); + +/** + * richacl_chmod - filesystem chmod helper + * @inode: inode whose file permission bits to change + * @mode: new file permission bits including the file type + * + * Helper for filesystems to use to perform a chmod on the richacl of an inode. + */ +int +richacl_chmod(struct inode *inode, umode_t mode) +{ + struct richacl *acl; + int retval; + + if (S_ISLNK(mode)) + return -EOPNOTSUPP; + if (!inode->i_op->set_richacl) + return -EOPNOTSUPP; + acl = get_richacl(inode); + if (IS_ERR_OR_NULL(acl)) + return PTR_ERR(acl); + acl = __richacl_chmod(acl, mode); + if (IS_ERR(acl)) + return PTR_ERR(acl); + retval = inode->i_op->set_richacl(inode, acl); + richacl_put(acl); + + return retval; +} +EXPORT_SYMBOL(richacl_chmod); diff --git a/include/linux/richacl.h b/include/linux/richacl.h index 7bf912b6..a2d5600 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -184,8 +184,10 @@ extern int richacl_masks_to_mode(const struct richacl *); extern unsigned int richacl_mode_to_mask(umode_t); extern unsigned int richacl_want_to_mask(unsigned int); extern void richacl_compute_max_masks(struct richacl *); +extern struct richacl *__richacl_chmod(struct richacl *, umode_t); /* richacl_inode.c */ extern int richacl_permission(struct inode *, const struct richacl *, int); +extern int richacl_chmod(struct inode *, umode_t); #endif /* __RICHACL_H */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:50 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 4913029E0C for ; Mon, 11 Jan 2016 09:23:50 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 297028F8040 for ; Mon, 11 Jan 2016 07:23:50 -0800 (PST) X-ASG-Debug-ID: 1452525823-04cbb05f7d74a70001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id bssJp8RXAk3sdeXx (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:44 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 81C268EA39; Mon, 11 Jan 2016 15:23:43 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbU000775; Mon, 11 Jan 2016 10:23:37 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 15/22] richacl: Check if an acl is equivalent to a file mode Date: Mon, 11 Jan 2016 16:21:46 +0100 X-ASG-Orig-Subj: [PATCH v17 15/22] richacl: Check if an acl is equivalent to a file mode Message-Id: <1452525713-11348-16-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525824 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 ACLs are considered equivalent to file modes if they only consist of owner@, group@, and everyone@ entries, the owner@ permissions do not depend on whether the owner is a member in the owning group, and no inheritance flags are set. This test is used to avoid storing richacls if the acl can be computed from the file permission bits. Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/richacl_base.c | 104 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/richacl.h | 1 + 2 files changed, 105 insertions(+) diff --git a/fs/richacl_base.c b/fs/richacl_base.c index e4dd779..74e5cb5 100644 --- a/fs/richacl_base.c +++ b/fs/richacl_base.c @@ -382,3 +382,107 @@ __richacl_chmod(struct richacl *acl, umode_t mode) return clone; } EXPORT_SYMBOL_GPL(__richacl_chmod); + +/** + * richacl_equiv_mode - compute the mode equivalent of @acl + * + * An acl is considered equivalent to a file mode if it only consists of + * owner@, group@, and everyone@ entries and the owner@ permissions do not + * depend on whether the owner is a member in the owning group. + */ +int +richacl_equiv_mode(const struct richacl *acl, umode_t *mode_p) +{ + umode_t mode = *mode_p; + + /* + * The RICHACE_DELETE_CHILD flag is meaningless for non-directories, so + * we ignore it. + */ + unsigned int x = S_ISDIR(mode) ? 0 : RICHACE_DELETE_CHILD; + struct { + unsigned int allowed; + unsigned int defined; /* allowed or denied */ + } owner = { + .defined = RICHACE_POSIX_ALWAYS_ALLOWED | + RICHACE_POSIX_OWNER_ALLOWED | x, + }, group = { + .defined = RICHACE_POSIX_ALWAYS_ALLOWED | x, + }, everyone = { + .defined = RICHACE_POSIX_ALWAYS_ALLOWED | x, + }; + const struct richace *ace; + + if (acl->a_flags & ~(RICHACL_WRITE_THROUGH | RICHACL_MASKED)) + return -1; + + richacl_for_each_entry(ace, acl) { + if (ace->e_flags & ~RICHACE_SPECIAL_WHO) + return -1; + + if (richace_is_owner(ace) || richace_is_everyone(ace)) { + x = ace->e_mask & ~owner.defined; + if (richace_is_allow(ace)) { + unsigned int group_denied = + group.defined & ~group.allowed; + + if (x & group_denied) + return -1; + owner.allowed |= x; + } else /* if (richace_is_deny(ace)) */ { + if (x & group.allowed) + return -1; + } + owner.defined |= x; + + if (richace_is_everyone(ace)) { + x = ace->e_mask; + if (richace_is_allow(ace)) { + group.allowed |= + x & ~group.defined; + everyone.allowed |= + x & ~everyone.defined; + } + group.defined |= x; + everyone.defined |= x; + } + } else if (richace_is_group(ace)) { + x = ace->e_mask & ~group.defined; + if (richace_is_allow(ace)) + group.allowed |= x; + group.defined |= x; + } else + return -1; + } + + if (group.allowed & ~owner.defined) + return -1; + + if (acl->a_flags & RICHACL_MASKED) { + if (acl->a_flags & RICHACL_WRITE_THROUGH) { + owner.allowed = acl->a_owner_mask; + everyone.allowed = acl->a_other_mask; + } else { + owner.allowed &= acl->a_owner_mask; + everyone.allowed &= acl->a_other_mask; + } + group.allowed &= acl->a_group_mask; + } + + mode = (mode & ~S_IRWXUGO) | + (richacl_mask_to_mode(owner.allowed) << 6) | + (richacl_mask_to_mode(group.allowed) << 3) | + richacl_mask_to_mode(everyone.allowed); + + /* Mask flags we can ignore */ + x = S_ISDIR(mode) ? 0 : RICHACE_DELETE_CHILD; + + if (((richacl_mode_to_mask(mode >> 6) ^ owner.allowed) & ~x) || + ((richacl_mode_to_mask(mode >> 3) ^ group.allowed) & ~x) || + ((richacl_mode_to_mask(mode) ^ everyone.allowed) & ~x)) + return -1; + + *mode_p = mode; + return 0; +} +EXPORT_SYMBOL_GPL(richacl_equiv_mode); diff --git a/include/linux/richacl.h b/include/linux/richacl.h index a2d5600..b7128bf 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -185,6 +185,7 @@ extern unsigned int richacl_mode_to_mask(umode_t); extern unsigned int richacl_want_to_mask(unsigned int); extern void richacl_compute_max_masks(struct richacl *); extern struct richacl *__richacl_chmod(struct richacl *, umode_t); +extern int richacl_equiv_mode(const struct richacl *, umode_t *); /* richacl_inode.c */ extern int richacl_permission(struct inode *, const struct richacl *, int); -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:52 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 297F529E22 for ; Mon, 11 Jan 2016 09:23:52 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id F0F5330405F for ; Mon, 11 Jan 2016 07:23:51 -0800 (PST) X-ASG-Debug-ID: 1452525830-04bdf06f6e74180001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id CFaDmbMeUz3q3LNb (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:50 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 1BFBBC1C7B8A; Mon, 11 Jan 2016 15:23:50 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbV000775; Mon, 11 Jan 2016 10:23:44 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 16/22] richacl: Create-time inheritance Date: Mon, 11 Jan 2016 16:21:47 +0100 X-ASG-Orig-Subj: [PATCH v17 16/22] richacl: Create-time inheritance Message-Id: <1452525713-11348-17-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525830 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 When a new file is created, it can inherit an acl from its parent directory; this is similar to how default acls work in POSIX (draft) ACLs. As with POSIX ACLs, if a file inherits an acl from its parent directory, the intersection between the create mode and the permissions granted by the inherited acl determines the file masks and file permission bits, and the umask is ignored. Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/richacl_base.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++ fs/richacl_inode.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/richacl.h | 2 ++ 3 files changed, 140 insertions(+) diff --git a/fs/richacl_base.c b/fs/richacl_base.c index 74e5cb5..3753216 100644 --- a/fs/richacl_base.c +++ b/fs/richacl_base.c @@ -486,3 +486,71 @@ richacl_equiv_mode(const struct richacl *acl, umode_t *mode_p) return 0; } EXPORT_SYMBOL_GPL(richacl_equiv_mode); + +/** + * richacl_inherit - compute the inherited acl of a new file + * @dir_acl: acl of the containing directory + * @isdir: inherit by a directory or non-directory? + * + * A directory can have acl entries which files and/or directories created + * inside the directory will inherit. This function computes the acl for such + * a new file. If there is no inheritable acl, it will return %NULL. + */ +struct richacl * +richacl_inherit(const struct richacl *dir_acl, int isdir) +{ + const struct richace *dir_ace; + struct richacl *acl = NULL; + struct richace *ace; + int count = 0; + + if (isdir) { + richacl_for_each_entry(dir_ace, dir_acl) { + if (!richace_is_inheritable(dir_ace)) + continue; + count++; + } + if (!count) + return NULL; + acl = richacl_alloc(count, GFP_KERNEL); + if (!acl) + return ERR_PTR(-ENOMEM); + ace = acl->a_entries; + richacl_for_each_entry(dir_ace, dir_acl) { + if (!richace_is_inheritable(dir_ace)) + continue; + richace_copy(ace, dir_ace); + if (dir_ace->e_flags & RICHACE_NO_PROPAGATE_INHERIT_ACE) + ace->e_flags &= ~RICHACE_INHERITANCE_FLAGS; + else if (!(dir_ace->e_flags & RICHACE_DIRECTORY_INHERIT_ACE)) + ace->e_flags |= RICHACE_INHERIT_ONLY_ACE; + ace++; + } + } else { + richacl_for_each_entry(dir_ace, dir_acl) { + if (!(dir_ace->e_flags & RICHACE_FILE_INHERIT_ACE)) + continue; + count++; + } + if (!count) + return NULL; + acl = richacl_alloc(count, GFP_KERNEL); + if (!acl) + return ERR_PTR(-ENOMEM); + ace = acl->a_entries; + richacl_for_each_entry(dir_ace, dir_acl) { + if (!(dir_ace->e_flags & RICHACE_FILE_INHERIT_ACE)) + continue; + richace_copy(ace, dir_ace); + ace->e_flags &= ~RICHACE_INHERITANCE_FLAGS; + /* + * RICHACE_DELETE_CHILD is meaningless for + * non-directories, so clear it. + */ + ace->e_mask &= ~RICHACE_DELETE_CHILD; + ace++; + } + } + + return acl; +} diff --git a/fs/richacl_inode.c b/fs/richacl_inode.c index e329826..ec3d2c8 100644 --- a/fs/richacl_inode.c +++ b/fs/richacl_inode.c @@ -254,3 +254,73 @@ richacl_chmod(struct inode *inode, umode_t mode) return retval; } EXPORT_SYMBOL(richacl_chmod); + +/* + * richacl_inherit_inode - compute inherited acl and file mode + * @dir_acl: acl of the containing directory + * @mode_p: mode of the new inode + * + * The file permission bits in @mode_p must be set to the create mode by the + * caller. + * + * If there is an inheritable acl, the maximum permissions that the acl grants + * are computed and the file masks of the new acl are set accordingly. + */ +static struct richacl * +richacl_inherit_inode(const struct richacl *dir_acl, umode_t *mode_p) +{ + struct richacl *acl; + umode_t mode = *mode_p; + + acl = richacl_inherit(dir_acl, S_ISDIR(mode)); + if (acl) { + if (richacl_equiv_mode(acl, &mode) == 0) { + *mode_p &= mode; + richacl_put(acl); + acl = NULL; + } else { + richacl_compute_max_masks(acl); + /* + * Ensure that the acl will not grant any permissions + * beyond the create mode. + */ + acl->a_flags |= RICHACL_MASKED; + acl->a_owner_mask &= + richacl_mode_to_mask(mode >> 6); + acl->a_group_mask &= + richacl_mode_to_mask(mode >> 3); + acl->a_other_mask &= + richacl_mode_to_mask(mode); + } + } else + *mode_p &= ~current_umask(); + + return acl; +} + +/** + * richacl_create - filesystem create helper + * @mode_p: mode of the new inode + * @dir: containing directory + * + * Compute the inherited acl for a new inode. If there is no acl to inherit, + * apply the umask. Use when creating a new inode on a richacl enabled file + * system. + */ +struct richacl *richacl_create(umode_t *mode_p, struct inode *dir) +{ + struct richacl *dir_acl, *acl = NULL; + + if (S_ISLNK(*mode_p)) + return NULL; + dir_acl = get_richacl(dir); + if (dir_acl) { + if (IS_ERR(dir_acl)) + return dir_acl; + acl = richacl_inherit_inode(dir_acl, mode_p); + richacl_put(dir_acl); + } else + *mode_p &= ~current_umask(); + return acl; +} +EXPORT_SYMBOL_GPL(richacl_create); diff --git a/include/linux/richacl.h b/include/linux/richacl.h index b7128bf..c8fae91a 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -186,9 +186,11 @@ extern unsigned int richacl_want_to_mask(unsigned int); extern void richacl_compute_max_masks(struct richacl *); extern struct richacl *__richacl_chmod(struct richacl *, umode_t); extern int richacl_equiv_mode(const struct richacl *, umode_t *); +extern struct richacl *richacl_inherit(const struct richacl *, int); /* richacl_inode.c */ extern int richacl_permission(struct inode *, const struct richacl *, int); extern int richacl_chmod(struct inode *, umode_t); +extern struct richacl *richacl_create(umode_t *, struct inode *); #endif /* __RICHACL_H */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:23:59 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9C99829E0C for ; Mon, 11 Jan 2016 09:23:59 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 234AFAC004 for ; Mon, 11 Jan 2016 07:23:59 -0800 (PST) X-ASG-Debug-ID: 1452525837-04cb6c75db70910001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id C5BbNkEJ2Kh0EUcs (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:23:57 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id C62C28F281; Mon, 11 Jan 2016 15:23:56 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbW000775; Mon, 11 Jan 2016 10:23:50 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 17/22] richacl: Automatic Inheritance Date: Mon, 11 Jan 2016 16:21:48 +0100 X-ASG-Orig-Subj: [PATCH v17 17/22] richacl: Automatic Inheritance Message-Id: <1452525713-11348-18-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525837 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Automatic Inheritance (AI) allows changes to the acl of a directory to propagate down to children. This is mostly implemented in user space: when a process changes the permissions of a directory and Automatic Inheritance is enabled for that directory, the process must propagate those changes to all children, recursively. The kernel enables this by keeping track of which permissions have been inherited at create time. In addition, it makes sure that permission propagation is turned off when the permissions are set explicitly (for example, upon create or chmod). Automatic Inheritance works as follows: - When the RICHACL_AUTO_INHERIT flag in the acl of a file or directory is not set, the file or directory is not affected by AI. - When the RICHACL_AUTO_INHERIT flag in the acl of a directory is set and a file or subdirectory is created in that directory, the inherited acl will have the RICHACL_AUTO_INHERIT flag set, and all inherited aces will have the RICHACE_INHERITED_ACE flag set. This allows user space to distinguish between aces which have been inherited and aces which have been explicitly added. - When the RICHACL_PROTECTED acl flag in the acl of a file or directory is set, AI will not modify the acl. This does not affect propagation of permissions from the file to its children (if the file is a directory). Linux does not have a way of creating files or directories without setting the file permission bits, so all files created inside a directory with RICHACL_AUTO_INHERIT set will have the RICHACL_PROTECTED flag set. This effectively disables Automatic Inheritance. Protocols which support creating files without specifying permissions can explicitly clear the RICHACL_PROTECTED flag after creating a file and reset the file masks to "undo" applying the create mode; see richacl_compute_max_masks(). They should set the RICHACL_DEFAULTED flag. (A mechanism that would allow to indicate to the kernel to ignore the create mode in the first place when there are inherited permissions would be nice to have.) Signed-off-by: Andreas Gruenbacher Reviewed-by: J. Bruce Fields --- fs/richacl_base.c | 10 +++++++++- fs/richacl_inode.c | 7 +++++++ include/linux/richacl.h | 12 ++++++++++++ include/uapi/linux/richacl.h | 11 ++++++++++- 4 files changed, 38 insertions(+), 2 deletions(-) diff --git a/fs/richacl_base.c b/fs/richacl_base.c index 3753216..b67280d 100644 --- a/fs/richacl_base.c +++ b/fs/richacl_base.c @@ -366,7 +366,8 @@ __richacl_chmod(struct richacl *acl, umode_t mode) acl->a_group_mask == group_mask && acl->a_other_mask == other_mask && (acl->a_flags & RICHACL_MASKED) && - (acl->a_flags & RICHACL_WRITE_THROUGH)) + (acl->a_flags & RICHACL_WRITE_THROUGH) && + (!richacl_is_auto_inherit(acl) || richacl_is_protected(acl))) return acl; clone = richacl_clone(acl, GFP_KERNEL); @@ -378,6 +379,8 @@ __richacl_chmod(struct richacl *acl, umode_t mode) clone->a_owner_mask = owner_mask; clone->a_group_mask = group_mask; clone->a_other_mask = other_mask; + if (richacl_is_auto_inherit(clone)) + clone->a_flags |= RICHACL_PROTECTED; return clone; } @@ -551,6 +554,11 @@ richacl_inherit(const struct richacl *dir_acl, int isdir) ace++; } } + if (richacl_is_auto_inherit(dir_acl)) { + acl->a_flags = RICHACL_AUTO_INHERIT; + richacl_for_each_entry(ace, acl) + ace->e_flags |= RICHACE_INHERITED_ACE; + } return acl; } diff --git a/fs/richacl_inode.c b/fs/richacl_inode.c index ec3d2c8..99a1ab6 100644 --- a/fs/richacl_inode.c +++ b/fs/richacl_inode.c @@ -279,6 +279,13 @@ richacl_inherit_inode(const struct richacl *dir_acl, umode_t *mode_p) richacl_put(acl); acl = NULL; } else { + /* + * We need to set RICHACL_PROTECTED because we are + * doing an implicit chmod + */ + if (richacl_is_auto_inherit(acl)) + acl->a_flags |= RICHACL_PROTECTED; + richacl_compute_max_masks(acl); /* * Ensure that the acl will not grant any permissions diff --git a/include/linux/richacl.h b/include/linux/richacl.h index c8fae91a..c524a4b 100644 --- a/include/linux/richacl.h +++ b/include/linux/richacl.h @@ -76,6 +76,18 @@ extern void set_cached_richacl(struct inode *, struct richacl *); extern void forget_cached_richacl(struct inode *); extern struct richacl *get_richacl(struct inode *); +static inline int +richacl_is_auto_inherit(const struct richacl *acl) +{ + return acl->a_flags & RICHACL_AUTO_INHERIT; +} + +static inline int +richacl_is_protected(const struct richacl *acl) +{ + return acl->a_flags & RICHACL_PROTECTED; +} + /** * richace_is_owner - check if @ace is an OWNER@ entry */ diff --git a/include/uapi/linux/richacl.h b/include/uapi/linux/richacl.h index 1ed48ac..8849a53 100644 --- a/include/uapi/linux/richacl.h +++ b/include/uapi/linux/richacl.h @@ -18,6 +18,9 @@ #define __UAPI_RICHACL_H /* a_flags values */ +#define RICHACL_AUTO_INHERIT 0x01 +#define RICHACL_PROTECTED 0x02 +#define RICHACL_DEFAULTED 0x04 #define RICHACL_WRITE_THROUGH 0x40 #define RICHACL_MASKED 0x80 @@ -31,6 +34,7 @@ #define RICHACE_NO_PROPAGATE_INHERIT_ACE 0x0004 #define RICHACE_INHERIT_ONLY_ACE 0x0008 #define RICHACE_IDENTIFIER_GROUP 0x0040 +#define RICHACE_INHERITED_ACE 0x0080 #define RICHACE_SPECIAL_WHO 0x4000 /* e_mask bitflags */ @@ -60,6 +64,9 @@ #define RICHACE_EVERYONE_SPECIAL_ID 2 #define RICHACL_VALID_FLAGS ( \ + RICHACL_AUTO_INHERIT | \ + RICHACL_PROTECTED | \ + RICHACL_DEFAULTED | \ RICHACL_WRITE_THROUGH | \ RICHACL_MASKED ) @@ -69,13 +76,15 @@ RICHACE_NO_PROPAGATE_INHERIT_ACE | \ RICHACE_INHERIT_ONLY_ACE | \ RICHACE_IDENTIFIER_GROUP | \ + RICHACE_INHERITED_ACE | \ RICHACE_SPECIAL_WHO ) #define RICHACE_INHERITANCE_FLAGS ( \ RICHACE_FILE_INHERIT_ACE | \ RICHACE_DIRECTORY_INHERIT_ACE | \ RICHACE_NO_PROPAGATE_INHERIT_ACE | \ - RICHACE_INHERIT_ONLY_ACE ) + RICHACE_INHERIT_ONLY_ACE | \ + RICHACE_INHERITED_ACE ) /* Valid RICHACE_* flags for directories and non-directories */ #define RICHACE_VALID_MASK ( \ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:24:05 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id EAEAA29DFE for ; Mon, 11 Jan 2016 09:24:05 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id BF3058F8040 for ; Mon, 11 Jan 2016 07:24:05 -0800 (PST) X-ASG-Debug-ID: 1452525843-04bdf06f6e741a0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id h7irRA52WtGtAllL (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:24:03 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 87A2D2DC340; Mon, 11 Jan 2016 15:24:03 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbX000775; Mon, 11 Jan 2016 10:23:57 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 18/22] richacl: xattr mapping functions Date: Mon, 11 Jan 2016 16:21:49 +0100 X-ASG-Orig-Subj: [PATCH v17 18/22] richacl: xattr mapping functions Message-Id: <1452525713-11348-19-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525843 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Map between "system.richacl" xattrs and the in-kernel representation. Signed-off-by: Andreas Gruenbacher --- fs/Makefile | 2 +- fs/richacl_xattr.c | 162 +++++++++++++++++++++++++++++++++++++ fs/xattr.c | 29 +++++-- include/linux/richacl_xattr.h | 29 +++++++ include/uapi/linux/Kbuild | 1 + include/uapi/linux/richacl_xattr.h | 44 ++++++++++ include/uapi/linux/xattr.h | 2 + 7 files changed, 262 insertions(+), 7 deletions(-) create mode 100644 fs/richacl_xattr.c create mode 100644 include/linux/richacl_xattr.h create mode 100644 include/uapi/linux/richacl_xattr.h diff --git a/fs/Makefile b/fs/Makefile index d5b45ca..2d6d224 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -49,7 +49,7 @@ obj-$(CONFIG_SYSCTL) += drop_caches.o obj-$(CONFIG_FHANDLE) += fhandle.o obj-$(CONFIG_FS_RICHACL) += richacl.o -richacl-y := richacl_base.o richacl_inode.o +richacl-y := richacl_base.o richacl_inode.o richacl_xattr.o obj-y += quota/ diff --git a/fs/richacl_xattr.c b/fs/richacl_xattr.c new file mode 100644 index 0000000..a273139 --- /dev/null +++ b/fs/richacl_xattr.c @@ -0,0 +1,162 @@ +/* + * Copyright (C) 2006, 2010 Novell, Inc. + * Copyright (C) 2015 Red Hat, Inc. + * Written by Andreas Gruenbacher + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2, or (at your option) any + * later version. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ + +#include +#include +#include +#include +#include + +MODULE_LICENSE("GPL"); + +/** + * richacl_from_xattr - convert a richacl xattr into the in-memory representation + */ +struct richacl * +richacl_from_xattr(struct user_namespace *user_ns, + const void *value, size_t size) +{ + const struct richacl_xattr *xattr_acl = value; + const struct richace_xattr *xattr_ace = (void *)(xattr_acl + 1); + struct richacl *acl; + struct richace *ace; + int count; + + if (size < sizeof(*xattr_acl) || + xattr_acl->a_version != RICHACL_XATTR_VERSION || + (xattr_acl->a_flags & ~RICHACL_VALID_FLAGS)) + return ERR_PTR(-EINVAL); + size -= sizeof(*xattr_acl); + count = le16_to_cpu(xattr_acl->a_count); + if (count > RICHACL_XATTR_MAX_COUNT) + return ERR_PTR(-EINVAL); + if (size != count * sizeof(*xattr_ace)) + return ERR_PTR(-EINVAL); + + acl = richacl_alloc(count, GFP_NOFS); + if (!acl) + return ERR_PTR(-ENOMEM); + + acl->a_flags = xattr_acl->a_flags; + acl->a_owner_mask = le32_to_cpu(xattr_acl->a_owner_mask); + if (acl->a_owner_mask & ~RICHACE_VALID_MASK) + goto fail_einval; + acl->a_group_mask = le32_to_cpu(xattr_acl->a_group_mask); + if (acl->a_group_mask & ~RICHACE_VALID_MASK) + goto fail_einval; + acl->a_other_mask = le32_to_cpu(xattr_acl->a_other_mask); + if (acl->a_other_mask & ~RICHACE_VALID_MASK) + goto fail_einval; + + richacl_for_each_entry(ace, acl) { + ace->e_type = le16_to_cpu(xattr_ace->e_type); + ace->e_flags = le16_to_cpu(xattr_ace->e_flags); + ace->e_mask = le32_to_cpu(xattr_ace->e_mask); + + if (ace->e_flags & ~RICHACE_VALID_FLAGS) + goto fail_einval; + if (ace->e_flags & RICHACE_SPECIAL_WHO) { + ace->e_id.special = le32_to_cpu(xattr_ace->e_id); + if (ace->e_id.special > RICHACE_EVERYONE_SPECIAL_ID) + goto fail_einval; + } else if (ace->e_flags & RICHACE_IDENTIFIER_GROUP) { + u32 id = le32_to_cpu(xattr_ace->e_id); + + ace->e_id.gid = make_kgid(user_ns, id); + if (!gid_valid(ace->e_id.gid)) + goto fail_einval; + } else { + u32 id = le32_to_cpu(xattr_ace->e_id); + + ace->e_id.uid = make_kuid(user_ns, id); + if (!uid_valid(ace->e_id.uid)) + goto fail_einval; + } + if (ace->e_type > RICHACE_ACCESS_DENIED_ACE_TYPE || + (ace->e_mask & ~RICHACE_VALID_MASK)) + goto fail_einval; + + xattr_ace++; + } + + return acl; + +fail_einval: + richacl_put(acl); + return ERR_PTR(-EINVAL); +} +EXPORT_SYMBOL_GPL(richacl_from_xattr); + +/** + * richacl_xattr_size - compute the size of the xattr representation of @acl + */ +size_t +richacl_xattr_size(const struct richacl *acl) +{ + size_t size = sizeof(struct richacl_xattr); + + size += sizeof(struct richace_xattr) * acl->a_count; + return size; +} +EXPORT_SYMBOL_GPL(richacl_xattr_size); + +/** + * richacl_to_xattr - convert @acl into its xattr representation + * @acl: the richacl to convert + * @buffer: buffer for the result + * @size: size of @buffer + */ +int +richacl_to_xattr(struct user_namespace *user_ns, + const struct richacl *acl, void *buffer, size_t size) +{ + struct richacl_xattr *xattr_acl = buffer; + struct richace_xattr *xattr_ace; + const struct richace *ace; + size_t real_size; + + real_size = richacl_xattr_size(acl); + if (!buffer) + return real_size; + if (real_size > size) + return -ERANGE; + + xattr_acl->a_version = RICHACL_XATTR_VERSION; + xattr_acl->a_flags = acl->a_flags; + xattr_acl->a_count = cpu_to_le16(acl->a_count); + + xattr_acl->a_owner_mask = cpu_to_le32(acl->a_owner_mask); + xattr_acl->a_group_mask = cpu_to_le32(acl->a_group_mask); + xattr_acl->a_other_mask = cpu_to_le32(acl->a_other_mask); + + xattr_ace = (void *)(xattr_acl + 1); + richacl_for_each_entry(ace, acl) { + xattr_ace->e_type = cpu_to_le16(ace->e_type); + xattr_ace->e_flags = cpu_to_le16(ace->e_flags); + xattr_ace->e_mask = cpu_to_le32(ace->e_mask); + if (ace->e_flags & RICHACE_SPECIAL_WHO) + xattr_ace->e_id = cpu_to_le32(ace->e_id.special); + else if (ace->e_flags & RICHACE_IDENTIFIER_GROUP) + xattr_ace->e_id = + cpu_to_le32(from_kgid(user_ns, ace->e_id.gid)); + else + xattr_ace->e_id = + cpu_to_le32(from_kuid(user_ns, ace->e_id.uid)); + xattr_ace++; + } + return real_size; +} +EXPORT_SYMBOL_GPL(richacl_to_xattr); diff --git a/fs/xattr.c b/fs/xattr.c index d7f5037..58df156 100644 --- a/fs/xattr.c +++ b/fs/xattr.c @@ -295,6 +295,16 @@ out: } EXPORT_SYMBOL_GPL(vfs_removexattr); +static void +fix_xattr_from_user(const char *kname, void *kvalue, size_t size) +{ + if (strncmp(kname, XATTR_SYSTEM_PREFIX, XATTR_SYSTEM_PREFIX_LEN)) + return; + kname += XATTR_SYSTEM_PREFIX_LEN; + if (!strcmp(kname, XATTR_POSIX_ACL_ACCESS) || + !strcmp(kname, XATTR_POSIX_ACL_DEFAULT)) + posix_acl_fix_xattr_from_user(kvalue, size); +} /* * Extended attribute SET operations @@ -331,9 +341,7 @@ setxattr(struct dentry *d, const char __user *name, const void __user *value, error = -EFAULT; goto out; } - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) - posix_acl_fix_xattr_from_user(kvalue, size); + fix_xattr_from_user(kname, kvalue, size); } error = vfs_setxattr(d, kname, kvalue, size, flags); @@ -400,6 +408,17 @@ SYSCALL_DEFINE5(fsetxattr, int, fd, const char __user *, name, return error; } +static void +fix_xattr_to_user(const char *kname, void *kvalue, size_t size) +{ + if (strncmp(kname, XATTR_SYSTEM_PREFIX, XATTR_SYSTEM_PREFIX_LEN)) + return; + kname += XATTR_SYSTEM_PREFIX_LEN; + if (!strcmp(kname, XATTR_POSIX_ACL_ACCESS) || + !strcmp(kname, XATTR_POSIX_ACL_DEFAULT)) + posix_acl_fix_xattr_to_user(kvalue, size); +} + /* * Extended attribute GET operations */ @@ -432,9 +451,7 @@ getxattr(struct dentry *d, const char __user *name, void __user *value, error = vfs_getxattr(d, kname, kvalue, size); if (error > 0) { - if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) || - (strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0)) - posix_acl_fix_xattr_to_user(kvalue, size); + fix_xattr_to_user(kname, kvalue, size); if (size && copy_to_user(value, kvalue, error)) error = -EFAULT; } else if (error == -ERANGE && size >= XATTR_SIZE_MAX) { diff --git a/include/linux/richacl_xattr.h b/include/linux/richacl_xattr.h new file mode 100644 index 0000000..ab67af2 --- /dev/null +++ b/include/linux/richacl_xattr.h @@ -0,0 +1,29 @@ +/* + * Copyright (C) 2006, 2010 Novell, Inc. + * Copyright (C) 2015 Red Hat, Inc. + * Written by Andreas Gruenbacher + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2, or (at your option) any + * later version. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ + +#ifndef __RICHACL_XATTR_H +#define __RICHACL_XATTR_H + +#include +#include + +extern struct richacl *richacl_from_xattr(struct user_namespace *, const void *, + size_t); +extern size_t richacl_xattr_size(const struct richacl *); +extern int richacl_to_xattr(struct user_namespace *, const struct richacl *, + void *, size_t); + +#endif /* __RICHACL_XATTR_H */ diff --git a/include/uapi/linux/Kbuild b/include/uapi/linux/Kbuild index 55699fa..f45190f 100644 --- a/include/uapi/linux/Kbuild +++ b/include/uapi/linux/Kbuild @@ -351,6 +351,7 @@ header-y += reiserfs_fs.h header-y += reiserfs_xattr.h header-y += resource.h header-y += richacl.h +header-y += richacl_xattr.h header-y += rfkill.h header-y += romfs_fs.h header-y += rose.h diff --git a/include/uapi/linux/richacl_xattr.h b/include/uapi/linux/richacl_xattr.h new file mode 100644 index 0000000..5178ca6 --- /dev/null +++ b/include/uapi/linux/richacl_xattr.h @@ -0,0 +1,44 @@ +/* + * Copyright (C) 2006, 2010 Novell, Inc. + * Copyright (C) 2015 Red Hat, Inc. + * Written by Andreas Gruenbacher + * + * This file is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This file is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + */ + +#ifndef __UAPI_RICHACL_XATTR_H +#define __UAPI_RICHACL_XATTR_H + +#include +#include + +struct richace_xattr { + __le16 e_type; + __le16 e_flags; + __le32 e_mask; + __le32 e_id; +}; + +struct richacl_xattr { + unsigned char a_version; + unsigned char a_flags; + __le16 a_count; + __le32 a_owner_mask; + __le32 a_group_mask; + __le32 a_other_mask; +}; + +#define RICHACL_XATTR_VERSION 0 +#define RICHACL_XATTR_MAX_COUNT \ + ((XATTR_SIZE_MAX - sizeof(struct richacl_xattr)) / \ + sizeof(struct richace_xattr)) + +#endif /* __UAPI_RICHACL_XATTR_H */ diff --git a/include/uapi/linux/xattr.h b/include/uapi/linux/xattr.h index 1590c49..1996903 100644 --- a/include/uapi/linux/xattr.h +++ b/include/uapi/linux/xattr.h @@ -73,5 +73,7 @@ #define XATTR_POSIX_ACL_DEFAULT "posix_acl_default" #define XATTR_NAME_POSIX_ACL_DEFAULT XATTR_SYSTEM_PREFIX XATTR_POSIX_ACL_DEFAULT +#define XATTR_RICHACL "richacl" +#define XATTR_NAME_RICHACL XATTR_SYSTEM_PREFIX XATTR_RICHACL #endif /* _UAPI_LINUX_XATTR_H */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:24:12 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 0F3C629DFE for ; Mon, 11 Jan 2016 09:24:12 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id F2D2B304053 for ; Mon, 11 Jan 2016 07:24:11 -0800 (PST) X-ASG-Debug-ID: 1452525850-04cbb05f7c74aa0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id SjBMm3MSexhbJtBK (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:24:11 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 2E0B71304; Mon, 11 Jan 2016 15:24:10 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbY000775; Mon, 11 Jan 2016 10:24:04 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 19/22] richacl: Add richacl xattr handler Date: Mon, 11 Jan 2016 16:21:50 +0100 X-ASG-Orig-Subj: [PATCH v17 19/22] richacl: Add richacl xattr handler Message-Id: <1452525713-11348-20-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525850 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Add richacl xattr handler implementing the xattr operations based on the get_richacl and set_richacl inode operations. Signed-off-by: Andreas Gruenbacher --- fs/richacl_xattr.c | 73 +++++++++++++++++++++++++++++++++++++++++++ include/linux/richacl_xattr.h | 2 ++ 2 files changed, 75 insertions(+) diff --git a/fs/richacl_xattr.c b/fs/richacl_xattr.c index a273139..afa2859 100644 --- a/fs/richacl_xattr.c +++ b/fs/richacl_xattr.c @@ -18,7 +18,9 @@ #include #include #include +#include #include +#include MODULE_LICENSE("GPL"); @@ -160,3 +162,74 @@ richacl_to_xattr(struct user_namespace *user_ns, return real_size; } EXPORT_SYMBOL_GPL(richacl_to_xattr); + +static bool +richacl_xattr_list(struct dentry *dentry) +{ + return IS_RICHACL(d_backing_inode(dentry)); +} + +static int +richacl_xattr_get(const struct xattr_handler *handler, + struct dentry *dentry, const char *name, void *buffer, + size_t buffer_size) +{ + struct inode *inode = d_backing_inode(dentry); + struct richacl *acl; + int error; + + if (*name) + return -EINVAL; + if (!IS_RICHACL(inode)) + return -EOPNOTSUPP; + if (S_ISLNK(inode->i_mode)) + return -EOPNOTSUPP; + acl = get_richacl(inode); + if (IS_ERR(acl)) + return PTR_ERR(acl); + if (acl == NULL) + return -ENODATA; + error = richacl_to_xattr(current_user_ns(), acl, buffer, buffer_size); + richacl_put(acl); + return error; +} + +static int +richacl_xattr_set(const struct xattr_handler *handler, + struct dentry *dentry, const char *name, + const void *value, size_t size, int flags) +{ + struct inode *inode = d_backing_inode(dentry); + struct richacl *acl = NULL; + int ret; + + if (*name) + return -EINVAL; + if (!IS_RICHACL(inode)) + return -EOPNOTSUPP; + if (!inode->i_op->set_richacl) + return -EOPNOTSUPP; + + if (!uid_eq(current_fsuid(), inode->i_uid) && + inode_permission(inode, MAY_CHMOD) && + !capable(CAP_FOWNER)) + return -EPERM; + + if (value) { + acl = richacl_from_xattr(current_user_ns(), value, size); + if (IS_ERR(acl)) + return PTR_ERR(acl); + } + + ret = inode->i_op->set_richacl(inode, acl); + richacl_put(acl); + return ret; +} + +struct xattr_handler richacl_xattr_handler = { + .name = XATTR_NAME_RICHACL, + .list = richacl_xattr_list, + .get = richacl_xattr_get, + .set = richacl_xattr_set, +}; +EXPORT_SYMBOL(richacl_xattr_handler); diff --git a/include/linux/richacl_xattr.h b/include/linux/richacl_xattr.h index ab67af2..ad4a56e 100644 --- a/include/linux/richacl_xattr.h +++ b/include/linux/richacl_xattr.h @@ -26,4 +26,6 @@ extern size_t richacl_xattr_size(const struct richacl *); extern int richacl_to_xattr(struct user_namespace *, const struct richacl *, void *, size_t); +extern struct xattr_handler richacl_xattr_handler; + #endif /* __RICHACL_XATTR_H */ -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:24:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9047429DFE for ; Mon, 11 Jan 2016 09:24:18 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 6597C30405F for ; Mon, 11 Jan 2016 07:24:18 -0800 (PST) X-ASG-Debug-ID: 1452525857-04bdf06f6c741b0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 3i1tH301bxPYALk0 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:24:17 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id D8E41C0BF2B4; Mon, 11 Jan 2016 15:24:16 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbZ000775; Mon, 11 Jan 2016 10:24:10 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: Andreas Gruenbacher Subject: [PATCH v17 20/22] vfs: Add richacl permission checking Date: Mon, 11 Jan 2016 16:21:51 +0100 X-ASG-Orig-Subj: [PATCH v17 20/22] vfs: Add richacl permission checking Message-Id: <1452525713-11348-21-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525857 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 Hook the richacl permission checking function into the vfs. Signed-off-by: Andreas Gruenbacher --- fs/namei.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- fs/posix_acl.c | 6 +++--- 2 files changed, 52 insertions(+), 5 deletions(-) diff --git a/fs/namei.c b/fs/namei.c index d242992..0335b3d 100644 --- a/fs/namei.c +++ b/fs/namei.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include "internal.h" @@ -255,7 +256,40 @@ void putname(struct filename *name) __putname(name); } -static int check_acl(struct inode *inode, int mask) +static int check_richacl(struct inode *inode, int mask) +{ +#ifdef CONFIG_FS_RICHACL + struct richacl *acl; + + if (mask & MAY_NOT_BLOCK) { + acl = get_cached_richacl_rcu(inode); + if (!acl) + goto no_acl; + /* no ->get_richacl() calls in RCU mode... */ + if (acl == ACL_NOT_CACHED) + return -ECHILD; + return richacl_permission(inode, acl, mask & ~MAY_NOT_BLOCK); + } + + acl = get_richacl(inode); + if (IS_ERR(acl)) + return PTR_ERR(acl); + if (acl) { + int error = richacl_permission(inode, acl, mask); + richacl_put(acl); + return error; + } +no_acl: +#endif + if (mask & (MAY_DELETE_SELF | MAY_TAKE_OWNERSHIP | + MAY_CHMOD | MAY_SET_TIMES)) { + /* File permission bits cannot grant this. */ + return -EACCES; + } + return -EAGAIN; +} + +static int check_posix_acl(struct inode *inode, int mask) { #ifdef CONFIG_FS_POSIX_ACL struct posix_acl *acl; @@ -290,11 +324,24 @@ static int acl_permission_check(struct inode *inode, int mask) { unsigned int mode = inode->i_mode; + /* + * With POSIX ACLs, the (mode & S_IRWXU) bits exactly match the owner + * permissions, and we can skip checking posix acls for the owner. + * With richacls, the owner may be granted fewer permissions than the + * mode bits seem to suggest (for example, append but not write), and + * we always need to check the richacl. + */ + + if (IS_RICHACL(inode)) { + int error = check_richacl(inode, mask); + if (error != -EAGAIN) + return error; + } if (likely(uid_eq(current_fsuid(), inode->i_uid))) mode >>= 6; else { if (IS_POSIXACL(inode) && (mode & S_IRWXG)) { - int error = check_acl(inode, mask); + int error = check_posix_acl(inode, mask); if (error != -EAGAIN) return error; } diff --git a/fs/posix_acl.c b/fs/posix_acl.c index f24646e..7810c6f 100644 --- a/fs/posix_acl.c +++ b/fs/posix_acl.c @@ -100,13 +100,13 @@ struct posix_acl *get_acl(struct inode *inode, int type) { struct posix_acl *acl; + if (!IS_POSIXACL(inode)) + return NULL; + acl = get_cached_acl(inode, type); if (acl != ACL_NOT_CACHED) return acl; - if (!IS_POSIXACL(inode)) - return NULL; - /* * A filesystem can force a ACL callback by just never filling the * ACL cache. But normally you'd fill the cache either at inode -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:24:26 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6DD4829DFD for ; Mon, 11 Jan 2016 09:24:26 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 2A1A78F8040 for ; Mon, 11 Jan 2016 07:24:26 -0800 (PST) X-ASG-Debug-ID: 1452525864-04bdf06f6d741c0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id AfMnC5bqGth3FiGs (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:24:24 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id D662F8F26E; Mon, 11 Jan 2016 15:24:23 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsba000775; Mon, 11 Jan 2016 10:24:17 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: "Aneesh Kumar K.V" , Andreas Gruenbacher Subject: [PATCH v17 21/22] ext4: Add richacl support Date: Mon, 11 Jan 2016 16:21:52 +0100 X-ASG-Orig-Subj: [PATCH v17 21/22] ext4: Add richacl support Message-Id: <1452525713-11348-22-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525864 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: "Aneesh Kumar K.V" Support the richacl permission model in ext4. The richacls are stored in "system.richacl" xattrs. Richacls need to be enabled by tune2fs or at file system create time. Signed-off-by: Aneesh Kumar K.V Signed-off-by: Andreas Gruenbacher Reviewed-by: Andreas Dilger --- fs/ext4/Kconfig | 11 +++++ fs/ext4/Makefile | 1 + fs/ext4/file.c | 3 ++ fs/ext4/ialloc.c | 11 ++++- fs/ext4/inode.c | 12 ++++- fs/ext4/namei.c | 5 ++ fs/ext4/richacl.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ fs/ext4/richacl.h | 40 +++++++++++++++ fs/ext4/xattr.c | 7 +++ 9 files changed, 229 insertions(+), 3 deletions(-) create mode 100644 fs/ext4/richacl.c create mode 100644 fs/ext4/richacl.h diff --git a/fs/ext4/Kconfig b/fs/ext4/Kconfig index b46e9fc..4e21c18 100644 --- a/fs/ext4/Kconfig +++ b/fs/ext4/Kconfig @@ -22,6 +22,17 @@ config EXT3_FS_POSIX_ACL This config option is here only for backward compatibility. ext3 filesystem is now handled by the ext4 driver. +config EXT4_FS_RICHACL + bool "Ext4 Rich Access Control Lists" + depends on EXT4_FS + select FS_RICHACL + help + Richacls are an implementation of NFSv4 ACLs, extended by file masks + to cleanly integrate into the POSIX file permission model. To learn + more about them, see http://www.bestbits.at/richacl/. + + If you don't know what Richacls are, say N. + config EXT3_FS_SECURITY bool "Ext3 Security Labels" depends on EXT3_FS diff --git a/fs/ext4/Makefile b/fs/ext4/Makefile index f52cf54..1fb7f11 100644 --- a/fs/ext4/Makefile +++ b/fs/ext4/Makefile @@ -14,3 +14,4 @@ ext4-$(CONFIG_EXT4_FS_POSIX_ACL) += acl.o ext4-$(CONFIG_EXT4_FS_SECURITY) += xattr_security.o ext4-$(CONFIG_EXT4_FS_ENCRYPTION) += crypto_policy.o crypto.o \ crypto_key.o crypto_fname.o +ext4-$(CONFIG_EXT4_FS_RICHACL) += richacl.o diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 113837e..a03b4a5 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -30,6 +30,7 @@ #include "ext4_jbd2.h" #include "xattr.h" #include "acl.h" +#include "richacl.h" /* * Called when an inode is released. Note that this is different @@ -719,6 +720,8 @@ const struct inode_operations ext4_file_inode_operations = { .removexattr = generic_removexattr, .get_acl = ext4_get_acl, .set_acl = ext4_set_acl, + .get_richacl = ext4_get_richacl, + .set_richacl = ext4_set_richacl, .fiemap = ext4_fiemap, }; diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c index 1b8024d..58d9f7d 100644 --- a/fs/ext4/ialloc.c +++ b/fs/ext4/ialloc.c @@ -27,6 +27,7 @@ #include "ext4_jbd2.h" #include "xattr.h" #include "acl.h" +#include "richacl.h" #include @@ -727,6 +728,14 @@ out: return ret; } +static inline int +ext4_new_acl(handle_t *handle, struct inode *inode, struct inode *dir) +{ + if (IS_RICHACL(dir)) + return ext4_init_richacl(handle, inode, dir); + return ext4_init_acl(handle, inode, dir); +} + /* * There are two policies for allocating an inode. If the new inode is * a directory, then a forward search is made for a block group with both @@ -1084,7 +1093,7 @@ got: if (err) goto fail_drop; - err = ext4_init_acl(handle, inode, dir); + err = ext4_new_acl(handle, inode, dir); if (err) goto fail_free_drop; diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index ea433a7..d68d979 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -42,6 +42,7 @@ #include "xattr.h" #include "acl.h" #include "truncate.h" +#include "richacl.h" #include @@ -4677,6 +4678,14 @@ static void ext4_wait_for_tail_page_commit(struct inode *inode) } } +static inline int +ext4_acl_chmod(struct inode *inode, umode_t mode) +{ + if (IS_RICHACL(inode)) + return richacl_chmod(inode, inode->i_mode); + return posix_acl_chmod(inode, inode->i_mode); +} + /* * ext4_setattr() * @@ -4845,8 +4854,7 @@ int ext4_setattr(struct dentry *dentry, struct iattr *attr) ext4_orphan_del(NULL, inode); if (!rc && (ia_valid & ATTR_MODE)) - rc = posix_acl_chmod(inode, inode->i_mode); - + rc = ext4_acl_chmod(inode, inode->i_mode); err_out: ext4_std_error(inode->i_sb, error); if (!error) diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c index a969ab3..3edeeaa 100644 --- a/fs/ext4/namei.c +++ b/fs/ext4/namei.c @@ -38,6 +38,7 @@ #include "xattr.h" #include "acl.h" +#include "richacl.h" #include /* @@ -3849,6 +3850,8 @@ const struct inode_operations ext4_dir_inode_operations = { .removexattr = generic_removexattr, .get_acl = ext4_get_acl, .set_acl = ext4_set_acl, + .get_richacl = ext4_get_richacl, + .set_richacl = ext4_set_richacl, .fiemap = ext4_fiemap, }; @@ -3860,4 +3863,6 @@ const struct inode_operations ext4_special_inode_operations = { .removexattr = generic_removexattr, .get_acl = ext4_get_acl, .set_acl = ext4_set_acl, + .get_richacl = ext4_get_richacl, + .set_richacl = ext4_set_richacl, }; diff --git a/fs/ext4/richacl.c b/fs/ext4/richacl.c new file mode 100644 index 0000000..d581be4 --- /dev/null +++ b/fs/ext4/richacl.c @@ -0,0 +1,142 @@ +/* + * Copyright IBM Corporation, 2010 + * Copyright (C) 2015 Red Hat, Inc. + * Author: Aneesh Kumar K.V , + * Andreas Gruenbacher + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of version 2.1 of the GNU Lesser General Public License + * as published by the Free Software Foundation. + * + * This program is distributed in the hope that it would be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + * + */ + +#include +#include +#include + +#include "ext4.h" +#include "ext4_jbd2.h" +#include "xattr.h" +#include "acl.h" +#include "richacl.h" + +struct richacl * +ext4_get_richacl(struct inode *inode) +{ + const int name_index = EXT4_XATTR_INDEX_RICHACL; + void *value = NULL; + struct richacl *acl = NULL; + int retval; + + retval = ext4_xattr_get(inode, name_index, "", NULL, 0); + if (retval > 0) { + value = kmalloc(retval, GFP_NOFS); + if (!value) + return ERR_PTR(-ENOMEM); + retval = ext4_xattr_get(inode, name_index, "", value, retval); + } + if (retval > 0) { + acl = richacl_from_xattr(&init_user_ns, value, retval); + if (acl == ERR_PTR(-EINVAL)) + acl = ERR_PTR(-EIO); + } else if (retval != -ENODATA && retval != -ENOSYS) { + acl = ERR_PTR(retval); + } + kfree(value); + + if (!IS_ERR(acl)) + set_cached_richacl(inode, acl); + + return acl; +} + +static int +__ext4_remove_richacl(handle_t *handle, struct inode *inode) +{ + const int name_index = EXT4_XATTR_INDEX_RICHACL; + int retval; + + retval = ext4_xattr_set_handle(handle, inode, name_index, "", + NULL, 0, 0); + if (!retval) + set_cached_richacl(inode, NULL); + return retval; +} + +static int +__ext4_set_richacl(handle_t *handle, struct inode *inode, struct richacl *acl) +{ + const int name_index = EXT4_XATTR_INDEX_RICHACL; + umode_t mode = inode->i_mode; + int retval, size; + void *value; + + if (richacl_equiv_mode(acl, &mode) == 0) { + inode->i_ctime = ext4_current_time(inode); + inode->i_mode = mode; + ext4_mark_inode_dirty(handle, inode); + return __ext4_remove_richacl(handle, inode); + } + + mode &= ~S_IRWXUGO; + mode |= richacl_masks_to_mode(acl); + + size = richacl_xattr_size(acl); + value = kmalloc(size, GFP_NOFS); + if (!value) + return -ENOMEM; + richacl_to_xattr(&init_user_ns, acl, value, size); + inode->i_mode = mode; + retval = ext4_xattr_set_handle(handle, inode, name_index, "", + value, size, 0); + kfree(value); + if (retval) + return retval; + + set_cached_richacl(inode, acl); + + return 0; +} + +int +ext4_set_richacl(struct inode *inode, struct richacl *acl) +{ + handle_t *handle; + int retval, retries = 0; + +retry: + handle = ext4_journal_start(inode, EXT4_HT_XATTR, + ext4_jbd2_credits_xattr(inode)); + if (IS_ERR(handle)) + return PTR_ERR(handle); + + if (acl) + retval = __ext4_set_richacl(handle, inode, acl); + else + retval = __ext4_remove_richacl(handle, inode); + + ext4_journal_stop(handle); + if (retval == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries)) + goto retry; + return retval; +} + +int +ext4_init_richacl(handle_t *handle, struct inode *inode, struct inode *dir) +{ + struct richacl *acl = richacl_create(&inode->i_mode, dir); + int error; + + error = PTR_ERR(acl); + if (IS_ERR(acl)) + return error; + if (acl) { + error = __ext4_set_richacl(handle, inode, acl); + richacl_put(acl); + } + return error; +} diff --git a/fs/ext4/richacl.h b/fs/ext4/richacl.h new file mode 100644 index 0000000..6fe9a92 --- /dev/null +++ b/fs/ext4/richacl.h @@ -0,0 +1,40 @@ +/* + * Copyright IBM Corporation, 2010 + * Copyright (C) 2015 Red Hat, Inc. + * Author Aneesh Kumar K.V + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of version 2.1 of the GNU Lesser General Public License + * as published by the Free Software Foundation. + * + * This program is distributed in the hope that it would be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + * + */ + +#ifndef __FS_EXT4_RICHACL_H +#define __FS_EXT4_RICHACL_H + +#include + +#ifdef CONFIG_EXT4_FS_RICHACL + +extern struct richacl *ext4_get_richacl(struct inode *); +extern int ext4_set_richacl(struct inode *, struct richacl *); + +extern int ext4_init_richacl(handle_t *, struct inode *, struct inode *); + +#else /* CONFIG_EXT4_FS_RICHACL */ + +#define ext4_get_richacl NULL +#define ext4_set_richacl NULL + +static inline int +ext4_init_richacl(handle_t *handle, struct inode *inode, struct inode *dir) +{ + return 0; +} + +#endif /* CONFIG_EXT4_FS_RICHACL */ +#endif /* __FS_EXT4_RICHACL_H */ diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c index e9b9afd..91f0ff2 100644 --- a/fs/ext4/xattr.c +++ b/fs/ext4/xattr.c @@ -55,6 +55,7 @@ #include #include #include +#include #include "ext4_jbd2.h" #include "ext4.h" #include "xattr.h" @@ -99,6 +100,9 @@ static const struct xattr_handler *ext4_xattr_handler_map[] = { #ifdef CONFIG_EXT4_FS_SECURITY [EXT4_XATTR_INDEX_SECURITY] = &ext4_xattr_security_handler, #endif +#ifdef CONFIG_EXT4_FS_RICHACL + [EXT4_XATTR_INDEX_RICHACL] = &richacl_xattr_handler, +#endif }; const struct xattr_handler *ext4_xattr_handlers[] = { @@ -111,6 +115,9 @@ const struct xattr_handler *ext4_xattr_handlers[] = { #ifdef CONFIG_EXT4_FS_SECURITY &ext4_xattr_security_handler, #endif +#ifdef CONFIG_EXT4_FS_RICHACL + &richacl_xattr_handler, +#endif NULL }; -- 2.5.0 From agruenba@redhat.com Mon Jan 11 09:24:33 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0F74629DFD for ; Mon, 11 Jan 2016 09:24:33 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id E4CBF8F8040 for ; Mon, 11 Jan 2016 07:24:32 -0800 (PST) X-ASG-Debug-ID: 1452525871-04cb6c75dc70960001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id HddJBGMaSTXLzVhK (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:24:31 -0800 (PST) X-Barracuda-Envelope-From: agruenba@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id D885D19D60F; Mon, 11 Jan 2016 15:24:30 +0000 (UTC) Received: from nux.redhat.com (vpn1-7-125.ams2.redhat.com [10.36.7.125]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BFLsbb000775; Mon, 11 Jan 2016 10:24:24 -0500 From: Andreas Gruenbacher To: Alexander Viro , "Theodore Ts'o" , Andreas Dilger , "J. Bruce Fields" , Jeff Layton , Trond Myklebust , Anna Schumaker , Dave Chinner , linux-ext4@vger.kernel.org, xfs@oss.sgi.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-cifs@vger.kernel.org, linux-api@vger.kernel.org Cc: "Aneesh Kumar K.V" , Andreas Gruenbacher Subject: [PATCH v17 22/22] ext4: Add richacl feature flag Date: Mon, 11 Jan 2016 16:21:53 +0100 X-ASG-Orig-Subj: [PATCH v17 22/22] ext4: Add richacl feature flag Message-Id: <1452525713-11348-23-git-send-email-agruenba@redhat.com> In-Reply-To: <1452525713-11348-1-git-send-email-agruenba@redhat.com> References: <1452525713-11348-1-git-send-email-agruenba@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452525871 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: "Aneesh Kumar K.V" This feature flag selects richacl instead of POSIX ACL support on the filesystem. When this feature is off, the "acl" and "noacl" mount options control whether POSIX ACLs are enabled. When it is on, richacls are automatically enabled and using the "noacl" mount option leads to an error. Signed-off-by: Aneesh Kumar K.V Signed-off-by: Andreas Gruenbacher Reviewed-by: Andreas Dilger --- fs/ext4/ext4.h | 6 ++++-- fs/ext4/super.c | 49 ++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 44 insertions(+), 11 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 750063f..7072ff5 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -995,7 +995,7 @@ struct ext4_inode_info { #define EXT4_MOUNT_UPDATE_JOURNAL 0x01000 /* Update the journal format */ #define EXT4_MOUNT_NO_UID32 0x02000 /* Disable 32-bit UIDs */ #define EXT4_MOUNT_XATTR_USER 0x04000 /* Extended user attributes */ -#define EXT4_MOUNT_POSIX_ACL 0x08000 /* POSIX Access Control Lists */ +#define EXT4_MOUNT_ACL 0x08000 /* Access Control Lists */ #define EXT4_MOUNT_NO_AUTO_DA_ALLOC 0x10000 /* No auto delalloc mapping */ #define EXT4_MOUNT_BARRIER 0x20000 /* Use block barriers */ #define EXT4_MOUNT_QUOTA 0x80000 /* Some quota option set */ @@ -1593,6 +1593,7 @@ static inline int ext4_encrypted_inode(struct inode *inode) #define EXT4_FEATURE_INCOMPAT_LARGEDIR 0x4000 /* >2GB or 3-lvl htree */ #define EXT4_FEATURE_INCOMPAT_INLINE_DATA 0x8000 /* data in inode */ #define EXT4_FEATURE_INCOMPAT_ENCRYPT 0x10000 +#define EXT4_FEATURE_INCOMPAT_RICHACL 0x20000 #define EXT4_FEATURE_COMPAT_FUNCS(name, flagname) \ static inline bool ext4_has_feature_##name(struct super_block *sb) \ @@ -1707,7 +1708,8 @@ EXT4_FEATURE_INCOMPAT_FUNCS(encrypt, ENCRYPT) EXT4_FEATURE_INCOMPAT_MMP | \ EXT4_FEATURE_INCOMPAT_INLINE_DATA | \ EXT4_FEATURE_INCOMPAT_ENCRYPT | \ - EXT4_FEATURE_INCOMPAT_CSUM_SEED) + EXT4_FEATURE_INCOMPAT_CSUM_SEED | \ + EXT4_FEATURE_INCOMPAT_RICHACL) #define EXT4_FEATURE_RO_COMPAT_SUPP (EXT4_FEATURE_RO_COMPAT_SPARSE_SUPER| \ EXT4_FEATURE_RO_COMPAT_LARGE_FILE| \ EXT4_FEATURE_RO_COMPAT_GDT_CSUM| \ diff --git a/fs/ext4/super.c b/fs/ext4/super.c index c9ab67d..9486b8a 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -1273,6 +1273,28 @@ static ext4_fsblk_t get_sb_block(void **data) return sb_block; } +static int enable_acl(struct super_block *sb) +{ + sb->s_flags &= ~(MS_POSIXACL | MS_RICHACL); + if (test_opt(sb, ACL)) { + if (EXT4_HAS_INCOMPAT_FEATURE(sb, + EXT4_FEATURE_INCOMPAT_RICHACL)) { +#ifdef CONFIG_EXT4_FS_RICHACL + sb->s_flags |= MS_RICHACL; +#else + return -EOPNOTSUPP; +#endif + } else { +#ifdef CONFIG_EXT4_FS_POSIX_ACL + sb->s_flags |= MS_POSIXACL; +#else + return -EOPNOTSUPP; +#endif + } + } + return 0; +} + #define DEFAULT_JOURNAL_IOPRIO (IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, 3)) static char deprecated_msg[] = "Mount option \"%s\" will be removed by %s\n" "Contact linux-ext4@vger.kernel.org if you think we should keep it.\n"; @@ -1419,9 +1441,9 @@ static const struct mount_opts { MOPT_NO_EXT2 | MOPT_DATAJ}, {Opt_user_xattr, EXT4_MOUNT_XATTR_USER, MOPT_SET}, {Opt_nouser_xattr, EXT4_MOUNT_XATTR_USER, MOPT_CLEAR}, -#ifdef CONFIG_EXT4_FS_POSIX_ACL - {Opt_acl, EXT4_MOUNT_POSIX_ACL, MOPT_SET}, - {Opt_noacl, EXT4_MOUNT_POSIX_ACL, MOPT_CLEAR}, +#if defined(CONFIG_EXT4_FS_POSIX_ACL) || defined(CONFIG_EXT4_FS_RICHACL) + {Opt_acl, EXT4_MOUNT_ACL, MOPT_SET}, + {Opt_noacl, EXT4_MOUNT_ACL, MOPT_CLEAR}, #else {Opt_acl, 0, MOPT_NOSUPPORT}, {Opt_noacl, 0, MOPT_NOSUPPORT}, @@ -1469,6 +1491,13 @@ static int handle_mount_opt(struct super_block *sb, char *opt, int token, #endif switch (token) { case Opt_noacl: +#ifdef CONFIG_EXT4_FS_RICHACL + if (EXT4_HAS_INCOMPAT_FEATURE(sb, EXT4_FEATURE_INCOMPAT_RICHACL)) { + ext4_msg(sb, KERN_ERR, "Mount option \"%s\" incompatible " + "with richacl feature", opt); + return -1; + } +#endif case Opt_nouser_xattr: ext4_msg(sb, KERN_WARNING, deprecated_msg, opt, "3.5"); break; @@ -3229,8 +3258,8 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent) set_opt(sb, NO_UID32); /* xattr user namespace & acls are now defaulted on */ set_opt(sb, XATTR_USER); -#ifdef CONFIG_EXT4_FS_POSIX_ACL - set_opt(sb, POSIX_ACL); +#if defined(CONFIG_EXT4_FS_POSIX_ACL) || defined(CONFIG_EXT4_FS_RICHACL) + set_opt(sb, ACL); #endif /* don't forget to enable journal_csum when metadata_csum is enabled. */ if (ext4_has_metadata_csum(sb)) @@ -3313,8 +3342,9 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent) sb->s_iflags |= SB_I_CGROUPWB; } - sb->s_flags = (sb->s_flags & ~MS_POSIXACL) | - (test_opt(sb, POSIX_ACL) ? MS_POSIXACL : 0); + err = enable_acl(sb); + if (err) + goto failed_mount; if (le32_to_cpu(es->s_rev_level) == EXT4_GOOD_OLD_REV && (ext4_has_compat_features(sb) || @@ -4630,8 +4660,9 @@ static int ext4_remount(struct super_block *sb, int *flags, char *data) if (sbi->s_mount_flags & EXT4_MF_FS_ABORTED) ext4_abort(sb, "Abort forced by user"); - sb->s_flags = (sb->s_flags & ~MS_POSIXACL) | - (test_opt(sb, POSIX_ACL) ? MS_POSIXACL : 0); + err = enable_acl(sb); + if (err) + goto restore_opts; es = sbi->s_es; -- 2.5.0 From hare@suse.de Mon Jan 11 09:24:43 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: * X-Spam-Status: No, score=1.0 required=5.0 tests=FRT_OFFER2 autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id D1D7429DFD for ; Mon, 11 Jan 2016 09:24:43 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id B365B8F8050 for ; Mon, 11 Jan 2016 07:24:43 -0800 (PST) X-ASG-Debug-ID: 1452525880-04cbb05f7d74af0001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id gsIuCsw0tMf73SGY (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:24:41 -0800 (PST) X-Barracuda-Envelope-From: hare@suse.de X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id CDFF9AABB; Mon, 11 Jan 2016 15:24:37 +0000 (UTC) Subject: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life To: Al Viro , Dan Williams X-ASG-Orig-Subj: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> <20160109075414.GA5008@ZenIV.linux.org.uk> Cc: xfs@oss.sgi.com, linux-block@vger.kernel.org, linux-nvdimm@ml01.01.org, Dave Chinner , Jens Axboe , Jan Kara , linux-fsdevel@vger.kernel.org, Matthew Wilcox , Ross Zwisler From: Hannes Reinecke Message-ID: <5693C935.3060701@suse.de> Date: Mon, 11 Jan 2016 16:24:37 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <20160109075414.GA5008@ZenIV.linux.org.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452525881 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 1.59 X-Barracuda-Spam-Status: No, SCORE=1.59 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, FRT_OFFER2 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26032 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 1.59 FRT_OFFER2 BODY: ReplaceTags: Offer (2) On 01/09/2016 08:54 AM, Al Viro wrote: > On Mon, Jan 04, 2016 at 10:20:05AM -0800, Dan Williams wrote: >> Historically we have waited for filesystem specific heuristics to >> attempt to guess when a block device is gone. Sometimes this works, but >> in other cases the system can hang waiting for the fs to trigger its >> shutdown protocol. >> >> The initial motivation for this investigation was to prevent DAX >> mappings (direct mmap access to persistent memory) from leaking past the >> lifetime of the hosting block device. However, Dave points out that >> these shutdown operations are needed in other scenarios. Quoting Dave: >> >> For example, if we detect a free space corruption during allocation, >> it is not safe to trust *any active mapping* because we can't trust >> that we having handed out the same block to multiple owners. Hence >> on such a filesystem shutdown, we have to prevent any new DAX >> mapping from occurring and invalidate all existing mappings as we >> cannot allow userspace to modify any data or metadata until we've >> resolved the corruption situation. >> >> The current block device shutdown sequence of del_gendisk + >> blk_cleanup_queue is problematic. We want to tell the fs after >> blk_cleanup_queue that there is no possibility of recovery, but by that >> time we have deleted partitions and lost the ability to find all the >> super-blocks on a block device. >> >> Introduce del_gendisk_queue to trigger ->quiesce() and ->bdi_gone() >> notifications to all the filesystems hosted on the disk. Where >> ->quiesce() are 'shutdown' operations while the bdev may still be alive, >> and ->bdi_gone() is a set of actions to take after the backing device >> is known to be permanently dead. > > Would you mind explaining what the hell is _the_ backing device > of a filesystem? What does that translate into in case of e.g. btrfs > spanning several disks? Or ext4 with journal on a different device, for > that matter? > > If anything, I would argue that filesystem is out of place here - > general situation is "IO on X may require IO on device Y and X needs to do > something when Y goes away". Consider e.g. /dev/loop backed by a device > that went away. Or by a file on fs that has run down the curtain and joined > the bleedin choir invisible. With another fs partially hosted by that > loopback device. Or by RAID0 containing said device. > > You are given Y and attempt to locate the affected X. _Then_ > you assume that X is a filesystem and has "something to be done" independent > from the role Y played for it, so you can pick that action from superblock > method. > > IMO you are placing the burden in the wrong place. _Recepient_ > knows what it depends upon and what should be done for each source of > trouble. So make it recepient's responsibility to request notifications. > At which point the superblock method goes away, along with the requirement > to handle all sources of trouble the same way, etc. > > What's more, things like RAID5 (also interested in knowing when > a component has been ripped out) might or might not decide to propagate > the event further - after all, that's exactly the point of redundancy. > > I'd look into something along the lines of notifier chain per > gendisk, with potential victims registering a callback when they decide > that from now on such and such device might screw them over... Fully support this. I was planning on something similar to transport device changes (resizing, topology change etc). And it might even be an idea to convert the block device events to a notifier chain, too. Dan, can you keep me in the loop here? Thanks. Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg) From dan.j.williams@intel.com Mon Jan 11 09:55:18 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6244D29DFC for ; Mon, 11 Jan 2016 09:55:18 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 264478F804C for ; Mon, 11 Jan 2016 07:55:18 -0800 (PST) X-ASG-Debug-ID: 1452527715-04bdf06f6b75260001-NocioJ Received: from mail-yk0-f172.google.com (mail-yk0-f172.google.com [209.85.160.172]) by cuda.sgi.com with ESMTP id Tg0lNu38sOHE4ejU (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 11 Jan 2016 07:55:15 -0800 (PST) X-Barracuda-Envelope-From: dan.j.williams@intel.com X-Barracuda-Apparent-Source-IP: 209.85.160.172 Received: by mail-yk0-f172.google.com with SMTP id a85so357523963ykb.1 for ; Mon, 11 Jan 2016 07:55:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Fo3D7nLKO0hgRFNo8zMa6Q0pfcqa992ElF7TRdz8gRQ=; b=ECHkp47KWt6ZoVVrGmgXjPgqz+nJQa2oqq52CXH4SXhz8yTCMCei83JanjhHw1N3Bt 9Su5Dj+Gt7Sje/U0HXGNb47FOO2OT52yuio3PHQgMH9LWKFfsUk/P4O6COvhb3uPeOm+ htaHDBTKS8SxGPg4glSDH3ygjiHQoeUABFglytIBJRYqfohAMr9KT/wukqVjh0Df9pOC TuDwf8hRD42EwS60+srP9VrUW932P+Zrv2R0Nz1LaaLWvhHWA+5uTAHG+lmWXBVzYCV7 okqOYgMCASCZIN9t9d4tsckBGnnSmWgS2Gp7u1vTV352BXKZuTxsBkfZsc5U+dvZq+h4 KL4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Fo3D7nLKO0hgRFNo8zMa6Q0pfcqa992ElF7TRdz8gRQ=; b=aBWNx9MR+ETZ5HStCkaMDx83u35ynDLhIcCSVo4XsGQU2i3z3eYFth+IovMxay5c8X kyTBNkp8eSL/Qbo+Ei0QrPwggrkuj6uZTa/sXWZ8A465YW4a2Edi83rG69hCggPD9Q+B eEvmHogReBQ//twlTJgNosRDSTHeY8ytIjIvu9eaTAu5MYXs/382491yjV2+B6Nq/X2/ eUYIYXQWelFU6xLAY9R3aNX5/VlqhsYO0lqHGZ3G1XaLuLgMBiVDQXvIwhCfhSY8Of6I NLnNzGguMDfTOLjuH65fnSjWab+dAiZTP3CZngvvefzHcEBZhc+STOxBzfjlorBxOBPV 4GRw== X-Gm-Message-State: ALoCoQlgvgrBXPDj3a86HovEOlf6zxsHSIP5YBF8d68JwxHwWn+2NumtPRCF4z85EwJFcK3WQu8WML32z7Da5YKbnIqF3Hbp/8s7tTRqxtjO2HKwPZ8OabU= MIME-Version: 1.0 X-Received: by 10.129.132.22 with SMTP id u22mr81987339ywf.198.1452527714981; Mon, 11 Jan 2016 07:55:14 -0800 (PST) Received: by 10.37.202.20 with HTTP; Mon, 11 Jan 2016 07:55:14 -0800 (PST) In-Reply-To: <5693C935.3060701@suse.de> References: <20160104181220.24118.96661.stgit@dwillia2-desk3.amr.corp.intel.com> <20160104182005.24118.50361.stgit@dwillia2-desk3.amr.corp.intel.com> <20160109075414.GA5008@ZenIV.linux.org.uk> <5693C935.3060701@suse.de> Date: Mon, 11 Jan 2016 07:55:14 -0800 Message-ID: Subject: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life From: Dan Williams X-ASG-Orig-Subj: Re: [resend PATCH 1/3] block, fs: reliably communicate bdev end-of-life To: Hannes Reinecke Cc: Al Viro , XFS Developers , linux-block@vger.kernel.org, linux-nvdimm , Dave Chinner , Jens Axboe , Jan Kara , linux-fsdevel , Matthew Wilcox , Ross Zwisler Content-Type: text/plain; charset=UTF-8 X-Barracuda-Connect: mail-yk0-f172.google.com[209.85.160.172] X-Barracuda-Start-Time: 1452527715 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26032 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature On Mon, Jan 11, 2016 at 7:24 AM, Hannes Reinecke wrote: > On 01/09/2016 08:54 AM, Al Viro wrote: >> >> On Mon, Jan 04, 2016 at 10:20:05AM -0800, Dan Williams wrote: >>> >>> Historically we have waited for filesystem specific heuristics to >>> attempt to guess when a block device is gone. Sometimes this works, but >>> in other cases the system can hang waiting for the fs to trigger its >>> shutdown protocol. >>> >>> The initial motivation for this investigation was to prevent DAX >>> mappings (direct mmap access to persistent memory) from leaking past the >>> lifetime of the hosting block device. However, Dave points out that >>> these shutdown operations are needed in other scenarios. Quoting Dave: >>> >>> For example, if we detect a free space corruption during allocation, >>> it is not safe to trust *any active mapping* because we can't trust >>> that we having handed out the same block to multiple owners. Hence >>> on such a filesystem shutdown, we have to prevent any new DAX >>> mapping from occurring and invalidate all existing mappings as we >>> cannot allow userspace to modify any data or metadata until we've >>> resolved the corruption situation. >>> >>> The current block device shutdown sequence of del_gendisk + >>> blk_cleanup_queue is problematic. We want to tell the fs after >>> blk_cleanup_queue that there is no possibility of recovery, but by that >>> time we have deleted partitions and lost the ability to find all the >>> super-blocks on a block device. >>> >>> Introduce del_gendisk_queue to trigger ->quiesce() and ->bdi_gone() >>> notifications to all the filesystems hosted on the disk. Where >>> ->quiesce() are 'shutdown' operations while the bdev may still be alive, >>> and ->bdi_gone() is a set of actions to take after the backing device >>> is known to be permanently dead. >> >> >> Would you mind explaining what the hell is _the_ backing device >> of a filesystem? What does that translate into in case of e.g. btrfs >> spanning several disks? Or ext4 with journal on a different device, for >> that matter? >> >> If anything, I would argue that filesystem is out of place here - >> general situation is "IO on X may require IO on device Y and X needs to do >> something when Y goes away". Consider e.g. /dev/loop backed by a device >> that went away. Or by a file on fs that has run down the curtain and >> joined >> the bleedin choir invisible. With another fs partially hosted by that >> loopback device. Or by RAID0 containing said device. >> >> You are given Y and attempt to locate the affected X. _Then_ >> you assume that X is a filesystem and has "something to be done" >> independent >> from the role Y played for it, so you can pick that action from superblock >> method. >> >> IMO you are placing the burden in the wrong place. _Recepient_ >> knows what it depends upon and what should be done for each source of >> trouble. So make it recepient's responsibility to request notifications. >> At which point the superblock method goes away, along with the requirement >> to handle all sources of trouble the same way, etc. >> >> What's more, things like RAID5 (also interested in knowing when >> a component has been ripped out) might or might not decide to propagate >> the event further - after all, that's exactly the point of redundancy. >> >> I'd look into something along the lines of notifier chain per >> gendisk, with potential victims registering a callback when they decide >> that from now on such and such device might screw them over... > > > Fully support this. I was planning on something similar to transport device > changes (resizing, topology change etc). > > And it might even be an idea to convert the block device events to a > notifier chain, too. > > Dan, can you keep me in the loop here? Yes, will do. From jack@suse.cz Mon Jan 11 09:57:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id CDB6D29DFC for ; Mon, 11 Jan 2016 09:57:01 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id BBF348F804B for ; Mon, 11 Jan 2016 07:57:01 -0800 (PST) X-ASG-Debug-ID: 1452527818-04cb6c75de71850001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 5uENVmvk6flEc5dA (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 11 Jan 2016 07:56:59 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 5045AAD0E; Mon, 11 Jan 2016 15:56:57 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id 63B55823D8; Mon, 11 Jan 2016 16:57:09 +0100 (CET) Date: Mon, 11 Jan 2016 16:57:09 +0100 From: Jan Kara To: Eric Sandeen Cc: fsdevel , xfs@oss.sgi.com, Jan Kara Subject: Re: [PATCH 3/4] xfs: Factor xfs_seek_hole_data into helper Message-ID: <20160111155709.GI6262@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH 3/4] xfs: Factor xfs_seek_hole_data into helper References: <568FEA2C.6080708@redhat.com> <568FEAD1.5040302@sandeen.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <568FEAD1.5040302@sandeen.net> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452527819 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26032 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Fri 08-01-16 10:58:57, Eric Sandeen wrote: > Factor xfs_seek_hole_data into an unlocked helper which takes > an xfs inode rather than a file for internal use. > > Also allow specification of "end" - the vfs lseek interface is > defined such that any offset past eof/i_size shall return -ENXIO, > but we will use this for quota code which does not maintain i_size, > and we want to be able to SEEK_DATA past i_size as well. So the > lseek path can send in i_size, and the quota code can determine > its own ending offset. This patch looks whitespace damaged making it difficult to read... Honza > Signed-off-by: Eric Sandeen > --- > fs/xfs/xfs_file.c | 82 ++++++++++++++++++++++++++++++++++++---------------- > fs/xfs/xfs_inode.h | 2 + > 2 files changed, 59 insertions(+), 25 deletions(-) > > diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c > index ebe9b82..5dc7113 100644 > --- a/fs/xfs/xfs_file.c > +++ b/fs/xfs/xfs_file.c > @@ -1337,31 +1337,31 @@ out: > return found; > } > > -STATIC loff_t > -xfs_seek_hole_data( > - struct file *file, > +/* > + * caller must lock inode with xfs_ilock_data_map_shared, > + * can we craft an appropriate ASSERT? > + * > + * end is because the VFS-level lseek interface is defined such that any > + * offset past i_size shall return -ENXIO, but we use this for quota code > + * which does not maintain i_size, and we want to SEEK_DATA past i_size. > + */ > +loff_t > +__xfs_seek_hole_data( > + struct inode *inode, > loff_t start, > + loff_t end, > int whence) > { > - struct inode *inode = file->f_mapping->host; > struct xfs_inode *ip = XFS_I(inode); > struct xfs_mount *mp = ip->i_mount; > loff_t uninitialized_var(offset); > - xfs_fsize_t isize; > xfs_fileoff_t fsbno; > - xfs_filblks_t end; > - uint lock; > + xfs_filblks_t lastbno; > int error; > > - if (XFS_FORCED_SHUTDOWN(mp)) > - return -EIO; > - > - lock = xfs_ilock_data_map_shared(ip); > - > - isize = i_size_read(inode); > - if (start >= isize) { > + if (start >= end) { > error = -ENXIO; > - goto out_unlock; > + goto out_error; > } > > /* > @@ -1369,22 +1369,22 @@ xfs_seek_hole_data( > * by fsbno to the end block of the file. > */ > fsbno = XFS_B_TO_FSBT(mp, start); > - end = XFS_B_TO_FSB(mp, isize); > + lastbno = XFS_B_TO_FSB(mp, end); > > for (;;) { > struct xfs_bmbt_irec map[2]; > int nmap = 2; > unsigned int i; > > - error = xfs_bmapi_read(ip, fsbno, end - fsbno, map, &nmap, > + error = xfs_bmapi_read(ip, fsbno, lastbno - fsbno, map, &nmap, > XFS_BMAPI_ENTIRE); > if (error) > - goto out_unlock; > + goto out_error; > > /* No extents at given offset, must be beyond EOF */ > if (nmap == 0) { > error = -ENXIO; > - goto out_unlock; > + goto out_error; > } > > for (i = 0; i < nmap; i++) { > @@ -1426,7 +1426,7 @@ xfs_seek_hole_data( > * hole at the end of any file). > */ > if (whence == SEEK_HOLE) { > - offset = isize; > + offset = end; > break; > } > /* > @@ -1434,7 +1434,7 @@ xfs_seek_hole_data( > */ > ASSERT(whence == SEEK_DATA); > error = -ENXIO; > - goto out_unlock; > + goto out_error; > } > > ASSERT(i > 1); > @@ -1445,14 +1445,14 @@ xfs_seek_hole_data( > */ > fsbno = map[i - 1].br_startoff + map[i - 1].br_blockcount; > start = XFS_FSB_TO_B(mp, fsbno); > - if (start >= isize) { > + if (start >= end) { > if (whence == SEEK_HOLE) { > - offset = isize; > + offset = end; > break; > } > ASSERT(whence == SEEK_DATA); > error = -ENXIO; > - goto out_unlock; > + goto out_error; > } > } > > @@ -1464,7 +1464,39 @@ out: > * situation in particular. > */ > if (whence == SEEK_HOLE) > - offset = min_t(loff_t, offset, isize); > + offset = min_t(loff_t, offset, end); > + > + return offset; > + > +out_error: > + return error; > +} > + > +STATIC loff_t > +xfs_seek_hole_data( > + struct file *file, > + loff_t start, > + int whence) > +{ > + struct inode *inode = file->f_mapping->host; > + struct xfs_inode *ip = XFS_I(inode); > + struct xfs_mount *mp = ip->i_mount; > + uint lock; > + loff_t offset, end; > + int error = 0; > + > + if (XFS_FORCED_SHUTDOWN(mp)) > + return -EIO; > + > + lock = xfs_ilock_data_map_shared(ip); > + > + end = i_size_read(inode); > + offset = __xfs_seek_hole_data(inode, start, end, whence); > + if (offset < 0) { > + error = offset; > + goto out_unlock; > + } > + > offset = vfs_setpos(file, offset, inode->i_sb->s_maxbytes); > > out_unlock: > diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h > index ca9e119..ed7e933 100644 > --- a/fs/xfs/xfs_inode.h > +++ b/fs/xfs/xfs_inode.h > @@ -437,6 +437,8 @@ int xfs_update_prealloc_flags(struct xfs_inode *ip, > int xfs_zero_eof(struct xfs_inode *ip, xfs_off_t offset, > xfs_fsize_t isize, bool *did_zeroing); > int xfs_iozero(struct xfs_inode *ip, loff_t pos, size_t count); > +loff_t __xfs_seek_hole_data(struct inode *inode, loff_t start, > + loff_t eof, int whence); > > > /* from xfs_iops.c */ > -- > 1.7.1 > > -- Jan Kara SUSE Labs, CR From sandeen@sandeen.net Mon Jan 11 10:01:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 52D1D29DFC for ; Mon, 11 Jan 2016 10:01:16 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 329AC8F804B for ; Mon, 11 Jan 2016 08:01:12 -0800 (PST) X-ASG-Debug-ID: 1452528070-04cbb05f7d75d60001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id jS8P5G6cZ6HmwOqv for ; Mon, 11 Jan 2016 08:01:10 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id CF5976CEEE07; Mon, 11 Jan 2016 10:01:09 -0600 (CST) Subject: [PATCH 3/4 V3] xfs: Factor xfs_seek_hole_data into helper To: fsdevel , xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 3/4 V3] xfs: Factor xfs_seek_hole_data into helper References: <568FEA2C.6080708@redhat.com> <568FEAD1.5040302@sandeen.net> Cc: Jan Kara From: Eric Sandeen Message-ID: <5693D1C6.6010606@sandeen.net> Date: Mon, 11 Jan 2016 10:01:10 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <568FEAD1.5040302@sandeen.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452528070 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26032 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Factor xfs_seek_hole_data into an unlocked helper which takes an xfs inode rather than a file for internal use. Also allow specification of "end" - the vfs lseek interface is defined such that any offset past eof/i_size shall return -ENXIO, but we will use this for quota code which does not maintain i_size, and we want to be able to SEEK_DATA past i_size as well. So the lseek path can send in i_size, and the quota code can determine its own ending offset. Signed-off-by: Eric Sandeen --- (Ugh, no idea what happened to that formatting!) (V3 same as V2 but actually hitting both lists - email is hard!) fs/xfs/xfs_file.c | 82 ++++++++++++++++++++++++++++++++++++---------------- fs/xfs/xfs_inode.h | 2 + 2 files changed, 59 insertions(+), 25 deletions(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index ebe9b82..5dc7113 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1337,31 +1337,31 @@ out: return found; } -STATIC loff_t -xfs_seek_hole_data( - struct file *file, +/* + * caller must lock inode with xfs_ilock_data_map_shared, + * can we craft an appropriate ASSERT? + * + * end is because the VFS-level lseek interface is defined such that any + * offset past i_size shall return -ENXIO, but we use this for quota code + * which does not maintain i_size, and we want to SEEK_DATA past i_size. + */ +loff_t +__xfs_seek_hole_data( + struct inode *inode, loff_t start, + loff_t end, int whence) { - struct inode *inode = file->f_mapping->host; struct xfs_inode *ip = XFS_I(inode); struct xfs_mount *mp = ip->i_mount; loff_t uninitialized_var(offset); - xfs_fsize_t isize; xfs_fileoff_t fsbno; - xfs_filblks_t end; - uint lock; + xfs_filblks_t lastbno; int error; - if (XFS_FORCED_SHUTDOWN(mp)) - return -EIO; - - lock = xfs_ilock_data_map_shared(ip); - - isize = i_size_read(inode); - if (start >= isize) { + if (start >= end) { error = -ENXIO; - goto out_unlock; + goto out_error; } /* @@ -1369,22 +1369,22 @@ xfs_seek_hole_data( * by fsbno to the end block of the file. */ fsbno = XFS_B_TO_FSBT(mp, start); - end = XFS_B_TO_FSB(mp, isize); + lastbno = XFS_B_TO_FSB(mp, end); for (;;) { struct xfs_bmbt_irec map[2]; int nmap = 2; unsigned int i; - error = xfs_bmapi_read(ip, fsbno, end - fsbno, map, &nmap, + error = xfs_bmapi_read(ip, fsbno, lastbno - fsbno, map, &nmap, XFS_BMAPI_ENTIRE); if (error) - goto out_unlock; + goto out_error; /* No extents at given offset, must be beyond EOF */ if (nmap == 0) { error = -ENXIO; - goto out_unlock; + goto out_error; } for (i = 0; i < nmap; i++) { @@ -1426,7 +1426,7 @@ xfs_seek_hole_data( * hole at the end of any file). */ if (whence == SEEK_HOLE) { - offset = isize; + offset = end; break; } /* @@ -1434,7 +1434,7 @@ xfs_seek_hole_data( */ ASSERT(whence == SEEK_DATA); error = -ENXIO; - goto out_unlock; + goto out_error; } ASSERT(i > 1); @@ -1445,14 +1445,14 @@ xfs_seek_hole_data( */ fsbno = map[i - 1].br_startoff + map[i - 1].br_blockcount; start = XFS_FSB_TO_B(mp, fsbno); - if (start >= isize) { + if (start >= end) { if (whence == SEEK_HOLE) { - offset = isize; + offset = end; break; } ASSERT(whence == SEEK_DATA); error = -ENXIO; - goto out_unlock; + goto out_error; } } @@ -1464,7 +1464,39 @@ out: * situation in particular. */ if (whence == SEEK_HOLE) - offset = min_t(loff_t, offset, isize); + offset = min_t(loff_t, offset, end); + + return offset; + +out_error: + return error; +} + +STATIC loff_t +xfs_seek_hole_data( + struct file *file, + loff_t start, + int whence) +{ + struct inode *inode = file->f_mapping->host; + struct xfs_inode *ip = XFS_I(inode); + struct xfs_mount *mp = ip->i_mount; + uint lock; + loff_t offset, end; + int error = 0; + + if (XFS_FORCED_SHUTDOWN(mp)) + return -EIO; + + lock = xfs_ilock_data_map_shared(ip); + + end = i_size_read(inode); + offset = __xfs_seek_hole_data(inode, start, end, whence); + if (offset < 0) { + error = offset; + goto out_unlock; + } + offset = vfs_setpos(file, offset, inode->i_sb->s_maxbytes); out_unlock: diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h index ca9e119..ed7e933 100644 --- a/fs/xfs/xfs_inode.h +++ b/fs/xfs/xfs_inode.h @@ -437,6 +437,8 @@ int xfs_update_prealloc_flags(struct xfs_inode *ip, int xfs_zero_eof(struct xfs_inode *ip, xfs_off_t offset, xfs_fsize_t isize, bool *did_zeroing); int xfs_iozero(struct xfs_inode *ip, loff_t pos, size_t count); +loff_t __xfs_seek_hole_data(struct inode *inode, loff_t start, + loff_t eof, int whence); /* from xfs_iops.c */ -- 1.7.1 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Mon Jan 11 10:03:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 5F48629DF6 for ; Mon, 11 Jan 2016 10:03:41 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id D201DAC001 for ; Mon, 11 Jan 2016 08:03:37 -0800 (PST) X-ASG-Debug-ID: 1452528215-04cbb05f7d75ea0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id wbgYBA3iShNMfb4L (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 08:03:36 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id B0926C0B7E05; Mon, 11 Jan 2016 16:03:35 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BG3Z8B011749; Mon, 11 Jan 2016 11:03:35 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id C416B1201A8; Mon, 11 Jan 2016 11:03:34 -0500 (EST) Date: Mon, 11 Jan 2016 11:03:34 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH 1/2] xfs: inode recovery readahead can race with inode buffer creation Message-ID: <20160111160332.GA4118@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH 1/2] xfs: inode recovery readahead can race with inode buffer creation References: <1452482694-27314-1-git-send-email-david@fromorbit.com> <1452482694-27314-2-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452482694-27314-2-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452528216 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Mon, Jan 11, 2016 at 02:24:53PM +1100, Dave Chinner wrote: > From: Dave Chinner > > When we do inode readahead in log recovery, we do can do the > readahead before we've replayed the icreate transaction that stamps > the buffer with inode cores. The inode readahead verifier catches > this and marks the buffer as !done to indicate that it doesn't yet > contain valid inodes. > > In adding buffer error notification (i.e. setting b_error = -EIO at > the same time as as we clear the done flag) to such a readahead > verifier failure, we can then get subsequent inode recovery failing > with this error: > > XFS (dm-0): metadata I/O error: block 0xa00060 ("xlog_recover_do..(read#2)") error 5 numblks 32 > > This occurs when readahead completion races with icreate item replay > such as: > > inode readahead > find buffer > lock buffer > submit RA io > .... > icreate recovery > xfs_trans_get_buffer > find buffer > lock buffer > > ..... > > fails verifier > clear XBF_DONE > set bp->b_error = -EIO > release and unlock buffer > > icreate initialises buffer > marks buffer as done > adds buffer to delayed write queue > releases buffer > > At this point, we have an initialised inode buffer that is up to > date but has an -EIO state registered against it. When we finally > get to recovering an inode in that buffer: > > inode item recovery > xfs_trans_read_buffer > find buffer > lock buffer > sees XBF_DONE is set, returns buffer > sees bp->b_error is set > fail log recovery! > > Essentially, we need xfs_trans_get_buf_map() to clear the error status of > the buffer when doing a lookup. This function returns uninitialised > buffers, so the buffer returned can not be in an error state and > none of the code that uses this function expects b_error to be set > on return. Indeed, there is an ASSERT(!bp->b_error); in the > transaction case in xfs_trans_get_buf_map() that would have caught > this if log recovery used transactions.... > > This patch firstly changes the inode readahead failure to set -EIO > on the buffer, and secondly changes xfs_buf_get_map() to never > return a buffer with an error state set so this first change doesn't > cause unexpected log recovery failures. > > Signed-off-by: Dave Chinner > --- Reviewed-by: Brian Foster > fs/xfs/libxfs/xfs_inode_buf.c | 12 +++++++----- > fs/xfs/xfs_buf.c | 7 +++++++ > 2 files changed, 14 insertions(+), 5 deletions(-) > > diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c > index 1b8d98a..ff17c48 100644 > --- a/fs/xfs/libxfs/xfs_inode_buf.c > +++ b/fs/xfs/libxfs/xfs_inode_buf.c > @@ -62,11 +62,12 @@ xfs_inobp_check( > * has not had the inode cores stamped into it. Hence for readahead, the buffer > * may be potentially invalid. > * > - * If the readahead buffer is invalid, we don't want to mark it with an error, > - * but we do want to clear the DONE status of the buffer so that a followup read > - * will re-read it from disk. This will ensure that we don't get an unnecessary > - * warnings during log recovery and we don't get unnecssary panics on debug > - * kernels. > + * If the readahead buffer is invalid, we need to mark it with an error and > + * clear the DONE status of the buffer so that a followup read will re-read it > + * from disk. We don't report the error otherwise to avoid warnings during log > + * recovery and we don't get unnecssary panics on debug kernels. We use EIO here > + * because all we want to do is say readahead failed; there is no-one to report > + * the error to, so this will distinguish it from a non-ra verifier failure. > */ > static void > xfs_inode_buf_verify( > @@ -93,6 +94,7 @@ xfs_inode_buf_verify( > XFS_RANDOM_ITOBP_INOTOBP))) { > if (readahead) { > bp->b_flags &= ~XBF_DONE; > + xfs_buf_ioerror(bp, -EIO); > return; > } > > diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c > index 45a8ea7..ae86b16 100644 > --- a/fs/xfs/xfs_buf.c > +++ b/fs/xfs/xfs_buf.c > @@ -604,6 +604,13 @@ found: > } > } > > + /* > + * Clear b_error if this is a lookup from a caller that doesn't expect > + * valid data to be found in the buffer. > + */ > + if (!(flags & XBF_READ)) > + xfs_buf_ioerror(bp, 0); > + > XFS_STATS_INC(target->bt_mount, xb_get); > trace_xfs_buf_get(bp, flags, _RET_IP_); > return bp; > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From sandeen@sandeen.net Mon Jan 11 10:07:27 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 59E7D29DF6 for ; Mon, 11 Jan 2016 10:07:27 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3B5D7304059 for ; Mon, 11 Jan 2016 08:07:24 -0800 (PST) X-ASG-Debug-ID: 1452528442-04cbb05f7c76050001-NocioJ Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id vXzrKDttjHZuABNG for ; Mon, 11 Jan 2016 08:07:23 -0800 (PST) X-Barracuda-Envelope-From: sandeen@sandeen.net X-Barracuda-Apparent-Source-IP: 63.231.237.45 Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by sandeen.net (Postfix) with ESMTPSA id A962C6CEEE07; Mon, 11 Jan 2016 10:07:22 -0600 (CST) Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 To: Jan Kara , Christoph Hellwig X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 References: <568FEA2C.6080708@redhat.com> <20160109072600.GA21636@infradead.org> <20160111132617.GD6262@quack.suse.cz> Cc: fsdevel , Eric Sandeen , xfs@oss.sgi.com From: Eric Sandeen Message-ID: <5693D33A.5090307@sandeen.net> Date: Mon, 11 Jan 2016 10:07:22 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <20160111132617.GD6262@quack.suse.cz> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Barracuda-Connect: sandeen.net[63.231.237.45] X-Barracuda-Start-Time: 1452528442 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26032 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On 1/11/16 7:26 AM, Jan Kara wrote: > On Fri 08-01-16 23:26:00, Christoph Hellwig wrote: >> On Fri, Jan 08, 2016 at 10:56:12AM -0600, Eric Sandeen wrote: >>> This adds a new quotactl, Q_XGETQUOTA2. >>> >>> Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will >>> return quota information for the id equal to or greater than >>> the id requested. In other words, if the specified id has >>> no quota, the command will return quota information for the >>> next higher id which does have a quota set. If no higher id >>> has an active quota, -ESRCH is returned. >> >> Please add a flags argument to Q_XGETQUOTA2, and then make the >> new behavior the first flag. Keep Q_XGETQUOTA behavior for the >> flag-less case. That way we get future etensibility for free. > > So this is what I wanted to suggest at first as well. What I somewhat > dislike is that 'addr' must now point to something like: > > struct getquota_args { > __u64 flags; > struct fs_disk_quota ret; > }; Ok... > which is not as nice as passing pointer to fs_disk_quota directly. But > probably still OK. So I agree with the flags idea. I understand that flags are for future-proofing, for possibly-not-yet-imagined scenarios, but on the other hand, I wonder how many different flavors of "get me a disk quota" we really expect to have? > Another issue is that OCFS2 and ext4 with quota in hidden inodes would need > call with these capabilities as well (they have the same problem as xfs). > For reporting VFS quota we use bytes for space (since some filesystems need > it) and we don't need RT fields (easy so zero-fill) and warning fields > (there zeros may be confusing). So we would need something like struct > qc_dqblk (currently only internal) extended with ID and maybe flags field > XFS is using to return quota type which can contain all the information and > tell which info is actually valid as return structure. Hohum, yeah - I forgot that ext4 has (can have?) a hidden quota inode. And I didn't know about OCFS2. And TBH I did the xfs-specific one because it was trivial to extend, with the id already in the returned quota struct. So ok, it sounds like this needs a more significant overhaul at the vfs quota level... -Eric > > Honza > From jack@suse.cz Mon Jan 11 10:28:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 513A029DF6 for ; Mon, 11 Jan 2016 10:28:00 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 0F2A5304053 for ; Mon, 11 Jan 2016 08:27:59 -0800 (PST) X-ASG-Debug-ID: 1452529675-04bdf06f6b761b0001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id WGJYY3NyxEjonxmo (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 11 Jan 2016 08:27:57 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 78B01ACD2; Mon, 11 Jan 2016 16:27:54 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id 4385A823D8; Mon, 11 Jan 2016 17:28:07 +0100 (CET) Date: Mon, 11 Jan 2016 17:28:07 +0100 From: Jan Kara To: Eric Sandeen Cc: Jan Kara , Christoph Hellwig , fsdevel , Eric Sandeen , xfs@oss.sgi.com Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 Message-ID: <20160111162807.GK6262@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2 References: <568FEA2C.6080708@redhat.com> <20160109072600.GA21636@infradead.org> <20160111132617.GD6262@quack.suse.cz> <5693D33A.5090307@sandeen.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5693D33A.5090307@sandeen.net> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452529677 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26033 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Mon 11-01-16 10:07:22, Eric Sandeen wrote: > On 1/11/16 7:26 AM, Jan Kara wrote: > > On Fri 08-01-16 23:26:00, Christoph Hellwig wrote: > >> On Fri, Jan 08, 2016 at 10:56:12AM -0600, Eric Sandeen wrote: > >>> This adds a new quotactl, Q_XGETQUOTA2. > >>> > >>> Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will > >>> return quota information for the id equal to or greater than > >>> the id requested. In other words, if the specified id has > >>> no quota, the command will return quota information for the > >>> next higher id which does have a quota set. If no higher id > >>> has an active quota, -ESRCH is returned. > >> > >> Please add a flags argument to Q_XGETQUOTA2, and then make the > >> new behavior the first flag. Keep Q_XGETQUOTA behavior for the > >> flag-less case. That way we get future etensibility for free. > > > > So this is what I wanted to suggest at first as well. What I somewhat > > dislike is that 'addr' must now point to something like: > > > > struct getquota_args { > > __u64 flags; > > struct fs_disk_quota ret; > > }; > > Ok... > > > which is not as nice as passing pointer to fs_disk_quota directly. But > > probably still OK. So I agree with the flags idea. > > I understand that flags are for future-proofing, for possibly-not-yet-imagined > scenarios, but on the other hand, I wonder how many different flavors of > "get me a disk quota" we really expect to have? > > > Another issue is that OCFS2 and ext4 with quota in hidden inodes would need > > call with these capabilities as well (they have the same problem as xfs). > > For reporting VFS quota we use bytes for space (since some filesystems need > > it) and we don't need RT fields (easy so zero-fill) and warning fields > > (there zeros may be confusing). So we would need something like struct > > qc_dqblk (currently only internal) extended with ID and maybe flags field > > XFS is using to return quota type which can contain all the information and > > tell which info is actually valid as return structure. > > Hohum, yeah - I forgot that ext4 has (can have?) a hidden quota inode. > And I didn't know about OCFS2. > > And TBH I did the xfs-specific one because it was trivial to extend, with the > id already in the returned quota struct. > > So ok, it sounds like this needs a more significant overhaul at the vfs > quota level... Actually, what I want from you is just an interface which is usable for VFS quotas as well since I'd like to avoid adding GETQUOTA2 quotactl shortly after XGETQUOTA2 :). Using id as a cooking is fine for VFS quotas as well so that part can stay. I can have a look at implementing the new handler for VFS quota format(s) which should be pretty easy once the interface part is merged or you can do it if you're interested in learning more about quota internals. Honza -- Jan Kara SUSE Labs, CR From mchristi@redhat.com Mon Jan 11 14:21:48 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 7FFB57F37 for ; Mon, 11 Jan 2016 14:21:48 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 6ECE3304062 for ; Mon, 11 Jan 2016 12:21:48 -0800 (PST) X-ASG-Debug-ID: 1452543706-04cb6c75de79020001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 1uBLkfuBDlCeqzJ2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:21:47 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 2D94C3F3D4; Mon, 11 Jan 2016 20:21:46 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEV008385; Mon, 11 Jan 2016 15:21:43 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields Date: Mon, 11 Jan 2016 14:21:00 -0600 X-ASG-Orig-Subj: [PATCH 02/35] block: add REQ_OP definitions and bi_op/op fields Message-Id: <1452543693-4440-3-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543707 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The following patches separate the operation (write, read, discard, etc) from the flags in bi_rw/cmd_flags. This patch adds definitions for request/bio operations, adds fields to the request/bio to set them, and some temporary compat code so the kernel/modules can use either one. In the final patches this compat code will be removed when everything is converted. Also, in this patch the REQ_OPs match the REQ rq_flag_bits ones for compat reasons while all the code is converted in this set. In the last patches that will also be removed. Signed-off-by: Mike Christie --- block/blk-core.c | 19 ++++++++++++++++--- include/linux/blk_types.h | 15 ++++++++++++++- include/linux/blkdev.h | 1 + include/linux/fs.h | 37 +++++++++++++++++++++++++++++++++++-- 4 files changed, 66 insertions(+), 6 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 9b887e3..954a450 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1697,7 +1697,8 @@ void init_request_from_bio(struct request *req, struct bio *bio) { req->cmd_type = REQ_TYPE_FS; - req->cmd_flags |= bio->bi_rw & REQ_COMMON_MASK; + /* tmp compat. Allow users to set bi_op or bi_rw */ + req->cmd_flags |= (bio->bi_rw | bio->bi_op) & REQ_COMMON_MASK; if (bio->bi_rw & REQ_RAHEAD) req->cmd_flags |= REQ_FAILFAST_MASK; @@ -2032,6 +2033,12 @@ blk_qc_t generic_make_request(struct bio *bio) struct bio_list bio_list_on_stack; blk_qc_t ret = BLK_QC_T_NONE; + /* tmp compat. Allow users to set either one or both. + * This will be removed when we have converted + * everyone in the next patches. + */ + bio->bi_rw |= bio->bi_op; + if (!generic_make_request_checks(bio)) goto out; @@ -2101,6 +2108,12 @@ EXPORT_SYMBOL(generic_make_request); */ blk_qc_t submit_bio(struct bio *bio) { + /* tmp compat. Allow users to set either one or both. + * This will be removed when we have converted + * everyone in the next patches. + */ + bio->bi_rw |= bio->bi_op; + /* * If it's a regular read/write or a barrier with data attached, * go through the normal accounting stuff before submission. @@ -2972,8 +2985,8 @@ EXPORT_SYMBOL_GPL(__blk_end_request_err); void blk_rq_bio_prep(struct request_queue *q, struct request *rq, struct bio *bio) { - /* Bit 0 (R/W) is identical in rq->cmd_flags and bio->bi_rw */ - rq->cmd_flags |= bio->bi_rw & REQ_WRITE; + /* tmp compat. Allow users to set bi_op or bi_rw */ + rq->cmd_flags |= bio_data_dir(bio); if (bio_has_data(bio)) rq->nr_phys_segments = bio_phys_segments(q, bio); diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 86a38ea..6e49c91 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -48,9 +48,15 @@ struct bio { struct block_device *bi_bdev; unsigned int bi_flags; /* status, command, etc */ int bi_error; - unsigned long bi_rw; /* bottom bits READ/WRITE, + unsigned long bi_rw; /* bottom bits rq_flags_bits * top bits priority */ + /* + * this will be a u8 in the next patches and bi_rw can be shrunk to + * a u32. For compat in these transistional patches op is a int here. + */ + int bi_op; /* REQ_OP */ + struct bvec_iter bi_iter; @@ -242,6 +248,13 @@ enum rq_flag_bits { #define REQ_HASHED (1ULL << __REQ_HASHED) #define REQ_MQ_INFLIGHT (1ULL << __REQ_MQ_INFLIGHT) +enum req_op { + REQ_OP_READ, + REQ_OP_WRITE = REQ_WRITE, + REQ_OP_DISCARD = REQ_DISCARD, + REQ_OP_WRITE_SAME = REQ_WRITE_SAME, +}; + typedef unsigned int blk_qc_t; #define BLK_QC_T_NONE -1U #define BLK_QC_T_SHIFT 16 diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 29189ae..35b9eb3 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -96,6 +96,7 @@ struct request { struct request_queue *q; struct blk_mq_ctx *mq_ctx; + int op; u64 cmd_flags; unsigned cmd_type; unsigned long atomic_flags; diff --git a/include/linux/fs.h b/include/linux/fs.h index afa9df4..afca758 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2405,15 +2405,48 @@ extern void make_bad_inode(struct inode *); extern bool is_bad_inode(struct inode *); #ifdef CONFIG_BLOCK + +static inline bool op_is_write(int op) +{ + switch (op) { + case REQ_OP_WRITE: + case REQ_OP_WRITE_SAME: + case REQ_OP_DISCARD: + return true; + default: + return false; + } +} + /* * return READ, READA, or WRITE */ -#define bio_rw(bio) ((bio)->bi_rw & (RW_MASK | RWA_MASK)) +static inline int bio_rw(struct bio *bio) +{ + /* + * tmp cpmpat. Allow users to set either op or rw, until + * all code is converted in the next patches. + */ + if (op_is_write(bio->bi_op)) + return WRITE; + + return bio->bi_rw & (RW_MASK | RWA_MASK); +} /* * return data direction, READ or WRITE */ -#define bio_data_dir(bio) ((bio)->bi_rw & 1) +static inline int bio_data_dir(struct bio *bio) +{ + /* + * tmp cpmpat. Allow users to set either op or rw, until + * all code is converted in the next patches. + */ + if (op_is_write(bio->bi_op)) + return WRITE; + + return bio->bi_rw & 1; +} extern void check_disk_size_change(struct gendisk *disk, struct block_device *bdev); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:21:49 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id AA7B57F37 for ; Mon, 11 Jan 2016 14:21:49 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 9BFE3304053 for ; Mon, 11 Jan 2016 12:21:49 -0800 (PST) X-ASG-Debug-ID: 1452543704-04cb6c75dd79010001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id fNlKM8k9ELrp4Gia (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:21:44 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id C161AABB0C; Mon, 11 Jan 2016 20:21:43 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEU008385; Mon, 11 Jan 2016 15:21:41 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio Date: Mon, 11 Jan 2016 14:20:59 -0600 X-ASG-Orig-Subj: [PATCH 01/35] block/fs/drivers: remove rw argument from submit_bio Message-Id: <1452543693-4440-2-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543704 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This has callers of submit_bio/submit_bio_wait set the bio->bi_rw instead of passing it in. This makes that use the same as generic_make_request and how we set the other bio fields. v2. 1. Set bi_rw instead of ORing it. For cloned bios, I still OR it to keep the old behavior incase there bits we wanted to keep. Signed-off-by: Mike Christie --- block/bio.c | 7 +++---- block/blk-core.c | 11 ++++------- block/blk-flush.c | 3 ++- block/blk-lib.c | 9 ++++++--- drivers/block/drbd/drbd_actlog.c | 2 +- drivers/block/drbd/drbd_bitmap.c | 4 ++-- drivers/block/floppy.c | 3 ++- drivers/block/xen-blkback/blkback.c | 4 +++- drivers/block/xen-blkfront.c | 4 ++-- drivers/md/bcache/debug.c | 6 ++++-- drivers/md/bcache/journal.c | 2 +- drivers/md/bcache/super.c | 4 ++-- drivers/md/dm-bufio.c | 3 ++- drivers/md/dm-io.c | 3 ++- drivers/md/dm-log-writes.c | 9 ++++++--- drivers/md/dm-thin.c | 3 ++- drivers/md/md.c | 10 +++++++--- drivers/md/raid1.c | 3 ++- drivers/md/raid10.c | 4 +++- drivers/md/raid5-cache.c | 7 ++++--- drivers/target/target_core_iblock.c | 24 +++++++++++++----------- fs/btrfs/check-integrity.c | 18 ++++++++++-------- fs/btrfs/check-integrity.h | 4 ++-- fs/btrfs/disk-io.c | 3 ++- fs/btrfs/extent_io.c | 7 ++++--- fs/btrfs/raid56.c | 16 +++++++++++----- fs/btrfs/scrub.c | 16 +++++++++++----- fs/btrfs/volumes.c | 14 +++++++------- fs/buffer.c | 3 ++- fs/direct-io.c | 3 ++- fs/ext4/crypto.c | 3 ++- fs/ext4/page-io.c | 3 ++- fs/ext4/readpage.c | 9 +++++---- fs/f2fs/data.c | 13 ++++++++----- fs/f2fs/segment.c | 6 ++++-- fs/gfs2/lops.c | 3 ++- fs/gfs2/meta_io.c | 3 ++- fs/gfs2/ops_fstype.c | 3 ++- fs/hfsplus/wrapper.c | 3 ++- fs/jfs/jfs_logmgr.c | 6 ++++-- fs/jfs/jfs_metapage.c | 10 ++++++---- fs/logfs/dev_bdev.c | 15 ++++++++++----- fs/mpage.c | 3 ++- fs/nfs/blocklayout/blocklayout.c | 22 ++++++++++++---------- fs/nilfs2/segbuf.c | 3 ++- fs/ocfs2/cluster/heartbeat.c | 12 +++++++----- fs/xfs/xfs_aops.c | 3 ++- fs/xfs/xfs_buf.c | 4 ++-- include/linux/bio.h | 2 +- include/linux/fs.h | 2 +- kernel/power/swap.c | 5 +++-- mm/page_io.c | 10 ++++++---- 52 files changed, 211 insertions(+), 141 deletions(-) diff --git a/block/bio.c b/block/bio.c index dbabd48..921112b 100644 --- a/block/bio.c +++ b/block/bio.c @@ -859,21 +859,20 @@ static void submit_bio_wait_endio(struct bio *bio) /** * submit_bio_wait - submit a bio, and wait until it completes - * @rw: whether to %READ or %WRITE, or maybe to %READA (read ahead) * @bio: The &struct bio which describes the I/O * * Simple wrapper around submit_bio(). Returns 0 on success, or the error from * bio_endio() on failure. */ -int submit_bio_wait(int rw, struct bio *bio) +int submit_bio_wait(struct bio *bio) { struct submit_bio_ret ret; - rw |= REQ_SYNC; init_completion(&ret.event); bio->bi_private = &ret; bio->bi_end_io = submit_bio_wait_endio; - submit_bio(rw, bio); + bio->bi_rw |= REQ_SYNC; + submit_bio(bio); wait_for_completion(&ret.event); return ret.error; diff --git a/block/blk-core.c b/block/blk-core.c index ab51685..9b887e3 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -2092,7 +2092,6 @@ EXPORT_SYMBOL(generic_make_request); /** * submit_bio - submit a bio to the block device layer for I/O - * @rw: whether to %READ or %WRITE, or maybe to %READA (read ahead) * @bio: The &struct bio which describes the I/O * * submit_bio() is very similar in purpose to generic_make_request(), and @@ -2100,10 +2099,8 @@ EXPORT_SYMBOL(generic_make_request); * interfaces; @bio must be presetup and ready for I/O. * */ -blk_qc_t submit_bio(int rw, struct bio *bio) +blk_qc_t submit_bio(struct bio *bio) { - bio->bi_rw |= rw; - /* * If it's a regular read/write or a barrier with data attached, * go through the normal accounting stuff before submission. @@ -2111,12 +2108,12 @@ blk_qc_t submit_bio(int rw, struct bio *bio) if (bio_has_data(bio)) { unsigned int count; - if (unlikely(rw & REQ_WRITE_SAME)) + if (unlikely(bio->bi_rw & REQ_WRITE_SAME)) count = bdev_logical_block_size(bio->bi_bdev) >> 9; else count = bio_sectors(bio); - if (rw & WRITE) { + if (bio->bi_rw & WRITE) { count_vm_events(PGPGOUT, count); } else { task_io_account_read(bio->bi_iter.bi_size); @@ -2127,7 +2124,7 @@ blk_qc_t submit_bio(int rw, struct bio *bio) char b[BDEVNAME_SIZE]; printk(KERN_DEBUG "%s(%d): %s block %Lu on %s (%u sectors)\n", current->comm, task_pid_nr(current), - (rw & WRITE) ? "WRITE" : "READ", + (bio->bi_rw & WRITE) ? "WRITE" : "READ", (unsigned long long)bio->bi_iter.bi_sector, bdevname(bio->bi_bdev, b), count); diff --git a/block/blk-flush.c b/block/blk-flush.c index 9c423e5..f2fbf9a 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -484,8 +484,9 @@ int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, bio = bio_alloc(gfp_mask, 0); bio->bi_bdev = bdev; + bio->bi_rw = WRITE_FLUSH; - ret = submit_bio_wait(WRITE_FLUSH, bio); + ret = submit_bio_wait(bio); /* * The driver must store the error location in ->bi_sector, if diff --git a/block/blk-lib.c b/block/blk-lib.c index 9ebf653..87e3de4 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -102,13 +102,14 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, bio->bi_end_io = bio_batch_end_io; bio->bi_bdev = bdev; bio->bi_private = &bb; + bio->bi_rw = type; bio->bi_iter.bi_size = req_sects << 9; nr_sects -= req_sects; sector = end_sect; atomic_inc(&bb.done); - submit_bio(type, bio); + submit_bio(bio); /* * We can loop for a long time in here, if someone does @@ -177,6 +178,7 @@ int blkdev_issue_write_same(struct block_device *bdev, sector_t sector, bio->bi_io_vec->bv_page = page; bio->bi_io_vec->bv_offset = 0; bio->bi_io_vec->bv_len = bdev_logical_block_size(bdev); + bio->bi_rw = REQ_WRITE | REQ_WRITE_SAME; if (nr_sects > max_write_same_sectors) { bio->bi_iter.bi_size = max_write_same_sectors << 9; @@ -188,7 +190,7 @@ int blkdev_issue_write_same(struct block_device *bdev, sector_t sector, } atomic_inc(&bb.done); - submit_bio(REQ_WRITE | REQ_WRITE_SAME, bio); + submit_bio(bio); } /* Wait for bios in-flight */ @@ -238,6 +240,7 @@ static int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, bio->bi_bdev = bdev; bio->bi_end_io = bio_batch_end_io; bio->bi_private = &bb; + bio->bi_rw = WRITE; while (nr_sects != 0) { sz = min((sector_t) PAGE_SIZE >> 9 , nr_sects); @@ -249,7 +252,7 @@ static int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, } ret = 0; atomic_inc(&bb.done); - submit_bio(WRITE, bio); + submit_bio(bio); } /* Wait for bios in-flight */ diff --git a/drivers/block/drbd/drbd_actlog.c b/drivers/block/drbd/drbd_actlog.c index 10459a1..6069e15 100644 --- a/drivers/block/drbd/drbd_actlog.c +++ b/drivers/block/drbd/drbd_actlog.c @@ -177,7 +177,7 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, if (drbd_insert_fault(device, (rw & WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) bio_io_error(bio); else - submit_bio(rw, bio); + submit_bio(bio); wait_until_done_or_force_detached(device, bdev, &device->md_io.done); if (!bio->bi_error) err = device->md_io.error; diff --git a/drivers/block/drbd/drbd_bitmap.c b/drivers/block/drbd/drbd_bitmap.c index 92d6fc0..e8959fe 100644 --- a/drivers/block/drbd/drbd_bitmap.c +++ b/drivers/block/drbd/drbd_bitmap.c @@ -1011,12 +1011,12 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho bio_add_page(bio, page, len, 0); bio->bi_private = ctx; bio->bi_end_io = drbd_bm_endio; + bio->bi_rw = rw; if (drbd_insert_fault(device, (rw & WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) { - bio->bi_rw |= rw; bio_io_error(bio); } else { - submit_bio(rw, bio); + submit_bio(bio); /* this should not count as user activity and cause the * resync to throttle -- see drbd_rs_should_slow_down(). */ atomic_add(len >> 9, &device->rs_sect_ev); diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c index 9e25120..aebbdb9 100644 --- a/drivers/block/floppy.c +++ b/drivers/block/floppy.c @@ -3817,8 +3817,9 @@ static int __floppy_read_block_0(struct block_device *bdev, int drive) bio.bi_flags |= (1 << BIO_QUIET); bio.bi_private = &cbdata; bio.bi_end_io = floppy_rb0_cb; + bio.bi_rw = READ; - submit_bio(READ, &bio); + submit_bio(&bio); process_fd_request(); init_completion(&cbdata.complete); diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index 41fb1a9..733d397 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -1348,6 +1348,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, bio->bi_private = pending_req; bio->bi_end_io = end_block_io_op; bio->bi_iter.bi_sector = preq.sector_number; + bio->bi_rw = operation; } preq.sector_number += seg[i].nsec; @@ -1365,13 +1366,14 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, bio->bi_bdev = preq.bdev; bio->bi_private = pending_req; bio->bi_end_io = end_block_io_op; + bio->bi_rw = operation; } atomic_set(&pending_req->pendcnt, nbio); blk_start_plug(&plug); for (i = 0; i < nbio; i++) - submit_bio(operation, biolist[i]); + submit_bio(biolist[i]); /* Let the I/Os go.. */ blk_finish_plug(&plug); diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 2fee2ee..81444fc 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -1749,7 +1749,7 @@ static int blkif_recover(struct blkfront_info *info) bio_trim(cloned_bio, offset, size); cloned_bio->bi_private = split_bio; cloned_bio->bi_end_io = split_bio_end; - submit_bio(cloned_bio->bi_rw, cloned_bio); + submit_bio(cloned_bio); } /* * Now we have to wait for all those smaller bios to @@ -1758,7 +1758,7 @@ static int blkif_recover(struct blkfront_info *info) continue; } /* We don't need to split this bio */ - submit_bio(bio->bi_rw, bio); + submit_bio(bio); } return 0; diff --git a/drivers/md/bcache/debug.c b/drivers/md/bcache/debug.c index 8b1f1d5..52b6bcf 100644 --- a/drivers/md/bcache/debug.c +++ b/drivers/md/bcache/debug.c @@ -52,9 +52,10 @@ void bch_btree_verify(struct btree *b) bio->bi_bdev = PTR_CACHE(b->c, &b->key, 0)->bdev; bio->bi_iter.bi_sector = PTR_OFFSET(&b->key, 0); bio->bi_iter.bi_size = KEY_SIZE(&v->key) << 9; + bio->bi_rw = REQ_META|READ_SYNC; bch_bio_map(bio, sorted); - submit_bio_wait(REQ_META|READ_SYNC, bio); + submit_bio_wait(bio); bch_bbio_free(bio, b->c); memcpy(ondisk, sorted, KEY_SIZE(&v->key) << 9); @@ -113,11 +114,12 @@ void bch_data_verify(struct cached_dev *dc, struct bio *bio) check = bio_clone(bio, GFP_NOIO); if (!check) return; + check->bi_rw |= READ_SYNC; if (bio_alloc_pages(check, GFP_NOIO)) goto out_put; - submit_bio_wait(READ_SYNC, check); + submit_bio_wait(check); bio_for_each_segment(bv, bio, iter) { void *p1 = kmap_atomic(bv.bv_page); diff --git a/drivers/md/bcache/journal.c b/drivers/md/bcache/journal.c index 29eba72..af3f9f7 100644 --- a/drivers/md/bcache/journal.c +++ b/drivers/md/bcache/journal.c @@ -418,7 +418,7 @@ static void journal_discard_work(struct work_struct *work) struct journal_device *ja = container_of(work, struct journal_device, discard_work); - submit_bio(0, &ja->discard_bio); + submit_bio(&ja->discard_bio); } static void do_journal_discard(struct cache *ca) diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index 8d0ead9..9e2eac8 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -212,7 +212,7 @@ static void __write_super(struct cache_sb *sb, struct bio *bio) unsigned i; bio->bi_iter.bi_sector = SB_SECTOR; - bio->bi_rw = REQ_SYNC|REQ_META; + bio->bi_rw = REQ_WRITE|REQ_SYNC|REQ_META; bio->bi_iter.bi_size = SB_SIZE; bch_bio_map(bio, NULL); @@ -238,7 +238,7 @@ static void __write_super(struct cache_sb *sb, struct bio *bio) pr_debug("ver %llu, flags %llu, seq %llu", sb->version, sb->flags, sb->seq); - submit_bio(REQ_WRITE, bio); + submit_bio(bio); } static void bch_write_bdev_super_unlock(struct closure *cl) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index cd77216..9d3ee7f 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -634,6 +634,7 @@ static void use_inline_bio(struct dm_buffer *b, int rw, sector_t block, * the dm_buffer's inline bio is local to bufio. */ b->bio.bi_private = end_io; + b->bio.bi_rw = rw; /* * We assume that if len >= PAGE_SIZE ptr is page-aligned. @@ -660,7 +661,7 @@ static void use_inline_bio(struct dm_buffer *b, int rw, sector_t block, ptr += PAGE_SIZE; } while (len > 0); - submit_bio(rw, &b->bio); + submit_bio(&b->bio); } static void submit_io(struct dm_buffer *b, int rw, sector_t block, diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c index 06d426e..50f17e3 100644 --- a/drivers/md/dm-io.c +++ b/drivers/md/dm-io.c @@ -322,6 +322,7 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, bio->bi_iter.bi_sector = where->sector + (where->count - remaining); bio->bi_bdev = where->bdev; bio->bi_end_io = endio; + bio->bi_rw = rw; store_io_and_region_in_bio(bio, io, region); if (rw & REQ_DISCARD) { @@ -355,7 +356,7 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, } atomic_inc(&io->count); - submit_bio(rw, bio); + submit_bio(bio); } while (remaining); } diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c index 624589d..58c2a3a 100644 --- a/drivers/md/dm-log-writes.c +++ b/drivers/md/dm-log-writes.c @@ -205,6 +205,7 @@ static int write_metadata(struct log_writes_c *lc, void *entry, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; + bio->bi_rw = WRITE; page = alloc_page(GFP_KERNEL); if (!page) { @@ -226,7 +227,7 @@ static int write_metadata(struct log_writes_c *lc, void *entry, DMERR("Couldn't add page to the log block"); goto error_bio; } - submit_bio(WRITE, bio); + submit_bio(bio); return 0; error_bio: bio_put(bio); @@ -269,6 +270,7 @@ static int log_one_block(struct log_writes_c *lc, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; + bio->bi_rw = WRITE; for (i = 0; i < block->vec_cnt; i++) { /* @@ -279,7 +281,7 @@ static int log_one_block(struct log_writes_c *lc, block->vecs[i].bv_len, 0); if (ret != block->vecs[i].bv_len) { atomic_inc(&lc->io_blocks); - submit_bio(WRITE, bio); + submit_bio(bio); bio = bio_alloc(GFP_KERNEL, block->vec_cnt - i); if (!bio) { DMERR("Couldn't alloc log bio"); @@ -290,6 +292,7 @@ static int log_one_block(struct log_writes_c *lc, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; + bio->bi_rw = WRITE; ret = bio_add_page(bio, block->vecs[i].bv_page, block->vecs[i].bv_len, 0); @@ -301,7 +304,7 @@ static int log_one_block(struct log_writes_c *lc, } sector += block->vecs[i].bv_len >> SECTOR_SHIFT; } - submit_bio(WRITE, bio); + submit_bio(bio); out: kfree(block->data); kfree(block); diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index 72d91f4..a40ef76b 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -365,8 +365,9 @@ static int __blkdev_issue_discard_async(struct block_device *bdev, sector_t sect bio->bi_iter.bi_sector = sector; bio->bi_bdev = bdev; bio->bi_iter.bi_size = nr_sects << 9; + bio->bi_rw = type; - submit_bio(type, bio); + submit_bio(bio); return 0; } diff --git a/drivers/md/md.c b/drivers/md/md.c index 61268bb..58ebe4b 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -391,8 +391,9 @@ static void submit_flushes(struct work_struct *ws) bi->bi_end_io = md_end_flush; bi->bi_private = rdev; bi->bi_bdev = rdev->bdev; + bi->bi_rw = WRITE_FLUSH; atomic_inc(&mddev->flush_pending); - submit_bio(WRITE_FLUSH, bi); + submit_bio(bi); rcu_read_lock(); rdev_dec_pending(rdev, mddev); } @@ -736,9 +737,10 @@ void md_super_write(struct mddev *mddev, struct md_rdev *rdev, bio_add_page(bio, page, size, 0); bio->bi_private = rdev; bio->bi_end_io = super_written; + bio->bi_rw = WRITE_FLUSH_FUA; atomic_inc(&mddev->pending_writes); - submit_bio(WRITE_FLUSH_FUA, bio); + submit_bio(bio); } void md_super_wait(struct mddev *mddev) @@ -755,6 +757,7 @@ int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, bio->bi_bdev = (metadata_op && rdev->meta_bdev) ? rdev->meta_bdev : rdev->bdev; + bio->bi_rw = rw; if (metadata_op) bio->bi_iter.bi_sector = sector + rdev->sb_start; else if (rdev->mddev->reshape_position != MaxSector && @@ -764,7 +767,8 @@ int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, else bio->bi_iter.bi_sector = sector + rdev->data_offset; bio_add_page(bio, page, size, 0); - submit_bio_wait(rw, bio); + + submit_bio_wait(bio); ret = !bio->bi_error; bio_put(bio); diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index e2169ff..01e6e74 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2208,7 +2208,8 @@ static int narrow_write_error(struct r1bio *r1_bio, int i) bio_trim(wbio, sector - r1_bio->sector, sectors); wbio->bi_iter.bi_sector += rdev->data_offset; wbio->bi_bdev = rdev->bdev; - if (submit_bio_wait(WRITE, wbio) < 0) + + if (submit_bio_wait(wbio) < 0) /* failure! */ ok = rdev_set_badblocks(rdev, sector, sectors, 0) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 84e597e..5b81668 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2474,7 +2474,9 @@ static int narrow_write_error(struct r10bio *r10_bio, int i) choose_data_offset(r10_bio, rdev) + (sector - r10_bio->sector)); wbio->bi_bdev = rdev->bdev; - if (submit_bio_wait(WRITE, wbio) < 0) + wbio->bi_rw = WRITE; + + if (submit_bio_wait(wbio) < 0) /* Failure! */ ok = rdev_set_badblocks(rdev, sector, sectors, 0) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 81dc3d7..517ffd7 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -254,7 +254,7 @@ static void r5l_submit_current_io(struct r5l_log *log) __r5l_set_io_unit_state(io, IO_UNIT_IO_START); spin_unlock_irqrestore(&log->io_list_lock, flags); - submit_bio(WRITE, io->current_bio); + submit_bio(io->current_bio); } static struct bio *r5l_bio_alloc(struct r5l_log *log) @@ -373,7 +373,7 @@ static void r5l_append_payload_page(struct r5l_log *log, struct page *page) io->current_bio = r5l_bio_alloc(log); bio_chain(io->current_bio, prev); - submit_bio(WRITE, prev); + submit_bio(prev); } if (!bio_add_page(io->current_bio, page, PAGE_SIZE, 0)) @@ -691,7 +691,8 @@ void r5l_flush_stripe_to_raid(struct r5conf *conf) bio_reset(&log->flush_bio); log->flush_bio.bi_bdev = log->rdev->bdev; log->flush_bio.bi_end_io = r5l_log_flush_endio; - submit_bio(WRITE_FLUSH, &log->flush_bio); + log->flush_bio.bi_rw = WRITE_FLUSH; + submit_bio(&log->flush_bio); } static void r5l_write_super(struct r5l_log *log, sector_t cp); diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c index 5a2899f..7455df9 100644 --- a/drivers/target/target_core_iblock.c +++ b/drivers/target/target_core_iblock.c @@ -330,7 +330,7 @@ static void iblock_bio_done(struct bio *bio) } static struct bio * -iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num) +iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, int rw) { struct iblock_dev *ib_dev = IBLOCK_DEV(cmd->se_dev); struct bio *bio; @@ -352,18 +352,19 @@ iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num) bio->bi_private = cmd; bio->bi_end_io = &iblock_bio_done; bio->bi_iter.bi_sector = lba; + bio->bi_rw = rw; return bio; } -static void iblock_submit_bios(struct bio_list *list, int rw) +static void iblock_submit_bios(struct bio_list *list) { struct blk_plug plug; struct bio *bio; blk_start_plug(&plug); while ((bio = bio_list_pop(list))) - submit_bio(rw, bio); + submit_bio(bio); blk_finish_plug(&plug); } @@ -405,9 +406,10 @@ iblock_execute_sync_cache(struct se_cmd *cmd) bio = bio_alloc(GFP_KERNEL, 0); bio->bi_end_io = iblock_end_io_flush; bio->bi_bdev = ib_dev->ibd_bd; + bio->bi_rw = WRITE_FLUSH; if (!immed) bio->bi_private = cmd; - submit_bio(WRITE_FLUSH, bio); + submit_bio(bio); return 0; } @@ -456,7 +458,7 @@ iblock_execute_write_same(struct se_cmd *cmd) goto fail; cmd->priv = ibr; - bio = iblock_get_bio(cmd, block_lba, 1); + bio = iblock_get_bio(cmd, block_lba, 1, WRITE); if (!bio) goto fail_free_ibr; @@ -469,7 +471,7 @@ iblock_execute_write_same(struct se_cmd *cmd) while (bio_add_page(bio, sg_page(sg), sg->length, sg->offset) != sg->length) { - bio = iblock_get_bio(cmd, block_lba, 1); + bio = iblock_get_bio(cmd, block_lba, 1, WRITE); if (!bio) goto fail_put_bios; @@ -482,7 +484,7 @@ iblock_execute_write_same(struct se_cmd *cmd) sectors -= 1; } - iblock_submit_bios(&list, WRITE); + iblock_submit_bios(&list); return 0; fail_put_bios: @@ -708,7 +710,7 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, return 0; } - bio = iblock_get_bio(cmd, block_lba, sgl_nents); + bio = iblock_get_bio(cmd, block_lba, sgl_nents, rw); if (!bio) goto fail_free_ibr; @@ -728,11 +730,11 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, while (bio_add_page(bio, sg_page(sg), sg->length, sg->offset) != sg->length) { if (bio_cnt >= IBLOCK_MAX_BIO_PER_TASK) { - iblock_submit_bios(&list, rw); + iblock_submit_bios(&list); bio_cnt = 0; } - bio = iblock_get_bio(cmd, block_lba, sg_num); + bio = iblock_get_bio(cmd, block_lba, sg_num, rw); if (!bio) goto fail_put_bios; @@ -752,7 +754,7 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, goto fail_put_bios; } - iblock_submit_bios(&list, rw); + iblock_submit_bios(&list); iblock_complete_cmd(cmd); return 0; diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index 0340c57..dff4ab4 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -1683,6 +1683,7 @@ static int btrfsic_read_block(struct btrfsic_state *state, } bio->bi_bdev = block_ctx->dev->bdev; bio->bi_iter.bi_sector = dev_bytenr >> 9; + bio->bi_rw = READ; for (j = i; j < num_pages; j++) { ret = bio_add_page(bio, block_ctx->pagev[j], @@ -1695,7 +1696,7 @@ static int btrfsic_read_block(struct btrfsic_state *state, "btrfsic: error, failed to add a single page!\n"); return -1; } - if (submit_bio_wait(READ, bio)) { + if (submit_bio_wait(bio)) { printk(KERN_INFO "btrfsic: read error at logical %llu dev %s!\n", block_ctx->start, block_ctx->dev->name); @@ -2960,9 +2961,10 @@ int btrfsic_submit_bh(int rw, struct buffer_head *bh) return submit_bh(rw, bh); } -static void __btrfsic_submit_bio(int rw, struct bio *bio) +static void __btrfsic_submit_bio(struct bio *bio) { struct btrfsic_dev_state *dev_state; + int rw = bio->bi_rw; if (!btrfsic_is_initialized) return; @@ -3058,16 +3060,16 @@ leave: mutex_unlock(&btrfsic_mutex); } -void btrfsic_submit_bio(int rw, struct bio *bio) +void btrfsic_submit_bio(struct bio *bio) { - __btrfsic_submit_bio(rw, bio); - submit_bio(rw, bio); + __btrfsic_submit_bio(bio); + submit_bio(bio); } -int btrfsic_submit_bio_wait(int rw, struct bio *bio) +int btrfsic_submit_bio_wait(struct bio *bio) { - __btrfsic_submit_bio(rw, bio); - return submit_bio_wait(rw, bio); + __btrfsic_submit_bio(bio); + return submit_bio_wait(bio); } int btrfsic_mount(struct btrfs_root *root, diff --git a/fs/btrfs/check-integrity.h b/fs/btrfs/check-integrity.h index 13b8566..c04e249 100644 --- a/fs/btrfs/check-integrity.h +++ b/fs/btrfs/check-integrity.h @@ -21,8 +21,8 @@ #ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY int btrfsic_submit_bh(int rw, struct buffer_head *bh); -void btrfsic_submit_bio(int rw, struct bio *bio); -int btrfsic_submit_bio_wait(int rw, struct bio *bio); +void btrfsic_submit_bio(struct bio *bio); +int btrfsic_submit_bio_wait(struct bio *bio); #else #define btrfsic_submit_bh submit_bh #define btrfsic_submit_bio submit_bio diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 6dd6774..23f45f3 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -3454,12 +3454,13 @@ static int write_dev_flush(struct btrfs_device *device, int wait) bio->bi_end_io = btrfs_end_empty_barrier; bio->bi_bdev = device->bdev; + bio->bi_rw = WRITE_FLUSH; init_completion(&device->flush_wait); bio->bi_private = &device->flush_wait; device->flush_bio = bio; bio_get(bio); - btrfsic_submit_bio(WRITE_FLUSH, bio); + btrfsic_submit_bio(bio); return 0; } diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index a2356e2..8e7bf941 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2040,9 +2040,10 @@ int repair_io_failure(struct inode *inode, u64 start, u64 length, u64 logical, return -EIO; } bio->bi_bdev = dev->bdev; + bio->bi_rw = WRITE_SYNC; bio_add_page(bio, page, length, pg_offset); - if (btrfsic_submit_bio_wait(WRITE_SYNC, bio)) { + if (btrfsic_submit_bio_wait(bio)) { /* try to remap that extent elsewhere? */ bio_put(bio); btrfs_dev_stat_inc_and_print(dev, BTRFS_DEV_STAT_WRITE_ERRS); @@ -2730,14 +2731,14 @@ static int __must_check submit_one_bio(int rw, struct bio *bio, start = page_offset(page) + bvec->bv_offset; bio->bi_private = NULL; - + bio->bi_rw = rw; bio_get(bio); if (tree->ops && tree->ops->submit_bio_hook) ret = tree->ops->submit_bio_hook(page->mapping->host, rw, bio, mirror_num, bio_flags, start); else - btrfsic_submit_bio(rw, bio); + btrfsic_submit_bio(bio); bio_put(bio); return ret; diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c index 1a33d3e..3290fa1 100644 --- a/fs/btrfs/raid56.c +++ b/fs/btrfs/raid56.c @@ -1323,7 +1323,9 @@ write_data: bio->bi_private = rbio; bio->bi_end_io = raid_write_end_io; - submit_bio(WRITE, bio); + bio->bi_rw = WRITE; + + submit_bio(bio); } return; @@ -1577,11 +1579,12 @@ static int raid56_rmw_stripe(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid_rmw_end_io; + bio->bi_rw = READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); - submit_bio(READ, bio); + submit_bio(bio); } /* the actual write will happen once the reads are done */ return 0; @@ -2103,11 +2106,12 @@ static int __raid56_parity_recover(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid_recover_end_io; + bio->bi_rw = READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); - submit_bio(READ, bio); + submit_bio(bio); } out: return 0; @@ -2465,7 +2469,8 @@ submit_write: bio->bi_private = rbio; bio->bi_end_io = raid_write_parity_end_io; - submit_bio(WRITE, bio); + bio->bi_rw = WRITE; + submit_bio(bio); } return; @@ -2642,11 +2647,12 @@ static void raid56_parity_scrub_stripe(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid56_parity_scrub_end_io; + bio->bi_rw = READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); - submit_bio(READ, bio); + submit_bio(bio); } /* the actual write will happen once the reads are done */ return; diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c index 6998715..8855cfe 100644 --- a/fs/btrfs/scrub.c +++ b/fs/btrfs/scrub.c @@ -1504,8 +1504,9 @@ static void scrub_recheck_block(struct btrfs_fs_info *fs_info, sblock->no_io_error_seen = 0; } else { bio->bi_iter.bi_sector = page->physical >> 9; + bio->bi_rw = READ; - if (btrfsic_submit_bio_wait(READ, bio)) + if (btrfsic_submit_bio_wait(bio)) sblock->no_io_error_seen = 0; } @@ -1585,6 +1586,7 @@ static int scrub_repair_page_from_good_copy(struct scrub_block *sblock_bad, return -EIO; bio->bi_bdev = page_bad->dev->bdev; bio->bi_iter.bi_sector = page_bad->physical >> 9; + bio->bi_rw = WRITE; ret = bio_add_page(bio, page_good->page, PAGE_SIZE, 0); if (PAGE_SIZE != ret) { @@ -1592,7 +1594,7 @@ static int scrub_repair_page_from_good_copy(struct scrub_block *sblock_bad, return -EIO; } - if (btrfsic_submit_bio_wait(WRITE, bio)) { + if (btrfsic_submit_bio_wait(bio)) { btrfs_dev_stat_inc_and_print(page_bad->dev, BTRFS_DEV_STAT_WRITE_ERRS); btrfs_dev_replace_stats_inc( @@ -1685,6 +1687,7 @@ again: bio->bi_end_io = scrub_wr_bio_end_io; bio->bi_bdev = sbio->dev->bdev; bio->bi_iter.bi_sector = sbio->physical >> 9; + bio->bi_rw = WRITE; sbio->err = 0; } else if (sbio->physical + sbio->page_count * PAGE_SIZE != spage->physical_for_dev_replace || @@ -1732,7 +1735,7 @@ static void scrub_wr_submit(struct scrub_ctx *sctx) * orders the requests before sending them to the driver which * doubled the write performance on spinning disks when measured * with Linux 3.5 */ - btrfsic_submit_bio(WRITE, sbio->bio); + btrfsic_submit_bio(sbio->bio); } static void scrub_wr_bio_end_io(struct bio *bio) @@ -2042,7 +2045,7 @@ static void scrub_submit(struct scrub_ctx *sctx) sbio = sctx->bios[sctx->curr]; sctx->curr = -1; scrub_pending_bio_inc(sctx); - btrfsic_submit_bio(READ, sbio->bio); + btrfsic_submit_bio(sbio->bio); } static int scrub_add_page_to_rd_bio(struct scrub_ctx *sctx, @@ -2088,6 +2091,7 @@ again: bio->bi_end_io = scrub_bio_end_io; bio->bi_bdev = sbio->dev->bdev; bio->bi_iter.bi_sector = sbio->physical >> 9; + bio->bi_rw = READ; sbio->err = 0; } else if (sbio->physical + sbio->page_count * PAGE_SIZE != spage->physical || @@ -4390,6 +4394,8 @@ static int write_page_nocow(struct scrub_ctx *sctx, bio->bi_iter.bi_size = 0; bio->bi_iter.bi_sector = physical_for_dev_replace >> 9; bio->bi_bdev = dev->bdev; + bio->bi_rw = WRITE_SYNC; + ret = bio_add_page(bio, page, PAGE_CACHE_SIZE, 0); if (ret != PAGE_CACHE_SIZE) { leave_with_eio: @@ -4398,7 +4404,7 @@ leave_with_eio: return -EIO; } - if (btrfsic_submit_bio_wait(WRITE_SYNC, bio)) + if (btrfsic_submit_bio_wait(bio)) goto leave_with_eio; bio_put(bio); diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index a114b7b..2463761 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -446,7 +446,7 @@ loop_lock: sync_pending = 0; } - btrfsic_submit_bio(cur->bi_rw, cur); + btrfsic_submit_bio(cur); num_run++; batch_run++; @@ -5942,7 +5942,7 @@ static void btrfs_end_bio(struct bio *bio) */ static noinline void btrfs_schedule_bio(struct btrfs_root *root, struct btrfs_device *device, - int rw, struct bio *bio) + struct bio *bio) { int should_queue = 1; struct btrfs_pending_bios *pending_bios; @@ -5953,9 +5953,9 @@ static noinline void btrfs_schedule_bio(struct btrfs_root *root, } /* don't bother with additional async steps for reads, right now */ - if (!(rw & REQ_WRITE)) { + if (!(bio->bi_rw & REQ_WRITE)) { bio_get(bio); - btrfsic_submit_bio(rw, bio); + btrfsic_submit_bio(bio); bio_put(bio); return; } @@ -5969,7 +5969,6 @@ static noinline void btrfs_schedule_bio(struct btrfs_root *root, atomic_inc(&root->fs_info->nr_async_bios); WARN_ON(bio->bi_next); bio->bi_next = NULL; - bio->bi_rw |= rw; spin_lock(&device->io_lock); if (bio->bi_rw & REQ_SYNC) @@ -6003,6 +6002,7 @@ static void submit_stripe_bio(struct btrfs_root *root, struct btrfs_bio *bbio, btrfs_io_bio(bio)->stripe_index = dev_nr; bio->bi_end_io = btrfs_end_bio; bio->bi_iter.bi_sector = physical >> 9; + bio->bi_rw |= rw; #ifdef DEBUG { struct rcu_string *name; @@ -6021,9 +6021,9 @@ static void submit_stripe_bio(struct btrfs_root *root, struct btrfs_bio *bbio, btrfs_bio_counter_inc_noblocked(root->fs_info); if (async) - btrfs_schedule_bio(root, dev, rw, bio); + btrfs_schedule_bio(root, dev, bio); else - btrfsic_submit_bio(rw, bio); + btrfsic_submit_bio(bio); } static void bbio_error(struct btrfs_bio *bbio, struct bio *bio, u64 logical) diff --git a/fs/buffer.c b/fs/buffer.c index e1632ab..4349245 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -3032,8 +3032,9 @@ static int submit_bh_wbc(int rw, struct buffer_head *bh, rw |= REQ_META; if (buffer_prio(bh)) rw |= REQ_PRIO; + bio->bi_rw = rw; - submit_bio(rw, bio); + submit_bio(bio); return 0; } diff --git a/fs/direct-io.c b/fs/direct-io.c index 602e844..4400836 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -369,6 +369,7 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_bdev = bdev; bio->bi_iter.bi_sector = first_sector; + bio->bi_rw = dio->rw; if (dio->is_async) bio->bi_end_io = dio_bio_end_aio; else @@ -406,7 +407,7 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) sdio->logical_offset_in_bio); dio->bio_cookie = BLK_QC_T_NONE; } else - dio->bio_cookie = submit_bio(dio->rw, bio); + dio->bio_cookie = submit_bio(bio); sdio->bio = NULL; sdio->boundary = 0; diff --git a/fs/ext4/crypto.c b/fs/ext4/crypto.c index c802120..02f2070 100644 --- a/fs/ext4/crypto.c +++ b/fs/ext4/crypto.c @@ -424,6 +424,7 @@ int ext4_encrypted_zeroout(struct inode *inode, ext4_lblk_t lblk, bio->bi_bdev = inode->i_sb->s_bdev; bio->bi_iter.bi_sector = pblk << (inode->i_sb->s_blocksize_bits - 9); + bio->bi_rw = WRITE; ret = bio_add_page(bio, ciphertext_page, inode->i_sb->s_blocksize, 0); if (ret != inode->i_sb->s_blocksize) { @@ -435,7 +436,7 @@ int ext4_encrypted_zeroout(struct inode *inode, ext4_lblk_t lblk, err = -EIO; goto errout; } - err = submit_bio_wait(WRITE, bio); + err = submit_bio_wait(bio); if ((err == 0) && bio->bi_error) err = -EIO; bio_put(bio); diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index 090b349..cf63ac3 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -356,7 +356,8 @@ void ext4_io_submit(struct ext4_io_submit *io) int io_op = io->io_wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE; bio_get(io->io_bio); - submit_bio(io_op, io->io_bio); + io->io_bio->bi_rw = io_op; + submit_bio(io->io_bio); bio_put(io->io_bio); } io->io_bio = NULL; diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c index 5dc5e95..893dc15 100644 --- a/fs/ext4/readpage.c +++ b/fs/ext4/readpage.c @@ -271,7 +271,7 @@ int ext4_mpage_readpages(struct address_space *mapping, */ if (bio && (last_block_in_bio != blocks[0] - 1)) { submit_and_realloc: - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } if (bio == NULL) { @@ -294,6 +294,7 @@ int ext4_mpage_readpages(struct address_space *mapping, bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9); bio->bi_end_io = mpage_end_io; bio->bi_private = ctx; + bio->bi_rw = READ; } length = first_hole << blkbits; @@ -303,14 +304,14 @@ int ext4_mpage_readpages(struct address_space *mapping, if (((map.m_flags & EXT4_MAP_BOUNDARY) && (relative_block == map.m_len)) || (first_hole != blocks_per_page)) { - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } else last_block_in_bio = blocks[blocks_per_page - 1]; goto next_page; confused: if (bio) { - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } if (!PageUptodate(page)) @@ -323,6 +324,6 @@ int ext4_mpage_readpages(struct address_space *mapping, } BUG_ON(pages && !list_empty(pages)); if (bio) - submit_bio(READ, bio); + submit_bio(bio); return 0; } diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 5c38f44..e1c64ed 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -111,8 +111,9 @@ static void __submit_merged_bio(struct f2fs_bio_info *io) trace_f2fs_submit_read_bio(io->sbi->sb, fio, io->bio); else trace_f2fs_submit_write_bio(io->sbi->sb, fio, io->bio); + io->bio->bi_rw = fio->rw; - submit_bio(fio->rw, io->bio); + submit_bio(io->bio); io->bio = NULL; } @@ -157,8 +158,9 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio) bio_put(bio); return -EFAULT; } + bio->bi_rw = fio->rw; - submit_bio(fio->rw, bio); + submit_bio(bio); return 0; } @@ -961,7 +963,7 @@ got_it: */ if (bio && (last_block_in_bio != block_nr - 1)) { submit_and_realloc: - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } if (bio == NULL) { @@ -990,6 +992,7 @@ submit_and_realloc: bio->bi_iter.bi_sector = SECTOR_FROM_BLOCK(block_nr); bio->bi_end_io = f2fs_read_end_io; bio->bi_private = ctx; + bio->bi_rw = READ; } if (bio_add_page(bio, page, blocksize, 0) < blocksize) @@ -1004,7 +1007,7 @@ set_error_page: goto next_page; confused: if (bio) { - submit_bio(READ, bio); + submit_bio(bio); bio = NULL; } unlock_page(page); @@ -1014,7 +1017,7 @@ next_page: } BUG_ON(pages && !list_empty(pages)); if (bio) - submit_bio(READ, bio); + submit_bio(bio); return 0; } diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index a3474ba..f518cab 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -318,7 +318,8 @@ repeat: fcc->dispatch_list = llist_reverse_order(fcc->dispatch_list); bio->bi_bdev = sbi->sb->s_bdev; - ret = submit_bio_wait(WRITE_FLUSH, bio); + bio->bi_rw = WRITE_FLUSH; + ret = submit_bio_wait(bio); llist_for_each_entry_safe(cmd, next, fcc->dispatch_list, llnode) { @@ -350,7 +351,8 @@ int f2fs_issue_flush(struct f2fs_sb_info *sbi) int ret; bio->bi_bdev = sbi->sb->s_bdev; - ret = submit_bio_wait(WRITE_FLUSH, bio); + bio->bi_rw = WRITE_FLUSH; + ret = submit_bio_wait(bio); bio_put(bio); return ret; } diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index d5369a1..ce28242 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -240,7 +240,8 @@ void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int rw) { if (sdp->sd_log_bio) { atomic_inc(&sdp->sd_log_in_flight); - submit_bio(rw, sdp->sd_log_bio); + sdp->sd_log_bio->bi_rw = rw; + submit_bio(sdp->sd_log_bio); sdp->sd_log_bio = NULL; } } diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index e137d96..87a3abc 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -230,7 +230,8 @@ static void gfs2_submit_bhs(int rw, struct buffer_head *bhs[], int num) bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh)); } bio->bi_end_io = gfs2_meta_read_endio; - submit_bio(rw, bio); + bio->bi_rw = rw; + submit_bio(bio); } /** diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c index dbed9e2..4872a61 100644 --- a/fs/gfs2/ops_fstype.c +++ b/fs/gfs2/ops_fstype.c @@ -246,7 +246,8 @@ static int gfs2_read_super(struct gfs2_sbd *sdp, sector_t sector, int silent) bio->bi_end_io = end_bio_io_page; bio->bi_private = page; - submit_bio(READ_SYNC | REQ_META, bio); + bio->bi_rw = READ_SYNC | REQ_META; + submit_bio(bio); wait_on_page_locked(page); bio_put(bio); if (!PageUptodate(page)) { diff --git a/fs/hfsplus/wrapper.c b/fs/hfsplus/wrapper.c index cc62356..d026bb3 100644 --- a/fs/hfsplus/wrapper.c +++ b/fs/hfsplus/wrapper.c @@ -65,6 +65,7 @@ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, bio = bio_alloc(GFP_NOIO, 1); bio->bi_iter.bi_sector = sector; bio->bi_bdev = sb->s_bdev; + bio->bi_rw = rw; if (!(rw & WRITE) && data) *data = (u8 *)buf + offset; @@ -83,7 +84,7 @@ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, buf = (u8 *)buf + len; } - ret = submit_bio_wait(rw, bio); + ret = submit_bio_wait(bio); out: bio_put(bio); return ret < 0 ? ret : 0; diff --git a/fs/jfs/jfs_logmgr.c b/fs/jfs/jfs_logmgr.c index a270cb7..b0e068f 100644 --- a/fs/jfs/jfs_logmgr.c +++ b/fs/jfs/jfs_logmgr.c @@ -2004,12 +2004,13 @@ static int lbmRead(struct jfs_log * log, int pn, struct lbuf ** bpp) bio->bi_end_io = lbmIODone; bio->bi_private = bp; + bio->bi_rw = READ_SYNC; /*check if journaling to disk has been disabled*/ if (log->no_integrity) { bio->bi_iter.bi_size = 0; lbmIODone(bio); } else { - submit_bio(READ_SYNC, bio); + submit_bio(bio); } wait_event(bp->l_ioevent, (bp->l_flag != lbmREAD)); @@ -2147,13 +2148,14 @@ static void lbmStartIO(struct lbuf * bp) bio->bi_end_io = lbmIODone; bio->bi_private = bp; + bio->bi_rw = WRITE_SYNC; /* check if journaling to disk has been disabled */ if (log->no_integrity) { bio->bi_iter.bi_size = 0; lbmIODone(bio); } else { - submit_bio(WRITE_SYNC, bio); + submit_bio(bio); INCREMENT(lmStat.submitted); } } diff --git a/fs/jfs/jfs_metapage.c b/fs/jfs/jfs_metapage.c index a3eb316..bc29972 100644 --- a/fs/jfs/jfs_metapage.c +++ b/fs/jfs/jfs_metapage.c @@ -411,7 +411,7 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) inc_io(page); if (!bio->bi_iter.bi_size) goto dump_bio; - submit_bio(WRITE, bio); + submit_bio(bio); nr_underway++; bio = NULL; } else @@ -434,6 +434,7 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) bio->bi_iter.bi_sector = pblock << (inode->i_blkbits - 9); bio->bi_end_io = metapage_write_end_io; bio->bi_private = page; + bio->bi_rw = WRITE; /* Don't call bio_add_page yet, we may add to this vec */ bio_offset = offset; @@ -448,7 +449,7 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) if (!bio->bi_iter.bi_size) goto dump_bio; - submit_bio(WRITE, bio); + submit_bio(bio); nr_underway++; } if (redirty) @@ -506,7 +507,7 @@ static int metapage_readpage(struct file *fp, struct page *page) insert_metapage(page, NULL); inc_io(page); if (bio) - submit_bio(READ, bio); + submit_bio(bio); bio = bio_alloc(GFP_NOFS, 1); bio->bi_bdev = inode->i_sb->s_bdev; @@ -514,6 +515,7 @@ static int metapage_readpage(struct file *fp, struct page *page) pblock << (inode->i_blkbits - 9); bio->bi_end_io = metapage_read_end_io; bio->bi_private = page; + bio->bi_rw = READ; len = xlen << inode->i_blkbits; offset = block_offset << inode->i_blkbits; if (bio_add_page(bio, page, len, offset) < len) @@ -523,7 +525,7 @@ static int metapage_readpage(struct file *fp, struct page *page) block_offset++; } if (bio) - submit_bio(READ, bio); + submit_bio(bio); else unlock_page(page); diff --git a/fs/logfs/dev_bdev.c b/fs/logfs/dev_bdev.c index a709d80..9e690ad 100644 --- a/fs/logfs/dev_bdev.c +++ b/fs/logfs/dev_bdev.c @@ -29,8 +29,9 @@ static int sync_request(struct page *page, struct block_device *bdev, int rw) bio.bi_bdev = bdev; bio.bi_iter.bi_sector = page->index * (PAGE_SIZE >> 9); bio.bi_iter.bi_size = PAGE_SIZE; + bio.bi_rw = rw; - return submit_bio_wait(rw, &bio); + return submit_bio_wait(&bio); } static int bdev_readpage(void *_sb, struct page *page) @@ -95,8 +96,9 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = writeseg_end_io; + bio->bi_rw = WRITE; atomic_inc(&super->s_pending_writes); - submit_bio(WRITE, bio); + submit_bio(bio); ofs += i * PAGE_SIZE; index += i; @@ -122,8 +124,9 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = writeseg_end_io; + bio->bi_rw = WRITE; atomic_inc(&super->s_pending_writes); - submit_bio(WRITE, bio); + submit_bio(bio); return 0; } @@ -185,8 +188,9 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = erase_end_io; + bio->bi_rw = WRITE; atomic_inc(&super->s_pending_writes); - submit_bio(WRITE, bio); + submit_bio(bio); ofs += i * PAGE_SIZE; index += i; @@ -206,8 +210,9 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = erase_end_io; + bio->bi_rw = WRITE; atomic_inc(&super->s_pending_writes); - submit_bio(WRITE, bio); + submit_bio(bio); return 0; } diff --git a/fs/mpage.c b/fs/mpage.c index 1480d3a..d9adde2 100644 --- a/fs/mpage.c +++ b/fs/mpage.c @@ -58,8 +58,9 @@ static void mpage_end_io(struct bio *bio) static struct bio *mpage_bio_submit(int rw, struct bio *bio) { bio->bi_end_io = mpage_end_io; + bio->bi_rw = rw; guard_bio_eod(rw, bio); - submit_bio(rw, bio); + submit_bio(bio); return NULL; } diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index ddd0138..8c0443c 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -102,14 +102,15 @@ static inline void put_parallel(struct parallel_io *p) } static struct bio * -bl_submit_bio(int rw, struct bio *bio) +bl_submit_bio(struct bio *bio) { if (bio) { get_parallel(bio->bi_private); dprintk("%s submitting %s bio %u@%llu\n", __func__, - rw == READ ? "read" : "write", bio->bi_iter.bi_size, + bio->bi_rw == READ ? "read" : "write", + bio->bi_iter.bi_size, (unsigned long long)bio->bi_iter.bi_sector); - submit_bio(rw, bio); + submit_bio(bio); } return NULL; } @@ -158,7 +159,7 @@ do_add_page_to_bio(struct bio *bio, int npg, int rw, sector_t isect, if (disk_addr < map->start || disk_addr >= map->start + map->len) { if (!dev->map(dev, disk_addr, map)) return ERR_PTR(-EIO); - bio = bl_submit_bio(rw, bio); + bio = bl_submit_bio(bio); } disk_addr += map->disk_offset; disk_addr -= map->start; @@ -174,9 +175,10 @@ retry: disk_addr >> SECTOR_SHIFT, end_io, par); if (!bio) return ERR_PTR(-ENOMEM); + bio->bi_rw = rw; } if (bio_add_page(bio, page, *len, offset) < *len) { - bio = bl_submit_bio(rw, bio); + bio = bl_submit_bio(bio); goto retry; } return bio; @@ -252,7 +254,7 @@ bl_read_pagelist(struct nfs_pgio_header *header) for (i = pg_index; i < header->page_array.npages; i++) { if (extent_length <= 0) { /* We've used up the previous extent */ - bio = bl_submit_bio(READ, bio); + bio = bl_submit_bio(bio); /* Get the next one */ if (!ext_tree_lookup(bl, isect, &be, false)) { @@ -273,7 +275,7 @@ bl_read_pagelist(struct nfs_pgio_header *header) } if (is_hole(&be)) { - bio = bl_submit_bio(READ, bio); + bio = bl_submit_bio(bio); /* Fill hole w/ zeroes w/o accessing device */ dprintk("%s Zeroing page for hole\n", __func__); zero_user_segment(pages[i], pg_offset, pg_len); @@ -306,7 +308,7 @@ bl_read_pagelist(struct nfs_pgio_header *header) header->res.count = (isect << SECTOR_SHIFT) - header->args.offset; } out: - bl_submit_bio(READ, bio); + bl_submit_bio(bio); blk_finish_plug(&plug); put_parallel(par); return PNFS_ATTEMPTED; @@ -398,7 +400,7 @@ bl_write_pagelist(struct nfs_pgio_header *header, int sync) for (i = pg_index; i < header->page_array.npages; i++) { if (extent_length <= 0) { /* We've used up the previous extent */ - bio = bl_submit_bio(WRITE, bio); + bio = bl_submit_bio(bio); /* Get the next one */ if (!ext_tree_lookup(bl, isect, &be, true)) { header->pnfs_error = -EINVAL; @@ -427,7 +429,7 @@ bl_write_pagelist(struct nfs_pgio_header *header, int sync) header->res.count = header->args.count; out: - bl_submit_bio(WRITE, bio); + bl_submit_bio(bio); blk_finish_plug(&plug); put_parallel(par); return PNFS_ATTEMPTED; diff --git a/fs/nilfs2/segbuf.c b/fs/nilfs2/segbuf.c index f63620c..7666f1d 100644 --- a/fs/nilfs2/segbuf.c +++ b/fs/nilfs2/segbuf.c @@ -368,7 +368,8 @@ static int nilfs_segbuf_submit_bio(struct nilfs_segment_buffer *segbuf, bio->bi_end_io = nilfs_end_bio_write; bio->bi_private = segbuf; - submit_bio(mode, bio); + bio->bi_rw = mode; + submit_bio(bio); segbuf->sb_nbio++; wi->bio = NULL; diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c index a3cc6d2..119bd85 100644 --- a/fs/ocfs2/cluster/heartbeat.c +++ b/fs/ocfs2/cluster/heartbeat.c @@ -391,7 +391,7 @@ static void o2hb_bio_end_io(struct bio *bio) static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, struct o2hb_bio_wait_ctxt *wc, unsigned int *current_slot, - unsigned int max_slots) + unsigned int max_slots, int rw) { int len, current_page; unsigned int vec_len, vec_start; @@ -417,6 +417,7 @@ static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, bio->bi_bdev = reg->hr_bdev; bio->bi_private = wc; bio->bi_end_io = o2hb_bio_end_io; + bio->bi_rw = rw; vec_start = (cs << bits) % PAGE_CACHE_SIZE; while(cs < max_slots) { @@ -452,7 +453,8 @@ static int o2hb_read_slots(struct o2hb_region *reg, o2hb_bio_wait_init(&wc); while(current_slot < max_slots) { - bio = o2hb_setup_one_bio(reg, &wc, ¤t_slot, max_slots); + bio = o2hb_setup_one_bio(reg, &wc, ¤t_slot, max_slots, + READ); if (IS_ERR(bio)) { status = PTR_ERR(bio); mlog_errno(status); @@ -460,7 +462,7 @@ static int o2hb_read_slots(struct o2hb_region *reg, } atomic_inc(&wc.wc_num_reqs); - submit_bio(READ, bio); + submit_bio(bio); } status = 0; @@ -484,7 +486,7 @@ static int o2hb_issue_node_write(struct o2hb_region *reg, slot = o2nm_this_node(); - bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1); + bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1, WRITE_SYNC); if (IS_ERR(bio)) { status = PTR_ERR(bio); mlog_errno(status); @@ -492,7 +494,7 @@ static int o2hb_issue_node_write(struct o2hb_region *reg, } atomic_inc(&write_wc->wc_num_reqs); - submit_bio(WRITE_SYNC, bio); + submit_bio(bio); status = 0; bail: diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 29e7e5d..5525a1b 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -382,7 +382,8 @@ xfs_submit_ioend_bio( atomic_inc(&ioend->io_remaining); bio->bi_private = ioend; bio->bi_end_io = xfs_end_bio; - submit_bio(wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE, bio); + bio->bi_rw = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + submit_bio(bio); } STATIC struct bio * diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index def8860..9fd85d5 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1163,7 +1163,7 @@ next_chunk: bio->bi_iter.bi_sector = sector; bio->bi_end_io = xfs_buf_bio_end_io; bio->bi_private = bp; - + bio->bi_rw = rw; for (; size && nr_pages; nr_pages--, page_index++) { int rbytes, nbytes = PAGE_SIZE - offset; @@ -1187,7 +1187,7 @@ next_chunk: flush_kernel_vmap_range(bp->b_addr, xfs_buf_vmap_len(bp)); } - submit_bio(rw, bio); + submit_bio(bio); if (size) goto next_chunk; } else { diff --git a/include/linux/bio.h b/include/linux/bio.h index 5349e68..9f0e2cf 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -441,7 +441,7 @@ static inline void bio_io_error(struct bio *bio) struct request_queue; extern int bio_phys_segments(struct request_queue *, struct bio *); -extern int submit_bio_wait(int rw, struct bio *bio); +extern int submit_bio_wait(struct bio *bio); extern void bio_advance(struct bio *, unsigned); extern void bio_init(struct bio *); diff --git a/include/linux/fs.h b/include/linux/fs.h index 94dfe09..afa9df4 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2644,7 +2644,7 @@ static inline void remove_inode_hash(struct inode *inode) extern void inode_sb_list_add(struct inode *inode); #ifdef CONFIG_BLOCK -extern blk_qc_t submit_bio(int, struct bio *); +extern blk_qc_t submit_bio(struct bio *); extern int bdev_read_only(struct block_device *); #endif extern int set_blocksize(struct block_device *, int); diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 12cd989..4d050eb 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -260,6 +260,7 @@ static int hib_submit_io(int rw, pgoff_t page_off, void *addr, bio = bio_alloc(__GFP_RECLAIM | __GFP_HIGH, 1); bio->bi_iter.bi_sector = page_off * (PAGE_SIZE >> 9); bio->bi_bdev = hib_resume_bdev; + bio->bi_rw = rw; if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) { printk(KERN_ERR "PM: Adding page to bio failed at %llu\n", @@ -272,9 +273,9 @@ static int hib_submit_io(int rw, pgoff_t page_off, void *addr, bio->bi_end_io = hib_end_io; bio->bi_private = hb; atomic_inc(&hb->count); - submit_bio(rw, bio); + submit_bio(bio); } else { - error = submit_bio_wait(rw, bio); + error = submit_bio_wait(bio); bio_put(bio); } diff --git a/mm/page_io.c b/mm/page_io.c index b995a5b..e8a18a7 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -253,7 +253,7 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, bio_end_io_t end_write_func) { struct bio *bio; - int ret, rw = WRITE; + int ret; struct swap_info_struct *sis = page_swap_info(page); if (sis->flags & SWP_FILE) { @@ -311,12 +311,13 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, ret = -ENOMEM; goto out; } + bio->bi_rw = WRITE; if (wbc->sync_mode == WB_SYNC_ALL) - rw |= REQ_SYNC; + bio->bi_rw |= REQ_SYNC; count_vm_event(PSWPOUT); set_page_writeback(page); unlock_page(page); - submit_bio(rw, bio); + submit_bio(bio); out: return ret; } @@ -358,8 +359,9 @@ int swap_readpage(struct page *page) ret = -ENOMEM; goto out; } + bio->bi_rw = READ; count_vm_event(PSWPIN); - submit_bio(READ, bio); + submit_bio(bio); out: return ret; } -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:21:51 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 8AE6C29DFA for ; Mon, 11 Jan 2016 14:21:51 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 7AF1030405F for ; Mon, 11 Jan 2016 12:21:51 -0800 (PST) X-ASG-Debug-ID: 1452543709-04cb6c75dd79020001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id kjEHs84i8AgxvmFh (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:21:49 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 6F2B6A0E5C; Mon, 11 Jan 2016 20:21:48 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEW008385; Mon, 11 Jan 2016 15:21:46 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 03/35] block, fs, mm, drivers: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:01 -0600 X-ASG-Orig-Subj: [PATCH 03/35] block, fs, mm, drivers: set bi_op to REQ_OP Message-Id: <1452543693-4440-4-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543709 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch converts the simple bi_rw use cases in the block, drivers, mm and fs code to use bi_op for a REQ_OP and bi_rw for rq_flag_bits. These should be simple one liner cases, so I just did them in one patch. The next patches handle the more complicated cases in a module per patch. Signed-off-by: Mike Christie --- block/bio.c | 8 +++++--- block/blk-flush.c | 1 + block/blk-lib.c | 7 ++++--- block/blk-map.c | 2 +- drivers/block/floppy.c | 2 +- drivers/block/pktcdvd.c | 4 ++-- drivers/lightnvm/rrpc.c | 4 ++-- drivers/scsi/osd/osd_initiator.c | 8 ++++---- fs/exofs/ore.c | 2 +- fs/ext4/crypto.c | 2 +- fs/ext4/page-io.c | 8 +++++--- fs/ext4/readpage.c | 2 +- fs/jfs/jfs_logmgr.c | 2 ++ fs/jfs/jfs_metapage.c | 4 ++-- fs/logfs/dev_bdev.c | 12 ++++++------ fs/nfs/blocklayout/blocklayout.c | 2 +- mm/page_io.c | 4 ++-- 17 files changed, 41 insertions(+), 33 deletions(-) diff --git a/block/bio.c b/block/bio.c index 921112b..3b8e970 100644 --- a/block/bio.c +++ b/block/bio.c @@ -581,6 +581,7 @@ void __bio_clone_fast(struct bio *bio, struct bio *bio_src) */ bio->bi_bdev = bio_src->bi_bdev; bio_set_flag(bio, BIO_CLONED); + bio->bi_op = bio_src->bi_op; bio->bi_rw = bio_src->bi_rw; bio->bi_iter = bio_src->bi_iter; bio->bi_io_vec = bio_src->bi_io_vec; @@ -663,6 +664,7 @@ struct bio *bio_clone_bioset(struct bio *bio_src, gfp_t gfp_mask, return NULL; bio->bi_bdev = bio_src->bi_bdev; + bio->bi_op = bio_src->bi_op; bio->bi_rw = bio_src->bi_rw; bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; bio->bi_iter.bi_size = bio_src->bi_iter.bi_size; @@ -1168,7 +1170,7 @@ struct bio *bio_copy_user_iov(struct request_queue *q, goto out_bmd; if (iter->type & WRITE) - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; ret = 0; @@ -1338,7 +1340,7 @@ struct bio *bio_map_user_iov(struct request_queue *q, * set data direction, and check if mapped pages need bouncing */ if (iter->type & WRITE) - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; bio_set_flag(bio, BIO_USER_MAPPED); @@ -1531,7 +1533,7 @@ struct bio *bio_copy_kern(struct request_queue *q, void *data, unsigned int len, bio->bi_private = data; } else { bio->bi_end_io = bio_copy_kern_endio; - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; } return bio; diff --git a/block/blk-flush.c b/block/blk-flush.c index f2fbf9a..b05acca 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -484,6 +484,7 @@ int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, bio = bio_alloc(gfp_mask, 0); bio->bi_bdev = bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw = WRITE_FLUSH; ret = submit_bio_wait(bio); diff --git a/block/blk-lib.c b/block/blk-lib.c index 87e3de4..d01b5f2 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -42,7 +42,7 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, { DECLARE_COMPLETION_ONSTACK(wait); struct request_queue *q = bdev_get_queue(bdev); - int type = REQ_WRITE | REQ_DISCARD; + int type = 0; unsigned int granularity; int alignment; struct bio_batch bb; @@ -102,6 +102,7 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, bio->bi_end_io = bio_batch_end_io; bio->bi_bdev = bdev; bio->bi_private = &bb; + bio->bi_op = REQ_OP_DISCARD; bio->bi_rw = type; bio->bi_iter.bi_size = req_sects << 9; @@ -178,7 +179,7 @@ int blkdev_issue_write_same(struct block_device *bdev, sector_t sector, bio->bi_io_vec->bv_page = page; bio->bi_io_vec->bv_offset = 0; bio->bi_io_vec->bv_len = bdev_logical_block_size(bdev); - bio->bi_rw = REQ_WRITE | REQ_WRITE_SAME; + bio->bi_op = REQ_OP_WRITE_SAME; if (nr_sects > max_write_same_sectors) { bio->bi_iter.bi_size = max_write_same_sectors << 9; @@ -240,7 +241,7 @@ static int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, bio->bi_bdev = bdev; bio->bi_end_io = bio_batch_end_io; bio->bi_private = &bb; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; while (nr_sects != 0) { sz = min((sector_t) PAGE_SIZE >> 9 , nr_sects); diff --git a/block/blk-map.c b/block/blk-map.c index f565e11..9021a8f 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -224,7 +224,7 @@ int blk_rq_map_kern(struct request_queue *q, struct request *rq, void *kbuf, return PTR_ERR(bio); if (!reading) - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; if (do_copy) rq->cmd_flags |= REQ_COPY_USER; diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c index aebbdb9..3fda035 100644 --- a/drivers/block/floppy.c +++ b/drivers/block/floppy.c @@ -3817,7 +3817,7 @@ static int __floppy_read_block_0(struct block_device *bdev, int drive) bio.bi_flags |= (1 << BIO_QUIET); bio.bi_private = &cbdata; bio.bi_end_io = floppy_rb0_cb; - bio.bi_rw = READ; + bio.bi_op = REQ_OP_READ; submit_bio(&bio); process_fd_request(); diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c index d06c62e..a4ecbd3 100644 --- a/drivers/block/pktcdvd.c +++ b/drivers/block/pktcdvd.c @@ -1074,7 +1074,7 @@ static void pkt_gather_data(struct pktcdvd_device *pd, struct packet_data *pkt) BUG(); atomic_inc(&pkt->io_wait); - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; pkt_queue_bio(pd, bio); frames_read++; } @@ -1336,7 +1336,7 @@ static void pkt_start_write(struct pktcdvd_device *pd, struct packet_data *pkt) /* Start the write request */ atomic_set(&pkt->io_wait, 1); - pkt->w_bio->bi_rw = WRITE; + pkt->w_bio->bi_op = REQ_OP_WRITE; pkt_queue_bio(pd, pkt->w_bio); } diff --git a/drivers/lightnvm/rrpc.c b/drivers/lightnvm/rrpc.c index 134e4fa..c20cb18 100644 --- a/drivers/lightnvm/rrpc.c +++ b/drivers/lightnvm/rrpc.c @@ -315,7 +315,7 @@ try: /* Perform read to do GC */ bio->bi_iter.bi_sector = rrpc_get_sector(rev->addr); - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_private = &wait; bio->bi_end_io = rrpc_end_sync_bio; @@ -333,7 +333,7 @@ try: reinit_completion(&wait); bio->bi_iter.bi_sector = rrpc_get_sector(rev->addr); - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_private = &wait; bio->bi_end_io = rrpc_end_sync_bio; diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c index d8a2b51..4e7b440 100644 --- a/drivers/scsi/osd/osd_initiator.c +++ b/drivers/scsi/osd/osd_initiator.c @@ -726,7 +726,7 @@ static int _osd_req_list_objects(struct osd_request *or, return PTR_ERR(bio); } - bio->bi_rw &= ~REQ_WRITE; + bio->bi_op = REQ_OP_READ; or->in.bio = bio; or->in.total_bytes = bio->bi_iter.bi_size; return 0; @@ -839,7 +839,7 @@ int osd_req_write_kern(struct osd_request *or, if (IS_ERR(bio)) return PTR_ERR(bio); - bio->bi_rw |= REQ_WRITE; /* FIXME: bio_set_dir() */ + bio->bi_op = REQ_OP_WRITE; osd_req_write(or, obj, offset, bio, len); return 0; } @@ -956,7 +956,7 @@ static int _osd_req_finalize_cdb_cont(struct osd_request *or, const u8 *cap_key) if (IS_ERR(bio)) return PTR_ERR(bio); - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; /* integrity check the continuation before the bio is linked * with the other data segments since the continuation @@ -1077,7 +1077,7 @@ int osd_req_write_sg_kern(struct osd_request *or, if (IS_ERR(bio)) return PTR_ERR(bio); - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; osd_req_write_sg(or, obj, bio, sglist, numentries); return 0; diff --git a/fs/exofs/ore.c b/fs/exofs/ore.c index 7bd8ac8..c40ed74 100644 --- a/fs/exofs/ore.c +++ b/fs/exofs/ore.c @@ -878,7 +878,7 @@ static int _write_mirror(struct ore_io_state *ios, int cur_comp) } else { bio = master_dev->bio; /* FIXME: bio_set_dir() */ - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; } osd_req_write(or, _ios_obj(ios, cur_comp), diff --git a/fs/ext4/crypto.c b/fs/ext4/crypto.c index 02f2070..cc884e2 100644 --- a/fs/ext4/crypto.c +++ b/fs/ext4/crypto.c @@ -424,7 +424,7 @@ int ext4_encrypted_zeroout(struct inode *inode, ext4_lblk_t lblk, bio->bi_bdev = inode->i_sb->s_bdev; bio->bi_iter.bi_sector = pblk << (inode->i_sb->s_blocksize_bits - 9); - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; ret = bio_add_page(bio, ciphertext_page, inode->i_sb->s_blocksize, 0); if (ret != inode->i_sb->s_blocksize) { diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index cf63ac3..046863c 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -353,10 +353,12 @@ void ext4_io_submit(struct ext4_io_submit *io) struct bio *bio = io->io_bio; if (bio) { - int io_op = io->io_wbc->sync_mode == WB_SYNC_ALL ? - WRITE_SYNC : WRITE; + int io_op_flags = io->io_wbc->sync_mode == WB_SYNC_ALL ? + WRITE_SYNC : 0; + bio_get(io->io_bio); - io->io_bio->bi_rw = io_op; + io->io_bio->bi_op = REQ_OP_WRITE; + io->io_bio->bi_rw = io_op_flags; submit_bio(io->io_bio); bio_put(io->io_bio); } diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c index 893dc15..fb36434 100644 --- a/fs/ext4/readpage.c +++ b/fs/ext4/readpage.c @@ -294,7 +294,7 @@ int ext4_mpage_readpages(struct address_space *mapping, bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9); bio->bi_end_io = mpage_end_io; bio->bi_private = ctx; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; } length = first_hole << blkbits; diff --git a/fs/jfs/jfs_logmgr.c b/fs/jfs/jfs_logmgr.c index b0e068f..00227f8 100644 --- a/fs/jfs/jfs_logmgr.c +++ b/fs/jfs/jfs_logmgr.c @@ -2004,6 +2004,7 @@ static int lbmRead(struct jfs_log * log, int pn, struct lbuf ** bpp) bio->bi_end_io = lbmIODone; bio->bi_private = bp; + bio->bi_op = REQ_OP_READ; bio->bi_rw = READ_SYNC; /*check if journaling to disk has been disabled*/ if (log->no_integrity) { @@ -2148,6 +2149,7 @@ static void lbmStartIO(struct lbuf * bp) bio->bi_end_io = lbmIODone; bio->bi_private = bp; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw = WRITE_SYNC; /* check if journaling to disk has been disabled */ diff --git a/fs/jfs/jfs_metapage.c b/fs/jfs/jfs_metapage.c index bc29972..115d99e 100644 --- a/fs/jfs/jfs_metapage.c +++ b/fs/jfs/jfs_metapage.c @@ -434,7 +434,7 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) bio->bi_iter.bi_sector = pblock << (inode->i_blkbits - 9); bio->bi_end_io = metapage_write_end_io; bio->bi_private = page; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; /* Don't call bio_add_page yet, we may add to this vec */ bio_offset = offset; @@ -515,7 +515,7 @@ static int metapage_readpage(struct file *fp, struct page *page) pblock << (inode->i_blkbits - 9); bio->bi_end_io = metapage_read_end_io; bio->bi_private = page; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; len = xlen << inode->i_blkbits; offset = block_offset << inode->i_blkbits; if (bio_add_page(bio, page, len, offset) < len) diff --git a/fs/logfs/dev_bdev.c b/fs/logfs/dev_bdev.c index 9e690ad..130b7e2 100644 --- a/fs/logfs/dev_bdev.c +++ b/fs/logfs/dev_bdev.c @@ -14,7 +14,7 @@ #define PAGE_OFS(ofs) ((ofs) & (PAGE_SIZE-1)) -static int sync_request(struct page *page, struct block_device *bdev, int rw) +static int sync_request(struct page *page, struct block_device *bdev, int op) { struct bio bio; struct bio_vec bio_vec; @@ -29,7 +29,7 @@ static int sync_request(struct page *page, struct block_device *bdev, int rw) bio.bi_bdev = bdev; bio.bi_iter.bi_sector = page->index * (PAGE_SIZE >> 9); bio.bi_iter.bi_size = PAGE_SIZE; - bio.bi_rw = rw; + bio.bi_op = op; return submit_bio_wait(&bio); } @@ -96,7 +96,7 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = writeseg_end_io; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; atomic_inc(&super->s_pending_writes); submit_bio(bio); @@ -124,7 +124,7 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = writeseg_end_io; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; atomic_inc(&super->s_pending_writes); submit_bio(bio); return 0; @@ -188,7 +188,7 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = erase_end_io; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; atomic_inc(&super->s_pending_writes); submit_bio(bio); @@ -210,7 +210,7 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index, bio->bi_iter.bi_sector = ofs >> 9; bio->bi_private = sb; bio->bi_end_io = erase_end_io; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; atomic_inc(&super->s_pending_writes); submit_bio(bio); return 0; diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index 8c0443c..3832909 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -175,7 +175,7 @@ retry: disk_addr >> SECTOR_SHIFT, end_io, par); if (!bio) return ERR_PTR(-ENOMEM); - bio->bi_rw = rw; + bio->bi_op = rw; } if (bio_add_page(bio, page, *len, offset) < *len) { bio = bl_submit_bio(bio); diff --git a/mm/page_io.c b/mm/page_io.c index e8a18a7..e48b84c 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -311,7 +311,7 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, ret = -ENOMEM; goto out; } - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; if (wbc->sync_mode == WB_SYNC_ALL) bio->bi_rw |= REQ_SYNC; count_vm_event(PSWPOUT); @@ -359,7 +359,7 @@ int swap_readpage(struct page *page) ret = -ENOMEM; goto out; } - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; count_vm_event(PSWPIN); submit_bio(bio); out: -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:21:54 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 68D9C29E0A for ; Mon, 11 Jan 2016 14:21:54 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 09618AC002 for ; Mon, 11 Jan 2016 12:21:47 -0800 (PST) X-ASG-Debug-ID: 1452543702-04cb6c75dc79000001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id g7zvjOQCKBLtYU73 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:21:42 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 2D3386E799; Mon, 11 Jan 2016 20:21:41 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZET008385; Mon, 11 Jan 2016 15:21:35 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Subject: [PATCH 00/35 v3] eparate operations from flags in the bio/request structs Date: Mon, 11 Jan 2016 14:20:58 -0600 X-ASG-Orig-Subj: [PATCH 00/35 v3] eparate operations from flags in the bio/request structs Message-Id: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543702 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 The following patches begin to cleanup the request->cmd_flags and bio->bi_rw mess. We currently use cmd_flags to specify the operation, attributes and state of the request. For bi_rw we use it for similar info and also the priority but then also have another bi_flags field for state. At some point, we abused them so much we just made cmd_flags 64 bits, so we could add more. The following patches seperate the operation (read, write discard, flush, etc) from cmd_flags/bi_rw. This patchset was made against linux-next from today Jan 11 2016. (git tag next-20160111). I put a git tree here: https://github.com/mikechristie/linux-kernel.git v3: 1. Used "=" instead of "|=" to setup bio bi_rw. 2. Removed __get_request cmd_flags compat code. 3. Merged initial dm related changes requested by Mike Snitzer. 4. Fixed ubd kbuild errors in flush related patches. 5. Fix 80 char col issues in several patches. 6. Fix issue with one of the btrfs patches where it looks like I reverted a patch when trying to fix a merge error. v2 1. Dropped arguments from submit_bio, and had callers setup bio. 2. Add REQ_OP_FLUSH for request_fn users and renamed REQ_FLUSH to REQ_PREFLUSH for make_request_fn users. 3. Dropped bio/rq_data_dir functions, and added a op_is_write function instead. From mchristi@redhat.com Mon Jan 11 14:21:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id C05BF29E0E for ; Mon, 11 Jan 2016 14:21:55 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 34A6BAC005 for ; Mon, 11 Jan 2016 12:21:55 -0800 (PST) X-ASG-Debug-ID: 1452543711-04cbb05f7f97890001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id jEFpLlM28op8ZgoE (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:21:51 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id C83F6C0BF2A1; Mon, 11 Jan 2016 20:21:50 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEX008385; Mon, 11 Jan 2016 15:21:48 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 04/35] fs: have submit_bh users pass in op and flags separately Date: Mon, 11 Jan 2016 14:21:02 -0600 X-ASG-Orig-Subj: [PATCH 04/35] fs: have submit_bh users pass in op and flags separately Message-Id: <1452543693-4440-5-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543711 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This has submit_bh users pass in the operation and flags separately, so we can setup the bio->bi_op and bio-bi_rw flags. Signed-off-by: Mike Christie --- drivers/md/bitmap.c | 4 ++-- fs/btrfs/check-integrity.c | 24 ++++++++++---------- fs/btrfs/check-integrity.h | 2 +- fs/btrfs/disk-io.c | 4 ++-- fs/buffer.c | 54 +++++++++++++++++++++++---------------------- fs/ext4/balloc.c | 2 +- fs/ext4/ialloc.c | 2 +- fs/ext4/inode.c | 2 +- fs/ext4/mmp.c | 4 ++-- fs/fat/misc.c | 2 +- fs/gfs2/bmap.c | 2 +- fs/gfs2/dir.c | 2 +- fs/gfs2/meta_io.c | 6 ++--- fs/jbd2/commit.c | 6 ++--- fs/jbd2/journal.c | 8 +++---- fs/nilfs2/btnode.c | 6 ++--- fs/nilfs2/btnode.h | 2 +- fs/nilfs2/btree.c | 6 +++-- fs/nilfs2/gcinode.c | 5 +++-- fs/nilfs2/mdt.c | 11 ++++----- fs/ntfs/aops.c | 6 ++--- fs/ntfs/compress.c | 2 +- fs/ntfs/file.c | 2 +- fs/ntfs/logfile.c | 2 +- fs/ntfs/mft.c | 4 ++-- fs/ocfs2/buffer_head_io.c | 8 +++---- fs/reiserfs/inode.c | 4 ++-- fs/reiserfs/journal.c | 6 ++--- fs/ufs/util.c | 2 +- include/linux/buffer_head.h | 9 ++++---- 30 files changed, 103 insertions(+), 96 deletions(-) diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c index 4f22e91..13811fc 100644 --- a/drivers/md/bitmap.c +++ b/drivers/md/bitmap.c @@ -299,7 +299,7 @@ static void write_page(struct bitmap *bitmap, struct page *page, int wait) atomic_inc(&bitmap->pending_writes); set_buffer_locked(bh); set_buffer_mapped(bh); - submit_bh(WRITE | REQ_SYNC, bh); + submit_bh(REQ_OP_WRITE, REQ_SYNC, bh); bh = bh->b_this_page; } @@ -394,7 +394,7 @@ static int read_page(struct file *file, unsigned long index, atomic_inc(&bitmap->pending_writes); set_buffer_locked(bh); set_buffer_mapped(bh); - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); } block++; bh = bh->b_this_page; diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index dff4ab4..036598f 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -2898,12 +2898,12 @@ static struct btrfsic_dev_state *btrfsic_dev_state_lookup( return ds; } -int btrfsic_submit_bh(int rw, struct buffer_head *bh) +int btrfsic_submit_bh(int op, int op_flags, struct buffer_head *bh) { struct btrfsic_dev_state *dev_state; if (!btrfsic_is_initialized) - return submit_bh(rw, bh); + return submit_bh(op, op_flags, bh); mutex_lock(&btrfsic_mutex); /* since btrfsic_submit_bh() might also be called before @@ -2912,26 +2912,26 @@ int btrfsic_submit_bh(int rw, struct buffer_head *bh) /* Only called to write the superblock (incl. FLUSH/FUA) */ if (NULL != dev_state && - (rw & WRITE) && bh->b_size > 0) { + (op == REQ_OP_WRITE) && bh->b_size > 0) { u64 dev_bytenr; dev_bytenr = 4096 * bh->b_blocknr; if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bh(rw=0x%x, blocknr=%llu (bytenr %llu)," - " size=%zu, data=%p, bdev=%p)\n", - rw, (unsigned long long)bh->b_blocknr, + "submit_bh(op=0x%x,0x%x, blocknr=%llu " + "(bytenr %llu), size=%zu, data=%p, bdev=%p)\n", + op, op_flags, (unsigned long long)bh->b_blocknr, dev_bytenr, bh->b_size, bh->b_data, bh->b_bdev); btrfsic_process_written_block(dev_state, dev_bytenr, &bh->b_data, 1, NULL, - NULL, bh, rw); - } else if (NULL != dev_state && (rw & REQ_FLUSH)) { + NULL, bh, op_flags); + } else if (NULL != dev_state && (op_flags & REQ_FLUSH)) { if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bh(rw=0x%x FLUSH, bdev=%p)\n", - rw, bh->b_bdev); + "submit_bh(op=0x%x,0x%x FLUSH, bdev=%p)\n", + op, op_flags, bh->b_bdev); if (!dev_state->dummy_block_for_bio_bh_flush.is_iodone) { if ((dev_state->state->print_mask & (BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH | @@ -2949,7 +2949,7 @@ int btrfsic_submit_bh(int rw, struct buffer_head *bh) block->never_written = 0; block->iodone_w_error = 0; block->flush_gen = dev_state->last_flush_gen + 1; - block->submit_bio_bh_rw = rw; + block->submit_bio_bh_rw = op_flags; block->orig_bio_bh_private = bh->b_private; block->orig_bio_bh_end_io.bh = bh->b_end_io; block->next_in_same_bio = NULL; @@ -2958,7 +2958,7 @@ int btrfsic_submit_bh(int rw, struct buffer_head *bh) } } mutex_unlock(&btrfsic_mutex); - return submit_bh(rw, bh); + return submit_bh(op, op_flags, bh); } static void __btrfsic_submit_bio(struct bio *bio) diff --git a/fs/btrfs/check-integrity.h b/fs/btrfs/check-integrity.h index c04e249..f78dff1 100644 --- a/fs/btrfs/check-integrity.h +++ b/fs/btrfs/check-integrity.h @@ -20,7 +20,7 @@ #define __BTRFS_CHECK_INTEGRITY__ #ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY -int btrfsic_submit_bh(int rw, struct buffer_head *bh); +int btrfsic_submit_bh(int op, int op_flags, struct buffer_head *bh); void btrfsic_submit_bio(struct bio *bio); int btrfsic_submit_bio_wait(struct bio *bio); #else diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 23f45f3..2eb3f10 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -3388,9 +3388,9 @@ static int write_dev_supers(struct btrfs_device *device, * to go down lazy. */ if (i == 0) - ret = btrfsic_submit_bh(WRITE_FUA, bh); + ret = btrfsic_submit_bh(REQ_OP_WRITE, WRITE_FUA, bh); else - ret = btrfsic_submit_bh(WRITE_SYNC, bh); + ret = btrfsic_submit_bh(REQ_OP_WRITE, WRITE_SYNC, bh); if (ret) errors++; } diff --git a/fs/buffer.c b/fs/buffer.c index 4349245..2343a40 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -45,7 +45,7 @@ #include static int fsync_buffers_list(spinlock_t *lock, struct list_head *list); -static int submit_bh_wbc(int rw, struct buffer_head *bh, +static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, unsigned long bio_flags, struct writeback_control *wbc); @@ -1227,7 +1227,7 @@ static struct buffer_head *__bread_slow(struct buffer_head *bh) } else { get_bh(bh); bh->b_end_io = end_buffer_read_sync; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); wait_on_buffer(bh); if (buffer_uptodate(bh)) return bh; @@ -1699,7 +1699,7 @@ static int __block_write_full_page(struct inode *inode, struct page *page, struct buffer_head *bh, *head; unsigned int blocksize, bbits; int nr_underway = 0; - int write_op = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + int write_flags = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : 0); head = create_page_buffers(page, inode, (1 << BH_Dirty)|(1 << BH_Uptodate)); @@ -1788,7 +1788,7 @@ static int __block_write_full_page(struct inode *inode, struct page *page, do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh_wbc(write_op, bh, 0, wbc); + submit_bh_wbc(REQ_OP_WRITE, write_flags, bh, 0, wbc); nr_underway++; } bh = next; @@ -1842,7 +1842,7 @@ recover: struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { clear_buffer_dirty(bh); - submit_bh_wbc(write_op, bh, 0, wbc); + submit_bh_wbc(REQ_OP_WRITE, write_flags, bh, 0, wbc); nr_underway++; } bh = next; @@ -2250,7 +2250,7 @@ int block_read_full_page(struct page *page, get_block_t *get_block) if (buffer_uptodate(bh)) end_buffer_async_read(bh, 1); else - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); } return 0; } @@ -2584,7 +2584,7 @@ int nobh_write_begin(struct address_space *mapping, if (block_start < from || block_end > to) { lock_buffer(bh); bh->b_end_io = end_buffer_read_nobh; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); nr_reads++; } } @@ -2951,7 +2951,7 @@ static void end_bio_bh_io_sync(struct bio *bio) * errors, this only handles the "we need to be able to * do IO at the final sector" case. */ -void guard_bio_eod(int rw, struct bio *bio) +void guard_bio_eod(int op, struct bio *bio) { sector_t maxsector; struct bio_vec *bvec = &bio->bi_io_vec[bio->bi_vcnt - 1]; @@ -2981,13 +2981,13 @@ void guard_bio_eod(int rw, struct bio *bio) bvec->bv_len -= truncated_bytes; /* ..and clear the end of the buffer for reads */ - if ((rw & RW_MASK) == READ) { + if (op == REQ_OP_READ) { zero_user(bvec->bv_page, bvec->bv_offset + bvec->bv_len, truncated_bytes); } } -static int submit_bh_wbc(int rw, struct buffer_head *bh, +static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, unsigned long bio_flags, struct writeback_control *wbc) { struct bio *bio; @@ -3001,7 +3001,7 @@ static int submit_bh_wbc(int rw, struct buffer_head *bh, /* * Only clear out a write error when rewriting */ - if (test_set_buffer_req(bh) && (rw & WRITE)) + if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE)) clear_buffer_write_io_error(bh); /* @@ -3024,29 +3024,31 @@ static int submit_bh_wbc(int rw, struct buffer_head *bh, bio->bi_end_io = end_bio_bh_io_sync; bio->bi_private = bh; bio->bi_flags |= bio_flags; + bio->bi_op = op; /* Take care of bh's that straddle the end of the device */ - guard_bio_eod(rw, bio); + guard_bio_eod(op, bio); if (buffer_meta(bh)) - rw |= REQ_META; + op_flags |= REQ_META; if (buffer_prio(bh)) - rw |= REQ_PRIO; - bio->bi_rw = rw; + op_flags |= REQ_PRIO; + bio->bi_rw = op_flags; submit_bio(bio); return 0; } -int _submit_bh(int rw, struct buffer_head *bh, unsigned long bio_flags) +int _submit_bh(int op, int op_flags, struct buffer_head *bh, + unsigned long bio_flags) { - return submit_bh_wbc(rw, bh, bio_flags, NULL); + return submit_bh_wbc(op, op_flags, bh, bio_flags, NULL); } EXPORT_SYMBOL_GPL(_submit_bh); -int submit_bh(int rw, struct buffer_head *bh) +int submit_bh(int op, int op_flags, struct buffer_head *bh) { - return submit_bh_wbc(rw, bh, 0, NULL); + return submit_bh_wbc(op, op_flags, bh, 0, NULL); } EXPORT_SYMBOL(submit_bh); @@ -3088,14 +3090,14 @@ void ll_rw_block(int rw, int nr, struct buffer_head *bhs[]) if (test_clear_buffer_dirty(bh)) { bh->b_end_io = end_buffer_write_sync; get_bh(bh); - submit_bh(WRITE, bh); + submit_bh(rw, 0, bh); continue; } } else { if (!buffer_uptodate(bh)) { bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(rw, bh); + submit_bh(rw, 0, bh); continue; } } @@ -3104,7 +3106,7 @@ void ll_rw_block(int rw, int nr, struct buffer_head *bhs[]) } EXPORT_SYMBOL(ll_rw_block); -void write_dirty_buffer(struct buffer_head *bh, int rw) +void write_dirty_buffer(struct buffer_head *bh, int op_flags) { lock_buffer(bh); if (!test_clear_buffer_dirty(bh)) { @@ -3113,7 +3115,7 @@ void write_dirty_buffer(struct buffer_head *bh, int rw) } bh->b_end_io = end_buffer_write_sync; get_bh(bh); - submit_bh(rw, bh); + submit_bh(REQ_OP_WRITE, op_flags, bh); } EXPORT_SYMBOL(write_dirty_buffer); @@ -3122,7 +3124,7 @@ EXPORT_SYMBOL(write_dirty_buffer); * and then start new I/O and then wait upon it. The caller must have a ref on * the buffer_head. */ -int __sync_dirty_buffer(struct buffer_head *bh, int rw) +int __sync_dirty_buffer(struct buffer_head *bh, int op_flags) { int ret = 0; @@ -3131,7 +3133,7 @@ int __sync_dirty_buffer(struct buffer_head *bh, int rw) if (test_clear_buffer_dirty(bh)) { get_bh(bh); bh->b_end_io = end_buffer_write_sync; - ret = submit_bh(rw, bh); + ret = submit_bh(REQ_OP_WRITE, op_flags, bh); wait_on_buffer(bh); if (!ret && !buffer_uptodate(bh)) ret = -EIO; @@ -3394,7 +3396,7 @@ int bh_submit_read(struct buffer_head *bh) get_bh(bh); bh->b_end_io = end_buffer_read_sync; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); wait_on_buffer(bh); if (buffer_uptodate(bh)) return 0; diff --git a/fs/ext4/balloc.c b/fs/ext4/balloc.c index ec0668a..b03bfd8 100644 --- a/fs/ext4/balloc.c +++ b/fs/ext4/balloc.c @@ -469,7 +469,7 @@ ext4_read_block_bitmap_nowait(struct super_block *sb, ext4_group_t block_group) trace_ext4_read_block_bitmap_load(sb, block_group); bh->b_end_io = ext4_end_bitmap_read; get_bh(bh); - submit_bh(READ | REQ_META | REQ_PRIO, bh); + submit_bh(REQ_OP_READ, REQ_META | REQ_PRIO, bh); return bh; verify: err = ext4_validate_block_bitmap(sb, desc, block_group, bh); diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c index e86e33c..0001b19 100644 --- a/fs/ext4/ialloc.c +++ b/fs/ext4/ialloc.c @@ -212,7 +212,7 @@ ext4_read_inode_bitmap(struct super_block *sb, ext4_group_t block_group) trace_ext4_load_inode_bitmap(sb, block_group); bh->b_end_io = ext4_end_bitmap_read; get_bh(bh); - submit_bh(READ | REQ_META | REQ_PRIO, bh); + submit_bh(REQ_OP_READ, REQ_META | REQ_PRIO, bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) { put_bh(bh); diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 032e6f2..4fc178a 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -4094,7 +4094,7 @@ make_io: trace_ext4_load_inode(inode); get_bh(bh); bh->b_end_io = end_buffer_read_sync; - submit_bh(READ | REQ_META | REQ_PRIO, bh); + submit_bh(REQ_OP_READ, REQ_META | REQ_PRIO, bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) { EXT4_ERROR_INODE_BLOCK(inode, block, diff --git a/fs/ext4/mmp.c b/fs/ext4/mmp.c index 0a512aa..bd90abf 100644 --- a/fs/ext4/mmp.c +++ b/fs/ext4/mmp.c @@ -52,7 +52,7 @@ static int write_mmp_block(struct super_block *sb, struct buffer_head *bh) lock_buffer(bh); bh->b_end_io = end_buffer_write_sync; get_bh(bh); - submit_bh(WRITE_SYNC | REQ_META | REQ_PRIO, bh); + submit_bh(REQ_OP_WRITE, WRITE_SYNC | REQ_META | REQ_PRIO, bh); wait_on_buffer(bh); sb_end_write(sb); if (unlikely(!buffer_uptodate(bh))) @@ -88,7 +88,7 @@ static int read_mmp_block(struct super_block *sb, struct buffer_head **bh, get_bh(*bh); lock_buffer(*bh); (*bh)->b_end_io = end_buffer_read_sync; - submit_bh(READ_SYNC | REQ_META | REQ_PRIO, *bh); + submit_bh(REQ_OP_READ, READ_SYNC | REQ_META | REQ_PRIO, *bh); wait_on_buffer(*bh); if (!buffer_uptodate(*bh)) { brelse(*bh); diff --git a/fs/fat/misc.c b/fs/fat/misc.c index c4589e9..8a86981 100644 --- a/fs/fat/misc.c +++ b/fs/fat/misc.c @@ -267,7 +267,7 @@ int fat_sync_bhs(struct buffer_head **bhs, int nr_bhs) int i, err = 0; for (i = 0; i < nr_bhs; i++) - write_dirty_buffer(bhs[i], WRITE); + write_dirty_buffer(bhs[i], 0); for (i = 0; i < nr_bhs; i++) { wait_on_buffer(bhs[i]); diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c index 0860f0b..29faf72 100644 --- a/fs/gfs2/bmap.c +++ b/fs/gfs2/bmap.c @@ -285,7 +285,7 @@ static void gfs2_metapath_ra(struct gfs2_glock *gl, if (trylock_buffer(rabh)) { if (!buffer_uptodate(rabh)) { rabh->b_end_io = end_buffer_read_sync; - submit_bh(READA | REQ_META, rabh); + submit_bh(REQ_OP_READ, READA | REQ_META, rabh); continue; } unlock_buffer(rabh); diff --git a/fs/gfs2/dir.c b/fs/gfs2/dir.c index 6a92592..2b81841 100644 --- a/fs/gfs2/dir.c +++ b/fs/gfs2/dir.c @@ -1510,7 +1510,7 @@ static void gfs2_dir_readahead(struct inode *inode, unsigned hsize, u32 index, continue; } bh->b_end_io = end_buffer_read_sync; - submit_bh(READA | REQ_META, bh); + submit_bh(REQ_OP_READ, READA | REQ_META, bh); continue; } brelse(bh); diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index 87a3abc..dc1c9db 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -37,8 +37,8 @@ static int gfs2_aspace_writepage(struct page *page, struct writeback_control *wb { struct buffer_head *bh, *head; int nr_underway = 0; - int write_op = REQ_META | REQ_PRIO | - (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + int write_flags = REQ_META | REQ_PRIO | + (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : 0); BUG_ON(!PageLocked(page)); BUG_ON(!page_has_buffers(page)); @@ -79,7 +79,7 @@ static int gfs2_aspace_writepage(struct page *page, struct writeback_control *wb do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh(write_op, bh); + submit_bh(REQ_OP_WRITE, write_flags, bh); nr_underway++; } bh = next; diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c index 36345fe..7390e3a 100644 --- a/fs/jbd2/commit.c +++ b/fs/jbd2/commit.c @@ -157,9 +157,9 @@ static int journal_submit_commit_record(journal_t *journal, if (journal->j_flags & JBD2_BARRIER && !jbd2_has_feature_async_commit(journal)) - ret = submit_bh(WRITE_SYNC | WRITE_FLUSH_FUA, bh); + ret = submit_bh(REQ_OP_WRITE, WRITE_SYNC | WRITE_FLUSH_FUA, bh); else - ret = submit_bh(WRITE_SYNC, bh); + ret = submit_bh(REQ_OP_WRITE, WRITE_SYNC, bh); *cbh = bh; return ret; @@ -737,7 +737,7 @@ start_journal_io: clear_buffer_dirty(bh); set_buffer_uptodate(bh); bh->b_end_io = journal_end_buffer_io_sync; - submit_bh(WRITE_SYNC, bh); + submit_bh(REQ_OP_WRITE, WRITE_SYNC, bh); } cond_resched(); stats.run.rs_blocks_logged += bufs; diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c index 81e6226..3b7db0b 100644 --- a/fs/jbd2/journal.c +++ b/fs/jbd2/journal.c @@ -1323,15 +1323,15 @@ static int journal_reset(journal_t *journal) return jbd2_journal_start_thread(journal); } -static int jbd2_write_superblock(journal_t *journal, int write_op) +static int jbd2_write_superblock(journal_t *journal, int write_flags) { struct buffer_head *bh = journal->j_sb_buffer; journal_superblock_t *sb = journal->j_superblock; int ret; - trace_jbd2_write_superblock(journal, write_op); + trace_jbd2_write_superblock(journal, write_flags); if (!(journal->j_flags & JBD2_BARRIER)) - write_op &= ~(REQ_FUA | REQ_FLUSH); + write_flags &= ~(REQ_FUA | REQ_FLUSH); lock_buffer(bh); if (buffer_write_io_error(bh)) { /* @@ -1351,7 +1351,7 @@ static int jbd2_write_superblock(journal_t *journal, int write_op) jbd2_superblock_csum_set(journal, sb); get_bh(bh); bh->b_end_io = end_buffer_write_sync; - ret = submit_bh(write_op, bh); + ret = submit_bh(REQ_OP_WRITE, write_flags, bh); wait_on_buffer(bh); if (buffer_write_io_error(bh)) { clear_buffer_write_io_error(bh); diff --git a/fs/nilfs2/btnode.c b/fs/nilfs2/btnode.c index a35ae35..07fe874 100644 --- a/fs/nilfs2/btnode.c +++ b/fs/nilfs2/btnode.c @@ -67,7 +67,7 @@ nilfs_btnode_create_block(struct address_space *btnc, __u64 blocknr) } int nilfs_btnode_submit_block(struct address_space *btnc, __u64 blocknr, - sector_t pblocknr, int mode, + sector_t pblocknr, int mode, int mode_flags, struct buffer_head **pbh, sector_t *submit_ptr) { struct buffer_head *bh; @@ -100,7 +100,7 @@ int nilfs_btnode_submit_block(struct address_space *btnc, __u64 blocknr, } } - if (mode == READA) { + if (mode_flags & REQ_RAHEAD) { if (pblocknr != *submit_ptr + 1 || !trylock_buffer(bh)) { err = -EBUSY; /* internal code */ brelse(bh); @@ -119,7 +119,7 @@ int nilfs_btnode_submit_block(struct address_space *btnc, __u64 blocknr, bh->b_blocknr = pblocknr; /* set block address for read */ bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(mode, bh); + submit_bh(mode, mode_flags, bh); bh->b_blocknr = blocknr; /* set back to the given block address */ *submit_ptr = pblocknr; err = 0; diff --git a/fs/nilfs2/btnode.h b/fs/nilfs2/btnode.h index d876b56..3f93197 100644 --- a/fs/nilfs2/btnode.h +++ b/fs/nilfs2/btnode.h @@ -47,7 +47,7 @@ void nilfs_btnode_cache_clear(struct address_space *); struct buffer_head *nilfs_btnode_create_block(struct address_space *btnc, __u64 blocknr); int nilfs_btnode_submit_block(struct address_space *, __u64, sector_t, int, - struct buffer_head **, sector_t *); + int, struct buffer_head **, sector_t *); void nilfs_btnode_delete(struct buffer_head *); int nilfs_btnode_prepare_change_key(struct address_space *, struct nilfs_btnode_chkey_ctxt *); diff --git a/fs/nilfs2/btree.c b/fs/nilfs2/btree.c index 3a3821b..5d6a2c6 100644 --- a/fs/nilfs2/btree.c +++ b/fs/nilfs2/btree.c @@ -480,7 +480,8 @@ static int __nilfs_btree_get_block(const struct nilfs_bmap *btree, __u64 ptr, sector_t submit_ptr = 0; int ret; - ret = nilfs_btnode_submit_block(btnc, ptr, 0, READ, &bh, &submit_ptr); + ret = nilfs_btnode_submit_block(btnc, ptr, 0, REQ_OP_READ, 0, &bh, + &submit_ptr); if (ret) { if (ret != -EEXIST) return ret; @@ -496,7 +497,8 @@ static int __nilfs_btree_get_block(const struct nilfs_bmap *btree, __u64 ptr, n > 0 && i < ra->ncmax; n--, i++) { ptr2 = nilfs_btree_node_get_ptr(ra->node, i, ra->ncmax); - ret = nilfs_btnode_submit_block(btnc, ptr2, 0, READA, + ret = nilfs_btnode_submit_block(btnc, ptr2, 0, + REQ_OP_READ, REQ_RAHEAD, &ra_bh, &submit_ptr); if (likely(!ret || ret == -EEXIST)) brelse(ra_bh); diff --git a/fs/nilfs2/gcinode.c b/fs/nilfs2/gcinode.c index 748ca23..1f18ffc 100644 --- a/fs/nilfs2/gcinode.c +++ b/fs/nilfs2/gcinode.c @@ -106,7 +106,7 @@ int nilfs_gccache_submit_read_data(struct inode *inode, sector_t blkoff, bh->b_blocknr = pbn; bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); if (vbn) bh->b_blocknr = vbn; out: @@ -143,7 +143,8 @@ int nilfs_gccache_submit_read_node(struct inode *inode, sector_t pbn, int ret; ret = nilfs_btnode_submit_block(&NILFS_I(inode)->i_btnode_cache, - vbn ? : pbn, pbn, READ, out_bh, &pbn); + vbn ? : pbn, pbn, REQ_OP_READ, 0, + out_bh, &pbn); if (ret == -EEXIST) /* internal code (cache hit) */ ret = 0; return ret; diff --git a/fs/nilfs2/mdt.c b/fs/nilfs2/mdt.c index 1125f40..92e627d 100644 --- a/fs/nilfs2/mdt.c +++ b/fs/nilfs2/mdt.c @@ -124,7 +124,7 @@ static int nilfs_mdt_create_block(struct inode *inode, unsigned long block, static int nilfs_mdt_submit_block(struct inode *inode, unsigned long blkoff, - int mode, struct buffer_head **out_bh) + int mode, int mode_flags, struct buffer_head **out_bh) { struct buffer_head *bh; __u64 blknum = 0; @@ -138,7 +138,7 @@ nilfs_mdt_submit_block(struct inode *inode, unsigned long blkoff, if (buffer_uptodate(bh)) goto out; - if (mode == READA) { + if (mode_flags & REQ_RAHEAD) { if (!trylock_buffer(bh)) { ret = -EBUSY; goto failed_bh; @@ -160,7 +160,7 @@ nilfs_mdt_submit_block(struct inode *inode, unsigned long blkoff, bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(mode, bh); + submit_bh(mode, mode_flags, bh); ret = 0; trace_nilfs2_mdt_submit_block(inode, inode->i_ino, blkoff, mode); @@ -184,7 +184,7 @@ static int nilfs_mdt_read_block(struct inode *inode, unsigned long block, int i, nr_ra_blocks = NILFS_MDT_MAX_RA_BLOCKS; int err; - err = nilfs_mdt_submit_block(inode, block, READ, &first_bh); + err = nilfs_mdt_submit_block(inode, block, REQ_OP_READ, 0, &first_bh); if (err == -EEXIST) /* internal code */ goto out; @@ -194,7 +194,8 @@ static int nilfs_mdt_read_block(struct inode *inode, unsigned long block, if (readahead) { blkoff = block + 1; for (i = 0; i < nr_ra_blocks; i++, blkoff++) { - err = nilfs_mdt_submit_block(inode, blkoff, READA, &bh); + err = nilfs_mdt_submit_block(inode, blkoff, REQ_OP_READ, + REQ_RAHEAD, &bh); if (likely(!err || err == -EEXIST)) brelse(bh); else if (err != -EBUSY) diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c index 7521e11..57c64bd 100644 --- a/fs/ntfs/aops.c +++ b/fs/ntfs/aops.c @@ -362,7 +362,7 @@ handle_zblock: for (i = 0; i < nr; i++) { tbh = arr[i]; if (likely(!buffer_uptodate(tbh))) - submit_bh(READ, tbh); + submit_bh(REQ_OP_READ, 0, tbh); else ntfs_end_buffer_async_read(tbh, 1); } @@ -877,7 +877,7 @@ lock_retry_remap: do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); need_end_writeback = false; } bh = next; @@ -1202,7 +1202,7 @@ lock_retry_remap: BUG_ON(!buffer_mapped(tbh)); get_bh(tbh); tbh->b_end_io = end_buffer_write_sync; - submit_bh(WRITE, tbh); + submit_bh(REQ_OP_WRITE, 0, tbh); } /* Synchronize the mft mirror now if not @sync. */ if (is_mft && !sync) diff --git a/fs/ntfs/compress.c b/fs/ntfs/compress.c index f82498c..1fd482c 100644 --- a/fs/ntfs/compress.c +++ b/fs/ntfs/compress.c @@ -674,7 +674,7 @@ lock_retry_remap: } get_bh(tbh); tbh->b_end_io = end_buffer_read_sync; - submit_bh(READ, tbh); + submit_bh(REQ_OP_READ, 0, tbh); } /* Wait for io completion on all buffer heads. */ diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c index 9d383e5..03a1a32 100644 --- a/fs/ntfs/file.c +++ b/fs/ntfs/file.c @@ -553,7 +553,7 @@ static inline int ntfs_submit_bh_for_read(struct buffer_head *bh) lock_buffer(bh); get_bh(bh); bh->b_end_io = end_buffer_read_sync; - return submit_bh(READ, bh); + return submit_bh(REQ_OP_READ, 0, bh); } /** diff --git a/fs/ntfs/logfile.c b/fs/ntfs/logfile.c index c71de29..1c95c41 100644 --- a/fs/ntfs/logfile.c +++ b/fs/ntfs/logfile.c @@ -821,7 +821,7 @@ map_vcn: * completed ignore errors afterwards as we can assume * that if one buffer worked all of them will work. */ - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); if (should_wait) { should_wait = false; wait_on_buffer(bh); diff --git a/fs/ntfs/mft.c b/fs/ntfs/mft.c index 3014a36..38c6f7a 100644 --- a/fs/ntfs/mft.c +++ b/fs/ntfs/mft.c @@ -592,7 +592,7 @@ int ntfs_sync_mft_mirror(ntfs_volume *vol, const unsigned long mft_no, clear_buffer_dirty(tbh); get_bh(tbh); tbh->b_end_io = end_buffer_write_sync; - submit_bh(WRITE, tbh); + submit_bh(REQ_OP_WRITE, 0, tbh); } /* Wait on i/o completion of buffers. */ for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) { @@ -785,7 +785,7 @@ int write_mft_record_nolock(ntfs_inode *ni, MFT_RECORD *m, int sync) clear_buffer_dirty(tbh); get_bh(tbh); tbh->b_end_io = end_buffer_write_sync; - submit_bh(WRITE, tbh); + submit_bh(REQ_OP_WRITE, 0, tbh); } /* Synchronize the mft mirror now if not @sync. */ if (!sync && ni->mft_no < vol->mftmirr_size) diff --git a/fs/ocfs2/buffer_head_io.c b/fs/ocfs2/buffer_head_io.c index fe50ded..fb775c9 100644 --- a/fs/ocfs2/buffer_head_io.c +++ b/fs/ocfs2/buffer_head_io.c @@ -79,7 +79,7 @@ int ocfs2_write_block(struct ocfs2_super *osb, struct buffer_head *bh, get_bh(bh); /* for end_buffer_write_sync() */ bh->b_end_io = end_buffer_write_sync; - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); wait_on_buffer(bh); @@ -149,7 +149,7 @@ int ocfs2_read_blocks_sync(struct ocfs2_super *osb, u64 block, clear_buffer_uptodate(bh); get_bh(bh); /* for end_buffer_read_sync() */ bh->b_end_io = end_buffer_read_sync; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); } for (i = nr; i > 0; i--) { @@ -305,7 +305,7 @@ int ocfs2_read_blocks(struct ocfs2_caching_info *ci, u64 block, int nr, if (validate) set_buffer_needs_validate(bh); bh->b_end_io = end_buffer_read_sync; - submit_bh(READ, bh); + submit_bh(REQ_OP_READ, 0, bh); continue; } } @@ -419,7 +419,7 @@ int ocfs2_write_super_or_backup(struct ocfs2_super *osb, get_bh(bh); /* for end_buffer_write_sync() */ bh->b_end_io = end_buffer_write_sync; ocfs2_compute_meta_ecc(osb->sb, bh->b_data, &di->i_check); - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); wait_on_buffer(bh); diff --git a/fs/reiserfs/inode.c b/fs/reiserfs/inode.c index ae9e5b3..91155ed 100644 --- a/fs/reiserfs/inode.c +++ b/fs/reiserfs/inode.c @@ -2668,7 +2668,7 @@ static int reiserfs_write_full_page(struct page *page, do { struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); nr++; } put_bh(bh); @@ -2728,7 +2728,7 @@ fail: struct buffer_head *next = bh->b_this_page; if (buffer_async_write(bh)) { clear_buffer_dirty(bh); - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); nr++; } put_bh(bh); diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c index 44c2bdc..688dce9 100644 --- a/fs/reiserfs/journal.c +++ b/fs/reiserfs/journal.c @@ -652,7 +652,7 @@ static void submit_logged_buffer(struct buffer_head *bh) BUG(); if (!buffer_uptodate(bh)) BUG(); - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); } static void submit_ordered_buffer(struct buffer_head *bh) @@ -662,7 +662,7 @@ static void submit_ordered_buffer(struct buffer_head *bh) clear_buffer_dirty(bh); if (!buffer_uptodate(bh)) BUG(); - submit_bh(WRITE, bh); + submit_bh(REQ_OP_WRITE, 0, bh); } #define CHUNK_SIZE 32 @@ -2269,7 +2269,7 @@ abort_replay: /* flush out the real blocks */ for (i = 0; i < get_desc_trans_len(desc); i++) { set_buffer_dirty(real_blocks[i]); - write_dirty_buffer(real_blocks[i], WRITE); + write_dirty_buffer(real_blocks[i], 0); } for (i = 0; i < get_desc_trans_len(desc); i++) { wait_on_buffer(real_blocks[i]); diff --git a/fs/ufs/util.c b/fs/ufs/util.c index b6c2f94..5f68b1a 100644 --- a/fs/ufs/util.c +++ b/fs/ufs/util.c @@ -118,7 +118,7 @@ void ubh_sync_block(struct ufs_buffer_head *ubh) unsigned i; for (i = 0; i < ubh->count; i++) - write_dirty_buffer(ubh->bh[i], WRITE); + write_dirty_buffer(ubh->bh[i], 0); for (i = 0; i < ubh->count; i++) wait_on_buffer(ubh->bh[i]); diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index 89d9aa9..f7cc163 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -189,10 +189,11 @@ void unlock_buffer(struct buffer_head *bh); void __lock_buffer(struct buffer_head *bh); void ll_rw_block(int, int, struct buffer_head * bh[]); int sync_dirty_buffer(struct buffer_head *bh); -int __sync_dirty_buffer(struct buffer_head *bh, int rw); -void write_dirty_buffer(struct buffer_head *bh, int rw); -int _submit_bh(int rw, struct buffer_head *bh, unsigned long bio_flags); -int submit_bh(int, struct buffer_head *); +int __sync_dirty_buffer(struct buffer_head *bh, int op_flags); +void write_dirty_buffer(struct buffer_head *bh, int op_flags); +int _submit_bh(int op, int op_flags, struct buffer_head *bh, + unsigned long bio_flags); +int submit_bh(int, int, struct buffer_head *); void write_boundary_block(struct block_device *bdev, sector_t bblock, unsigned blocksize); int bh_uptodate_or_lock(struct buffer_head *bh); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:21:56 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id A67F57F3F for ; Mon, 11 Jan 2016 14:21:56 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 27590AC002 for ; Mon, 11 Jan 2016 12:21:56 -0800 (PST) X-ASG-Debug-ID: 1452543713-04cbb05f7e978b0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id mu9Tao5SuUyt12yz (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:21:54 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 3BAAC42E5B5; Mon, 11 Jan 2016 20:21:53 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEY008385; Mon, 11 Jan 2016 15:21:50 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 05/35] fs: have ll_rw_block users pass in op and flags separately Date: Mon, 11 Jan 2016 14:21:03 -0600 X-ASG-Orig-Subj: [PATCH 05/35] fs: have ll_rw_block users pass in op and flags separately Message-Id: <1452543693-4440-6-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543713 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This has ll_rw_block users pass in the operation and flags separately, so we can setup the bio->bi_op and bio-bi_rw flags. v2: 1. Fix for kbuild error in ll_rw_block comments. Signed-off-by: Mike Christie --- fs/buffer.c | 19 ++++++++++--------- fs/ext4/inode.c | 6 +++--- fs/ext4/namei.c | 3 ++- fs/ext4/super.c | 2 +- fs/gfs2/bmap.c | 2 +- fs/gfs2/meta_io.c | 4 ++-- fs/gfs2/quota.c | 2 +- fs/isofs/compress.c | 2 +- fs/jbd2/journal.c | 2 +- fs/jbd2/recovery.c | 4 ++-- fs/ocfs2/aops.c | 2 +- fs/ocfs2/super.c | 2 +- fs/reiserfs/journal.c | 8 ++++---- fs/reiserfs/stree.c | 4 ++-- fs/reiserfs/super.c | 2 +- fs/squashfs/block.c | 4 ++-- fs/udf/dir.c | 2 +- fs/udf/directory.c | 2 +- fs/udf/inode.c | 2 +- fs/ufs/balloc.c | 2 +- include/linux/buffer_head.h | 2 +- 21 files changed, 40 insertions(+), 38 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 2343a40..5c6ab92 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -588,7 +588,7 @@ void write_boundary_block(struct block_device *bdev, struct buffer_head *bh = __find_get_block(bdev, bblock + 1, blocksize); if (bh) { if (buffer_dirty(bh)) - ll_rw_block(WRITE, 1, &bh); + ll_rw_block(REQ_OP_WRITE, 0, 1, &bh); put_bh(bh); } } @@ -1397,7 +1397,7 @@ void __breadahead(struct block_device *bdev, sector_t block, unsigned size) { struct buffer_head *bh = __getblk(bdev, block, size); if (likely(bh)) { - ll_rw_block(READA, 1, &bh); + ll_rw_block(REQ_OP_READ, READA, 1, &bh); brelse(bh); } } @@ -1957,7 +1957,7 @@ int __block_write_begin(struct page *page, loff_t pos, unsigned len, if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh) && (block_start < from || block_end > to)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); *wait_bh++=bh; } } @@ -2854,7 +2854,7 @@ int block_truncate_page(struct address_space *mapping, if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh)) { err = -EIO; - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); /* Uhhuh. Read error. Complain and punt. */ if (!buffer_uptodate(bh)) @@ -3054,7 +3054,8 @@ EXPORT_SYMBOL(submit_bh); /** * ll_rw_block: low-level access to block devices (DEPRECATED) - * @rw: whether to %READ or %WRITE or maybe %READA (readahead) + * @op: whether to %READ or %WRITE + * @op_flags: rq_flag_bits or %READA (readahead) * @nr: number of &struct buffer_heads in the array * @bhs: array of pointers to &struct buffer_head * @@ -3077,7 +3078,7 @@ EXPORT_SYMBOL(submit_bh); * All of the buffers must be for the same device, and must also be a * multiple of the current approved size for the device. */ -void ll_rw_block(int rw, int nr, struct buffer_head *bhs[]) +void ll_rw_block(int op, int op_flags, int nr, struct buffer_head *bhs[]) { int i; @@ -3086,18 +3087,18 @@ void ll_rw_block(int rw, int nr, struct buffer_head *bhs[]) if (!trylock_buffer(bh)) continue; - if (rw == WRITE) { + if (op == WRITE) { if (test_clear_buffer_dirty(bh)) { bh->b_end_io = end_buffer_write_sync; get_bh(bh); - submit_bh(rw, 0, bh); + submit_bh(op, op_flags, bh); continue; } } else { if (!buffer_uptodate(bh)) { bh->b_end_io = end_buffer_read_sync; get_bh(bh); - submit_bh(rw, 0, bh); + submit_bh(op, op_flags, bh); continue; } } diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 4fc178a..26a07cb 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -810,7 +810,7 @@ struct buffer_head *ext4_bread(handle_t *handle, struct inode *inode, return bh; if (!bh || buffer_uptodate(bh)) return bh; - ll_rw_block(READ | REQ_META | REQ_PRIO, 1, &bh); + ll_rw_block(REQ_OP_READ, REQ_META | REQ_PRIO, 1, &bh); wait_on_buffer(bh); if (buffer_uptodate(bh)) return bh; @@ -964,7 +964,7 @@ static int ext4_block_write_begin(struct page *page, loff_t pos, unsigned len, if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh) && (block_start < from || block_end > to)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); *wait_bh++ = bh; decrypt = ext4_encrypted_inode(inode) && S_ISREG(inode->i_mode); @@ -3511,7 +3511,7 @@ static int __ext4_block_zero_page_range(handle_t *handle, if (!buffer_uptodate(bh)) { err = -EIO; - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); /* Uhhuh. Read error. Complain and punt. */ if (!buffer_uptodate(bh)) diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c index 71d1021..1e35dc8 100644 --- a/fs/ext4/namei.c +++ b/fs/ext4/namei.c @@ -1440,7 +1440,8 @@ restart: } bh_use[ra_max] = bh; if (bh) - ll_rw_block(READ | REQ_META | REQ_PRIO, + ll_rw_block(REQ_OP_READ, + REQ_META | REQ_PRIO, 1, &bh); } } diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 50b2207..380aadc 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -4204,7 +4204,7 @@ static journal_t *ext4_get_dev_journal(struct super_block *sb, goto out_bdev; } journal->j_private = sb; - ll_rw_block(READ | REQ_META | REQ_PRIO, 1, &journal->j_sb_buffer); + ll_rw_block(REQ_OP_READ, REQ_META | REQ_PRIO, 1, &journal->j_sb_buffer); wait_on_buffer(journal->j_sb_buffer); if (!buffer_uptodate(journal->j_sb_buffer)) { ext4_msg(sb, KERN_ERR, "I/O error on journal device"); diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c index 29faf72..628d4a4 100644 --- a/fs/gfs2/bmap.c +++ b/fs/gfs2/bmap.c @@ -974,7 +974,7 @@ static int gfs2_block_truncate_page(struct address_space *mapping, loff_t from) if (!buffer_uptodate(bh)) { err = -EIO; - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); /* Uhhuh. Read error. Complain and punt. */ if (!buffer_uptodate(bh)) diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index dc1c9db..96c8140 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -448,7 +448,7 @@ struct buffer_head *gfs2_meta_ra(struct gfs2_glock *gl, u64 dblock, u32 extlen) if (buffer_uptodate(first_bh)) goto out; if (!buffer_locked(first_bh)) - ll_rw_block(READ_SYNC | REQ_META, 1, &first_bh); + ll_rw_block(REQ_OP_READ, READ_SYNC | REQ_META, 1, &first_bh); dblock++; extlen--; @@ -457,7 +457,7 @@ struct buffer_head *gfs2_meta_ra(struct gfs2_glock *gl, u64 dblock, u32 extlen) bh = gfs2_getbuf(gl, dblock, CREATE); if (!buffer_uptodate(bh) && !buffer_locked(bh)) - ll_rw_block(READA | REQ_META, 1, &bh); + ll_rw_block(REQ_OP_READ, READA | REQ_META, 1, &bh); brelse(bh); dblock++; extlen--; diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c index be6d9c4..13be922 100644 --- a/fs/gfs2/quota.c +++ b/fs/gfs2/quota.c @@ -730,7 +730,7 @@ static int gfs2_write_buf_to_page(struct gfs2_inode *ip, unsigned long index, if (PageUptodate(page)) set_buffer_uptodate(bh); if (!buffer_uptodate(bh)) { - ll_rw_block(READ | REQ_META, 1, &bh); + ll_rw_block(REQ_OP_READ, REQ_META, 1, &bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) goto unlock_out; diff --git a/fs/isofs/compress.c b/fs/isofs/compress.c index f311bf0..f9b5fc8 100644 --- a/fs/isofs/compress.c +++ b/fs/isofs/compress.c @@ -81,7 +81,7 @@ static loff_t zisofs_uncompress_block(struct inode *inode, loff_t block_start, blocknum = block_start >> bufshift; memset(bhs, 0, (needblocks + 1) * sizeof(struct buffer_head *)); haveblocks = isofs_get_blocks(inode, blocknum, bhs, needblocks); - ll_rw_block(READ, haveblocks, bhs); + ll_rw_block(REQ_OP_READ, 0, haveblocks, bhs); curbh = 0; curpage = 0; diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c index 3b7db0b..7104d76 100644 --- a/fs/jbd2/journal.c +++ b/fs/jbd2/journal.c @@ -1474,7 +1474,7 @@ static int journal_get_superblock(journal_t *journal) J_ASSERT(bh != NULL); if (!buffer_uptodate(bh)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) { printk(KERN_ERR diff --git a/fs/jbd2/recovery.c b/fs/jbd2/recovery.c index 7f277e4..8d66ea9 100644 --- a/fs/jbd2/recovery.c +++ b/fs/jbd2/recovery.c @@ -104,7 +104,7 @@ static int do_readahead(journal_t *journal, unsigned int start) if (!buffer_uptodate(bh) && !buffer_locked(bh)) { bufs[nbufs++] = bh; if (nbufs == MAXBUF) { - ll_rw_block(READ, nbufs, bufs); + ll_rw_block(REQ_OP_READ, 0, nbufs, bufs); journal_brelse_array(bufs, nbufs); nbufs = 0; } @@ -113,7 +113,7 @@ static int do_readahead(journal_t *journal, unsigned int start) } if (nbufs) - ll_rw_block(READ, nbufs, bufs); + ll_rw_block(REQ_OP_READ, 0, nbufs, bufs); err = 0; failed: diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c index 182f754..2e761fa 100644 --- a/fs/ocfs2/aops.c +++ b/fs/ocfs2/aops.c @@ -640,7 +640,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, !buffer_new(bh) && ocfs2_should_read_blk(inode, page, block_start) && (block_start < from || block_end > to)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); *wait_bh++=bh; } diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index 45b6139..9550593 100644 --- a/fs/ocfs2/super.c +++ b/fs/ocfs2/super.c @@ -1818,7 +1818,7 @@ static int ocfs2_get_sector(struct super_block *sb, if (!buffer_dirty(*bh)) clear_buffer_uptodate(*bh); unlock_buffer(*bh); - ll_rw_block(READ, 1, bh); + ll_rw_block(REQ_OP_READ, 0, 1, bh); wait_on_buffer(*bh); if (!buffer_uptodate(*bh)) { mlog_errno(-EIO); diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c index 688dce9..09682cc 100644 --- a/fs/reiserfs/journal.c +++ b/fs/reiserfs/journal.c @@ -870,7 +870,7 @@ loop_next: */ if (buffer_dirty(bh) && unlikely(bh->b_page->mapping == NULL)) { spin_unlock(lock); - ll_rw_block(WRITE, 1, &bh); + ll_rw_block(REQ_OP_WRITE, 0, 1, &bh); spin_lock(lock); } put_bh(bh); @@ -1057,7 +1057,7 @@ static int flush_commit_list(struct super_block *s, if (tbh) { if (buffer_dirty(tbh)) { depth = reiserfs_write_unlock_nested(s); - ll_rw_block(WRITE, 1, &tbh); + ll_rw_block(REQ_OP_WRITE, 0, 1, &tbh); reiserfs_write_lock_nested(s, depth); } put_bh(tbh) ; @@ -2244,7 +2244,7 @@ abort_replay: } } /* read in the log blocks, memcpy to the corresponding real block */ - ll_rw_block(READ, get_desc_trans_len(desc), log_blocks); + ll_rw_block(REQ_OP_READ, 0, get_desc_trans_len(desc), log_blocks); for (i = 0; i < get_desc_trans_len(desc); i++) { wait_on_buffer(log_blocks[i]); @@ -2346,7 +2346,7 @@ static struct buffer_head *reiserfs_breada(struct block_device *dev, } else bhlist[j++] = bh; } - ll_rw_block(READ, j, bhlist); + ll_rw_block(REQ_OP_READ, 0, j, bhlist); for (i = 1; i < j; i++) brelse(bhlist[i]); bh = bhlist[0]; diff --git a/fs/reiserfs/stree.c b/fs/reiserfs/stree.c index 24cbe01..5f9b6df 100644 --- a/fs/reiserfs/stree.c +++ b/fs/reiserfs/stree.c @@ -551,7 +551,7 @@ static int search_by_key_reada(struct super_block *s, if (!buffer_uptodate(bh[j])) { if (depth == -1) depth = reiserfs_write_unlock_nested(s); - ll_rw_block(READA, 1, bh + j); + ll_rw_block(REQ_OP_READ, READA, 1, bh + j); } brelse(bh[j]); } @@ -660,7 +660,7 @@ int search_by_key(struct super_block *sb, const struct cpu_key *key, if (!buffer_uptodate(bh) && depth == -1) depth = reiserfs_write_unlock_nested(sb); - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); if (depth != -1) diff --git a/fs/reiserfs/super.c b/fs/reiserfs/super.c index 05db747..bd04525 100644 --- a/fs/reiserfs/super.c +++ b/fs/reiserfs/super.c @@ -1660,7 +1660,7 @@ static int read_super_block(struct super_block *s, int offset) /* after journal replay, reread all bitmap and super blocks */ static int reread_meta_blocks(struct super_block *s) { - ll_rw_block(READ, 1, &SB_BUFFER_WITH_SB(s)); + ll_rw_block(REQ_OP_READ, 0, 1, &SB_BUFFER_WITH_SB(s)); wait_on_buffer(SB_BUFFER_WITH_SB(s)); if (!buffer_uptodate(SB_BUFFER_WITH_SB(s))) { reiserfs_warning(s, "reiserfs-2504", "error reading the super"); diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c index 0cea9b9..59375be 100644 --- a/fs/squashfs/block.c +++ b/fs/squashfs/block.c @@ -124,7 +124,7 @@ int squashfs_read_data(struct super_block *sb, u64 index, int length, goto block_release; bytes += msblk->devblksize; } - ll_rw_block(READ, b, bh); + ll_rw_block(REQ_OP_READ, 0, b, bh); } else { /* * Metadata block. @@ -156,7 +156,7 @@ int squashfs_read_data(struct super_block *sb, u64 index, int length, goto block_release; bytes += msblk->devblksize; } - ll_rw_block(READ, b - 1, bh + 1); + ll_rw_block(REQ_OP_READ, 0, b - 1, bh + 1); } for (i = 0; i < b; i++) { diff --git a/fs/udf/dir.c b/fs/udf/dir.c index 541d9c6..91b9f02 100644 --- a/fs/udf/dir.c +++ b/fs/udf/dir.c @@ -113,7 +113,7 @@ static int udf_readdir(struct file *file, struct dir_context *ctx) brelse(tmp); } if (num) { - ll_rw_block(READA, num, bha); + ll_rw_block(REQ_OP_READ, READA, num, bha); for (i = 0; i < num; i++) brelse(bha[i]); } diff --git a/fs/udf/directory.c b/fs/udf/directory.c index c763fda..71f3e0b 100644 --- a/fs/udf/directory.c +++ b/fs/udf/directory.c @@ -87,7 +87,7 @@ struct fileIdentDesc *udf_fileident_read(struct inode *dir, loff_t *nf_pos, brelse(tmp); } if (num) { - ll_rw_block(READA, num, bha); + ll_rw_block(REQ_OP_READ, READA, num, bha); for (i = 0; i < num; i++) brelse(bha[i]); } diff --git a/fs/udf/inode.c b/fs/udf/inode.c index 87dc16d..e49036b 100644 --- a/fs/udf/inode.c +++ b/fs/udf/inode.c @@ -1200,7 +1200,7 @@ struct buffer_head *udf_bread(struct inode *inode, int block, if (buffer_uptodate(bh)) return bh; - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); if (buffer_uptodate(bh)) diff --git a/fs/ufs/balloc.c b/fs/ufs/balloc.c index dc5fae6..26a3bb1 100644 --- a/fs/ufs/balloc.c +++ b/fs/ufs/balloc.c @@ -292,7 +292,7 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, if (!buffer_mapped(bh)) map_bh(bh, inode->i_sb, oldb + pos); if (!buffer_uptodate(bh)) { - ll_rw_block(READ, 1, &bh); + ll_rw_block(REQ_OP_READ, 0, 1, &bh); wait_on_buffer(bh); if (!buffer_uptodate(bh)) { ufs_error(inode->i_sb, __func__, diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h index f7cc163..15af19f 100644 --- a/include/linux/buffer_head.h +++ b/include/linux/buffer_head.h @@ -187,7 +187,7 @@ struct buffer_head *alloc_buffer_head(gfp_t gfp_flags); void free_buffer_head(struct buffer_head * bh); void unlock_buffer(struct buffer_head *bh); void __lock_buffer(struct buffer_head *bh); -void ll_rw_block(int, int, struct buffer_head * bh[]); +void ll_rw_block(int, int, int, struct buffer_head * bh[]); int sync_dirty_buffer(struct buffer_head *bh); int __sync_dirty_buffer(struct buffer_head *bh, int op_flags); void write_dirty_buffer(struct buffer_head *bh, int op_flags); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:21:58 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 31EA729E0E for ; Mon, 11 Jan 2016 14:21:58 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id A55A4AC005 for ; Mon, 11 Jan 2016 12:21:57 -0800 (PST) X-ASG-Debug-ID: 1452543715-04cb6c75db79020001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id z7etgpdnkepoilwz (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:21:56 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 6D628744; Mon, 11 Jan 2016 20:21:55 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEZ008385; Mon, 11 Jan 2016 15:21:53 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 06/35] direct-io: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:04 -0600 X-ASG-Orig-Subj: [PATCH 06/35] direct-io: set bi_op to REQ_OP Message-Id: <1452543693-4440-7-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543716 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the dio code set the bio bi_op to a REQ_OP. It also begins to convert btrfs's dio_submit_t related code, because of the submit_io callout use. In the btrfs_submit_direct change, I OR'd the op and flag back together. It is only temporary. The next patch will completely convert all the btrfs code paths. Signed-off-by: Mike Christie --- fs/btrfs/inode.c | 9 +++++---- fs/direct-io.c | 35 +++++++++++++++++++++-------------- include/linux/fs.h | 2 +- 3 files changed, 27 insertions(+), 19 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 00f27eb..06f88bf 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8300,14 +8300,14 @@ out_err: return 0; } -static void btrfs_submit_direct(int rw, struct bio *dio_bio, - struct inode *inode, loff_t file_offset) +static void btrfs_submit_direct(struct bio *dio_bio, struct inode *inode, + loff_t file_offset) { struct btrfs_dio_private *dip = NULL; struct bio *io_bio = NULL; struct btrfs_io_bio *btrfs_bio; int skip_sum; - int write = rw & REQ_WRITE; + bool write = (dio_bio->bi_op == REQ_OP_WRITE); int ret = 0; skip_sum = BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM; @@ -8358,7 +8358,8 @@ static void btrfs_submit_direct(int rw, struct bio *dio_bio, dio_data->unsubmitted_oe_range_end; } - ret = btrfs_submit_direct_hook(rw, dip, skip_sum); + ret = btrfs_submit_direct_hook(dio_bio->bi_op | dio_bio->bi_rw, dip, + skip_sum); if (!ret) return; diff --git a/fs/direct-io.c b/fs/direct-io.c index 4400836..52134ba 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -108,7 +108,8 @@ struct dio_submit { /* dio_state communicated between submission path and end_io */ struct dio { int flags; /* doesn't change */ - int rw; + int op; + int op_flags; blk_qc_t bio_cookie; struct block_device *bio_bdev; struct inode *inode; @@ -163,7 +164,7 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) ret = iov_iter_get_pages(sdio->iter, dio->pages, LONG_MAX, DIO_PAGES, &sdio->from); - if (ret < 0 && sdio->blocks_available && (dio->rw & WRITE)) { + if (ret < 0 && sdio->blocks_available && (dio->op == REQ_OP_WRITE)) { struct page *page = ZERO_PAGE(0); /* * A memory fault, but the filesystem has some outstanding @@ -242,7 +243,8 @@ static ssize_t dio_complete(struct dio *dio, loff_t offset, ssize_t ret, transferred = dio->result; /* Check for short read case */ - if ((dio->rw == READ) && ((offset + transferred) > dio->i_size)) + if ((dio->op == REQ_OP_READ) && + ((offset + transferred) > dio->i_size)) transferred = dio->i_size - offset; } @@ -260,7 +262,7 @@ static ssize_t dio_complete(struct dio *dio, loff_t offset, ssize_t ret, inode_dio_end(dio->inode); if (is_async) { - if (dio->rw & WRITE) { + if (dio->op == REQ_OP_WRITE) { int err; err = generic_write_sync(dio->iocb->ki_filp, offset, @@ -369,7 +371,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_bdev = bdev; bio->bi_iter.bi_sector = first_sector; - bio->bi_rw = dio->rw; + bio->bi_op = dio->op; + bio->bi_rw = dio->op_flags; if (dio->is_async) bio->bi_end_io = dio_bio_end_aio; else @@ -397,14 +400,13 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) dio->refcount++; spin_unlock_irqrestore(&dio->bio_lock, flags); - if (dio->is_async && dio->rw == READ && dio->should_dirty) + if (dio->is_async && dio->op == REQ_OP_READ && dio->should_dirty) bio_set_pages_dirty(bio); dio->bio_bdev = bio->bi_bdev; if (sdio->submit_io) { - sdio->submit_io(dio->rw, bio, dio->inode, - sdio->logical_offset_in_bio); + sdio->submit_io(bio, dio->inode, sdio->logical_offset_in_bio); dio->bio_cookie = BLK_QC_T_NONE; } else dio->bio_cookie = submit_bio(bio); @@ -472,14 +474,14 @@ static int dio_bio_complete(struct dio *dio, struct bio *bio) if (bio->bi_error) dio->io_error = -EIO; - if (dio->is_async && dio->rw == READ && dio->should_dirty) { + if (dio->is_async && dio->op == REQ_OP_READ && dio->should_dirty) { bio_check_pages_dirty(bio); /* transfers ownership */ err = bio->bi_error; } else { bio_for_each_segment_all(bvec, bio, i) { struct page *page = bvec->bv_page; - if (dio->rw == READ && !PageCompound(page) && + if (dio->op == REQ_OP_READ && !PageCompound(page) && dio->should_dirty) set_page_dirty_lock(page); page_cache_release(page); @@ -632,7 +634,7 @@ static int get_more_blocks(struct dio *dio, struct dio_submit *sdio, * which may decide to handle it or also return an unmapped * buffer head. */ - create = dio->rw & WRITE; + create = dio->op == REQ_OP_WRITE; if (dio->flags & DIO_SKIP_HOLES) { if (sdio->block_in_file < (i_size_read(dio->inode) >> sdio->blkbits)) @@ -782,7 +784,7 @@ submit_page_section(struct dio *dio, struct dio_submit *sdio, struct page *page, { int ret = 0; - if (dio->rw & WRITE) { + if (dio->op == REQ_OP_WRITE) { /* * Read accounting is performed in submit_bio() */ @@ -982,7 +984,7 @@ do_holes: loff_t i_size_aligned; /* AKPM: eargh, -ENOTBLK is a hack */ - if (dio->rw & WRITE) { + if (dio->op == REQ_OP_WRITE) { page_cache_release(page); return -ENOTBLK; } @@ -1195,7 +1197,12 @@ do_blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, dio->is_async = true; dio->inode = inode; - dio->rw = iov_iter_rw(iter) == WRITE ? WRITE_ODIRECT : READ; + if (iov_iter_rw(iter) == WRITE) { + dio->op = REQ_OP_WRITE; + dio->op_flags = WRITE_ODIRECT; + } else { + dio->op = REQ_OP_READ; + } /* * For AIO O_(D)SYNC writes we need to defer completions to a workqueue diff --git a/include/linux/fs.h b/include/linux/fs.h index afca758..204ea4c 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2732,7 +2732,7 @@ extern int generic_file_open(struct inode * inode, struct file * filp); extern int nonseekable_open(struct inode * inode, struct file * filp); #ifdef CONFIG_BLOCK -typedef void (dio_submit_t)(int rw, struct bio *bio, struct inode *inode, +typedef void (dio_submit_t)(struct bio *bio, struct inode *inode, loff_t file_offset); enum { -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:01 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E42D629E0E for ; Mon, 11 Jan 2016 14:22:00 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 5838EAC002 for ; Mon, 11 Jan 2016 12:22:00 -0800 (PST) X-ASG-Debug-ID: 1452543718-04cb6c75de79030001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id zE14L94fe1hLXFpZ (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:21:58 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id CD87A8E010; Mon, 11 Jan 2016 20:21:57 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEa008385; Mon, 11 Jan 2016 15:21:55 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 07/35] btrfs: have submit_one_bio users setup bio bi_op Date: Mon, 11 Jan 2016 14:21:05 -0600 X-ASG-Orig-Subj: [PATCH 07/35] btrfs: have submit_one_bio users setup bio bi_op Message-Id: <1452543693-4440-8-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543718 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has btrfs's submit_one_bio callers set the bio->bi_op to a REQ_OP and the bi_rw to rq_flag_bits. The next patches will continue to convert btrfs, so submit_bio_hook and merge_bio_hook related code will be modified to take only the bio. I did not do it in this patch to try and keep it smaller. Note: I have run xfs tests on these btrfs patches. There were some failures with and without the patches. I have not had time to track down why xfstest fails without the patches. Signed-off-by: Mike Christie --- fs/btrfs/extent_io.c | 92 +++++++++++++++++++++++++++------------------------- 1 file changed, 47 insertions(+), 45 deletions(-) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index 8e7bf941..a4830f0 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2382,7 +2382,7 @@ static int bio_readpage_error(struct bio *failed_bio, u64 phy_offset, int read_mode; int ret; - BUG_ON(failed_bio->bi_rw & REQ_WRITE); + BUG_ON(failed_bio->bi_op == REQ_OP_WRITE); ret = btrfs_get_io_failure_record(inode, start, end, &failrec); if (ret) @@ -2408,6 +2408,8 @@ static int bio_readpage_error(struct bio *failed_bio, u64 phy_offset, free_io_failure(inode, failrec); return -EIO; } + bio->bi_op = REQ_OP_READ; + bio->bi_rw = read_mode; pr_debug("Repair Read Error: submitting new read[%#x] to this_mirror=%d, in_validation=%d\n", read_mode, failrec->this_mirror, failrec->in_validation); @@ -2719,8 +2721,8 @@ struct bio *btrfs_io_bio_alloc(gfp_t gfp_mask, unsigned int nr_iovecs) } -static int __must_check submit_one_bio(int rw, struct bio *bio, - int mirror_num, unsigned long bio_flags) +static int __must_check submit_one_bio(struct bio *bio, int mirror_num, + unsigned long bio_flags) { int ret = 0; struct bio_vec *bvec = bio->bi_io_vec + bio->bi_vcnt - 1; @@ -2731,12 +2733,12 @@ static int __must_check submit_one_bio(int rw, struct bio *bio, start = page_offset(page) + bvec->bv_offset; bio->bi_private = NULL; - bio->bi_rw = rw; bio_get(bio); if (tree->ops && tree->ops->submit_bio_hook) - ret = tree->ops->submit_bio_hook(page->mapping->host, rw, bio, - mirror_num, bio_flags, start); + ret = tree->ops->submit_bio_hook(page->mapping->host, + bio->bi_rw, bio, mirror_num, + bio_flags, start); else btrfsic_submit_bio(bio); @@ -2744,20 +2746,20 @@ static int __must_check submit_one_bio(int rw, struct bio *bio, return ret; } -static int merge_bio(int rw, struct extent_io_tree *tree, struct page *page, +static int merge_bio(struct extent_io_tree *tree, struct page *page, unsigned long offset, size_t size, struct bio *bio, unsigned long bio_flags) { int ret = 0; if (tree->ops && tree->ops->merge_bio_hook) - ret = tree->ops->merge_bio_hook(rw, page, offset, size, bio, - bio_flags); + ret = tree->ops->merge_bio_hook(bio->bi_op, page, offset, size, + bio, bio_flags); BUG_ON(ret < 0); return ret; } -static int submit_extent_page(int rw, struct extent_io_tree *tree, +static int submit_extent_page(int op, int op_flags, struct extent_io_tree *tree, struct writeback_control *wbc, struct page *page, sector_t sector, size_t size, unsigned long offset, @@ -2785,10 +2787,9 @@ static int submit_extent_page(int rw, struct extent_io_tree *tree, if (prev_bio_flags != bio_flags || !contig || force_bio_submit || - merge_bio(rw, tree, page, offset, page_size, bio, bio_flags) || + merge_bio(tree, page, offset, page_size, bio, bio_flags) || bio_add_page(bio, page, page_size, offset) < page_size) { - ret = submit_one_bio(rw, bio, mirror_num, - prev_bio_flags); + ret = submit_one_bio(bio, mirror_num, prev_bio_flags); if (ret < 0) { *bio_ret = NULL; return ret; @@ -2809,6 +2810,8 @@ static int submit_extent_page(int rw, struct extent_io_tree *tree, bio_add_page(bio, page, page_size, offset); bio->bi_end_io = end_io_func; bio->bi_private = tree; + bio->bi_op = op; + bio->bi_rw = op_flags; if (wbc) { wbc_init_bio(wbc, bio); wbc_account_io(wbc, page, page_size); @@ -2817,7 +2820,7 @@ static int submit_extent_page(int rw, struct extent_io_tree *tree, if (bio_ret) *bio_ret = bio; else - ret = submit_one_bio(rw, bio, mirror_num, bio_flags); + ret = submit_one_bio(bio, mirror_num, bio_flags); return ret; } @@ -2881,7 +2884,7 @@ static int __do_readpage(struct extent_io_tree *tree, get_extent_t *get_extent, struct extent_map **em_cached, struct bio **bio, int mirror_num, - unsigned long *bio_flags, int rw, + unsigned long *bio_flags, int read_flags, u64 *prev_em_start) { struct inode *inode = page->mapping->host; @@ -3072,8 +3075,8 @@ static int __do_readpage(struct extent_io_tree *tree, } pnr -= page->index; - ret = submit_extent_page(rw, tree, NULL, page, - sector, disk_io_size, pg_offset, + ret = submit_extent_page(REQ_OP_READ, read_flags, tree, NULL, + page, sector, disk_io_size, pg_offset, bdev, bio, pnr, end_bio_extent_readpage, mirror_num, *bio_flags, @@ -3105,7 +3108,7 @@ static inline void __do_contiguous_readpages(struct extent_io_tree *tree, get_extent_t *get_extent, struct extent_map **em_cached, struct bio **bio, int mirror_num, - unsigned long *bio_flags, int rw, + unsigned long *bio_flags, u64 *prev_em_start) { struct inode *inode; @@ -3126,7 +3129,7 @@ static inline void __do_contiguous_readpages(struct extent_io_tree *tree, for (index = 0; index < nr_pages; index++) { __do_readpage(tree, pages[index], get_extent, em_cached, bio, - mirror_num, bio_flags, rw, prev_em_start); + mirror_num, bio_flags, 0, prev_em_start); page_cache_release(pages[index]); } } @@ -3136,7 +3139,7 @@ static void __extent_readpages(struct extent_io_tree *tree, int nr_pages, get_extent_t *get_extent, struct extent_map **em_cached, struct bio **bio, int mirror_num, - unsigned long *bio_flags, int rw, + unsigned long *bio_flags, u64 *prev_em_start) { u64 start = 0; @@ -3158,7 +3161,7 @@ static void __extent_readpages(struct extent_io_tree *tree, index - first_index, start, end, get_extent, em_cached, bio, mirror_num, bio_flags, - rw, prev_em_start); + prev_em_start); start = page_start; end = start + PAGE_CACHE_SIZE - 1; first_index = index; @@ -3169,7 +3172,7 @@ static void __extent_readpages(struct extent_io_tree *tree, __do_contiguous_readpages(tree, &pages[first_index], index - first_index, start, end, get_extent, em_cached, bio, - mirror_num, bio_flags, rw, + mirror_num, bio_flags, prev_em_start); } @@ -3177,7 +3180,7 @@ static int __extent_read_full_page(struct extent_io_tree *tree, struct page *page, get_extent_t *get_extent, struct bio **bio, int mirror_num, - unsigned long *bio_flags, int rw) + unsigned long *bio_flags, int read_flags) { struct inode *inode = page->mapping->host; struct btrfs_ordered_extent *ordered; @@ -3196,7 +3199,7 @@ static int __extent_read_full_page(struct extent_io_tree *tree, } ret = __do_readpage(tree, page, get_extent, NULL, bio, mirror_num, - bio_flags, rw, NULL); + bio_flags, read_flags, NULL); return ret; } @@ -3208,9 +3211,9 @@ int extent_read_full_page(struct extent_io_tree *tree, struct page *page, int ret; ret = __extent_read_full_page(tree, page, get_extent, &bio, mirror_num, - &bio_flags, READ); + &bio_flags, 0); if (bio) - ret = submit_one_bio(READ, bio, mirror_num, bio_flags); + ret = submit_one_bio(bio, mirror_num, bio_flags); return ret; } @@ -3222,9 +3225,9 @@ int extent_read_full_page_nolock(struct extent_io_tree *tree, struct page *page, int ret; ret = __do_readpage(tree, page, get_extent, NULL, &bio, mirror_num, - &bio_flags, READ, NULL); + &bio_flags, 0, NULL); if (bio) - ret = submit_one_bio(READ, bio, mirror_num, bio_flags); + ret = submit_one_bio(bio, mirror_num, bio_flags); return ret; } @@ -3471,7 +3474,8 @@ static noinline_for_stack int __extent_writepage_io(struct inode *inode, page->index, cur, end); } - ret = submit_extent_page(write_flags, tree, wbc, page, + ret = submit_extent_page(REQ_OP_WRITE, write_flags, + tree, wbc, page, sector, iosize, pg_offset, bdev, &epd->bio, max_nr, end_bio_extent_writepage, @@ -3511,13 +3515,11 @@ static int __extent_writepage(struct page *page, struct writeback_control *wbc, size_t pg_offset = 0; loff_t i_size = i_size_read(inode); unsigned long end_index = i_size >> PAGE_CACHE_SHIFT; - int write_flags; + int write_flags = 0; unsigned long nr_written = 0; if (wbc->sync_mode == WB_SYNC_ALL) write_flags = WRITE_SYNC; - else - write_flags = WRITE; trace___extent_writepage(page, inode, wbc); @@ -3761,7 +3763,7 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb, u64 offset = eb->start; unsigned long i, num_pages; unsigned long bio_flags = 0; - int rw = (epd->sync_io ? WRITE_SYNC : WRITE) | REQ_META; + int write_flags = (epd->sync_io ? WRITE_SYNC : 0) | REQ_META; int ret = 0; clear_bit(EXTENT_BUFFER_WRITE_ERR, &eb->bflags); @@ -3775,9 +3777,10 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb, clear_page_dirty_for_io(p); set_page_writeback(p); - ret = submit_extent_page(rw, tree, wbc, p, offset >> 9, - PAGE_CACHE_SIZE, 0, bdev, &epd->bio, - -1, end_bio_extent_buffer_writepage, + ret = submit_extent_page(REQ_OP_WRITE, write_flags, tree, wbc, + p, offset >> 9, PAGE_CACHE_SIZE, 0, + bdev, &epd->bio, -1, + end_bio_extent_buffer_writepage, 0, epd->bio_flags, bio_flags, false); epd->bio_flags = bio_flags; if (ret) { @@ -4066,13 +4069,13 @@ retry: static void flush_epd_write_bio(struct extent_page_data *epd) { if (epd->bio) { - int rw = WRITE; int ret; + epd->bio->bi_op = REQ_OP_WRITE; if (epd->sync_io) - rw = WRITE_SYNC; + epd->bio->bi_rw = WRITE_SYNC; - ret = submit_one_bio(rw, epd->bio, 0, epd->bio_flags); + ret = submit_one_bio(epd->bio, 0, epd->bio_flags); BUG_ON(ret < 0); /* -ENOMEM */ epd->bio = NULL; } @@ -4199,19 +4202,19 @@ int extent_readpages(struct extent_io_tree *tree, if (nr < ARRAY_SIZE(pagepool)) continue; __extent_readpages(tree, pagepool, nr, get_extent, &em_cached, - &bio, 0, &bio_flags, READ, &prev_em_start); + &bio, 0, &bio_flags, &prev_em_start); nr = 0; } if (nr) __extent_readpages(tree, pagepool, nr, get_extent, &em_cached, - &bio, 0, &bio_flags, READ, &prev_em_start); + &bio, 0, &bio_flags, &prev_em_start); if (em_cached) free_extent_map(em_cached); BUG_ON(!list_empty(pages)); if (bio) - return submit_one_bio(READ, bio, 0, bio_flags); + return submit_one_bio(bio, 0, bio_flags); return 0; } @@ -5233,7 +5236,7 @@ int read_extent_buffer_pages(struct extent_io_tree *tree, err = __extent_read_full_page(tree, page, get_extent, &bio, mirror_num, &bio_flags, - READ | REQ_META); + REQ_META); if (err) ret = err; } else { @@ -5242,8 +5245,7 @@ int read_extent_buffer_pages(struct extent_io_tree *tree, } if (bio) { - err = submit_one_bio(READ | REQ_META, bio, mirror_num, - bio_flags); + err = submit_one_bio(bio, mirror_num, bio_flags); if (err) return err; } -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:03 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 5A90329E1E for ; Mon, 11 Jan 2016 14:22:03 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id CEE51AC002 for ; Mon, 11 Jan 2016 12:22:02 -0800 (PST) X-ASG-Debug-ID: 1452543720-04cb6c75dc79030001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id pqoXD3aFQnagWem6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:00 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 1A4058E74E; Mon, 11 Jan 2016 20:22:00 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEb008385; Mon, 11 Jan 2016 15:21:57 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 08/35] btrfs: set bi_op tp REQ_OP Date: Mon, 11 Jan 2016 14:21:06 -0600 X-ASG-Orig-Subj: [PATCH 08/35] btrfs: set bi_op tp REQ_OP Message-Id: <1452543693-4440-9-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543720 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has btrfs set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. Signed-off-by: Mike Christie --- fs/btrfs/check-integrity.c | 19 +++++++++---------- fs/btrfs/compression.c | 4 ++++ fs/btrfs/disk-io.c | 7 ++++--- fs/btrfs/inode.c | 20 +++++++++++++------- fs/btrfs/raid56.c | 10 +++++----- fs/btrfs/scrub.c | 9 +++++---- fs/btrfs/volumes.c | 18 +++++++++--------- 7 files changed, 49 insertions(+), 38 deletions(-) diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index 036598f..e409d1f 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -1683,7 +1683,7 @@ static int btrfsic_read_block(struct btrfsic_state *state, } bio->bi_bdev = block_ctx->dev->bdev; bio->bi_iter.bi_sector = dev_bytenr >> 9; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; for (j = i; j < num_pages; j++) { ret = bio_add_page(bio, block_ctx->pagev[j], @@ -2964,7 +2964,6 @@ int btrfsic_submit_bh(int op, int op_flags, struct buffer_head *bh) static void __btrfsic_submit_bio(struct bio *bio) { struct btrfsic_dev_state *dev_state; - int rw = bio->bi_rw; if (!btrfsic_is_initialized) return; @@ -2974,7 +2973,7 @@ static void __btrfsic_submit_bio(struct bio *bio) * btrfsic_mount(), this might return NULL */ dev_state = btrfsic_dev_state_lookup(bio->bi_bdev); if (NULL != dev_state && - (rw & WRITE) && NULL != bio->bi_io_vec) { + (bio->bi_op == REQ_OP_WRITE) && NULL != bio->bi_io_vec) { unsigned int i; u64 dev_bytenr; u64 cur_bytenr; @@ -2986,9 +2985,9 @@ static void __btrfsic_submit_bio(struct bio *bio) if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bio(rw=0x%x, bi_vcnt=%u," + "submit_bio(rw=%d,0x%lx, bi_vcnt=%u," " bi_sector=%llu (bytenr %llu), bi_bdev=%p)\n", - rw, bio->bi_vcnt, + bio->bi_op, bio->bi_rw, bio->bi_vcnt, (unsigned long long)bio->bi_iter.bi_sector, dev_bytenr, bio->bi_bdev); @@ -3019,18 +3018,18 @@ static void __btrfsic_submit_bio(struct bio *bio) btrfsic_process_written_block(dev_state, dev_bytenr, mapped_datav, bio->bi_vcnt, bio, &bio_is_patched, - NULL, rw); + NULL, bio->bi_rw); while (i > 0) { i--; kunmap(bio->bi_io_vec[i].bv_page); } kfree(mapped_datav); - } else if (NULL != dev_state && (rw & REQ_FLUSH)) { + } else if (NULL != dev_state && (bio->bi_rw & REQ_FLUSH)) { if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bio(rw=0x%x FLUSH, bdev=%p)\n", - rw, bio->bi_bdev); + "submit_bio(rw=%d,0x%lx FLUSH, bdev=%p)\n", + bio->bi_op, bio->bi_rw, bio->bi_bdev); if (!dev_state->dummy_block_for_bio_bh_flush.is_iodone) { if ((dev_state->state->print_mask & (BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH | @@ -3048,7 +3047,7 @@ static void __btrfsic_submit_bio(struct bio *bio) block->never_written = 0; block->iodone_w_error = 0; block->flush_gen = dev_state->last_flush_gen + 1; - block->submit_bio_bh_rw = rw; + block->submit_bio_bh_rw = bio->bi_rw; block->orig_bio_bh_private = bio->bi_private; block->orig_bio_bh_end_io.bio = bio->bi_end_io; block->next_in_same_bio = NULL; diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index c473c42..25bf179 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -363,6 +363,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, kfree(cb); return -ENOMEM; } + bio->bi_op = REQ_OP_WRITE; bio->bi_private = cb; bio->bi_end_io = end_compressed_bio_write; atomic_inc(&cb->pending_bios); @@ -408,6 +409,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, bio = compressed_bio_alloc(bdev, first_byte, GFP_NOFS); BUG_ON(!bio); + bio->bi_op = REQ_OP_WRITE; bio->bi_private = cb; bio->bi_end_io = end_compressed_bio_write; bio_add_page(bio, page, PAGE_CACHE_SIZE, 0); @@ -650,6 +652,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, comp_bio = compressed_bio_alloc(bdev, cur_disk_byte, GFP_NOFS); if (!comp_bio) goto fail2; + comp_bio->bi_op = REQ_OP_READ; comp_bio->bi_private = cb; comp_bio->bi_end_io = end_compressed_bio_read; atomic_inc(&cb->pending_bios); @@ -703,6 +706,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, comp_bio = compressed_bio_alloc(bdev, cur_disk_byte, GFP_NOFS); BUG_ON(!comp_bio); + comp_bio->bi_op = REQ_OP_READ; comp_bio->bi_private = cb; comp_bio->bi_end_io = end_compressed_bio_read; diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 2eb3f10..28cf697 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -713,7 +713,7 @@ static void end_workqueue_bio(struct bio *bio) fs_info = end_io_wq->info; end_io_wq->error = bio->bi_error; - if (bio->bi_rw & REQ_WRITE) { + if (bio->bi_op == REQ_OP_WRITE) { if (end_io_wq->metadata == BTRFS_WQ_ENDIO_METADATA) { wq = fs_info->endio_meta_write_workers; func = btrfs_endio_meta_write_helper; @@ -859,7 +859,7 @@ int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode, atomic_inc(&fs_info->nr_async_submits); - if (rw & REQ_SYNC) + if (bio->bi_rw & REQ_SYNC) btrfs_set_work_high_priority(&async->work); btrfs_queue_work(fs_info->workers, &async->work); @@ -937,7 +937,7 @@ static int btree_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, int async = check_async_write(inode, bio_flags); int ret; - if (!(rw & REQ_WRITE)) { + if (bio->bi_op != REQ_OP_WRITE) { /* * called for a read, do the setup so that checksum validation * can happen in the async kernel threads @@ -3454,6 +3454,7 @@ static int write_dev_flush(struct btrfs_device *device, int wait) bio->bi_end_io = btrfs_end_empty_barrier; bio->bi_bdev = device->bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw = WRITE_FLUSH; init_completion(&device->flush_wait); bio->bi_private = &device->flush_wait; diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 06f88bf..dd435ee 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -1895,7 +1895,7 @@ static int btrfs_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, if (btrfs_is_free_space_inode(inode)) metadata = BTRFS_WQ_ENDIO_FREE_SPACE; - if (!(rw & REQ_WRITE)) { + if (bio->bi_op != REQ_OP_WRITE) { ret = btrfs_bio_wq_end_io(root->fs_info, bio, metadata); if (ret) goto out; @@ -7713,7 +7713,7 @@ static inline int submit_dio_repair_bio(struct inode *inode, struct bio *bio, struct btrfs_root *root = BTRFS_I(inode)->root; int ret; - BUG_ON(rw & REQ_WRITE); + BUG_ON(bio->bi_op == REQ_OP_WRITE); bio_get(bio); @@ -7773,7 +7773,7 @@ static int dio_read_error(struct inode *inode, struct bio *failed_bio, int read_mode; int ret; - BUG_ON(failed_bio->bi_rw & REQ_WRITE); + BUG_ON(failed_bio->bi_op == REQ_OP_WRITE); ret = btrfs_get_io_failure_record(inode, start, end, &failrec); if (ret) @@ -7799,6 +7799,8 @@ static int dio_read_error(struct inode *inode, struct bio *failed_bio, free_io_failure(inode, failrec); return -EIO; } + bio->bi_op = REQ_OP_READ; + bio->bi_rw = read_mode; btrfs_debug(BTRFS_I(inode)->root->fs_info, "Repair DIO Read Error: submitting new dio read[%#x] to this_mirror=%d, in_validation=%d\n", @@ -8061,8 +8063,8 @@ static void btrfs_end_dio_bio(struct bio *bio) if (err) btrfs_warn(BTRFS_I(dip->inode)->root->fs_info, - "direct IO failed ino %llu rw %lu sector %#Lx len %u err no %d", - btrfs_ino(dip->inode), bio->bi_rw, + "direct IO failed ino %llu rw %d,%lu sector %#Lx len %u err no %d", + btrfs_ino(dip->inode), bio->bi_op, bio->bi_rw, (unsigned long long)bio->bi_iter.bi_sector, bio->bi_iter.bi_size, err); @@ -8140,7 +8142,7 @@ static inline int __btrfs_submit_dio_bio(struct bio *bio, struct inode *inode, int async_submit) { struct btrfs_dio_private *dip = bio->bi_private; - int write = rw & REQ_WRITE; + bool write = bio->bi_op == REQ_OP_WRITE; struct btrfs_root *root = BTRFS_I(inode)->root; int ret; @@ -8225,6 +8227,8 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, if (!bio) return -ENOMEM; + bio->bi_op = orig_bio->bi_op; + bio->bi_rw = orig_bio->bi_rw; bio->bi_private = dip; bio->bi_end_io = btrfs_end_dio_bio; btrfs_io_bio(bio)->logical = file_offset; @@ -8260,12 +8264,14 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, start_sector, GFP_NOFS); if (!bio) goto out_err; + bio->bi_op = orig_bio->bi_op; + bio->bi_rw = orig_bio->bi_rw; bio->bi_private = dip; bio->bi_end_io = btrfs_end_dio_bio; btrfs_io_bio(bio)->logical = file_offset; map_length = orig_bio->bi_iter.bi_size; - ret = btrfs_map_block(root->fs_info, rw, + ret = btrfs_map_block(root->fs_info, orig_bio->bi_op, start_sector << 9, &map_length, NULL, 0); if (ret) { diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c index 3290fa1..54e4941 100644 --- a/fs/btrfs/raid56.c +++ b/fs/btrfs/raid56.c @@ -1323,7 +1323,7 @@ write_data: bio->bi_private = rbio; bio->bi_end_io = raid_write_end_io; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; submit_bio(bio); } @@ -1579,7 +1579,7 @@ static int raid56_rmw_stripe(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid_rmw_end_io; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); @@ -2106,7 +2106,7 @@ static int __raid56_parity_recover(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid_recover_end_io; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); @@ -2469,7 +2469,7 @@ submit_write: bio->bi_private = rbio; bio->bi_end_io = raid_write_parity_end_io; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; submit_bio(bio); } return; @@ -2647,7 +2647,7 @@ static void raid56_parity_scrub_stripe(struct btrfs_raid_bio *rbio) bio->bi_private = rbio; bio->bi_end_io = raid56_parity_scrub_end_io; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; btrfs_bio_wq_end_io(rbio->fs_info, bio, BTRFS_WQ_ENDIO_RAID56); diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c index 8855cfe..abba2e1 100644 --- a/fs/btrfs/scrub.c +++ b/fs/btrfs/scrub.c @@ -1504,7 +1504,7 @@ static void scrub_recheck_block(struct btrfs_fs_info *fs_info, sblock->no_io_error_seen = 0; } else { bio->bi_iter.bi_sector = page->physical >> 9; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; if (btrfsic_submit_bio_wait(bio)) sblock->no_io_error_seen = 0; @@ -1586,7 +1586,7 @@ static int scrub_repair_page_from_good_copy(struct scrub_block *sblock_bad, return -EIO; bio->bi_bdev = page_bad->dev->bdev; bio->bi_iter.bi_sector = page_bad->physical >> 9; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; ret = bio_add_page(bio, page_good->page, PAGE_SIZE, 0); if (PAGE_SIZE != ret) { @@ -1687,7 +1687,7 @@ again: bio->bi_end_io = scrub_wr_bio_end_io; bio->bi_bdev = sbio->dev->bdev; bio->bi_iter.bi_sector = sbio->physical >> 9; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; sbio->err = 0; } else if (sbio->physical + sbio->page_count * PAGE_SIZE != spage->physical_for_dev_replace || @@ -2091,7 +2091,7 @@ again: bio->bi_end_io = scrub_bio_end_io; bio->bi_bdev = sbio->dev->bdev; bio->bi_iter.bi_sector = sbio->physical >> 9; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; sbio->err = 0; } else if (sbio->physical + sbio->page_count * PAGE_SIZE != spage->physical || @@ -4394,6 +4394,7 @@ static int write_page_nocow(struct scrub_ctx *sctx, bio->bi_iter.bi_size = 0; bio->bi_iter.bi_sector = physical_for_dev_replace >> 9; bio->bi_bdev = dev->bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw = WRITE_SYNC; ret = bio_add_page(bio, page, PAGE_CACHE_SIZE, 0); diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 2463761..0da1d32 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -5888,7 +5888,7 @@ static void btrfs_end_bio(struct bio *bio) BUG_ON(stripe_index >= bbio->num_stripes); dev = bbio->stripes[stripe_index].dev; if (dev->bdev) { - if (bio->bi_rw & WRITE) + if (bio->bi_op == REQ_OP_WRITE) btrfs_dev_stat_inc(dev, BTRFS_DEV_STAT_WRITE_ERRS); else @@ -5953,7 +5953,7 @@ static noinline void btrfs_schedule_bio(struct btrfs_root *root, } /* don't bother with additional async steps for reads, right now */ - if (!(bio->bi_rw & REQ_WRITE)) { + if (bio->bi_op == REQ_OP_READ) { bio_get(bio); btrfsic_submit_bio(bio); bio_put(bio); @@ -6002,15 +6002,14 @@ static void submit_stripe_bio(struct btrfs_root *root, struct btrfs_bio *bbio, btrfs_io_bio(bio)->stripe_index = dev_nr; bio->bi_end_io = btrfs_end_bio; bio->bi_iter.bi_sector = physical >> 9; - bio->bi_rw |= rw; #ifdef DEBUG { struct rcu_string *name; rcu_read_lock(); name = rcu_dereference(dev->name); - pr_debug("btrfs_map_bio: rw %d, sector=%llu, dev=%lu " - "(%s id %llu), size=%u\n", rw, + pr_debug("btrfs_map_bio: rw %d 0x%x, sector=%llu, dev=%lu " + "(%s id %llu), size=%u\n", bio->bi_op, bio->bi_rw, (u64)bio->bi_iter.bi_sector, (u_long)dev->bdev->bd_dev, name->str, dev->devid, bio->bi_iter.bi_size); rcu_read_unlock(); @@ -6057,8 +6056,8 @@ int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, map_length = length; btrfs_bio_counter_inc_blocked(root->fs_info); - ret = __btrfs_map_block(root->fs_info, rw, logical, &map_length, &bbio, - mirror_num, 1); + ret = __btrfs_map_block(root->fs_info, bio->bi_op, logical, + &map_length, &bbio, mirror_num, 1); if (ret) { btrfs_bio_counter_dec(root->fs_info); return ret; @@ -6074,7 +6073,7 @@ int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, if (bbio->raid_map) { /* In this case, map_length has been set to the length of a single stripe; not the whole write */ - if (rw & WRITE) { + if (bio->bi_op == REQ_OP_WRITE) { ret = raid56_parity_write(root, bio, bbio, map_length); } else { ret = raid56_parity_recover(root, bio, bbio, map_length, @@ -6093,7 +6092,8 @@ int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, for (dev_nr = 0; dev_nr < total_devs; dev_nr++) { dev = bbio->stripes[dev_nr].dev; - if (!dev || !dev->bdev || (rw & WRITE && !dev->writeable)) { + if (!dev || !dev->bdev || + (bio->bi_op == REQ_OP_WRITE && !dev->writeable)) { bbio_error(bbio, first_bio, logical); continue; } -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:04 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 72CF529E23 for ; Mon, 11 Jan 2016 14:22:04 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 63127304053 for ; Mon, 11 Jan 2016 12:22:04 -0800 (PST) X-ASG-Debug-ID: 1452543722-04cbb05f7c97900001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id a3vs1EOHWpkFk8h0 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:03 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 7026BA37FC; Mon, 11 Jan 2016 20:22:02 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEc008385; Mon, 11 Jan 2016 15:22:00 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 09/35] btrfs: update __btrfs_map_block for bi_op transition Date: Mon, 11 Jan 2016 14:21:07 -0600 X-ASG-Orig-Subj: [PATCH 09/35] btrfs: update __btrfs_map_block for bi_op transition Message-Id: <1452543693-4440-10-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543723 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie We no longer pass in a bitmap of rq_flag_bits bits to __btrfs_map_block. It will always be a REQ_OP, or the btrfs specific REQ_GET_READ_MIRRORS, so this drops the bit tests. Signed-off-by: Mike Christie --- fs/btrfs/extent-tree.c | 2 +- fs/btrfs/inode.c | 2 +- fs/btrfs/volumes.c | 55 +++++++++++++++++++++++++++----------------------- fs/btrfs/volumes.h | 4 ++-- 4 files changed, 34 insertions(+), 29 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index add4af6..4d503d0 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2043,7 +2043,7 @@ int btrfs_discard_extent(struct btrfs_root *root, u64 bytenr, /* Tell the block device(s) that the sectors can be discarded */ - ret = btrfs_map_block(root->fs_info, REQ_DISCARD, + ret = btrfs_map_block(root->fs_info, REQ_OP_DISCARD, bytenr, &num_bytes, &bbio, 0); /* Error condition is -ENOMEM */ if (!ret) { diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index dd435ee..32a9aa1 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8206,7 +8206,7 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, int async_submit = 0; map_length = orig_bio->bi_iter.bi_size; - ret = btrfs_map_block(root->fs_info, rw, start_sector << 9, + ret = btrfs_map_block(root->fs_info, orig_bio->bi_op, start_sector << 9, &map_length, NULL, 0); if (ret) return -EIO; diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 0da1d32..bf1e9af 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -5218,7 +5218,7 @@ void btrfs_put_bbio(struct btrfs_bio *bbio) kfree(bbio); } -static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, +static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num, int need_raid_map) @@ -5296,7 +5296,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, raid56_full_stripe_start *= full_stripe_len; } - if (rw & REQ_DISCARD) { + if (op == REQ_OP_DISCARD) { /* we don't discard raid56 yet */ if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) { ret = -EOPNOTSUPP; @@ -5309,7 +5309,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, For other RAID types and for RAID[56] reads, just allow a single stripe (on a single disk). */ if ((map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) && - (rw & REQ_WRITE)) { + (op == REQ_OP_WRITE)) { max_len = stripe_len * nr_data_stripes(map) - (offset - raid56_full_stripe_start); } else { @@ -5332,8 +5332,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, btrfs_dev_replace_unlock(dev_replace); if (dev_replace_is_ongoing && mirror_num == map->num_stripes + 1 && - !(rw & (REQ_WRITE | REQ_DISCARD | REQ_GET_READ_MIRRORS)) && - dev_replace->tgtdev != NULL) { + op != REQ_OP_WRITE && op != REQ_OP_DISCARD && + op != REQ_GET_READ_MIRRORS && dev_replace->tgtdev != NULL) { /* * in dev-replace case, for repair case (that's the only * case where the mirror is selected explicitly when @@ -5422,15 +5422,17 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, (offset + *length); if (map->type & BTRFS_BLOCK_GROUP_RAID0) { - if (rw & REQ_DISCARD) + if (op == REQ_OP_DISCARD) num_stripes = min_t(u64, map->num_stripes, stripe_nr_end - stripe_nr_orig); stripe_nr = div_u64_rem(stripe_nr, map->num_stripes, &stripe_index); - if (!(rw & (REQ_WRITE | REQ_DISCARD | REQ_GET_READ_MIRRORS))) + if (op != REQ_OP_WRITE && op != REQ_OP_DISCARD && + op != REQ_GET_READ_MIRRORS) mirror_num = 1; } else if (map->type & BTRFS_BLOCK_GROUP_RAID1) { - if (rw & (REQ_WRITE | REQ_DISCARD | REQ_GET_READ_MIRRORS)) + if (op == REQ_OP_WRITE || op == REQ_OP_DISCARD || + op == REQ_GET_READ_MIRRORS) num_stripes = map->num_stripes; else if (mirror_num) stripe_index = mirror_num - 1; @@ -5443,7 +5445,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, } } else if (map->type & BTRFS_BLOCK_GROUP_DUP) { - if (rw & (REQ_WRITE | REQ_DISCARD | REQ_GET_READ_MIRRORS)) { + if (op == REQ_OP_WRITE || REQ_OP_DISCARD || + op == REQ_GET_READ_MIRRORS) { num_stripes = map->num_stripes; } else if (mirror_num) { stripe_index = mirror_num - 1; @@ -5457,9 +5460,9 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, stripe_nr = div_u64_rem(stripe_nr, factor, &stripe_index); stripe_index *= map->sub_stripes; - if (rw & (REQ_WRITE | REQ_GET_READ_MIRRORS)) + if (op == REQ_OP_WRITE || op == REQ_GET_READ_MIRRORS) num_stripes = map->sub_stripes; - else if (rw & REQ_DISCARD) + else if (op == REQ_OP_DISCARD) num_stripes = min_t(u64, map->sub_stripes * (stripe_nr_end - stripe_nr_orig), map->num_stripes); @@ -5477,7 +5480,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, } else if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) { if (need_raid_map && - ((rw & (REQ_WRITE | REQ_GET_READ_MIRRORS)) || + (op == REQ_OP_WRITE || op == REQ_GET_READ_MIRRORS || mirror_num > 1)) { /* push stripe_nr back to the start of the full stripe */ stripe_nr = div_u64(raid56_full_stripe_start, @@ -5505,8 +5508,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, /* We distribute the parity blocks across stripes */ div_u64_rem(stripe_nr + stripe_index, map->num_stripes, &stripe_index); - if (!(rw & (REQ_WRITE | REQ_DISCARD | - REQ_GET_READ_MIRRORS)) && mirror_num <= 1) + if ((op != REQ_OP_WRITE && op != REQ_OP_DISCARD && + op != REQ_GET_READ_MIRRORS) && mirror_num <= 1) mirror_num = 1; } } else { @@ -5523,9 +5526,9 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, num_alloc_stripes = num_stripes; if (dev_replace_is_ongoing) { - if (rw & (REQ_WRITE | REQ_DISCARD)) + if (op == REQ_OP_WRITE || op == REQ_OP_DISCARD) num_alloc_stripes <<= 1; - if (rw & REQ_GET_READ_MIRRORS) + if (op == REQ_GET_READ_MIRRORS) num_alloc_stripes++; tgtdev_indexes = num_stripes; } @@ -5540,7 +5543,8 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, /* build raid_map */ if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK && - need_raid_map && ((rw & (REQ_WRITE | REQ_GET_READ_MIRRORS)) || + need_raid_map && + ((op == REQ_OP_WRITE || op == REQ_GET_READ_MIRRORS) || mirror_num > 1)) { u64 tmp; unsigned rot; @@ -5565,7 +5569,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, RAID6_Q_STRIPE; } - if (rw & REQ_DISCARD) { + if (op == REQ_OP_DISCARD) { u32 factor = 0; u32 sub_stripes = 0; u64 stripes_per_dev = 0; @@ -5645,14 +5649,15 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, } } - if (rw & (REQ_WRITE | REQ_GET_READ_MIRRORS)) + if (op == REQ_OP_WRITE || op == REQ_GET_READ_MIRRORS) max_errors = btrfs_chunk_max_errors(map); if (bbio->raid_map) sort_parity_stripes(bbio, num_stripes); tgtdev_indexes = 0; - if (dev_replace_is_ongoing && (rw & (REQ_WRITE | REQ_DISCARD)) && + if (dev_replace_is_ongoing && + (op == REQ_OP_WRITE || op == REQ_OP_DISCARD) && dev_replace->tgtdev != NULL) { int index_where_to_add; u64 srcdev_devid = dev_replace->srcdev->devid; @@ -5687,7 +5692,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, } } num_stripes = index_where_to_add; - } else if (dev_replace_is_ongoing && (rw & REQ_GET_READ_MIRRORS) && + } else if (dev_replace_is_ongoing && (op == REQ_GET_READ_MIRRORS) && dev_replace->tgtdev != NULL) { u64 srcdev_devid = dev_replace->srcdev->devid; int index_srcdev = 0; @@ -5760,21 +5765,21 @@ out: return ret; } -int btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, +int btrfs_map_block(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num) { - return __btrfs_map_block(fs_info, rw, logical, length, bbio_ret, + return __btrfs_map_block(fs_info, op, logical, length, bbio_ret, mirror_num, 0); } /* For Scrub/replace */ -int btrfs_map_sblock(struct btrfs_fs_info *fs_info, int rw, +int btrfs_map_sblock(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num, int need_raid_map) { - return __btrfs_map_block(fs_info, rw, logical, length, bbio_ret, + return __btrfs_map_block(fs_info, op, logical, length, bbio_ret, mirror_num, need_raid_map); } diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h index 6a4375a..8ef796a 100644 --- a/fs/btrfs/volumes.h +++ b/fs/btrfs/volumes.h @@ -421,10 +421,10 @@ int btrfs_account_dev_extents_size(struct btrfs_device *device, u64 start, u64 end, u64 *length); void btrfs_get_bbio(struct btrfs_bio *bbio); void btrfs_put_bbio(struct btrfs_bio *bbio); -int btrfs_map_block(struct btrfs_fs_info *fs_info, int rw, +int btrfs_map_block(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num); -int btrfs_map_sblock(struct btrfs_fs_info *fs_info, int rw, +int btrfs_map_sblock(struct btrfs_fs_info *fs_info, int op, u64 logical, u64 *length, struct btrfs_bio **bbio_ret, int mirror_num, int need_raid_map); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id A53DF29E06 for ; Mon, 11 Jan 2016 14:22:10 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 95CD28F8040 for ; Mon, 11 Jan 2016 12:22:07 -0800 (PST) X-ASG-Debug-ID: 1452543725-04cb6c75de79040001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id eHwyxKrg9aSnRV4L (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:05 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id E72C83F3D4; Mon, 11 Jan 2016 20:22:04 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEd008385; Mon, 11 Jan 2016 15:22:02 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 10/35] btrfs: don't pass rq_flag_bits if there is a bio Date: Mon, 11 Jan 2016 14:21:08 -0600 X-ASG-Orig-Subj: [PATCH 10/35] btrfs: don't pass rq_flag_bits if there is a bio Message-Id: <1452543693-4440-11-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543725 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The bio bi_op and bi_rw is now setup, so there is no need to pass around the rq_flag_bits bits too. v2: 1. Fix merge_bio issue where instead of removing rw/op argument I passed it in again to the merge_bio related functions. Signed-off-by: Mike Christie --- fs/btrfs/compression.c | 13 ++++++------- fs/btrfs/ctree.h | 2 +- fs/btrfs/disk-io.c | 30 ++++++++++++------------------ fs/btrfs/disk-io.h | 2 +- fs/btrfs/extent_io.c | 12 +++++------- fs/btrfs/extent_io.h | 8 ++++---- fs/btrfs/inode.c | 44 ++++++++++++++++++++------------------------ fs/btrfs/volumes.c | 6 +++--- fs/btrfs/volumes.h | 2 +- 9 files changed, 53 insertions(+), 66 deletions(-) diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index 25bf179..fd1837b 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -374,7 +374,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, page = compressed_pages[pg_index]; page->mapping = inode->i_mapping; if (bio->bi_iter.bi_size) - ret = io_tree->ops->merge_bio_hook(WRITE, page, 0, + ret = io_tree->ops->merge_bio_hook(page, 0, PAGE_CACHE_SIZE, bio, 0); else @@ -402,7 +402,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, BUG_ON(ret); /* -ENOMEM */ } - ret = btrfs_map_bio(root, WRITE, bio, 0, 1); + ret = btrfs_map_bio(root, bio, 0, 1); BUG_ON(ret); /* -ENOMEM */ bio_put(bio); @@ -433,7 +433,7 @@ int btrfs_submit_compressed_write(struct inode *inode, u64 start, BUG_ON(ret); /* -ENOMEM */ } - ret = btrfs_map_bio(root, WRITE, bio, 0, 1); + ret = btrfs_map_bio(root, bio, 0, 1); BUG_ON(ret); /* -ENOMEM */ bio_put(bio); @@ -663,7 +663,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, page->index = em_start >> PAGE_CACHE_SHIFT; if (comp_bio->bi_iter.bi_size) - ret = tree->ops->merge_bio_hook(READ, page, 0, + ret = tree->ops->merge_bio_hook(page, 0, PAGE_CACHE_SIZE, comp_bio, 0); else @@ -694,8 +694,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, sums += DIV_ROUND_UP(comp_bio->bi_iter.bi_size, root->sectorsize); - ret = btrfs_map_bio(root, READ, comp_bio, - mirror_num, 0); + ret = btrfs_map_bio(root, comp_bio, mirror_num, 0); if (ret) { bio->bi_error = ret; bio_endio(comp_bio); @@ -725,7 +724,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, BUG_ON(ret); /* -ENOMEM */ } - ret = btrfs_map_bio(root, READ, comp_bio, mirror_num, 0); + ret = btrfs_map_bio(root, comp_bio, mirror_num, 0); if (ret) { bio->bi_error = ret; bio_endio(comp_bio); diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index f2f0578..038fcf5 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -4040,7 +4040,7 @@ int btrfs_create_subvol_root(struct btrfs_trans_handle *trans, struct btrfs_root *new_root, struct btrfs_root *parent_root, u64 new_dirid); -int btrfs_merge_bio_hook(int rw, struct page *page, unsigned long offset, +int btrfs_merge_bio_hook(struct page *page, unsigned long offset, size_t size, struct bio *bio, unsigned long bio_flags); int btrfs_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf); diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 28cf697..cb719bc 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -119,7 +119,6 @@ struct async_submit_bio { struct list_head list; extent_submit_bio_hook_t *submit_bio_start; extent_submit_bio_hook_t *submit_bio_done; - int rw; int mirror_num; unsigned long bio_flags; /* @@ -783,7 +782,7 @@ static void run_one_async_start(struct btrfs_work *work) int ret; async = container_of(work, struct async_submit_bio, work); - ret = async->submit_bio_start(async->inode, async->rw, async->bio, + ret = async->submit_bio_start(async->inode, async->bio, async->mirror_num, async->bio_flags, async->bio_offset); if (ret) @@ -816,9 +815,8 @@ static void run_one_async_done(struct btrfs_work *work) return; } - async->submit_bio_done(async->inode, async->rw, async->bio, - async->mirror_num, async->bio_flags, - async->bio_offset); + async->submit_bio_done(async->inode, async->bio, async->mirror_num, + async->bio_flags, async->bio_offset); } static void run_one_async_free(struct btrfs_work *work) @@ -830,7 +828,7 @@ static void run_one_async_free(struct btrfs_work *work) } int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode, - int rw, struct bio *bio, int mirror_num, + struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset, extent_submit_bio_hook_t *submit_bio_start, @@ -843,7 +841,6 @@ int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode, return -ENOMEM; async->inode = inode; - async->rw = rw; async->bio = bio; async->mirror_num = mirror_num; async->submit_bio_start = submit_bio_start; @@ -889,9 +886,8 @@ static int btree_csum_one_bio(struct bio *bio) return ret; } -static int __btree_submit_bio_start(struct inode *inode, int rw, - struct bio *bio, int mirror_num, - unsigned long bio_flags, +static int __btree_submit_bio_start(struct inode *inode, struct bio *bio, + int mirror_num, unsigned long bio_flags, u64 bio_offset) { /* @@ -901,7 +897,7 @@ static int __btree_submit_bio_start(struct inode *inode, int rw, return btree_csum_one_bio(bio); } -static int __btree_submit_bio_done(struct inode *inode, int rw, struct bio *bio, +static int __btree_submit_bio_done(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset) { @@ -911,7 +907,7 @@ static int __btree_submit_bio_done(struct inode *inode, int rw, struct bio *bio, * when we're called for a write, we're already in the async * submission context. Just jump into btrfs_map_bio */ - ret = btrfs_map_bio(BTRFS_I(inode)->root, rw, bio, mirror_num, 1); + ret = btrfs_map_bio(BTRFS_I(inode)->root, bio, mirror_num, 1); if (ret) { bio->bi_error = ret; bio_endio(bio); @@ -930,7 +926,7 @@ static int check_async_write(struct inode *inode, unsigned long bio_flags) return 1; } -static int btree_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, +static int btree_submit_bio_hook(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset) { @@ -946,21 +942,19 @@ static int btree_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, bio, BTRFS_WQ_ENDIO_METADATA); if (ret) goto out_w_error; - ret = btrfs_map_bio(BTRFS_I(inode)->root, rw, bio, - mirror_num, 0); + ret = btrfs_map_bio(BTRFS_I(inode)->root, bio, mirror_num, 0); } else if (!async) { ret = btree_csum_one_bio(bio); if (ret) goto out_w_error; - ret = btrfs_map_bio(BTRFS_I(inode)->root, rw, bio, - mirror_num, 0); + ret = btrfs_map_bio(BTRFS_I(inode)->root, bio, mirror_num, 0); } else { /* * kthread helpers are used to submit writes so that * checksumming can happen in parallel across all CPUs */ ret = btrfs_wq_submit_bio(BTRFS_I(inode)->root->fs_info, - inode, rw, bio, mirror_num, 0, + inode, bio, mirror_num, 0, bio_offset, __btree_submit_bio_start, __btree_submit_bio_done); diff --git a/fs/btrfs/disk-io.h b/fs/btrfs/disk-io.h index 7c52e29..4a5d5df 100644 --- a/fs/btrfs/disk-io.h +++ b/fs/btrfs/disk-io.h @@ -122,7 +122,7 @@ void btrfs_csum_final(u32 crc, char *result); int btrfs_bio_wq_end_io(struct btrfs_fs_info *info, struct bio *bio, enum btrfs_wq_endio_type metadata); int btrfs_wq_submit_bio(struct btrfs_fs_info *fs_info, struct inode *inode, - int rw, struct bio *bio, int mirror_num, + struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset, extent_submit_bio_hook_t *submit_bio_start, extent_submit_bio_hook_t *submit_bio_done); diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index a4830f0..8c8d4f7 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2414,8 +2414,7 @@ static int bio_readpage_error(struct bio *failed_bio, u64 phy_offset, pr_debug("Repair Read Error: submitting new read[%#x] to this_mirror=%d, in_validation=%d\n", read_mode, failrec->this_mirror, failrec->in_validation); - ret = tree->ops->submit_bio_hook(inode, read_mode, bio, - failrec->this_mirror, + ret = tree->ops->submit_bio_hook(inode, bio, failrec->this_mirror, failrec->bio_flags, 0); if (ret) { free_io_failure(inode, failrec); @@ -2736,9 +2735,8 @@ static int __must_check submit_one_bio(struct bio *bio, int mirror_num, bio_get(bio); if (tree->ops && tree->ops->submit_bio_hook) - ret = tree->ops->submit_bio_hook(page->mapping->host, - bio->bi_rw, bio, mirror_num, - bio_flags, start); + ret = tree->ops->submit_bio_hook(page->mapping->host, bio, + mirror_num, bio_flags, start); else btrfsic_submit_bio(bio); @@ -2752,8 +2750,8 @@ static int merge_bio(struct extent_io_tree *tree, struct page *page, { int ret = 0; if (tree->ops && tree->ops->merge_bio_hook) - ret = tree->ops->merge_bio_hook(bio->bi_op, page, offset, size, - bio, bio_flags); + ret = tree->ops->merge_bio_hook(page, offset, size, bio, + bio_flags); BUG_ON(ret < 0); return ret; diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h index 0377413..062195d 100644 --- a/fs/btrfs/extent_io.h +++ b/fs/btrfs/extent_io.h @@ -63,9 +63,9 @@ struct extent_state; struct btrfs_root; struct btrfs_io_bio; -typedef int (extent_submit_bio_hook_t)(struct inode *inode, int rw, - struct bio *bio, int mirror_num, - unsigned long bio_flags, u64 bio_offset); +typedef int (extent_submit_bio_hook_t)(struct inode *inode, struct bio *bio, + int mirror_num, unsigned long bio_flags, + u64 bio_offset); struct extent_io_ops { int (*fill_delalloc)(struct inode *inode, struct page *locked_page, u64 start, u64 end, int *page_started, @@ -73,7 +73,7 @@ struct extent_io_ops { int (*writepage_start_hook)(struct page *page, u64 start, u64 end); int (*writepage_io_hook)(struct page *page, u64 start, u64 end); extent_submit_bio_hook_t *submit_bio_hook; - int (*merge_bio_hook)(int rw, struct page *page, unsigned long offset, + int (*merge_bio_hook)(struct page *page, unsigned long offset, size_t size, struct bio *bio, unsigned long bio_flags); int (*readpage_io_failed_hook)(struct page *page, int failed_mirror); diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 32a9aa1..bb35ed2 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -1808,7 +1808,7 @@ static void btrfs_clear_bit_hook(struct inode *inode, * extent_io.c merge_bio_hook, this must check the chunk tree to make sure * we don't create bios that span stripes or chunks */ -int btrfs_merge_bio_hook(int rw, struct page *page, unsigned long offset, +int btrfs_merge_bio_hook(struct page *page, unsigned long offset, size_t size, struct bio *bio, unsigned long bio_flags) { @@ -1823,7 +1823,7 @@ int btrfs_merge_bio_hook(int rw, struct page *page, unsigned long offset, length = bio->bi_iter.bi_size; map_length = length; - ret = btrfs_map_block(root->fs_info, rw, logical, + ret = btrfs_map_block(root->fs_info, bio->bi_op, logical, &map_length, NULL, 0); /* Will always return 0 with map_multi == NULL */ BUG_ON(ret < 0); @@ -1840,9 +1840,8 @@ int btrfs_merge_bio_hook(int rw, struct page *page, unsigned long offset, * At IO completion time the cums attached on the ordered extent record * are inserted into the btree */ -static int __btrfs_submit_bio_start(struct inode *inode, int rw, - struct bio *bio, int mirror_num, - unsigned long bio_flags, +static int __btrfs_submit_bio_start(struct inode *inode, struct bio *bio, + int mirror_num, unsigned long bio_flags, u64 bio_offset) { struct btrfs_root *root = BTRFS_I(inode)->root; @@ -1861,14 +1860,14 @@ static int __btrfs_submit_bio_start(struct inode *inode, int rw, * At IO completion time the cums attached on the ordered extent record * are inserted into the btree */ -static int __btrfs_submit_bio_done(struct inode *inode, int rw, struct bio *bio, +static int __btrfs_submit_bio_done(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset) { struct btrfs_root *root = BTRFS_I(inode)->root; int ret; - ret = btrfs_map_bio(root, rw, bio, mirror_num, 1); + ret = btrfs_map_bio(root, bio, mirror_num, 1); if (ret) { bio->bi_error = ret; bio_endio(bio); @@ -1880,7 +1879,7 @@ static int __btrfs_submit_bio_done(struct inode *inode, int rw, struct bio *bio, * extent_io.c submission hook. This does the right thing for csum calculation * on write, or reading the csums from the tree before a read */ -static int btrfs_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, +static int btrfs_submit_bio_hook(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 bio_offset) { @@ -1917,7 +1916,7 @@ static int btrfs_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, goto mapit; /* we're doing a write, do the async checksumming */ ret = btrfs_wq_submit_bio(BTRFS_I(inode)->root->fs_info, - inode, rw, bio, mirror_num, + inode, bio, mirror_num, bio_flags, bio_offset, __btrfs_submit_bio_start, __btrfs_submit_bio_done); @@ -1929,7 +1928,7 @@ static int btrfs_submit_bio_hook(struct inode *inode, int rw, struct bio *bio, } mapit: - ret = btrfs_map_bio(root, rw, bio, mirror_num, 0); + ret = btrfs_map_bio(root, bio, mirror_num, 0); out: if (ret < 0) { @@ -7708,7 +7707,7 @@ err: } static inline int submit_dio_repair_bio(struct inode *inode, struct bio *bio, - int rw, int mirror_num) + int mirror_num) { struct btrfs_root *root = BTRFS_I(inode)->root; int ret; @@ -7722,7 +7721,7 @@ static inline int submit_dio_repair_bio(struct inode *inode, struct bio *bio, if (ret) goto err; - ret = btrfs_map_bio(root, rw, bio, mirror_num, 0); + ret = btrfs_map_bio(root, bio, mirror_num, 0); err: bio_put(bio); return ret; @@ -7806,8 +7805,7 @@ static int dio_read_error(struct inode *inode, struct bio *failed_bio, "Repair DIO Read Error: submitting new dio read[%#x] to this_mirror=%d, in_validation=%d\n", read_mode, failrec->this_mirror, failrec->in_validation); - ret = submit_dio_repair_bio(inode, bio, read_mode, - failrec->this_mirror); + ret = submit_dio_repair_bio(inode, bio, failrec->this_mirror); if (ret) { free_io_failure(inode, failrec); bio_put(bio); @@ -8045,7 +8043,7 @@ static void btrfs_endio_direct_write(struct bio *bio) bio_put(bio); } -static int __btrfs_submit_bio_start_direct_io(struct inode *inode, int rw, +static int __btrfs_submit_bio_start_direct_io(struct inode *inode, struct bio *bio, int mirror_num, unsigned long bio_flags, u64 offset) { @@ -8138,7 +8136,7 @@ static inline int btrfs_lookup_and_bind_dio_csum(struct btrfs_root *root, } static inline int __btrfs_submit_dio_bio(struct bio *bio, struct inode *inode, - int rw, u64 file_offset, int skip_sum, + u64 file_offset, int skip_sum, int async_submit) { struct btrfs_dio_private *dip = bio->bi_private; @@ -8163,8 +8161,7 @@ static inline int __btrfs_submit_dio_bio(struct bio *bio, struct inode *inode, if (write && async_submit) { ret = btrfs_wq_submit_bio(root->fs_info, - inode, rw, bio, 0, 0, - file_offset, + inode, bio, 0, 0, file_offset, __btrfs_submit_bio_start_direct_io, __btrfs_submit_bio_done); goto err; @@ -8183,13 +8180,13 @@ static inline int __btrfs_submit_dio_bio(struct bio *bio, struct inode *inode, goto err; } map: - ret = btrfs_map_bio(root, rw, bio, 0, async_submit); + ret = btrfs_map_bio(root, bio, 0, async_submit); err: bio_put(bio); return ret; } -static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, +static int btrfs_submit_direct_hook(struct btrfs_dio_private *dip, int skip_sum) { struct inode *inode = dip->inode; @@ -8245,7 +8242,7 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, * before we're done setting it up */ atomic_inc(&dip->pending_bios); - ret = __btrfs_submit_dio_bio(bio, inode, rw, + ret = __btrfs_submit_dio_bio(bio, inode, file_offset, skip_sum, async_submit); if (ret) { @@ -8286,7 +8283,7 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip, } submit: - ret = __btrfs_submit_dio_bio(bio, inode, rw, file_offset, skip_sum, + ret = __btrfs_submit_dio_bio(bio, inode, file_offset, skip_sum, async_submit); if (!ret) return 0; @@ -8364,8 +8361,7 @@ static void btrfs_submit_direct(struct bio *dio_bio, struct inode *inode, dio_data->unsubmitted_oe_range_end; } - ret = btrfs_submit_direct_hook(dio_bio->bi_op | dio_bio->bi_rw, dip, - skip_sum); + ret = btrfs_submit_direct_hook(dip, skip_sum); if (!ret) return; diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index bf1e9af..3535659 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -5999,7 +5999,7 @@ static noinline void btrfs_schedule_bio(struct btrfs_root *root, static void submit_stripe_bio(struct btrfs_root *root, struct btrfs_bio *bbio, struct bio *bio, u64 physical, int dev_nr, - int rw, int async) + int async) { struct btrfs_device *dev = bbio->stripes[dev_nr].dev; @@ -6044,7 +6044,7 @@ static void bbio_error(struct btrfs_bio *bbio, struct bio *bio, u64 logical) } } -int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, +int btrfs_map_bio(struct btrfs_root *root, struct bio *bio, int mirror_num, int async_submit) { struct btrfs_device *dev; @@ -6110,7 +6110,7 @@ int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, bio = first_bio; submit_stripe_bio(root, bbio, bio, - bbio->stripes[dev_nr].physical, dev_nr, rw, + bbio->stripes[dev_nr].physical, dev_nr, async_submit); } btrfs_bio_counter_dec(root->fs_info); diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h index 8ef796a..5a2da8a 100644 --- a/fs/btrfs/volumes.h +++ b/fs/btrfs/volumes.h @@ -437,7 +437,7 @@ int btrfs_alloc_chunk(struct btrfs_trans_handle *trans, struct btrfs_root *extent_root, u64 type); void btrfs_mapping_init(struct btrfs_mapping_tree *tree); void btrfs_mapping_tree_free(struct btrfs_mapping_tree *tree); -int btrfs_map_bio(struct btrfs_root *root, int rw, struct bio *bio, +int btrfs_map_bio(struct btrfs_root *root, struct bio *bio, int mirror_num, int async_submit); int btrfs_open_devices(struct btrfs_fs_devices *fs_devices, fmode_t flags, void *holder); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:10 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id ACB7D29E0A for ; Mon, 11 Jan 2016 14:22:10 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 1CA27AC006 for ; Mon, 11 Jan 2016 12:22:10 -0800 (PST) X-ASG-Debug-ID: 1452543727-04cbb05f7d97930001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id YJSFB9ZDY1kZdcUW (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:08 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 44981ABB16; Mon, 11 Jan 2016 20:22:07 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEe008385; Mon, 11 Jan 2016 15:22:04 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 11/35] f2fs: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:09 -0600 X-ASG-Orig-Subj: [PATCH 11/35] f2fs: set bi_op to REQ_OP Message-Id: <1452543693-4440-12-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543727 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has f2fs set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- fs/f2fs/checkpoint.c | 10 ++++++---- fs/f2fs/data.c | 33 ++++++++++++++++++++------------- fs/f2fs/f2fs.h | 5 +++-- fs/f2fs/gc.c | 9 ++++++--- fs/f2fs/inline.c | 3 ++- fs/f2fs/node.c | 8 +++++--- fs/f2fs/segment.c | 10 +++++++--- fs/f2fs/trace.c | 8 +++++--- include/trace/events/f2fs.h | 34 +++++++++++++++++++++------------- 9 files changed, 75 insertions(+), 45 deletions(-) diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index 5dbafd5..98bf767 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -55,13 +55,14 @@ static struct page *__get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index, struct f2fs_io_info fio = { .sbi = sbi, .type = META, - .rw = READ_SYNC | REQ_META | REQ_PRIO, + .op = REQ_OP_READ, + .op_flags = READ_SYNC | REQ_META | REQ_PRIO, .blk_addr = index, .encrypted_page = NULL, }; if (unlikely(!is_meta)) - fio.rw &= ~REQ_META; + fio.op_flags &= ~REQ_META; repeat: page = grab_cache_page(mapping, index); if (!page) { @@ -149,12 +150,13 @@ int ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages, struct f2fs_io_info fio = { .sbi = sbi, .type = META, - .rw = sync ? (READ_SYNC | REQ_META | REQ_PRIO) : READA, + .op = REQ_OP_READ, + .op_flags = sync ? (READ_SYNC | REQ_META | REQ_PRIO) : READA, .encrypted_page = NULL, }; if (unlikely(type == META_POR)) - fio.rw &= ~REQ_META; + fio.op_flags &= ~REQ_META; for (; nrpages-- > 0; blkno++) { diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index e1c64ed..4c0f054 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -107,11 +107,12 @@ static void __submit_merged_bio(struct f2fs_bio_info *io) if (!io->bio) return; - if (is_read_io(fio->rw)) + if (is_read_io(fio->op)) trace_f2fs_submit_read_bio(io->sbi->sb, fio, io->bio); else trace_f2fs_submit_write_bio(io->sbi->sb, fio, io->bio); - io->bio->bi_rw = fio->rw; + io->bio->bi_op = fio->op; + io->bio->bi_rw = fio->op_flags; submit_bio(io->bio); io->bio = NULL; @@ -130,10 +131,12 @@ void f2fs_submit_merged_bio(struct f2fs_sb_info *sbi, /* change META to META_FLUSH in the checkpoint procedure */ if (type >= META_FLUSH) { io->fio.type = META_FLUSH; + io->fio.op = REQ_OP_WRITE; if (test_opt(sbi, NOBARRIER)) - io->fio.rw = WRITE_FLUSH | REQ_META | REQ_PRIO; + io->fio.op_flags = WRITE_FLUSH | REQ_META | REQ_PRIO; else - io->fio.rw = WRITE_FLUSH_FUA | REQ_META | REQ_PRIO; + io->fio.op_flags = WRITE_FLUSH_FUA | REQ_META | + REQ_PRIO; } __submit_merged_bio(io); up_write(&io->io_rwsem); @@ -152,13 +155,14 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio) f2fs_trace_ios(fio, 0); /* Allocate a new bio */ - bio = __bio_alloc(fio->sbi, fio->blk_addr, 1, is_read_io(fio->rw)); + bio = __bio_alloc(fio->sbi, fio->blk_addr, 1, is_read_io(fio->op)); if (bio_add_page(bio, page, PAGE_CACHE_SIZE, 0) < PAGE_CACHE_SIZE) { bio_put(bio); return -EFAULT; } - bio->bi_rw = fio->rw; + bio->bi_op = fio->op; + bio->bi_rw = fio->op_flags; submit_bio(bio); return 0; @@ -169,7 +173,7 @@ void f2fs_submit_page_mbio(struct f2fs_io_info *fio) struct f2fs_sb_info *sbi = fio->sbi; enum page_type btype = PAGE_TYPE_OF_BIO(fio->type); struct f2fs_bio_info *io; - bool is_read = is_read_io(fio->rw); + bool is_read = is_read_io(fio->op); struct page *bio_page; io = is_read ? &sbi->read_io : &sbi->write_io[btype]; @@ -182,7 +186,7 @@ void f2fs_submit_page_mbio(struct f2fs_io_info *fio) inc_page_count(sbi, F2FS_WRITEBACK); if (io->bio && (io->last_block_in_bio != fio->blk_addr - 1 || - io->fio.rw != fio->rw)) + (io->fio.op != fio->op || io->fio.op_flags != fio->op_flags))) __submit_merged_bio(io); alloc_new: if (io->bio == NULL) { @@ -279,7 +283,7 @@ int f2fs_get_block(struct dnode_of_data *dn, pgoff_t index) } struct page *get_read_data_page(struct inode *inode, pgoff_t index, - int rw, bool for_write) + int op_flags, bool for_write) { struct address_space *mapping = inode->i_mapping; struct dnode_of_data dn; @@ -289,7 +293,8 @@ struct page *get_read_data_page(struct inode *inode, pgoff_t index, struct f2fs_io_info fio = { .sbi = F2FS_I_SB(inode), .type = DATA, - .rw = rw, + .op = REQ_OP_READ, + .op_flags = op_flags, .encrypted_page = NULL, }; @@ -992,7 +997,7 @@ submit_and_realloc: bio->bi_iter.bi_sector = SECTOR_FROM_BLOCK(block_nr); bio->bi_end_io = f2fs_read_end_io; bio->bi_private = ctx; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; } if (bio_add_page(bio, page, blocksize, 0) < blocksize) @@ -1126,7 +1131,8 @@ static int f2fs_write_data_page(struct page *page, struct f2fs_io_info fio = { .sbi = sbi, .type = DATA, - .rw = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : WRITE, + .op = REQ_OP_WRITE, + .op_flags = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : 0, .page = page, .encrypted_page = NULL, }; @@ -1553,7 +1559,8 @@ repeat: struct f2fs_io_info fio = { .sbi = sbi, .type = DATA, - .rw = READ_SYNC, + .op = REQ_OP_READ, + .op_flags = READ_SYNC, .blk_addr = blkaddr, .page = page, .encrypted_page = NULL, diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 5e3d0f0..31abc37 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -684,13 +684,14 @@ enum page_type { struct f2fs_io_info { struct f2fs_sb_info *sbi; /* f2fs_sb_info pointer */ enum page_type type; /* contains DATA/NODE/META/META_FLUSH */ - int rw; /* contains R/RS/W/WS with REQ_META/REQ_PRIO */ + int op; /* contains REQ_OP_ */ + int op_flags; /* rq_flag_bits */ block_t blk_addr; /* block address to be written */ struct page *page; /* page to be written */ struct page *encrypted_page; /* encrypted page */ }; -#define is_read_io(rw) (((rw) & 1) == READ) +#define is_read_io(rw) (rw == READ) struct f2fs_bio_info { struct f2fs_sb_info *sbi; /* f2fs superblock */ struct bio *bio; /* bios to merge */ diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index c09be33..29a5a12 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -540,7 +540,8 @@ static void move_encrypted_block(struct inode *inode, block_t bidx) struct f2fs_io_info fio = { .sbi = F2FS_I_SB(inode), .type = DATA, - .rw = READ_SYNC, + .op = REQ_OP_READ, + .op_flags = READ_SYNC, .encrypted_page = NULL, }; struct dnode_of_data dn; @@ -607,7 +608,8 @@ static void move_encrypted_block(struct inode *inode, block_t bidx) f2fs_wait_on_page_writeback(dn.node_page, NODE); allocate_data_block(fio.sbi, NULL, fio.blk_addr, &fio.blk_addr, &sum, CURSEG_COLD_DATA); - fio.rw = WRITE_SYNC; + fio.op = REQ_OP_WRITE; + fio.op_flags = WRITE_SYNC; f2fs_submit_page_mbio(&fio); dn.data_blkaddr = fio.blk_addr; @@ -641,7 +643,8 @@ static void move_data_page(struct inode *inode, block_t bidx, int gc_type) struct f2fs_io_info fio = { .sbi = F2FS_I_SB(inode), .type = DATA, - .rw = WRITE_SYNC, + .op = REQ_OP_WRITE, + .op_flags = WRITE_SYNC, .page = page, .encrypted_page = NULL, }; diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c index 5ffbd16..ef863b5 100644 --- a/fs/f2fs/inline.c +++ b/fs/f2fs/inline.c @@ -109,7 +109,8 @@ int f2fs_convert_inline_page(struct dnode_of_data *dn, struct page *page) struct f2fs_io_info fio = { .sbi = F2FS_I_SB(dn->inode), .type = DATA, - .rw = WRITE_SYNC | REQ_PRIO, + .op = REQ_OP_WRITE, + .op_flags = WRITE_SYNC | REQ_PRIO, .page = page, .encrypted_page = NULL, }; diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 6d5f548..3eb4919 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -1007,14 +1007,15 @@ fail: * 0: f2fs_put_page(page, 0) * LOCKED_PAGE or error: f2fs_put_page(page, 1) */ -static int read_node_page(struct page *page, int rw) +static int read_node_page(struct page *page, int op_flags) { struct f2fs_sb_info *sbi = F2FS_P_SB(page); struct node_info ni; struct f2fs_io_info fio = { .sbi = sbi, .type = NODE, - .rw = rw, + .op = REQ_OP_READ, + .op_flags = op_flags, .page = page, .encrypted_page = NULL, }; @@ -1325,7 +1326,8 @@ static int f2fs_write_node_page(struct page *page, struct f2fs_io_info fio = { .sbi = sbi, .type = NODE, - .rw = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : WRITE, + .op = REQ_OP_WRITE, + .op_flags = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : 0, .page = page, .encrypted_page = NULL, }; diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index f518cab..e7f1a6f 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -200,7 +200,8 @@ int commit_inmem_pages(struct inode *inode, bool abort) struct f2fs_io_info fio = { .sbi = sbi, .type = DATA, - .rw = WRITE_SYNC | REQ_PRIO, + .op = REQ_OP_WRITE, + .op_flags = WRITE_SYNC | REQ_PRIO, .encrypted_page = NULL, }; int err = 0; @@ -318,6 +319,7 @@ repeat: fcc->dispatch_list = llist_reverse_order(fcc->dispatch_list); bio->bi_bdev = sbi->sb->s_bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw = WRITE_FLUSH; ret = submit_bio_wait(bio); @@ -351,6 +353,7 @@ int f2fs_issue_flush(struct f2fs_sb_info *sbi) int ret; bio->bi_bdev = sbi->sb->s_bdev; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw = WRITE_FLUSH; ret = submit_bio_wait(bio); bio_put(bio); @@ -1291,14 +1294,15 @@ void write_meta_page(struct f2fs_sb_info *sbi, struct page *page) struct f2fs_io_info fio = { .sbi = sbi, .type = META, - .rw = WRITE_SYNC | REQ_META | REQ_PRIO, + .op = REQ_OP_WRITE, + .op_flags = WRITE_SYNC | REQ_META | REQ_PRIO, .blk_addr = page->index, .page = page, .encrypted_page = NULL, }; if (unlikely(page->index >= MAIN_BLKADDR(sbi))) - fio.rw &= ~REQ_META; + fio.op_flags &= ~REQ_META; set_page_writeback(page); f2fs_submit_page_mbio(&fio); diff --git a/fs/f2fs/trace.c b/fs/f2fs/trace.c index 145fb65..5b7edca 100644 --- a/fs/f2fs/trace.c +++ b/fs/f2fs/trace.c @@ -25,11 +25,12 @@ static inline void __print_last_io(void) if (!last_io.len) return; - trace_printk("%3x:%3x %4x %-16s %2x %5x %12x %4x\n", + trace_printk("%3x:%3x %4x %-16s %2x %5x %5x %12x %4x\n", last_io.major, last_io.minor, last_io.pid, "----------------", last_io.type, - last_io.fio.rw, last_io.fio.blk_addr, + last_io.fio.op, last_io.fio.op_flags, + last_io.fio.blk_addr, last_io.len); memset(&last_io, 0, sizeof(last_io)); } @@ -100,7 +101,8 @@ void f2fs_trace_ios(struct f2fs_io_info *fio, int flush) if (last_io.major == major && last_io.minor == minor && last_io.pid == pid && last_io.type == __file_type(inode, pid) && - last_io.fio.rw == fio->rw && + last_io.fio.op == fio->op && + last_io.fio.op_flags == fio->op_flags && last_io.fio.blk_addr + last_io.len == fio->blk_addr) { last_io.len++; return; diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index a1b4888..851c335 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -55,17 +55,21 @@ TRACE_DEFINE_ENUM(CP_DISCARD); { IPU, "IN-PLACE" }, \ { OPU, "OUT-OF-PLACE" }) -#define F2FS_BIO_MASK(t) (t & (READA | WRITE_FLUSH_FUA)) +#define F2FS_BIO_FLAG_MASK(t) (t & (READA | WRITE_FLUSH_FUA)) #define F2FS_BIO_EXTRA_MASK(t) (t & (REQ_META | REQ_PRIO)) -#define show_bio_type(type) show_bio_base(type), show_bio_extra(type) +#define show_bio_type(op, op_flags) show_bio_op(op), \ + show_bio_op_flags(op_flags), show_bio_extra(op_flags) -#define show_bio_base(type) \ - __print_symbolic(F2FS_BIO_MASK(type), \ +#define show_bio_op(op) \ + __print_symbolic(op, \ { READ, "READ" }, \ + { WRITE, "WRITE" }) + +#define show_bio_op_flags(flags) \ + __print_symbolic(F2FS_BIO_FLAG_MASK(flags), \ { READA, "READAHEAD" }, \ { READ_SYNC, "READ_SYNC" }, \ - { WRITE, "WRITE" }, \ { WRITE_SYNC, "WRITE_SYNC" }, \ { WRITE_FLUSH, "WRITE_FLUSH" }, \ { WRITE_FUA, "WRITE_FUA" }, \ @@ -728,7 +732,8 @@ DECLARE_EVENT_CLASS(f2fs__submit_page_bio, __field(ino_t, ino) __field(pgoff_t, index) __field(block_t, blkaddr) - __field(int, rw) + __field(int, op) + __field(int, op_flags) __field(int, type) ), @@ -737,16 +742,17 @@ DECLARE_EVENT_CLASS(f2fs__submit_page_bio, __entry->ino = page->mapping->host->i_ino; __entry->index = page->index; __entry->blkaddr = fio->blk_addr; - __entry->rw = fio->rw; + __entry->op = fio->op; + __entry->op_flags = fio->op_flags; __entry->type = fio->type; ), TP_printk("dev = (%d,%d), ino = %lu, page_index = 0x%lx, " - "blkaddr = 0x%llx, rw = %s%s, type = %s", + "blkaddr = 0x%llx, rw = %s%s%s, type = %s", show_dev_ino(__entry), (unsigned long)__entry->index, (unsigned long long)__entry->blkaddr, - show_bio_type(__entry->rw), + show_bio_type(__entry->op, __entry->op_flags), show_block_type(__entry->type)) ); @@ -777,7 +783,8 @@ DECLARE_EVENT_CLASS(f2fs__submit_bio, TP_STRUCT__entry( __field(dev_t, dev) - __field(int, rw) + __field(int, op) + __field(int, op_flags) __field(int, type) __field(sector_t, sector) __field(unsigned int, size) @@ -785,15 +792,16 @@ DECLARE_EVENT_CLASS(f2fs__submit_bio, TP_fast_assign( __entry->dev = sb->s_dev; - __entry->rw = fio->rw; + __entry->op = fio->op; + __entry->op_flags = fio->op_flags; __entry->type = fio->type; __entry->sector = bio->bi_iter.bi_sector; __entry->size = bio->bi_iter.bi_size; ), - TP_printk("dev = (%d,%d), %s%s, %s, sector = %lld, size = %u", + TP_printk("dev = (%d,%d), %s%s%s, %s, sector = %lld, size = %u", show_dev(__entry), - show_bio_type(__entry->rw), + show_bio_type(__entry->op, __entry->op_flags), show_block_type(__entry->type), (unsigned long long)__entry->sector, __entry->size) -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:12 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id EC8E529E20 for ; Mon, 11 Jan 2016 14:22:11 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 70BB3AC005 for ; Mon, 11 Jan 2016 12:22:11 -0800 (PST) X-ASG-Debug-ID: 1452543729-04cbb05f7c97940001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id vKPCRkWztK49b4WB (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:10 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 5FF358DFE0; Mon, 11 Jan 2016 20:22:09 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEf008385; Mon, 11 Jan 2016 15:22:07 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 12/35] gfs2: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:10 -0600 X-ASG-Orig-Subj: [PATCH 12/35] gfs2: set bi_op to REQ_OP Message-Id: <1452543693-4440-13-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543729 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has gfs2 set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. v2: Bob, I did not add your signed off, because there was the gfs2_submit_bhs changes since last time you reviewed it. Signed-off-by: Mike Christie --- fs/gfs2/log.c | 8 ++++---- fs/gfs2/lops.c | 12 +++++++----- fs/gfs2/lops.h | 2 +- fs/gfs2/meta_io.c | 8 +++++--- fs/gfs2/ops_fstype.c | 1 + 5 files changed, 18 insertions(+), 13 deletions(-) diff --git a/fs/gfs2/log.c b/fs/gfs2/log.c index 0ff028c..e58ccef0 100644 --- a/fs/gfs2/log.c +++ b/fs/gfs2/log.c @@ -657,7 +657,7 @@ static void log_write_header(struct gfs2_sbd *sdp, u32 flags) struct gfs2_log_header *lh; unsigned int tail; u32 hash; - int rw = WRITE_FLUSH_FUA | REQ_META; + int op_flags = WRITE_FLUSH_FUA | REQ_META; struct page *page = mempool_alloc(gfs2_page_pool, GFP_NOIO); enum gfs2_freeze_state state = atomic_read(&sdp->sd_freeze_state); lh = page_address(page); @@ -682,12 +682,12 @@ static void log_write_header(struct gfs2_sbd *sdp, u32 flags) if (test_bit(SDF_NOBARRIERS, &sdp->sd_flags)) { gfs2_ordered_wait(sdp); log_flush_wait(sdp); - rw = WRITE_SYNC | REQ_META | REQ_PRIO; + op_flags = WRITE_SYNC | REQ_META | REQ_PRIO; } sdp->sd_log_idle = (tail == sdp->sd_log_flush_head); gfs2_log_write_page(sdp, page); - gfs2_log_flush_bio(sdp, rw); + gfs2_log_flush_bio(sdp, REQ_OP_WRITE, op_flags); log_flush_wait(sdp); if (sdp->sd_log_tail != tail) @@ -738,7 +738,7 @@ void gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl, gfs2_ordered_write(sdp); lops_before_commit(sdp, tr); - gfs2_log_flush_bio(sdp, WRITE); + gfs2_log_flush_bio(sdp, REQ_OP_WRITE, 0); if (sdp->sd_log_head != sdp->sd_log_flush_head) { log_flush_wait(sdp); diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index ce28242..c1099b4 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -230,17 +230,19 @@ static void gfs2_end_log_write(struct bio *bio) /** * gfs2_log_flush_bio - Submit any pending log bio * @sdp: The superblock - * @rw: The rw flags + * @op: REQ_OP + * @op_flags: rq_flag_bits * * Submit any pending part-built or full bio to the block device. If * there is no pending bio, then this is a no-op. */ -void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int rw) +void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int op, int op_flags) { if (sdp->sd_log_bio) { atomic_inc(&sdp->sd_log_in_flight); - sdp->sd_log_bio->bi_rw = rw; + sdp->sd_log_bio->bi_op = op; + sdp->sd_log_bio->bi_rw = op_flags; submit_bio(sdp->sd_log_bio); sdp->sd_log_bio = NULL; } @@ -300,7 +302,7 @@ static struct bio *gfs2_log_get_bio(struct gfs2_sbd *sdp, u64 blkno) nblk >>= sdp->sd_fsb2bb_shift; if (blkno == nblk) return bio; - gfs2_log_flush_bio(sdp, WRITE); + gfs2_log_flush_bio(sdp, REQ_OP_WRITE, 0); } return gfs2_log_alloc_bio(sdp, blkno); @@ -329,7 +331,7 @@ static void gfs2_log_write(struct gfs2_sbd *sdp, struct page *page, bio = gfs2_log_get_bio(sdp, blkno); ret = bio_add_page(bio, page, size, offset); if (ret == 0) { - gfs2_log_flush_bio(sdp, WRITE); + gfs2_log_flush_bio(sdp, REQ_OP_WRITE, 0); bio = gfs2_log_alloc_bio(sdp, blkno); ret = bio_add_page(bio, page, size, offset); WARN_ON(ret == 0); diff --git a/fs/gfs2/lops.h b/fs/gfs2/lops.h index a65a7ba..e529f53 100644 --- a/fs/gfs2/lops.h +++ b/fs/gfs2/lops.h @@ -27,7 +27,7 @@ extern const struct gfs2_log_operations gfs2_databuf_lops; extern const struct gfs2_log_operations *gfs2_log_ops[]; extern void gfs2_log_write_page(struct gfs2_sbd *sdp, struct page *page); -extern void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int rw); +extern void gfs2_log_flush_bio(struct gfs2_sbd *sdp, int op, int op_flags); extern void gfs2_pin(struct gfs2_sbd *sdp, struct buffer_head *bh); static inline unsigned int buf_limit(struct gfs2_sbd *sdp) diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index 96c8140..f790f38 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -213,7 +213,8 @@ static void gfs2_meta_read_endio(struct bio *bio) * Submit several consecutive buffer head I/O requests as a single bio I/O * request. (See submit_bh_wbc.) */ -static void gfs2_submit_bhs(int rw, struct buffer_head *bhs[], int num) +static void gfs2_submit_bhs(int op, int op_flags, struct buffer_head *bhs[], + int num) { struct buffer_head *bh = bhs[0]; struct bio *bio; @@ -230,7 +231,8 @@ static void gfs2_submit_bhs(int rw, struct buffer_head *bhs[], int num) bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh)); } bio->bi_end_io = gfs2_meta_read_endio; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; submit_bio(bio); } @@ -281,7 +283,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags, } } - gfs2_submit_bhs(READ_SYNC | REQ_META | REQ_PRIO, bhs, num); + gfs2_submit_bhs(REQ_OP_READ, READ_SYNC | REQ_META | REQ_PRIO, bhs, num); if (!(flags & DIO_WAIT)) return 0; diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c index 4872a61..f3e1e0e 100644 --- a/fs/gfs2/ops_fstype.c +++ b/fs/gfs2/ops_fstype.c @@ -246,6 +246,7 @@ static int gfs2_read_super(struct gfs2_sbd *sdp, sector_t sector, int silent) bio->bi_end_io = end_bio_io_page; bio->bi_private = page; + bio->bi_op = REQ_OP_READ; bio->bi_rw = READ_SYNC | REQ_META; submit_bio(bio); wait_on_page_locked(page); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9505629E2C for ; Mon, 11 Jan 2016 14:22:13 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 583E030405F for ; Mon, 11 Jan 2016 12:22:13 -0800 (PST) X-ASG-Debug-ID: 1452543731-04bdf06f6b7dad0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 5KxVKdixW0o18ofS (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:12 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id B55B2AACF2; Mon, 11 Jan 2016 20:22:11 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEg008385; Mon, 11 Jan 2016 15:22:09 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 13/35] xfs: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:11 -0600 X-ASG-Orig-Subj: [PATCH 13/35] xfs: set bi_op to REQ_OP Message-Id: <1452543693-4440-14-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543732 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has xfs set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. Signed-off-by: Mike Christie --- fs/xfs/xfs_aops.c | 3 ++- fs/xfs/xfs_buf.c | 27 +++++++++++++++------------ 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 5525a1b..25673e0 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -382,7 +382,8 @@ xfs_submit_ioend_bio( atomic_inc(&ioend->io_remaining); bio->bi_private = ioend; bio->bi_end_io = xfs_end_bio; - bio->bi_rw = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + bio->bi_op = REQ_OP_WRITE; + bio->bi_rw = WB_SYNC_ALL ? WRITE_SYNC : 0; submit_bio(bio); } diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 9fd85d5..de35a3c 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1124,7 +1124,8 @@ xfs_buf_ioapply_map( int map, int *buf_offset, int *count, - int rw) + int op, + int op_flags) { int page_index; int total_nr_pages = bp->b_page_count; @@ -1163,7 +1164,8 @@ next_chunk: bio->bi_iter.bi_sector = sector; bio->bi_end_io = xfs_buf_bio_end_io; bio->bi_private = bp; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; for (; size && nr_pages; nr_pages--, page_index++) { int rbytes, nbytes = PAGE_SIZE - offset; @@ -1207,7 +1209,8 @@ _xfs_buf_ioapply( struct xfs_buf *bp) { struct blk_plug plug; - int rw; + int op; + int op_flags = 0; int offset; int size; int i; @@ -1226,14 +1229,13 @@ _xfs_buf_ioapply( bp->b_ioend_wq = bp->b_target->bt_mount->m_buf_workqueue; if (bp->b_flags & XBF_WRITE) { + op = REQ_OP_WRITE; if (bp->b_flags & XBF_SYNCIO) - rw = WRITE_SYNC; - else - rw = WRITE; + op_flags = WRITE_SYNC; if (bp->b_flags & XBF_FUA) - rw |= REQ_FUA; + op_flags |= REQ_FUA; if (bp->b_flags & XBF_FLUSH) - rw |= REQ_FLUSH; + op_flags |= REQ_FLUSH; /* * Run the write verifier callback function if it exists. If @@ -1263,13 +1265,14 @@ _xfs_buf_ioapply( } } } else if (bp->b_flags & XBF_READ_AHEAD) { - rw = READA; + op = REQ_OP_READ; + op_flags = REQ_RAHEAD; } else { - rw = READ; + op = REQ_OP_READ; } /* we only use the buffer cache for meta-data */ - rw |= REQ_META; + op_flags |= REQ_META; /* * Walk all the vectors issuing IO on them. Set up the initial offset @@ -1281,7 +1284,7 @@ _xfs_buf_ioapply( size = BBTOB(bp->b_io_length); blk_start_plug(&plug); for (i = 0; i < bp->b_map_count; i++) { - xfs_buf_ioapply_map(bp, i, &offset, &size, rw); + xfs_buf_ioapply_map(bp, i, &offset, &size, op, op_flags); if (bp->b_error) break; if (size <= 0) -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:15 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id BD6C529E2C for ; Mon, 11 Jan 2016 14:22:15 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 7BEF48F804B for ; Mon, 11 Jan 2016 12:22:15 -0800 (PST) X-ASG-Debug-ID: 1452543734-04bdf06f6c7dae0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 0iNQI24UV9z4YwGp (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:14 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id F2D55C0BF2B2; Mon, 11 Jan 2016 20:22:13 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEh008385; Mon, 11 Jan 2016 15:22:11 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 14/35] hfsplus: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:12 -0600 X-ASG-Orig-Subj: [PATCH 14/35] hfsplus: set bi_op to REQ_OP Message-Id: <1452543693-4440-15-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543734 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has hfsplus set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- fs/hfsplus/hfsplus_fs.h | 2 +- fs/hfsplus/part_tbl.c | 5 +++-- fs/hfsplus/super.c | 6 ++++-- fs/hfsplus/wrapper.c | 15 +++++++++------ 4 files changed, 17 insertions(+), 11 deletions(-) diff --git a/fs/hfsplus/hfsplus_fs.h b/fs/hfsplus/hfsplus_fs.h index f91a1fa..80154aa 100644 --- a/fs/hfsplus/hfsplus_fs.h +++ b/fs/hfsplus/hfsplus_fs.h @@ -525,7 +525,7 @@ int hfsplus_compare_dentry(const struct dentry *parent, /* wrapper.c */ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, void *buf, - void **data, int rw); + void **data, int op, int op_flags); int hfsplus_read_wrapper(struct super_block *sb); /* time macros */ diff --git a/fs/hfsplus/part_tbl.c b/fs/hfsplus/part_tbl.c index eb355d8..63164eb 100644 --- a/fs/hfsplus/part_tbl.c +++ b/fs/hfsplus/part_tbl.c @@ -112,7 +112,8 @@ static int hfs_parse_new_pmap(struct super_block *sb, void *buf, if ((u8 *)pm - (u8 *)buf >= buf_size) { res = hfsplus_submit_bio(sb, *part_start + HFS_PMAP_BLK + i, - buf, (void **)&pm, READ); + buf, (void **)&pm, REQ_OP_READ, + 0); if (res) return res; } @@ -136,7 +137,7 @@ int hfs_part_find(struct super_block *sb, return -ENOMEM; res = hfsplus_submit_bio(sb, *part_start + HFS_PMAP_BLK, - buf, &data, READ); + buf, &data, REQ_OP_READ, 0); if (res) goto out; diff --git a/fs/hfsplus/super.c b/fs/hfsplus/super.c index 5d54490..01cf313 100644 --- a/fs/hfsplus/super.c +++ b/fs/hfsplus/super.c @@ -219,7 +219,8 @@ static int hfsplus_sync_fs(struct super_block *sb, int wait) error2 = hfsplus_submit_bio(sb, sbi->part_start + HFSPLUS_VOLHEAD_SECTOR, - sbi->s_vhdr_buf, NULL, WRITE_SYNC); + sbi->s_vhdr_buf, NULL, REQ_OP_WRITE, + WRITE_SYNC); if (!error) error = error2; if (!write_backup) @@ -227,7 +228,8 @@ static int hfsplus_sync_fs(struct super_block *sb, int wait) error2 = hfsplus_submit_bio(sb, sbi->part_start + sbi->sect_count - 2, - sbi->s_backup_vhdr_buf, NULL, WRITE_SYNC); + sbi->s_backup_vhdr_buf, NULL, REQ_OP_WRITE, + WRITE_SYNC); if (!error) error2 = error; out: diff --git a/fs/hfsplus/wrapper.c b/fs/hfsplus/wrapper.c index d026bb3..c5c916d 100644 --- a/fs/hfsplus/wrapper.c +++ b/fs/hfsplus/wrapper.c @@ -30,7 +30,8 @@ struct hfsplus_wd { * @sector: block to read or write, for blocks of HFSPLUS_SECTOR_SIZE bytes * @buf: buffer for I/O * @data: output pointer for location of requested data - * @rw: direction of I/O + * @op: direction of I/O + * @op_flags: request op flags * * The unit of I/O is hfsplus_min_io_size(sb), which may be bigger than * HFSPLUS_SECTOR_SIZE, and @buf must be sized accordingly. On reads @@ -44,7 +45,7 @@ struct hfsplus_wd { * will work correctly. */ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, - void *buf, void **data, int rw) + void *buf, void **data, int op, int op_flags) { struct bio *bio; int ret = 0; @@ -65,9 +66,10 @@ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, bio = bio_alloc(GFP_NOIO, 1); bio->bi_iter.bi_sector = sector; bio->bi_bdev = sb->s_bdev; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; - if (!(rw & WRITE) && data) + if (op != WRITE && data) *data = (u8 *)buf + offset; while (io_size > 0) { @@ -182,7 +184,7 @@ int hfsplus_read_wrapper(struct super_block *sb) reread: error = hfsplus_submit_bio(sb, part_start + HFSPLUS_VOLHEAD_SECTOR, sbi->s_vhdr_buf, (void **)&sbi->s_vhdr, - READ); + REQ_OP_READ, 0); if (error) goto out_free_backup_vhdr; @@ -214,7 +216,8 @@ reread: error = hfsplus_submit_bio(sb, part_start + part_size - 2, sbi->s_backup_vhdr_buf, - (void **)&sbi->s_backup_vhdr, READ); + (void **)&sbi->s_backup_vhdr, REQ_OP_READ, + 0); if (error) goto out_free_backup_vhdr; -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:17 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id F1DE329E37 for ; Mon, 11 Jan 2016 14:22:17 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id C5FA2304053 for ; Mon, 11 Jan 2016 12:22:17 -0800 (PST) X-ASG-Debug-ID: 1452543736-04bdf06f6b7dae0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id VrFiYExAqpTmFmVT (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:16 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 4AE7BA04BC; Mon, 11 Jan 2016 20:22:16 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEi008385; Mon, 11 Jan 2016 15:22:14 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 15/35] mpage: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:13 -0600 X-ASG-Orig-Subj: [PATCH 15/35] mpage: set bi_op to REQ_OP Message-Id: <1452543693-4440-16-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543736 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the mpage.c code set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. I have run xfstest with xfs, but I am not sure if I have stressed these code paths well. Signed-off-by: Mike Christie --- fs/mpage.c | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/fs/mpage.c b/fs/mpage.c index d9adde2..006a3ef 100644 --- a/fs/mpage.c +++ b/fs/mpage.c @@ -55,11 +55,12 @@ static void mpage_end_io(struct bio *bio) bio_put(bio); } -static struct bio *mpage_bio_submit(int rw, struct bio *bio) +static struct bio *mpage_bio_submit(int op, int op_flags, struct bio *bio) { bio->bi_end_io = mpage_end_io; - bio->bi_rw = rw; - guard_bio_eod(rw, bio); + bio->bi_op = op; + bio->bi_rw = op_flags; + guard_bio_eod(op, bio); submit_bio(bio); return NULL; } @@ -269,7 +270,7 @@ do_mpage_readpage(struct bio *bio, struct page *page, unsigned nr_pages, * This page will go to BIO. Do we need to send this BIO off first? */ if (bio && (*last_block_in_bio != blocks[0] - 1)) - bio = mpage_bio_submit(READ, bio); + bio = mpage_bio_submit(REQ_OP_READ, 0, bio); alloc_new: if (bio == NULL) { @@ -286,7 +287,7 @@ alloc_new: length = first_hole << blkbits; if (bio_add_page(bio, page, length, 0) < length) { - bio = mpage_bio_submit(READ, bio); + bio = mpage_bio_submit(REQ_OP_READ, 0, bio); goto alloc_new; } @@ -294,7 +295,7 @@ alloc_new: nblocks = map_bh->b_size >> blkbits; if ((buffer_boundary(map_bh) && relative_block == nblocks) || (first_hole != blocks_per_page)) - bio = mpage_bio_submit(READ, bio); + bio = mpage_bio_submit(REQ_OP_READ, 0, bio); else *last_block_in_bio = blocks[blocks_per_page - 1]; out: @@ -302,7 +303,7 @@ out: confused: if (bio) - bio = mpage_bio_submit(READ, bio); + bio = mpage_bio_submit(REQ_OP_READ, 0, bio); if (!PageUptodate(page)) block_read_full_page(page, get_block); else @@ -384,7 +385,7 @@ mpage_readpages(struct address_space *mapping, struct list_head *pages, } BUG_ON(!list_empty(pages)); if (bio) - mpage_bio_submit(READ, bio); + mpage_bio_submit(REQ_OP_READ, 0, bio); return 0; } EXPORT_SYMBOL(mpage_readpages); @@ -405,7 +406,7 @@ int mpage_readpage(struct page *page, get_block_t get_block) bio = do_mpage_readpage(bio, page, 1, &last_block_in_bio, &map_bh, &first_logical_block, get_block, gfp); if (bio) - mpage_bio_submit(READ, bio); + mpage_bio_submit(REQ_OP_READ, 0, bio); return 0; } EXPORT_SYMBOL(mpage_readpage); @@ -486,7 +487,7 @@ static int __mpage_writepage(struct page *page, struct writeback_control *wbc, struct buffer_head map_bh; loff_t i_size = i_size_read(inode); int ret = 0; - int wr = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : WRITE); + int op_flags = (wbc->sync_mode == WB_SYNC_ALL ? WRITE_SYNC : 0); if (page_has_buffers(page)) { struct buffer_head *head = page_buffers(page); @@ -595,7 +596,7 @@ page_is_mapped: * This page will go to BIO. Do we need to send this BIO off first? */ if (bio && mpd->last_block_in_bio != blocks[0] - 1) - bio = mpage_bio_submit(wr, bio); + bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); alloc_new: if (bio == NULL) { @@ -622,7 +623,7 @@ alloc_new: wbc_account_io(wbc, page, PAGE_SIZE); length = first_unmapped << blkbits; if (bio_add_page(bio, page, length, 0) < length) { - bio = mpage_bio_submit(wr, bio); + bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); goto alloc_new; } @@ -632,7 +633,7 @@ alloc_new: set_page_writeback(page); unlock_page(page); if (boundary || (first_unmapped != blocks_per_page)) { - bio = mpage_bio_submit(wr, bio); + bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); if (boundary_block) { write_boundary_block(boundary_bdev, boundary_block, 1 << blkbits); @@ -644,7 +645,7 @@ alloc_new: confused: if (bio) - bio = mpage_bio_submit(wr, bio); + bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); if (mpd->use_writepage) { ret = mapping->a_ops->writepage(page, wbc); @@ -701,9 +702,9 @@ mpage_writepages(struct address_space *mapping, ret = write_cache_pages(mapping, wbc, __mpage_writepage, &mpd); if (mpd.bio) { - int wr = (wbc->sync_mode == WB_SYNC_ALL ? - WRITE_SYNC : WRITE); - mpage_bio_submit(wr, mpd.bio); + int op_flags = (wbc->sync_mode == WB_SYNC_ALL ? + WRITE_SYNC : 0); + mpage_bio_submit(REQ_OP_WRITE, op_flags, mpd.bio); } } blk_finish_plug(&plug); @@ -722,9 +723,9 @@ int mpage_writepage(struct page *page, get_block_t get_block, }; int ret = __mpage_writepage(page, wbc, &mpd); if (mpd.bio) { - int wr = (wbc->sync_mode == WB_SYNC_ALL ? - WRITE_SYNC : WRITE); - mpage_bio_submit(wr, mpd.bio); + int op_flags = (wbc->sync_mode == WB_SYNC_ALL ? + WRITE_SYNC : 0); + mpage_bio_submit(REQ_OP_WRITE, op_flags, mpd.bio); } return ret; } -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:20 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 597E929E09 for ; Mon, 11 Jan 2016 14:22:20 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 339AB8F804B for ; Mon, 11 Jan 2016 12:22:20 -0800 (PST) X-ASG-Debug-ID: 1452543738-04cb6c75de79050001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 7F0kVTeRCAcUn0Aa (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:19 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id A11AA6E799; Mon, 11 Jan 2016 20:22:18 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEj008385; Mon, 11 Jan 2016 15:22:16 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 16/35] nilfs: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:14 -0600 X-ASG-Orig-Subj: [PATCH 16/35] nilfs: set bi_op to REQ_OP Message-Id: <1452543693-4440-17-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543739 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has nilfs set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- fs/nilfs2/segbuf.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/fs/nilfs2/segbuf.c b/fs/nilfs2/segbuf.c index 7666f1d..7b13e14 100644 --- a/fs/nilfs2/segbuf.c +++ b/fs/nilfs2/segbuf.c @@ -350,7 +350,8 @@ static void nilfs_end_bio_write(struct bio *bio) } static int nilfs_segbuf_submit_bio(struct nilfs_segment_buffer *segbuf, - struct nilfs_write_info *wi, int mode) + struct nilfs_write_info *wi, int mode, + int mode_flags) { struct bio *bio = wi->bio; int err; @@ -368,7 +369,8 @@ static int nilfs_segbuf_submit_bio(struct nilfs_segment_buffer *segbuf, bio->bi_end_io = nilfs_end_bio_write; bio->bi_private = segbuf; - bio->bi_rw = mode; + bio->bi_op = mode; + bio->bi_rw = mode_flags; submit_bio(bio); segbuf->sb_nbio++; @@ -442,7 +444,7 @@ static int nilfs_segbuf_submit_bh(struct nilfs_segment_buffer *segbuf, return 0; } /* bio is FULL */ - err = nilfs_segbuf_submit_bio(segbuf, wi, mode); + err = nilfs_segbuf_submit_bio(segbuf, wi, mode, 0); /* never submit current bh */ if (likely(!err)) goto repeat; @@ -466,19 +468,19 @@ static int nilfs_segbuf_write(struct nilfs_segment_buffer *segbuf, { struct nilfs_write_info wi; struct buffer_head *bh; - int res = 0, rw = WRITE; + int res = 0; wi.nilfs = nilfs; nilfs_segbuf_prepare_write(segbuf, &wi); list_for_each_entry(bh, &segbuf->sb_segsum_buffers, b_assoc_buffers) { - res = nilfs_segbuf_submit_bh(segbuf, &wi, bh, rw); + res = nilfs_segbuf_submit_bh(segbuf, &wi, bh, REQ_OP_WRITE); if (unlikely(res)) goto failed_bio; } list_for_each_entry(bh, &segbuf->sb_payload_buffers, b_assoc_buffers) { - res = nilfs_segbuf_submit_bh(segbuf, &wi, bh, rw); + res = nilfs_segbuf_submit_bh(segbuf, &wi, bh, REQ_OP_WRITE); if (unlikely(res)) goto failed_bio; } @@ -488,8 +490,8 @@ static int nilfs_segbuf_write(struct nilfs_segment_buffer *segbuf, * Last BIO is always sent through the following * submission. */ - rw |= REQ_SYNC; - res = nilfs_segbuf_submit_bio(segbuf, &wi, rw); + res = nilfs_segbuf_submit_bio(segbuf, &wi, REQ_OP_WRITE, + REQ_SYNC); } failed_bio: -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:22 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 832B129E08 for ; Mon, 11 Jan 2016 14:22:22 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 1FD22AC005 for ; Mon, 11 Jan 2016 12:22:22 -0800 (PST) X-ASG-Debug-ID: 1452543741-04bdf06f6b7db00001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id BWcBpqQOpOaKnbNi (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:21 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id F32F183F6E; Mon, 11 Jan 2016 20:22:20 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEk008385; Mon, 11 Jan 2016 15:22:18 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 17/35] ocfs2: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:15 -0600 X-ASG-Orig-Subj: [PATCH 17/35] ocfs2: set bi_op to REQ_OP Message-Id: <1452543693-4440-18-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543741 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has ocfs2 set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- fs/ocfs2/cluster/heartbeat.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c index 119bd85..271ef2a 100644 --- a/fs/ocfs2/cluster/heartbeat.c +++ b/fs/ocfs2/cluster/heartbeat.c @@ -391,7 +391,8 @@ static void o2hb_bio_end_io(struct bio *bio) static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, struct o2hb_bio_wait_ctxt *wc, unsigned int *current_slot, - unsigned int max_slots, int rw) + unsigned int max_slots, int op, + int op_flags) { int len, current_page; unsigned int vec_len, vec_start; @@ -417,7 +418,8 @@ static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, bio->bi_bdev = reg->hr_bdev; bio->bi_private = wc; bio->bi_end_io = o2hb_bio_end_io; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; vec_start = (cs << bits) % PAGE_CACHE_SIZE; while(cs < max_slots) { @@ -454,7 +456,7 @@ static int o2hb_read_slots(struct o2hb_region *reg, while(current_slot < max_slots) { bio = o2hb_setup_one_bio(reg, &wc, ¤t_slot, max_slots, - READ); + REQ_OP_READ, 0); if (IS_ERR(bio)) { status = PTR_ERR(bio); mlog_errno(status); @@ -486,7 +488,8 @@ static int o2hb_issue_node_write(struct o2hb_region *reg, slot = o2nm_this_node(); - bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1, WRITE_SYNC); + bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1, REQ_OP_WRITE, + WRITE_SYNC); if (IS_ERR(bio)) { status = PTR_ERR(bio); mlog_errno(status); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:25 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 2853129E43 for ; Mon, 11 Jan 2016 14:22:25 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 190778F804B for ; Mon, 11 Jan 2016 12:22:25 -0800 (PST) X-ASG-Debug-ID: 1452543743-04cbb05f7c97a60001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id nP6mkUiCsVN6WPfS (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:24 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 3A9BBA10CD; Mon, 11 Jan 2016 20:22:23 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEl008385; Mon, 11 Jan 2016 15:22:21 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 18/35] pm: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:16 -0600 X-ASG-Orig-Subj: [PATCH 18/35] pm: set bi_op to REQ_OP Message-Id: <1452543693-4440-19-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543743 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the pm swap code set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- kernel/power/swap.c | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 4d050eb..adbcb1b 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -250,7 +250,7 @@ static void hib_end_io(struct bio *bio) bio_put(bio); } -static int hib_submit_io(int rw, pgoff_t page_off, void *addr, +static int hib_submit_io(int op, int op_flags, pgoff_t page_off, void *addr, struct hib_bio_batch *hb) { struct page *page = virt_to_page(addr); @@ -260,7 +260,8 @@ static int hib_submit_io(int rw, pgoff_t page_off, void *addr, bio = bio_alloc(__GFP_RECLAIM | __GFP_HIGH, 1); bio->bi_iter.bi_sector = page_off * (PAGE_SIZE >> 9); bio->bi_bdev = hib_resume_bdev; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) { printk(KERN_ERR "PM: Adding page to bio failed at %llu\n", @@ -296,7 +297,8 @@ static int mark_swapfiles(struct swap_map_handle *handle, unsigned int flags) { int error; - hib_submit_io(READ_SYNC, swsusp_resume_block, swsusp_header, NULL); + hib_submit_io(REQ_OP_READ, READ_SYNC, swsusp_resume_block, + swsusp_header, NULL); if (!memcmp("SWAP-SPACE",swsusp_header->sig, 10) || !memcmp("SWAPSPACE2",swsusp_header->sig, 10)) { memcpy(swsusp_header->orig_sig,swsusp_header->sig, 10); @@ -305,8 +307,8 @@ static int mark_swapfiles(struct swap_map_handle *handle, unsigned int flags) swsusp_header->flags = flags; if (flags & SF_CRC32_MODE) swsusp_header->crc32 = handle->crc32; - error = hib_submit_io(WRITE_SYNC, swsusp_resume_block, - swsusp_header, NULL); + error = hib_submit_io(REQ_OP_WRITE, WRITE_SYNC, + swsusp_resume_block, swsusp_header, NULL); } else { printk(KERN_ERR "PM: Swap header not found!\n"); error = -ENODEV; @@ -379,7 +381,7 @@ static int write_page(void *buf, sector_t offset, struct hib_bio_batch *hb) } else { src = buf; } - return hib_submit_io(WRITE_SYNC, offset, src, hb); + return hib_submit_io(REQ_OP_WRITE, WRITE_SYNC, offset, src, hb); } static void release_swap_writer(struct swap_map_handle *handle) @@ -982,7 +984,8 @@ static int get_swap_reader(struct swap_map_handle *handle, return -ENOMEM; } - error = hib_submit_io(READ_SYNC, offset, tmp->map, NULL); + error = hib_submit_io(REQ_OP_READ, READ_SYNC, offset, + tmp->map, NULL); if (error) { release_swap_reader(handle); return error; @@ -1006,7 +1009,7 @@ static int swap_read_page(struct swap_map_handle *handle, void *buf, offset = handle->cur->entries[handle->k]; if (!offset) return -EFAULT; - error = hib_submit_io(READ_SYNC, offset, buf, hb); + error = hib_submit_io(REQ_OP_READ, READ_SYNC, offset, buf, hb); if (error) return error; if (++handle->k >= MAP_PAGE_ENTRIES) { @@ -1508,7 +1511,8 @@ int swsusp_check(void) if (!IS_ERR(hib_resume_bdev)) { set_blocksize(hib_resume_bdev, PAGE_SIZE); clear_page(swsusp_header); - error = hib_submit_io(READ_SYNC, swsusp_resume_block, + error = hib_submit_io(REQ_OP_READ, READ_SYNC, + swsusp_resume_block, swsusp_header, NULL); if (error) goto put; @@ -1516,7 +1520,8 @@ int swsusp_check(void) if (!memcmp(HIBERNATE_SIG, swsusp_header->sig, 10)) { memcpy(swsusp_header->sig, swsusp_header->orig_sig, 10); /* Reset swap signature now */ - error = hib_submit_io(WRITE_SYNC, swsusp_resume_block, + error = hib_submit_io(REQ_OP_WRITE, WRITE_SYNC, + swsusp_resume_block, swsusp_header, NULL); } else { error = -EINVAL; @@ -1560,10 +1565,12 @@ int swsusp_unmark(void) { int error; - hib_submit_io(READ_SYNC, swsusp_resume_block, swsusp_header, NULL); + hib_submit_io(REQ_OP_READ, READ_SYNC, swsusp_resume_block, + swsusp_header, NULL); if (!memcmp(HIBERNATE_SIG,swsusp_header->sig, 10)) { memcpy(swsusp_header->sig,swsusp_header->orig_sig, 10); - error = hib_submit_io(WRITE_SYNC, swsusp_resume_block, + error = hib_submit_io(REQ_OP_WRITE, WRITE_SYNC, + swsusp_resume_block, swsusp_header, NULL); } else { printk(KERN_ERR "PM: Cannot find swsusp signature!\n"); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:29 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9826D29E40 for ; Mon, 11 Jan 2016 14:22:29 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 5E5D8304059 for ; Mon, 11 Jan 2016 12:22:29 -0800 (PST) X-ASG-Debug-ID: 1452543748-04bdf06f6b7db10001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 2hpyj14i3czHhTB8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:28 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id E1CD18DFEA; Mon, 11 Jan 2016 20:22:27 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEn008385; Mon, 11 Jan 2016 15:22:25 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 20/35] dm: pass dm stats data dir instead of bi_rw Date: Mon, 11 Jan 2016 14:21:18 -0600 X-ASG-Orig-Subj: [PATCH 20/35] dm: pass dm stats data dir instead of bi_rw Message-Id: <1452543693-4440-21-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543748 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie It looks like dm stats cares about the data direction (READ vs WRITE) and does not need the bio/request flags. Commands like REQ_FLUSH, REQ_DISCARD and REQ_WRITE_SAME are currently always set with REQ_WRITE, so the extra check for REQ_DISCARD in dm_stats_account_io is not needed. This patch has it use the bio and request data_dir helpers instead of accessing the bi_rw/cmd_flags directly. This makes the next patches that remove the operation from the cmd_flags and bi_rw easier, because we will no longer have the REQ_WRITE bit set for operations like discards. This patch is compile tested only. v2: 1. Merged Mike Snitzer's fixes to pass in int instead of unsigned long. 2. Fix 80 char col issues. Signed-off-by: Mike Christie --- drivers/md/dm-stats.c | 9 ++++----- drivers/md/dm.c | 21 ++++++++++++--------- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/drivers/md/dm-stats.c b/drivers/md/dm-stats.c index 8289804..4fba26c 100644 --- a/drivers/md/dm-stats.c +++ b/drivers/md/dm-stats.c @@ -514,11 +514,10 @@ static void dm_stat_round(struct dm_stat *s, struct dm_stat_shared *shared, } static void dm_stat_for_entry(struct dm_stat *s, size_t entry, - unsigned long bi_rw, sector_t len, + int idx, sector_t len, struct dm_stats_aux *stats_aux, bool end, unsigned long duration_jiffies) { - unsigned long idx = bi_rw & REQ_WRITE; struct dm_stat_shared *shared = &s->stat_shared[entry]; struct dm_stat_percpu *p; @@ -584,7 +583,7 @@ static void dm_stat_for_entry(struct dm_stat *s, size_t entry, #endif } -static void __dm_stat_bio(struct dm_stat *s, unsigned long bi_rw, +static void __dm_stat_bio(struct dm_stat *s, int bi_rw, sector_t bi_sector, sector_t end_sector, bool end, unsigned long duration_jiffies, struct dm_stats_aux *stats_aux) @@ -645,8 +644,8 @@ void dm_stats_account_io(struct dm_stats *stats, unsigned long bi_rw, last = raw_cpu_ptr(stats->last); stats_aux->merged = (bi_sector == (ACCESS_ONCE(last->last_sector) && - ((bi_rw & (REQ_WRITE | REQ_DISCARD)) == - (ACCESS_ONCE(last->last_rw) & (REQ_WRITE | REQ_DISCARD))) + ((bi_rw == WRITE) == + (ACCESS_ONCE(last->last_rw) == WRITE)) )); ACCESS_ONCE(last->last_sector) = end_sector; ACCESS_ONCE(last->last_rw) = bi_rw; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 1777c9c..b882e56c 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -686,8 +686,9 @@ static void start_io_acct(struct dm_io *io) atomic_inc_return(&md->pending[rw])); if (unlikely(dm_stats_used(&md->stats))) - dm_stats_account_io(&md->stats, bio->bi_rw, bio->bi_iter.bi_sector, - bio_sectors(bio), false, 0, &io->stats_aux); + dm_stats_account_io(&md->stats, bio_data_dir(bio), + bio->bi_iter.bi_sector, bio_sectors(bio), + false, 0, &io->stats_aux); } static void end_io_acct(struct dm_io *io) @@ -701,8 +702,9 @@ static void end_io_acct(struct dm_io *io) generic_end_io_acct(rw, &dm_disk(md)->part0, io->start_time); if (unlikely(dm_stats_used(&md->stats))) - dm_stats_account_io(&md->stats, bio->bi_rw, bio->bi_iter.bi_sector, - bio_sectors(bio), true, duration, &io->stats_aux); + dm_stats_account_io(&md->stats, bio_data_dir(bio), + bio->bi_iter.bi_sector, bio_sectors(bio), + true, duration, &io->stats_aux); /* * After this is decremented the bio must not be touched if it is @@ -1084,9 +1086,9 @@ static void rq_end_stats(struct mapped_device *md, struct request *orig) if (unlikely(dm_stats_used(&md->stats))) { struct dm_rq_target_io *tio = tio_from_request(orig); tio->duration_jiffies = jiffies - tio->duration_jiffies; - dm_stats_account_io(&md->stats, orig->cmd_flags, blk_rq_pos(orig), - tio->n_sectors, true, tio->duration_jiffies, - &tio->stats_aux); + dm_stats_account_io(&md->stats, rq_data_dir(orig), + blk_rq_pos(orig), tio->n_sectors, true, + tio->duration_jiffies, &tio->stats_aux); } } @@ -2017,8 +2019,9 @@ static void dm_start_request(struct mapped_device *md, struct request *orig) struct dm_rq_target_io *tio = tio_from_request(orig); tio->duration_jiffies = jiffies; tio->n_sectors = blk_rq_sectors(orig); - dm_stats_account_io(&md->stats, orig->cmd_flags, blk_rq_pos(orig), - tio->n_sectors, false, 0, &tio->stats_aux); + dm_stats_account_io(&md->stats, rq_data_dir(orig), + blk_rq_pos(orig), tio->n_sectors, false, 0, + &tio->stats_aux); } /* -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:29 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 401F429DFD for ; Mon, 11 Jan 2016 14:22:29 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 9CE3FAC002 for ; Mon, 11 Jan 2016 12:22:28 -0800 (PST) X-ASG-Debug-ID: 1452543745-04cb6c75dd79060001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Qm6GH9Fwrpym7mxM (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:26 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 94F5F8E370; Mon, 11 Jan 2016 20:22:25 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEm008385; Mon, 11 Jan 2016 15:22:23 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 19/35] dm: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:17 -0600 X-ASG-Orig-Subj: [PATCH 19/35] dm: set bi_op to REQ_OP Message-Id: <1452543693-4440-20-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543746 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has dm set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. I did some basic dm tests, but I think this patch should be considered compile tested only. I have not tested all the dm targets and I did not stress every code path I have touched. Signed-off-by: Mike Christie --- drivers/md/dm-bufio.c | 8 +++--- drivers/md/dm-crypt.c | 1 + drivers/md/dm-io.c | 57 ++++++++++++++++++++++------------------- drivers/md/dm-kcopyd.c | 25 +++++++++--------- drivers/md/dm-log-writes.c | 6 ++--- drivers/md/dm-log.c | 5 ++-- drivers/md/dm-raid1.c | 11 +++++--- drivers/md/dm-snap-persistent.c | 24 +++++++++-------- drivers/md/dm-thin.c | 7 ++--- drivers/md/dm.c | 1 + include/linux/dm-io.h | 3 ++- 11 files changed, 82 insertions(+), 66 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index 9d3ee7f..b6055f2 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -574,7 +574,8 @@ static void use_dmio(struct dm_buffer *b, int rw, sector_t block, { int r; struct dm_io_request io_req = { - .bi_rw = rw, + .bi_op = rw, + .bi_op_flags = 0, .notify.fn = dmio_complete, .notify.context = b, .client = b->c->dm_io, @@ -634,7 +635,7 @@ static void use_inline_bio(struct dm_buffer *b, int rw, sector_t block, * the dm_buffer's inline bio is local to bufio. */ b->bio.bi_private = end_io; - b->bio.bi_rw = rw; + b->bio.bi_op = rw; /* * We assume that if len >= PAGE_SIZE ptr is page-aligned. @@ -1327,7 +1328,8 @@ EXPORT_SYMBOL_GPL(dm_bufio_write_dirty_buffers); int dm_bufio_issue_flush(struct dm_bufio_client *c) { struct dm_io_request io_req = { - .bi_rw = WRITE_FLUSH, + .bi_op = REQ_OP_WRITE, + .bi_op_flags = WRITE_FLUSH, .mem.type = DM_IO_KMEM, .mem.ptr.addr = NULL, .client = c->dm_io, diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 3147c8d..f466fec 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1133,6 +1133,7 @@ static void clone_init(struct dm_crypt_io *io, struct bio *clone) clone->bi_private = io; clone->bi_end_io = crypt_endio; clone->bi_bdev = cc->dev->bdev; + clone->bi_op = io->base_bio->bi_op; clone->bi_rw = io->base_bio->bi_rw; } diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c index 50f17e3..0f723ca 100644 --- a/drivers/md/dm-io.c +++ b/drivers/md/dm-io.c @@ -278,8 +278,9 @@ static void km_dp_init(struct dpages *dp, void *data) /*----------------------------------------------------------------- * IO routines that accept a list of pages. *---------------------------------------------------------------*/ -static void do_region(int rw, unsigned region, struct dm_io_region *where, - struct dpages *dp, struct io *io) +static void do_region(int op, int op_flags, unsigned region, + struct dm_io_region *where, struct dpages *dp, + struct io *io) { struct bio *bio; struct page *page; @@ -295,24 +296,25 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, /* * Reject unsupported discard and write same requests. */ - if (rw & REQ_DISCARD) + if (op == REQ_OP_DISCARD) special_cmd_max_sectors = q->limits.max_discard_sectors; - else if (rw & REQ_WRITE_SAME) + else if (op == REQ_OP_WRITE_SAME) special_cmd_max_sectors = q->limits.max_write_same_sectors; - if ((rw & (REQ_DISCARD | REQ_WRITE_SAME)) && special_cmd_max_sectors == 0) { + if ((op == REQ_OP_DISCARD || op == REQ_OP_WRITE_SAME) && + special_cmd_max_sectors == 0) { dec_count(io, region, -EOPNOTSUPP); return; } /* - * where->count may be zero if rw holds a flush and we need to + * where->count may be zero if op holds a flush and we need to * send a zero-sized flush. */ do { /* * Allocate a suitably sized-bio. */ - if ((rw & REQ_DISCARD) || (rw & REQ_WRITE_SAME)) + if ((op == REQ_OP_DISCARD) || (op == REQ_OP_WRITE_SAME)) num_bvecs = 1; else num_bvecs = min_t(int, BIO_MAX_PAGES, @@ -322,14 +324,15 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, bio->bi_iter.bi_sector = where->sector + (where->count - remaining); bio->bi_bdev = where->bdev; bio->bi_end_io = endio; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; store_io_and_region_in_bio(bio, io, region); - if (rw & REQ_DISCARD) { + if (op == REQ_OP_DISCARD) { num_sectors = min_t(sector_t, special_cmd_max_sectors, remaining); bio->bi_iter.bi_size = num_sectors << SECTOR_SHIFT; remaining -= num_sectors; - } else if (rw & REQ_WRITE_SAME) { + } else if (op == REQ_OP_WRITE_SAME) { /* * WRITE SAME only uses a single page. */ @@ -360,7 +363,7 @@ static void do_region(int rw, unsigned region, struct dm_io_region *where, } while (remaining); } -static void dispatch_io(int rw, unsigned int num_regions, +static void dispatch_io(int op, int op_flags, unsigned int num_regions, struct dm_io_region *where, struct dpages *dp, struct io *io, int sync) { @@ -370,7 +373,7 @@ static void dispatch_io(int rw, unsigned int num_regions, BUG_ON(num_regions > DM_IO_MAX_REGIONS); if (sync) - rw |= REQ_SYNC; + op_flags |= REQ_SYNC; /* * For multiple regions we need to be careful to rewind @@ -378,8 +381,8 @@ static void dispatch_io(int rw, unsigned int num_regions, */ for (i = 0; i < num_regions; i++) { *dp = old_pages; - if (where[i].count || (rw & REQ_FLUSH)) - do_region(rw, i, where + i, dp, io); + if (where[i].count || (op_flags & REQ_FLUSH)) + do_region(op, op_flags, i, where + i, dp, io); } /* @@ -403,13 +406,13 @@ static void sync_io_complete(unsigned long error, void *context) } static int sync_io(struct dm_io_client *client, unsigned int num_regions, - struct dm_io_region *where, int rw, struct dpages *dp, - unsigned long *error_bits) + struct dm_io_region *where, int op, int op_flags, + struct dpages *dp, unsigned long *error_bits) { struct io *io; struct sync_io sio; - if (num_regions > 1 && (rw & RW_MASK) != WRITE) { + if (num_regions > 1 && !op_is_write(op)) { WARN_ON(1); return -EIO; } @@ -426,7 +429,7 @@ static int sync_io(struct dm_io_client *client, unsigned int num_regions, io->vma_invalidate_address = dp->vma_invalidate_address; io->vma_invalidate_size = dp->vma_invalidate_size; - dispatch_io(rw, num_regions, where, dp, io, 1); + dispatch_io(op, op_flags, num_regions, where, dp, io, 1); wait_for_completion_io(&sio.wait); @@ -437,12 +440,12 @@ static int sync_io(struct dm_io_client *client, unsigned int num_regions, } static int async_io(struct dm_io_client *client, unsigned int num_regions, - struct dm_io_region *where, int rw, struct dpages *dp, - io_notify_fn fn, void *context) + struct dm_io_region *where, int op, int op_flags, + struct dpages *dp, io_notify_fn fn, void *context) { struct io *io; - if (num_regions > 1 && (rw & RW_MASK) != WRITE) { + if (num_regions > 1 && !op_is_write(op)) { WARN_ON(1); fn(1, context); return -EIO; @@ -458,7 +461,7 @@ static int async_io(struct dm_io_client *client, unsigned int num_regions, io->vma_invalidate_address = dp->vma_invalidate_address; io->vma_invalidate_size = dp->vma_invalidate_size; - dispatch_io(rw, num_regions, where, dp, io, 0); + dispatch_io(op, op_flags, num_regions, where, dp, io, 0); return 0; } @@ -481,7 +484,7 @@ static int dp_init(struct dm_io_request *io_req, struct dpages *dp, case DM_IO_VMA: flush_kernel_vmap_range(io_req->mem.ptr.vma, size); - if ((io_req->bi_rw & RW_MASK) == READ) { + if (io_req->bi_op == REQ_OP_READ) { dp->vma_invalidate_address = io_req->mem.ptr.vma; dp->vma_invalidate_size = size; } @@ -519,10 +522,12 @@ int dm_io(struct dm_io_request *io_req, unsigned num_regions, if (!io_req->notify.fn) return sync_io(io_req->client, num_regions, where, - io_req->bi_rw, &dp, sync_error_bits); + io_req->bi_op, io_req->bi_op_flags, &dp, + sync_error_bits); - return async_io(io_req->client, num_regions, where, io_req->bi_rw, - &dp, io_req->notify.fn, io_req->notify.context); + return async_io(io_req->client, num_regions, where, io_req->bi_op, + io_req->bi_op_flags, &dp, io_req->notify.fn, + io_req->notify.context); } EXPORT_SYMBOL(dm_io); diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c index 1452ed9..4182bc7d 100644 --- a/drivers/md/dm-kcopyd.c +++ b/drivers/md/dm-kcopyd.c @@ -465,10 +465,10 @@ static void complete_io(unsigned long error, void *context) io_job_finish(kc->throttle); if (error) { - if (job->rw & WRITE) - job->write_err |= error; - else + if (job->rw == READ) job->read_err = 1; + else + job->write_err |= error; if (!test_bit(DM_KCOPYD_IGNORE_ERROR, &job->flags)) { push(&kc->complete_jobs, job); @@ -477,13 +477,11 @@ static void complete_io(unsigned long error, void *context) } } - if (job->rw & WRITE) - push(&kc->complete_jobs, job); - - else { + if (job->rw == READ) { job->rw = WRITE; push(&kc->io_jobs, job); - } + } else + push(&kc->complete_jobs, job); wake(kc); } @@ -496,7 +494,8 @@ static int run_io_job(struct kcopyd_job *job) { int r; struct dm_io_request io_req = { - .bi_rw = job->rw, + .bi_op = job->rw, + .bi_op_flags = 0, .mem.type = DM_IO_PAGE_LIST, .mem.ptr.pl = job->pages, .mem.offset = 0, @@ -550,10 +549,10 @@ static int process_jobs(struct list_head *jobs, struct dm_kcopyd_client *kc, if (r < 0) { /* error this rogue job */ - if (job->rw & WRITE) - job->write_err = (unsigned long) -1L; - else + if (job->rw == READ) job->read_err = 1; + else + job->write_err = (unsigned long) -1L; push(&kc->complete_jobs, job); break; } @@ -734,7 +733,7 @@ int dm_kcopyd_copy(struct dm_kcopyd_client *kc, struct dm_io_region *from, /* * Use WRITE SAME to optimize zeroing if all dests support it. */ - job->rw = WRITE | REQ_WRITE_SAME; + job->rw = REQ_OP_WRITE_SAME; for (i = 0; i < job->num_dests; i++) if (!bdev_write_same(job->dests[i].bdev)) { job->rw = WRITE; diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c index 58c2a3a..f63bbf4 100644 --- a/drivers/md/dm-log-writes.c +++ b/drivers/md/dm-log-writes.c @@ -205,7 +205,7 @@ static int write_metadata(struct log_writes_c *lc, void *entry, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; page = alloc_page(GFP_KERNEL); if (!page) { @@ -270,7 +270,7 @@ static int log_one_block(struct log_writes_c *lc, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; for (i = 0; i < block->vec_cnt; i++) { /* @@ -292,7 +292,7 @@ static int log_one_block(struct log_writes_c *lc, bio->bi_bdev = lc->logdev->bdev; bio->bi_end_io = log_end_io; bio->bi_private = lc; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; ret = bio_add_page(bio, block->vecs[i].bv_page, block->vecs[i].bv_len, 0); diff --git a/drivers/md/dm-log.c b/drivers/md/dm-log.c index 627d191..4ca2d1d 100644 --- a/drivers/md/dm-log.c +++ b/drivers/md/dm-log.c @@ -293,7 +293,7 @@ static void header_from_disk(struct log_header_core *core, struct log_header_dis static int rw_header(struct log_c *lc, int rw) { - lc->io_req.bi_rw = rw; + lc->io_req.bi_op = rw; return dm_io(&lc->io_req, 1, &lc->header_location, NULL); } @@ -306,7 +306,8 @@ static int flush_header(struct log_c *lc) .count = 0, }; - lc->io_req.bi_rw = WRITE_FLUSH; + lc->io_req.bi_op = REQ_OP_WRITE; + lc->io_req.bi_op_flags = WRITE_FLUSH; return dm_io(&lc->io_req, 1, &null_location, NULL); } diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c index f2a363a..6b61b62 100644 --- a/drivers/md/dm-raid1.c +++ b/drivers/md/dm-raid1.c @@ -260,7 +260,8 @@ static int mirror_flush(struct dm_target *ti) struct dm_io_region io[ms->nr_mirrors]; struct mirror *m; struct dm_io_request io_req = { - .bi_rw = WRITE_FLUSH, + .bi_op = REQ_OP_WRITE, + .bi_op_flags = WRITE_FLUSH, .mem.type = DM_IO_KMEM, .mem.ptr.addr = NULL, .client = ms->io_client, @@ -541,7 +542,8 @@ static void read_async_bio(struct mirror *m, struct bio *bio) { struct dm_io_region io; struct dm_io_request io_req = { - .bi_rw = READ, + .bi_op = REQ_OP_READ, + .bi_op_flags = 0, .mem.type = DM_IO_BIO, .mem.ptr.bio = bio, .notify.fn = read_callback, @@ -654,7 +656,8 @@ static void do_write(struct mirror_set *ms, struct bio *bio) struct dm_io_region io[ms->nr_mirrors], *dest = io; struct mirror *m; struct dm_io_request io_req = { - .bi_rw = WRITE | (bio->bi_rw & WRITE_FLUSH_FUA), + .bi_op = REQ_OP_WRITE, + .bi_op_flags = bio->bi_rw & WRITE_FLUSH_FUA, .mem.type = DM_IO_BIO, .mem.ptr.bio = bio, .notify.fn = write_callback, @@ -663,7 +666,7 @@ static void do_write(struct mirror_set *ms, struct bio *bio) }; if (bio->bi_rw & REQ_DISCARD) { - io_req.bi_rw |= REQ_DISCARD; + io_req.bi_op = REQ_OP_DISCARD; io_req.mem.type = DM_IO_KMEM; io_req.mem.ptr.addr = NULL; } diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persistent.c index 4d39093..b8cf956 100644 --- a/drivers/md/dm-snap-persistent.c +++ b/drivers/md/dm-snap-persistent.c @@ -226,8 +226,8 @@ static void do_metadata(struct work_struct *work) /* * Read or write a chunk aligned and sized block of data from a device. */ -static int chunk_io(struct pstore *ps, void *area, chunk_t chunk, int rw, - int metadata) +static int chunk_io(struct pstore *ps, void *area, chunk_t chunk, int op, + int op_flags, int metadata) { struct dm_io_region where = { .bdev = dm_snap_cow(ps->store->snap)->bdev, @@ -235,7 +235,8 @@ static int chunk_io(struct pstore *ps, void *area, chunk_t chunk, int rw, .count = ps->store->chunk_size, }; struct dm_io_request io_req = { - .bi_rw = rw, + .bi_op = op, + .bi_op_flags = op_flags, .mem.type = DM_IO_VMA, .mem.ptr.vma = area, .client = ps->io_client, @@ -281,14 +282,14 @@ static void skip_metadata(struct pstore *ps) * Read or write a metadata area. Remembering to skip the first * chunk which holds the header. */ -static int area_io(struct pstore *ps, int rw) +static int area_io(struct pstore *ps, int op, int op_flags) { int r; chunk_t chunk; chunk = area_location(ps, ps->current_area); - r = chunk_io(ps, ps->area, chunk, rw, 0); + r = chunk_io(ps, ps->area, chunk, op, op_flags, 0); if (r) return r; @@ -302,7 +303,8 @@ static void zero_memory_area(struct pstore *ps) static int zero_disk_area(struct pstore *ps, chunk_t area) { - return chunk_io(ps, ps->zero_area, area_location(ps, area), WRITE, 0); + return chunk_io(ps, ps->zero_area, area_location(ps, area), + REQ_OP_WRITE, 0, 0); } static int read_header(struct pstore *ps, int *new_snapshot) @@ -334,7 +336,7 @@ static int read_header(struct pstore *ps, int *new_snapshot) if (r) return r; - r = chunk_io(ps, ps->header_area, 0, READ, 1); + r = chunk_io(ps, ps->header_area, 0, REQ_OP_READ, 0, 1); if (r) goto bad; @@ -395,7 +397,7 @@ static int write_header(struct pstore *ps) dh->version = cpu_to_le32(ps->version); dh->chunk_size = cpu_to_le32(ps->store->chunk_size); - return chunk_io(ps, ps->header_area, 0, WRITE, 1); + return chunk_io(ps, ps->header_area, 0, REQ_OP_WRITE, 0, 1); } /* @@ -739,7 +741,7 @@ static void persistent_commit_exception(struct dm_exception_store *store, /* * Commit exceptions to disk. */ - if (ps->valid && area_io(ps, WRITE_FLUSH_FUA)) + if (ps->valid && area_io(ps, REQ_OP_WRITE, WRITE_FLUSH_FUA)) ps->valid = 0; /* @@ -779,7 +781,7 @@ static int persistent_prepare_merge(struct dm_exception_store *store, return 0; ps->current_area--; - r = area_io(ps, READ); + r = area_io(ps, REQ_OP_READ, 0); if (r < 0) return r; ps->current_committed = ps->exceptions_per_area; @@ -816,7 +818,7 @@ static int persistent_commit_merge(struct dm_exception_store *store, for (i = 0; i < nr_merged; i++) clear_exception(ps, ps->current_committed - 1 - i); - r = area_io(ps, WRITE_FLUSH_FUA); + r = area_io(ps, REQ_OP_WRITE, WRITE_FLUSH_FUA); if (r < 0) return r; diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index a40ef76b..5e1d281 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -338,7 +338,7 @@ static int __blkdev_issue_discard_async(struct block_device *bdev, sector_t sect struct bio *parent_bio) { struct request_queue *q = bdev_get_queue(bdev); - int type = REQ_WRITE | REQ_DISCARD; + int op_flags = 0; struct bio *bio; if (!q || !nr_sects) @@ -350,7 +350,7 @@ static int __blkdev_issue_discard_async(struct block_device *bdev, sector_t sect if (flags & BLKDEV_DISCARD_SECURE) { if (!blk_queue_secdiscard(q)) return -EOPNOTSUPP; - type |= REQ_SECURE; + op_flags |= REQ_SECURE; } /* @@ -365,7 +365,8 @@ static int __blkdev_issue_discard_async(struct block_device *bdev, sector_t sect bio->bi_iter.bi_sector = sector; bio->bi_bdev = bdev; bio->bi_iter.bi_size = nr_sects << 9; - bio->bi_rw = type; + bio->bi_op = REQ_OP_DISCARD; + bio->bi_rw = op_flags; submit_bio(bio); diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 5df4048..1777c9c 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -2358,6 +2358,7 @@ static struct mapped_device *alloc_dev(int minor) bio_init(&md->flush_bio); md->flush_bio.bi_bdev = md->bdev; + md->flush_bio.bi_op = REQ_OP_WRITE; md->flush_bio.bi_rw = WRITE_FLUSH; dm_stats_init(&md->stats); diff --git a/include/linux/dm-io.h b/include/linux/dm-io.h index a68cbe5..b91b023 100644 --- a/include/linux/dm-io.h +++ b/include/linux/dm-io.h @@ -57,7 +57,8 @@ struct dm_io_notify { */ struct dm_io_client; struct dm_io_request { - int bi_rw; /* READ|WRITE - not READA */ + int bi_op; /* REQ_OP */ + int bi_op_flags; /* rq_flag_bits */ struct dm_io_memory mem; /* Memory to use for io */ struct dm_io_notify notify; /* Synchronous if notify.fn is NULL */ struct dm_io_client *client; /* Client memory handler */ -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:33 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 0919A29E22 for ; Mon, 11 Jan 2016 14:22:33 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 838C0AC002 for ; Mon, 11 Jan 2016 12:22:32 -0800 (PST) X-ASG-Debug-ID: 1452543750-04cbb05f7c97ae0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id c8Txhc9u1HA2Uxjx (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:30 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 3A26A8E74E; Mon, 11 Jan 2016 20:22:30 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEo008385; Mon, 11 Jan 2016 15:22:27 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 21/35] bcache: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:19 -0600 X-ASG-Orig-Subj: [PATCH 21/35] bcache: set bi_op to REQ_OP Message-Id: <1452543693-4440-22-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543750 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has bcache set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only Signed-off-by: Mike Christie --- drivers/md/bcache/btree.c | 2 ++ drivers/md/bcache/debug.c | 2 ++ drivers/md/bcache/io.c | 2 +- drivers/md/bcache/journal.c | 7 ++++--- drivers/md/bcache/movinggc.c | 2 +- drivers/md/bcache/request.c | 9 +++++---- drivers/md/bcache/super.c | 26 +++++++++++++++----------- drivers/md/bcache/writeback.c | 4 ++-- 8 files changed, 32 insertions(+), 22 deletions(-) diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c index 22b9e34..752a44f 100644 --- a/drivers/md/bcache/btree.c +++ b/drivers/md/bcache/btree.c @@ -295,6 +295,7 @@ static void bch_btree_node_read(struct btree *b) closure_init_stack(&cl); bio = bch_bbio_alloc(b->c); + bio->bi_op = REQ_OP_READ; bio->bi_rw = REQ_META|READ_SYNC; bio->bi_iter.bi_size = KEY_SIZE(&b->key) << 9; bio->bi_end_io = btree_node_read_endio; @@ -397,6 +398,7 @@ static void do_btree_node_write(struct btree *b) b->bio->bi_end_io = btree_node_write_endio; b->bio->bi_private = cl; + b->bio->bi_op = REQ_OP_WRITE; b->bio->bi_rw = REQ_META|WRITE_SYNC|REQ_FUA; b->bio->bi_iter.bi_size = roundup(set_bytes(i), block_bytes(b->c)); bch_bio_map(b->bio, i); diff --git a/drivers/md/bcache/debug.c b/drivers/md/bcache/debug.c index 52b6bcf..8df9e66 100644 --- a/drivers/md/bcache/debug.c +++ b/drivers/md/bcache/debug.c @@ -52,6 +52,7 @@ void bch_btree_verify(struct btree *b) bio->bi_bdev = PTR_CACHE(b->c, &b->key, 0)->bdev; bio->bi_iter.bi_sector = PTR_OFFSET(&b->key, 0); bio->bi_iter.bi_size = KEY_SIZE(&v->key) << 9; + bio->bi_op = REQ_OP_READ; bio->bi_rw = REQ_META|READ_SYNC; bch_bio_map(bio, sorted); @@ -114,6 +115,7 @@ void bch_data_verify(struct cached_dev *dc, struct bio *bio) check = bio_clone(bio, GFP_NOIO); if (!check) return; + check->bi_op = REQ_OP_READ; check->bi_rw |= READ_SYNC; if (bio_alloc_pages(check, GFP_NOIO)) diff --git a/drivers/md/bcache/io.c b/drivers/md/bcache/io.c index 86a0bb8..f10a9a0 100644 --- a/drivers/md/bcache/io.c +++ b/drivers/md/bcache/io.c @@ -111,7 +111,7 @@ void bch_bbio_count_io_errors(struct cache_set *c, struct bio *bio, struct bbio *b = container_of(bio, struct bbio, bio); struct cache *ca = PTR_CACHE(c, &b->key, 0); - unsigned threshold = bio->bi_rw & REQ_WRITE + unsigned threshold = op_is_write(bio->bi_op) ? c->congested_write_threshold_us : c->congested_read_threshold_us; diff --git a/drivers/md/bcache/journal.c b/drivers/md/bcache/journal.c index af3f9f7..68fa0f0 100644 --- a/drivers/md/bcache/journal.c +++ b/drivers/md/bcache/journal.c @@ -54,7 +54,7 @@ reread: left = ca->sb.bucket_size - offset; bio_reset(bio); bio->bi_iter.bi_sector = bucket + offset; bio->bi_bdev = ca->bdev; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_iter.bi_size = len << 9; bio->bi_end_io = journal_read_endio; @@ -452,7 +452,7 @@ static void do_journal_discard(struct cache *ca) bio->bi_iter.bi_sector = bucket_to_sector(ca->set, ca->sb.d[ja->discard_idx]); bio->bi_bdev = ca->bdev; - bio->bi_rw = REQ_WRITE|REQ_DISCARD; + bio->bi_op = REQ_OP_DISCARD; bio->bi_max_vecs = 1; bio->bi_io_vec = bio->bi_inline_vecs; bio->bi_iter.bi_size = bucket_bytes(ca); @@ -626,7 +626,8 @@ static void journal_write_unlocked(struct closure *cl) bio_reset(bio); bio->bi_iter.bi_sector = PTR_OFFSET(k, i); bio->bi_bdev = ca->bdev; - bio->bi_rw = REQ_WRITE|REQ_SYNC|REQ_META|REQ_FLUSH|REQ_FUA; + bio->bi_op = REQ_OP_WRITE; + bio->bi_rw = REQ_SYNC|REQ_META|REQ_FLUSH|REQ_FUA; bio->bi_iter.bi_size = sectors << 9; bio->bi_end_io = journal_write_endio; diff --git a/drivers/md/bcache/movinggc.c b/drivers/md/bcache/movinggc.c index b929fc9..f33860a 100644 --- a/drivers/md/bcache/movinggc.c +++ b/drivers/md/bcache/movinggc.c @@ -163,7 +163,7 @@ static void read_moving(struct cache_set *c) moving_init(io); bio = &io->bio.bio; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_end_io = read_moving_endio; if (bio_alloc_pages(bio, GFP_KERNEL)) diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index 25fa844..cd6837c 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -253,7 +253,7 @@ static void bch_data_insert_start(struct closure *cl) trace_bcache_cache_insert(k); bch_keylist_push(&op->insert_keys); - n->bi_rw |= REQ_WRITE; + n->bi_op = REQ_OP_WRITE; bch_submit_bbio(n, op->c, k, 0); } while (n != bio); @@ -383,7 +383,7 @@ static bool check_should_bypass(struct cached_dev *dc, struct bio *bio) if (mode == CACHE_MODE_NONE || (mode == CACHE_MODE_WRITEAROUND && - (bio->bi_rw & REQ_WRITE))) + op_is_write(bio->bi_op))) goto skip; if (bio->bi_iter.bi_sector & (c->sb.block_size - 1) || @@ -404,7 +404,7 @@ static bool check_should_bypass(struct cached_dev *dc, struct bio *bio) if (!congested && mode == CACHE_MODE_WRITEBACK && - (bio->bi_rw & REQ_WRITE) && + op_is_write(bio->bi_op) && (bio->bi_rw & REQ_SYNC)) goto rescale; @@ -657,7 +657,7 @@ static inline struct search *search_alloc(struct bio *bio, s->cache_miss = NULL; s->d = d; s->recoverable = 1; - s->write = (bio->bi_rw & REQ_WRITE) != 0; + s->write = op_is_write(bio->bi_op); s->read_dirty_data = 0; s->start_time = jiffies; @@ -925,6 +925,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) struct bio *flush = bio_alloc_bioset(GFP_NOIO, 0, dc->disk.bio_split); + flush->bi_op = REQ_OP_WRITE; flush->bi_rw = WRITE_FLUSH; flush->bi_bdev = bio->bi_bdev; flush->bi_end_io = request_endio; diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index 9e2eac8..b11f7f4 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -212,7 +212,8 @@ static void __write_super(struct cache_sb *sb, struct bio *bio) unsigned i; bio->bi_iter.bi_sector = SB_SECTOR; - bio->bi_rw = REQ_WRITE|REQ_SYNC|REQ_META; + bio->bi_op = REQ_OP_WRITE; + bio->bi_rw = REQ_SYNC|REQ_META; bio->bi_iter.bi_size = SB_SIZE; bch_bio_map(bio, NULL); @@ -333,7 +334,7 @@ static void uuid_io_unlock(struct closure *cl) up(&c->uuid_write_mutex); } -static void uuid_io(struct cache_set *c, unsigned long rw, +static void uuid_io(struct cache_set *c, int op, unsigned long op_flags, struct bkey *k, struct closure *parent) { struct closure *cl = &c->uuid_write; @@ -348,7 +349,8 @@ static void uuid_io(struct cache_set *c, unsigned long rw, for (i = 0; i < KEY_PTRS(k); i++) { struct bio *bio = bch_bbio_alloc(c); - bio->bi_rw = REQ_SYNC|REQ_META|rw; + bio->bi_op = op; + bio->bi_rw = REQ_SYNC|REQ_META|op_flags; bio->bi_iter.bi_size = KEY_SIZE(k) << 9; bio->bi_end_io = uuid_endio; @@ -357,12 +359,12 @@ static void uuid_io(struct cache_set *c, unsigned long rw, bch_submit_bbio(bio, c, k, i); - if (!(rw & WRITE)) + if (op != REQ_OP_WRITE) break; } bch_extent_to_text(buf, sizeof(buf), k); - pr_debug("%s UUIDs at %s", rw & REQ_WRITE ? "wrote" : "read", buf); + pr_debug("%s UUIDs at %s", op == REQ_OP_WRITE ? "wrote" : "read", buf); for (u = c->uuids; u < c->uuids + c->nr_uuids; u++) if (!bch_is_zero(u->uuid, 16)) @@ -381,7 +383,7 @@ static char *uuid_read(struct cache_set *c, struct jset *j, struct closure *cl) return "bad uuid pointer"; bkey_copy(&c->uuid_bucket, k); - uuid_io(c, READ_SYNC, k, cl); + uuid_io(c, REQ_OP_READ, READ_SYNC, k, cl); if (j->version < BCACHE_JSET_VERSION_UUIDv1) { struct uuid_entry_v0 *u0 = (void *) c->uuids; @@ -426,7 +428,7 @@ static int __uuid_write(struct cache_set *c) return 1; SET_KEY_SIZE(&k.key, c->sb.bucket_size); - uuid_io(c, REQ_WRITE, &k.key, &cl); + uuid_io(c, REQ_OP_WRITE, 0, &k.key, &cl); closure_sync(&cl); bkey_copy(&c->uuid_bucket, &k.key); @@ -498,7 +500,8 @@ static void prio_endio(struct bio *bio) closure_put(&ca->prio); } -static void prio_io(struct cache *ca, uint64_t bucket, unsigned long rw) +static void prio_io(struct cache *ca, uint64_t bucket, int op, + unsigned long op_flags) { struct closure *cl = &ca->prio; struct bio *bio = bch_bbio_alloc(ca->set); @@ -507,7 +510,8 @@ static void prio_io(struct cache *ca, uint64_t bucket, unsigned long rw) bio->bi_iter.bi_sector = bucket * ca->sb.bucket_size; bio->bi_bdev = ca->bdev; - bio->bi_rw = REQ_SYNC|REQ_META|rw; + bio->bi_op = op; + bio->bi_rw = REQ_SYNC|REQ_META|op_flags; bio->bi_iter.bi_size = bucket_bytes(ca); bio->bi_end_io = prio_endio; @@ -557,7 +561,7 @@ void bch_prio_write(struct cache *ca) BUG_ON(bucket == -1); mutex_unlock(&ca->set->bucket_lock); - prio_io(ca, bucket, REQ_WRITE); + prio_io(ca, bucket, REQ_OP_WRITE, 0); mutex_lock(&ca->set->bucket_lock); ca->prio_buckets[i] = bucket; @@ -599,7 +603,7 @@ static void prio_read(struct cache *ca, uint64_t bucket) ca->prio_last_buckets[bucket_nr] = bucket; bucket_nr++; - prio_io(ca, bucket, READ_SYNC); + prio_io(ca, bucket, REQ_OP_READ, READ_SYNC); if (p->csum != bch_crc64(&p->magic, bucket_bytes(ca) - 8)) pr_warn("bad csum reading priorities"); diff --git a/drivers/md/bcache/writeback.c b/drivers/md/bcache/writeback.c index b9346cd..96ddf28 100644 --- a/drivers/md/bcache/writeback.c +++ b/drivers/md/bcache/writeback.c @@ -183,7 +183,7 @@ static void write_dirty(struct closure *cl) struct keybuf_key *w = io->bio.bi_private; dirty_init(w); - io->bio.bi_rw = WRITE; + io->bio.bi_op = REQ_OP_WRITE; io->bio.bi_iter.bi_sector = KEY_START(&w->key); io->bio.bi_bdev = io->dc->bdev; io->bio.bi_end_io = dirty_endio; @@ -256,7 +256,7 @@ static void read_dirty(struct cached_dev *dc) io->bio.bi_iter.bi_sector = PTR_OFFSET(&w->key, 0); io->bio.bi_bdev = PTR_CACHE(dc->disk.c, &w->key, 0)->bdev; - io->bio.bi_rw = READ; + io->bio.bi_op = REQ_OP_READ; io->bio.bi_end_io = read_dirty_endio; if (bio_alloc_pages(&io->bio, GFP_KERNEL)) -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:36 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 85AAC29E0A for ; Mon, 11 Jan 2016 14:22:36 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 4CBA1304059 for ; Mon, 11 Jan 2016 12:22:36 -0800 (PST) X-ASG-Debug-ID: 1452543752-04bdf06f6b7db20001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id hImtH46H8LHOIt2n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:32 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 88BF1744; Mon, 11 Jan 2016 20:22:32 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEp008385; Mon, 11 Jan 2016 15:22:30 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 22/35] drbd: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:20 -0600 X-ASG-Orig-Subj: [PATCH 22/35] drbd: set bi_op to REQ_OP Message-Id: <1452543693-4440-23-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543752 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has drbd set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. Lars and Philip, I might have split this patch up a little weird. Thisi patch handles setting up the bio, and then patch 30 (0030-block-fs-drivers-do-not-test-bi_rw-for-REQ_OPs.patch) handles where were check/read bio->bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/block/drbd/drbd_actlog.c | 29 ++++++++++++++++------------- drivers/block/drbd/drbd_bitmap.c | 6 +++--- drivers/block/drbd/drbd_int.h | 4 ++-- drivers/block/drbd/drbd_main.c | 5 +++-- drivers/block/drbd/drbd_receiver.c | 37 +++++++++++++++++++++++++------------ drivers/block/drbd/drbd_worker.c | 3 ++- 6 files changed, 51 insertions(+), 33 deletions(-) diff --git a/drivers/block/drbd/drbd_actlog.c b/drivers/block/drbd/drbd_actlog.c index 6069e15..2fa8534 100644 --- a/drivers/block/drbd/drbd_actlog.c +++ b/drivers/block/drbd/drbd_actlog.c @@ -137,19 +137,19 @@ void wait_until_done_or_force_detached(struct drbd_device *device, struct drbd_b static int _drbd_md_sync_page_io(struct drbd_device *device, struct drbd_backing_dev *bdev, - sector_t sector, int rw) + sector_t sector, int op) { struct bio *bio; /* we do all our meta data IO in aligned 4k blocks. */ const int size = 4096; - int err; + int err, op_flags = 0; device->md_io.done = 0; device->md_io.error = -ENODEV; - if ((rw & WRITE) && !test_bit(MD_NO_FUA, &device->flags)) - rw |= REQ_FUA | REQ_FLUSH; - rw |= REQ_SYNC | REQ_NOIDLE; + if ((op == REQ_OP_WRITE) && !test_bit(MD_NO_FUA, &device->flags)) + op_flags |= REQ_FUA | REQ_FLUSH; + op_flags |= REQ_SYNC | REQ_NOIDLE; bio = bio_alloc_drbd(GFP_NOIO); bio->bi_bdev = bdev->md_bdev; @@ -159,9 +159,10 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, goto out; bio->bi_private = device; bio->bi_end_io = drbd_md_endio; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; - if (!(rw & WRITE) && device->state.disk == D_DISKLESS && device->ldev == NULL) + if (op != REQ_OP_WRITE && device->state.disk == D_DISKLESS && device->ldev == NULL) /* special case, drbd_md_read() during drbd_adm_attach(): no get_ldev */ ; else if (!get_ldev_if_state(device, D_ATTACHING)) { @@ -174,7 +175,7 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, bio_get(bio); /* one bio_put() is in the completion handler */ atomic_inc(&device->md_io.in_use); /* drbd_md_put_buffer() is in the completion handler */ device->md_io.submit_jif = jiffies; - if (drbd_insert_fault(device, (rw & WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) + if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) bio_io_error(bio); else submit_bio(bio); @@ -188,7 +189,7 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, } int drbd_md_sync_page_io(struct drbd_device *device, struct drbd_backing_dev *bdev, - sector_t sector, int rw) + sector_t sector, int op) { int err; D_ASSERT(device, atomic_read(&device->md_io.in_use) == 1); @@ -197,19 +198,21 @@ int drbd_md_sync_page_io(struct drbd_device *device, struct drbd_backing_dev *bd dynamic_drbd_dbg(device, "meta_data io: %s [%d]:%s(,%llus,%s) %pS\n", current->comm, current->pid, __func__, - (unsigned long long)sector, (rw & WRITE) ? "WRITE" : "READ", + (unsigned long long)sector, (op == REQ_OP_WRITE) ? "WRITE" : "READ", (void*)_RET_IP_ ); if (sector < drbd_md_first_sector(bdev) || sector + 7 > drbd_md_last_sector(bdev)) drbd_alert(device, "%s [%d]:%s(,%llus,%s) out of range md access!\n", current->comm, current->pid, __func__, - (unsigned long long)sector, (rw & WRITE) ? "WRITE" : "READ"); + (unsigned long long)sector, + (op == REQ_OP_WRITE) ? "WRITE" : "READ"); - err = _drbd_md_sync_page_io(device, bdev, sector, rw); + err = _drbd_md_sync_page_io(device, bdev, sector, op); if (err) { drbd_err(device, "drbd_md_sync_page_io(,%llus,%s) failed with error %d\n", - (unsigned long long)sector, (rw & WRITE) ? "WRITE" : "READ", err); + (unsigned long long)sector, + (op == REQ_OP_WRITE) ? "WRITE" : "READ", err); } return err; } diff --git a/drivers/block/drbd/drbd_bitmap.c b/drivers/block/drbd/drbd_bitmap.c index e8959fe..126bf4a 100644 --- a/drivers/block/drbd/drbd_bitmap.c +++ b/drivers/block/drbd/drbd_bitmap.c @@ -980,7 +980,7 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho struct drbd_bitmap *b = device->bitmap; struct page *page; unsigned int len; - unsigned int rw = (ctx->flags & BM_AIO_READ) ? READ : WRITE; + unsigned int op = (ctx->flags & BM_AIO_READ) ? REQ_OP_READ : REQ_OP_WRITE; sector_t on_disk_sector = device->ldev->md.md_offset + device->ldev->md.bm_offset; @@ -1011,9 +1011,9 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho bio_add_page(bio, page, len, 0); bio->bi_private = ctx; bio->bi_end_io = drbd_bm_endio; - bio->bi_rw = rw; + bio->bi_op = op; - if (drbd_insert_fault(device, (rw & WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) { + if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) { bio_io_error(bio); } else { submit_bio(bio); diff --git a/drivers/block/drbd/drbd_int.h b/drivers/block/drbd/drbd_int.h index 34bc84e..5204079 100644 --- a/drivers/block/drbd/drbd_int.h +++ b/drivers/block/drbd/drbd_int.h @@ -1507,7 +1507,7 @@ extern int drbd_resync_finished(struct drbd_device *device); extern void *drbd_md_get_buffer(struct drbd_device *device, const char *intent); extern void drbd_md_put_buffer(struct drbd_device *device); extern int drbd_md_sync_page_io(struct drbd_device *device, - struct drbd_backing_dev *bdev, sector_t sector, int rw); + struct drbd_backing_dev *bdev, sector_t sector, int op); extern void drbd_ov_out_of_sync_found(struct drbd_device *, sector_t, int); extern void wait_until_done_or_force_detached(struct drbd_device *device, struct drbd_backing_dev *bdev, unsigned int *done); @@ -1557,7 +1557,7 @@ extern bool drbd_rs_should_slow_down(struct drbd_device *device, sector_t sector bool throttle_if_app_is_waiting); extern int drbd_submit_peer_request(struct drbd_device *, struct drbd_peer_request *, const unsigned, - const int); + const unsigned, const int); extern int drbd_free_peer_reqs(struct drbd_device *, struct list_head *); extern struct drbd_peer_request *drbd_alloc_peer_req(struct drbd_peer_device *, u64, sector_t, unsigned int, diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 5b43dfb..f6626e5 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -3061,7 +3061,7 @@ void drbd_md_write(struct drbd_device *device, void *b) D_ASSERT(device, drbd_md_ss(device->ldev) == device->ldev->md.md_offset); sector = device->ldev->md.md_offset; - if (drbd_md_sync_page_io(device, device->ldev, sector, WRITE)) { + if (drbd_md_sync_page_io(device, device->ldev, sector, REQ_OP_WRITE)) { /* this was a try anyways ... */ drbd_err(device, "meta data update failed!\n"); drbd_chk_io_error(device, 1, DRBD_META_IO_ERROR); @@ -3263,7 +3263,8 @@ int drbd_md_read(struct drbd_device *device, struct drbd_backing_dev *bdev) * Affects the paranoia out-of-range access check in drbd_md_sync_page_io(). */ bdev->md.md_size_sect = 8; - if (drbd_md_sync_page_io(device, bdev, bdev->md.md_offset, READ)) { + if (drbd_md_sync_page_io(device, bdev, bdev->md.md_offset, + REQ_OP_READ)) { /* NOTE: can't do normal error processing here as this is called BEFORE disk is attached */ drbd_err(device, "Error while reading metadata.\n"); diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c index 1957fe8..880de0d 100644 --- a/drivers/block/drbd/drbd_receiver.c +++ b/drivers/block/drbd/drbd_receiver.c @@ -1398,7 +1398,8 @@ void drbd_bump_write_ordering(struct drbd_resource *resource, struct drbd_backin /* TODO allocate from our own bio_set. */ int drbd_submit_peer_request(struct drbd_device *device, struct drbd_peer_request *peer_req, - const unsigned rw, const int fault_type) + const unsigned op, const unsigned op_flags, + const int fault_type) { struct bio *bios = NULL; struct bio *bio; @@ -1450,7 +1451,8 @@ next_bio: /* > peer_req->i.sector, unless this is the first bio */ bio->bi_iter.bi_sector = sector; bio->bi_bdev = device->ldev->backing_bdev; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; bio->bi_private = peer_req; bio->bi_end_io = drbd_peer_request_endio; @@ -1458,7 +1460,7 @@ next_bio: bios = bio; ++n_bios; - if (rw & REQ_DISCARD) { + if (op == REQ_OP_DISCARD) { bio->bi_iter.bi_size = data_size; goto submit; } @@ -1830,7 +1832,8 @@ static int recv_resync_read(struct drbd_peer_device *peer_device, sector_t secto spin_unlock_irq(&device->resource->req_lock); atomic_add(pi->size >> 9, &device->rs_sect_ev); - if (drbd_submit_peer_request(device, peer_req, WRITE, DRBD_FAULT_RS_WR) == 0) + if (drbd_submit_peer_request(device, peer_req, REQ_OP_WRITE, 0, + DRBD_FAULT_RS_WR) == 0) return 0; /* don't care for the reason here */ @@ -2152,12 +2155,19 @@ static int wait_for_and_update_peer_seq(struct drbd_peer_device *peer_device, co /* see also bio_flags_to_wire() * DRBD_REQ_*, because we need to semantically map the flags to data packet * flags and back. We may replicate to other kernel versions. */ -static unsigned long wire_flags_to_bio(u32 dpf) +static unsigned long wire_flags_to_bio_flags(u32 dpf) { return (dpf & DP_RW_SYNC ? REQ_SYNC : 0) | (dpf & DP_FUA ? REQ_FUA : 0) | - (dpf & DP_FLUSH ? REQ_FLUSH : 0) | - (dpf & DP_DISCARD ? REQ_DISCARD : 0); + (dpf & DP_FLUSH ? REQ_FLUSH : 0); +} + +static unsigned long wire_flags_to_bio_op(u32 dpf) +{ + if (dpf & DP_DISCARD) + return REQ_OP_DISCARD; + else + return REQ_OP_WRITE; } static void fail_postponed_requests(struct drbd_device *device, sector_t sector, @@ -2303,7 +2313,7 @@ static int receive_Data(struct drbd_connection *connection, struct packet_info * struct drbd_peer_request *peer_req; struct p_data *p = pi->data; u32 peer_seq = be32_to_cpu(p->seq_num); - int rw = WRITE; + int op, op_flags; u32 dp_flags; int err, tp; @@ -2342,14 +2352,15 @@ static int receive_Data(struct drbd_connection *connection, struct packet_info * peer_req->flags |= EE_APPLICATION; dp_flags = be32_to_cpu(p->dp_flags); - rw |= wire_flags_to_bio(dp_flags); + op = wire_flags_to_bio_op(dp_flags); + op_flags = wire_flags_to_bio_flags(dp_flags); if (pi->cmd == P_TRIM) { struct request_queue *q = bdev_get_queue(device->ldev->backing_bdev); peer_req->flags |= EE_IS_TRIM; if (!blk_queue_discard(q)) peer_req->flags |= EE_IS_TRIM_USE_ZEROOUT; D_ASSERT(peer_device, peer_req->i.size > 0); - D_ASSERT(peer_device, rw & REQ_DISCARD); + D_ASSERT(peer_device, op == REQ_OP_DISCARD); D_ASSERT(peer_device, peer_req->pages == NULL); } else if (peer_req->pages == NULL) { D_ASSERT(device, peer_req->i.size == 0); @@ -2433,7 +2444,8 @@ static int receive_Data(struct drbd_connection *connection, struct packet_info * peer_req->flags |= EE_CALL_AL_COMPLETE_IO; } - err = drbd_submit_peer_request(device, peer_req, rw, DRBD_FAULT_DT_WR); + err = drbd_submit_peer_request(device, peer_req, op, op_flags, + DRBD_FAULT_DT_WR); if (!err) return 0; @@ -2723,7 +2735,8 @@ submit_for_resync: submit: update_receiver_timing_details(connection, drbd_submit_peer_request); inc_unacked(device); - if (drbd_submit_peer_request(device, peer_req, READ, fault_type) == 0) + if (drbd_submit_peer_request(device, peer_req, REQ_OP_READ, 0, + fault_type) == 0) return 0; /* don't care for the reason here */ diff --git a/drivers/block/drbd/drbd_worker.c b/drivers/block/drbd/drbd_worker.c index eff716c..f92f533 100644 --- a/drivers/block/drbd/drbd_worker.c +++ b/drivers/block/drbd/drbd_worker.c @@ -392,7 +392,8 @@ static int read_for_csum(struct drbd_peer_device *peer_device, sector_t sector, spin_unlock_irq(&device->resource->req_lock); atomic_add(size >> 9, &device->rs_sect_ev); - if (drbd_submit_peer_request(device, peer_req, READ, DRBD_FAULT_RS_RD) == 0) + if (drbd_submit_peer_request(device, peer_req, REQ_OP_READ, 0, + DRBD_FAULT_RS_RD) == 0) return 0; /* If it failed because of ENOMEM, retry should help. If it failed -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:38 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 3795F29E0A for ; Mon, 11 Jan 2016 14:22:38 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 2791B304066 for ; Mon, 11 Jan 2016 12:22:38 -0800 (PST) X-ASG-Debug-ID: 1452543755-04cbb05f7d97b00001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id EEBNB0ZB47Vkl2oK (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:35 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id D744619EF1B; Mon, 11 Jan 2016 20:22:34 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEq008385; Mon, 11 Jan 2016 15:22:32 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 23/35] md/raid: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:21 -0600 X-ASG-Orig-Subj: [PATCH 23/35] md/raid: set bi_op to REQ_OP Message-Id: <1452543693-4440-24-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543755 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has md/raid set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/md/bitmap.c | 2 +- drivers/md/dm-raid.c | 5 +++-- drivers/md/md.c | 11 +++++++---- drivers/md/md.h | 3 ++- drivers/md/raid1.c | 34 ++++++++++++++++---------------- drivers/md/raid10.c | 50 ++++++++++++++++++++++++++---------------------- drivers/md/raid5-cache.c | 25 +++++++++++++++--------- drivers/md/raid5.c | 48 ++++++++++++++++++++++++++-------------------- 8 files changed, 101 insertions(+), 77 deletions(-) diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c index 13811fc..18458f2 100644 --- a/drivers/md/bitmap.c +++ b/drivers/md/bitmap.c @@ -160,7 +160,7 @@ static int read_sb_page(struct mddev *mddev, loff_t offset, if (sync_page_io(rdev, target, roundup(size, bdev_logical_block_size(rdev->bdev)), - page, READ, true)) { + page, REQ_OP_READ, 0, true)) { page->index = index; return 0; } diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c index a090121..43a749c 100644 --- a/drivers/md/dm-raid.c +++ b/drivers/md/dm-raid.c @@ -792,7 +792,7 @@ static int read_disk_sb(struct md_rdev *rdev, int size) if (rdev->sb_loaded) return 0; - if (!sync_page_io(rdev, 0, size, rdev->sb_page, READ, 1)) { + if (!sync_page_io(rdev, 0, size, rdev->sb_page, REQ_OP_READ, 0, 1)) { DMERR("Failed to read superblock of device at position %d", rdev->raid_disk); md_error(rdev->mddev, rdev); @@ -1646,7 +1646,8 @@ static void attempt_restore_of_faulty_devices(struct raid_set *rs) for (i = 0; i < rs->md.raid_disks; i++) { r = &rs->dev[i].rdev; if (test_bit(Faulty, &r->flags) && r->sb_page && - sync_page_io(r, 0, r->sb_size, r->sb_page, READ, 1)) { + sync_page_io(r, 0, r->sb_size, r->sb_page, REQ_OP_READ, 0, + 1)) { DMINFO("Faulty %s device #%d has readable super block." " Attempting to revive it.", rs->raid_type->name, i); diff --git a/drivers/md/md.c b/drivers/md/md.c index 58ebe4b..a2199ad 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -391,6 +391,7 @@ static void submit_flushes(struct work_struct *ws) bi->bi_end_io = md_end_flush; bi->bi_private = rdev; bi->bi_bdev = rdev->bdev; + bi->bi_op = REQ_OP_WRITE; bi->bi_rw = WRITE_FLUSH; atomic_inc(&mddev->flush_pending); submit_bio(bi); @@ -737,6 +738,7 @@ void md_super_write(struct mddev *mddev, struct md_rdev *rdev, bio_add_page(bio, page, size, 0); bio->bi_private = rdev; bio->bi_end_io = super_written; + bio->bi_op = REQ_OP_WRITE; bio->bi_rw = WRITE_FLUSH_FUA; atomic_inc(&mddev->pending_writes); @@ -750,14 +752,15 @@ void md_super_wait(struct mddev *mddev) } int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, - struct page *page, int rw, bool metadata_op) + struct page *page, int op, int op_flags, bool metadata_op) { struct bio *bio = bio_alloc_mddev(GFP_NOIO, 1, rdev->mddev); int ret; bio->bi_bdev = (metadata_op && rdev->meta_bdev) ? rdev->meta_bdev : rdev->bdev; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; if (metadata_op) bio->bi_iter.bi_sector = sector + rdev->sb_start; else if (rdev->mddev->reshape_position != MaxSector && @@ -783,7 +786,7 @@ static int read_disk_sb(struct md_rdev *rdev, int size) if (rdev->sb_loaded) return 0; - if (!sync_page_io(rdev, 0, size, rdev->sb_page, READ, true)) + if (!sync_page_io(rdev, 0, size, rdev->sb_page, REQ_OP_READ, 0, true)) goto fail; rdev->sb_loaded = 1; return 0; @@ -1469,7 +1472,7 @@ static int super_1_load(struct md_rdev *rdev, struct md_rdev *refdev, int minor_ return -EINVAL; bb_sector = (long long)offset; if (!sync_page_io(rdev, bb_sector, sectors << 9, - rdev->bb_page, READ, true)) + rdev->bb_page, REQ_OP_READ, 0, true)) return -EIO; bbp = (u64 *)page_address(rdev->bb_page); rdev->badblocks.shift = sb->bblog_shift; diff --git a/drivers/md/md.h b/drivers/md/md.h index 25c821b..6dc5f66 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -618,7 +618,8 @@ extern void md_super_write(struct mddev *mddev, struct md_rdev *rdev, sector_t sector, int size, struct page *page); extern void md_super_wait(struct mddev *mddev); extern int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, - struct page *page, int rw, bool metadata_op); + struct page *page, int op, int op_flags, + bool metadata_op); extern void md_do_sync(struct md_thread *thread); extern void md_new_event(struct mddev *mddev); extern int md_allow_write(struct mddev *mddev); diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 01e6e74..2a42daa 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -759,7 +759,7 @@ static void flush_pending_writes(struct r1conf *conf) while (bio) { /* submit pending writes */ struct bio *next = bio->bi_next; bio->bi_next = NULL; - if (unlikely((bio->bi_rw & REQ_DISCARD) && + if (unlikely((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) /* Just ignore it */ bio_endio(bio); @@ -1033,7 +1033,7 @@ static void raid1_unplug(struct blk_plug_cb *cb, bool from_schedule) while (bio) { /* submit pending writes */ struct bio *next = bio->bi_next; bio->bi_next = NULL; - if (unlikely((bio->bi_rw & REQ_DISCARD) && + if (unlikely((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) /* Just ignore it */ bio_endio(bio); @@ -1053,12 +1053,11 @@ static void make_request(struct mddev *mddev, struct bio * bio) int i, disks; struct bitmap *bitmap; unsigned long flags; + const int op = bio->bi_op; const int rw = bio_data_dir(bio); const unsigned long do_sync = (bio->bi_rw & REQ_SYNC); const unsigned long do_flush_fua = (bio->bi_rw & (REQ_FLUSH | REQ_FUA)); - const unsigned long do_discard = (bio->bi_rw - & (REQ_DISCARD | REQ_SECURE)); - const unsigned long do_same = (bio->bi_rw & REQ_WRITE_SAME); + const unsigned long do_sec = (bio->bi_rw & REQ_SECURE); struct md_rdev *blocked_rdev; struct blk_plug_cb *cb; struct raid1_plug_cb *plug = NULL; @@ -1166,7 +1165,8 @@ read_again: mirror->rdev->data_offset; read_bio->bi_bdev = mirror->rdev->bdev; read_bio->bi_end_io = raid1_end_read_request; - read_bio->bi_rw = READ | do_sync; + read_bio->bi_op = op; + read_bio->bi_rw = do_sync; read_bio->bi_private = r1_bio; if (max_sectors < r1_bio->sectors) { @@ -1376,8 +1376,9 @@ read_again: conf->mirrors[i].rdev->data_offset); mbio->bi_bdev = conf->mirrors[i].rdev->bdev; mbio->bi_end_io = raid1_end_write_request; + mbio->bi_op = op; mbio->bi_rw = - WRITE | do_flush_fua | do_sync | do_discard | do_same; + do_flush_fua | do_sync | do_sec; mbio->bi_private = r1_bio; atomic_inc(&r1_bio->remaining); @@ -1771,7 +1772,7 @@ static void end_sync_write(struct bio *bio) static int r1_sync_page_io(struct md_rdev *rdev, sector_t sector, int sectors, struct page *page, int rw) { - if (sync_page_io(rdev, sector, sectors << 9, page, rw, false)) + if (sync_page_io(rdev, sector, sectors << 9, page, rw, 0, false)) /* success */ return 1; if (rw == WRITE) { @@ -1825,7 +1826,7 @@ static int fix_sync_read_error(struct r1bio *r1_bio) rdev = conf->mirrors[d].rdev; if (sync_page_io(rdev, sect, s<<9, bio->bi_io_vec[idx].bv_page, - READ, false)) { + REQ_OP_READ, 0, false)) { success = 1; break; } @@ -2030,7 +2031,7 @@ static void sync_request_write(struct mddev *mddev, struct r1bio *r1_bio) !test_bit(MD_RECOVERY_SYNC, &mddev->recovery)))) continue; - wbio->bi_rw = WRITE; + wbio->bi_op = REQ_OP_WRITE; wbio->bi_end_io = end_sync_write; atomic_inc(&r1_bio->remaining); md_sync_acct(conf->mirrors[i].rdev->bdev, bio_sectors(wbio)); @@ -2090,7 +2091,7 @@ static void fix_read_error(struct r1conf *conf, int read_disk, is_badblock(rdev, sect, s, &first_bad, &bad_sectors) == 0 && sync_page_io(rdev, sect, s<<9, - conf->tmppage, READ, false)) + conf->tmppage, REQ_OP_READ, 0, false)) success = 1; else { d++; @@ -2201,7 +2202,7 @@ static int narrow_write_error(struct r1bio *r1_bio, int i) wbio = bio_clone_mddev(r1_bio->master_bio, GFP_NOIO, mddev); } - wbio->bi_rw = WRITE; + wbio->bi_op = REQ_OP_WRITE; wbio->bi_iter.bi_sector = r1_bio->sector; wbio->bi_iter.bi_size = r1_bio->sectors << 9; @@ -2343,7 +2344,8 @@ read_more: bio->bi_iter.bi_sector = r1_bio->sector + rdev->data_offset; bio->bi_bdev = rdev->bdev; bio->bi_end_io = raid1_end_read_request; - bio->bi_rw = READ | do_sync; + bio->bi_op = REQ_OP_READ; + bio->bi_rw = do_sync; bio->bi_private = r1_bio; if (max_sectors < r1_bio->sectors) { /* Drat - have to split this up more */ @@ -2568,7 +2570,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipp if (i < conf->raid_disks) still_degraded = 1; } else if (!test_bit(In_sync, &rdev->flags)) { - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_end_io = end_sync_write; write_targets ++; } else { @@ -2595,7 +2597,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipp if (disk < 0) disk = i; } - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_end_io = end_sync_read; read_targets++; } else if (!test_bit(WriteErrorSeen, &rdev->flags) && @@ -2607,7 +2609,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipp * if we are doing resync or repair. Otherwise, leave * this device alone for this sync request. */ - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_end_io = end_sync_write; write_targets++; } diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 5b81668..1cfc061 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -865,7 +865,7 @@ static void flush_pending_writes(struct r10conf *conf) while (bio) { /* submit pending writes */ struct bio *next = bio->bi_next; bio->bi_next = NULL; - if (unlikely((bio->bi_rw & REQ_DISCARD) && + if (unlikely((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) /* Just ignore it */ bio_endio(bio); @@ -1041,7 +1041,7 @@ static void raid10_unplug(struct blk_plug_cb *cb, bool from_schedule) while (bio) { /* submit pending writes */ struct bio *next = bio->bi_next; bio->bi_next = NULL; - if (unlikely((bio->bi_rw & REQ_DISCARD) && + if (unlikely((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) /* Just ignore it */ bio_endio(bio); @@ -1058,12 +1058,11 @@ static void __make_request(struct mddev *mddev, struct bio *bio) struct r10bio *r10_bio; struct bio *read_bio; int i; + const int op = bio->bi_op; const int rw = bio_data_dir(bio); const unsigned long do_sync = (bio->bi_rw & REQ_SYNC); const unsigned long do_fua = (bio->bi_rw & REQ_FUA); - const unsigned long do_discard = (bio->bi_rw - & (REQ_DISCARD | REQ_SECURE)); - const unsigned long do_same = (bio->bi_rw & REQ_WRITE_SAME); + const unsigned long do_sec = (bio->bi_rw & REQ_SECURE); unsigned long flags; struct md_rdev *blocked_rdev; struct blk_plug_cb *cb; @@ -1156,7 +1155,8 @@ read_again: choose_data_offset(r10_bio, rdev); read_bio->bi_bdev = rdev->bdev; read_bio->bi_end_io = raid10_end_read_request; - read_bio->bi_rw = READ | do_sync; + read_bio->bi_op = op; + read_bio->bi_rw = do_sync; read_bio->bi_private = r10_bio; if (max_sectors < r10_bio->sectors) { @@ -1363,8 +1363,9 @@ retry_write: rdev)); mbio->bi_bdev = rdev->bdev; mbio->bi_end_io = raid10_end_write_request; + mbio->bi_op = op; mbio->bi_rw = - WRITE | do_sync | do_fua | do_discard | do_same; + do_sync | do_fua | do_sec; mbio->bi_private = r10_bio; atomic_inc(&r10_bio->remaining); @@ -1406,8 +1407,9 @@ retry_write: r10_bio, rdev)); mbio->bi_bdev = rdev->bdev; mbio->bi_end_io = raid10_end_write_request; + mbio->bi_op = op; mbio->bi_rw = - WRITE | do_sync | do_fua | do_discard | do_same; + do_sync | do_fua | do_sec; mbio->bi_private = r10_bio; atomic_inc(&r10_bio->remaining); @@ -1992,7 +1994,7 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio) tbio->bi_vcnt = vcnt; tbio->bi_iter.bi_size = fbio->bi_iter.bi_size; - tbio->bi_rw = WRITE; + tbio->bi_op = REQ_OP_WRITE; tbio->bi_private = r10_bio; tbio->bi_iter.bi_sector = r10_bio->devs[i].addr; tbio->bi_end_io = end_sync_write; @@ -2078,7 +2080,7 @@ static void fix_recovery_read_error(struct r10bio *r10_bio) addr, s << 9, bio->bi_io_vec[idx].bv_page, - READ, false); + REQ_OP_READ, 0, false); if (ok) { rdev = conf->mirrors[dw].rdev; addr = r10_bio->devs[1].addr + sect; @@ -2086,7 +2088,7 @@ static void fix_recovery_read_error(struct r10bio *r10_bio) addr, s << 9, bio->bi_io_vec[idx].bv_page, - WRITE, false); + REQ_OP_WRITE, 0, false); if (!ok) { set_bit(WriteErrorSeen, &rdev->flags); if (!test_and_set_bit(WantReplacement, @@ -2213,7 +2215,7 @@ static int r10_sync_page_io(struct md_rdev *rdev, sector_t sector, if (is_badblock(rdev, sector, sectors, &first_bad, &bad_sectors) && (rw == READ || test_bit(WriteErrorSeen, &rdev->flags))) return -1; - if (sync_page_io(rdev, sector, sectors << 9, page, rw, false)) + if (sync_page_io(rdev, sector, sectors << 9, page, rw, 0, false)) /* success */ return 1; if (rw == WRITE) { @@ -2299,7 +2301,8 @@ static void fix_read_error(struct r10conf *conf, struct mddev *mddev, struct r10 r10_bio->devs[sl].addr + sect, s<<9, - conf->tmppage, READ, false); + conf->tmppage, + REQ_OP_READ, 0, false); rdev_dec_pending(rdev, mddev); rcu_read_lock(); if (success) @@ -2474,7 +2477,7 @@ static int narrow_write_error(struct r10bio *r10_bio, int i) choose_data_offset(r10_bio, rdev) + (sector - r10_bio->sector)); wbio->bi_bdev = rdev->bdev; - wbio->bi_rw = WRITE; + wbio->bi_op = REQ_OP_WRITE; if (submit_bio_wait(wbio) < 0) /* Failure! */ @@ -2550,7 +2553,8 @@ read_more: bio->bi_iter.bi_sector = r10_bio->devs[slot].addr + choose_data_offset(r10_bio, rdev); bio->bi_bdev = rdev->bdev; - bio->bi_rw = READ | do_sync; + bio->bi_op = REQ_OP_READ; + bio->bi_rw = do_sync; bio->bi_private = r10_bio; bio->bi_end_io = raid10_end_read_request; if (max_sectors < r10_bio->sectors) { @@ -3037,7 +3041,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_read; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; from_addr = r10_bio->devs[j].addr; bio->bi_iter.bi_sector = from_addr + rdev->data_offset; @@ -3063,7 +3067,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_write; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_iter.bi_sector = to_addr + rdev->data_offset; bio->bi_bdev = rdev->bdev; @@ -3092,7 +3096,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_write; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_iter.bi_sector = to_addr + rdev->data_offset; bio->bi_bdev = rdev->bdev; @@ -3212,7 +3216,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_read; - bio->bi_rw = READ; + bio->bi_op = REQ_OP_READ; bio->bi_iter.bi_sector = sector + conf->mirrors[d].rdev->data_offset; bio->bi_bdev = conf->mirrors[d].rdev->bdev; @@ -3234,7 +3238,7 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, biolist = bio; bio->bi_private = r10_bio; bio->bi_end_io = end_sync_write; - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_iter.bi_sector = sector + conf->mirrors[d].replacement->data_offset; bio->bi_bdev = conf->mirrors[d].replacement->bdev; @@ -4317,7 +4321,7 @@ read_more: + rdev->data_offset); read_bio->bi_private = r10_bio; read_bio->bi_end_io = end_sync_read; - read_bio->bi_rw = READ; + read_bio->bi_op = REQ_OP_READ; read_bio->bi_flags &= (~0UL << BIO_RESET_BITS); read_bio->bi_error = 0; read_bio->bi_vcnt = 0; @@ -4351,7 +4355,7 @@ read_more: rdev2->new_data_offset; b->bi_private = r10_bio; b->bi_end_io = end_reshape_write; - b->bi_rw = WRITE; + b->bi_op = REQ_OP_WRITE; b->bi_next = blist; blist = b; } @@ -4519,7 +4523,7 @@ static int handle_reshape_read_error(struct mddev *mddev, addr, s << 9, bvec[idx].bv_page, - READ, false); + REQ_OP_READ, 0, false); if (success) break; failed: diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 517ffd7..c4624b3 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -261,7 +261,7 @@ static struct bio *r5l_bio_alloc(struct r5l_log *log) { struct bio *bio = bio_alloc_bioset(GFP_NOIO, BIO_MAX_PAGES, log->bs); - bio->bi_rw = WRITE; + bio->bi_op = REQ_OP_WRITE; bio->bi_bdev = log->rdev->bdev; bio->bi_iter.bi_sector = log->rdev->data_offset + log->log_start; @@ -691,6 +691,7 @@ void r5l_flush_stripe_to_raid(struct r5conf *conf) bio_reset(&log->flush_bio); log->flush_bio.bi_bdev = log->rdev->bdev; log->flush_bio.bi_end_io = r5l_log_flush_endio; + log->flush_bio.bi_op = REQ_OP_WRITE; log->flush_bio.bi_rw = WRITE_FLUSH; submit_bio(&log->flush_bio); } @@ -889,7 +890,8 @@ static int r5l_read_meta_block(struct r5l_log *log, struct r5l_meta_block *mb; u32 crc, stored_crc; - if (!sync_page_io(log->rdev, ctx->pos, PAGE_SIZE, page, READ, false)) + if (!sync_page_io(log->rdev, ctx->pos, PAGE_SIZE, page, REQ_OP_READ, 0, + false)) return -EIO; mb = page_address(page); @@ -934,7 +936,8 @@ static int r5l_recovery_flush_one_stripe(struct r5l_log *log, &disk_index, sh); sync_page_io(log->rdev, *log_offset, PAGE_SIZE, - sh->dev[disk_index].page, READ, false); + sh->dev[disk_index].page, REQ_OP_READ, 0, + false); sh->dev[disk_index].log_checksum = le32_to_cpu(payload->checksum[0]); set_bit(R5_Wantwrite, &sh->dev[disk_index].flags); @@ -942,7 +945,8 @@ static int r5l_recovery_flush_one_stripe(struct r5l_log *log, } else { disk_index = sh->pd_idx; sync_page_io(log->rdev, *log_offset, PAGE_SIZE, - sh->dev[disk_index].page, READ, false); + sh->dev[disk_index].page, REQ_OP_READ, 0, + false); sh->dev[disk_index].log_checksum = le32_to_cpu(payload->checksum[0]); set_bit(R5_Wantwrite, &sh->dev[disk_index].flags); @@ -952,7 +956,7 @@ static int r5l_recovery_flush_one_stripe(struct r5l_log *log, sync_page_io(log->rdev, r5l_ring_add(log, *log_offset, BLOCK_SECTORS), PAGE_SIZE, sh->dev[disk_index].page, - READ, false); + REQ_OP_READ, 0, false); sh->dev[disk_index].log_checksum = le32_to_cpu(payload->checksum[1]); set_bit(R5_Wantwrite, @@ -994,11 +998,13 @@ static int r5l_recovery_flush_one_stripe(struct r5l_log *log, rdev = rcu_dereference(conf->disks[disk_index].rdev); if (rdev) sync_page_io(rdev, stripe_sect, PAGE_SIZE, - sh->dev[disk_index].page, WRITE, false); + sh->dev[disk_index].page, REQ_OP_WRITE, 0, + false); rrdev = rcu_dereference(conf->disks[disk_index].replacement); if (rrdev) sync_page_io(rrdev, stripe_sect, PAGE_SIZE, - sh->dev[disk_index].page, WRITE, false); + sh->dev[disk_index].page, REQ_OP_WRITE, 0, + false); } raid5_release_stripe(sh); return 0; @@ -1070,7 +1076,8 @@ static int r5l_log_write_empty_meta_block(struct r5l_log *log, sector_t pos, crc = crc32c_le(log->uuid_checksum, mb, PAGE_SIZE); mb->checksum = cpu_to_le32(crc); - if (!sync_page_io(log->rdev, pos, PAGE_SIZE, page, WRITE_FUA, false)) { + if (!sync_page_io(log->rdev, pos, PAGE_SIZE, page, REQ_OP_WRITE, + WRITE_FUA, false)) { __free_page(page); return -EIO; } @@ -1145,7 +1152,7 @@ static int r5l_load_log(struct r5l_log *log) if (!page) return -ENOMEM; - if (!sync_page_io(rdev, cp, PAGE_SIZE, page, READ, false)) { + if (!sync_page_io(rdev, cp, PAGE_SIZE, page, REQ_OP_READ, 0, false)) { ret = -EIO; goto ioerr; } diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index e74ead1..1a5edba 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -815,7 +815,8 @@ static void stripe_add_to_batch_list(struct r5conf *conf, struct stripe_head *sh dd_idx = 0; while (dd_idx == sh->pd_idx || dd_idx == sh->qd_idx) dd_idx++; - if (head->dev[dd_idx].towrite->bi_rw != sh->dev[dd_idx].towrite->bi_rw) + if (head->dev[dd_idx].towrite->bi_rw != sh->dev[dd_idx].towrite->bi_rw || + head->dev[dd_idx].towrite->bi_op != sh->dev[dd_idx].towrite->bi_op) goto unlock_out; if (head->batch_head) { @@ -900,29 +901,32 @@ static void ops_run_io(struct stripe_head *sh, struct stripe_head_state *s) if (r5l_write_stripe(conf, sh) == 0) return; for (i = disks; i--; ) { - int rw; + int op; + int op_flags = 0; int replace_only = 0; struct bio *bi, *rbi; struct md_rdev *rdev, *rrdev = NULL; sh = head_sh; if (test_and_clear_bit(R5_Wantwrite, &sh->dev[i].flags)) { - if (test_and_clear_bit(R5_WantFUA, &sh->dev[i].flags)) - rw = WRITE_FUA; - else - rw = WRITE; + if (test_and_clear_bit(R5_WantFUA, &sh->dev[i].flags)) { + op = REQ_OP_WRITE; + op_flags = WRITE_FUA; + } else { + op = REQ_OP_WRITE; + } if (test_bit(R5_Discard, &sh->dev[i].flags)) - rw |= REQ_DISCARD; + op = REQ_OP_DISCARD; } else if (test_and_clear_bit(R5_Wantread, &sh->dev[i].flags)) - rw = READ; + op = REQ_OP_READ; else if (test_and_clear_bit(R5_WantReplace, &sh->dev[i].flags)) { - rw = WRITE; + op = REQ_OP_WRITE; replace_only = 1; } else continue; if (test_and_clear_bit(R5_SyncIO, &sh->dev[i].flags)) - rw |= REQ_SYNC; + op_flags |= REQ_SYNC; again: bi = &sh->dev[i].req; @@ -936,7 +940,7 @@ again: rdev = rrdev; rrdev = NULL; } - if (rw & WRITE) { + if (op_is_write(op)) { if (replace_only) rdev = NULL; if (rdev == rrdev) @@ -962,7 +966,7 @@ again: * need to check for writes. We never accept write errors * on the replacement, so we don't to check rrdev. */ - while ((rw & WRITE) && rdev && + while (op_is_write(op) && rdev && test_bit(WriteErrorSeen, &rdev->flags)) { sector_t first_bad; int bad_sectors; @@ -1004,8 +1008,9 @@ again: bio_reset(bi); bi->bi_bdev = rdev->bdev; - bi->bi_rw = rw; - bi->bi_end_io = (rw & WRITE) + bi->bi_op = op; + bi->bi_rw = op_flags; + bi->bi_end_io = op_is_write(op) ? raid5_end_write_request : raid5_end_read_request; bi->bi_private = sh; @@ -1036,7 +1041,7 @@ again: * If this is discard request, set bi_vcnt 0. We don't * want to confuse SCSI because SCSI will replace payload */ - if (rw & REQ_DISCARD) + if (op == REQ_OP_DISCARD) bi->bi_vcnt = 0; if (rrdev) set_bit(R5_DOUBLE_LOCKED, &sh->dev[i].flags); @@ -1056,8 +1061,9 @@ again: bio_reset(rbi); rbi->bi_bdev = rrdev->bdev; - rbi->bi_rw = rw; - BUG_ON(!(rw & WRITE)); + rbi->bi_op = op; + rbi->bi_rw = op_flags; + BUG_ON(!op_is_write(op)); rbi->bi_end_io = raid5_end_write_request; rbi->bi_private = sh; @@ -1085,7 +1091,7 @@ again: * If this is discard request, set bi_vcnt 0. We don't * want to confuse SCSI because SCSI will replace payload */ - if (rw & REQ_DISCARD) + if (op == REQ_OP_DISCARD) rbi->bi_vcnt = 0; if (conf->mddev->gendisk) trace_block_bio_remap(bdev_get_queue(rbi->bi_bdev), @@ -1094,7 +1100,7 @@ again: generic_make_request(rbi); } if (!rdev && !rrdev) { - if (rw & WRITE) + if (op_is_write(op)) set_bit(STRIPE_DEGRADED, &sh->state); pr_debug("skip op %ld on disc %d for sector %llu\n", bi->bi_rw, i, (unsigned long long)sh->sector); @@ -1632,7 +1638,7 @@ again: set_bit(R5_WantFUA, &dev->flags); if (wbi->bi_rw & REQ_SYNC) set_bit(R5_SyncIO, &dev->flags); - if (wbi->bi_rw & REQ_DISCARD) + if (wbi->bi_op == REQ_OP_DISCARD) set_bit(R5_Discard, &dev->flags); else { tx = async_copy_data(1, wbi, &dev->page, @@ -5173,7 +5179,7 @@ static void make_request(struct mddev *mddev, struct bio * bi) return; } - if (unlikely(bi->bi_rw & REQ_DISCARD)) { + if (unlikely(bi->bi_op == REQ_OP_DISCARD)) { make_discard_request(mddev, bi); return; } -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6567629E59 for ; Mon, 11 Jan 2016 14:22:41 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id 47A148F8049 for ; Mon, 11 Jan 2016 12:22:41 -0800 (PST) X-ASG-Debug-ID: 1452543759-04cb6c75db79090001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id xyd2CuGOupr1aKsk (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:40 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 88BA591FD2; Mon, 11 Jan 2016 20:22:39 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEs008385; Mon, 11 Jan 2016 15:22:37 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 25/35] target: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:23 -0600 X-ASG-Orig-Subj: [PATCH 25/35] target: set bi_op to REQ_OP Message-Id: <1452543693-4440-26-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543760 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the target modules set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie Acked-by: Nicholas Bellinger --- drivers/target/target_core_iblock.c | 38 ++++++++++++++++++++++--------------- drivers/target/target_core_pscsi.c | 2 +- 2 files changed, 24 insertions(+), 16 deletions(-) diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c index 7455df9..30f4f41 100644 --- a/drivers/target/target_core_iblock.c +++ b/drivers/target/target_core_iblock.c @@ -330,7 +330,8 @@ static void iblock_bio_done(struct bio *bio) } static struct bio * -iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, int rw) +iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, int op, + int op_flags) { struct iblock_dev *ib_dev = IBLOCK_DEV(cmd->se_dev); struct bio *bio; @@ -352,7 +353,8 @@ iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, int rw) bio->bi_private = cmd; bio->bi_end_io = &iblock_bio_done; bio->bi_iter.bi_sector = lba; - bio->bi_rw = rw; + bio->bi_op = op; + bio->bi_rw = op_flags; return bio; } @@ -458,7 +460,7 @@ iblock_execute_write_same(struct se_cmd *cmd) goto fail; cmd->priv = ibr; - bio = iblock_get_bio(cmd, block_lba, 1, WRITE); + bio = iblock_get_bio(cmd, block_lba, 1, REQ_OP_WRITE, 0); if (!bio) goto fail_free_ibr; @@ -471,7 +473,8 @@ iblock_execute_write_same(struct se_cmd *cmd) while (bio_add_page(bio, sg_page(sg), sg->length, sg->offset) != sg->length) { - bio = iblock_get_bio(cmd, block_lba, 1, WRITE); + bio = iblock_get_bio(cmd, block_lba, 1, REQ_OP_WRITE, + 0); if (!bio) goto fail_put_bios; @@ -657,7 +660,8 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, u32 sg_num = sgl_nents; sector_t block_lba; unsigned bio_cnt; - int rw = 0; + int op_flags = 0; + int op = 0; int i; if (data_direction == DMA_TO_DEVICE) { @@ -668,17 +672,20 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, * is not enabled, or if initiator set the Force Unit Access bit. */ if (q->flush_flags & REQ_FUA) { - if (cmd->se_cmd_flags & SCF_FUA) - rw = WRITE_FUA; - else if (!(q->flush_flags & REQ_FLUSH)) - rw = WRITE_FUA; - else - rw = WRITE; + if (cmd->se_cmd_flags & SCF_FUA) { + op = REQ_OP_WRITE; + op_flags = WRITE_FUA; + } else if (!(q->flush_flags & REQ_FLUSH)) { + op = REQ_OP_WRITE; + op_flags = WRITE_FUA; + } else { + op = REQ_OP_WRITE; + } } else { - rw = WRITE; + op = REQ_OP_WRITE; } } else { - rw = READ; + op = REQ_OP_READ; } /* @@ -710,7 +717,7 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, return 0; } - bio = iblock_get_bio(cmd, block_lba, sgl_nents, rw); + bio = iblock_get_bio(cmd, block_lba, sgl_nents, op, op_flags); if (!bio) goto fail_free_ibr; @@ -734,7 +741,8 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, bio_cnt = 0; } - bio = iblock_get_bio(cmd, block_lba, sg_num, rw); + bio = iblock_get_bio(cmd, block_lba, sg_num, op, + op_flags); if (!bio) goto fail_put_bios; diff --git a/drivers/target/target_core_pscsi.c b/drivers/target/target_core_pscsi.c index de18790..2cf915c 100644 --- a/drivers/target/target_core_pscsi.c +++ b/drivers/target/target_core_pscsi.c @@ -922,7 +922,7 @@ pscsi_map_sg(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, goto fail; if (rw) - bio->bi_rw |= REQ_WRITE; + bio->bi_op = REQ_OP_WRITE; pr_debug("PSCSI: Allocated bio: %p," " dir: %s nr_vecs: %d\n", bio, -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:41 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 9F6C029E59 for ; Mon, 11 Jan 2016 14:22:41 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id 64A018F804B for ; Mon, 11 Jan 2016 12:22:41 -0800 (PST) X-ASG-Debug-ID: 1452543757-04bdf06f6b7db30001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id JelTrIsx9PLdhSg2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:37 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 41A4BC0BF2AC; Mon, 11 Jan 2016 20:22:37 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEr008385; Mon, 11 Jan 2016 15:22:34 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 24/35] xen: set bi_op to REQ_OP Date: Mon, 11 Jan 2016 14:21:22 -0600 X-ASG-Orig-Subj: [PATCH 24/35] xen: set bi_op to REQ_OP Message-Id: <1452543693-4440-25-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543757 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has xen set the bio bi_op to a REQ_OP, and rq_flag_bits to bi_rw. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/block/xen-blkback/blkback.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index 733d397..f977d29 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -488,7 +488,7 @@ static int xen_vbd_translate(struct phys_req *req, struct xen_blkif *blkif, struct xen_vbd *vbd = &blkif->vbd; int rc = -EACCES; - if ((operation != READ) && vbd->readonly) + if ((operation != REQ_OP_READ) && vbd->readonly) goto out; if (likely(req->nr_sects)) { @@ -995,7 +995,7 @@ static int dispatch_discard_io(struct xen_blkif *blkif, preq.sector_number = req->u.discard.sector_number; preq.nr_sects = req->u.discard.nr_sectors; - err = xen_vbd_translate(&preq, blkif, WRITE); + err = xen_vbd_translate(&preq, blkif, REQ_OP_WRITE); if (err) { pr_warn("access denied: DISCARD [%llu->%llu] on dev=%04x\n", preq.sector_number, @@ -1208,6 +1208,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, struct bio **biolist = pending_req->biolist; int i, nbio = 0; int operation; + int operation_flags = 0; struct blk_plug plug; bool drain = false; struct grant_page **pages = pending_req->segments; @@ -1226,17 +1227,19 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, switch (req_operation) { case BLKIF_OP_READ: blkif->st_rd_req++; - operation = READ; + operation = REQ_OP_READ; break; case BLKIF_OP_WRITE: blkif->st_wr_req++; - operation = WRITE_ODIRECT; + operation = REQ_OP_WRITE; + operation_flags = WRITE_ODIRECT; break; case BLKIF_OP_WRITE_BARRIER: drain = true; case BLKIF_OP_FLUSH_DISKCACHE: blkif->st_f_req++; - operation = WRITE_FLUSH; + operation = REQ_OP_WRITE; + operation_flags = WRITE_FLUSH; break; default: operation = 0; /* make gcc happy */ @@ -1248,7 +1251,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, nseg = req->operation == BLKIF_OP_INDIRECT ? req->u.indirect.nr_segments : req->u.rw.nr_segments; - if (unlikely(nseg == 0 && operation != WRITE_FLUSH) || + if (unlikely(nseg == 0 && operation_flags != WRITE_FLUSH) || unlikely((req->operation != BLKIF_OP_INDIRECT) && (nseg > BLKIF_MAX_SEGMENTS_PER_REQUEST)) || unlikely((req->operation == BLKIF_OP_INDIRECT) && @@ -1289,7 +1292,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, if (xen_vbd_translate(&preq, blkif, operation) != 0) { pr_debug("access denied: %s of [%llu,%llu] on dev=%04x\n", - operation == READ ? "read" : "write", + operation == REQ_OP_READ ? "read" : "write", preq.sector_number, preq.sector_number + preq.nr_sects, blkif->vbd.pdevice); @@ -1348,7 +1351,8 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, bio->bi_private = pending_req; bio->bi_end_io = end_block_io_op; bio->bi_iter.bi_sector = preq.sector_number; - bio->bi_rw = operation; + bio->bi_op = operation; + bio->bi_rw = operation_flags; } preq.sector_number += seg[i].nsec; @@ -1356,7 +1360,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, /* This will be hit if the operation was a flush or discard. */ if (!bio) { - BUG_ON(operation != WRITE_FLUSH); + BUG_ON(operation_flags != WRITE_FLUSH); bio = bio_alloc(GFP_KERNEL, 0); if (unlikely(bio == NULL)) @@ -1366,7 +1370,8 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, bio->bi_bdev = preq.bdev; bio->bi_private = pending_req; bio->bi_end_io = end_block_io_op; - bio->bi_rw = operation; + bio->bi_op = operation; + bio->bi_rw = operation_flags; } atomic_set(&pending_req->pendcnt, nbio); @@ -1378,9 +1383,9 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif, /* Let the I/Os go.. */ blk_finish_plug(&plug); - if (operation == READ) + if (operation == REQ_OP_READ) blkif->st_rd_sect += preq.nr_sects; - else if (operation & WRITE) + else if (operation == REQ_OP_WRITE) blkif->st_wr_sect += preq.nr_sects; return 0; -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:46 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 22E917F37 for ; Mon, 11 Jan 2016 14:22:46 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 7A03DAC005 for ; Mon, 11 Jan 2016 12:22:45 -0800 (PST) X-ASG-Debug-ID: 1452543762-04cbb05f7d97b70001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id GYYmwDgWuSiQd9G1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:42 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id D7BBBC1C7516; Mon, 11 Jan 2016 20:22:41 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEt008385; Mon, 11 Jan 2016 15:22:39 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 26/35] block: set op to REQ_OP Date: Mon, 11 Jan 2016 14:21:24 -0600 X-ASG-Orig-Subj: [PATCH 26/35] block: set op to REQ_OP Message-Id: <1452543693-4440-27-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543762 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch converts the request related block layer code to set request->op to a REQ_OP and cmd_flags to rq_flag_bits. There is some tmp compat code when setting up cmd_flags so it still carries both the op and flags. It will be removed in in later patches in this set when I have converted all drivers. I have not been able to test the mq paths with real mq hardware. Signed-off-by: Mike Christie --- block/blk-core.c | 60 ++++++++++++++++++++++++++-------------------- block/blk-flush.c | 1 + block/blk-merge.c | 10 ++++---- block/blk-mq.c | 38 ++++++++++++++++------------- block/cfq-iosched.c | 53 +++++++++++++++++++++++----------------- block/elevator.c | 8 +++---- include/linux/blk-cgroup.h | 13 +++++----- include/linux/blkdev.h | 28 +++++++++++----------- include/linux/elevator.h | 4 ++-- 9 files changed, 120 insertions(+), 95 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 954a450..dacbd68 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -959,10 +959,10 @@ static void __freed_request(struct request_list *rl, int sync) * A request has just been released. Account for it, update the full and * congestion status, wake up any waiters. Called under q->queue_lock. */ -static void freed_request(struct request_list *rl, unsigned int flags) +static void freed_request(struct request_list *rl, int op, unsigned int flags) { struct request_queue *q = rl->q; - int sync = rw_is_sync(flags); + int sync = rw_is_sync(op, flags); q->nr_rqs[sync]--; rl->count[sync]--; @@ -1054,7 +1054,8 @@ static struct io_context *rq_ioc(struct bio *bio) /** * __get_request - get a free request * @rl: request list to allocate from - * @rw_flags: RW and SYNC flags + * @op: REQ_OP_READ/REQ_OP_WRITE + * @op_flags: rq_flag_bits * @bio: bio to allocate request for (can be %NULL) * @gfp_mask: allocation mask * @@ -1065,21 +1066,22 @@ static struct io_context *rq_ioc(struct bio *bio) * Returns ERR_PTR on failure, with @q->queue_lock held. * Returns request pointer on success, with @q->queue_lock *not held*. */ -static struct request *__get_request(struct request_list *rl, int rw_flags, - struct bio *bio, gfp_t gfp_mask) +static struct request *__get_request(struct request_list *rl, int op, + int op_flags, struct bio *bio, + gfp_t gfp_mask) { struct request_queue *q = rl->q; struct request *rq; struct elevator_type *et = q->elevator->type; struct io_context *ioc = rq_ioc(bio); struct io_cq *icq = NULL; - const bool is_sync = rw_is_sync(rw_flags) != 0; + const bool is_sync = rw_is_sync(op, op_flags) != 0; int may_queue; if (unlikely(blk_queue_dying(q))) return ERR_PTR(-ENODEV); - may_queue = elv_may_queue(q, rw_flags); + may_queue = elv_may_queue(q, op, op_flags); if (may_queue == ELV_MQUEUE_NO) goto rq_starved; @@ -1123,7 +1125,7 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, /* * Decide whether the new request will be managed by elevator. If - * so, mark @rw_flags and increment elvpriv. Non-zero elvpriv will + * so, mark @op_flags and increment elvpriv. Non-zero elvpriv will * prevent the current elevator from being destroyed until the new * request is freed. This guarantees icq's won't be destroyed and * makes creating new ones safe. @@ -1132,14 +1134,14 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, * it will be created after releasing queue_lock. */ if (blk_rq_should_init_elevator(bio) && !blk_queue_bypass(q)) { - rw_flags |= REQ_ELVPRIV; + op_flags |= REQ_ELVPRIV; q->nr_rqs_elvpriv++; if (et->icq_cache && ioc) icq = ioc_lookup_icq(ioc, q); } if (blk_queue_io_stat(q)) - rw_flags |= REQ_IO_STAT; + op_flags |= REQ_IO_STAT; spin_unlock_irq(q->queue_lock); /* allocate and init request */ @@ -1149,10 +1151,12 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, blk_rq_init(q, rq); blk_rq_set_rl(rq, rl); - rq->cmd_flags = rw_flags | REQ_ALLOCED; + /* tmp compat - allow users to check either one for the op */ + rq->cmd_flags = op | op_flags | REQ_ALLOCED; + rq->op = op; /* init elvpriv */ - if (rw_flags & REQ_ELVPRIV) { + if (op_flags & REQ_ELVPRIV) { if (unlikely(et->icq_cache && !icq)) { if (ioc) icq = ioc_create_icq(ioc, q, gfp_mask); @@ -1178,7 +1182,7 @@ out: if (ioc_batching(q, ioc)) ioc->nr_batch_requests--; - trace_block_getrq(q, bio, rw_flags & 1); + trace_block_getrq(q, bio, op); return rq; fail_elvpriv: @@ -1208,7 +1212,7 @@ fail_alloc: * queue, but this is pretty rare. */ spin_lock_irq(q->queue_lock); - freed_request(rl, rw_flags); + freed_request(rl, op, op_flags); /* * in the very unlikely event that allocation failed and no @@ -1226,7 +1230,8 @@ rq_starved: /** * get_request - get a free request * @q: request_queue to allocate request from - * @rw_flags: RW and SYNC flags + * op: REQ_OP_READ/REQ_OP_WRITE + * @op_flags: rq_flag_bits * @bio: bio to allocate request for (can be %NULL) * @gfp_mask: allocation mask * @@ -1237,17 +1242,18 @@ rq_starved: * Returns ERR_PTR on failure, with @q->queue_lock held. * Returns request pointer on success, with @q->queue_lock *not held*. */ -static struct request *get_request(struct request_queue *q, int rw_flags, - struct bio *bio, gfp_t gfp_mask) +static struct request *get_request(struct request_queue *q, int op, + int op_flags, struct bio *bio, + gfp_t gfp_mask) { - const bool is_sync = rw_is_sync(rw_flags) != 0; + const bool is_sync = rw_is_sync(op, op_flags) != 0; DEFINE_WAIT(wait); struct request_list *rl; struct request *rq; rl = blk_get_rl(q, bio); /* transferred to @rq on success */ retry: - rq = __get_request(rl, rw_flags, bio, gfp_mask); + rq = __get_request(rl, op, op_flags, bio, gfp_mask); if (!IS_ERR(rq)) return rq; @@ -1260,7 +1266,7 @@ retry: prepare_to_wait_exclusive(&rl->wait[is_sync], &wait, TASK_UNINTERRUPTIBLE); - trace_block_sleeprq(q, bio, rw_flags & 1); + trace_block_sleeprq(q, bio, op); spin_unlock_irq(q->queue_lock); io_schedule(); @@ -1289,7 +1295,7 @@ static struct request *blk_old_get_request(struct request_queue *q, int rw, create_io_context(gfp_mask, q->node); spin_lock_irq(q->queue_lock); - rq = get_request(q, rw, NULL, gfp_mask); + rq = get_request(q, rw, 0, NULL, gfp_mask); if (IS_ERR(rq)) spin_unlock_irq(q->queue_lock); /* q->queue_lock is unlocked at this point */ @@ -1491,13 +1497,14 @@ void __blk_put_request(struct request_queue *q, struct request *req) */ if (req->cmd_flags & REQ_ALLOCED) { unsigned int flags = req->cmd_flags; + int op = req->op; struct request_list *rl = blk_rq_rl(req); BUG_ON(!list_empty(&req->queuelist)); BUG_ON(ELV_ON_HASH(req)); blk_free_request(rl, req); - freed_request(rl, flags); + freed_request(rl, op, flags); blk_put_rl(rl); } } @@ -1712,7 +1719,7 @@ static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio) { const bool sync = !!(bio->bi_rw & REQ_SYNC); struct blk_plug *plug; - int el_ret, rw_flags, where = ELEVATOR_INSERT_SORT; + int el_ret, rw_flags = 0, where = ELEVATOR_INSERT_SORT; struct request *req; unsigned int request_count = 0; @@ -1772,7 +1779,6 @@ get_rq: * but we need to set it earlier to expose the sync flag to the * rq allocator and io schedulers. */ - rw_flags = bio_data_dir(bio); if (sync) rw_flags |= REQ_SYNC; @@ -1780,7 +1786,7 @@ get_rq: * Grab a free request. This is might sleep but can not fail. * Returns with the queue unlocked. */ - req = get_request(q, rw_flags, bio, GFP_NOIO); + req = get_request(q, bio_data_dir(bio), rw_flags, bio, GFP_NOIO); if (IS_ERR(req)) { bio->bi_error = PTR_ERR(req); bio_endio(bio); @@ -2168,7 +2174,7 @@ EXPORT_SYMBOL(submit_bio); static int blk_cloned_rq_check_limits(struct request_queue *q, struct request *rq) { - if (blk_rq_sectors(rq) > blk_queue_get_max_sectors(q, rq->cmd_flags)) { + if (blk_rq_sectors(rq) > blk_queue_get_max_sectors(q, rq->op)) { printk(KERN_ERR "%s: over max size limit.\n", __func__); return -EIO; } @@ -2987,6 +2993,7 @@ void blk_rq_bio_prep(struct request_queue *q, struct request *rq, { /* tmp compat. Allow users to set bi_op or bi_rw */ rq->cmd_flags |= bio_data_dir(bio); + rq->op = bio->bi_op; if (bio_has_data(bio)) rq->nr_phys_segments = bio_phys_segments(q, bio); @@ -3071,6 +3078,7 @@ EXPORT_SYMBOL_GPL(blk_rq_unprep_clone); static void __blk_rq_prep_clone(struct request *dst, struct request *src) { dst->cpu = src->cpu; + dst->op = src->op; dst->cmd_flags |= (src->cmd_flags & REQ_CLONE_MASK) | REQ_NOMERGE; dst->cmd_type = src->cmd_type; dst->__sector = blk_rq_pos(src); diff --git a/block/blk-flush.c b/block/blk-flush.c index b05acca..e01d3ac 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -330,6 +330,7 @@ static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) flush_rq->cmd_type = REQ_TYPE_FS; flush_rq->cmd_flags = WRITE_FLUSH | REQ_FLUSH_SEQ; + flush_rq->op = REQ_OP_WRITE; flush_rq->rq_disk = first_rq->rq_disk; flush_rq->end_io = flush_end_io; diff --git a/block/blk-merge.c b/block/blk-merge.c index b66f095..de73ed1 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -628,7 +628,8 @@ static int attempt_merge(struct request_queue *q, struct request *req, if (!rq_mergeable(req) || !rq_mergeable(next)) return 0; - if (!blk_check_merge_flags(req->cmd_flags, next->cmd_flags)) + if (!blk_check_merge_flags(req->cmd_flags, req->op, next->cmd_flags, + next->op)) return 0; /* @@ -642,7 +643,7 @@ static int attempt_merge(struct request_queue *q, struct request *req, || req_no_special_merge(next)) return 0; - if (req->cmd_flags & REQ_WRITE_SAME && + if (req->op == REQ_OP_WRITE_SAME && !blk_write_same_mergeable(req->bio, next->bio)) return 0; @@ -730,7 +731,8 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) if (!rq_mergeable(rq) || !bio_mergeable(bio)) return false; - if (!blk_check_merge_flags(rq->cmd_flags, bio->bi_rw)) + if (!blk_check_merge_flags(rq->cmd_flags, rq->op, bio->bi_rw, + bio->bi_op)) return false; /* different data direction or already started, don't merge */ @@ -746,7 +748,7 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) return false; /* must be using the same buffer */ - if (rq->cmd_flags & REQ_WRITE_SAME && + if (rq->op == REQ_OP_WRITE_SAME && !blk_write_same_mergeable(rq->bio, bio)) return false; diff --git a/block/blk-mq.c b/block/blk-mq.c index 4c0622f..745dae8 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -159,16 +159,19 @@ bool blk_mq_can_queue(struct blk_mq_hw_ctx *hctx) EXPORT_SYMBOL(blk_mq_can_queue); static void blk_mq_rq_ctx_init(struct request_queue *q, struct blk_mq_ctx *ctx, - struct request *rq, unsigned int rw_flags) + struct request *rq, int op, + unsigned int op_flags) { if (blk_queue_io_stat(q)) - rw_flags |= REQ_IO_STAT; + op_flags |= REQ_IO_STAT; INIT_LIST_HEAD(&rq->queuelist); /* csd/requeue_work/fifo_time is initialized before use */ rq->q = q; rq->mq_ctx = ctx; - rq->cmd_flags |= rw_flags; + rq->op = op; + /* tmp compat - allow users to check either one for the op */ + rq->cmd_flags |= op | op_flags; /* do not touch atomic flags, it needs atomic ops against the timer */ rq->cpu = -1; INIT_HLIST_NODE(&rq->hash); @@ -203,11 +206,11 @@ static void blk_mq_rq_ctx_init(struct request_queue *q, struct blk_mq_ctx *ctx, rq->end_io_data = NULL; rq->next_rq = NULL; - ctx->rq_dispatched[rw_is_sync(rw_flags)]++; + ctx->rq_dispatched[rw_is_sync(op, op_flags)]++; } static struct request * -__blk_mq_alloc_request(struct blk_mq_alloc_data *data, int rw) +__blk_mq_alloc_request(struct blk_mq_alloc_data *data, int op, int op_flags) { struct request *rq; unsigned int tag; @@ -222,7 +225,7 @@ __blk_mq_alloc_request(struct blk_mq_alloc_data *data, int rw) } rq->tag = tag; - blk_mq_rq_ctx_init(data->q, data->ctx, rq, rw); + blk_mq_rq_ctx_init(data->q, data->ctx, rq, op, op_flags); return rq; } @@ -246,7 +249,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, hctx = q->mq_ops->map_queue(q, ctx->cpu); blk_mq_set_alloc_data(&alloc_data, q, flags, ctx, hctx); - rq = __blk_mq_alloc_request(&alloc_data, rw); + rq = __blk_mq_alloc_request(&alloc_data, rw, 0); if (!rq && !(flags & BLK_MQ_REQ_NOWAIT)) { __blk_mq_run_hw_queue(hctx); blk_mq_put_ctx(ctx); @@ -254,7 +257,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, ctx = blk_mq_get_ctx(q); hctx = q->mq_ops->map_queue(q, ctx->cpu); blk_mq_set_alloc_data(&alloc_data, q, flags, ctx, hctx); - rq = __blk_mq_alloc_request(&alloc_data, rw); + rq = __blk_mq_alloc_request(&alloc_data, rw, 0); ctx = alloc_data.ctx; } blk_mq_put_ctx(ctx); @@ -1165,28 +1168,29 @@ static struct request *blk_mq_map_request(struct request_queue *q, struct blk_mq_hw_ctx *hctx; struct blk_mq_ctx *ctx; struct request *rq; - int rw = bio_data_dir(bio); + int op = bio_data_dir(bio); + int op_flags = 0; struct blk_mq_alloc_data alloc_data; blk_queue_enter_live(q); ctx = blk_mq_get_ctx(q); hctx = q->mq_ops->map_queue(q, ctx->cpu); - if (rw_is_sync(bio->bi_rw)) - rw |= REQ_SYNC; + if (rw_is_sync(bio->bi_op, bio->bi_rw)) + op_flags |= REQ_SYNC; - trace_block_getrq(q, bio, rw); + trace_block_getrq(q, bio, op); blk_mq_set_alloc_data(&alloc_data, q, BLK_MQ_REQ_NOWAIT, ctx, hctx); - rq = __blk_mq_alloc_request(&alloc_data, rw); + rq = __blk_mq_alloc_request(&alloc_data, op, op_flags); if (unlikely(!rq)) { __blk_mq_run_hw_queue(hctx); blk_mq_put_ctx(ctx); - trace_block_sleeprq(q, bio, rw); + trace_block_sleeprq(q, bio, op); ctx = blk_mq_get_ctx(q); hctx = q->mq_ops->map_queue(q, ctx->cpu); blk_mq_set_alloc_data(&alloc_data, q, 0, ctx, hctx); - rq = __blk_mq_alloc_request(&alloc_data, rw); + rq = __blk_mq_alloc_request(&alloc_data, op, op_flags); ctx = alloc_data.ctx; hctx = alloc_data.hctx; } @@ -1240,7 +1244,7 @@ static int blk_mq_direct_issue_request(struct request *rq, blk_qc_t *cookie) */ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) { - const int is_sync = rw_is_sync(bio->bi_rw); + const int is_sync = rw_is_sync(bio->bi_op, bio->bi_rw); const int is_flush_fua = bio->bi_rw & (REQ_FLUSH | REQ_FUA); struct blk_map_ctx data; struct request *rq; @@ -1337,7 +1341,7 @@ done: */ static blk_qc_t blk_sq_make_request(struct request_queue *q, struct bio *bio) { - const int is_sync = rw_is_sync(bio->bi_rw); + const int is_sync = rw_is_sync(bio->bi_op, bio->bi_rw); const int is_flush_fua = bio->bi_rw & (REQ_FLUSH | REQ_FUA); struct blk_plug *plug; unsigned int request_count = 0; diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c index 1f9093e..d300aa1 100644 --- a/block/cfq-iosched.c +++ b/block/cfq-iosched.c @@ -660,9 +660,10 @@ static inline void cfqg_put(struct cfq_group *cfqg) } while (0) static inline void cfqg_stats_update_io_add(struct cfq_group *cfqg, - struct cfq_group *curr_cfqg, int rw) + struct cfq_group *curr_cfqg, int op, + int op_flags) { - blkg_rwstat_add(&cfqg->stats.queued, rw, 1); + blkg_rwstat_add(&cfqg->stats.queued, op, op_flags, 1); cfqg_stats_end_empty_time(&cfqg->stats); cfqg_stats_set_start_group_wait_time(cfqg, curr_cfqg); } @@ -676,26 +677,30 @@ static inline void cfqg_stats_update_timeslice_used(struct cfq_group *cfqg, #endif } -static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, int rw) +static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, int op, + int op_flags) { - blkg_rwstat_add(&cfqg->stats.queued, rw, -1); + blkg_rwstat_add(&cfqg->stats.queued, op, op_flags, -1); } -static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, int rw) +static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, int op, + int op_flags) { - blkg_rwstat_add(&cfqg->stats.merged, rw, 1); + blkg_rwstat_add(&cfqg->stats.merged, op, op_flags, 1); } static inline void cfqg_stats_update_completion(struct cfq_group *cfqg, - uint64_t start_time, uint64_t io_start_time, int rw) + uint64_t start_time, uint64_t io_start_time, int op, + int op_flags) { struct cfqg_stats *stats = &cfqg->stats; unsigned long long now = sched_clock(); if (time_after64(now, io_start_time)) - blkg_rwstat_add(&stats->service_time, rw, now - io_start_time); + blkg_rwstat_add(&stats->service_time, op, op_flags, + now - io_start_time); if (time_after64(io_start_time, start_time)) - blkg_rwstat_add(&stats->wait_time, rw, + blkg_rwstat_add(&stats->wait_time, op, op_flags, io_start_time - start_time); } @@ -769,13 +774,16 @@ static inline void cfqg_put(struct cfq_group *cfqg) { } #define cfq_log_cfqg(cfqd, cfqg, fmt, args...) do {} while (0) static inline void cfqg_stats_update_io_add(struct cfq_group *cfqg, - struct cfq_group *curr_cfqg, int rw) { } + struct cfq_group *curr_cfqg, int op, int op_flags) { } static inline void cfqg_stats_update_timeslice_used(struct cfq_group *cfqg, unsigned long time, unsigned long unaccounted_time) { } -static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, int rw) { } -static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, int rw) { } +static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, int op, + int op_flags) { } +static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, int op, + int op_flags) { } static inline void cfqg_stats_update_completion(struct cfq_group *cfqg, - uint64_t start_time, uint64_t io_start_time, int rw) { } + uint64_t start_time, uint64_t io_start_time, int op, + int op_flags) { } #endif /* CONFIG_CFQ_GROUP_IOSCHED */ @@ -2449,10 +2457,10 @@ static void cfq_reposition_rq_rb(struct cfq_queue *cfqq, struct request *rq) { elv_rb_del(&cfqq->sort_list, rq); cfqq->queued[rq_is_sync(rq)]--; - cfqg_stats_update_io_remove(RQ_CFQG(rq), rq->cmd_flags); + cfqg_stats_update_io_remove(RQ_CFQG(rq), rq->op, rq->cmd_flags); cfq_add_rq_rb(rq); cfqg_stats_update_io_add(RQ_CFQG(rq), cfqq->cfqd->serving_group, - rq->cmd_flags); + rq->op, rq->cmd_flags); } static struct request * @@ -2505,7 +2513,7 @@ static void cfq_remove_request(struct request *rq) cfq_del_rq_rb(rq); cfqq->cfqd->rq_queued--; - cfqg_stats_update_io_remove(RQ_CFQG(rq), rq->cmd_flags); + cfqg_stats_update_io_remove(RQ_CFQG(rq), rq->op, rq->cmd_flags); if (rq->cmd_flags & REQ_PRIO) { WARN_ON(!cfqq->prio_pending); cfqq->prio_pending--; @@ -2540,7 +2548,7 @@ static void cfq_merged_request(struct request_queue *q, struct request *req, static void cfq_bio_merged(struct request_queue *q, struct request *req, struct bio *bio) { - cfqg_stats_update_io_merged(RQ_CFQG(req), bio->bi_rw); + cfqg_stats_update_io_merged(RQ_CFQG(req), bio->bi_op, bio->bi_rw); } static void @@ -2563,7 +2571,7 @@ cfq_merged_requests(struct request_queue *q, struct request *rq, if (cfqq->next_rq == next) cfqq->next_rq = rq; cfq_remove_request(next); - cfqg_stats_update_io_merged(RQ_CFQG(rq), next->cmd_flags); + cfqg_stats_update_io_merged(RQ_CFQG(rq), next->op, next->cmd_flags); cfqq = RQ_CFQQ(next); /* @@ -4085,7 +4093,7 @@ static void cfq_insert_request(struct request_queue *q, struct request *rq) rq->fifo_time = jiffies + cfqd->cfq_fifo_expire[rq_is_sync(rq)]; list_add_tail(&rq->queuelist, &cfqq->fifo); cfq_add_rq_rb(rq); - cfqg_stats_update_io_add(RQ_CFQG(rq), cfqd->serving_group, + cfqg_stats_update_io_add(RQ_CFQG(rq), cfqd->serving_group, rq->op, rq->cmd_flags); cfq_rq_enqueued(cfqd, cfqq, rq); } @@ -4183,7 +4191,8 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq) cfqq->dispatched--; (RQ_CFQG(rq))->dispatched--; cfqg_stats_update_completion(cfqq->cfqg, rq_start_time_ns(rq), - rq_io_start_time_ns(rq), rq->cmd_flags); + rq_io_start_time_ns(rq), rq->op, + rq->cmd_flags); cfqd->rq_in_flight[cfq_cfqq_sync(cfqq)]--; @@ -4262,7 +4271,7 @@ static inline int __cfq_may_queue(struct cfq_queue *cfqq) return ELV_MQUEUE_MAY; } -static int cfq_may_queue(struct request_queue *q, int rw) +static int cfq_may_queue(struct request_queue *q, int op, int op_flags) { struct cfq_data *cfqd = q->elevator->elevator_data; struct task_struct *tsk = current; @@ -4279,7 +4288,7 @@ static int cfq_may_queue(struct request_queue *q, int rw) if (!cic) return ELV_MQUEUE_MAY; - cfqq = cic_to_cfqq(cic, rw_is_sync(rw)); + cfqq = cic_to_cfqq(cic, rw_is_sync(op, op_flags)); if (cfqq) { cfq_init_prio_data(cfqq, cic); diff --git a/block/elevator.c b/block/elevator.c index c3555c9..f6279ca 100644 --- a/block/elevator.c +++ b/block/elevator.c @@ -366,8 +366,8 @@ void elv_dispatch_sort(struct request_queue *q, struct request *rq) list_for_each_prev(entry, &q->queue_head) { struct request *pos = list_entry_rq(entry); - if ((rq->cmd_flags & REQ_DISCARD) != - (pos->cmd_flags & REQ_DISCARD)) + if ((rq->op == REQ_OP_DISCARD) != + (pos->op == REQ_OP_DISCARD)) break; if (rq_data_dir(rq) != rq_data_dir(pos)) break; @@ -717,12 +717,12 @@ void elv_put_request(struct request_queue *q, struct request *rq) e->type->ops.elevator_put_req_fn(rq); } -int elv_may_queue(struct request_queue *q, int rw) +int elv_may_queue(struct request_queue *q, int op, int op_flags) { struct elevator_queue *e = q->elevator; if (e->type->ops.elevator_may_queue_fn) - return e->type->ops.elevator_may_queue_fn(q, rw); + return e->type->ops.elevator_may_queue_fn(q, op, op_flags); return ELV_MQUEUE_MAY; } diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index c02e669..9071feb 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -590,25 +590,26 @@ static inline void blkg_rwstat_exit(struct blkg_rwstat *rwstat) /** * blkg_rwstat_add - add a value to a blkg_rwstat * @rwstat: target blkg_rwstat - * @rw: mask of REQ_{WRITE|SYNC} + * @op: REQ_OP + * @op_flags: rq_flag_bits * @val: value to add * * Add @val to @rwstat. The counters are chosen according to @rw. The * caller is responsible for synchronizing calls to this function. */ static inline void blkg_rwstat_add(struct blkg_rwstat *rwstat, - int rw, uint64_t val) + int op, int op_flags, uint64_t val) { struct percpu_counter *cnt; - if (rw & REQ_WRITE) + if (op_is_write(op)) cnt = &rwstat->cpu_cnt[BLKG_RWSTAT_WRITE]; else cnt = &rwstat->cpu_cnt[BLKG_RWSTAT_READ]; __percpu_counter_add(cnt, val, BLKG_STAT_CPU_BATCH); - if (rw & REQ_SYNC) + if (op_flags & REQ_SYNC) cnt = &rwstat->cpu_cnt[BLKG_RWSTAT_SYNC]; else cnt = &rwstat->cpu_cnt[BLKG_RWSTAT_ASYNC]; @@ -713,9 +714,9 @@ static inline bool blkcg_bio_issue_check(struct request_queue *q, if (!throtl) { blkg = blkg ?: q->root_blkg; - blkg_rwstat_add(&blkg->stat_bytes, bio->bi_rw, + blkg_rwstat_add(&blkg->stat_bytes, bio->bi_op, bio->bi_rw, bio->bi_iter.bi_size); - blkg_rwstat_add(&blkg->stat_ios, bio->bi_rw, 1); + blkg_rwstat_add(&blkg->stat_ios, bio->bi_op, bio->bi_rw, 1); } rcu_read_unlock(); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 35b9eb3..bc024c7 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -596,7 +596,7 @@ static inline void queue_flag_clear(unsigned int flag, struct request_queue *q) #define list_entry_rq(ptr) list_entry((ptr), struct request, queuelist) -#define rq_data_dir(rq) ((int)((rq)->cmd_flags & 1)) +#define rq_data_dir(rq) (op_is_write(rq->op) ? WRITE : READ) /* * Driver can handle struct request, if it either has an old style @@ -615,14 +615,14 @@ static inline unsigned int blk_queue_cluster(struct request_queue *q) /* * We regard a request as sync, if either a read or a sync write */ -static inline bool rw_is_sync(unsigned int rw_flags) +static inline bool rw_is_sync(int op, unsigned int rw_flags) { - return !(rw_flags & REQ_WRITE) || (rw_flags & REQ_SYNC); + return op == REQ_OP_READ || (rw_flags & REQ_SYNC); } static inline bool rq_is_sync(struct request *rq) { - return rw_is_sync(rq->cmd_flags); + return rw_is_sync(rq->op, rq->cmd_flags); } static inline bool blk_rl_full(struct request_list *rl, bool sync) @@ -657,16 +657,16 @@ static inline bool rq_mergeable(struct request *rq) return true; } -static inline bool blk_check_merge_flags(unsigned int flags1, - unsigned int flags2) +static inline bool blk_check_merge_flags(unsigned int flags1, unsigned int op1, + unsigned int flags2, unsigned int op2) { - if ((flags1 & REQ_DISCARD) != (flags2 & REQ_DISCARD)) + if ((op1 == REQ_OP_DISCARD) != (op2 == REQ_OP_DISCARD)) return false; if ((flags1 & REQ_SECURE) != (flags2 & REQ_SECURE)) return false; - if ((flags1 & REQ_WRITE_SAME) != (flags2 & REQ_WRITE_SAME)) + if ((op1 == REQ_OP_WRITE_SAME) != (op2 == REQ_OP_WRITE_SAME)) return false; return true; @@ -864,12 +864,12 @@ static inline unsigned int blk_rq_cur_sectors(const struct request *rq) } static inline unsigned int blk_queue_get_max_sectors(struct request_queue *q, - unsigned int cmd_flags) + int op) { - if (unlikely(cmd_flags & REQ_DISCARD)) + if (unlikely(op == REQ_OP_DISCARD)) return min(q->limits.max_discard_sectors, UINT_MAX >> 9); - if (unlikely(cmd_flags & REQ_WRITE_SAME)) + if (unlikely(op == REQ_OP_WRITE_SAME)) return q->limits.max_write_same_sectors; return q->limits.max_sectors; @@ -896,11 +896,11 @@ static inline unsigned int blk_rq_get_max_sectors(struct request *rq) if (unlikely(rq->cmd_type == REQ_TYPE_BLOCK_PC)) return q->limits.max_hw_sectors; - if (!q->limits.chunk_sectors || (rq->cmd_flags & REQ_DISCARD)) - return blk_queue_get_max_sectors(q, rq->cmd_flags); + if (!q->limits.chunk_sectors || (rq->op == REQ_OP_DISCARD)) + return blk_queue_get_max_sectors(q, rq->op); return min(blk_max_size_offset(q, blk_rq_pos(rq)), - blk_queue_get_max_sectors(q, rq->cmd_flags)); + blk_queue_get_max_sectors(q, rq->op)); } static inline unsigned int blk_rq_count_bios(struct request *rq) diff --git a/include/linux/elevator.h b/include/linux/elevator.h index 638b324..953d286 100644 --- a/include/linux/elevator.h +++ b/include/linux/elevator.h @@ -26,7 +26,7 @@ typedef int (elevator_dispatch_fn) (struct request_queue *, int); typedef void (elevator_add_req_fn) (struct request_queue *, struct request *); typedef struct request *(elevator_request_list_fn) (struct request_queue *, struct request *); typedef void (elevator_completed_req_fn) (struct request_queue *, struct request *); -typedef int (elevator_may_queue_fn) (struct request_queue *, int); +typedef int (elevator_may_queue_fn) (struct request_queue *, int, int); typedef void (elevator_init_icq_fn) (struct io_cq *); typedef void (elevator_exit_icq_fn) (struct io_cq *); @@ -134,7 +134,7 @@ extern struct request *elv_former_request(struct request_queue *, struct request extern struct request *elv_latter_request(struct request_queue *, struct request *); extern int elv_register_queue(struct request_queue *q); extern void elv_unregister_queue(struct request_queue *q); -extern int elv_may_queue(struct request_queue *, int); +extern int elv_may_queue(struct request_queue *, int, int); extern void elv_completed_request(struct request_queue *, struct request *); extern int elv_set_request(struct request_queue *q, struct request *rq, struct bio *bio, gfp_t gfp_mask); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:48 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E82B57F37 for ; Mon, 11 Jan 2016 14:22:47 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 96AB4AC005 for ; Mon, 11 Jan 2016 12:22:47 -0800 (PST) X-ASG-Debug-ID: 1452543764-04bdf06f6e7db60001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id mndFcBmY14MwzJvE (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:44 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 62404C1C7518; Mon, 11 Jan 2016 20:22:44 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEu008385; Mon, 11 Jan 2016 15:22:41 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 27/35] drivers: set request op to REQ_OP Date: Mon, 11 Jan 2016 14:21:25 -0600 X-ASG-Orig-Subj: [PATCH 27/35] drivers: set request op to REQ_OP Message-Id: <1452543693-4440-28-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543764 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This patch has the block driver use the request->op for REQ_OP operations and cmd_flags for rq_flag_bits. I have only tested scsi and rbd. Signed-off-by: Mike Christie --- drivers/block/loop.c | 6 +++--- drivers/block/mtip32xx/mtip32xx.c | 2 +- drivers/block/nbd.c | 2 +- drivers/block/rbd.c | 2 +- drivers/block/skd_main.c | 11 ++++------- drivers/block/xen-blkfront.c | 8 +++++--- drivers/md/dm.c | 2 +- drivers/mmc/card/block.c | 7 +++---- drivers/mmc/card/queue.c | 6 ++---- drivers/mmc/card/queue.h | 5 ++++- drivers/mtd/mtd_blkdevs.c | 2 +- drivers/nvme/host/pci.c | 4 ++-- drivers/scsi/sd.c | 25 ++++++++++++++++--------- 13 files changed, 44 insertions(+), 38 deletions(-) diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 423f4ca..e771bab 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -538,7 +538,7 @@ static int do_req_filebacked(struct loop_device *lo, struct request *rq) if (rq->cmd_flags & REQ_WRITE) { if (rq->cmd_flags & REQ_FLUSH) ret = lo_req_flush(lo, rq); - else if (rq->cmd_flags & REQ_DISCARD) + else if (rq->op == REQ_OP_DISCARD) ret = lo_discard(lo, rq, pos); else if (lo->transfer) ret = lo_write_transfer(lo, rq, pos); @@ -1653,8 +1653,8 @@ static int loop_queue_rq(struct blk_mq_hw_ctx *hctx, if (lo->lo_state != Lo_bound) return -EIO; - if (lo->use_dio && !(cmd->rq->cmd_flags & (REQ_FLUSH | - REQ_DISCARD))) + if (lo->use_dio && (!(cmd->rq->cmd_flags & REQ_FLUSH) || + cmd->rq->op == REQ_OP_DISCARD)) cmd->use_aio = true; else cmd->use_aio = false; diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c index 9b180db..3995a9e 100644 --- a/drivers/block/mtip32xx/mtip32xx.c +++ b/drivers/block/mtip32xx/mtip32xx.c @@ -3670,7 +3670,7 @@ static int mtip_submit_request(struct blk_mq_hw_ctx *hctx, struct request *rq) return -ENXIO; } - if (rq->cmd_flags & REQ_DISCARD) { + if (rq->op == REQ_OP_DISCARD) { int err; err = mtip_send_trim(dd, blk_rq_pos(rq), blk_rq_sectors(rq)); diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index e4c5cc1..dd8f3e9 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -242,7 +242,7 @@ static int nbd_send_req(struct nbd_device *nbd, struct request *req) if (req->cmd_type == REQ_TYPE_DRV_PRIV) type = NBD_CMD_DISC; - else if (req->cmd_flags & REQ_DISCARD) + else if (req->op == REQ_OP_DISCARD) type = NBD_CMD_TRIM; else if (req->cmd_flags & REQ_FLUSH) type = NBD_CMD_FLUSH; diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 81ea69f..ea326ef 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -3373,7 +3373,7 @@ static void rbd_queue_workfn(struct work_struct *work) goto err; } - if (rq->cmd_flags & REQ_DISCARD) + if (rq->op == REQ_OP_DISCARD) op_type = OBJ_OP_DISCARD; else if (rq->cmd_flags & REQ_WRITE) op_type = OBJ_OP_WRITE; diff --git a/drivers/block/skd_main.c b/drivers/block/skd_main.c index 586f916..f89a0c8 100644 --- a/drivers/block/skd_main.c +++ b/drivers/block/skd_main.c @@ -576,7 +576,6 @@ static void skd_request_fn(struct request_queue *q) struct request *req = NULL; struct skd_scsi_request *scsi_req; struct page *page; - unsigned long io_flags; int error; u32 lba; u32 count; @@ -624,12 +623,11 @@ static void skd_request_fn(struct request_queue *q) lba = (u32)blk_rq_pos(req); count = blk_rq_sectors(req); data_dir = rq_data_dir(req); - io_flags = req->cmd_flags; - if (io_flags & REQ_FLUSH) + if (req->cmd_flags & REQ_FLUSH) flush++; - if (io_flags & REQ_FUA) + if (req->cmd_flags & REQ_FUA) fua++; pr_debug("%s:%s:%d new req=%p lba=%u(0x%x) " @@ -735,7 +733,7 @@ static void skd_request_fn(struct request_queue *q) else skreq->sg_data_dir = SKD_DATA_DIR_HOST_TO_CARD; - if (io_flags & REQ_DISCARD) { + if (req->op == REQ_OP_DISCARD) { page = alloc_page(GFP_ATOMIC | __GFP_ZERO); if (!page) { pr_err("request_fn:Page allocation failed.\n"); @@ -852,9 +850,8 @@ static void skd_end_request(struct skd_device *skdev, struct skd_request_context *skreq, int error) { struct request *req = skreq->req; - unsigned int io_flags = req->cmd_flags; - if ((io_flags & REQ_DISCARD) && + if ((req->op == REQ_OP_DISCARD) && (skreq->discard_page == 1)) { pr_debug("%s:%s:%d, free the page!", skdev->name, __func__, __LINE__); diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 81444fc..d4aa473 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -718,7 +718,8 @@ static int blkif_queue_request(struct request *req) if (unlikely(info->connected != BLKIF_STATE_CONNECTED)) return 1; - if (unlikely(req->cmd_flags & (REQ_DISCARD | REQ_SECURE))) + if (unlikely(req->op == REQ_OP_DISCARD || + req->cmd_flags & REQ_SECURE)) return blkif_queue_discard_req(req); else return blkif_queue_rw_req(req); @@ -1691,8 +1692,9 @@ static int blkif_recover(struct blkfront_info *info) /* * Get the bios in the request so we can re-queue them. */ - if (copy[i].request->cmd_flags & - (REQ_FLUSH | REQ_FUA | REQ_DISCARD | REQ_SECURE)) { + if (copy[i].request->cmd_flags & REQ_FLUSH || + copy[i].request->op == REQ_OP_DISCARD || + copy[i].request->cmd_flags & (REQ_FUA | REQ_SECURE)) { /* * Flush operations don't contain bios, so * we need to requeue the whole request diff --git a/drivers/md/dm.c b/drivers/md/dm.c index b882e56c..0e7ddc4 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1278,7 +1278,7 @@ static void dm_done(struct request *clone, int error, bool mapped) r = rq_end_io(tio->ti, clone, error, &tio->info); } - if (unlikely(r == -EREMOTEIO && (clone->cmd_flags & REQ_WRITE_SAME) && + if (unlikely(r == -EREMOTEIO && (clone->op == REQ_OP_WRITE_SAME) && !clone->q->limits.max_write_same_sectors)) disable_write_same(tio->md); diff --git a/drivers/mmc/card/block.c b/drivers/mmc/card/block.c index 5914263..cc9b1ff 100644 --- a/drivers/mmc/card/block.c +++ b/drivers/mmc/card/block.c @@ -1691,8 +1691,7 @@ static u8 mmc_blk_prep_packed_list(struct mmc_queue *mq, struct request *req) !IS_ALIGNED(blk_rq_sectors(next), 8)) break; - if (next->cmd_flags & REQ_DISCARD || - next->cmd_flags & REQ_FLUSH) + if (next->op == REQ_OP_DISCARD || next->cmd_flags & REQ_FLUSH) break; if (rq_data_dir(cur) != rq_data_dir(next)) @@ -2133,7 +2132,7 @@ static int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) } mq->flags &= ~MMC_QUEUE_NEW_REQUEST; - if (cmd_flags & REQ_DISCARD) { + if (req && req->op == REQ_OP_DISCARD) { /* complete ongoing async transfer before issuing discard */ if (card->host->areq) mmc_blk_issue_rw_rq(mq, NULL); @@ -2157,7 +2156,7 @@ static int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) out: if ((!req && !(mq->flags & MMC_QUEUE_NEW_REQUEST)) || - (cmd_flags & MMC_REQ_SPECIAL_MASK)) + mmc_req_is_special(req)) /* * Release host when there are no more requests * and after special request(discard, flush) is done. diff --git a/drivers/mmc/card/queue.c b/drivers/mmc/card/queue.c index 6f4323c..9fb8d21 100644 --- a/drivers/mmc/card/queue.c +++ b/drivers/mmc/card/queue.c @@ -33,7 +33,7 @@ static int mmc_prep_request(struct request_queue *q, struct request *req) /* * We only like normal block requests and discards. */ - if (req->cmd_type != REQ_TYPE_FS && !(req->cmd_flags & REQ_DISCARD)) { + if (req->cmd_type != REQ_TYPE_FS && req->op != REQ_OP_DISCARD) { blk_dump_rq_flags(req, "MMC bad request"); return BLKPREP_KILL; } @@ -56,7 +56,6 @@ static int mmc_queue_thread(void *d) down(&mq->thread_sem); do { struct request *req = NULL; - unsigned int cmd_flags = 0; spin_lock_irq(q->queue_lock); set_current_state(TASK_INTERRUPTIBLE); @@ -66,7 +65,6 @@ static int mmc_queue_thread(void *d) if (req || mq->mqrq_prev->req) { set_current_state(TASK_RUNNING); - cmd_flags = req ? req->cmd_flags : 0; mq->issue_fn(mq, req); cond_resched(); if (mq->flags & MMC_QUEUE_NEW_REQUEST) { @@ -81,7 +79,7 @@ static int mmc_queue_thread(void *d) * has been finished. Do not assign it to previous * request. */ - if (cmd_flags & MMC_REQ_SPECIAL_MASK) + if (mmc_req_is_special(req)) mq->mqrq_cur->req = NULL; mq->mqrq_prev->brq.mrq.data = NULL; diff --git a/drivers/mmc/card/queue.h b/drivers/mmc/card/queue.h index 36cddab..f166e5b 100644 --- a/drivers/mmc/card/queue.h +++ b/drivers/mmc/card/queue.h @@ -1,7 +1,10 @@ #ifndef MMC_QUEUE_H #define MMC_QUEUE_H -#define MMC_REQ_SPECIAL_MASK (REQ_DISCARD | REQ_FLUSH) +static inline bool mmc_req_is_special(struct request *req) +{ + return req && (req->cmd_flags & REQ_FLUSH || req->op == REQ_OP_DISCARD); +} struct request; struct task_struct; diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c index f470118..b2e0bce 100644 --- a/drivers/mtd/mtd_blkdevs.c +++ b/drivers/mtd/mtd_blkdevs.c @@ -94,7 +94,7 @@ static int do_blktrans_request(struct mtd_blktrans_ops *tr, get_capacity(req->rq_disk)) return -EIO; - if (req->cmd_flags & REQ_DISCARD) + if (req->op == REQ_OP_DISCARD) return tr->discard(dev, block, nsect); if (rq_data_dir(req) == READ) { diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 92fa1a6..c51205a 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -347,7 +347,7 @@ static int nvme_init_iod(struct request *rq, struct nvme_dev *dev) int nseg = rq->nr_phys_segments; unsigned size; - if (rq->cmd_flags & REQ_DISCARD) + if (rq->op == REQ_OP_DISCARD) size = sizeof(struct nvme_dsm_range); else size = blk_rq_bytes(rq); @@ -663,7 +663,7 @@ static int nvme_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) return ret; - if (req->cmd_flags & REQ_DISCARD) { + if (req->op == REQ_OP_DISCARD) { ret = nvme_setup_discard(nvmeq, ns, req, &cmnd); } else { if (req->cmd_type == REQ_TYPE_DRV_PRIV) diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 4e08d1cd..f6eda6d 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -1012,7 +1012,8 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) } else if (rq_data_dir(rq) == READ) { SCpnt->cmnd[0] = READ_6; } else { - scmd_printk(KERN_ERR, SCpnt, "Unknown command %llx\n", (unsigned long long) rq->cmd_flags); + scmd_printk(KERN_ERR, SCpnt, "Unknown command %d,%llx\n", + rq->op, (unsigned long long) rq->cmd_flags); goto out; } @@ -1137,21 +1138,27 @@ static int sd_init_command(struct scsi_cmnd *cmd) { struct request *rq = cmd->request; - if (rq->cmd_flags & REQ_DISCARD) + switch (rq->op) { + case REQ_OP_DISCARD: return sd_setup_discard_cmnd(cmd); - else if (rq->cmd_flags & REQ_WRITE_SAME) + case REQ_OP_WRITE_SAME: return sd_setup_write_same_cmnd(cmd); - else if (rq->cmd_flags & REQ_FLUSH) - return sd_setup_flush_cmnd(cmd); - else - return sd_setup_read_write_cmnd(cmd); + case REQ_OP_READ: + case REQ_OP_WRITE: + if (rq->cmd_flags & REQ_FLUSH) + return sd_setup_flush_cmnd(cmd); + else + return sd_setup_read_write_cmnd(cmd); + default: + BUG(); + } } static void sd_uninit_command(struct scsi_cmnd *SCpnt) { struct request *rq = SCpnt->request; - if (rq->cmd_flags & REQ_DISCARD) + if (rq->op == REQ_OP_DISCARD) __free_page(rq->completion_data); if (SCpnt->cmnd != rq->cmd) { @@ -1768,7 +1775,7 @@ static int sd_done(struct scsi_cmnd *SCpnt) unsigned char op = SCpnt->cmnd[0]; unsigned char unmap = SCpnt->cmnd[1] & 8; - if (req->cmd_flags & REQ_DISCARD || req->cmd_flags & REQ_WRITE_SAME) { + if (req->op == REQ_OP_DISCARD || req->op == REQ_OP_WRITE_SAME) { if (!result) { good_bytes = blk_rq_bytes(req); scsi_set_resid(SCpnt, 0); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:49 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9544D29E66 for ; Mon, 11 Jan 2016 14:22:49 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 26D0EAC005 for ; Mon, 11 Jan 2016 12:22:49 -0800 (PST) X-ASG-Debug-ID: 1452543766-04cbb05f7c97b90001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 50yWb0lMMJQhL4TD (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:47 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 8EAE38DFF2; Mon, 11 Jan 2016 20:22:46 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEv008385; Mon, 11 Jan 2016 15:22:44 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 28/35] blktrace: get op from req->op/bio->bi_op Date: Mon, 11 Jan 2016 14:21:26 -0600 X-ASG-Orig-Subj: [PATCH 28/35] blktrace: get op from req->op/bio->bi_op Message-Id: <1452543693-4440-29-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543767 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The bio and request struct now store the operation in bio->bi_op/request->op. This patch has blktrace not check bi_rw/cmd_flags. This patch is only compile tested. Signed-off-by: Mike Christie --- include/linux/blktrace_api.h | 2 +- include/trace/events/bcache.h | 12 ++++++---- include/trace/events/block.h | 31 +++++++++++++++++--------- kernel/trace/blktrace.c | 52 +++++++++++++++++++++++-------------------- 4 files changed, 57 insertions(+), 40 deletions(-) diff --git a/include/linux/blktrace_api.h b/include/linux/blktrace_api.h index afc1343..ee25ba4 100644 --- a/include/linux/blktrace_api.h +++ b/include/linux/blktrace_api.h @@ -109,7 +109,7 @@ static inline int blk_cmd_buf_len(struct request *rq) } extern void blk_dump_cmd(char *buf, struct request *rq); -extern void blk_fill_rwbs(char *rwbs, u32 rw, int bytes); +extern void blk_fill_rwbs(char *rwbs, int op, u32 rw, int bytes); #endif /* CONFIG_EVENT_TRACING && CONFIG_BLOCK */ diff --git a/include/trace/events/bcache.h b/include/trace/events/bcache.h index 981acf7..8abe564 100644 --- a/include/trace/events/bcache.h +++ b/include/trace/events/bcache.h @@ -27,7 +27,8 @@ DECLARE_EVENT_CLASS(bcache_request, __entry->sector = bio->bi_iter.bi_sector; __entry->orig_sector = bio->bi_iter.bi_sector - 16; __entry->nr_sector = bio->bi_iter.bi_size >> 9; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); ), TP_printk("%d,%d %s %llu + %u (from %d,%d @ %llu)", @@ -101,7 +102,8 @@ DECLARE_EVENT_CLASS(bcache_bio, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio->bi_iter.bi_size >> 9; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); ), TP_printk("%d,%d %s %llu + %u", @@ -136,7 +138,8 @@ TRACE_EVENT(bcache_read, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio->bi_iter.bi_size >> 9; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); __entry->cache_hit = hit; __entry->bypass = bypass; ), @@ -167,7 +170,8 @@ TRACE_EVENT(bcache_write, __entry->inode = inode; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio->bi_iter.bi_size >> 9; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); __entry->writeback = writeback; __entry->bypass = bypass; ), diff --git a/include/trace/events/block.h b/include/trace/events/block.h index e8a5eca..4416dcd 100644 --- a/include/trace/events/block.h +++ b/include/trace/events/block.h @@ -84,7 +84,8 @@ DECLARE_EVENT_CLASS(block_rq_with_error, 0 : blk_rq_sectors(rq); __entry->errors = rq->errors; - blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, blk_rq_bytes(rq)); + blk_fill_rwbs(__entry->rwbs, rq->op, rq->cmd_flags, + blk_rq_bytes(rq)); blk_dump_cmd(__get_str(cmd), rq); ), @@ -162,7 +163,7 @@ TRACE_EVENT(block_rq_complete, __entry->nr_sector = nr_bytes >> 9; __entry->errors = rq->errors; - blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, nr_bytes); + blk_fill_rwbs(__entry->rwbs, rq->op, rq->cmd_flags, nr_bytes); blk_dump_cmd(__get_str(cmd), rq); ), @@ -198,7 +199,8 @@ DECLARE_EVENT_CLASS(block_rq, __entry->bytes = (rq->cmd_type == REQ_TYPE_BLOCK_PC) ? blk_rq_bytes(rq) : 0; - blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, blk_rq_bytes(rq)); + blk_fill_rwbs(__entry->rwbs, rq->op, rq->cmd_flags, + blk_rq_bytes(rq)); blk_dump_cmd(__get_str(cmd), rq); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -272,7 +274,8 @@ TRACE_EVENT(block_bio_bounce, bio->bi_bdev->bd_dev : 0; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio_sectors(bio); - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -310,7 +313,8 @@ TRACE_EVENT(block_bio_complete, __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio_sectors(bio); __entry->error = error; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); ), TP_printk("%d,%d %s %llu + %u [%d]", @@ -337,7 +341,8 @@ DECLARE_EVENT_CLASS(block_bio_merge, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio_sectors(bio); - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -404,7 +409,8 @@ TRACE_EVENT(block_bio_queue, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->nr_sector = bio_sectors(bio); - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -432,7 +438,7 @@ DECLARE_EVENT_CLASS(block_get_rq, __entry->dev = bio ? bio->bi_bdev->bd_dev : 0; __entry->sector = bio ? bio->bi_iter.bi_sector : 0; __entry->nr_sector = bio ? bio_sectors(bio) : 0; - blk_fill_rwbs(__entry->rwbs, + blk_fill_rwbs(__entry->rwbs, bio ? bio->bi_op : 0, bio ? bio->bi_rw : 0, __entry->nr_sector); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -567,7 +573,8 @@ TRACE_EVENT(block_split, __entry->dev = bio->bi_bdev->bd_dev; __entry->sector = bio->bi_iter.bi_sector; __entry->new_sector = new_sector; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); memcpy(__entry->comm, current->comm, TASK_COMM_LEN); ), @@ -610,7 +617,8 @@ TRACE_EVENT(block_bio_remap, __entry->nr_sector = bio_sectors(bio); __entry->old_dev = dev; __entry->old_sector = from; - blk_fill_rwbs(__entry->rwbs, bio->bi_rw, bio->bi_iter.bi_size); + blk_fill_rwbs(__entry->rwbs, bio->bi_op, bio->bi_rw, + bio->bi_iter.bi_size); ), TP_printk("%d,%d %s %llu + %u <- (%d,%d) %llu", @@ -656,7 +664,8 @@ TRACE_EVENT(block_rq_remap, __entry->old_dev = dev; __entry->old_sector = from; __entry->nr_bios = blk_rq_count_bios(rq); - blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, blk_rq_bytes(rq)); + blk_fill_rwbs(__entry->rwbs, rq->op, rq->cmd_flags, + blk_rq_bytes(rq)); ), TP_printk("%d,%d %s %llu + %u <- (%d,%d) %llu %u", diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index 2aeb6ff..4a9ac83 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -199,7 +199,8 @@ static const u32 ddir_act[2] = { BLK_TC_ACT(BLK_TC_READ), * blk_io_trace structure and places it in a per-cpu subbuffer. */ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes, - int rw, u32 what, int error, int pdu_len, void *pdu_data) + int op, int op_flags, u32 what, int error, int pdu_len, + void *pdu_data) { struct task_struct *tsk = current; struct ring_buffer_event *event = NULL; @@ -214,13 +215,14 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes, if (unlikely(bt->trace_state != Blktrace_running && !blk_tracer)) return; - what |= ddir_act[rw & WRITE]; - what |= MASK_TC_BIT(rw, SYNC); - what |= MASK_TC_BIT(rw, RAHEAD); - what |= MASK_TC_BIT(rw, META); - what |= MASK_TC_BIT(rw, DISCARD); - what |= MASK_TC_BIT(rw, FLUSH); - what |= MASK_TC_BIT(rw, FUA); + what |= ddir_act[op_is_write(op) ? WRITE : READ]; + what |= MASK_TC_BIT(op_flags, SYNC); + what |= MASK_TC_BIT(op_flags, RAHEAD); + what |= MASK_TC_BIT(op_flags, META); + what |= MASK_TC_BIT(op_flags, FLUSH); + what |= MASK_TC_BIT(op_flags, FUA); + if (op == REQ_OP_DISCARD) + what |= BLK_TC_ACT(BLK_TC_DISCARD); pid = tsk->pid; if (act_log_check(bt, what, sector, pid)) @@ -708,11 +710,11 @@ static void blk_add_trace_rq(struct request_queue *q, struct request *rq, if (rq->cmd_type == REQ_TYPE_BLOCK_PC) { what |= BLK_TC_ACT(BLK_TC_PC); - __blk_add_trace(bt, 0, nr_bytes, rq->cmd_flags, + __blk_add_trace(bt, 0, nr_bytes, rq->op, rq->cmd_flags, what, rq->errors, rq->cmd_len, rq->cmd); } else { what |= BLK_TC_ACT(BLK_TC_FS); - __blk_add_trace(bt, blk_rq_pos(rq), nr_bytes, + __blk_add_trace(bt, blk_rq_pos(rq), nr_bytes, rq->op, rq->cmd_flags, what, rq->errors, 0, NULL); } } @@ -770,7 +772,7 @@ static void blk_add_trace_bio(struct request_queue *q, struct bio *bio, return; __blk_add_trace(bt, bio->bi_iter.bi_sector, bio->bi_iter.bi_size, - bio->bi_rw, what, error, 0, NULL); + bio->bi_op, bio->bi_rw, what, error, 0, NULL); } static void blk_add_trace_bio_bounce(void *ignore, @@ -818,7 +820,8 @@ static void blk_add_trace_getrq(void *ignore, struct blk_trace *bt = q->blk_trace; if (bt) - __blk_add_trace(bt, 0, 0, rw, BLK_TA_GETRQ, 0, 0, NULL); + __blk_add_trace(bt, 0, 0, rw, 0, BLK_TA_GETRQ, 0, 0, + NULL); } } @@ -833,7 +836,7 @@ static void blk_add_trace_sleeprq(void *ignore, struct blk_trace *bt = q->blk_trace; if (bt) - __blk_add_trace(bt, 0, 0, rw, BLK_TA_SLEEPRQ, + __blk_add_trace(bt, 0, 0, rw, 0, BLK_TA_SLEEPRQ, 0, 0, NULL); } } @@ -843,7 +846,7 @@ static void blk_add_trace_plug(void *ignore, struct request_queue *q) struct blk_trace *bt = q->blk_trace; if (bt) - __blk_add_trace(bt, 0, 0, 0, BLK_TA_PLUG, 0, 0, NULL); + __blk_add_trace(bt, 0, 0, 0, 0, BLK_TA_PLUG, 0, 0, NULL); } static void blk_add_trace_unplug(void *ignore, struct request_queue *q, @@ -860,7 +863,7 @@ static void blk_add_trace_unplug(void *ignore, struct request_queue *q, else what = BLK_TA_UNPLUG_TIMER; - __blk_add_trace(bt, 0, 0, 0, what, 0, sizeof(rpdu), &rpdu); + __blk_add_trace(bt, 0, 0, 0, 0, what, 0, sizeof(rpdu), &rpdu); } } @@ -874,8 +877,9 @@ static void blk_add_trace_split(void *ignore, __be64 rpdu = cpu_to_be64(pdu); __blk_add_trace(bt, bio->bi_iter.bi_sector, - bio->bi_iter.bi_size, bio->bi_rw, BLK_TA_SPLIT, - bio->bi_error, sizeof(rpdu), &rpdu); + bio->bi_iter.bi_size, bio->bi_op, bio->bi_rw, + BLK_TA_SPLIT, bio->bi_error, sizeof(rpdu), + &rpdu); } } @@ -907,7 +911,7 @@ static void blk_add_trace_bio_remap(void *ignore, r.sector_from = cpu_to_be64(from); __blk_add_trace(bt, bio->bi_iter.bi_sector, bio->bi_iter.bi_size, - bio->bi_rw, BLK_TA_REMAP, bio->bi_error, + bio->bi_op, bio->bi_rw, BLK_TA_REMAP, bio->bi_error, sizeof(r), &r); } @@ -940,7 +944,7 @@ static void blk_add_trace_rq_remap(void *ignore, r.sector_from = cpu_to_be64(from); __blk_add_trace(bt, blk_rq_pos(rq), blk_rq_bytes(rq), - rq_data_dir(rq), BLK_TA_REMAP, !!rq->errors, + rq_data_dir(rq), 0, BLK_TA_REMAP, !!rq->errors, sizeof(r), &r); } @@ -965,10 +969,10 @@ void blk_add_driver_data(struct request_queue *q, return; if (rq->cmd_type == REQ_TYPE_BLOCK_PC) - __blk_add_trace(bt, 0, blk_rq_bytes(rq), 0, + __blk_add_trace(bt, 0, blk_rq_bytes(rq), 0, 0, BLK_TA_DRV_DATA, rq->errors, len, data); else - __blk_add_trace(bt, blk_rq_pos(rq), blk_rq_bytes(rq), 0, + __blk_add_trace(bt, blk_rq_pos(rq), blk_rq_bytes(rq), 0, 0, BLK_TA_DRV_DATA, rq->errors, len, data); } EXPORT_SYMBOL_GPL(blk_add_driver_data); @@ -1767,16 +1771,16 @@ void blk_dump_cmd(char *buf, struct request *rq) } } -void blk_fill_rwbs(char *rwbs, u32 rw, int bytes) +void blk_fill_rwbs(char *rwbs, int op, u32 rw, int bytes) { int i = 0; if (rw & REQ_FLUSH) rwbs[i++] = 'F'; - if (rw & WRITE) + if (op == REQ_OP_WRITE) rwbs[i++] = 'W'; - else if (rw & REQ_DISCARD) + else if (op == REQ_OP_DISCARD) rwbs[i++] = 'D'; else if (bytes) rwbs[i++] = 'R'; -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:51 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id F25D729E03 for ; Mon, 11 Jan 2016 14:22:50 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 85A97AC001 for ; Mon, 11 Jan 2016 12:22:50 -0800 (PST) X-ASG-Debug-ID: 1452543769-04cbb05f7c97bd0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id EjOFTQPAXFkC0vuI (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:49 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id D1EE28E38D; Mon, 11 Jan 2016 20:22:48 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEw008385; Mon, 11 Jan 2016 15:22:46 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 29/35] ide cd: do not set REQ_WRITE on requests. Date: Mon, 11 Jan 2016 14:21:27 -0600 X-ASG-Orig-Subj: [PATCH 29/35] ide cd: do not set REQ_WRITE on requests. Message-Id: <1452543693-4440-30-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543769 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The block layer will set the correct READ/WRITE operation flags/fields when creating a request, so there is not need for drivers to set the REQ_WRITE flag. This patch is compile tested only. Signed-off-by: Mike Christie --- drivers/ide/ide-cd_ioctl.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/drivers/ide/ide-cd_ioctl.c b/drivers/ide/ide-cd_ioctl.c index 474173e..5887a7a 100644 --- a/drivers/ide/ide-cd_ioctl.c +++ b/drivers/ide/ide-cd_ioctl.c @@ -459,9 +459,6 @@ int ide_cdrom_packet(struct cdrom_device_info *cdi, layer. the packet must be complete, as we do not touch it at all. */ - if (cgc->data_direction == CGC_DATA_WRITE) - flags |= REQ_WRITE; - if (cgc->sense) memset(cgc->sense, 0, sizeof(struct request_sense)); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:55 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9F9B929DF6 for ; Mon, 11 Jan 2016 14:22:55 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 17C3AAC005 for ; Mon, 11 Jan 2016 12:22:54 -0800 (PST) X-ASG-Debug-ID: 1452543771-04cb6c75dc790b0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id fAiLr6jH14M8yAwA (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:51 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 3ED3AC0BF2B3; Mon, 11 Jan 2016 20:22:51 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZEx008385; Mon, 11 Jan 2016 15:22:48 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 30/35] block, fs, drivers: do not test bi_rw for REQ_OPs Date: Mon, 11 Jan 2016 14:21:28 -0600 X-ASG-Orig-Subj: [PATCH 30/35] block, fs, drivers: do not test bi_rw for REQ_OPs Message-Id: <1452543693-4440-31-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543771 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie We no longer use the bio->bi_rw field for REQ_OPs: REQ_WRITE, REQ_DISCARD, REQ_WRITE_SAME, so this patch stops checking for them in bi_rw and also removes the related compat code. v2: 1. Remove compat code in __get_request. Signed-off-by: Mike Christie --- block/bio.c | 6 ++--- block/blk-core.c | 34 ++++++++--------------------- block/blk-merge.c | 14 ++++++------ block/blk-mq.c | 3 +-- drivers/ata/libata-scsi.c | 2 +- drivers/block/brd.c | 2 +- drivers/block/drbd/drbd_main.c | 15 +++++++------ drivers/block/drbd/drbd_worker.c | 4 ++-- drivers/block/loop.c | 6 ++--- drivers/block/rbd.c | 2 +- drivers/block/rsxx/dma.c | 2 +- drivers/block/umem.c | 2 +- drivers/block/zram/zram_drv.c | 2 +- drivers/ide/ide-floppy.c | 2 +- drivers/lightnvm/rrpc.c | 2 +- drivers/md/bcache/request.c | 10 ++++----- drivers/md/dm-cache-target.c | 10 +++++---- drivers/md/dm-crypt.c | 2 +- drivers/md/dm-log-writes.c | 2 +- drivers/md/dm-raid1.c | 8 +++---- drivers/md/dm-region-hash.c | 4 ++-- drivers/md/dm-stripe.c | 4 ++-- drivers/md/dm-thin.c | 15 ++++++++----- drivers/md/dm.c | 6 ++--- drivers/md/linear.c | 2 +- drivers/md/raid0.c | 2 +- drivers/scsi/osd/osd_initiator.c | 4 ++-- drivers/staging/lustre/lustre/llite/lloop.c | 8 +++---- include/linux/bio.h | 15 ++++++++----- include/linux/fs.h | 25 +++++++-------------- 30 files changed, 100 insertions(+), 115 deletions(-) diff --git a/block/bio.c b/block/bio.c index 3b8e970..ca0c52d 100644 --- a/block/bio.c +++ b/block/bio.c @@ -669,10 +669,10 @@ struct bio *bio_clone_bioset(struct bio *bio_src, gfp_t gfp_mask, bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; bio->bi_iter.bi_size = bio_src->bi_iter.bi_size; - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) goto integrity_clone; - if (bio->bi_rw & REQ_WRITE_SAME) { + if (bio->bi_op == REQ_OP_WRITE_SAME) { bio->bi_io_vec[bio->bi_vcnt++] = bio_src->bi_io_vec[0]; goto integrity_clone; } @@ -1792,7 +1792,7 @@ struct bio *bio_split(struct bio *bio, int sectors, * Discards need a mutable bio_vec to accommodate the payload * required by the DSM TRIM and UNMAP commands. */ - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) split = bio_clone_bioset(bio, gfp, bs); else split = bio_clone_fast(bio, gfp, bs); diff --git a/block/blk-core.c b/block/blk-core.c index dacbd68..b3db65c 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1151,8 +1151,7 @@ static struct request *__get_request(struct request_list *rl, int op, blk_rq_init(q, rq); blk_rq_set_rl(rq, rl); - /* tmp compat - allow users to check either one for the op */ - rq->cmd_flags = op | op_flags | REQ_ALLOCED; + rq->cmd_flags = op_flags | REQ_ALLOCED; rq->op = op; /* init elvpriv */ @@ -1704,8 +1703,7 @@ void init_request_from_bio(struct request *req, struct bio *bio) { req->cmd_type = REQ_TYPE_FS; - /* tmp compat. Allow users to set bi_op or bi_rw */ - req->cmd_flags |= (bio->bi_rw | bio->bi_op) & REQ_COMMON_MASK; + req->cmd_flags |= bio->bi_rw & REQ_COMMON_MASK; if (bio->bi_rw & REQ_RAHEAD) req->cmd_flags |= REQ_FAILFAST_MASK; @@ -1855,9 +1853,9 @@ static void handle_bad_sector(struct bio *bio) char b[BDEVNAME_SIZE]; printk(KERN_INFO "attempt to access beyond end of device\n"); - printk(KERN_INFO "%s: rw=%ld, want=%Lu, limit=%Lu\n", + printk(KERN_INFO "%s: rw=%d,%ld, want=%Lu, limit=%Lu\n", bdevname(bio->bi_bdev, b), - bio->bi_rw, + bio->bi_op, bio->bi_rw, (unsigned long long)bio_end_sector(bio), (long long)(i_size_read(bio->bi_bdev->bd_inode) >> 9)); } @@ -1978,14 +1976,14 @@ generic_make_request_checks(struct bio *bio) } } - if ((bio->bi_rw & REQ_DISCARD) && + if ((bio->bi_op == REQ_OP_DISCARD) && (!blk_queue_discard(q) || ((bio->bi_rw & REQ_SECURE) && !blk_queue_secdiscard(q)))) { err = -EOPNOTSUPP; goto end_io; } - if (bio->bi_rw & REQ_WRITE_SAME && !bdev_write_same(bio->bi_bdev)) { + if (bio->bi_op == REQ_OP_WRITE_SAME && !bdev_write_same(bio->bi_bdev)) { err = -EOPNOTSUPP; goto end_io; } @@ -2039,12 +2037,6 @@ blk_qc_t generic_make_request(struct bio *bio) struct bio_list bio_list_on_stack; blk_qc_t ret = BLK_QC_T_NONE; - /* tmp compat. Allow users to set either one or both. - * This will be removed when we have converted - * everyone in the next patches. - */ - bio->bi_rw |= bio->bi_op; - if (!generic_make_request_checks(bio)) goto out; @@ -2114,12 +2106,6 @@ EXPORT_SYMBOL(generic_make_request); */ blk_qc_t submit_bio(struct bio *bio) { - /* tmp compat. Allow users to set either one or both. - * This will be removed when we have converted - * everyone in the next patches. - */ - bio->bi_rw |= bio->bi_op; - /* * If it's a regular read/write or a barrier with data attached, * go through the normal accounting stuff before submission. @@ -2127,12 +2113,12 @@ blk_qc_t submit_bio(struct bio *bio) if (bio_has_data(bio)) { unsigned int count; - if (unlikely(bio->bi_rw & REQ_WRITE_SAME)) + if (unlikely(bio->bi_op == REQ_OP_WRITE_SAME)) count = bdev_logical_block_size(bio->bi_bdev) >> 9; else count = bio_sectors(bio); - if (bio->bi_rw & WRITE) { + if (op_is_write(bio->bi_op)) { count_vm_events(PGPGOUT, count); } else { task_io_account_read(bio->bi_iter.bi_size); @@ -2143,7 +2129,7 @@ blk_qc_t submit_bio(struct bio *bio) char b[BDEVNAME_SIZE]; printk(KERN_DEBUG "%s(%d): %s block %Lu on %s (%u sectors)\n", current->comm, task_pid_nr(current), - (bio->bi_rw & WRITE) ? "WRITE" : "READ", + op_is_write(bio->bi_op) ? "WRITE" : "READ", (unsigned long long)bio->bi_iter.bi_sector, bdevname(bio->bi_bdev, b), count); @@ -2991,8 +2977,6 @@ EXPORT_SYMBOL_GPL(__blk_end_request_err); void blk_rq_bio_prep(struct request_queue *q, struct request *rq, struct bio *bio) { - /* tmp compat. Allow users to set bi_op or bi_rw */ - rq->cmd_flags |= bio_data_dir(bio); rq->op = bio->bi_op; if (bio_has_data(bio)) diff --git a/block/blk-merge.c b/block/blk-merge.c index de73ed1..f53fefc 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -147,9 +147,9 @@ void blk_queue_split(struct request_queue *q, struct bio **bio, struct bio *split, *res; unsigned nsegs; - if ((*bio)->bi_rw & REQ_DISCARD) + if ((*bio)->bi_op == REQ_OP_DISCARD) split = blk_bio_discard_split(q, *bio, bs, &nsegs); - else if ((*bio)->bi_rw & REQ_WRITE_SAME) + else if ((*bio)->bi_op == REQ_OP_WRITE_SAME) split = blk_bio_write_same_split(q, *bio, bs, &nsegs); else split = blk_bio_segment_split(q, *bio, q->bio_split, &nsegs); @@ -188,10 +188,10 @@ static unsigned int __blk_recalc_rq_segments(struct request_queue *q, * This should probably be returning 0, but blk_add_request_payload() * (Christoph!!!!) */ - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) return 1; - if (bio->bi_rw & REQ_WRITE_SAME) + if (bio->bi_op == REQ_OP_WRITE_SAME) return 1; fbio = bio; @@ -364,7 +364,7 @@ static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio, nsegs = 0; cluster = blk_queue_cluster(q); - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { /* * This is a hack - drivers should be neither modifying the * biovec, nor relying on bi_vcnt - but because of @@ -379,7 +379,7 @@ static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio, return 0; } - if (bio->bi_rw & REQ_WRITE_SAME) { + if (bio->bi_op == REQ_OP_WRITE_SAME) { single_segment: *sg = sglist; bvec = bio_iovec(bio); @@ -418,7 +418,7 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq, } if (q->dma_drain_size && q->dma_drain_needed(rq)) { - if (rq->cmd_flags & REQ_WRITE) + if (op_is_write(rq->op)) memset(q->dma_drain_buffer, 0, q->dma_drain_size); sg_unmark_end(sg); diff --git a/block/blk-mq.c b/block/blk-mq.c index 745dae8..72a028b 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -170,8 +170,7 @@ static void blk_mq_rq_ctx_init(struct request_queue *q, struct blk_mq_ctx *ctx, rq->q = q; rq->mq_ctx = ctx; rq->op = op; - /* tmp compat - allow users to check either one for the op */ - rq->cmd_flags |= op | op_flags; + rq->cmd_flags |= op_flags; /* do not touch atomic flags, it needs atomic ops against the timer */ rq->cpu = -1; INIT_HLIST_NODE(&rq->hash); diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c index 7e959f9..05ab70e 100644 --- a/drivers/ata/libata-scsi.c +++ b/drivers/ata/libata-scsi.c @@ -1125,7 +1125,7 @@ static int atapi_drain_needed(struct request *rq) if (likely(rq->cmd_type != REQ_TYPE_BLOCK_PC)) return 0; - if (!blk_rq_bytes(rq) || (rq->cmd_flags & REQ_WRITE)) + if (!blk_rq_bytes(rq) || op_is_write(rq->op)) return 0; return atapi_cmd_type(rq->cmd[0]) == ATAPI_MISC; diff --git a/drivers/block/brd.c b/drivers/block/brd.c index 5e6eb98..c7c25b1 100644 --- a/drivers/block/brd.c +++ b/drivers/block/brd.c @@ -339,7 +339,7 @@ static blk_qc_t brd_make_request(struct request_queue *q, struct bio *bio) if (bio_end_sector(bio) > get_capacity(bdev->bd_disk)) goto io_error; - if (unlikely(bio->bi_rw & REQ_DISCARD)) { + if (unlikely(bio->bi_op == REQ_OP_DISCARD)) { if (sector & ((PAGE_SIZE >> SECTOR_SHIFT) - 1) || bio->bi_iter.bi_size & PAGE_MASK) goto io_error; diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index f6626e5..76ff804 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1603,15 +1603,16 @@ static int _drbd_send_zc_ee(struct drbd_peer_device *peer_device, return 0; } -static u32 bio_flags_to_wire(struct drbd_connection *connection, unsigned long bi_rw) +static u32 bio_flags_to_wire(struct drbd_connection *connection, + struct bio *bio) { if (connection->agreed_pro_version >= 95) - return (bi_rw & REQ_SYNC ? DP_RW_SYNC : 0) | - (bi_rw & REQ_FUA ? DP_FUA : 0) | - (bi_rw & REQ_FLUSH ? DP_FLUSH : 0) | - (bi_rw & REQ_DISCARD ? DP_DISCARD : 0); + return (bio->bi_rw & REQ_SYNC ? DP_RW_SYNC : 0) | + (bio->bi_rw & REQ_FUA ? DP_FUA : 0) | + (bio->bi_rw & REQ_FLUSH ? DP_FLUSH : 0) | + (bio->bi_op == REQ_OP_DISCARD ? DP_DISCARD : 0); else - return bi_rw & REQ_SYNC ? DP_RW_SYNC : 0; + return bio->bi_rw & REQ_SYNC ? DP_RW_SYNC : 0; } /* Used to send write or TRIM aka REQ_DISCARD requests @@ -1636,7 +1637,7 @@ int drbd_send_dblock(struct drbd_peer_device *peer_device, struct drbd_request * p->sector = cpu_to_be64(req->i.sector); p->block_id = (unsigned long)req; p->seq_num = cpu_to_be32(atomic_inc_return(&device->packet_seq)); - dp_flags = bio_flags_to_wire(peer_device->connection, req->master_bio->bi_rw); + dp_flags = bio_flags_to_wire(peer_device->connection, req->master_bio); if (device->state.conn >= C_SYNC_SOURCE && device->state.conn <= C_PAUSED_SYNC_T) dp_flags |= DP_MAY_SET_IN_SYNC; diff --git a/drivers/block/drbd/drbd_worker.c b/drivers/block/drbd/drbd_worker.c index f92f533..4335f2d 100644 --- a/drivers/block/drbd/drbd_worker.c +++ b/drivers/block/drbd/drbd_worker.c @@ -174,7 +174,7 @@ void drbd_peer_request_endio(struct bio *bio) struct drbd_peer_request *peer_req = bio->bi_private; struct drbd_device *device = peer_req->peer_device->device; int is_write = bio_data_dir(bio) == WRITE; - int is_discard = !!(bio->bi_rw & REQ_DISCARD); + int is_discard = !!(bio->bi_op == REQ_OP_DISCARD); if (bio->bi_error && __ratelimit(&drbd_ratelimit_state)) drbd_warn(device, "%s: error=%d s=%llus\n", @@ -248,7 +248,7 @@ void drbd_request_endio(struct bio *bio) /* to avoid recursion in __req_mod */ if (unlikely(bio->bi_error)) { - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) what = (bio->bi_error == -EOPNOTSUPP) ? DISCARD_COMPLETED_NOTSUPP : DISCARD_COMPLETED_WITH_ERROR; diff --git a/drivers/block/loop.c b/drivers/block/loop.c index e771bab..1afc03c 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -447,7 +447,7 @@ static int lo_req_flush(struct loop_device *lo, struct request *rq) static inline void handle_partial_read(struct loop_cmd *cmd, long bytes) { - if (bytes < 0 || (cmd->rq->cmd_flags & REQ_WRITE)) + if (bytes < 0 || op_is_write(cmd->rq->op)) return; if (unlikely(bytes < blk_rq_bytes(cmd->rq))) { @@ -535,7 +535,7 @@ static int do_req_filebacked(struct loop_device *lo, struct request *rq) pos = ((loff_t) blk_rq_pos(rq) << 9) + lo->lo_offset; - if (rq->cmd_flags & REQ_WRITE) { + if (op_is_write(rq->op)) { if (rq->cmd_flags & REQ_FLUSH) ret = lo_req_flush(lo, rq); else if (rq->op == REQ_OP_DISCARD) @@ -1666,7 +1666,7 @@ static int loop_queue_rq(struct blk_mq_hw_ctx *hctx, static void loop_handle_cmd(struct loop_cmd *cmd) { - const bool write = cmd->rq->cmd_flags & REQ_WRITE; + const bool write = op_is_write(cmd->rq->op); struct loop_device *lo = cmd->rq->q->queuedata; int ret = 0; diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index ea326ef..0afccb1 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -3375,7 +3375,7 @@ static void rbd_queue_workfn(struct work_struct *work) if (rq->op == REQ_OP_DISCARD) op_type = OBJ_OP_DISCARD; - else if (rq->cmd_flags & REQ_WRITE) + else if (rq->op == REQ_OP_WRITE) op_type = OBJ_OP_WRITE; else op_type = OBJ_OP_READ; diff --git a/drivers/block/rsxx/dma.c b/drivers/block/rsxx/dma.c index cf8cd29..dfc189e 100644 --- a/drivers/block/rsxx/dma.c +++ b/drivers/block/rsxx/dma.c @@ -705,7 +705,7 @@ int rsxx_dma_queue_bio(struct rsxx_cardinfo *card, dma_cnt[i] = 0; } - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { bv_len = bio->bi_iter.bi_size; while (bv_len > 0) { diff --git a/drivers/block/umem.c b/drivers/block/umem.c index 7939b9f..a24ccbc 100644 --- a/drivers/block/umem.c +++ b/drivers/block/umem.c @@ -462,7 +462,7 @@ static void process_page(unsigned long data) le32_to_cpu(desc->local_addr)>>9, le32_to_cpu(desc->transfer_size)); dump_dmastat(card, control); - } else if ((bio->bi_rw & REQ_WRITE) && + } else if (op_is_write(bio->bi_op) && le32_to_cpu(desc->local_addr) >> 9 == card->init_size) { card->init_size += le32_to_cpu(desc->transfer_size) >> 9; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 47915d7..8d14cea 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -849,7 +849,7 @@ static void __zram_make_request(struct zram *zram, struct bio *bio) offset = (bio->bi_iter.bi_sector & (SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT; - if (unlikely(bio->bi_rw & REQ_DISCARD)) { + if (unlikely(bio->bi_op == REQ_OP_DISCARD)) { zram_bio_discard(zram, index, offset, bio); bio_endio(bio); return; diff --git a/drivers/ide/ide-floppy.c b/drivers/ide/ide-floppy.c index 2fb5350..f079d8d 100644 --- a/drivers/ide/ide-floppy.c +++ b/drivers/ide/ide-floppy.c @@ -206,7 +206,7 @@ static void idefloppy_create_rw_cmd(ide_drive_t *drive, memcpy(rq->cmd, pc->c, 12); pc->rq = rq; - if (rq->cmd_flags & REQ_WRITE) + if (cmd == WRITE) pc->flags |= PC_FLAG_WRITING; pc->flags |= PC_FLAG_DMA_OK; diff --git a/drivers/lightnvm/rrpc.c b/drivers/lightnvm/rrpc.c index c20cb18..9b5a8c3 100644 --- a/drivers/lightnvm/rrpc.c +++ b/drivers/lightnvm/rrpc.c @@ -853,7 +853,7 @@ static blk_qc_t rrpc_make_rq(struct request_queue *q, struct bio *bio) struct nvm_rq *rqd; int err; - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { rrpc_discard(rrpc, bio); return BLK_QC_T_NONE; } diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index cd6837c..be8cd63f 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -378,7 +378,7 @@ static bool check_should_bypass(struct cached_dev *dc, struct bio *bio) if (test_bit(BCACHE_DEV_DETACHING, &dc->disk.flags) || c->gc_stats.in_use > CUTOFF_CACHE_ADD || - (bio->bi_rw & REQ_DISCARD)) + (bio->bi_op == REQ_OP_DISCARD)) goto skip; if (mode == CACHE_MODE_NONE || @@ -899,7 +899,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) * But check_overlapping drops dirty keys for which io hasn't started, * so we still want to call it. */ - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) s->iop.bypass = true; if (should_writeback(dc, s->orig_bio, @@ -913,7 +913,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) s->iop.bio = s->orig_bio; bio_get(s->iop.bio); - if (!(bio->bi_rw & REQ_DISCARD) || + if (!(bio->bi_op == REQ_OP_DISCARD) || blk_queue_discard(bdev_get_queue(dc->bdev))) closure_bio_submit(bio, cl); } else if (s->iop.writeback) { @@ -993,7 +993,7 @@ static blk_qc_t cached_dev_make_request(struct request_queue *q, cached_dev_read(dc, s); } } else { - if ((bio->bi_rw & REQ_DISCARD) && + if ((bio->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(dc->bdev))) bio_endio(bio); else @@ -1104,7 +1104,7 @@ static blk_qc_t flash_dev_make_request(struct request_queue *q, &KEY(d->id, bio->bi_iter.bi_sector, 0), &KEY(d->id, bio_end_sector(bio), 0)); - s->iop.bypass = (bio->bi_rw & REQ_DISCARD) != 0; + s->iop.bypass = (bio->bi_op == REQ_OP_DISCARD) != 0; s->iop.writeback = true; s->iop.bio = bio; diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c index 5780acc..4c54326 100644 --- a/drivers/md/dm-cache-target.c +++ b/drivers/md/dm-cache-target.c @@ -788,7 +788,8 @@ static void check_if_tick_bio_needed(struct cache *cache, struct bio *bio) spin_lock_irqsave(&cache->lock, flags); if (cache->need_tick_bio && - !(bio->bi_rw & (REQ_FUA | REQ_FLUSH | REQ_DISCARD))) { + !(bio->bi_rw & (REQ_FUA | REQ_FLUSH)) && + bio->bi_op != REQ_OP_DISCARD) { pb->tick = true; cache->need_tick_bio = false; } @@ -851,7 +852,7 @@ static void inc_ds(struct cache *cache, struct bio *bio, static bool accountable_bio(struct cache *cache, struct bio *bio) { return ((bio->bi_bdev == cache->origin_dev->bdev) && - !(bio->bi_rw & REQ_DISCARD)); + bio->bi_op != REQ_OP_DISCARD); } static void accounted_begin(struct cache *cache, struct bio *bio) @@ -1062,7 +1063,8 @@ static void dec_io_migrations(struct cache *cache) static bool discard_or_flush(struct bio *bio) { - return bio->bi_rw & (REQ_FLUSH | REQ_FUA | REQ_DISCARD); + return bio->bi_op == REQ_OP_DISCARD || + bio->bi_rw & (REQ_FLUSH | REQ_FUA); } static void __cell_defer(struct cache *cache, struct dm_bio_prison_cell *cell) @@ -1975,7 +1977,7 @@ static void process_deferred_bios(struct cache *cache) if (bio->bi_rw & REQ_FLUSH) process_flush_bio(cache, bio); - else if (bio->bi_rw & REQ_DISCARD) + else if (bio->bi_op == REQ_OP_DISCARD) process_discard_bio(cache, &structs, bio); else process_bio(cache, &structs, bio); diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index f466fec..b7cbd39 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1913,7 +1913,7 @@ static int crypt_map(struct dm_target *ti, struct bio *bio) * - for REQ_FLUSH device-mapper core ensures that no IO is in-flight * - for REQ_DISCARD caller must use flush if IO ordering matters */ - if (unlikely(bio->bi_rw & (REQ_FLUSH | REQ_DISCARD))) { + if (unlikely(bio->bi_rw & REQ_FLUSH || bio->bi_op == REQ_OP_DISCARD)) { bio->bi_bdev = cc->dev->bdev; if (bio_sectors(bio)) bio->bi_iter.bi_sector = cc->start + diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c index f63bbf4..813bb4d 100644 --- a/drivers/md/dm-log-writes.c +++ b/drivers/md/dm-log-writes.c @@ -557,7 +557,7 @@ static int log_writes_map(struct dm_target *ti, struct bio *bio) int i = 0; bool flush_bio = (bio->bi_rw & REQ_FLUSH); bool fua_bio = (bio->bi_rw & REQ_FUA); - bool discard_bio = (bio->bi_rw & REQ_DISCARD); + bool discard_bio = (bio->bi_op == REQ_OP_DISCARD); pb->block = NULL; diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c index 6b61b62..54cc0c7 100644 --- a/drivers/md/dm-raid1.c +++ b/drivers/md/dm-raid1.c @@ -626,7 +626,7 @@ static void write_callback(unsigned long error, void *context) * If the bio is discard, return an error, but do not * degrade the array. */ - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { bio->bi_error = -EOPNOTSUPP; bio_endio(bio); return; @@ -665,7 +665,7 @@ static void do_write(struct mirror_set *ms, struct bio *bio) .client = ms->io_client, }; - if (bio->bi_rw & REQ_DISCARD) { + if (bio->bi_op == REQ_OP_DISCARD) { io_req.bi_op = REQ_OP_DISCARD; io_req.mem.type = DM_IO_KMEM; io_req.mem.ptr.addr = NULL; @@ -705,7 +705,7 @@ static void do_writes(struct mirror_set *ms, struct bio_list *writes) while ((bio = bio_list_pop(writes))) { if ((bio->bi_rw & REQ_FLUSH) || - (bio->bi_rw & REQ_DISCARD)) { + (bio->bi_op == REQ_OP_DISCARD)) { bio_list_add(&sync, bio); continue; } @@ -1253,7 +1253,7 @@ static int mirror_end_io(struct dm_target *ti, struct bio *bio, int error) * We need to dec pending if this was a write. */ if (rw == WRITE) { - if (!(bio->bi_rw & (REQ_FLUSH | REQ_DISCARD))) + if (!(bio->bi_rw & REQ_FLUSH) && bio->bi_op != REQ_OP_DISCARD) dm_rh_dec(ms->rh, bio_record->write_region); return error; } diff --git a/drivers/md/dm-region-hash.c b/drivers/md/dm-region-hash.c index 74cb7b9..ce4af57 100644 --- a/drivers/md/dm-region-hash.c +++ b/drivers/md/dm-region-hash.c @@ -403,7 +403,7 @@ void dm_rh_mark_nosync(struct dm_region_hash *rh, struct bio *bio) return; } - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) return; /* We must inform the log that the sync count has changed. */ @@ -526,7 +526,7 @@ void dm_rh_inc_pending(struct dm_region_hash *rh, struct bio_list *bios) struct bio *bio; for (bio = bios->head; bio; bio = bio->bi_next) { - if (bio->bi_rw & (REQ_FLUSH | REQ_DISCARD)) + if (bio->bi_rw & REQ_FLUSH || bio->bi_op == REQ_OP_DISCARD) continue; rh_inc(rh, dm_rh_bio_to_region(rh, bio)); } diff --git a/drivers/md/dm-stripe.c b/drivers/md/dm-stripe.c index 797ddb9..12b1630 100644 --- a/drivers/md/dm-stripe.c +++ b/drivers/md/dm-stripe.c @@ -292,8 +292,8 @@ static int stripe_map(struct dm_target *ti, struct bio *bio) bio->bi_bdev = sc->stripe[target_bio_nr].dev->bdev; return DM_MAPIO_REMAPPED; } - if (unlikely(bio->bi_rw & REQ_DISCARD) || - unlikely(bio->bi_rw & REQ_WRITE_SAME)) { + if (unlikely(bio->bi_op == REQ_OP_DISCARD) || + unlikely(bio->bi_op == REQ_OP_WRITE_SAME)) { target_bio_nr = dm_bio_get_target_bio_nr(bio); BUG_ON(target_bio_nr >= sc->stripes); return stripe_map_range(sc, bio, target_bio_nr); diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index 5e1d281..0b2c6c1 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -705,7 +705,7 @@ static void inc_all_io_entry(struct pool *pool, struct bio *bio) { struct dm_thin_endio_hook *h; - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) return; h = dm_per_bio_data(bio, sizeof(struct dm_thin_endio_hook)); @@ -868,7 +868,8 @@ static void __inc_remap_and_issue_cell(void *context, struct bio *bio; while ((bio = bio_list_pop(&cell->bios))) { - if (bio->bi_rw & (REQ_DISCARD | REQ_FLUSH | REQ_FUA)) + if (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + bio->bi_op == REQ_OP_DISCARD) bio_list_add(&info->defer_bios, bio); else { inc_all_io_entry(info->tc->pool, bio); @@ -1646,7 +1647,8 @@ static void __remap_and_issue_shared_cell(void *context, while ((bio = bio_list_pop(&cell->bios))) { if ((bio_data_dir(bio) == WRITE) || - (bio->bi_rw & (REQ_DISCARD | REQ_FLUSH | REQ_FUA))) + (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + bio->bi_op == REQ_OP_DISCARD)) bio_list_add(&info->defer_bios, bio); else { struct dm_thin_endio_hook *h = dm_per_bio_data(bio, sizeof(struct dm_thin_endio_hook));; @@ -2035,7 +2037,7 @@ static void process_thin_deferred_bios(struct thin_c *tc) break; } - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) pool->process_discard(tc, bio); else pool->process_bio(tc, bio); @@ -2122,7 +2124,7 @@ static void process_thin_deferred_cells(struct thin_c *tc) return; } - if (cell->holder->bi_rw & REQ_DISCARD) + if (cell->holder->bi_op == REQ_OP_DISCARD) pool->process_discard_cell(tc, cell); else pool->process_cell(tc, cell); @@ -2558,7 +2560,8 @@ static int thin_bio_map(struct dm_target *ti, struct bio *bio) return DM_MAPIO_SUBMITTED; } - if (bio->bi_rw & (REQ_DISCARD | REQ_FLUSH | REQ_FUA)) { + if (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + bio->bi_op == REQ_OP_DISCARD) { thin_defer_bio_with_throttle(tc, bio); return DM_MAPIO_SUBMITTED; } diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 0e7ddc4..4200164 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1016,7 +1016,7 @@ static void clone_endio(struct bio *bio) } } - if (unlikely(r == -EREMOTEIO && (bio->bi_rw & REQ_WRITE_SAME) && + if (unlikely(r == -EREMOTEIO && (bio->bi_op == REQ_OP_WRITE_SAME) && !bdev_get_queue(bio->bi_bdev)->limits.max_write_same_sectors)) disable_write_same(md); @@ -1690,9 +1690,9 @@ static int __split_and_process_non_flush(struct clone_info *ci) struct dm_target *ti; unsigned len; - if (unlikely(bio->bi_rw & REQ_DISCARD)) + if (unlikely(bio->bi_op == REQ_OP_DISCARD)) return __send_discard(ci); - else if (unlikely(bio->bi_rw & REQ_WRITE_SAME)) + else if (unlikely(bio->bi_op == REQ_OP_WRITE_SAME)) return __send_write_same(ci); ti = dm_table_find_target(ci->map, ci->sector); diff --git a/drivers/md/linear.c b/drivers/md/linear.c index b7fe7e9..aad82c7 100644 --- a/drivers/md/linear.c +++ b/drivers/md/linear.c @@ -252,7 +252,7 @@ static void linear_make_request(struct mddev *mddev, struct bio *bio) split->bi_iter.bi_sector = split->bi_iter.bi_sector - start_sector + data_offset; - if (unlikely((split->bi_rw & REQ_DISCARD) && + if (unlikely((split->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(split->bi_bdev)))) { /* Just ignore it */ bio_endio(split); diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c index 2ea12c6..e0d1b8c 100644 --- a/drivers/md/raid0.c +++ b/drivers/md/raid0.c @@ -488,7 +488,7 @@ static void raid0_make_request(struct mddev *mddev, struct bio *bio) split->bi_iter.bi_sector = sector + zone->dev_start + tmp_dev->data_offset; - if (unlikely((split->bi_rw & REQ_DISCARD) && + if (unlikely((split->bi_op == REQ_OP_DISCARD) && !blk_queue_discard(bdev_get_queue(split->bi_bdev)))) { /* Just ignore it */ bio_endio(split); diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c index 4e7b440..680d9e4 100644 --- a/drivers/scsi/osd/osd_initiator.c +++ b/drivers/scsi/osd/osd_initiator.c @@ -824,7 +824,7 @@ void osd_req_write(struct osd_request *or, { _osd_req_encode_common(or, OSD_ACT_WRITE, obj, offset, len); WARN_ON(or->out.bio || or->out.total_bytes); - WARN_ON(0 == (bio->bi_rw & REQ_WRITE)); + WARN_ON(!op_is_write(bio->bi_op)); or->out.bio = bio; or->out.total_bytes = len; } @@ -875,7 +875,7 @@ void osd_req_read(struct osd_request *or, { _osd_req_encode_common(or, OSD_ACT_READ, obj, offset, len); WARN_ON(or->in.bio || or->in.total_bytes); - WARN_ON(bio->bi_rw & REQ_WRITE); + WARN_ON(op_is_write(bio->bi_op)); or->in.bio = bio; or->in.total_bytes = len; } diff --git a/drivers/staging/lustre/lustre/llite/lloop.c b/drivers/staging/lustre/lustre/llite/lloop.c index 420d391..37a52ed 100644 --- a/drivers/staging/lustre/lustre/llite/lloop.c +++ b/drivers/staging/lustre/lustre/llite/lloop.c @@ -212,9 +212,9 @@ static int do_bio_lustrebacked(struct lloop_device *lo, struct bio *head) io->ci_lockreq = CILR_NEVER; LASSERT(head != NULL); - rw = head->bi_rw; + rw = bio_data_dir(head); for (bio = head; bio != NULL; bio = bio->bi_next) { - LASSERT(rw == bio->bi_rw); + LASSERT(rw == bio_data_dir(bio)); offset = (pgoff_t)(bio->bi_iter.bi_sector << 9) + lo->lo_offset; bio_for_each_segment(bvec, bio, iter) { @@ -305,9 +305,9 @@ static unsigned int loop_get_bio(struct lloop_device *lo, struct bio **req) /* TODO: need to split the bio, too bad. */ LASSERT(first->bi_vcnt <= LLOOP_MAX_SEGMENTS); - rw = first->bi_rw; + rw = bio_data_dir(first); bio = &lo->lo_bio; - while (*bio && (*bio)->bi_rw == rw) { + while (*bio && bio_data_dir(*bio) == rw) { CDEBUG(D_INFO, "bio sector %llu size %u count %u vcnt%u \n", (unsigned long long)(*bio)->bi_iter.bi_sector, (*bio)->bi_iter.bi_size, diff --git a/include/linux/bio.h b/include/linux/bio.h index 9f0e2cf..1e4b3b5 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -106,18 +106,23 @@ static inline bool bio_has_data(struct bio *bio) { if (bio && bio->bi_iter.bi_size && - !(bio->bi_rw & REQ_DISCARD)) + !(bio->bi_op == REQ_OP_DISCARD)) return true; return false; } +static inline bool bio_no_advance_iter(struct bio *bio) +{ + return bio->bi_op == REQ_OP_DISCARD || bio->bi_op == REQ_OP_WRITE_SAME; +} + static inline bool bio_is_rw(struct bio *bio) { if (!bio_has_data(bio)) return false; - if (bio->bi_rw & BIO_NO_ADVANCE_ITER_MASK) + if (bio_no_advance_iter(bio)) return false; return true; @@ -225,7 +230,7 @@ static inline void bio_advance_iter(struct bio *bio, struct bvec_iter *iter, { iter->bi_sector += bytes >> 9; - if (bio->bi_rw & BIO_NO_ADVANCE_ITER_MASK) + if (bio_no_advance_iter(bio)) iter->bi_size -= bytes; else bvec_iter_advance(bio->bi_io_vec, iter, bytes); @@ -253,10 +258,10 @@ static inline unsigned bio_segments(struct bio *bio) * differently: */ - if (bio->bi_rw & REQ_DISCARD) + if (bio->bi_op == REQ_OP_DISCARD) return 1; - if (bio->bi_rw & REQ_WRITE_SAME) + if (bio->bi_op == REQ_OP_WRITE_SAME) return 1; bio_for_each_segment(bv, bio, iter) diff --git a/include/linux/fs.h b/include/linux/fs.h index 204ea4c..0a9fe5a 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2419,33 +2419,24 @@ static inline bool op_is_write(int op) } /* - * return READ, READA, or WRITE + * return data direction, READ or WRITE */ -static inline int bio_rw(struct bio *bio) +static inline int bio_data_dir(struct bio *bio) { - /* - * tmp cpmpat. Allow users to set either op or rw, until - * all code is converted in the next patches. - */ if (op_is_write(bio->bi_op)) return WRITE; - - return bio->bi_rw & (RW_MASK | RWA_MASK); + return READ; } /* - * return data direction, READ or WRITE + * return READ, READA, or WRITE */ -static inline int bio_data_dir(struct bio *bio) +static inline int bio_rw(struct bio *bio) { - /* - * tmp cpmpat. Allow users to set either op or rw, until - * all code is converted in the next patches. - */ - if (op_is_write(bio->bi_op)) - return WRITE; + if (bio->bi_rw & RWA_MASK) + return READA; - return bio->bi_rw & 1; + return bio_data_dir(bio); } extern void check_disk_size_change(struct gendisk *disk, -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 0372129E0B for ; Mon, 11 Jan 2016 14:22:57 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 91A5CAC001 for ; Mon, 11 Jan 2016 12:22:56 -0800 (PST) X-ASG-Debug-ID: 1452543774-04bdf06f6e7db80001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id qgh5OgAZwgAQ3Ayb (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:54 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id ED91E8F228; Mon, 11 Jan 2016 20:22:53 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZF0008385; Mon, 11 Jan 2016 15:22:51 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 31/35] block, fs: remove old REQ definitions. Date: Mon, 11 Jan 2016 14:21:29 -0600 X-ASG-Orig-Subj: [PATCH 31/35] block, fs: remove old REQ definitions. Message-Id: <1452543693-4440-32-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543774 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie We no longer use REQ_WRITE. REQ_WRITE_SAME and REQ_DISCARD, so this patch removes them. Signed-off-by: Mike Christie --- include/linux/blk_types.h | 21 ++++++--------------- include/linux/fs.h | 21 +++++++++++---------- include/trace/events/f2fs.h | 1 - 3 files changed, 17 insertions(+), 26 deletions(-) diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 6e49c91..b4251ed 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -151,7 +151,6 @@ struct bio { */ enum rq_flag_bits { /* common flags */ - __REQ_WRITE, /* not set, read. set, write */ __REQ_FAILFAST_DEV, /* no driver retries of device errors */ __REQ_FAILFAST_TRANSPORT, /* no driver retries of transport errors */ __REQ_FAILFAST_DRIVER, /* no driver retries of driver errors */ @@ -159,9 +158,7 @@ enum rq_flag_bits { __REQ_SYNC, /* request is sync (sync write or read) */ __REQ_META, /* metadata io request */ __REQ_PRIO, /* boost priority in cfq */ - __REQ_DISCARD, /* request to discard sectors */ - __REQ_SECURE, /* secure discard (used with __REQ_DISCARD) */ - __REQ_WRITE_SAME, /* write same block many times */ + __REQ_SECURE, /* secure discard (used with REQ_OP_DISCARD) */ __REQ_NOIDLE, /* don't anticipate more IO after this one */ __REQ_INTEGRITY, /* I/O includes block integrity payload */ @@ -197,28 +194,22 @@ enum rq_flag_bits { __REQ_NR_BITS, /* stops here */ }; -#define REQ_WRITE (1ULL << __REQ_WRITE) #define REQ_FAILFAST_DEV (1ULL << __REQ_FAILFAST_DEV) #define REQ_FAILFAST_TRANSPORT (1ULL << __REQ_FAILFAST_TRANSPORT) #define REQ_FAILFAST_DRIVER (1ULL << __REQ_FAILFAST_DRIVER) #define REQ_SYNC (1ULL << __REQ_SYNC) #define REQ_META (1ULL << __REQ_META) #define REQ_PRIO (1ULL << __REQ_PRIO) -#define REQ_DISCARD (1ULL << __REQ_DISCARD) -#define REQ_WRITE_SAME (1ULL << __REQ_WRITE_SAME) #define REQ_NOIDLE (1ULL << __REQ_NOIDLE) #define REQ_INTEGRITY (1ULL << __REQ_INTEGRITY) #define REQ_FAILFAST_MASK \ (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER) #define REQ_COMMON_MASK \ - (REQ_WRITE | REQ_FAILFAST_MASK | REQ_SYNC | REQ_META | REQ_PRIO | \ - REQ_DISCARD | REQ_WRITE_SAME | REQ_NOIDLE | REQ_FLUSH | REQ_FUA | \ - REQ_SECURE | REQ_INTEGRITY) + (REQ_FAILFAST_MASK | REQ_SYNC | REQ_META | REQ_PRIO | REQ_NOIDLE | \ + REQ_FLUSH | REQ_FUA | REQ_SECURE | REQ_INTEGRITY) #define REQ_CLONE_MASK REQ_COMMON_MASK -#define BIO_NO_ADVANCE_ITER_MASK (REQ_DISCARD|REQ_WRITE_SAME) - /* This mask is used for both bio and request merge checking */ #define REQ_NOMERGE_FLAGS \ (REQ_NOMERGE | REQ_STARTED | REQ_SOFTBARRIER | REQ_FLUSH | REQ_FUA | REQ_FLUSH_SEQ) @@ -250,9 +241,9 @@ enum rq_flag_bits { enum req_op { REQ_OP_READ, - REQ_OP_WRITE = REQ_WRITE, - REQ_OP_DISCARD = REQ_DISCARD, - REQ_OP_WRITE_SAME = REQ_WRITE_SAME, + REQ_OP_WRITE, + REQ_OP_DISCARD, /* request to discard sectors */ + REQ_OP_WRITE_SAME, /* write same block many times */ }; typedef unsigned int blk_qc_t; diff --git a/include/linux/fs.h b/include/linux/fs.h index 0a9fe5a..2fbed45 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -151,9 +151,10 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); #define CHECK_IOVEC_ONLY -1 /* - * The below are the various read and write types that we support. Some of + * The below are the various read and write flags that we support. Some of * them include behavioral modifiers that send information down to the - * block layer and IO scheduler. Terminology: + * block layer and IO scheduler. They should be used along with a req_op. + * Terminology: * * The block layer uses device plugging to defer IO a little bit, in * the hope that we will see more IO very shortly. This increases @@ -192,19 +193,19 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); * non-volatile media on completion. * */ -#define RW_MASK REQ_WRITE +#define RW_MASK REQ_OP_WRITE #define RWA_MASK REQ_RAHEAD -#define READ 0 +#define READ REQ_OP_READ #define WRITE RW_MASK #define READA RWA_MASK -#define READ_SYNC (READ | REQ_SYNC) -#define WRITE_SYNC (WRITE | REQ_SYNC | REQ_NOIDLE) -#define WRITE_ODIRECT (WRITE | REQ_SYNC) -#define WRITE_FLUSH (WRITE | REQ_SYNC | REQ_NOIDLE | REQ_FLUSH) -#define WRITE_FUA (WRITE | REQ_SYNC | REQ_NOIDLE | REQ_FUA) -#define WRITE_FLUSH_FUA (WRITE | REQ_SYNC | REQ_NOIDLE | REQ_FLUSH | REQ_FUA) +#define READ_SYNC REQ_SYNC +#define WRITE_SYNC (REQ_SYNC | REQ_NOIDLE) +#define WRITE_ODIRECT REQ_SYNC +#define WRITE_FLUSH (REQ_SYNC | REQ_NOIDLE | REQ_FLUSH) +#define WRITE_FUA (REQ_SYNC | REQ_NOIDLE | REQ_FUA) +#define WRITE_FLUSH_FUA (REQ_SYNC | REQ_NOIDLE | REQ_FLUSH | REQ_FUA) /* * Attribute flags. These should be or-ed together to figure out what diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index 851c335..1d568ee 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -31,7 +31,6 @@ TRACE_DEFINE_ENUM(BG_GC); TRACE_DEFINE_ENUM(LFS); TRACE_DEFINE_ENUM(SSR); TRACE_DEFINE_ENUM(__REQ_RAHEAD); -TRACE_DEFINE_ENUM(__REQ_WRITE); TRACE_DEFINE_ENUM(__REQ_SYNC); TRACE_DEFINE_ENUM(__REQ_NOIDLE); TRACE_DEFINE_ENUM(__REQ_FLUSH); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:22:57 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 9D86229DF6 for ; Mon, 11 Jan 2016 14:22:57 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 8C34E304059 for ; Mon, 11 Jan 2016 12:22:57 -0800 (PST) X-ASG-Debug-ID: 1452543776-04cb6c75dd790c0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id sXBKnoroxeKYak8F (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:56 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id D55D7C0A8058; Mon, 11 Jan 2016 20:22:55 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZF1008385; Mon, 11 Jan 2016 15:22:53 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 32/35] block: shrink bi_rw and bi_op Date: Mon, 11 Jan 2016 14:21:30 -0600 X-ASG-Orig-Subj: [PATCH 32/35] block: shrink bi_rw and bi_op Message-Id: <1452543693-4440-33-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543776 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie There is no need for bi_op/op and bi_rw to be so large now, so this patch shrinks them. Signed-off-by: Mike Christie --- block/blk-core.c | 2 +- drivers/md/dm-flakey.c | 2 +- drivers/md/raid5.c | 13 +++++++------ fs/btrfs/check-integrity.c | 4 ++-- fs/btrfs/inode.c | 2 +- include/linux/bio.h | 13 ++----------- include/linux/blk_types.h | 11 +++-------- include/linux/blkdev.h | 2 +- 8 files changed, 18 insertions(+), 31 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index b3db65c..1681f7e 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1853,7 +1853,7 @@ static void handle_bad_sector(struct bio *bio) char b[BDEVNAME_SIZE]; printk(KERN_INFO "attempt to access beyond end of device\n"); - printk(KERN_INFO "%s: rw=%d,%ld, want=%Lu, limit=%Lu\n", + printk(KERN_INFO "%s: rw=%d,%u, want=%Lu, limit=%Lu\n", bdevname(bio->bi_bdev, b), bio->bi_op, bio->bi_rw, (unsigned long long)bio_end_sector(bio), diff --git a/drivers/md/dm-flakey.c b/drivers/md/dm-flakey.c index 09e2afc..b831226 100644 --- a/drivers/md/dm-flakey.c +++ b/drivers/md/dm-flakey.c @@ -266,7 +266,7 @@ static void corrupt_bio_data(struct bio *bio, struct flakey_c *fc) data[fc->corrupt_bio_byte - 1] = fc->corrupt_bio_value; DMDEBUG("Corrupting data bio=%p by writing %u to byte %u " - "(rw=%c bi_rw=%lu bi_sector=%llu cur_bytes=%u)\n", + "(rw=%c bi_rw=%u bi_sector=%llu cur_bytes=%u)\n", bio, fc->corrupt_bio_value, fc->corrupt_bio_byte, (bio_data_dir(bio) == WRITE) ? 'w' : 'r', bio->bi_rw, (unsigned long long)bio->bi_iter.bi_sector, bio_bytes); diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 1a5edba..792366c 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -1015,9 +1015,9 @@ again: : raid5_end_read_request; bi->bi_private = sh; - pr_debug("%s: for %llu schedule op %ld on disc %d\n", + pr_debug("%s: for %llu schedule op %d,%u on disc %d\n", __func__, (unsigned long long)sh->sector, - bi->bi_rw, i); + bi->bi_op, bi->bi_rw, i); atomic_inc(&sh->count); if (sh != head_sh) atomic_inc(&head_sh->count); @@ -1067,10 +1067,10 @@ again: rbi->bi_end_io = raid5_end_write_request; rbi->bi_private = sh; - pr_debug("%s: for %llu schedule op %ld on " + pr_debug("%s: for %llu schedule op %d,%u on " "replacement disc %d\n", __func__, (unsigned long long)sh->sector, - rbi->bi_rw, i); + rbi->bi_op, rbi->bi_rw, i); atomic_inc(&sh->count); if (sh != head_sh) atomic_inc(&head_sh->count); @@ -1102,8 +1102,9 @@ again: if (!rdev && !rrdev) { if (op_is_write(op)) set_bit(STRIPE_DEGRADED, &sh->state); - pr_debug("skip op %ld on disc %d for sector %llu\n", - bi->bi_rw, i, (unsigned long long)sh->sector); + pr_debug("skip op %d,%u on disc %d for sector %llu\n", + bi->bi_op, bi->bi_rw, i, + (unsigned long long)sh->sector); clear_bit(R5_LOCKED, &sh->dev[i].flags); set_bit(STRIPE_HANDLE, &sh->state); } diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index e409d1f..1623d11 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -2985,7 +2985,7 @@ static void __btrfsic_submit_bio(struct bio *bio) if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bio(rw=%d,0x%lx, bi_vcnt=%u," + "submit_bio(rw=%d,0x%x, bi_vcnt=%u," " bi_sector=%llu (bytenr %llu), bi_bdev=%p)\n", bio->bi_op, bio->bi_rw, bio->bi_vcnt, (unsigned long long)bio->bi_iter.bi_sector, @@ -3028,7 +3028,7 @@ static void __btrfsic_submit_bio(struct bio *bio) if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO - "submit_bio(rw=%d,0x%lx FLUSH, bdev=%p)\n", + "submit_bio(rw=%d,0x%x FLUSH, bdev=%p)\n", bio->bi_op, bio->bi_rw, bio->bi_bdev); if (!dev_state->dummy_block_for_bio_bh_flush.is_iodone) { if ((dev_state->state->print_mask & diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index bb35ed2..22404a5 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8061,7 +8061,7 @@ static void btrfs_end_dio_bio(struct bio *bio) if (err) btrfs_warn(BTRFS_I(dip->inode)->root->fs_info, - "direct IO failed ino %llu rw %d,%lu sector %#Lx len %u err no %d", + "direct IO failed ino %llu rw %d,0x%x sector %#Lx len %u err no %d", btrfs_ino(dip->inode), bio->bi_op, bio->bi_rw, (unsigned long long)bio->bi_iter.bi_sector, bio->bi_iter.bi_size, err); diff --git a/include/linux/bio.h b/include/linux/bio.h index 1e4b3b5..c91afc1 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -44,18 +44,9 @@ #define BIO_MAX_SIZE (BIO_MAX_PAGES << PAGE_CACHE_SHIFT) #define BIO_MAX_SECTORS (BIO_MAX_SIZE >> 9) -/* - * upper 16 bits of bi_rw define the io priority of this bio - */ -#define BIO_PRIO_SHIFT (8 * sizeof(unsigned long) - IOPRIO_BITS) -#define bio_prio(bio) ((bio)->bi_rw >> BIO_PRIO_SHIFT) +#define bio_prio(bio) (bio)->bi_ioprio #define bio_prio_valid(bio) ioprio_valid(bio_prio(bio)) - -#define bio_set_prio(bio, prio) do { \ - WARN_ON(prio >= (1 << IOPRIO_BITS)); \ - (bio)->bi_rw &= ((1UL << BIO_PRIO_SHIFT) - 1); \ - (bio)->bi_rw |= ((unsigned long) (prio) << BIO_PRIO_SHIFT); \ -} while (0) +#define bio_set_prio(bio, prio) ((bio)->bi_ioprio = prio) /* * various member access, note that bio_data should of course not be used diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index b4251ed..498f395 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -48,14 +48,9 @@ struct bio { struct block_device *bi_bdev; unsigned int bi_flags; /* status, command, etc */ int bi_error; - unsigned long bi_rw; /* bottom bits rq_flags_bits - * top bits priority - */ - /* - * this will be a u8 in the next patches and bi_rw can be shrunk to - * a u32. For compat in these transistional patches op is a int here. - */ - int bi_op; /* REQ_OP */ + unsigned int bi_rw; /* rq_flag_bits */ + unsigned short bi_ioprio; + u8 bi_op; /* REQ_OP */ struct bvec_iter bi_iter; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index bc024c7..83d6f033 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -96,7 +96,7 @@ struct request { struct request_queue *q; struct blk_mq_ctx *mq_ctx; - int op; + u8 op; u64 cmd_flags; unsigned cmd_type; unsigned long atomic_flags; -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:23:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id AE0BE29E43 for ; Mon, 11 Jan 2016 14:23:00 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9CBBE8F8049 for ; Mon, 11 Jan 2016 12:23:00 -0800 (PST) X-ASG-Debug-ID: 1452543778-04cbb05f7c97c90001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id GRKGwdla1JFzWzwt (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:22:58 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 350E6C0A847C; Mon, 11 Jan 2016 20:22:58 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZF2008385; Mon, 11 Jan 2016 15:22:55 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 33/35] block, drivers: add REQ_OP_FLUSH operation Date: Mon, 11 Jan 2016 14:21:31 -0600 X-ASG-Orig-Subj: [PATCH 33/35] block, drivers: add REQ_OP_FLUSH operation Message-Id: <1452543693-4440-34-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543778 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie This adds a REQ_OP_FLUSH operation that is sent to request_fn based drivers by the block layer's flush code, instead of sending requests with the request->cmd_flags REQ_FLUSH bit set. For the following 3 flush related patches, I have not tested every driver. I have only tested scsi with xfs and btrfs. v2. 1. Fix kbuild failures. Forgot to update ubd driver. Signed-off-by: Mike Christie --- Documentation/block/writeback_cache_control.txt | 6 +++--- arch/um/drivers/ubd_kern.c | 2 +- block/blk-flush.c | 6 +++--- drivers/block/loop.c | 4 ++-- drivers/block/nbd.c | 2 +- drivers/block/osdblk.c | 2 +- drivers/block/ps3disk.c | 4 ++-- drivers/block/skd_main.c | 2 +- drivers/block/virtio_blk.c | 2 +- drivers/block/xen-blkfront.c | 8 ++++---- drivers/ide/ide-disk.c | 2 +- drivers/md/dm.c | 4 ++-- drivers/mmc/card/block.c | 5 ++--- drivers/mmc/card/queue.h | 2 +- drivers/mtd/mtd_blkdevs.c | 2 +- drivers/nvme/host/pci.c | 2 +- drivers/scsi/sd.c | 7 +++---- include/linux/blk_types.h | 1 + include/linux/blkdev.h | 3 +++ kernel/trace/blktrace.c | 5 ++++- 20 files changed, 38 insertions(+), 33 deletions(-) diff --git a/Documentation/block/writeback_cache_control.txt b/Documentation/block/writeback_cache_control.txt index 83407d3..ea5550f 100644 --- a/Documentation/block/writeback_cache_control.txt +++ b/Documentation/block/writeback_cache_control.txt @@ -73,9 +73,9 @@ doing: blk_queue_flush(sdkp->disk->queue, REQ_FLUSH); -and handle empty REQ_FLUSH requests in its prep_fn/request_fn. Note that +and handle empty REQ_OP_FLUSH requests in its prep_fn/request_fn. Note that REQ_FLUSH requests with a payload are automatically turned into a sequence -of an empty REQ_FLUSH request followed by the actual write by the block +of an empty REQ_OP_FLUSH request followed by the actual write by the block layer. For devices that also support the FUA bit the block layer needs to be told to pass through the REQ_FUA bit using: @@ -83,4 +83,4 @@ to be told to pass through the REQ_FUA bit using: and the driver must handle write requests that have the REQ_FUA bit set in prep_fn/request_fn. If the FUA bit is not natively supported the block -layer turns it into an empty REQ_FLUSH request after the actual write. +layer turns it into an empty REQ_OP_FLUSH request after the actual write. diff --git a/arch/um/drivers/ubd_kern.c b/arch/um/drivers/ubd_kern.c index 39ba207..a7dc382 100644 --- a/arch/um/drivers/ubd_kern.c +++ b/arch/um/drivers/ubd_kern.c @@ -1286,7 +1286,7 @@ static void do_ubd_request(struct request_queue *q) req = dev->request; - if (req->cmd_flags & REQ_FLUSH) { + if (req->op == REQ_OP_FLUSH) { io_req = kmalloc(sizeof(struct io_thread_req), GFP_ATOMIC); if (io_req == NULL) { diff --git a/block/blk-flush.c b/block/blk-flush.c index e01d3ac..070d7c7 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -29,7 +29,7 @@ * The actual execution of flush is double buffered. Whenever a request * needs to execute PRE or POSTFLUSH, it queues at * fq->flush_queue[fq->flush_pending_idx]. Once certain criteria are met, a - * flush is issued and the pending_idx is toggled. When the flush + * REQ_OP_FLUSH is issued and the pending_idx is toggled. When the flush * completes, all the requests which were pending are proceeded to the next * step. This allows arbitrary merging of different types of FLUSH/FUA * requests. @@ -329,8 +329,8 @@ static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) } flush_rq->cmd_type = REQ_TYPE_FS; - flush_rq->cmd_flags = WRITE_FLUSH | REQ_FLUSH_SEQ; - flush_rq->op = REQ_OP_WRITE; + flush_rq->cmd_flags = REQ_SYNC | REQ_NOIDLE | REQ_FLUSH_SEQ; + flush_rq->op = REQ_OP_FLUSH; flush_rq->rq_disk = first_rq->rq_disk; flush_rq->end_io = flush_end_io; diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 1afc03c..a3d1293 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -536,7 +536,7 @@ static int do_req_filebacked(struct loop_device *lo, struct request *rq) pos = ((loff_t) blk_rq_pos(rq) << 9) + lo->lo_offset; if (op_is_write(rq->op)) { - if (rq->cmd_flags & REQ_FLUSH) + if (rq->op == REQ_OP_FLUSH) ret = lo_req_flush(lo, rq); else if (rq->op == REQ_OP_DISCARD) ret = lo_discard(lo, rq, pos); @@ -1653,7 +1653,7 @@ static int loop_queue_rq(struct blk_mq_hw_ctx *hctx, if (lo->lo_state != Lo_bound) return -EIO; - if (lo->use_dio && (!(cmd->rq->cmd_flags & REQ_FLUSH) || + if (lo->use_dio && (cmd->rq->op != REQ_OP_FLUSH || cmd->rq->op == REQ_OP_DISCARD)) cmd->use_aio = true; else diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index dd8f3e9..3b4be71 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -244,7 +244,7 @@ static int nbd_send_req(struct nbd_device *nbd, struct request *req) type = NBD_CMD_DISC; else if (req->op == REQ_OP_DISCARD) type = NBD_CMD_TRIM; - else if (req->cmd_flags & REQ_FLUSH) + else if (req->op == REQ_OP_FLUSH) type = NBD_CMD_FLUSH; else if (rq_data_dir(req) == WRITE) type = NBD_CMD_WRITE; diff --git a/drivers/block/osdblk.c b/drivers/block/osdblk.c index 1b709a4..3f8a0a0 100644 --- a/drivers/block/osdblk.c +++ b/drivers/block/osdblk.c @@ -321,7 +321,7 @@ static void osdblk_rq_fn(struct request_queue *q) * driver-specific, etc. */ - do_flush = rq->cmd_flags & REQ_FLUSH; + do_flush = (rq->op == REQ_OP_FLUSH); do_write = (rq_data_dir(rq) == WRITE); if (!do_flush) { /* osd_flush does not use a bio */ diff --git a/drivers/block/ps3disk.c b/drivers/block/ps3disk.c index c120d70..95c524b 100644 --- a/drivers/block/ps3disk.c +++ b/drivers/block/ps3disk.c @@ -196,7 +196,7 @@ static void ps3disk_do_request(struct ps3_storage_device *dev, dev_dbg(&dev->sbd.core, "%s:%u\n", __func__, __LINE__); while ((req = blk_fetch_request(q))) { - if (req->cmd_flags & REQ_FLUSH) { + if (req->op == REQ_OP_FLUSH) { if (ps3disk_submit_flush_request(dev, req)) break; } else if (req->cmd_type == REQ_TYPE_FS) { @@ -256,7 +256,7 @@ static irqreturn_t ps3disk_interrupt(int irq, void *data) return IRQ_HANDLED; } - if (req->cmd_flags & REQ_FLUSH) { + if (req->op == REQ_OP_FLUSH) { read = 0; op = "flush"; } else { diff --git a/drivers/block/skd_main.c b/drivers/block/skd_main.c index f89a0c8..a15cc2e 100644 --- a/drivers/block/skd_main.c +++ b/drivers/block/skd_main.c @@ -624,7 +624,7 @@ static void skd_request_fn(struct request_queue *q) count = blk_rq_sectors(req); data_dir = rq_data_dir(req); - if (req->cmd_flags & REQ_FLUSH) + if (req->op == REQ_OP_FLUSH) flush++; if (req->cmd_flags & REQ_FUA) diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 6ca3549..c6d3bc2 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -172,7 +172,7 @@ static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx, BUG_ON(req->nr_phys_segments + 2 > vblk->sg_elems); vbr->req = req; - if (req->cmd_flags & REQ_FLUSH) { + if (req->op == REQ_OP_FLUSH) { vbr->out_hdr.type = cpu_to_virtio32(vblk->vdev, VIRTIO_BLK_T_FLUSH); vbr->out_hdr.sector = 0; vbr->out_hdr.ioprio = cpu_to_virtio32(vblk->vdev, req_get_ioprio(vbr->req)); diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index d4aa473..1d48f0a 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -635,7 +635,7 @@ static int blkif_queue_rw_req(struct request *req) * The indirect operation can only be a BLKIF_OP_READ or * BLKIF_OP_WRITE */ - BUG_ON(req->cmd_flags & (REQ_FLUSH | REQ_FUA)); + BUG_ON(req->op == REQ_OP_FLUSH || req->cmd_flags & REQ_FUA); ring_req->operation = BLKIF_OP_INDIRECT; ring_req->u.indirect.indirect_op = rq_data_dir(req) ? BLKIF_OP_WRITE : BLKIF_OP_READ; @@ -647,7 +647,7 @@ static int blkif_queue_rw_req(struct request *req) ring_req->u.rw.handle = info->handle; ring_req->operation = rq_data_dir(req) ? BLKIF_OP_WRITE : BLKIF_OP_READ; - if (req->cmd_flags & (REQ_FLUSH | REQ_FUA)) { + if (req->op == REQ_OP_FLUSH || req->cmd_flags & REQ_FUA) { /* * Ideally we can do an unordered flush-to-disk. * In case the backend onlysupports barriers, use that. @@ -739,7 +739,7 @@ static inline bool blkif_request_flush_invalid(struct request *req, struct blkfront_info *info) { return ((req->cmd_type != REQ_TYPE_FS) || - ((req->cmd_flags & REQ_FLUSH) && + ((req->op == REQ_OP_FLUSH) && !(info->feature_flush & REQ_FLUSH)) || ((req->cmd_flags & REQ_FUA) && !(info->feature_flush & REQ_FUA))); @@ -1692,7 +1692,7 @@ static int blkif_recover(struct blkfront_info *info) /* * Get the bios in the request so we can re-queue them. */ - if (copy[i].request->cmd_flags & REQ_FLUSH || + if (copy[i].request->op == REQ_OP_FLUSH || copy[i].request->op == REQ_OP_DISCARD || copy[i].request->cmd_flags & (REQ_FUA | REQ_SECURE)) { /* diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c index 37a8a90..6474ed3 100644 --- a/drivers/ide/ide-disk.c +++ b/drivers/ide/ide-disk.c @@ -431,7 +431,7 @@ static int idedisk_prep_fn(struct request_queue *q, struct request *rq) ide_drive_t *drive = q->queuedata; struct ide_cmd *cmd; - if (!(rq->cmd_flags & REQ_FLUSH)) + if (rq->op != REQ_OP_FLUSH) return BLKPREP_OK; if (rq->special) { diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 4200164..dfaaba2 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -2100,7 +2100,7 @@ static void dm_request_fn(struct request_queue *q) /* always use block 0 to find the target for flushes for now */ pos = 0; - if (!(rq->cmd_flags & REQ_FLUSH)) + if (rq->op != REQ_OP_FLUSH) pos = blk_rq_pos(rq); ti = dm_table_find_target(map, pos); @@ -2660,7 +2660,7 @@ static int dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx, /* always use block 0 to find the target for flushes for now */ pos = 0; - if (!(rq->cmd_flags & REQ_FLUSH)) + if (rq->op != REQ_OP_FLUSH) pos = blk_rq_pos(rq); ti = dm_table_find_target(map, pos); diff --git a/drivers/mmc/card/block.c b/drivers/mmc/card/block.c index cc9b1ff..7caf236 100644 --- a/drivers/mmc/card/block.c +++ b/drivers/mmc/card/block.c @@ -1691,7 +1691,7 @@ static u8 mmc_blk_prep_packed_list(struct mmc_queue *mq, struct request *req) !IS_ALIGNED(blk_rq_sectors(next), 8)) break; - if (next->op == REQ_OP_DISCARD || next->cmd_flags & REQ_FLUSH) + if (next->op == REQ_OP_DISCARD || next->op == REQ_OP_FLUSH) break; if (rq_data_dir(cur) != rq_data_dir(next)) @@ -2116,7 +2116,6 @@ static int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) struct mmc_card *card = md->queue.card; struct mmc_host *host = card->host; unsigned long flags; - unsigned int cmd_flags = req ? req->cmd_flags : 0; if (req && !mq->mqrq_prev->req) /* claim host only for the first request */ @@ -2140,7 +2139,7 @@ static int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) ret = mmc_blk_issue_secdiscard_rq(mq, req); else ret = mmc_blk_issue_discard_rq(mq, req); - } else if (cmd_flags & REQ_FLUSH) { + } else if (req && req->op == REQ_OP_FLUSH) { /* complete ongoing async transfer before issuing flush */ if (card->host->areq) mmc_blk_issue_rw_rq(mq, NULL); diff --git a/drivers/mmc/card/queue.h b/drivers/mmc/card/queue.h index f166e5b..6c67e4e 100644 --- a/drivers/mmc/card/queue.h +++ b/drivers/mmc/card/queue.h @@ -3,7 +3,7 @@ static inline bool mmc_req_is_special(struct request *req) { - return req && (req->cmd_flags & REQ_FLUSH || req->op == REQ_OP_DISCARD); + return req && (req->op == REQ_OP_FLUSH || req->op == REQ_OP_DISCARD); } struct request; diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c index b2e0bce..67da1cd 100644 --- a/drivers/mtd/mtd_blkdevs.c +++ b/drivers/mtd/mtd_blkdevs.c @@ -87,7 +87,7 @@ static int do_blktrans_request(struct mtd_blktrans_ops *tr, if (req->cmd_type != REQ_TYPE_FS) return -EIO; - if (req->cmd_flags & REQ_FLUSH) + if (req->op == REQ_OP_FLUSH) return tr->flush(dev); if (blk_rq_pos(req) + blk_rq_cur_sectors(req) > diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index c51205a..334a5a1 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -668,7 +668,7 @@ static int nvme_queue_rq(struct blk_mq_hw_ctx *hctx, } else { if (req->cmd_type == REQ_TYPE_DRV_PRIV) memcpy(&cmnd, req->cmd, sizeof(cmnd)); - else if (req->cmd_flags & REQ_FLUSH) + else if (req->op == REQ_OP_FLUSH) nvme_setup_flush(ns, &cmnd); else nvme_setup_rw(ns, req, &cmnd); diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index f6eda6d..8585b6f 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -1143,12 +1143,11 @@ static int sd_init_command(struct scsi_cmnd *cmd) return sd_setup_discard_cmnd(cmd); case REQ_OP_WRITE_SAME: return sd_setup_write_same_cmnd(cmd); + case REQ_OP_FLUSH: + return sd_setup_flush_cmnd(cmd); case REQ_OP_READ: case REQ_OP_WRITE: - if (rq->cmd_flags & REQ_FLUSH) - return sd_setup_flush_cmnd(cmd); - else - return sd_setup_read_write_cmnd(cmd); + return sd_setup_read_write_cmnd(cmd); default: BUG(); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 498f395..f9e53ca 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -239,6 +239,7 @@ enum req_op { REQ_OP_WRITE, REQ_OP_DISCARD, /* request to discard sectors */ REQ_OP_WRITE_SAME, /* write same block many times */ + REQ_OP_FLUSH, /* request for cache flush */ }; typedef unsigned int blk_qc_t; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 83d6f033..ec0d0d0 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -651,6 +651,9 @@ static inline bool rq_mergeable(struct request *rq) if (rq->cmd_type != REQ_TYPE_FS) return false; + if (rq->op == REQ_OP_FLUSH) + return false; + if (rq->cmd_flags & REQ_NOMERGE_FLAGS) return false; diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index 4a9ac83..3a54c83 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -223,6 +223,8 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes, what |= MASK_TC_BIT(op_flags, FUA); if (op == REQ_OP_DISCARD) what |= BLK_TC_ACT(BLK_TC_DISCARD); + if (op == REQ_OP_FLUSH) + what |= BLK_TC_ACT(BLK_TC_FLUSH); pid = tsk->pid; if (act_log_check(bt, what, sector, pid)) @@ -1775,7 +1777,8 @@ void blk_fill_rwbs(char *rwbs, int op, u32 rw, int bytes) { int i = 0; - if (rw & REQ_FLUSH) + if (rw & REQ_FLUSH || + op == REQ_OP_FLUSH) rwbs[i++] = 'F'; if (op == REQ_OP_WRITE) -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:23:05 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id BDD4329E78 for ; Mon, 11 Jan 2016 14:23:04 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 38E20AC001 for ; Mon, 11 Jan 2016 12:23:04 -0800 (PST) X-ASG-Debug-ID: 1452543781-04cb6c75dd790d0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 648OoA10IJGmskOF (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:23:01 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id B056A42E5B9; Mon, 11 Jan 2016 20:23:00 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZF3008385; Mon, 11 Jan 2016 15:22:58 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 34/35] block: add QUEUE_FLAGs for flush and fua Date: Mon, 11 Jan 2016 14:21:32 -0600 X-ASG-Orig-Subj: [PATCH 34/35] block: add QUEUE_FLAGs for flush and fua Message-Id: <1452543693-4440-35-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543781 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie The last patch added a REQ_OP_FLUSH for request_fn drivers and the next patch renames REQ_FLUSH to REQ_PREFLUSH which will be used by file systems and make_request_fn drivers. This leaves REQ_FLUSH/REQ_FUA defined for drivers to tell the block layer if flush/fua is supported. The names are confusing and I bet will will accidentally be used by people to request flushes. To avoid that, this patch adds QUEUE_FLAGs for flush and fua which drivers will use to indicate what they support. v2: 1. Fix kbuild failures. Forgot to update ubd driver. v3: 1. Rename dm_table_supports_flush callout function argument to callout_fn. Signed-off-by: Mike Christie --- arch/um/drivers/ubd_kern.c | 2 +- block/blk-core.c | 3 +- block/blk-flush.c | 12 ++++---- block/blk-settings.c | 20 -------------- drivers/block/drbd/drbd_main.c | 3 +- drivers/block/loop.c | 2 +- drivers/block/mtip32xx/mtip32xx.c | 3 +- drivers/block/nbd.c | 6 ++-- drivers/block/osdblk.c | 2 +- drivers/block/ps3disk.c | 2 +- drivers/block/skd_main.c | 3 +- drivers/block/virtio_blk.c | 4 +-- drivers/block/xen-blkback/xenbus.c | 2 +- drivers/block/xen-blkfront.c | 55 ++++++++++++++++++++++--------------- drivers/ide/ide-disk.c | 6 ++-- drivers/md/bcache/super.c | 4 +-- drivers/md/dm-table.c | 32 +++++++++++++-------- drivers/md/md.c | 3 +- drivers/md/raid5-cache.c | 3 +- drivers/mmc/card/block.c | 3 +- drivers/mtd/mtd_blkdevs.c | 2 +- drivers/nvme/host/core.c | 6 ++-- drivers/scsi/sd.c | 13 +++++---- drivers/target/target_core_iblock.c | 6 ++-- include/linux/blkdev.h | 6 ++-- 25 files changed, 108 insertions(+), 95 deletions(-) diff --git a/arch/um/drivers/ubd_kern.c b/arch/um/drivers/ubd_kern.c index a7dc382..44380d6 100644 --- a/arch/um/drivers/ubd_kern.c +++ b/arch/um/drivers/ubd_kern.c @@ -862,7 +862,7 @@ static int ubd_add(int n, char **error_out) goto out; } ubd_dev->queue->queuedata = ubd_dev; - blk_queue_flush(ubd_dev->queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, ubd_dev->queue); blk_queue_max_segments(ubd_dev->queue, MAX_SG); err = ubd_disk_register(UBD_MAJOR, ubd_dev->size, n, &ubd_gendisk[n]); diff --git a/block/blk-core.c b/block/blk-core.c index 1681f7e..cfa70f7 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1968,7 +1968,8 @@ generic_make_request_checks(struct bio *bio) * drivers without flush support don't have to worry * about them. */ - if ((bio->bi_rw & (REQ_FLUSH | REQ_FUA)) && !q->flush_flags) { + if ((bio->bi_rw & (REQ_FLUSH | REQ_FUA)) && + !(blk_queue_flush(q) || blk_queue_fua(q))) { bio->bi_rw &= ~(REQ_FLUSH | REQ_FUA); if (!nr_sectors) { err = 0; diff --git a/block/blk-flush.c b/block/blk-flush.c index 070d7c7..e07ca6c 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -95,17 +95,18 @@ enum { static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq); -static unsigned int blk_flush_policy(unsigned int fflags, struct request *rq) +static unsigned int blk_flush_policy(struct request *rq) { + struct request_queue *q = rq->q; unsigned int policy = 0; if (blk_rq_sectors(rq)) policy |= REQ_FSEQ_DATA; - if (fflags & REQ_FLUSH) { + if (blk_queue_flush(q)) { if (rq->cmd_flags & REQ_FLUSH) policy |= REQ_FSEQ_PREFLUSH; - if (!(fflags & REQ_FUA) && (rq->cmd_flags & REQ_FUA)) + if (!blk_queue_fua(q) && (rq->cmd_flags & REQ_FUA)) policy |= REQ_FSEQ_POSTFLUSH; } return policy; @@ -385,8 +386,7 @@ static void mq_flush_data_end_io(struct request *rq, int error) void blk_insert_flush(struct request *rq) { struct request_queue *q = rq->q; - unsigned int fflags = q->flush_flags; /* may change, cache */ - unsigned int policy = blk_flush_policy(fflags, rq); + unsigned int policy = blk_flush_policy(rq); struct blk_flush_queue *fq = blk_get_flush_queue(q, rq->mq_ctx); /* @@ -394,7 +394,7 @@ void blk_insert_flush(struct request *rq) * REQ_FLUSH and FUA for the driver. */ rq->cmd_flags &= ~REQ_FLUSH; - if (!(fflags & REQ_FUA)) + if (!blk_queue_fua(q)) rq->cmd_flags &= ~REQ_FUA; /* diff --git a/block/blk-settings.c b/block/blk-settings.c index dd49735..3cef016 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -820,26 +820,6 @@ void blk_queue_update_dma_alignment(struct request_queue *q, int mask) } EXPORT_SYMBOL(blk_queue_update_dma_alignment); -/** - * blk_queue_flush - configure queue's cache flush capability - * @q: the request queue for the device - * @flush: 0, REQ_FLUSH or REQ_FLUSH | REQ_FUA - * - * Tell block layer cache flush capability of @q. If it supports - * flushing, REQ_FLUSH should be set. If it supports bypassing - * write cache for individual writes, REQ_FUA should be set. - */ -void blk_queue_flush(struct request_queue *q, unsigned int flush) -{ - WARN_ON_ONCE(flush & ~(REQ_FLUSH | REQ_FUA)); - - if (WARN_ON_ONCE(!(flush & REQ_FLUSH) && (flush & REQ_FUA))) - flush &= ~REQ_FUA; - - q->flush_flags = flush & (REQ_FLUSH | REQ_FUA); -} -EXPORT_SYMBOL_GPL(blk_queue_flush); - void blk_queue_flush_queueable(struct request_queue *q, bool queueable) { q->flush_not_queueable = !queueable; diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 76ff804..8a767d3 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -2762,7 +2762,8 @@ enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsig q->backing_dev_info.congested_data = device; blk_queue_make_request(q, drbd_make_request); - blk_queue_flush(q, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); /* Setting the max_hw_sectors to an odd value of 8kibyte here This triggers a max_bio_size message upon first attach or connect */ blk_queue_max_hw_sectors(q, DRBD_MAX_BIO_SIZE_SAFE >> 8); diff --git a/drivers/block/loop.c b/drivers/block/loop.c index a3d1293..297a91f 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -937,7 +937,7 @@ static int loop_set_fd(struct loop_device *lo, fmode_t mode, mapping_set_gfp_mask(mapping, lo->old_gfp_mask & ~(__GFP_IO|__GFP_FS)); if (!(lo_flags & LO_FLAGS_READ_ONLY) && file->f_op->fsync) - blk_queue_flush(lo->lo_queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, lo->lo_queue); loop_update_dio(lo); set_capacity(lo->lo_disk, size); diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c index 3995a9e..0c41ce4 100644 --- a/drivers/block/mtip32xx/mtip32xx.c +++ b/drivers/block/mtip32xx/mtip32xx.c @@ -3910,7 +3910,8 @@ skip_create_disk: * write back cache is not supported in the device. FUA depends on * write back cache support, hence setting flush support to zero. */ - blk_queue_flush(dd->queue, 0); + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, dd->queue); + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, dd->queue); /* Signal trim support */ if (dd->trim_supp == true) { diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index 3b4be71..f6bda16 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -750,9 +750,11 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, nbd->disk->queue); if (nbd->flags & NBD_FLAG_SEND_FLUSH) - blk_queue_flush(nbd->disk->queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, + nbd->disk->queue); else - blk_queue_flush(nbd->disk->queue, 0); + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, + nbd->disk->queue); thread = kthread_run(nbd_thread_send, nbd, "%s", nbd_name(nbd)); diff --git a/drivers/block/osdblk.c b/drivers/block/osdblk.c index 3f8a0a0..d03f06a 100644 --- a/drivers/block/osdblk.c +++ b/drivers/block/osdblk.c @@ -437,7 +437,7 @@ static int osdblk_init_disk(struct osdblk_device *osdev) blk_queue_stack_limits(q, osd_request_queue(osdev->osd)); blk_queue_prep_rq(q, blk_queue_start_tag); - blk_queue_flush(q, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); disk->queue = q; diff --git a/drivers/block/ps3disk.c b/drivers/block/ps3disk.c index 95c524b..5985fdc 100644 --- a/drivers/block/ps3disk.c +++ b/drivers/block/ps3disk.c @@ -468,7 +468,7 @@ static int ps3disk_probe(struct ps3_system_bus_device *_dev) blk_queue_dma_alignment(queue, dev->blk_size-1); blk_queue_logical_block_size(queue, dev->blk_size); - blk_queue_flush(queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, queue); blk_queue_max_segments(queue, -1); blk_queue_max_segment_size(queue, dev->bounce_size); diff --git a/drivers/block/skd_main.c b/drivers/block/skd_main.c index a15cc2e..68d0e22 100644 --- a/drivers/block/skd_main.c +++ b/drivers/block/skd_main.c @@ -4409,7 +4409,8 @@ static int skd_cons_disk(struct skd_device *skdev) disk->queue = q; q->queuedata = skdev; - blk_queue_flush(q, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); blk_queue_max_segments(q, skdev->sgs_per_request); blk_queue_max_hw_sectors(q, SKD_N_MAX_SECTORS); diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index c6d3bc2..b8a508b 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -489,9 +489,9 @@ static void virtblk_update_cache_mode(struct virtio_device *vdev) struct virtio_blk *vblk = vdev->priv; if (writeback) - blk_queue_flush(vblk->disk->queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, vblk->disk->queue); else - blk_queue_flush(vblk->disk->queue, 0); + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, vblk->disk->queue); revalidate_disk(vblk->disk); } diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index f53cff4..4569692 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -413,7 +413,7 @@ static int xen_vbd_create(struct xen_blkif *blkif, blkif_vdev_t handle, vbd->type |= VDISK_REMOVABLE; q = bdev_get_queue(bdev); - if (q && q->flush_flags) + if (q && (blk_queue_flush(q) || blk_queue_fua(q))) vbd->flush_support = true; if (q && blk_queue_secdiscard(q)) diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 1d48f0a..6007c86 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -146,6 +146,7 @@ struct blkfront_info unsigned int persistent_gnts_c; unsigned long shadow_free; unsigned int feature_flush; + unsigned int feature_fua; unsigned int feature_discard:1; unsigned int feature_secdiscard:1; unsigned int discard_granularity; @@ -655,19 +656,15 @@ static int blkif_queue_rw_req(struct request *req) * implement it the same way. (It's also a FLUSH+FUA, * since it is guaranteed ordered WRT previous writes.) */ - switch (info->feature_flush & - ((REQ_FLUSH|REQ_FUA))) { - case REQ_FLUSH|REQ_FUA: + if (blk_queue_flush(info->rq) && + blk_queue_fua(info->rq)) ring_req->operation = BLKIF_OP_WRITE_BARRIER; - break; - case REQ_FLUSH: + else if (blk_queue_flush(info->rq)) ring_req->operation = BLKIF_OP_FLUSH_DISKCACHE; - break; - default: + else ring_req->operation = 0; - } } ring_req->u.rw.nr_segments = num_grant; } @@ -740,9 +737,9 @@ static inline bool blkif_request_flush_invalid(struct request *req, { return ((req->cmd_type != REQ_TYPE_FS) || ((req->op == REQ_OP_FLUSH) && - !(info->feature_flush & REQ_FLUSH)) || + !(blk_queue_flush(info->rq))) || ((req->cmd_flags & REQ_FUA) && - !(info->feature_flush & REQ_FUA))); + !(blk_queue_fua(info->rq)))); } static int blkif_queue_rq(struct blk_mq_hw_ctx *hctx, @@ -838,23 +835,30 @@ static int xlvbd_init_blk_queue(struct gendisk *gd, u16 sector_size, return 0; } -static const char *flush_info(unsigned int feature_flush) +static const char *flush_info(struct blkfront_info *info) { - switch (feature_flush & ((REQ_FLUSH | REQ_FUA))) { - case REQ_FLUSH|REQ_FUA: + if (blk_queue_flush(info->rq) && blk_queue_fua(info->rq)) return "barrier: enabled;"; - case REQ_FLUSH: + else if (blk_queue_flush(info->rq)) return "flush diskcache: enabled;"; - default: + else return "barrier or flush: disabled;"; - } } static void xlvbd_flush(struct blkfront_info *info) { - blk_queue_flush(info->rq, info->feature_flush); + if (info->feature_flush) + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, info->rq); + else + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, info->rq); + + if (info->feature_fua) + queue_flag_set_unlocked(QUEUE_FLAG_FUA, info->rq); + else + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, info->rq); + pr_info("blkfront: %s: %s %s %s %s %s\n", - info->gd->disk_name, flush_info(info->feature_flush), + info->gd->disk_name, flush_info(info), "persistent grants:", info->feature_persistent ? "enabled;" : "disabled;", "indirect descriptors:", info->max_indirect_segments ? "enabled;" : "disabled;"); @@ -1373,6 +1377,7 @@ static irqreturn_t blkif_interrupt(int irq, void *dev_id) if (unlikely(error)) { if (error == -EOPNOTSUPP) error = 0; + info->feature_fua = 0; info->feature_flush = 0; xlvbd_flush(info); } @@ -1936,6 +1941,7 @@ static int blkfront_gather_backend_features(struct blkfront_info *info) unsigned int indirect_segments; info->feature_flush = 0; + info->feature_fua = 0; err = xenbus_gather(XBT_NIL, info->xbdev->otherend, "feature-barrier", "%d", &barrier, @@ -1948,8 +1954,11 @@ static int blkfront_gather_backend_features(struct blkfront_info *info) * * If there are barriers, then we use flush. */ - if (!err && barrier) - info->feature_flush = REQ_FLUSH | REQ_FUA; + if (!err && barrier) { + info->feature_flush = 1; + info->feature_fua = 1; + } + /* * And if there is "feature-flush-cache" use that above * barriers. @@ -1958,8 +1967,10 @@ static int blkfront_gather_backend_features(struct blkfront_info *info) "feature-flush-cache", "%d", &flush, NULL); - if (!err && flush) - info->feature_flush = REQ_FLUSH; + if (!err && flush) { + info->feature_flush = 1; + info->feature_fua = 0; + } err = xenbus_gather(XBT_NIL, info->xbdev->otherend, "feature-discard", "%d", &discard, diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c index 6474ed3..4501ca7 100644 --- a/drivers/ide/ide-disk.c +++ b/drivers/ide/ide-disk.c @@ -522,8 +522,8 @@ static int ide_do_setfeature(ide_drive_t *drive, u8 feature, u8 nsect) static void update_flush(ide_drive_t *drive) { u16 *id = drive->id; - unsigned flush = 0; + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, drive->queue); if (drive->dev_flags & IDE_DFLAG_WCACHE) { unsigned long long capacity; int barrier; @@ -546,12 +546,10 @@ static void update_flush(ide_drive_t *drive) drive->name, barrier ? "" : "not "); if (barrier) { - flush = REQ_FLUSH; blk_queue_prep_rq(drive->queue, idedisk_prep_fn); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, drive->queue); } } - - blk_queue_flush(drive->queue, flush); } ide_devset_get_flag(wcache, IDE_DFLAG_WCACHE); diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index b11f7f4..3d75a0f 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -820,8 +820,8 @@ static int bcache_device_init(struct bcache_device *d, unsigned block_size, clear_bit(QUEUE_FLAG_ADD_RANDOM, &d->disk->queue->queue_flags); set_bit(QUEUE_FLAG_DISCARD, &d->disk->queue->queue_flags); - blk_queue_flush(q, REQ_FLUSH|REQ_FUA); - + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); return 0; } diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 061152a..9a4b3cb 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1312,13 +1312,21 @@ static void dm_table_verify_integrity(struct dm_table *t) static int device_flush_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { - unsigned flush = (*(unsigned *)data); struct request_queue *q = bdev_get_queue(dev->bdev); - return q && (q->flush_flags & flush); + return q && blk_queue_flush(q); } -static bool dm_table_supports_flush(struct dm_table *t, unsigned flush) +static int device_fua_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return q && blk_queue_fua(q); +} + +static bool dm_table_supports_flush(struct dm_table *t, + iterate_devices_callout_fn callout_fn) { struct dm_target *ti; unsigned i = 0; @@ -1339,7 +1347,7 @@ static bool dm_table_supports_flush(struct dm_table *t, unsigned flush) return true; if (ti->type->iterate_devices && - ti->type->iterate_devices(ti, device_flush_capable, &flush)) + ti->type->iterate_devices(ti, callout_fn, NULL)) return true; } @@ -1470,8 +1478,6 @@ static bool dm_table_supports_discards(struct dm_table *t) void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, struct queue_limits *limits) { - unsigned flush = 0; - /* * Copy table's limits to the DM device's request_queue */ @@ -1482,12 +1488,14 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, else queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, q); - if (dm_table_supports_flush(t, REQ_FLUSH)) { - flush |= REQ_FLUSH; - if (dm_table_supports_flush(t, REQ_FUA)) - flush |= REQ_FUA; - } - blk_queue_flush(q, flush); + if (dm_table_supports_flush(t, device_flush_capable)) { + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); + if (dm_table_supports_flush(t, device_fua_capable)) + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); + else + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, q); + } else + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, q); if (!dm_table_discard_zeroes_data(t)) q->limits.discard_zeroes_data = 0; diff --git a/drivers/md/md.c b/drivers/md/md.c index a2199ad..36b61e3 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -5036,7 +5036,8 @@ static int md_alloc(dev_t dev, char *name) disk->fops = &md_fops; disk->private_data = mddev; disk->queue = mddev->queue; - blk_queue_flush(mddev->queue, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, mddev->queue); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, mddev->queue); /* Allow extended partitions. This makes the * 'mdp' device redundant, but we can't really * remove it now. diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index c4624b3..a3e586d 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -1203,6 +1203,7 @@ ioerr: int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev) { + struct request_queue *q = rdev->bdev->bd_disk->queue; struct r5l_log *log; if (PAGE_SIZE != 4096) @@ -1212,7 +1213,7 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev) return -ENOMEM; log->rdev = rdev; - log->need_cache_flush = (rdev->bdev->bd_disk->queue->flush_flags != 0); + log->need_cache_flush = (blk_queue_flush(q) || blk_queue_fua(q)); log->uuid_checksum = crc32c_le(~0, rdev->mddev->uuid, sizeof(rdev->mddev->uuid)); diff --git a/drivers/mmc/card/block.c b/drivers/mmc/card/block.c index 7caf236..c1fad5f 100644 --- a/drivers/mmc/card/block.c +++ b/drivers/mmc/card/block.c @@ -2277,7 +2277,8 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct mmc_card *card, ((card->ext_csd.rel_param & EXT_CSD_WR_REL_PARAM_EN) || card->ext_csd.rel_sectors)) { md->flags |= MMC_BLK_REL_WR; - blk_queue_flush(md->queue.queue, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, md->queue.queue); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, md->queue.queue); } if (mmc_card_mmc(card) && diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c index 67da1cd..4462627 100644 --- a/drivers/mtd/mtd_blkdevs.c +++ b/drivers/mtd/mtd_blkdevs.c @@ -409,7 +409,7 @@ int add_mtd_blktrans_dev(struct mtd_blktrans_dev *new) goto error3; if (tr->flush) - blk_queue_flush(new->rq, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, new->rq); new->rq->queuedata = new; blk_queue_logical_block_size(new->rq, tr->blksize); diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 1375a83..9f253bc 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -1078,8 +1078,10 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) } if (ctrl->stripe_size) blk_queue_chunk_sectors(ns->queue, ctrl->stripe_size >> 9); - if (ctrl->vwc & NVME_CTRL_VWC_PRESENT) - blk_queue_flush(ns->queue, REQ_FLUSH | REQ_FUA); + if (ctrl->vwc & NVME_CTRL_VWC_PRESENT) { + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, ns->queue); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, ns->queue); + } blk_queue_virt_boundary(ns->queue, ctrl->page_size - 1); disk->major = nvme_major; diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 8585b6f..a4e5b6f 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -137,15 +137,18 @@ static const char *sd_cache_types[] = { static void sd_set_flush_flag(struct scsi_disk *sdkp) { - unsigned flush = 0; + struct request_queue *q = sdkp->disk->queue; if (sdkp->WCE) { - flush |= REQ_FLUSH; + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, q); if (sdkp->DPOFUA) - flush |= REQ_FUA; + queue_flag_set_unlocked(QUEUE_FLAG_FUA, q); + else + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, q); + } else { + queue_flag_clear_unlocked(QUEUE_FLAG_FUA, q); + queue_flag_clear_unlocked(QUEUE_FLAG_FLUSH, q); } - - blk_queue_flush(sdkp->disk->queue, flush); } static ssize_t diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c index 30f4f41..a85a391 100644 --- a/drivers/target/target_core_iblock.c +++ b/drivers/target/target_core_iblock.c @@ -671,11 +671,11 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, * Force writethrough using WRITE_FUA if a volatile write cache * is not enabled, or if initiator set the Force Unit Access bit. */ - if (q->flush_flags & REQ_FUA) { + if (blk_queue_fua(q)) { if (cmd->se_cmd_flags & SCF_FUA) { op = REQ_OP_WRITE; op_flags = WRITE_FUA; - } else if (!(q->flush_flags & REQ_FLUSH)) { + } else if (!blk_queue_flush(q)) { op = REQ_OP_WRITE; op_flags = WRITE_FUA; } else { @@ -842,7 +842,7 @@ static bool iblock_get_write_cache(struct se_device *dev) struct block_device *bd = ib_dev->ibd_bd; struct request_queue *q = bdev_get_queue(bd); - return q->flush_flags & REQ_FLUSH; + return blk_queue_flush(q); } static const struct target_backend_ops iblock_ops = { diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index ec0d0d0..09724ed 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -434,7 +434,6 @@ struct request_queue { /* * for flush operations */ - unsigned int flush_flags; unsigned int flush_not_queueable:1; struct blk_flush_queue *fq; @@ -492,6 +491,8 @@ struct request_queue { #define QUEUE_FLAG_INIT_DONE 20 /* queue is initialized */ #define QUEUE_FLAG_NO_SG_MERGE 21 /* don't attempt to merge SG segments*/ #define QUEUE_FLAG_POLL 22 /* IO polling enabled if set */ +#define QUEUE_FLAG_FLUSH 23 /* supports FLUSH/PREFLUSH */ +#define QUEUE_FLAG_FUA 24 /* supports FUA */ #define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \ (1 << QUEUE_FLAG_STACKABLE) | \ @@ -580,6 +581,8 @@ static inline void queue_flag_clear(unsigned int flag, struct request_queue *q) #define blk_queue_discard(q) test_bit(QUEUE_FLAG_DISCARD, &(q)->queue_flags) #define blk_queue_secdiscard(q) (blk_queue_discard(q) && \ test_bit(QUEUE_FLAG_SECDISCARD, &(q)->queue_flags)) +#define blk_queue_flush(q) test_bit(QUEUE_FLAG_FLUSH, &(q)->queue_flags) +#define blk_queue_fua(q) test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags) #define blk_noretry_request(rq) \ ((rq)->cmd_flags & (REQ_FAILFAST_DEV|REQ_FAILFAST_TRANSPORT| \ @@ -1007,7 +1010,6 @@ extern void blk_queue_update_dma_alignment(struct request_queue *, int); extern void blk_queue_softirq_done(struct request_queue *, softirq_done_fn *); extern void blk_queue_rq_timed_out(struct request_queue *, rq_timed_out_fn *); extern void blk_queue_rq_timeout(struct request_queue *, unsigned int); -extern void blk_queue_flush(struct request_queue *q, unsigned int flush); extern void blk_queue_flush_queueable(struct request_queue *q, bool queueable); extern struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev); -- 1.8.3.1 From mchristi@redhat.com Mon Jan 11 14:23:08 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 4873E29E74 for ; Mon, 11 Jan 2016 14:23:08 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id E19988F8049 for ; Mon, 11 Jan 2016 12:23:07 -0800 (PST) X-ASG-Debug-ID: 1452543783-04bdf06f6b7db80001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 5OWb6owumKFLASxP (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 12:23:03 -0800 (PST) X-Barracuda-Envelope-From: mchristi@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id DD947350102; Mon, 11 Jan 2016 20:23:02 +0000 (UTC) Received: from rh2.redhat.com (unused [10.10.50.149] (may be forged)) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0BKLZF4008385; Mon, 11 Jan 2016 15:23:00 -0500 From: mchristi@redhat.com To: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org, konrad.wilk@oracle.com, drbd-dev@lists.linbit.com, philipp.reisner@linbit.com, lars.ellenberg@linbit.com, linux-raid@vger.kernel.org, dm-devel@redhat.com, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org, target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org, osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com Cc: Mike Christie Subject: [PATCH 35/35] block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH Date: Mon, 11 Jan 2016 14:21:33 -0600 X-ASG-Orig-Subj: [PATCH 35/35] block, drivers, fs: rename REQ_FLUSH to REQ_PREFLUSH Message-Id: <1452543693-4440-36-git-send-email-mchristi@redhat.com> In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com> References: <1452543693-4440-1-git-send-email-mchristi@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452543783 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 From: Mike Christie To avoid confusion between REQ_OP_FLUSH, which is handled by request_fn drivers, and upper layers requesting the block layer perform a flush sequence along with possibly a WRITE, this patch renames REQ_FLUSH to REQ_PREFLUSH. Signed-off-by: Mike Christie --- Documentation/block/writeback_cache_control.txt | 31 +++++++++++++------------ Documentation/device-mapper/log-writes.txt | 10 ++++---- block/blk-core.c | 12 +++++----- block/blk-flush.c | 16 ++++++------- block/blk-mq.c | 4 ++-- drivers/block/drbd/drbd_actlog.c | 4 ++-- drivers/block/drbd/drbd_main.c | 2 +- drivers/block/drbd/drbd_protocol.h | 2 +- drivers/block/drbd/drbd_receiver.c | 2 +- drivers/block/drbd/drbd_req.c | 2 +- drivers/md/bcache/journal.c | 2 +- drivers/md/bcache/request.c | 8 +++---- drivers/md/dm-cache-target.c | 12 +++++----- drivers/md/dm-crypt.c | 7 +++--- drivers/md/dm-era-target.c | 4 ++-- drivers/md/dm-io.c | 2 +- drivers/md/dm-log-writes.c | 2 +- drivers/md/dm-raid1.c | 5 ++-- drivers/md/dm-region-hash.c | 4 ++-- drivers/md/dm-snap.c | 6 ++--- drivers/md/dm-stripe.c | 2 +- drivers/md/dm-thin.c | 8 +++---- drivers/md/dm.c | 12 +++++----- drivers/md/linear.c | 2 +- drivers/md/md.c | 2 +- drivers/md/md.h | 2 +- drivers/md/multipath.c | 2 +- drivers/md/raid0.c | 2 +- drivers/md/raid1.c | 3 ++- drivers/md/raid10.c | 2 +- drivers/md/raid5-cache.c | 2 +- drivers/md/raid5.c | 2 +- fs/btrfs/check-integrity.c | 8 +++---- fs/jbd2/journal.c | 2 +- fs/xfs/xfs_buf.c | 2 +- include/linux/blk_types.h | 8 +++---- include/linux/fs.h | 4 ++-- include/trace/events/f2fs.h | 2 +- kernel/trace/blktrace.c | 5 ++-- 39 files changed, 107 insertions(+), 102 deletions(-) diff --git a/Documentation/block/writeback_cache_control.txt b/Documentation/block/writeback_cache_control.txt index ea5550f..9869f18 100644 --- a/Documentation/block/writeback_cache_control.txt +++ b/Documentation/block/writeback_cache_control.txt @@ -20,11 +20,11 @@ a forced cache flush, and the Force Unit Access (FUA) flag for requests. Explicit cache flushes ---------------------- -The REQ_FLUSH flag can be OR ed into the r/w flags of a bio submitted from +The REQ_PREFLUSH flag can be OR ed into the r/w flags of a bio submitted from the filesystem and will make sure the volatile cache of the storage device has been flushed before the actual I/O operation is started. This explicitly guarantees that previously completed write requests are on non-volatile -storage before the flagged bio starts. In addition the REQ_FLUSH flag can be +storage before the flagged bio starts. In addition the REQ_PREFLUSH flag can be set on an otherwise empty bio structure, which causes only an explicit cache flush without any dependent I/O. It is recommend to use the blkdev_issue_flush() helper for a pure cache flush. @@ -41,21 +41,21 @@ signaled after the data has been committed to non-volatile storage. Implementation details for filesystems -------------------------------------- -Filesystems can simply set the REQ_FLUSH and REQ_FUA bits and do not have to +Filesystems can simply set the REQ_PREFLUSH and REQ_FUA bits and do not have to worry if the underlying devices need any explicit cache flushing and how -the Forced Unit Access is implemented. The REQ_FLUSH and REQ_FUA flags +the Forced Unit Access is implemented. The REQ_PREFLUSH and REQ_FUA flags may both be set on a single bio. Implementation details for make_request_fn based block drivers -------------------------------------------------------------- -These drivers will always see the REQ_FLUSH and REQ_FUA bits as they sit +These drivers will always see the REQ_PREFLUSH and REQ_FUA bits as they sit directly below the submit_bio interface. For remapping drivers the REQ_FUA bits need to be propagated to underlying devices, and a global flush needs -to be implemented for bios with the REQ_FLUSH bit set. For real device -drivers that do not have a volatile cache the REQ_FLUSH and REQ_FUA bits -on non-empty bios can simply be ignored, and REQ_FLUSH requests without +to be implemented for bios with the REQ_PREFLUSH bit set. For real device +drivers that do not have a volatile cache the REQ_PREFLUSH and REQ_FUA bits +on non-empty bios can simply be ignored, and REQ_PREFLUSH requests without data can be completed successfully without doing any work. Drivers for devices with volatile caches need to implement the support for these flags themselves without any help from the block layer. @@ -65,21 +65,22 @@ Implementation details for request_fn based block drivers -------------------------------------------------------------- For devices that do not support volatile write caches there is no driver -support required, the block layer completes empty REQ_FLUSH requests before -entering the driver and strips off the REQ_FLUSH and REQ_FUA bits from -requests that have a payload. For devices with volatile write caches the -driver needs to tell the block layer that it supports flushing caches by +support required, the block layer completes empty REQ_PREFLUSH requests +before entering the driver and strips off the REQ_PREFLUSH and REQ_FUA bits +from requests that have a payload. For devices with volatile write caches +the driver needs to tell the block layer that it supports flushing caches by doing: - blk_queue_flush(sdkp->disk->queue, REQ_FLUSH); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, sdkp->disk->queue); and handle empty REQ_OP_FLUSH requests in its prep_fn/request_fn. Note that -REQ_FLUSH requests with a payload are automatically turned into a sequence +REQ_PREFLUSH requests with a payload are automatically turned into a sequence of an empty REQ_OP_FLUSH request followed by the actual write by the block layer. For devices that also support the FUA bit the block layer needs to be told to pass through the REQ_FUA bit using: - blk_queue_flush(sdkp->disk->queue, REQ_FLUSH | REQ_FUA); + queue_flag_set_unlocked(QUEUE_FLAG_FLUSH, sdkp->disk->queue); + queue_flag_set_unlocked(QUEUE_FLAG_FUA, sdkp->disk->queue); and the driver must handle write requests that have the REQ_FUA bit set in prep_fn/request_fn. If the FUA bit is not natively supported the block diff --git a/Documentation/device-mapper/log-writes.txt b/Documentation/device-mapper/log-writes.txt index c10f30c..f4ebcba 100644 --- a/Documentation/device-mapper/log-writes.txt +++ b/Documentation/device-mapper/log-writes.txt @@ -14,14 +14,14 @@ Log Ordering We log things in order of completion once we are sure the write is no longer in cache. This means that normal WRITE requests are not actually logged until the -next REQ_FLUSH request. This is to make it easier for userspace to replay the -log in a way that correlates to what is on disk and not what is in cache, to -make it easier to detect improper waiting/flushing. +next REQ_PREFLUSH request. This is to make it easier for userspace to replay +the log in a way that correlates to what is on disk and not what is in cache, +to make it easier to detect improper waiting/flushing. This works by attaching all WRITE requests to a list once the write completes. -Once we see a REQ_FLUSH request we splice this list onto the request and once +Once we see a REQ_PREFLUSH request we splice this list onto the request and once the FLUSH request completes we log all of the WRITEs and then the FLUSH. Only -completed WRITEs, at the time the REQ_FLUSH is issued, are added in order to +completed WRITEs, at the time the REQ_PREFLUSH is issued, are added in order to simulate the worst case scenario with regard to power failures. Consider the following example (W means write, C means complete): diff --git a/block/blk-core.c b/block/blk-core.c index cfa70f7..1abda51 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1029,7 +1029,7 @@ static bool blk_rq_should_init_elevator(struct bio *bio) * Flush requests do not use the elevator so skip initialization. * This allows a request to share the flush and elevator data. */ - if (bio->bi_rw & (REQ_FLUSH | REQ_FUA)) + if (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA)) return false; return true; @@ -1736,7 +1736,7 @@ static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio) return BLK_QC_T_NONE; } - if (bio->bi_rw & (REQ_FLUSH | REQ_FUA)) { + if (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA)) { spin_lock_irq(q->queue_lock); where = ELEVATOR_INSERT_FLUSH; goto get_rq; @@ -1968,9 +1968,9 @@ generic_make_request_checks(struct bio *bio) * drivers without flush support don't have to worry * about them. */ - if ((bio->bi_rw & (REQ_FLUSH | REQ_FUA)) && + if ((bio->bi_rw & (REQ_PREFLUSH | REQ_FUA)) && !(blk_queue_flush(q) || blk_queue_fua(q))) { - bio->bi_rw &= ~(REQ_FLUSH | REQ_FUA); + bio->bi_rw &= ~(REQ_PREFLUSH | REQ_FUA); if (!nr_sectors) { err = 0; goto end_io; @@ -2217,7 +2217,7 @@ int blk_insert_cloned_request(struct request_queue *q, struct request *rq) */ BUG_ON(blk_queued_rq(rq)); - if (rq->cmd_flags & (REQ_FLUSH|REQ_FUA)) + if (rq->cmd_flags & (REQ_PREFLUSH | REQ_FUA)) where = ELEVATOR_INSERT_FLUSH; add_acct_request(q, rq, where); @@ -3309,7 +3309,7 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) /* * rq is already accounted, so use raw insert */ - if (rq->cmd_flags & (REQ_FLUSH | REQ_FUA)) + if (rq->cmd_flags & (REQ_PREFLUSH | REQ_FUA)) __elv_add_request(q, rq, ELEVATOR_INSERT_FLUSH); else __elv_add_request(q, rq, ELEVATOR_INSERT_SORT_MERGE); diff --git a/block/blk-flush.c b/block/blk-flush.c index e07ca6c..e784cd1 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -10,8 +10,8 @@ * optional steps - PREFLUSH, DATA and POSTFLUSH - according to the request * properties and hardware capability. * - * If a request doesn't have data, only REQ_FLUSH makes sense, which - * indicates a simple flush request. If there is data, REQ_FLUSH indicates + * If a request doesn't have data, only REQ_PREFLUSH makes sense, which + * indicates a simple flush request. If there is data, REQ_PREFLUSH indicates * that the device cache should be flushed before the data is executed, and * REQ_FUA means that the data must be on non-volatile media on request * completion. @@ -20,11 +20,11 @@ * difference. The requests are either completed immediately if there's no * data or executed as normal requests otherwise. * - * If the device has writeback cache and supports FUA, REQ_FLUSH is + * If the device has writeback cache and supports FUA, REQ_PREFLUSH is * translated to PREFLUSH but REQ_FUA is passed down directly with DATA. * - * If the device has writeback cache and doesn't support FUA, REQ_FLUSH is - * translated to PREFLUSH and REQ_FUA to POSTFLUSH. + * If the device has writeback cache and doesn't support FUA, REQ_PREFLUSH + * is translated to PREFLUSH and REQ_FUA to POSTFLUSH. * * The actual execution of flush is double buffered. Whenever a request * needs to execute PRE or POSTFLUSH, it queues at @@ -104,7 +104,7 @@ static unsigned int blk_flush_policy(struct request *rq) policy |= REQ_FSEQ_DATA; if (blk_queue_flush(q)) { - if (rq->cmd_flags & REQ_FLUSH) + if (rq->cmd_flags & REQ_PREFLUSH) policy |= REQ_FSEQ_PREFLUSH; if (!blk_queue_fua(q) && (rq->cmd_flags & REQ_FUA)) policy |= REQ_FSEQ_POSTFLUSH; @@ -391,9 +391,9 @@ void blk_insert_flush(struct request *rq) /* * @policy now records what operations need to be done. Adjust - * REQ_FLUSH and FUA for the driver. + * REQ_PREFLUSH and FUA for the driver. */ - rq->cmd_flags &= ~REQ_FLUSH; + rq->cmd_flags &= ~REQ_PREFLUSH; if (!blk_queue_fua(q)) rq->cmd_flags &= ~REQ_FUA; diff --git a/block/blk-mq.c b/block/blk-mq.c index 72a028b..bb376a8 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1244,7 +1244,7 @@ static int blk_mq_direct_issue_request(struct request *rq, blk_qc_t *cookie) static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) { const int is_sync = rw_is_sync(bio->bi_op, bio->bi_rw); - const int is_flush_fua = bio->bi_rw & (REQ_FLUSH | REQ_FUA); + const int is_flush_fua = bio->bi_rw & (REQ_PREFLUSH | REQ_FUA); struct blk_map_ctx data; struct request *rq; unsigned int request_count = 0; @@ -1341,7 +1341,7 @@ done: static blk_qc_t blk_sq_make_request(struct request_queue *q, struct bio *bio) { const int is_sync = rw_is_sync(bio->bi_op, bio->bi_rw); - const int is_flush_fua = bio->bi_rw & (REQ_FLUSH | REQ_FUA); + const int is_flush_fua = bio->bi_rw & (REQ_PREFLUSH | REQ_FUA); struct blk_plug *plug; unsigned int request_count = 0; struct blk_map_ctx data; diff --git a/drivers/block/drbd/drbd_actlog.c b/drivers/block/drbd/drbd_actlog.c index 2fa8534..b12ff76 100644 --- a/drivers/block/drbd/drbd_actlog.c +++ b/drivers/block/drbd/drbd_actlog.c @@ -148,7 +148,7 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, device->md_io.error = -ENODEV; if ((op == REQ_OP_WRITE) && !test_bit(MD_NO_FUA, &device->flags)) - op_flags |= REQ_FUA | REQ_FLUSH; + op_flags |= REQ_FUA | REQ_PREFLUSH; op_flags |= REQ_SYNC | REQ_NOIDLE; bio = bio_alloc_drbd(GFP_NOIO); @@ -848,7 +848,7 @@ int __drbd_change_sync(struct drbd_device *device, sector_t sector, int size, unsigned long count = 0; sector_t esector, nr_sectors; - /* This would be an empty REQ_FLUSH, be silent. */ + /* This would be an empty REQ_PREFLUSH, be silent. */ if ((mode == SET_OUT_OF_SYNC) && size == 0) return 0; diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 8a767d3..7a22208 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -1609,7 +1609,7 @@ static u32 bio_flags_to_wire(struct drbd_connection *connection, if (connection->agreed_pro_version >= 95) return (bio->bi_rw & REQ_SYNC ? DP_RW_SYNC : 0) | (bio->bi_rw & REQ_FUA ? DP_FUA : 0) | - (bio->bi_rw & REQ_FLUSH ? DP_FLUSH : 0) | + (bio->bi_rw & REQ_PREFLUSH ? DP_FLUSH : 0) | (bio->bi_op == REQ_OP_DISCARD ? DP_DISCARD : 0); else return bio->bi_rw & REQ_SYNC ? DP_RW_SYNC : 0; diff --git a/drivers/block/drbd/drbd_protocol.h b/drivers/block/drbd/drbd_protocol.h index ef92453..129f8c7 100644 --- a/drivers/block/drbd/drbd_protocol.h +++ b/drivers/block/drbd/drbd_protocol.h @@ -112,7 +112,7 @@ struct p_header100 { #define DP_MAY_SET_IN_SYNC 4 #define DP_UNPLUG 8 /* not used anymore */ #define DP_FUA 16 /* equals REQ_FUA */ -#define DP_FLUSH 32 /* equals REQ_FLUSH */ +#define DP_FLUSH 32 /* equals REQ_PREFLUSH */ #define DP_DISCARD 64 /* equals REQ_DISCARD */ #define DP_SEND_RECEIVE_ACK 128 /* This is a proto B write request */ #define DP_SEND_WRITE_ACK 256 /* This is a proto C write request */ diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c index 880de0d..752948f 100644 --- a/drivers/block/drbd/drbd_receiver.c +++ b/drivers/block/drbd/drbd_receiver.c @@ -2159,7 +2159,7 @@ static unsigned long wire_flags_to_bio_flags(u32 dpf) { return (dpf & DP_RW_SYNC ? REQ_SYNC : 0) | (dpf & DP_FUA ? REQ_FUA : 0) | - (dpf & DP_FLUSH ? REQ_FLUSH : 0); + (dpf & DP_FLUSH ? REQ_PREFLUSH : 0); } static unsigned long wire_flags_to_bio_op(u32 dpf) diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c index 2255dcf..eef6e95 100644 --- a/drivers/block/drbd/drbd_req.c +++ b/drivers/block/drbd/drbd_req.c @@ -1132,7 +1132,7 @@ static int drbd_process_write_request(struct drbd_request *req) * replicating, in which case there is no point. */ if (unlikely(req->i.size == 0)) { /* The only size==0 bios we expect are empty flushes. */ - D_ASSERT(device, req->master_bio->bi_rw & REQ_FLUSH); + D_ASSERT(device, req->master_bio->bi_rw & REQ_PREFLUSH); if (remote) _req_mod(req, QUEUE_AS_DRBD_BARRIER); return remote; diff --git a/drivers/md/bcache/journal.c b/drivers/md/bcache/journal.c index 68fa0f0..6e0a330 100644 --- a/drivers/md/bcache/journal.c +++ b/drivers/md/bcache/journal.c @@ -627,7 +627,7 @@ static void journal_write_unlocked(struct closure *cl) bio->bi_iter.bi_sector = PTR_OFFSET(k, i); bio->bi_bdev = ca->bdev; bio->bi_op = REQ_OP_WRITE; - bio->bi_rw = REQ_SYNC|REQ_META|REQ_FLUSH|REQ_FUA; + bio->bi_rw = REQ_SYNC|REQ_META|REQ_PREFLUSH|REQ_FUA; bio->bi_iter.bi_size = sectors << 9; bio->bi_end_io = journal_write_endio; diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index be8cd63f..3947114 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -205,10 +205,10 @@ static void bch_data_insert_start(struct closure *cl) return bch_data_invalidate(cl); /* - * Journal writes are marked REQ_FLUSH; if the original write was a + * Journal writes are marked REQ_PREFLUSH; if the original write was a * flush, it'll wait on the journal write. */ - bio->bi_rw &= ~(REQ_FLUSH|REQ_FUA); + bio->bi_rw &= ~(REQ_PREFLUSH|REQ_FUA); do { unsigned i; @@ -668,7 +668,7 @@ static inline struct search *search_alloc(struct bio *bio, s->iop.write_prio = 0; s->iop.error = 0; s->iop.flags = 0; - s->iop.flush_journal = (bio->bi_rw & (REQ_FLUSH|REQ_FUA)) != 0; + s->iop.flush_journal = (bio->bi_rw & (REQ_PREFLUSH|REQ_FUA)) != 0; s->iop.wq = bcache_wq; return s; @@ -920,7 +920,7 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) bch_writeback_add(dc); s->iop.bio = bio; - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { /* Also need to send a flush to the backing device */ struct bio *flush = bio_alloc_bioset(GFP_NOIO, 0, dc->disk.bio_split); diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c index 4c54326..af98fac 100644 --- a/drivers/md/dm-cache-target.c +++ b/drivers/md/dm-cache-target.c @@ -788,7 +788,7 @@ static void check_if_tick_bio_needed(struct cache *cache, struct bio *bio) spin_lock_irqsave(&cache->lock, flags); if (cache->need_tick_bio && - !(bio->bi_rw & (REQ_FUA | REQ_FLUSH)) && + !(bio->bi_rw & (REQ_FUA | REQ_PREFLUSH)) && bio->bi_op != REQ_OP_DISCARD) { pb->tick = true; cache->need_tick_bio = false; @@ -830,7 +830,7 @@ static dm_oblock_t get_bio_block(struct cache *cache, struct bio *bio) static int bio_triggers_commit(struct cache *cache, struct bio *bio) { - return bio->bi_rw & (REQ_FLUSH | REQ_FUA); + return bio->bi_rw & (REQ_PREFLUSH | REQ_FUA); } /* @@ -1064,7 +1064,7 @@ static void dec_io_migrations(struct cache *cache) static bool discard_or_flush(struct bio *bio) { return bio->bi_op == REQ_OP_DISCARD || - bio->bi_rw & (REQ_FLUSH | REQ_FUA); + bio->bi_rw & (REQ_PREFLUSH | REQ_FUA); } static void __cell_defer(struct cache *cache, struct dm_bio_prison_cell *cell) @@ -1609,8 +1609,8 @@ static void process_flush_bio(struct cache *cache, struct bio *bio) remap_to_cache(cache, bio, 0); /* - * REQ_FLUSH is not directed at any particular block so we don't - * need to inc_ds(). REQ_FUA's are split into a write + REQ_FLUSH + * REQ_PREFLUSH is not directed at any particular block so we don't + * need to inc_ds(). REQ_FUA's are split into a write + REQ_PREFLUSH * by dm-core. */ issue(cache, bio); @@ -1975,7 +1975,7 @@ static void process_deferred_bios(struct cache *cache) bio = bio_list_pop(&bios); - if (bio->bi_rw & REQ_FLUSH) + if (bio->bi_rw & REQ_PREFLUSH) process_flush_bio(cache, bio); else if (bio->bi_op == REQ_OP_DISCARD) process_discard_bio(cache, &structs, bio); diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index b7cbd39..7e39d3c 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1909,11 +1909,12 @@ static int crypt_map(struct dm_target *ti, struct bio *bio) struct crypt_config *cc = ti->private; /* - * If bio is REQ_FLUSH or REQ_DISCARD, just bypass crypt queues. - * - for REQ_FLUSH device-mapper core ensures that no IO is in-flight + * If bio is REQ_PREFLUSH or REQ_DISCARD, just bypass crypt queues. + * - for REQ_PREFLUSH device-mapper core ensures that no IO is in-flight * - for REQ_DISCARD caller must use flush if IO ordering matters */ - if (unlikely(bio->bi_rw & REQ_FLUSH || bio->bi_op == REQ_OP_DISCARD)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH || + bio->bi_op == REQ_OP_DISCARD)) { bio->bi_bdev = cc->dev->bdev; if (bio_sectors(bio)) bio->bi_iter.bi_sector = cc->start + diff --git a/drivers/md/dm-era-target.c b/drivers/md/dm-era-target.c index 665bf32..2faf49d8 100644 --- a/drivers/md/dm-era-target.c +++ b/drivers/md/dm-era-target.c @@ -1540,9 +1540,9 @@ static int era_map(struct dm_target *ti, struct bio *bio) remap_to_origin(era, bio); /* - * REQ_FLUSH bios carry no data, so we're not interested in them. + * REQ_PREFLUSH bios carry no data, so we're not interested in them. */ - if (!(bio->bi_rw & REQ_FLUSH) && + if (!(bio->bi_rw & REQ_PREFLUSH) && (bio_data_dir(bio) == WRITE) && !metadata_current_marked(era->md, block)) { defer_bio(era, bio); diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c index 0f723ca..f96ba41 100644 --- a/drivers/md/dm-io.c +++ b/drivers/md/dm-io.c @@ -381,7 +381,7 @@ static void dispatch_io(int op, int op_flags, unsigned int num_regions, */ for (i = 0; i < num_regions; i++) { *dp = old_pages; - if (where[i].count || (op_flags & REQ_FLUSH)) + if (where[i].count || (op_flags & REQ_PREFLUSH)) do_region(op, op_flags, i, where + i, dp, io); } diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c index 813bb4d..0848a43 100644 --- a/drivers/md/dm-log-writes.c +++ b/drivers/md/dm-log-writes.c @@ -555,7 +555,7 @@ static int log_writes_map(struct dm_target *ti, struct bio *bio) struct bio_vec bv; size_t alloc_size; int i = 0; - bool flush_bio = (bio->bi_rw & REQ_FLUSH); + bool flush_bio = (bio->bi_rw & REQ_PREFLUSH); bool fua_bio = (bio->bi_rw & REQ_FUA); bool discard_bio = (bio->bi_op == REQ_OP_DISCARD); diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c index 54cc0c7..9f34923 100644 --- a/drivers/md/dm-raid1.c +++ b/drivers/md/dm-raid1.c @@ -704,7 +704,7 @@ static void do_writes(struct mirror_set *ms, struct bio_list *writes) bio_list_init(&requeue); while ((bio = bio_list_pop(writes))) { - if ((bio->bi_rw & REQ_FLUSH) || + if ((bio->bi_rw & REQ_PREFLUSH) || (bio->bi_op == REQ_OP_DISCARD)) { bio_list_add(&sync, bio); continue; @@ -1253,7 +1253,8 @@ static int mirror_end_io(struct dm_target *ti, struct bio *bio, int error) * We need to dec pending if this was a write. */ if (rw == WRITE) { - if (!(bio->bi_rw & REQ_FLUSH) && bio->bi_op != REQ_OP_DISCARD) + if (!(bio->bi_rw & REQ_PREFLUSH) && + bio->bi_op != REQ_OP_DISCARD) dm_rh_dec(ms->rh, bio_record->write_region); return error; } diff --git a/drivers/md/dm-region-hash.c b/drivers/md/dm-region-hash.c index ce4af57..d257428 100644 --- a/drivers/md/dm-region-hash.c +++ b/drivers/md/dm-region-hash.c @@ -398,7 +398,7 @@ void dm_rh_mark_nosync(struct dm_region_hash *rh, struct bio *bio) region_t region = dm_rh_bio_to_region(rh, bio); int recovering = 0; - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { rh->flush_failure = 1; return; } @@ -526,7 +526,7 @@ void dm_rh_inc_pending(struct dm_region_hash *rh, struct bio_list *bios) struct bio *bio; for (bio = bios->head; bio; bio = bio->bi_next) { - if (bio->bi_rw & REQ_FLUSH || bio->bi_op == REQ_OP_DISCARD) + if (bio->bi_rw & REQ_PREFLUSH || bio->bi_op == REQ_OP_DISCARD) continue; rh_inc(rh, dm_rh_bio_to_region(rh, bio)); } diff --git a/drivers/md/dm-snap.c b/drivers/md/dm-snap.c index 376638608..ad80d73 100644 --- a/drivers/md/dm-snap.c +++ b/drivers/md/dm-snap.c @@ -1671,7 +1671,7 @@ static int snapshot_map(struct dm_target *ti, struct bio *bio) init_tracked_chunk(bio); - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { bio->bi_bdev = s->cow->bdev; return DM_MAPIO_REMAPPED; } @@ -1790,7 +1790,7 @@ static int snapshot_merge_map(struct dm_target *ti, struct bio *bio) init_tracked_chunk(bio); - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { if (!dm_bio_get_target_bio_nr(bio)) bio->bi_bdev = s->origin->bdev; else @@ -2276,7 +2276,7 @@ static int origin_map(struct dm_target *ti, struct bio *bio) bio->bi_bdev = o->dev->bdev; - if (unlikely(bio->bi_rw & REQ_FLUSH)) + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) return DM_MAPIO_REMAPPED; if (bio_rw(bio) != WRITE) diff --git a/drivers/md/dm-stripe.c b/drivers/md/dm-stripe.c index 12b1630..e6f330c 100644 --- a/drivers/md/dm-stripe.c +++ b/drivers/md/dm-stripe.c @@ -286,7 +286,7 @@ static int stripe_map(struct dm_target *ti, struct bio *bio) uint32_t stripe; unsigned target_bio_nr; - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { target_bio_nr = dm_bio_get_target_bio_nr(bio); BUG_ON(target_bio_nr >= sc->stripes); bio->bi_bdev = sc->stripe[target_bio_nr].dev->bdev; diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index 0b2c6c1..ce9b9b3 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -697,7 +697,7 @@ static void remap_to_origin(struct thin_c *tc, struct bio *bio) static int bio_triggers_commit(struct thin_c *tc, struct bio *bio) { - return (bio->bi_rw & (REQ_FLUSH | REQ_FUA)) && + return (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA)) && dm_thin_changed_this_transaction(tc->td); } @@ -868,7 +868,7 @@ static void __inc_remap_and_issue_cell(void *context, struct bio *bio; while ((bio = bio_list_pop(&cell->bios))) { - if (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + if (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA) || bio->bi_op == REQ_OP_DISCARD) bio_list_add(&info->defer_bios, bio); else { @@ -1647,7 +1647,7 @@ static void __remap_and_issue_shared_cell(void *context, while ((bio = bio_list_pop(&cell->bios))) { if ((bio_data_dir(bio) == WRITE) || - (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA) || bio->bi_op == REQ_OP_DISCARD)) bio_list_add(&info->defer_bios, bio); else { @@ -2560,7 +2560,7 @@ static int thin_bio_map(struct dm_target *ti, struct bio *bio) return DM_MAPIO_SUBMITTED; } - if (bio->bi_rw & (REQ_FLUSH | REQ_FUA) || + if (bio->bi_rw & (REQ_PREFLUSH | REQ_FUA) || bio->bi_op == REQ_OP_DISCARD) { thin_defer_bio_with_throttle(tc, bio); return DM_MAPIO_SUBMITTED; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index dfaaba2..d8dc3e7 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -966,12 +966,12 @@ static void dec_pending(struct dm_io *io, int error) if (io_error == DM_ENDIO_REQUEUE) return; - if ((bio->bi_rw & REQ_FLUSH) && bio->bi_iter.bi_size) { + if ((bio->bi_rw & REQ_PREFLUSH) && bio->bi_iter.bi_size) { /* * Preflush done for flush with data, reissue - * without REQ_FLUSH. + * without REQ_PREFLUSH. */ - bio->bi_rw &= ~REQ_FLUSH; + bio->bi_rw &= ~REQ_PREFLUSH; queue_io(md, bio); } else { /* done with normal IO or empty flush */ @@ -1430,7 +1430,7 @@ EXPORT_SYMBOL_GPL(dm_set_target_max_io_len); /* * A target may call dm_accept_partial_bio only from the map routine. It is - * allowed for all bio types except REQ_FLUSH. + * allowed for all bio types except REQ_PREFLUSH. * * dm_accept_partial_bio informs the dm that the target only wants to process * additional n_sectors sectors of the bio and the rest of the data should be @@ -1460,7 +1460,7 @@ void dm_accept_partial_bio(struct bio *bio, unsigned n_sectors) { struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone); unsigned bi_size = bio->bi_iter.bi_size >> SECTOR_SHIFT; - BUG_ON(bio->bi_rw & REQ_FLUSH); + BUG_ON(bio->bi_rw & REQ_PREFLUSH); BUG_ON(bi_size > *tio->len_ptr); BUG_ON(n_sectors > bi_size); *tio->len_ptr -= bi_size - n_sectors; @@ -1735,7 +1735,7 @@ static void __split_and_process_bio(struct mapped_device *md, start_io_acct(ci.io); - if (bio->bi_rw & REQ_FLUSH) { + if (bio->bi_rw & REQ_PREFLUSH) { ci.bio = &ci.md->flush_bio; ci.sector_count = 0; error = __send_empty_flush(&ci); diff --git a/drivers/md/linear.c b/drivers/md/linear.c index aad82c7..657053e 100644 --- a/drivers/md/linear.c +++ b/drivers/md/linear.c @@ -221,7 +221,7 @@ static void linear_make_request(struct mddev *mddev, struct bio *bio) struct bio *split; sector_t start_sector, end_sector, data_offset; - if (unlikely(bio->bi_rw & REQ_FLUSH)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) { md_flush_request(mddev, bio); return; } diff --git a/drivers/md/md.c b/drivers/md/md.c index 36b61e3..20a692e 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -412,7 +412,7 @@ static void md_submit_flush_data(struct work_struct *ws) /* an empty barrier - all done */ bio_endio(bio); else { - bio->bi_rw &= ~REQ_FLUSH; + bio->bi_rw &= ~REQ_PREFLUSH; mddev->pers->make_request(mddev, bio); } diff --git a/drivers/md/md.h b/drivers/md/md.h index 6dc5f66..a23964a 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -424,7 +424,7 @@ struct mddev { /* Generic flush handling. * The last to finish preflush schedules a worker to submit - * the rest of the request (without the REQ_FLUSH flag). + * the rest of the request (without the REQ_PREFLUSH flag). */ struct bio *flush_bio; atomic_t flush_pending; diff --git a/drivers/md/multipath.c b/drivers/md/multipath.c index 7331a80..b651af1 100644 --- a/drivers/md/multipath.c +++ b/drivers/md/multipath.c @@ -111,7 +111,7 @@ static void multipath_make_request(struct mddev *mddev, struct bio * bio) struct multipath_bh * mp_bh; struct multipath_info *multipath; - if (unlikely(bio->bi_rw & REQ_FLUSH)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) { md_flush_request(mddev, bio); return; } diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c index e0d1b8c..f95463d 100644 --- a/drivers/md/raid0.c +++ b/drivers/md/raid0.c @@ -458,7 +458,7 @@ static void raid0_make_request(struct mddev *mddev, struct bio *bio) struct md_rdev *tmp_dev; struct bio *split; - if (unlikely(bio->bi_rw & REQ_FLUSH)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) { md_flush_request(mddev, bio); return; } diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 2a42daa..2856b50 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1056,7 +1056,8 @@ static void make_request(struct mddev *mddev, struct bio * bio) const int op = bio->bi_op; const int rw = bio_data_dir(bio); const unsigned long do_sync = (bio->bi_rw & REQ_SYNC); - const unsigned long do_flush_fua = (bio->bi_rw & (REQ_FLUSH | REQ_FUA)); + const unsigned long do_flush_fua = (bio->bi_rw & + (REQ_PREFLUSH | REQ_FUA)); const unsigned long do_sec = (bio->bi_rw & REQ_SECURE); struct md_rdev *blocked_rdev; struct blk_plug_cb *cb; diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 1cfc061..6171ff7 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -1452,7 +1452,7 @@ static void make_request(struct mddev *mddev, struct bio *bio) struct bio *split; - if (unlikely(bio->bi_rw & REQ_FLUSH)) { + if (unlikely(bio->bi_rw & REQ_PREFLUSH)) { md_flush_request(mddev, bio); return; } diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index a3e586d..2c0c3b8 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -540,7 +540,7 @@ int r5l_handle_flush_request(struct r5conf *conf, struct bio *bio) bio_endio(bio); return 0; } - bio->bi_rw &= ~REQ_FLUSH; + bio->bi_rw &= ~REQ_PREFLUSH; return -EAGAIN; } diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 792366c..b7016d3 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -5154,7 +5154,7 @@ static void make_request(struct mddev *mddev, struct bio * bi) DEFINE_WAIT(w); bool do_prepare; - if (unlikely(bi->bi_rw & REQ_FLUSH)) { + if (unlikely(bi->bi_rw & REQ_PREFLUSH)) { int ret = r5l_handle_flush_request(conf, bi); if (ret == 0) diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c index 1623d11..e63ad0f 100644 --- a/fs/btrfs/check-integrity.c +++ b/fs/btrfs/check-integrity.c @@ -2237,7 +2237,7 @@ static void btrfsic_bio_end_io(struct bio *bp) block->dev_bytenr, block->mirror_num); next_block = block->next_in_same_bio; block->iodone_w_error = iodone_w_error; - if (block->submit_bio_bh_rw & REQ_FLUSH) { + if (block->submit_bio_bh_rw & REQ_PREFLUSH) { dev_state->last_flush_gen++; if ((dev_state->state->print_mask & BTRFSIC_PRINT_MASK_END_IO_BIO_BH)) @@ -2273,7 +2273,7 @@ static void btrfsic_bh_end_io(struct buffer_head *bh, int uptodate) block->dev_bytenr, block->mirror_num); block->iodone_w_error = iodone_w_error; - if (block->submit_bio_bh_rw & REQ_FLUSH) { + if (block->submit_bio_bh_rw & REQ_PREFLUSH) { dev_state->last_flush_gen++; if ((dev_state->state->print_mask & BTRFSIC_PRINT_MASK_END_IO_BIO_BH)) @@ -2926,7 +2926,7 @@ int btrfsic_submit_bh(int op, int op_flags, struct buffer_head *bh) btrfsic_process_written_block(dev_state, dev_bytenr, &bh->b_data, 1, NULL, NULL, bh, op_flags); - } else if (NULL != dev_state && (op_flags & REQ_FLUSH)) { + } else if (NULL != dev_state && (op_flags & REQ_PREFLUSH)) { if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO @@ -3024,7 +3024,7 @@ static void __btrfsic_submit_bio(struct bio *bio) kunmap(bio->bi_io_vec[i].bv_page); } kfree(mapped_datav); - } else if (NULL != dev_state && (bio->bi_rw & REQ_FLUSH)) { + } else if (NULL != dev_state && (bio->bi_rw & REQ_PREFLUSH)) { if (dev_state->state->print_mask & BTRFSIC_PRINT_MASK_SUBMIT_BIO_BH) printk(KERN_INFO diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c index 7104d76..2e4760b 100644 --- a/fs/jbd2/journal.c +++ b/fs/jbd2/journal.c @@ -1331,7 +1331,7 @@ static int jbd2_write_superblock(journal_t *journal, int write_flags) trace_jbd2_write_superblock(journal, write_flags); if (!(journal->j_flags & JBD2_BARRIER)) - write_flags &= ~(REQ_FUA | REQ_FLUSH); + write_flags &= ~(REQ_FUA | REQ_PREFLUSH); lock_buffer(bh); if (buffer_write_io_error(bh)) { /* diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index de35a3c..64c71f9 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1235,7 +1235,7 @@ _xfs_buf_ioapply( if (bp->b_flags & XBF_FUA) op_flags |= REQ_FUA; if (bp->b_flags & XBF_FLUSH) - op_flags |= REQ_FLUSH; + op_flags |= REQ_PREFLUSH; /* * Run the write verifier callback function if it exists. If diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index f9e53ca..2b6583a 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -158,7 +158,7 @@ enum rq_flag_bits { __REQ_NOIDLE, /* don't anticipate more IO after this one */ __REQ_INTEGRITY, /* I/O includes block integrity payload */ __REQ_FUA, /* forced unit access */ - __REQ_FLUSH, /* request for cache flush */ + __REQ_PREFLUSH, /* request for cache flush */ /* bio only flags */ __REQ_RAHEAD, /* read ahead, can fail anytime */ @@ -202,12 +202,12 @@ enum rq_flag_bits { (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER) #define REQ_COMMON_MASK \ (REQ_FAILFAST_MASK | REQ_SYNC | REQ_META | REQ_PRIO | REQ_NOIDLE | \ - REQ_FLUSH | REQ_FUA | REQ_SECURE | REQ_INTEGRITY) + REQ_PREFLUSH | REQ_FUA | REQ_SECURE | REQ_INTEGRITY) #define REQ_CLONE_MASK REQ_COMMON_MASK /* This mask is used for both bio and request merge checking */ #define REQ_NOMERGE_FLAGS \ - (REQ_NOMERGE | REQ_STARTED | REQ_SOFTBARRIER | REQ_FLUSH | REQ_FUA | REQ_FLUSH_SEQ) + (REQ_NOMERGE | REQ_STARTED | REQ_SOFTBARRIER | REQ_PREFLUSH | REQ_FUA | REQ_FLUSH_SEQ) #define REQ_RAHEAD (1ULL << __REQ_RAHEAD) #define REQ_THROTTLED (1ULL << __REQ_THROTTLED) @@ -225,7 +225,7 @@ enum rq_flag_bits { #define REQ_PREEMPT (1ULL << __REQ_PREEMPT) #define REQ_ALLOCED (1ULL << __REQ_ALLOCED) #define REQ_COPY_USER (1ULL << __REQ_COPY_USER) -#define REQ_FLUSH (1ULL << __REQ_FLUSH) +#define REQ_PREFLUSH (1ULL << __REQ_PREFLUSH) #define REQ_FLUSH_SEQ (1ULL << __REQ_FLUSH_SEQ) #define REQ_IO_STAT (1ULL << __REQ_IO_STAT) #define REQ_MIXED_MERGE (1ULL << __REQ_MIXED_MERGE) diff --git a/include/linux/fs.h b/include/linux/fs.h index 2fbed45..e8792be 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -203,9 +203,9 @@ typedef void (dax_iodone_t)(struct buffer_head *bh_map, int uptodate); #define READ_SYNC REQ_SYNC #define WRITE_SYNC (REQ_SYNC | REQ_NOIDLE) #define WRITE_ODIRECT REQ_SYNC -#define WRITE_FLUSH (REQ_SYNC | REQ_NOIDLE | REQ_FLUSH) +#define WRITE_FLUSH (REQ_SYNC | REQ_NOIDLE | REQ_PREFLUSH) #define WRITE_FUA (REQ_SYNC | REQ_NOIDLE | REQ_FUA) -#define WRITE_FLUSH_FUA (REQ_SYNC | REQ_NOIDLE | REQ_FLUSH | REQ_FUA) +#define WRITE_FLUSH_FUA (REQ_SYNC | REQ_NOIDLE | REQ_PREFLUSH | REQ_FUA) /* * Attribute flags. These should be or-ed together to figure out what diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index 1d568ee..0dc281a 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -33,7 +33,7 @@ TRACE_DEFINE_ENUM(SSR); TRACE_DEFINE_ENUM(__REQ_RAHEAD); TRACE_DEFINE_ENUM(__REQ_SYNC); TRACE_DEFINE_ENUM(__REQ_NOIDLE); -TRACE_DEFINE_ENUM(__REQ_FLUSH); +TRACE_DEFINE_ENUM(__REQ_PREFLUSH); TRACE_DEFINE_ENUM(__REQ_FUA); TRACE_DEFINE_ENUM(__REQ_PRIO); TRACE_DEFINE_ENUM(__REQ_META); diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index 3a54c83..c94dec7 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -189,6 +189,7 @@ static const u32 ddir_act[2] = { BLK_TC_ACT(BLK_TC_READ), BLK_TC_ACT(BLK_TC_WRITE) }; #define BLK_TC_RAHEAD BLK_TC_AHEAD +#define BLK_TC_PREFLUSH BLK_TC_FLUSH /* The ilog2() calls fall out because they're constant */ #define MASK_TC_BIT(rw, __name) ((rw & REQ_ ## __name) << \ @@ -219,7 +220,7 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes, what |= MASK_TC_BIT(op_flags, SYNC); what |= MASK_TC_BIT(op_flags, RAHEAD); what |= MASK_TC_BIT(op_flags, META); - what |= MASK_TC_BIT(op_flags, FLUSH); + what |= MASK_TC_BIT(op_flags, PREFLUSH); what |= MASK_TC_BIT(op_flags, FUA); if (op == REQ_OP_DISCARD) what |= BLK_TC_ACT(BLK_TC_DISCARD); @@ -1777,7 +1778,7 @@ void blk_fill_rwbs(char *rwbs, int op, u32 rw, int bytes) { int i = 0; - if (rw & REQ_FLUSH || + if (rw & REQ_PREFLUSH || op == REQ_OP_FLUSH) rwbs[i++] = 'F'; -- 1.8.3.1 From david@fromorbit.com Mon Jan 11 16:37:20 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E52297F56 for ; Mon, 11 Jan 2016 16:37:19 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 707D4AC002 for ; Mon, 11 Jan 2016 14:37:16 -0800 (PST) X-ASG-Debug-ID: 1452551833-04cbb05f7fb8910001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id aMXMc9FPDKYrTH9A for ; Mon, 11 Jan 2016 14:37:13 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AqEQATLpRWPGu7LXleKAECgw8iMG2CYoN+gXmgEQIWAQEBAQEBBoENilGFPYQHGAKFbwSBLE0BAQEBAQEHAQEBAUE/QRIBhD07JDQFJQMHLYgtn3GgRwkZhXWKaoIWDEGBNgWHZI8vhUOIDYFnjR0ChWSIbYIuCwEBAYJGKjSEL4IWAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 09:06:12 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aIl4N-0004Ae-5J for xfs@oss.sgi.com; Tue, 12 Jan 2016 09:36:11 +1100 Date: Tue, 12 Jan 2016 09:36:11 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [ANNOUNCE] xfs: for-next branch updated to dde7f55 Message-ID: <20160111223611.GJ10456@dastard> X-ASG-Orig-Subj: [ANNOUNCE] xfs: for-next branch updated to dde7f55 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="VrqPEDrXMn8OVzN4" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452551833 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26043 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- --VrqPEDrXMn8OVzN4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi folks, The for-next branch of the xfs kernel repository at git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git has just been updated. This should be the last update before asking Linus to pull the branch for the current merge window. If nothing shows up in the next couple of days, I'll send out a pull request... -Dave. The new head of the for-next branch is commit: dde7f55 Merge branch 'xfs-misc-fixes-for-4.5-2' into for-next New Commits: Dave Chinner (5): [121e213] xfs: add tracepoints to readpage calls [e354381] xfs: bmapbt checking on debug kernels too expensive [b79f4a1] xfs: inode recovery readahead can race with inode buffer cr= eation [7d6a13f] xfs: handle dquot buffer readahead in log recovery correctly [dde7f55] Merge branch 'xfs-misc-fixes-for-4.5-2' into for-next Eric Sandeen (1): [f6106ef] xfs: eliminate committed arg from xfs_bmap_finish Code Diffstat: fs/xfs/libxfs/xfs_attr.c | 141 +++++++-----------------------------= ---- fs/xfs/libxfs/xfs_attr_remote.c | 31 ++------- fs/xfs/libxfs/xfs_bmap.c | 16 +++-- fs/xfs/libxfs/xfs_bmap.h | 2 +- fs/xfs/libxfs/xfs_dquot_buf.c | 36 ++++++++-- fs/xfs/libxfs/xfs_inode_buf.c | 14 ++-- fs/xfs/libxfs/xfs_quota_defs.h | 2 +- fs/xfs/libxfs/xfs_shared.h | 1 + fs/xfs/xfs_aops.c | 2 + fs/xfs/xfs_bmap_util.c | 43 +++++------- fs/xfs/xfs_buf.c | 7 ++ fs/xfs/xfs_dquot.c | 13 ++-- fs/xfs/xfs_inode.c | 25 +++---- fs/xfs/xfs_iomap.c | 10 ++- fs/xfs/xfs_log_recover.c | 9 ++- fs/xfs/xfs_rtalloc.c | 3 +- fs/xfs/xfs_symlink.c | 12 +--- fs/xfs/xfs_trace.h | 26 ++++++++ 18 files changed, 159 insertions(+), 234 deletions(-) --=20 Dave Chinner david@fromorbit.com --VrqPEDrXMn8OVzN4 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBAgAGBQJWlC5aAAoJEK3oKUf0dfodQ3IP/3H9icA+AOk12PRwHHtaclA3 PWw3rdlVI7QLIPMr+2a+YBGAHKrwfnsESg02Tmb5lgDMpXZdz42f6565phakWFZH dMYDP0R9pbODwOF8jWmaFOPrcfVD3i5BiZxdqifUmzG5KEbChaco10Vgq+akLrMr NTWPYZIr/6yTvTApXyV+7g/jw1XyJgJTcXiKJPzJ/UKpqAlYX30Wy+HXvFu8oKLZ 1opPDPCHBiTjkdJ+wKZj8g6HGCSnwpbRWZRFjl0ltKEtVzs7vCrCLwrS+SqmJP3f YJ2Gm6EKq73yEpT8iQ5KY4rmAX63oiWAXAcUx1MunFIYyWdkyIuzYdzAevvVrDIy TiQicIBJV6f5LDkmBvE1xNiy7zZjILMNcoludQ8SlAIXCumlsAF5Uu0/vJ478fM3 bCu7MMFWtMSS43GacgMWA8IAL9sQQXdilCLZTqbYkrYrMdhM46uNeEcSzCKyrJCL zn2TPjpk5KMIW2eP2cWW0f55HiuIlxMyDQn5htcMNUYFS1XHAVvcu3Cgyn1RcIt0 WbPIJVN5jGBO1U6uBiEOzKZCtH26T03FdXKqlLEIoPkf4Un7CS/RgsyU1/ikLmwP VARFMQXlDHy5il45nqafUZxXRtGzCQd21JKBj/upGeGJm0fH9q1EBguGSUCe7RTX Bl74OTu4Ki4vVHuTmWI+ =KogP -----END PGP SIGNATURE----- --VrqPEDrXMn8OVzN4-- From darrick.wong@oracle.com Mon Jan 11 17:27:04 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id B1F6C29DF5 for ; Mon, 11 Jan 2016 17:27:04 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 9322A8F804C for ; Mon, 11 Jan 2016 15:27:01 -0800 (PST) X-ASG-Debug-ID: 1452554819-04cbb05f7dcbcb0001-NocioJ Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id FjE7fNKqmngT5xEj (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 15:26:59 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 141.146.126.69 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0BNQwuB011569 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 11 Jan 2016 23:26:58 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0BNQw8P016573 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 11 Jan 2016 23:26:58 GMT Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0BNQwTp012371; Mon, 11 Jan 2016 23:26:58 GMT Received: from localhost (/24.130.224.127) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 11 Jan 2016 15:26:58 -0800 Date: Mon, 11 Jan 2016 15:26:57 -0800 From: "Darrick J. Wong" To: Dave Chinner Cc: xfs@oss.sgi.com Subject: [PATCH] xfs: check sizes of XFS on-disk structures at compile time Message-ID: <20160111232657.GA7831@birch.djwong.org> X-ASG-Orig-Subj: [PATCH] xfs: check sizes of XFS on-disk structures at compile time MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69] X-Barracuda-Start-Time: 1452554819 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26045 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines Check the sizes of XFS on-disk structures when compiling the kernel. Use this to catch inadvertent changes in structure size due to padding and alignment issues, etc. Signed-off-by: Darrick J. Wong --- fs/xfs/libxfs/xfs_format.h | 4 ++++ fs/xfs/xfs_super.c | 45 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/fs/xfs/libxfs/xfs_format.h b/fs/xfs/libxfs/xfs_format.h index e2536bb..a589b47 100644 --- a/fs/xfs/libxfs/xfs_format.h +++ b/fs/xfs/libxfs/xfs_format.h @@ -1509,4 +1509,8 @@ struct xfs_acl { #define SGI_ACL_FILE_SIZE (sizeof(SGI_ACL_FILE)-1) #define SGI_ACL_DEFAULT_SIZE (sizeof(SGI_ACL_DEFAULT)-1) +#define XFS_CHECK_STRUCT_SIZE(structname, size) \ + BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(struct " \ + #structname ") is wrong, expected " #size) + #endif /* __XFS_FORMAT_H__ */ diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index 36bd882..31b69d1 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -1812,11 +1812,56 @@ xfs_destroy_workqueues(void) destroy_workqueue(xfs_alloc_wq); } +static void __init +xfs_check_ondisk_structs(void) +{ + /* on-disk structures */ + XFS_CHECK_STRUCT_SIZE(struct xfs_dsb, 264); + XFS_CHECK_STRUCT_SIZE(struct xfs_agf, 224); + XFS_CHECK_STRUCT_SIZE(struct xfs_agi, 336); + XFS_CHECK_STRUCT_SIZE(struct xfs_agfl, 36); + XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp, 8); + XFS_CHECK_STRUCT_SIZE(struct xfs_dinode, 176); + XFS_CHECK_STRUCT_SIZE(struct xfs_disk_dquot, 104); + XFS_CHECK_STRUCT_SIZE(struct xfs_dqblk, 136); + XFS_CHECK_STRUCT_SIZE(struct xfs_dsymlink_hdr, 56); + XFS_CHECK_STRUCT_SIZE(xfs_alloc_rec_t, 8); + XFS_CHECK_STRUCT_SIZE(xfs_alloc_key_t, 8); + XFS_CHECK_STRUCT_SIZE(xfs_alloc_ptr_t, 4); + XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_rec, 16); + XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_key, 4); + XFS_CHECK_STRUCT_SIZE(xfs_inobt_ptr_t, 4); + XFS_CHECK_STRUCT_SIZE(struct xfs_bmdr_block, 4); + XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_rec, 16); + XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_key, 8); + XFS_CHECK_STRUCT_SIZE(struct xfs_btree_block, 72); + XFS_CHECK_STRUCT_SIZE(struct xfs_acl_entry, 12); + XFS_CHECK_STRUCT_SIZE(struct xfs_acl, 4); + + /* log structures */ + XFS_CHECK_STRUCT_SIZE(struct xfs_trans_header, 16); + XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_32, 52); + XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_64, 56); + XFS_CHECK_STRUCT_SIZE(struct xfs_ictimestamp, 8); + XFS_CHECK_STRUCT_SIZE(struct xfs_icdinode, 176); + XFS_CHECK_STRUCT_SIZE(struct xfs_extent_32, 12); + XFS_CHECK_STRUCT_SIZE(struct xfs_extent_64, 16); + XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_32, 28); + XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64, 32); + XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32, 28); + XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64, 32); + XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat, 24); + XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat, 20); + XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log, 28); +} + STATIC int __init init_xfs_fs(void) { int error; + xfs_check_ondisk_structs(); + printk(KERN_INFO XFS_VERSION_STRING " with " XFS_BUILD_OPTIONS " enabled\n"); From darrick.wong@oracle.com Mon Jan 11 17:46:55 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 04C9B29DF5 for ; Mon, 11 Jan 2016 17:46:55 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 7B4C9AC004 for ; Mon, 11 Jan 2016 15:46:54 -0800 (PST) X-ASG-Debug-ID: 1452556010-04cb6c75db7d550001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id Gguo5H8A9xKiQPrr (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 11 Jan 2016 15:46:51 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0BNkjcQ020058 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 11 Jan 2016 23:46:46 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0BNkjws012582 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Mon, 11 Jan 2016 23:46:45 GMT Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0BNkjld020551; Mon, 11 Jan 2016 23:46:45 GMT Received: from localhost (/24.130.224.127) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 11 Jan 2016 15:46:45 -0800 Date: Mon, 11 Jan 2016 15:46:44 -0800 From: "Darrick J. Wong" To: Dave Chinner Cc: xfs@oss.sgi.com Subject: [PATCH] xfs_db: check on-disk structure sizes Message-ID: <20160111234644.GB7831@birch.djwong.org> X-ASG-Orig-Subj: [PATCH] xfs_db: check on-disk structure sizes MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0022.oracle.com [141.146.126.234] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1452556011 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26045 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines Check on-disk structure sizes against known values. Use this to catch inadvertent changes in structure size due to padding and alignment issues, etc. Signed-off-by: Darrick J. Wong --- db/Makefile | 2 - db/init.c | 3 + db/ondisk.c | 63 +++++++++++++++++++++++ db/ondisk.h | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++ libxfs/xfs_format.h | 4 + 5 files changed, 210 insertions(+), 1 deletion(-) create mode 100644 db/ondisk.c create mode 100644 db/ondisk.h diff --git a/db/Makefile b/db/Makefile index 8260da3..ba3e942 100644 --- a/db/Makefile +++ b/db/Makefile @@ -12,7 +12,7 @@ HFILES = addr.h agf.h agfl.h agi.h attr.h attrshort.h bit.h block.h bmap.h \ dir2.h dir2sf.h dquot.h echo.h faddr.h field.h \ flist.h fprint.h frag.h freesp.h hash.h help.h init.h inode.h input.h \ io.h logformat.h malloc.h metadump.h output.h print.h quit.h sb.h \ - sig.h strvec.h text.h type.h write.h attrset.h symlink.h + sig.h strvec.h text.h type.h write.h attrset.h symlink.h ondisk.h CFILES = $(HFILES:.h=.c) LSRCFILES = xfs_admin.sh xfs_ncheck.sh xfs_metadump.sh diff --git a/db/init.c b/db/init.c index c0472c8..d6df093 100644 --- a/db/init.c +++ b/db/init.c @@ -28,6 +28,7 @@ #include "output.h" #include "malloc.h" #include "type.h" +#include "ondisk.h" static char **cmdline; static int ncmdline; @@ -60,6 +61,8 @@ init( struct xfs_buf *bp; int c; + xfs_check_ondisk_structs(); + setlocale(LC_ALL, ""); bindtextdomain(PACKAGE, LOCALEDIR); textdomain(PACKAGE); diff --git a/db/ondisk.c b/db/ondisk.c new file mode 100644 index 0000000..532333d --- /dev/null +++ b/db/ondisk.c @@ -0,0 +1,63 @@ +/* + * Copyright (c) 2016 Oracle. + * All Rights Reserved. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it would be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libxfs.h" +#include "ondisk.h" + +void +xfs_check_ondisk_structs(void) +{ + /* on-disk structures */ + XFS_CHECK_STRUCT_SIZE(struct xfs_dsb, 264); + XFS_CHECK_STRUCT_SIZE(struct xfs_agf, 224); + XFS_CHECK_STRUCT_SIZE(struct xfs_agi, 336); + XFS_CHECK_STRUCT_SIZE(struct xfs_agfl, 36); + XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp, 8); + XFS_CHECK_STRUCT_SIZE(struct xfs_dinode, 176); + XFS_CHECK_STRUCT_SIZE(struct xfs_disk_dquot, 104); + XFS_CHECK_STRUCT_SIZE(struct xfs_dqblk, 136); + XFS_CHECK_STRUCT_SIZE(struct xfs_dsymlink_hdr, 56); + XFS_CHECK_STRUCT_SIZE(xfs_alloc_rec_t, 8); + XFS_CHECK_STRUCT_SIZE(xfs_alloc_key_t, 8); + XFS_CHECK_STRUCT_SIZE(xfs_alloc_ptr_t, 4); + XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_rec, 16); + XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_key, 4); + XFS_CHECK_STRUCT_SIZE(xfs_inobt_ptr_t, 4); + XFS_CHECK_STRUCT_SIZE(struct xfs_bmdr_block, 4); + XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_rec, 16); + XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_key, 8); + XFS_CHECK_STRUCT_SIZE(struct xfs_btree_block, 72); + XFS_CHECK_STRUCT_SIZE(struct xfs_acl_entry, 12); + XFS_CHECK_STRUCT_SIZE(struct xfs_acl, 4); + + /* log structures */ + XFS_CHECK_STRUCT_SIZE(struct xfs_trans_header, 16); + XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_32, 52); + XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_64, 56); + XFS_CHECK_STRUCT_SIZE(struct xfs_ictimestamp, 8); + XFS_CHECK_STRUCT_SIZE(struct xfs_icdinode, 176); + XFS_CHECK_STRUCT_SIZE(struct xfs_extent_32, 12); + XFS_CHECK_STRUCT_SIZE(struct xfs_extent_64, 16); + XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_32, 28); + XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64, 32); + XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32, 28); + XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64, 32); + XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat, 24); + XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat, 20); + XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log, 28); +} diff --git a/db/ondisk.h b/db/ondisk.h new file mode 100644 index 0000000..b5784d1 --- /dev/null +++ b/db/ondisk.h @@ -0,0 +1,139 @@ +/* + * Copyright (c) 2016 Oracle. + * All Rights Reserved. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it would be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef __XFS_DB_ONDISK_H +#define __XFS_DB_ONDISK_H + +/* Compile time object size, -1 for unknown */ +#ifndef __compiletime_error +# define __compiletime_error(message) +/* + * Sparse complains of variable sized arrays due to the temporary variable in + * __compiletime_assert. Unfortunately we can't just expand it out to make + * sparse see a constant array size without breaking compiletime_assert on old + * versions of GCC (e.g. 4.2.4), so hide the array from sparse altogether. + */ +# ifndef __CHECKER__ +# define __compiletime_error_fallback(condition) \ + do { ((void)sizeof(char[1 - 2 * condition])); } while (0) +# endif +#endif +#ifndef __compiletime_error_fallback +# define __compiletime_error_fallback(condition) do { } while (0) +#endif + +#define __compiletime_assert(condition, msg, prefix, suffix) \ + do { \ + bool __cond = !(condition); \ + extern void prefix ## suffix(void) __compiletime_error(msg); \ + if (__cond) \ + prefix ## suffix(); \ + __compiletime_error_fallback(__cond); \ + } while (0) + +#define _compiletime_assert(condition, msg, prefix, suffix) \ + __compiletime_assert(condition, msg, prefix, suffix) + +/** + * compiletime_assert - break build and emit msg if condition is false + * @condition: a compile-time constant condition to check + * @msg: a message to emit if condition is false + * + * In tradition of POSIX assert, this macro will break the build if the + * supplied condition is *false*, emitting the supplied error message if the + * compiler has support to do so. + */ +#define compiletime_assert(condition, msg) \ + _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__) + +#ifdef __CHECKER__ +#define BUILD_BUG_ON_NOT_POWER_OF_2(n) (0) +#define BUILD_BUG_ON_ZERO(e) (0) +#define BUILD_BUG_ON_NULL(e) ((void*)0) +#define BUILD_BUG_ON_INVALID(e) (0) +#define BUILD_BUG_ON_MSG(cond, msg) (0) +#define BUILD_BUG_ON(condition) (0) +#define BUILD_BUG() (0) +#else /* __CHECKER__ */ + +/* Force a compilation error if a constant expression is not a power of 2 */ +#define BUILD_BUG_ON_NOT_POWER_OF_2(n) \ + BUILD_BUG_ON((n) == 0 || (((n) & ((n) - 1)) != 0)) + +/* Force a compilation error if condition is true, but also produce a + result (of value 0 and type size_t), so the expression can be used + e.g. in a structure initializer (or where-ever else comma expressions + aren't permitted). */ +#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); })) +#define BUILD_BUG_ON_NULL(e) ((void *)sizeof(struct { int:-!!(e); })) + +/* + * BUILD_BUG_ON_INVALID() permits the compiler to check the validity of the + * expression but avoids the generation of any code, even if that expression + * has side-effects. + */ +#define BUILD_BUG_ON_INVALID(e) ((void)(sizeof((__force long)(e)))) + +/** + * BUILD_BUG_ON_MSG - break compile if a condition is true & emit supplied + * error message. + * @condition: the condition which the compiler should know is false. + * + * See BUILD_BUG_ON for description. + */ +#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg) + +/** + * BUILD_BUG_ON - break compile if a condition is true. + * @condition: the condition which the compiler should know is false. + * + * If you have some code which relies on certain constants being equal, or + * some other compile-time-evaluated condition, you should use BUILD_BUG_ON to + * detect if someone changes it. + * + * The implementation uses gcc's reluctance to create a negative array, but gcc + * (as of 4.4) only emits that error for obvious cases (e.g. not arguments to + * inline functions). Luckily, in 4.3 they added the "error" function + * attribute just for this type of case. Thus, we use a negative sized array + * (should always create an error on gcc versions older than 4.4) and then call + * an undefined function with the error attribute (should always create an + * error on gcc 4.3 and later). If for some reason, neither creates a + * compile-time error, we'll still have a link-time error, which is harder to + * track down. + */ +#ifndef __OPTIMIZE__ +#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)])) +#else +#define BUILD_BUG_ON(condition) \ + BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition) +#endif + +/** + * BUILD_BUG - break compile if used. + * + * If you have some code that you expect the compiler to eliminate at + * build time, you should use BUILD_BUG to detect if it is + * unexpectedly used. + */ +#define BUILD_BUG() BUILD_BUG_ON_MSG(1, "BUILD_BUG failed") + +#endif /* __CHECKER__ */ + +void xfs_check_ondisk_structs(void); + +#endif /* __XFS_DB_ONDISK_H */ diff --git a/libxfs/xfs_format.h b/libxfs/xfs_format.h index a35009a..c4c0c1c 100644 --- a/libxfs/xfs_format.h +++ b/libxfs/xfs_format.h @@ -1505,4 +1505,8 @@ struct xfs_acl { #define SGI_ACL_FILE_SIZE (sizeof(SGI_ACL_FILE)-1) #define SGI_ACL_DEFAULT_SIZE (sizeof(SGI_ACL_DEFAULT)-1) +#define XFS_CHECK_STRUCT_SIZE(structname, size) \ + BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(struct " \ + #structname ") is wrong, expected " #size) + #endif /* __XFS_FORMAT_H__ */ From dave@fromorbit.com Mon Jan 11 19:35:45 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4E0A67F4E for ; Mon, 11 Jan 2016 19:35:45 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 11BE6304053 for ; Mon, 11 Jan 2016 17:35:41 -0800 (PST) X-ASG-Debug-ID: 1452562538-04bdf06f6d84740001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id tmkWxwuaNhE3tVGQ for ; Mon, 11 Jan 2016 17:35:38 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CACQBXV5RWPGu7LXleKAECgw+BP4ZggXmgKQEBAQEBAQaRGotBTQEBAQEBAQcBAQEBQT+EYi8WJRhqAwctiC2fcaA/hg6PBAWHYYcSiCCcUwKOUYIuAQsBOhyBcSo0hkUBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 12:05:37 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIns0-0004XU-W2 for xfs@oss.sgi.com; Tue, 12 Jan 2016 12:35:37 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIns0-0008N8-Uz for xfs@oss.sgi.com; Tue, 12 Jan 2016 12:35:36 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH] xfs: log mount failures don't wait for buffers to be released Date: Tue, 12 Jan 2016 12:35:36 +1100 X-ASG-Orig-Subj: [PATCH] xfs: log mount failures don't wait for buffers to be released Message-Id: <1452562536-32145-1-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452562538 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26048 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner Recently I've been seeing xfs/051 fail on 1k block size filesystems. Trying to trace the events during the test lead to the problem going away, indicating that it was a race condition that lead to this ASSERT failure: XFS: Assertion failed: atomic_read(&pag->pag_ref) == 0, file: fs/xfs/xfs_mount.c, line: 156 ..... [] xfs_free_perag+0x87/0xb0 [] xfs_mountfs+0x4d9/0x900 [] xfs_fs_fill_super+0x3bf/0x4d0 [] mount_bdev+0x180/0x1b0 [] xfs_fs_mount+0x15/0x20 [] mount_fs+0x38/0x170 [] vfs_kern_mount+0x67/0x120 [] do_mount+0x218/0xd60 [] SyS_mount+0x8b/0xd0 When I finally caught it with tracing enabled, I saw that AG 2 had an elevated reference count and a buffer was responsible for it. I tracked down the specific buffer, and found that it was missing the final reference count release that would put it back on the LRU and hence be found by xfs_wait_buftarg() calls in the log mount failure handling. The last four traces for the buffer before the assert were (trimmed for relevance) kworker/0:1-5259 xfs_buf_iodone: hold 2 lock 0 flags ASYNC kworker/0:1-5259 xfs_buf_ioerror: hold 2 lock 0 error -5 mount-7163 xfs_buf_lock_done: hold 2 lock 0 flags ASYNC mount-7163 xfs_buf_unlock: hold 2 lock 1 flags ASYNC This is an async write that is completing, so there's nobody waiting for it directly. Hence we call xfs_buf_relse() once all the processing is complete. That does: static inline void xfs_buf_relse(xfs_buf_t *bp) { xfs_buf_unlock(bp); xfs_buf_rele(bp); } Now, it's clear that mount is waiting on the buffer lock, and that it has been released by xfs_buf_relse() and gained by mount. This is expected, because at this point the mount process is in xfs_buf_delwri_submit() waiting for all the IO it submitted to complete. The mount process, however, is waiting on the lock for the buffer because it is in xfs_buf_delwri_submit(). This waits for IO completion, but it doesn't wait for the buffer reference owned by the IO to go away. The mount process collects all the completions, fails the log recovery, and the higher level code then calls xfs_wait_buftarg() to free all the remaining buffers in the filesystem. The issue is that on unlocking the buffer, the scheduler has decided that the mount process has higher priority than the the kworker thread that is running the IO completion, and so immediately switched contexts to the mount process from the semaphore unlock code, hence preventing the kworker thread from finishing the IO completion and releasing the IO reference to the buffer. Hence by the time that xfs_wait_buftarg() is run, the buffer still has an active reference and so isn't on the LRU list that the function walks to free the remaining buffers. Hence we miss that buffer and continue onwards to tear down the mount structures, at which time we get find a stray reference count on the perag structure. On a non-debug kernel, this will be ignored and the structure torn down and freed. Hence when the kworker thread is then rescheduled and the buffer released and freed, it will access a freed perag structure. The problem here is that when the log mount fails, we still need to quiesce the log to ensure that the IO workqueues have returned to idle before we run xfs_wait_buftarg(). By synchronising the workqueues, we ensure that all IO completions are fully processed, not just to the point where buffers have been unlocked. This ensures we don't end up in the situation above. Signed-off-by: Dave Chinner --- fs/xfs/xfs_buf.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 04d2bdd..08ad701 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1529,6 +1529,16 @@ xfs_wait_buftarg( LIST_HEAD(dispose); int loop = 0; + /* + * We need to flush the buffer workqueue to ensure that all IO + * completion processing is 100% done. Just waiting on buffer locks is + * not sufficient for async IO as the reference count held over IO is + * not released until after the buffer lock is dropped. Hence we need to + * ensure here that all reference counts have been dropped before we + * start walking the LRU list. + */ + drain_workqueue(btp->bt_mount->m_buf_workqueue); + /* loop until there is nothing left on the lru list. */ while (list_lru_count(&btp->bt_lru)) { list_lru_walk(&btp->bt_lru, xfs_buftarg_wait_rele, -- 2.5.0 From philiowen2@gmail.com Mon Jan 11 20:31:02 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM,HTML_MESSAGE,SUBJ_ALL_CAPS,T_DKIM_INVALID autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 245A77F37 for ; Mon, 11 Jan 2016 20:31:02 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id 144F630405F for ; Mon, 11 Jan 2016 18:31:02 -0800 (PST) X-ASG-Debug-ID: 1452565857-04cbb05f7cfffa0001-NocioJ Received: from mail-io0-f194.google.com (mail-io0-f194.google.com [209.85.223.194]) by cuda.sgi.com with ESMTP id qqlRnekFyMDmSzgZ (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 11 Jan 2016 18:30:57 -0800 (PST) X-Barracuda-Envelope-From: philiowen2@gmail.com X-Barracuda-RBL-Trusted-Forwarder: 209.85.223.194 Received: by mail-io0-f194.google.com with SMTP id f81so11784690iof.0 for ; Mon, 11 Jan 2016 18:30:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=+rO4GY+ZGb51qZ8G0AiCBuHT8QGI3rSz56sSrVIZC4Q=; b=WLIA5O6633Q+P1pRtz7cE2JgiLDLHVMSn79KAxk0O49WX5GmQ7Ffb7utJUf0/7YrQl Urz2XYtjRSarVdEv6kTlIxHTMNG17zez0rVGiT5nLhq1HOEyQ5JnutQ+6jHOdEtSacHU 8VTRxjT1wcv6+gVyn9awC2TsuZ4rnTvfA2UPCDzXyCfN2e1Na52LrYMzVo7CYNqYjfMG gDUE0DlhW/l/jyewrdaB8wFzbo6z/k9PYGEpKe6lARA++JUBGqVT6h9rn7/7OIIkMlMM Bu+jIhlzV8yUhPLqEBfCu/KPJDGzo8UjAVUfKoG2tSVj4KioB9x+S4t8TLVYXFuQN0UO sjjg== MIME-Version: 1.0 X-Received: by 10.107.150.65 with SMTP id y62mr71239074iod.153.1452565857517; Mon, 11 Jan 2016 18:30:57 -0800 (PST) Received: by 10.79.106.135 with HTTP; Mon, 11 Jan 2016 18:30:57 -0800 (PST) Date: Tue, 12 Jan 2016 02:30:57 +0000 Message-ID: Subject: GOLD & DIAMOND FOR SALE From: Philip Owen X-ASG-Orig-Subj: GOLD & DIAMOND FOR SALE To: undisclosed-recipients:; Content-Type: multipart/alternative; boundary=001a1140f0aaf34413052919d7f4 X-Barracuda-Connect: mail-io0-f194.google.com[209.85.223.194] X-Barracuda-Start-Time: 1452565857 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 1.64 X-Barracuda-Spam-Status: No, SCORE=1.64 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=ADVANCE_FEE_1, BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE, SUBJ_ALL_CAPS, SUBJ_ALL_CAPS_2 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26049 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 0.00 HTML_MESSAGE BODY: HTML included in message 0.01 SUBJ_ALL_CAPS Subject is all capitals 0.00 ADVANCE_FEE_1 Appears to be advance fee fraud (Nigerian 419) 1.62 SUBJ_ALL_CAPS_2 SUBJ_ALL_CAPS_2 --001a1140f0aaf34413052919d7f4 Content-Type: text/plain; charset=UTF-8 Hello, My name is Mr.Philip Owen, I am contacting you in respect of Gold Bars/Dusts, and rough Diamonds of West African origin that is currently ready for sale. We have the capacity to supply substantial quantities of these products on a long term basis. Description Quantity Quality Purity Price Gold bars 300 kgs 22+ 96% Negotiable I shall send full details of our company and also details of our products location and procedures once you revert back to me showing your full interests for these products. Sincerely, Philip Owen. --001a1140f0aaf34413052919d7f4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable




Hello,

My name is Mr.Philip Owen, I am contacting you in respect of Gold=20 Bars/Dusts, and rough Diamonds of West African origin that is currently=20 ready for sale.

=C2=A0We have the capacity to supply substantial quantities of= these products on a long term basis.

Description=C2=A0=C2=A0=C2=A0 Quantity= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Quality=C2=A0=C2=A0 Purity=C2=A0=C2=A0= =C2=A0 Price

Gold bars=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 300 kgs=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 22+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 96%=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Negotiable

I shall send full details of our company and also details of our products location and procedures once you revert back to me showing your full=20 interests for these products.

Sincerely,
Philip Owen.
=

--001a1140f0aaf34413052919d7f4-- From ebiggers3@gmail.com Tue Jan 12 00:07:26 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.3 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM,T_DKIM_INVALID autolearn=no version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 328677F37 for ; Tue, 12 Jan 2016 00:07:26 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id EBB4C8F8049 for ; Mon, 11 Jan 2016 22:07:22 -0800 (PST) X-ASG-Debug-ID: 1452578837-04bdf06f6e8ac70001-NocioJ Received: from mail-io0-f195.google.com (mail-io0-f195.google.com [209.85.223.195]) by cuda.sgi.com with ESMTP id NlIZ7Lyew70riLuB (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 11 Jan 2016 22:07:17 -0800 (PST) X-Barracuda-Envelope-From: ebiggers3@gmail.com X-Barracuda-RBL-Trusted-Forwarder: 209.85.223.195 Received: by mail-io0-f195.google.com with SMTP id k127so44276678iok.1 for ; Mon, 11 Jan 2016 22:07:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=fDJ4FIw++86jXwpceWIj1olS7STjzKCxGAQ5Mw2Ewbk=; b=LVJOz/Nh76zYxY1098tBNROxat1ThW3XSJLrEBGenVwW2UaU4aopJNlwYrafFtyLCQ 2qZIbfRAZviNldr1M0DJhwu2TguXu1fNmGQtNvhf90onnoNq81mv7/qeL6vqwasmLUwZ blNt/4t5C4pbPbuGuhQTdoNwg6TsKMtsGnIu1wsyKvTckFYZ5fOekRzv36CTkNyn/+k3 26qn5VYA9AYlVytSwdr9HkL7JzQJRrRcW2tRRZSVO4XEYHChlqPQp9gK1SJbksNdO+fr ti0RLVsBOxwU9M4xYRKnk8yXVDISi96JxyGypdWt7q5r6HuHi4uk0VnS2i6ey2OkXGbU gfSw== X-Received: by 10.107.129.162 with SMTP id l34mr90653842ioi.105.1452578837229; Mon, 11 Jan 2016 22:07:17 -0800 (PST) Received: from zzz (c-24-7-245-123.hsd1.mn.comcast.net. [24.7.245.123]) by smtp.gmail.com with ESMTPSA id i73sm2747453ioi.6.2016.01.11.22.07.16 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 11 Jan 2016 22:07:16 -0800 (PST) Date: Tue, 12 Jan 2016 00:07:14 -0600 X-Barracuda-Apparent-Source-IP: 24.7.245.123 X-Barracuda-BBL-IP: 24.7.245.123 From: Eric Biggers To: "Darrick J. Wong" Cc: david@fromorbit.com, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH 8/9] vfs: hoist the btrfs deduplication ioctl to the vfs Message-ID: <20160112060714.GA4980@zzz> X-ASG-Orig-Subj: Re: [PATCH 8/9] vfs: hoist the btrfs deduplication ioctl to the vfs References: <20151219085505.12478.71157.stgit@birch.djwong.org> <20151219085559.12478.33700.stgit@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151219085559.12478.33700.stgit@birch.djwong.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mail-io0-f195.google.com[209.85.223.195] X-Barracuda-Start-Time: 1452578837 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26052 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header -0.00 DKIM_VERIFIED Domain Keys Identified Mail: signature passes verification 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature Some feedback on the VFS portion of the FIDEDUPERANGE ioctl and its man page... (note: I realize the patch is mostly just moving the code that already existed in btrfs, but in the VFS it deserves a more thorough review): At high level, I am confused about what is meant by the "source" and "destination" files. I understand that with more than two files, you effectively have to choose one file to treat specially and dedupe with all the other files (an NxN comparison isn't realistic). But with just two files, a deduplication operation should be completely symmetric, should it not? The end result should be that the data is deduplicated, regardless of the order in which I gave the file descriptors. So why is there some non-symmetric behavior? There are several examples but one is that the VFS is checking !S_ISREG() on the "source" file descriptor but not on the "destination" file descriptor. Another is that different permissions are required on the source versus on the destination. If there are good reasons for the nonsymmetry then this needs to be clearly explained in the man page; otherwise it may not be clear what to use as the "source" and what to use as the "destination". It seems odd to be adding "copy" as a system call but then have "dedupe" and "clone" as ioctls rather than system calls... it seems that they should all be one or the other (at least, if we put aside the fact that the ioctls already exist in btrfs). The range checking in clone_verify_area() appears incomplete. Someone could provide len=UINT64_MAX and all the checks would still pass even though 'pos+len' would overflow. Should the ioctl be interruptible? Right now it always goes through *all* the 'struct file_dedupe_range_info's you passed in --- potentially up to 65535 of them. Why 'info->bytes_deduped += deduped' rather than 'info->bytes_deduped = deduped'? 'bytes_deduped' is per file descriptor, not for the operation as a whole. What permissions do you need on the destination file descriptors? The man page implies they must be open for writing and not appending. The implementation differs: it requires FMODE_WRITE only for non-admin users, and it doesn't check for O_APPEND at all. The man page also says you get EPERM if "dest_fd is immutable" and ETXTBSY if "one of the files is a swap file", which I don't see actually happening in the implementation; it seems those error codes perhaps exist at all for this ioctl but rather be left to open(..., O_WRONLY). If the filesystem doesn't support deduplication, or I pass in a strange file descriptor such as one for a named pipe, do I get EINVAL or EOPNOTSUPP? The man page isn't clear. Under what circumstances will 'bytes_deduped' differ from the count that was passed in? If short counts are allowed, what will be the 'status' be in that case: FILE_DEDUP_RANGE_DIFFERS, FILE_DEDUPE_RANGE_SAME, or something else? Can data be deduped even if only a prefix of the data region matches? The man page doesn't mention FILE_DEDUPE_RANGE_SAME at all, instead calling it 0; it only mentions FILE_DEDUPE_RANGE_DIFFERS. The man page isn't clear about whether the ioctl stops early if an error occurs or always processes all the 'struct file_dedupe_range_info's you pass in. And if it were, hypothetically, to stop early, how is the user meant to know on which file it stopped? The man page says "logical_offset" but in the struct it is called "dest_offset". There are some variables named "same" which don't really make sense now that the ioctl is called FIDEDUPERANGE instead of EXTENT_SAME. Eric From dave@fromorbit.com Tue Jan 12 03:01:31 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 1D4C47F37 for ; Tue, 12 Jan 2016 03:01:31 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id A28ABAC001 for ; Tue, 12 Jan 2016 01:01:30 -0800 (PST) X-ASG-Debug-ID: 1452589285-04cb6c75de89ff0002-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id BQLXFAQFQKANbv5O for ; Tue, 12 Jan 2016 01:01:27 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DFDABTwJRWPGu7LXleKAECgw+BP4ZggXmffAaRGooWgSpNAQEBAQEBBwEBAQFBP4Q1AQUnLzMIGDE5AwcUGYgtwBOGDopEhEAFlxOcVUSOD4IuAQs4H4FxKjSGMQEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:24 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005K4-LE for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008Ad-Kg for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 7/8] xfs: move di_changecount to VFS inode Date: Tue, 12 Jan 2016 20:01:19 +1100 X-ASG-Orig-Subj: [PATCH 7/8] xfs: move di_changecount to VFS inode Message-Id: <1452589280-30167-8-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> References: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589287 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner We can store the di_changecount in the i_version field of the VFS inode and remove another 8 bytes from the xfs_icdinode. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_inode_buf.c | 4 ++-- fs/xfs/libxfs/xfs_inode_buf.h | 1 - fs/xfs/xfs_inode.c | 2 +- fs/xfs/xfs_inode_item.c | 2 +- fs/xfs/xfs_trans_inode.c | 2 +- 5 files changed, 5 insertions(+), 6 deletions(-) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index 0cf5a16..a2b10e3 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -245,7 +245,7 @@ xfs_inode_from_disk( to->di_flags = be16_to_cpu(from->di_flags); if (to->di_version == 3) { - to->di_changecount = be64_to_cpu(from->di_changecount); + inode->i_version = be64_to_cpu(from->di_changecount); to->di_crtime.t_sec = be32_to_cpu(from->di_crtime.t_sec); to->di_crtime.t_nsec = be32_to_cpu(from->di_crtime.t_nsec); to->di_flags2 = be64_to_cpu(from->di_flags2); @@ -294,7 +294,7 @@ xfs_inode_to_disk( to->di_flags = cpu_to_be16(from->di_flags); if (from->di_version == 3) { - to->di_changecount = cpu_to_be64(from->di_changecount); + to->di_changecount = cpu_to_be64(inode->i_version); to->di_crtime.t_sec = cpu_to_be32(from->di_crtime.t_sec); to->di_crtime.t_nsec = cpu_to_be32(from->di_crtime.t_nsec); to->di_flags2 = cpu_to_be64(from->di_flags2); diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h index 68da576..e49c229 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.h +++ b/fs/xfs/libxfs/xfs_inode_buf.h @@ -47,7 +47,6 @@ struct xfs_icdinode { __uint16_t di_dmstate; /* DMIG state info */ __uint16_t di_flags; /* random flags, XFS_DIFLAG_... */ - __uint64_t di_changecount; /* number of attribute changes */ __uint64_t di_flags2; /* more random flags */ xfs_ictimestamp_t di_crtime; /* time created */ diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index c8fa5d6..f0b9ccb 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -832,7 +832,7 @@ xfs_ialloc( ip->i_d.di_flags = 0; if (ip->i_d.di_version == 3) { - ip->i_d.di_changecount = 1; + inode->i_version = 1; ip->i_d.di_flags2 = 0; ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec; ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec; diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index 5cdf5ef..334d657 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -364,7 +364,7 @@ xfs_inode_to_log_dinode( to->di_flags = from->di_flags; if (from->di_version == 3) { - to->di_changecount = from->di_changecount; + to->di_changecount = inode->i_version; to->di_crtime.t_sec = from->di_crtime.t_sec; to->di_crtime.t_nsec = from->di_crtime.t_nsec; to->di_flags2 = from->di_flags2; diff --git a/fs/xfs/xfs_trans_inode.c b/fs/xfs/xfs_trans_inode.c index 3f0d466..11a3af0 100644 --- a/fs/xfs/xfs_trans_inode.c +++ b/fs/xfs/xfs_trans_inode.c @@ -117,7 +117,7 @@ xfs_trans_log_inode( */ if (!(ip->i_itemp->ili_item.li_desc->lid_flags & XFS_LID_DIRTY) && IS_I_VERSION(VFS_I(ip))) { - ip->i_d.di_changecount = ++VFS_I(ip)->i_version; + VFS_I(ip)->i_version++; flags |= XFS_ILOG_CORE; } -- 2.5.0 From dave@fromorbit.com Tue Jan 12 03:01:32 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 8E07C7F37 for ; Tue, 12 Jan 2016 03:01:32 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 20DDFAC001 for ; Tue, 12 Jan 2016 01:01:28 -0800 (PST) X-ASG-Debug-ID: 1452589285-04cb6c75de89ff0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id IkjjDOXjaQnYUB37 for ; Tue, 12 Jan 2016 01:01:26 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2C+DABTwJRWPGu7LXleKAECgw+IH4F5n3wGkRqLQE0BAQEBAQEHAQEBAUE/hRE7gQIDB0uID59PoESGDopqhBoFjT+JVJxVRI4Pgi4BCwE3AxyBcSqGZQEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:24 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005Jx-I6 for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008A6-HT for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [RFC PATCH 0/8] xfs: shrink the struct xfs_icdinode Date: Tue, 12 Jan 2016 20:01:12 +1100 X-ASG-Orig-Subj: [RFC PATCH 0/8] xfs: shrink the struct xfs_icdinode Message-Id: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589285 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_SA210e X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC5_SA210e Custom Rule SA210e Hi folks, This is a patchset I've written over the past couple of afternoons. It smokes badly if you try to run it, so I'm posting this for early feedback, not for testing and bug reports. Despite it's name, the struct xfs_icdinode is an on-disk structure - it gets copied directly to the log, and so we are kind of stuck with keeping it around and all it's duplicate information while we do this. Hence before we can shrink this structure, wehave ot break it's link with the logging code. That's what the first patch does - it introduces a struct xfs_log_dinode, and the log formatting and reocvery code use that for the structure that is placed physically in the log. We marshall the variables from the xfs_icdinode in and out individually so we break the dependence between the on-disk format and the xfs_icdinode. This now enables us to remove all the bits of the xfs_icdinode that are either unused (e.g. padding), only used to support old formats, or a duplicates with information stored in the VFS struct inode. As a result, I've been able to reduce the xfs_icdinode from 176 bytes down to: /* size: 72, cachelines: 2, members: 19 */ /* sum members: 68, holes: 1, sum holes: 4 */ 72 bytes. This means it takes 104 bytes off the size of the struct xfs_inode, which a 12% reduction in size. This will be a massive win for systems that cache lots of inodes! I have not yet addressed the di_uid or the di_gid because I haven't had time to look into the user namespace conversion issues that I'll have to handle correctly to use the VFS inode fieldsr. I think I can also pull up the di_nblocks field to the VFS inode as well. If I manage this, I will reduce the xfs_icdinode by another 16 bytes. With this change made, the xfs_icdinode is no long an "in core disk inode" so I'm wondering whether I should rename it or simply make it go away altogether and pull the remaining fields straight into the struct xfs_inode. Any thoughts on new names and/or getting rid of it woul dbe appreciated. Some of the code is pretty raw - I haven't cleaned it up at all, so do be surprised if there's things I've missed or changed in the wrong patch, etc. Comments, thoughts, flames, all welcome! -Dave. From dave@fromorbit.com Tue Jan 12 03:01:33 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 289677F3F for ; Tue, 12 Jan 2016 03:01:33 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id A23CBAC005 for ; Tue, 12 Jan 2016 01:01:32 -0800 (PST) X-ASG-Debug-ID: 1452589290-04cb6c75dd8a000001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id aMNd2KA4vTTHUWYI for ; Tue, 12 Jan 2016 01:01:30 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DFDABTwJRWPGu7LXleKAECgw+BP4ZggXmffAaRGooWgSpNAQEBAQEBBwEBAQFBP4Q1AQUnLzMIGDE5AwcUGYgtwBOGDopEhEAFlxOcVY5Tgi4BCzgfgXEqNIYxAQEB Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:24 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005K3-Kv for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008AY-KL for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 6/8] xfs: move inode generation count to VFS inode Date: Tue, 12 Jan 2016 20:01:18 +1100 X-ASG-Orig-Subj: [PATCH 6/8] xfs: move inode generation count to VFS inode Message-Id: <1452589280-30167-7-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> References: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589290 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner Pull another 4 bytes out of the xfs_icdinode. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_inode_buf.c | 8 ++++---- fs/xfs/libxfs/xfs_inode_buf.h | 1 - fs/xfs/xfs_export.c | 2 +- fs/xfs/xfs_inode.c | 5 +---- fs/xfs/xfs_inode_item.c | 2 +- fs/xfs/xfs_ioctl.c | 2 +- fs/xfs/xfs_iops.c | 1 - fs/xfs/xfs_itable.c | 2 +- 8 files changed, 9 insertions(+), 14 deletions(-) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index 6c00277..0cf5a16 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -231,6 +231,7 @@ xfs_inode_from_disk( inode->i_mtime.tv_nsec = be32_to_cpu(from->di_mtime.t_nsec); inode->i_ctime.tv_sec = be32_to_cpu(from->di_ctime.t_sec); inode->i_ctime.tv_nsec = be32_to_cpu(from->di_ctime.t_nsec); + inode->i_generation = be32_to_cpu(from->di_gen); to->di_size = be64_to_cpu(from->di_size); to->di_nblocks = be64_to_cpu(from->di_nblocks); @@ -242,7 +243,6 @@ xfs_inode_from_disk( to->di_dmevmask = be32_to_cpu(from->di_dmevmask); to->di_dmstate = be16_to_cpu(from->di_dmstate); to->di_flags = be16_to_cpu(from->di_flags); - to->di_gen = be32_to_cpu(from->di_gen); if (to->di_version == 3) { to->di_changecount = be64_to_cpu(from->di_changecount); @@ -280,6 +280,7 @@ xfs_inode_to_disk( to->di_ctime.t_sec = cpu_to_be32(inode->i_ctime.tv_sec); to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec); to->di_nlink = cpu_to_be32(inode->i_nlink); + to->di_gen = cpu_to_be32(inode->i_generation); to->di_size = cpu_to_be64(from->di_size); to->di_nblocks = cpu_to_be64(from->di_nblocks); @@ -291,7 +292,6 @@ xfs_inode_to_disk( to->di_dmevmask = cpu_to_be32(from->di_dmevmask); to->di_dmstate = cpu_to_be16(from->di_dmstate); to->di_flags = cpu_to_be16(from->di_flags); - to->di_gen = cpu_to_be32(from->di_gen); if (from->di_version == 3) { to->di_changecount = cpu_to_be64(from->di_changecount); @@ -437,7 +437,7 @@ xfs_iread( !(mp->m_flags & XFS_MOUNT_IKEEP)) { /* initialise the on-disk inode core */ memset(&ip->i_d, 0, sizeof(ip->i_d)); - ip->i_d.di_gen = prandom_u32(); + VFS_I(ip)->i_generation = prandom_u32(); if (xfs_sb_version_hascrc(&mp->m_sb)) ip->i_d.di_version = 3; else @@ -485,7 +485,7 @@ xfs_iread( * that xfs_ialloc won't overwrite or relies on being correct. */ ip->i_d.di_version = dip->di_version; - ip->i_d.di_gen = be32_to_cpu(dip->di_gen); + VFS_I(ip)->i_generation = be32_to_cpu(dip->di_gen); ip->i_d.di_flushiter = be16_to_cpu(dip->di_flushiter); /* diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h index 774d71f..68da576 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.h +++ b/fs/xfs/libxfs/xfs_inode_buf.h @@ -46,7 +46,6 @@ struct xfs_icdinode { __uint32_t di_dmevmask; /* DMIG event mask */ __uint16_t di_dmstate; /* DMIG state info */ __uint16_t di_flags; /* random flags, XFS_DIFLAG_... */ - __uint32_t di_gen; /* generation number */ __uint64_t di_changecount; /* number of attribute changes */ __uint64_t di_flags2; /* more random flags */ diff --git a/fs/xfs/xfs_export.c b/fs/xfs/xfs_export.c index 652cd3c..2816d42 100644 --- a/fs/xfs/xfs_export.c +++ b/fs/xfs/xfs_export.c @@ -152,7 +152,7 @@ xfs_nfs_get_inode( return ERR_PTR(error); } - if (ip->i_d.di_gen != generation) { + if (VFS_I(ip)->i_generation != generation) { IRELE(ip); return ERR_PTR(-ESTALE); } diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index d4f328a..c8fa5d6 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -826,9 +826,6 @@ xfs_ialloc( inode->i_atime = tv; inode->i_ctime = tv; - /* - * di_gen will have been taken care of in xfs_iread. - */ ip->i_d.di_extsize = 0; ip->i_d.di_dmevmask = 0; ip->i_d.di_dmstate = 0; @@ -2417,7 +2414,7 @@ xfs_ifree( * Bump the generation count so no one will be confused * by reincarnations of this inode. */ - ip->i_d.di_gen++; + VFS_I(ip)->i_generation++; xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE); if (xic.deleted) diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index 4e3611d..5cdf5ef 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -350,6 +350,7 @@ xfs_inode_to_log_dinode( to->di_ctime.t_sec = inode->i_ctime.tv_sec; to->di_ctime.t_nsec = inode->i_ctime.tv_nsec; to->di_nlink = inode->i_nlink; + to->di_gen = inode->i_generation; to->di_size = from->di_size; to->di_nblocks = from->di_nblocks; @@ -361,7 +362,6 @@ xfs_inode_to_log_dinode( to->di_dmevmask = from->di_dmevmask; to->di_dmstate = from->di_dmstate; to->di_flags = from->di_flags; - to->di_gen = from->di_gen; if (from->di_version == 3) { to->di_changecount = from->di_changecount; diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index d42738d..2c40c5f 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -114,7 +114,7 @@ xfs_find_handle( handle.ha_fid.fid_len = sizeof(xfs_fid_t) - sizeof(handle.ha_fid.fid_len); handle.ha_fid.fid_pad = 0; - handle.ha_fid.fid_gen = ip->i_d.di_gen; + handle.ha_fid.fid_gen = inode->i_generation; handle.ha_fid.fid_ino = ip->i_ino; hsize = XFS_HSIZE(handle); diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c index c424d4b..397ce85 100644 --- a/fs/xfs/xfs_iops.c +++ b/fs/xfs/xfs_iops.c @@ -1227,7 +1227,6 @@ xfs_setup_inode( break; } - inode->i_generation = ip->i_d.di_gen; i_size_write(inode, ip->i_d.di_size); xfs_diflags_to_iflags(inode, ip); diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c index cfb6527..6162e65 100644 --- a/fs/xfs/xfs_itable.c +++ b/fs/xfs/xfs_itable.c @@ -100,11 +100,11 @@ xfs_bulkstat_one_int( buf->bs_mtime.tv_nsec = inode->i_mtime.tv_nsec; buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec; buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec; + buf->bs_gen = inode->i_generation; buf->bs_xflags = xfs_ip2xflags(ip); buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog; buf->bs_extents = dic->di_nextents; - buf->bs_gen = dic->di_gen; memset(buf->bs_pad, 0, sizeof(buf->bs_pad)); buf->bs_dmevmask = dic->di_dmevmask; buf->bs_dmstate = dic->di_dmstate; -- 2.5.0 From dave@fromorbit.com Tue Jan 12 03:01:34 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 6AFC07F51 for ; Tue, 12 Jan 2016 03:01:34 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id 5BABE8F8035 for ; Tue, 12 Jan 2016 01:01:31 -0800 (PST) X-ASG-Debug-ID: 1452589288-04cbb05f7c11dbe0001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 60kcdfdtsytVim9G for ; Tue, 12 Jan 2016 01:01:28 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DIDABTwJRWPGu7LXleKAECgw+BP4ZggXmffAaBZo80ihaBKk0BAQEBAQEHAQEBAUE/hDUBBScvMwgYGRg5AwcUGYgtwBOGDopEIgyEEgWHZI8vnFWOU4IuAQsBNwMcgXEqNIRoBxcHgSQBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:24 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005K2-KX for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008AT-Jx for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 5/8] xfs: use vfs inode nlink field everywhere Date: Tue, 12 Jan 2016 20:01:17 +1100 X-ASG-Orig-Subj: [PATCH 5/8] xfs: use vfs inode nlink field everywhere Message-Id: <1452589280-30167-6-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> References: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589288 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner The Vfs tracks the inodenlink just like the xfs_icdinode. We can remove the variable from the icdinode and use the vfs inode variable everywhere, reducing the size of the xfs_icdinode by a further 4 bytes. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_inode_buf.c | 6 +++--- fs/xfs/libxfs/xfs_inode_buf.h | 1 - fs/xfs/xfs_inode.c | 30 ++++++++++++++---------------- fs/xfs/xfs_inode_item.c | 2 +- fs/xfs/xfs_iops.c | 3 +-- fs/xfs/xfs_itable.c | 2 +- fs/xfs/xfs_log_recover.c | 2 +- 7 files changed, 21 insertions(+), 25 deletions(-) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index a3e79a9..6c00277 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -210,12 +210,12 @@ xfs_inode_from_disk( * minimum inode version format we support in the rest of the code. */ if (to->di_version == 1) { - to->di_nlink = be16_to_cpu(from->di_onlink); + set_nlink(inode, be16_to_cpu(from->di_onlink)); to->di_projid_lo = 0; to->di_projid_hi = 0; to->di_version = 2; } else { - to->di_nlink = be32_to_cpu(from->di_nlink); + set_nlink(inode, be32_to_cpu(from->di_nlink)); to->di_projid_lo = be16_to_cpu(from->di_projid_lo); to->di_projid_hi = be16_to_cpu(from->di_projid_hi); } @@ -269,7 +269,6 @@ xfs_inode_to_disk( to->di_format = from->di_format; to->di_uid = cpu_to_be32(from->di_uid); to->di_gid = cpu_to_be32(from->di_gid); - to->di_nlink = cpu_to_be32(from->di_nlink); to->di_projid_lo = cpu_to_be16(from->di_projid_lo); to->di_projid_hi = cpu_to_be16(from->di_projid_hi); @@ -280,6 +279,7 @@ xfs_inode_to_disk( to->di_mtime.t_nsec = cpu_to_be32(inode->i_mtime.tv_nsec); to->di_ctime.t_sec = cpu_to_be32(inode->i_ctime.tv_sec); to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec); + to->di_nlink = cpu_to_be32(inode->i_nlink); to->di_size = cpu_to_be64(from->di_size); to->di_nblocks = cpu_to_be64(from->di_nblocks); diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h index feb04e6..774d71f 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.h +++ b/fs/xfs/libxfs/xfs_inode_buf.h @@ -34,7 +34,6 @@ struct xfs_icdinode { __uint16_t di_flushiter; /* incremented on flush */ __uint32_t di_uid; /* owner's user id */ __uint32_t di_gid; /* owner's group id */ - __uint32_t di_nlink; /* number of links to file */ __uint16_t di_projid_lo; /* lower part of owner's project id */ __uint16_t di_projid_hi; /* higher part of owner's project id */ xfs_fsize_t di_size; /* number of bytes in file */ diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index 914ec41..d4f328a 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -794,7 +794,7 @@ xfs_ialloc( ip->i_d.di_version = 2; ip->i_d.di_mode = mode; - ip->i_d.di_nlink = nlink; + set_nlink(inode, nlink); ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid()); ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid()); xfs_set_projid(ip, prid); @@ -1085,13 +1085,11 @@ xfs_droplink( xfs_trans_ichgtime(tp, ip, XFS_ICHGTIME_CHG); - ASSERT (ip->i_d.di_nlink > 0); - ip->i_d.di_nlink--; drop_nlink(VFS_I(ip)); xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE); error = 0; - if (ip->i_d.di_nlink == 0) { + if (VFS_I(ip)->i_nlink == 0) { /* * We're dropping the last link to this file. * Move the on-disk inode to the AGI unlinked list. @@ -1114,8 +1112,6 @@ xfs_bumplink( xfs_trans_ichgtime(tp, ip, XFS_ICHGTIME_CHG); ASSERT(ip->i_d.di_version > 1); - ASSERT(ip->i_d.di_nlink > 0 || (VFS_I(ip)->i_state & I_LINKABLE)); - ip->i_d.di_nlink++; inc_nlink(VFS_I(ip)); xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE); return 0; @@ -1373,7 +1369,6 @@ xfs_create_tmpfile( */ xfs_qm_vop_create_dqattach(tp, ip, udqp, gdqp, pdqp); - ip->i_d.di_nlink--; error = xfs_iunlink(tp, ip); if (error) goto out_trans_cancel; @@ -1472,7 +1467,10 @@ xfs_link( xfs_bmap_init(&free_list, &first_block); - if (sip->i_d.di_nlink == 0) { + /* + * Handle initial link state of O_TMPFILE inode + */ + if (VFS_I(sip)->i_nlink == 0) { error = xfs_iunlink_remove(tp, sip); if (error) goto error_return; @@ -1659,7 +1657,7 @@ xfs_release( } } - if (ip->i_d.di_nlink == 0) + if (VFS_I(ip)->i_nlink == 0) return 0; if (xfs_can_free_eofblocks(ip, false)) { @@ -1875,7 +1873,7 @@ xfs_inactive( if (mp->m_flags & XFS_MOUNT_RDONLY) return; - if (ip->i_d.di_nlink != 0) { + if (VFS_I(ip)->i_nlink != 0) { /* * force is true because we are evicting an inode from the * cache. Post-eof blocks must be freed, lest we end up with @@ -1951,7 +1949,7 @@ xfs_iunlink( int offset; int error; - ASSERT(ip->i_d.di_nlink == 0); + ASSERT(VFS_I(ip)->i_nlink == 0); ASSERT(ip->i_d.di_mode != 0); mp = tp->t_mountp; @@ -2392,7 +2390,7 @@ xfs_ifree( struct xfs_icluster xic = { 0 }; ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL)); - ASSERT(ip->i_d.di_nlink == 0); + ASSERT(VFS_I(ip)->i_nlink == 0); ASSERT(ip->i_d.di_nextents == 0); ASSERT(ip->i_d.di_anextents == 0); ASSERT(ip->i_d.di_size == 0 || !S_ISREG(ip->i_d.di_mode)); @@ -2560,8 +2558,8 @@ xfs_remove( * If we're removing a directory perform some additional validation. */ if (is_dir) { - ASSERT(ip->i_d.di_nlink >= 2); - if (ip->i_d.di_nlink != 2) { + ASSERT(VFS_I(ip)->i_nlink >= 2); + if (VFS_I(ip)->i_nlink != 2) { error = -ENOTEMPTY; goto out_trans_cancel; } @@ -3017,7 +3015,7 @@ xfs_rename( * Make sure target dir is empty. */ if (!(xfs_dir_isempty(target_ip)) || - (target_ip->i_d.di_nlink > 2)) { + (VFS_I(target_ip)->i_nlink > 2)) { error = -EEXIST; goto out_trans_cancel; } @@ -3124,7 +3122,7 @@ xfs_rename( * intermediate state on disk. */ if (wip) { - ASSERT(VFS_I(wip)->i_nlink == 0 && wip->i_d.di_nlink == 0); + ASSERT(VFS_I(wip)->i_nlink == 0); error = xfs_bumplink(tp, wip); if (error) goto out_bmap_cancel; diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index d777b7a..4e3611d 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -338,7 +338,6 @@ xfs_inode_to_log_dinode( to->di_format = from->di_format; to->di_uid = from->di_uid; to->di_gid = from->di_gid; - to->di_nlink = from->di_nlink; to->di_projid_lo = from->di_projid_lo; to->di_projid_hi = from->di_projid_hi; @@ -350,6 +349,7 @@ xfs_inode_to_log_dinode( to->di_mtime.t_nsec = inode->i_mtime.tv_nsec; to->di_ctime.t_sec = inode->i_ctime.tv_sec; to->di_ctime.t_nsec = inode->i_ctime.tv_nsec; + to->di_nlink = inode->i_nlink; to->di_size = from->di_size; to->di_nblocks = from->di_nblocks; diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c index b008677..c424d4b 100644 --- a/fs/xfs/xfs_iops.c +++ b/fs/xfs/xfs_iops.c @@ -455,7 +455,7 @@ xfs_vn_getattr( stat->size = XFS_ISIZE(ip); stat->dev = inode->i_sb->s_dev; stat->mode = ip->i_d.di_mode; - stat->nlink = ip->i_d.di_nlink; + stat->nlink = inode->i_nlink; stat->uid = inode->i_uid; stat->gid = inode->i_gid; stat->ino = ip->i_ino; @@ -1212,7 +1212,6 @@ xfs_setup_inode( hlist_add_fake(&inode->i_hash); inode->i_mode = ip->i_d.di_mode; - set_nlink(inode, ip->i_d.di_nlink); inode->i_uid = xfs_uid_to_kuid(ip->i_d.di_uid); inode->i_gid = xfs_gid_to_kgid(ip->i_d.di_gid); diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c index 2acda42..cfb6527 100644 --- a/fs/xfs/xfs_itable.c +++ b/fs/xfs/xfs_itable.c @@ -85,7 +85,6 @@ xfs_bulkstat_one_int( /* xfs_iget returns the following without needing * further change. */ - buf->bs_nlink = dic->di_nlink; buf->bs_projid_lo = dic->di_projid_lo; buf->bs_projid_hi = dic->di_projid_hi; buf->bs_ino = ino; @@ -94,6 +93,7 @@ xfs_bulkstat_one_int( buf->bs_gid = dic->di_gid; buf->bs_size = dic->di_size; + buf->bs_nlink = inode->i_nlink; buf->bs_atime.tv_sec = inode->i_atime.tv_sec; buf->bs_atime.tv_nsec = inode->i_atime.tv_nsec; buf->bs_mtime.tv_sec = inode->i_mtime.tv_sec; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 4b79cf0..611c25c 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -4337,7 +4337,7 @@ xlog_recover_process_one_iunlink( if (error) goto fail_iput; - ASSERT(ip->i_d.di_nlink == 0); + ASSERT(VFS_I(ip)->i_nlink == 0); ASSERT(ip->i_d.di_mode != 0); /* setup for the next pass */ -- 2.5.0 From dave@fromorbit.com Tue Jan 12 03:01:35 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 611477F55 for ; Tue, 12 Jan 2016 03:01:35 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 4A858304053 for ; Tue, 12 Jan 2016 01:01:32 -0800 (PST) X-ASG-Debug-ID: 1452589285-04cb6c75de89ff0003-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id Cz8ktz2Y0jPlCrTo for ; Tue, 12 Jan 2016 01:01:29 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DFDABTwJRWPGu7LXleKAECgw+BP4ZggXmffAaRGooWgSpNAQEBAQEBBwEBAQFBP4Q1AQUnLzMIGBkYOQMHFBmILcAThg6KRCIMhBIFh2SPL5xVRI4Pgi4BCzscgXEqNIRvgUIBAQE Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:24 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005K1-K6 for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008AO-JG for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 4/8] xfs: move v1 inode conversion to xfs_inode_from_disk Date: Tue, 12 Jan 2016 20:01:16 +1100 X-ASG-Orig-Subj: [PATCH 4/8] xfs: move v1 inode conversion to xfs_inode_from_disk Message-Id: <1452589280-30167-5-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> References: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589289 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner So we don't have to carry an di_onlink variable around anymore, move the inode conversion from v1 inode format to v2 inode format into xfs_inode_from_disk(). This means we can remove the di_onlink fields from the struct xfs_icdinode. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_inode_buf.c | 42 ++++++++++++++++++++---------------------- fs/xfs/libxfs/xfs_inode_buf.h | 1 - fs/xfs/libxfs/xfs_log_format.h | 2 +- fs/xfs/xfs_inode.c | 2 -- fs/xfs/xfs_inode_item.c | 2 +- 5 files changed, 22 insertions(+), 27 deletions(-) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index 920aeef..a3e79a9 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -204,13 +204,25 @@ xfs_inode_from_disk( to->di_mode = be16_to_cpu(from->di_mode); to->di_version = from ->di_version; + + /* + * Convert v1 inodes immediately to v2 inode format as this is the + * minimum inode version format we support in the rest of the code. + */ + if (to->di_version == 1) { + to->di_nlink = be16_to_cpu(from->di_onlink); + to->di_projid_lo = 0; + to->di_projid_hi = 0; + to->di_version = 2; + } else { + to->di_nlink = be32_to_cpu(from->di_nlink); + to->di_projid_lo = be16_to_cpu(from->di_projid_lo); + to->di_projid_hi = be16_to_cpu(from->di_projid_hi); + } + to->di_format = from->di_format; - to->di_onlink = be16_to_cpu(from->di_onlink); to->di_uid = be32_to_cpu(from->di_uid); to->di_gid = be32_to_cpu(from->di_gid); - to->di_nlink = be32_to_cpu(from->di_nlink); - to->di_projid_lo = be16_to_cpu(from->di_projid_lo); - to->di_projid_hi = be16_to_cpu(from->di_projid_hi); to->di_flushiter = be16_to_cpu(from->di_flushiter); inode->i_atime.tv_sec = be32_to_cpu(from->di_atime.t_sec); @@ -250,11 +262,11 @@ xfs_inode_to_disk( struct inode *inode = VFS_I(ip); to->di_magic = cpu_to_be16(XFS_DINODE_MAGIC); + to->di_onlink = 0; to->di_mode = cpu_to_be16(from->di_mode); to->di_version = from->di_version; to->di_format = from->di_format; - to->di_onlink = cpu_to_be16(from->di_onlink); to->di_uid = cpu_to_be32(from->di_uid); to->di_gid = cpu_to_be32(from->di_gid); to->di_nlink = cpu_to_be32(from->di_nlink); @@ -304,9 +316,9 @@ xfs_log_dinode_to_disk( { to->di_magic = cpu_to_be16(from->di_magic); to->di_mode = cpu_to_be16(from->di_mode); - to->di_version = from ->di_version; + to->di_version = from->di_version; to->di_format = from->di_format; - to->di_onlink = cpu_to_be16(from->di_onlink); + to->di_onlink = 0; to->di_uid = cpu_to_be32(from->di_uid); to->di_gid = cpu_to_be32(from->di_gid); to->di_nlink = cpu_to_be32(from->di_nlink); @@ -486,21 +498,7 @@ xfs_iread( ip->i_d.di_mode = 0; } - /* - * Automatically convert version 1 inode formats in memory to version 2 - * inode format. If the inode is modified, it will get logged and - * rewritten as a version 2 inode. We can do this because we set the - * superblock feature bit for v2 inodes unconditionally during mount - * and it means the reast of the code can assume the inode version is 2 - * or higher. - */ - if (ip->i_d.di_version == 1) { - ip->i_d.di_version = 2; - ip->i_d.di_nlink = ip->i_d.di_onlink; - ip->i_d.di_onlink = 0; - xfs_set_projid(ip, 0); - } - + ASSERT(ip->i_d.di_version >= 2); ip->i_delayed_blks = 0; /* diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h index 69d626e..feb04e6 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.h +++ b/fs/xfs/libxfs/xfs_inode_buf.h @@ -31,7 +31,6 @@ struct xfs_icdinode { __uint16_t di_mode; /* mode and type of file */ __int8_t di_version; /* inode version */ __int8_t di_format; /* format of di_c data */ - __uint16_t di_onlink; /* old number of links to file */ __uint16_t di_flushiter; /* incremented on flush */ __uint32_t di_uid; /* owner's user id */ __uint32_t di_gid; /* owner's group id */ diff --git a/fs/xfs/libxfs/xfs_log_format.h b/fs/xfs/libxfs/xfs_log_format.h index d00ed63..03f90b9 100644 --- a/fs/xfs/libxfs/xfs_log_format.h +++ b/fs/xfs/libxfs/xfs_log_format.h @@ -369,7 +369,7 @@ struct xfs_log_dinode { __uint16_t di_mode; /* mode and type of file */ __int8_t di_version; /* inode version */ __int8_t di_format; /* format of di_c data */ - __uint16_t di_onlink; /* old number of links to file */ + __uint8_t di_pad3[2]; /* unused in v2/3 inodes */ __uint32_t di_uid; /* owner's user id */ __uint32_t di_gid; /* owner's group id */ __uint32_t di_nlink; /* number of links to file */ diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index 3f020b5..914ec41 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -794,9 +794,7 @@ xfs_ialloc( ip->i_d.di_version = 2; ip->i_d.di_mode = mode; - ip->i_d.di_onlink = 0; ip->i_d.di_nlink = nlink; - ASSERT(ip->i_d.di_nlink == nlink); ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid()); ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid()); xfs_set_projid(ip, prid); diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index ae60087..d777b7a 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -336,7 +336,6 @@ xfs_inode_to_log_dinode( to->di_mode = from->di_mode; to->di_version = from->di_version; to->di_format = from->di_format; - to->di_onlink = from->di_onlink; to->di_uid = from->di_uid; to->di_gid = from->di_gid; to->di_nlink = from->di_nlink; @@ -344,6 +343,7 @@ xfs_inode_to_log_dinode( to->di_projid_hi = from->di_projid_hi; memset(to->di_pad, 0, sizeof(to->di_pad)); + memset(to->di_pad3, 0, sizeof(to->di_pad3)); to->di_atime.t_sec = inode->i_atime.tv_sec; to->di_atime.t_nsec = inode->i_atime.tv_nsec; to->di_mtime.t_sec = inode->i_mtime.tv_sec; -- 2.5.0 From dave@fromorbit.com Tue Jan 12 03:01:35 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 6922E7F58 for ; Tue, 12 Jan 2016 03:01:35 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id D9116AC001 for ; Tue, 12 Jan 2016 01:01:34 -0800 (PST) X-ASG-Debug-ID: 1452589288-04cbb05f7c11dbe0002-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id kMDwI9JNk1OdHuaf for ; Tue, 12 Jan 2016 01:01:30 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DCDABTwJRWPGu7LXleKAECgw+BP4ZggXmffAaRGotATQEBAQEBAQcBAQEBQT+ENQEFJy8zCBgZGDkDBxQZiC3AE4YOimYMhBIFlxOcVY5Tgi4BCwE3AxyBcSo0hG+BQgEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:24 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005Jy-IY for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008A9-Hx for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 1/8] xfs: introduce inode log format object Date: Tue, 12 Jan 2016 20:01:13 +1100 X-ASG-Orig-Subj: [PATCH 1/8] xfs: introduce inode log format object Message-Id: <1452589280-30167-2-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> References: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589290 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner We currently carry around and log an entire inode core in the struct xfs_inode. A lot of the information in the inode core is duplicated in the VFS inode, but we cannot remove this duplication of infomration because the inode core is logged directly in xfs_inode_item_format(). Add a new function xfs_inode_item_format_core() that copies the inode core data into a struct xfs_icdinode that is pulled directly from the log vector buffer. This means we no longer directly copy the inode core, but copy the structures one member at a time. This will be slightly less efficient than copying, but will allow us to remove duplicate and unnecessary items from the struct xfs_inode. To enable us to do this, call the new structure a xfs_log_dinode, so that we know it's different to the physical xfs_dinode and the in-core xfs_icdinode. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_inode_buf.c | 8 +-- fs/xfs/libxfs/xfs_inode_buf.h | 53 ++++++++++++++++- fs/xfs/libxfs/xfs_log_format.h | 15 ++--- fs/xfs/xfs_icache.c | 2 +- fs/xfs/xfs_inode.c | 6 +- fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 128 +++++++++++++++++++++++++++++++++++++++-- fs/xfs/xfs_inode_item.h | 2 + fs/xfs/xfs_log_recover.c | 52 +++++++++-------- 9 files changed, 221 insertions(+), 47 deletions(-) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index 1aabfda..63d46bf 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -196,8 +196,8 @@ xfs_imap_to_bp( void xfs_dinode_from_disk( - xfs_icdinode_t *to, - xfs_dinode_t *from) + struct xfs_icdinode *to, + struct xfs_dinode *from) { to->di_magic = be16_to_cpu(from->di_magic); to->di_mode = be16_to_cpu(from->di_mode); @@ -243,8 +243,8 @@ xfs_dinode_from_disk( void xfs_dinode_to_disk( - xfs_dinode_t *to, - xfs_icdinode_t *from) + struct xfs_dinode *to, + struct xfs_icdinode *from) { to->di_magic = cpu_to_be16(from->di_magic); to->di_mode = cpu_to_be16(from->di_mode); diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h index 9308c47..642f2a2 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.h +++ b/fs/xfs/libxfs/xfs_inode_buf.h @@ -20,7 +20,58 @@ struct xfs_inode; struct xfs_dinode; -struct xfs_icdinode; + +/* + * In memory representation of the XFS inode. This is held in the in-core + * struct xfs_inode to represent the on disk values, but no longer needs to be + * identical to the on-disk structure as it is always translated to on-disk + * format specific structures at the appropriate time. + */ +struct xfs_icdinode { + __uint16_t di_magic; /* inode magic # = XFS_DINODE_MAGIC */ + __uint16_t di_mode; /* mode and type of file */ + __int8_t di_version; /* inode version */ + __int8_t di_format; /* format of di_c data */ + __uint16_t di_onlink; /* old number of links to file */ + __uint32_t di_uid; /* owner's user id */ + __uint32_t di_gid; /* owner's group id */ + __uint32_t di_nlink; /* number of links to file */ + __uint16_t di_projid_lo; /* lower part of owner's project id */ + __uint16_t di_projid_hi; /* higher part of owner's project id */ + __uint8_t di_pad[6]; /* unused, zeroed space */ + __uint16_t di_flushiter; /* incremented on flush */ + xfs_ictimestamp_t di_atime; /* time last accessed */ + xfs_ictimestamp_t di_mtime; /* time last modified */ + xfs_ictimestamp_t di_ctime; /* time created/inode modified */ + xfs_fsize_t di_size; /* number of bytes in file */ + xfs_rfsblock_t di_nblocks; /* # of direct & btree blocks used */ + xfs_extlen_t di_extsize; /* basic/minimum extent size for file */ + xfs_extnum_t di_nextents; /* number of extents in data fork */ + xfs_aextnum_t di_anextents; /* number of extents in attribute fork*/ + __uint8_t di_forkoff; /* attr fork offs, <<3 for 64b align */ + __int8_t di_aformat; /* format of attr fork's data */ + __uint32_t di_dmevmask; /* DMIG event mask */ + __uint16_t di_dmstate; /* DMIG state info */ + __uint16_t di_flags; /* random flags, XFS_DIFLAG_... */ + __uint32_t di_gen; /* generation number */ + + /* di_next_unlinked is the only non-core field in the old dinode */ + xfs_agino_t di_next_unlinked;/* agi unlinked list ptr */ + + /* start of the extended dinode, writable fields */ + __uint32_t di_crc; /* CRC of the inode */ + __uint64_t di_changecount; /* number of attribute changes */ + xfs_lsn_t di_lsn; /* flush sequence */ + __uint64_t di_flags2; /* more random flags */ + __uint8_t di_pad2[16]; /* more padding for future expansion */ + + /* fields only written to during inode creation */ + xfs_ictimestamp_t di_crtime; /* time created */ + xfs_ino_t di_ino; /* inode number */ + uuid_t di_uuid; /* UUID of the filesystem */ + + /* structure must be padded to 64 bit alignment */ +}; /* * Inode location information. Stored in the inode and passed to diff --git a/fs/xfs/libxfs/xfs_log_format.h b/fs/xfs/libxfs/xfs_log_format.h index 2653146..d00ed63 100644 --- a/fs/xfs/libxfs/xfs_log_format.h +++ b/fs/xfs/libxfs/xfs_log_format.h @@ -290,6 +290,7 @@ typedef struct xfs_inode_log_format_64 { __int32_t ilf_boffset; /* off of inode in buffer */ } xfs_inode_log_format_64_t; + /* * Flags for xfs_trans_log_inode flags field. */ @@ -360,10 +361,10 @@ typedef struct xfs_ictimestamp { } xfs_ictimestamp_t; /* - * NOTE: This structure must be kept identical to struct xfs_dinode - * except for the endianness annotations. + * Define the format of the inode core that is logged. This structure must be + * kept identical to struct xfs_dinode except for the endianness annotations. */ -typedef struct xfs_icdinode { +struct xfs_log_dinode { __uint16_t di_magic; /* inode magic # = XFS_DINODE_MAGIC */ __uint16_t di_mode; /* mode and type of file */ __int8_t di_version; /* inode version */ @@ -407,13 +408,13 @@ typedef struct xfs_icdinode { uuid_t di_uuid; /* UUID of the filesystem */ /* structure must be padded to 64 bit alignment */ -} xfs_icdinode_t; +}; -static inline uint xfs_icdinode_size(int version) +static inline uint xfs_log_dinode_size(int version) { if (version == 3) - return sizeof(struct xfs_icdinode); - return offsetof(struct xfs_icdinode, di_next_unlinked); + return sizeof(struct xfs_log_dinode); + return offsetof(struct xfs_log_dinode, di_next_unlinked); } /* diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index d7a490f..7c26f86 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -79,7 +79,7 @@ xfs_inode_alloc( memset(&ip->i_df, 0, sizeof(xfs_ifork_t)); ip->i_flags = 0; ip->i_delayed_blks = 0; - memset(&ip->i_d, 0, sizeof(xfs_icdinode_t)); + memset(&ip->i_d, 0, sizeof(ip->i_d)); return ip; } diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index ae3758a..7e24232 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -650,9 +650,9 @@ _xfs_dic2xflags( uint xfs_ip2xflags( - xfs_inode_t *ip) + struct xfs_inode *ip) { - xfs_icdinode_t *dic = &ip->i_d; + struct xfs_icdinode *dic = &ip->i_d; return _xfs_dic2xflags(dic->di_flags) | (XFS_IFORK_Q(ip) ? XFS_XFLAG_HASATTR : 0); @@ -660,7 +660,7 @@ xfs_ip2xflags( uint xfs_dic2xflags( - xfs_dinode_t *dip) + struct xfs_dinode *dip) { return _xfs_dic2xflags(be16_to_cpu(dip->di_flags)) | (XFS_DFORK_Q(dip) ? XFS_XFLAG_HASATTR : 0); diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h index ca9e119..aef5452 100644 --- a/fs/xfs/xfs_inode.h +++ b/fs/xfs/xfs_inode.h @@ -63,7 +63,7 @@ typedef struct xfs_inode { unsigned long i_flags; /* see defined flags below */ unsigned int i_delayed_blks; /* count of delay alloc blks */ - xfs_icdinode_t i_d; /* most of ondisk inode */ + struct xfs_icdinode i_d; /* most of ondisk inode */ /* VFS inode */ struct inode i_vnode; /* embedded VFS inode */ diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index d14b12b..3ad9972 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -135,7 +135,7 @@ xfs_inode_item_size( *nvecs += 2; *nbytes += sizeof(struct xfs_inode_log_format) + - xfs_icdinode_size(ip->i_d.di_version); + xfs_log_dinode_size(ip->i_d.di_version); xfs_inode_item_data_fork_size(iip, nvecs, nbytes); if (XFS_IFORK_Q(ip)) @@ -322,6 +322,127 @@ xfs_inode_item_format_attr_fork( } } +static void +xfs_icdinode_to_log_dinode( + struct xfs_icdinode *from, + struct xfs_log_dinode *to) +{ + to->di_magic = from->di_magic; + to->di_mode = from->di_mode; + to->di_version = from->di_version; + to->di_format = from->di_format; + to->di_onlink = from->di_onlink; + to->di_uid = from->di_uid; + to->di_gid = from->di_gid; + to->di_nlink = from->di_nlink; + to->di_projid_lo = from->di_projid_lo; + to->di_projid_hi = from->di_projid_hi; + memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad)); + to->di_atime.t_sec = from->di_atime.t_sec; + to->di_atime.t_nsec = from->di_atime.t_nsec; + to->di_mtime.t_sec = from->di_mtime.t_sec; + to->di_mtime.t_nsec = from->di_mtime.t_nsec; + to->di_ctime.t_sec = from->di_ctime.t_sec; + to->di_ctime.t_nsec = from->di_ctime.t_nsec; + to->di_size = from->di_size; + to->di_nblocks = from->di_nblocks; + to->di_extsize = from->di_extsize; + to->di_nextents = from->di_nextents; + to->di_anextents = from->di_anextents; + to->di_forkoff = from->di_forkoff; + to->di_aformat = from->di_aformat; + to->di_dmevmask = from->di_dmevmask; + to->di_dmstate = from->di_dmstate; + to->di_flags = from->di_flags; + to->di_gen = from->di_gen; + + if (from->di_version == 3) { + to->di_changecount = from->di_changecount; + to->di_crtime.t_sec = from->di_crtime.t_sec; + to->di_crtime.t_nsec = from->di_crtime.t_nsec; + to->di_flags2 = from->di_flags2; + to->di_ino = from->di_ino; + to->di_lsn = from->di_lsn; + memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2)); + uuid_copy(&to->di_uuid, &from->di_uuid); + to->di_flushiter = 0; + } else { + to->di_flushiter = from->di_flushiter; + } +} + +/* + * Recovery needs to be able to convert a log dinode back to a real dinode + * for writeback we do that by converting a log dinode to a icdinode, and + * then passing that to the formatting function. + */ +void +xfs_log_dinode_to_icdinode( + struct xfs_log_dinode *from, + struct xfs_icdinode *to) +{ + to->di_magic = from->di_magic; + to->di_mode = from->di_mode; + to->di_version = from->di_version; + to->di_format = from->di_format; + to->di_onlink = from->di_onlink; + to->di_uid = from->di_uid; + to->di_gid = from->di_gid; + to->di_nlink = from->di_nlink; + to->di_projid_lo = from->di_projid_lo; + to->di_projid_hi = from->di_projid_hi; + memset(to->di_pad, 0, sizeof(to->di_pad)); + to->di_atime.t_sec = from->di_atime.t_sec; + to->di_atime.t_nsec = from->di_atime.t_nsec; + to->di_mtime.t_sec = from->di_mtime.t_sec; + to->di_mtime.t_nsec = from->di_mtime.t_nsec; + to->di_ctime.t_sec = from->di_ctime.t_sec; + to->di_ctime.t_nsec = from->di_ctime.t_nsec; + to->di_size = from->di_size; + to->di_nblocks = from->di_nblocks; + to->di_extsize = from->di_extsize; + to->di_nextents = from->di_nextents; + to->di_anextents = from->di_anextents; + to->di_forkoff = from->di_forkoff; + to->di_aformat = from->di_aformat; + to->di_dmevmask = from->di_dmevmask; + to->di_dmstate = from->di_dmstate; + to->di_flags = from->di_flags; + to->di_gen = from->di_gen; + + if (from->di_version == 3) { + to->di_changecount = from->di_changecount; + to->di_crtime.t_sec = from->di_crtime.t_sec; + to->di_crtime.t_nsec = from->di_crtime.t_nsec; + to->di_flags2 = from->di_flags2; + to->di_ino = from->di_ino; + to->di_lsn = from->di_lsn; + memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2)); + uuid_copy(&to->di_uuid, &from->di_uuid); + to->di_flushiter = 0; + } else { + to->di_flushiter = from->di_flushiter; + } +} + +/* + * Format the inode core. Current timestamp data is only in the VFS inode + * fields, so we need to grab them from there. Hence rather than just copying + * the XFS inode core structure, format the fields directly into the iovec. + */ +static void +xfs_inode_item_format_core( + struct xfs_inode *ip, + struct xfs_log_vec *lv, + struct xfs_log_iovec **vecp) +{ + struct xfs_log_dinode *dic; + + dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE); + xfs_icdinode_to_log_dinode(&ip->i_d, dic); + xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version)); +} + /* * This is called to fill in the vector of log iovecs for the given inode * log item. It fills the first item with an inode log format structure, @@ -351,10 +472,7 @@ xfs_inode_item_format( ilf->ilf_size = 2; /* format + core */ xlog_finish_iovec(lv, vecp, sizeof(struct xfs_inode_log_format)); - xlog_copy_iovec(lv, &vecp, XLOG_REG_TYPE_ICORE, - &ip->i_d, - xfs_icdinode_size(ip->i_d.di_version)); - + xfs_inode_item_format_core(ip, lv, &vecp); xfs_inode_item_format_data_fork(iip, ilf, lv, &vecp); if (XFS_IFORK_Q(ip)) { xfs_inode_item_format_attr_fork(iip, ilf, lv, &vecp); diff --git a/fs/xfs/xfs_inode_item.h b/fs/xfs/xfs_inode_item.h index 4c7722e..2426118 100644 --- a/fs/xfs/xfs_inode_item.h +++ b/fs/xfs/xfs_inode_item.h @@ -49,6 +49,8 @@ extern void xfs_istale_done(struct xfs_buf *, struct xfs_log_item *); extern void xfs_iflush_abort(struct xfs_inode *, bool); extern int xfs_inode_item_format_convert(xfs_log_iovec_t *, xfs_inode_log_format_t *); +extern void xfs_log_dinode_to_icdinode(struct xfs_log_dinode *from, + struct xfs_icdinode *to); extern struct kmem_zone *xfs_ili_zone; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index da37beb..3120f7b 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -2839,7 +2839,8 @@ xlog_recover_inode_pass2( int error; int attr_index; uint fields; - xfs_icdinode_t *dicp; + struct xfs_log_dinode *ldip; + struct xfs_icdinode icic; uint isize; int need_free = 0; @@ -2892,8 +2893,8 @@ xlog_recover_inode_pass2( error = -EFSCORRUPTED; goto out_release; } - dicp = item->ri_buf[1].i_addr; - if (unlikely(dicp->di_magic != XFS_DINODE_MAGIC)) { + ldip = item->ri_buf[1].i_addr; + if (unlikely(ldip->di_magic != XFS_DINODE_MAGIC)) { xfs_alert(mp, "%s: Bad inode log record, rec ptr 0x%p, ino %Ld", __func__, item, in_f->ilf_ino); @@ -2929,13 +2930,13 @@ xlog_recover_inode_pass2( * to skip replay when the on disk inode is newer than the log one */ if (!xfs_sb_version_hascrc(&mp->m_sb) && - dicp->di_flushiter < be16_to_cpu(dip->di_flushiter)) { + ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) { /* * Deal with the wrap case, DI_MAX_FLUSH is less * than smaller numbers */ if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH && - dicp->di_flushiter < (DI_MAX_FLUSH >> 1)) { + ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) { /* do nothing */ } else { trace_xfs_log_recover_inode_skip(log, in_f); @@ -2945,13 +2946,13 @@ xlog_recover_inode_pass2( } /* Take the opportunity to reset the flush iteration count */ - dicp->di_flushiter = 0; + ldip->di_flushiter = 0; - if (unlikely(S_ISREG(dicp->di_mode))) { - if ((dicp->di_format != XFS_DINODE_FMT_EXTENTS) && - (dicp->di_format != XFS_DINODE_FMT_BTREE)) { + if (unlikely(S_ISREG(ldip->di_mode))) { + if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) && + (ldip->di_format != XFS_DINODE_FMT_BTREE)) { XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(3)", - XFS_ERRLEVEL_LOW, mp, dicp); + XFS_ERRLEVEL_LOW, mp, ldip); xfs_alert(mp, "%s: Bad regular inode log record, rec ptr 0x%p, " "ino ptr = 0x%p, ino bp = 0x%p, ino %Ld", @@ -2959,12 +2960,12 @@ xlog_recover_inode_pass2( error = -EFSCORRUPTED; goto out_release; } - } else if (unlikely(S_ISDIR(dicp->di_mode))) { - if ((dicp->di_format != XFS_DINODE_FMT_EXTENTS) && - (dicp->di_format != XFS_DINODE_FMT_BTREE) && - (dicp->di_format != XFS_DINODE_FMT_LOCAL)) { + } else if (unlikely(S_ISDIR(ldip->di_mode))) { + if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) && + (ldip->di_format != XFS_DINODE_FMT_BTREE) && + (ldip->di_format != XFS_DINODE_FMT_LOCAL)) { XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(4)", - XFS_ERRLEVEL_LOW, mp, dicp); + XFS_ERRLEVEL_LOW, mp, ldip); xfs_alert(mp, "%s: Bad dir inode log record, rec ptr 0x%p, " "ino ptr = 0x%p, ino bp = 0x%p, ino %Ld", @@ -2973,32 +2974,32 @@ xlog_recover_inode_pass2( goto out_release; } } - if (unlikely(dicp->di_nextents + dicp->di_anextents > dicp->di_nblocks)){ + if (unlikely(ldip->di_nextents + ldip->di_anextents > ldip->di_nblocks)){ XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(5)", - XFS_ERRLEVEL_LOW, mp, dicp); + XFS_ERRLEVEL_LOW, mp, ldip); xfs_alert(mp, "%s: Bad inode log record, rec ptr 0x%p, dino ptr 0x%p, " "dino bp 0x%p, ino %Ld, total extents = %d, nblocks = %Ld", __func__, item, dip, bp, in_f->ilf_ino, - dicp->di_nextents + dicp->di_anextents, - dicp->di_nblocks); + ldip->di_nextents + ldip->di_anextents, + ldip->di_nblocks); error = -EFSCORRUPTED; goto out_release; } - if (unlikely(dicp->di_forkoff > mp->m_sb.sb_inodesize)) { + if (unlikely(ldip->di_forkoff > mp->m_sb.sb_inodesize)) { XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(6)", - XFS_ERRLEVEL_LOW, mp, dicp); + XFS_ERRLEVEL_LOW, mp, ldip); xfs_alert(mp, "%s: Bad inode log record, rec ptr 0x%p, dino ptr 0x%p, " "dino bp 0x%p, ino %Ld, forkoff 0x%x", __func__, - item, dip, bp, in_f->ilf_ino, dicp->di_forkoff); + item, dip, bp, in_f->ilf_ino, ldip->di_forkoff); error = -EFSCORRUPTED; goto out_release; } - isize = xfs_icdinode_size(dicp->di_version); + isize = xfs_log_dinode_size(ldip->di_version); if (unlikely(item->ri_buf[1].i_len > isize)) { XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(7)", - XFS_ERRLEVEL_LOW, mp, dicp); + XFS_ERRLEVEL_LOW, mp, ldip); xfs_alert(mp, "%s: Bad inode log record length %d, rec ptr 0x%p", __func__, item->ri_buf[1].i_len, item); @@ -3007,7 +3008,8 @@ xlog_recover_inode_pass2( } /* The core is in in-core format */ - xfs_dinode_to_disk(dip, dicp); + xfs_log_dinode_to_icdinode(ldip, &icic); + xfs_dinode_to_disk(dip, &icic); /* the rest is in on-disk format */ if (item->ri_buf[1].i_len > isize) { -- 2.5.0 From dave@fromorbit.com Tue Jan 12 03:01:36 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id C47887F59 for ; Tue, 12 Jan 2016 03:01:35 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id A77008F8035 for ; Tue, 12 Jan 2016 01:01:35 -0800 (PST) X-ASG-Debug-ID: 1452589285-04cb6c75de89ff0004-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 37xXOOAkByhyo3hu for ; Tue, 12 Jan 2016 01:01:31 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DFDABTwJRWPGu7LXleKAECgw+BP4ZggXmffAaRGooWgSpNAQEBAQEBBwEBAQFBP4Q1AQUnLzMIGDE5AwcUGYgtwBOGDopEhEAFh2SPL5xVjlOCLgELATcDHIFxKjSGMQEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:24 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005Jz-J3 for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008AE-IQ for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 2/8] xfs: remove timestamps from incore inode Date: Tue, 12 Jan 2016 20:01:14 +1100 X-ASG-Orig-Subj: [PATCH 2/8] xfs: remove timestamps from incore inode Message-Id: <1452589280-30167-3-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> References: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589291 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner The struct xfs_inode has two copies of the current timestamps in it, one in the vfs inode and one in the struct xfs_icdinode. Now that we no longer log the struct xfs_icdinode directly, we don't need to keep the timestamps in this structure. instead we can copy them straight out of the VFS inode when formatting the inode log item or the on-disk inode. This reduces the struct xfs_inode in size by 24 bytes. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_inode_buf.c | 85 +++++++++++++++++++++++++++++++++++++------ fs/xfs/libxfs/xfs_inode_buf.h | 9 ++--- fs/xfs/libxfs/xfs_rtbitmap.c | 2 +- fs/xfs/xfs_inode.c | 21 ++++++----- fs/xfs/xfs_inode_item.c | 73 ++++++------------------------------- fs/xfs/xfs_inode_item.h | 2 - fs/xfs/xfs_iops.c | 37 ++++--------------- fs/xfs/xfs_itable.c | 16 +++++--- fs/xfs/xfs_log_recover.c | 8 ++-- fs/xfs/xfs_rtalloc.c | 2 +- fs/xfs/xfs_trans_inode.c | 12 +----- 11 files changed, 124 insertions(+), 143 deletions(-) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index 63d46bf..0f33265 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -195,10 +195,13 @@ xfs_imap_to_bp( } void -xfs_dinode_from_disk( - struct xfs_icdinode *to, +xfs_inode_from_disk( + struct xfs_inode *ip, struct xfs_dinode *from) { + struct xfs_icdinode *to = &ip->i_d; + struct inode *inode = VFS_I(ip); + to->di_magic = be16_to_cpu(from->di_magic); to->di_mode = be16_to_cpu(from->di_mode); to->di_version = from ->di_version; @@ -211,12 +214,14 @@ xfs_dinode_from_disk( to->di_projid_hi = be16_to_cpu(from->di_projid_hi); memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad)); to->di_flushiter = be16_to_cpu(from->di_flushiter); - to->di_atime.t_sec = be32_to_cpu(from->di_atime.t_sec); - to->di_atime.t_nsec = be32_to_cpu(from->di_atime.t_nsec); - to->di_mtime.t_sec = be32_to_cpu(from->di_mtime.t_sec); - to->di_mtime.t_nsec = be32_to_cpu(from->di_mtime.t_nsec); - to->di_ctime.t_sec = be32_to_cpu(from->di_ctime.t_sec); - to->di_ctime.t_nsec = be32_to_cpu(from->di_ctime.t_nsec); + + inode->i_atime.tv_sec = be32_to_cpu(from->di_atime.t_sec); + inode->i_atime.tv_nsec = be32_to_cpu(from->di_atime.t_nsec); + inode->i_mtime.tv_sec = be32_to_cpu(from->di_mtime.t_sec); + inode->i_mtime.tv_nsec = be32_to_cpu(from->di_mtime.t_nsec); + inode->i_ctime.tv_sec = be32_to_cpu(from->di_ctime.t_sec); + inode->i_ctime.tv_nsec = be32_to_cpu(from->di_ctime.t_nsec); + to->di_size = be64_to_cpu(from->di_size); to->di_nblocks = be64_to_cpu(from->di_nblocks); to->di_extsize = be32_to_cpu(from->di_extsize); @@ -242,9 +247,63 @@ xfs_dinode_from_disk( } void -xfs_dinode_to_disk( - struct xfs_dinode *to, - struct xfs_icdinode *from) +xfs_inode_to_disk( + struct xfs_inode *ip, + struct xfs_dinode *to) +{ + struct xfs_icdinode *from = &ip->i_d; + struct inode *inode = VFS_I(ip); + + to->di_magic = cpu_to_be16(from->di_magic); + to->di_mode = cpu_to_be16(from->di_mode); + to->di_version = from ->di_version; + to->di_format = from->di_format; + to->di_onlink = cpu_to_be16(from->di_onlink); + to->di_uid = cpu_to_be32(from->di_uid); + to->di_gid = cpu_to_be32(from->di_gid); + to->di_nlink = cpu_to_be32(from->di_nlink); + to->di_projid_lo = cpu_to_be16(from->di_projid_lo); + to->di_projid_hi = cpu_to_be16(from->di_projid_hi); + memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad)); + + to->di_atime.t_sec = cpu_to_be32(inode->i_atime.tv_sec); + to->di_atime.t_nsec = cpu_to_be32(inode->i_atime.tv_nsec); + to->di_mtime.t_sec = cpu_to_be32(inode->i_mtime.tv_sec); + to->di_mtime.t_nsec = cpu_to_be32(inode->i_mtime.tv_nsec); + to->di_ctime.t_sec = cpu_to_be32(inode->i_ctime.tv_sec); + to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec); + + to->di_size = cpu_to_be64(from->di_size); + to->di_nblocks = cpu_to_be64(from->di_nblocks); + to->di_extsize = cpu_to_be32(from->di_extsize); + to->di_nextents = cpu_to_be32(from->di_nextents); + to->di_anextents = cpu_to_be16(from->di_anextents); + to->di_forkoff = from->di_forkoff; + to->di_aformat = from->di_aformat; + to->di_dmevmask = cpu_to_be32(from->di_dmevmask); + to->di_dmstate = cpu_to_be16(from->di_dmstate); + to->di_flags = cpu_to_be16(from->di_flags); + to->di_gen = cpu_to_be32(from->di_gen); + + if (from->di_version == 3) { + to->di_changecount = cpu_to_be64(from->di_changecount); + to->di_crtime.t_sec = cpu_to_be32(from->di_crtime.t_sec); + to->di_crtime.t_nsec = cpu_to_be32(from->di_crtime.t_nsec); + to->di_flags2 = cpu_to_be64(from->di_flags2); + to->di_ino = cpu_to_be64(from->di_ino); + to->di_lsn = cpu_to_be64(from->di_lsn); + memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2)); + uuid_copy(&to->di_uuid, &from->di_uuid); + to->di_flushiter = 0; + } else { + to->di_flushiter = cpu_to_be16(from->di_flushiter); + } +} + +void +xfs_log_dinode_to_disk( + struct xfs_log_dinode *from, + struct xfs_dinode *to) { to->di_magic = cpu_to_be16(from->di_magic); to->di_mode = cpu_to_be16(from->di_mode); @@ -257,12 +316,14 @@ xfs_dinode_to_disk( to->di_projid_lo = cpu_to_be16(from->di_projid_lo); to->di_projid_hi = cpu_to_be16(from->di_projid_hi); memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad)); + to->di_atime.t_sec = cpu_to_be32(from->di_atime.t_sec); to->di_atime.t_nsec = cpu_to_be32(from->di_atime.t_nsec); to->di_mtime.t_sec = cpu_to_be32(from->di_mtime.t_sec); to->di_mtime.t_nsec = cpu_to_be32(from->di_mtime.t_nsec); to->di_ctime.t_sec = cpu_to_be32(from->di_ctime.t_sec); to->di_ctime.t_nsec = cpu_to_be32(from->di_ctime.t_nsec); + to->di_size = cpu_to_be64(from->di_size); to->di_nblocks = cpu_to_be64(from->di_nblocks); to->di_extsize = cpu_to_be32(from->di_extsize); @@ -403,7 +464,7 @@ xfs_iread( * Otherwise, just get the truly permanent information. */ if (dip->di_mode) { - xfs_dinode_from_disk(&ip->i_d, dip); + xfs_inode_from_disk(ip, dip); error = xfs_iformat_fork(ip, dip); if (error) { #ifdef DEBUG diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h index 642f2a2..adcc9bf 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.h +++ b/fs/xfs/libxfs/xfs_inode_buf.h @@ -40,9 +40,6 @@ struct xfs_icdinode { __uint16_t di_projid_hi; /* higher part of owner's project id */ __uint8_t di_pad[6]; /* unused, zeroed space */ __uint16_t di_flushiter; /* incremented on flush */ - xfs_ictimestamp_t di_atime; /* time last accessed */ - xfs_ictimestamp_t di_mtime; /* time last modified */ - xfs_ictimestamp_t di_ctime; /* time created/inode modified */ xfs_fsize_t di_size; /* number of bytes in file */ xfs_rfsblock_t di_nblocks; /* # of direct & btree blocks used */ xfs_extlen_t di_extsize; /* basic/minimum extent size for file */ @@ -89,8 +86,10 @@ int xfs_imap_to_bp(struct xfs_mount *, struct xfs_trans *, int xfs_iread(struct xfs_mount *, struct xfs_trans *, struct xfs_inode *, uint); void xfs_dinode_calc_crc(struct xfs_mount *, struct xfs_dinode *); -void xfs_dinode_to_disk(struct xfs_dinode *to, struct xfs_icdinode *from); -void xfs_dinode_from_disk(struct xfs_icdinode *to, struct xfs_dinode *from); +void xfs_inode_to_disk(struct xfs_inode *ip, struct xfs_dinode *to); +void xfs_inode_from_disk(struct xfs_inode *ip, struct xfs_dinode *from); +void xfs_log_dinode_to_disk(struct xfs_log_dinode *from, + struct xfs_dinode *to); #if defined(DEBUG) void xfs_inobp_check(struct xfs_mount *, struct xfs_buf *); diff --git a/fs/xfs/libxfs/xfs_rtbitmap.c b/fs/xfs/libxfs/xfs_rtbitmap.c index 9b59ffa..acc71dd 100644 --- a/fs/xfs/libxfs/xfs_rtbitmap.c +++ b/fs/xfs/libxfs/xfs_rtbitmap.c @@ -983,7 +983,7 @@ xfs_rtfree_extent( mp->m_sb.sb_rextents) { if (!(mp->m_rbmip->i_d.di_flags & XFS_DIFLAG_NEWRTBM)) mp->m_rbmip->i_d.di_flags |= XFS_DIFLAG_NEWRTBM; - *(__uint64_t *)&mp->m_rbmip->i_d.di_atime = 0; + *(__uint64_t *)&VFS_I(mp->m_rbmip)->i_atime = 0; xfs_trans_log_inode(tp, mp->m_rbmip, XFS_ILOG_CORE); } return 0; diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index 7e24232..3e9b8a0 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -757,6 +757,7 @@ xfs_ialloc( uint flags; int error; struct timespec tv; + struct inode *inode; /* * Call the space management code to pick @@ -782,6 +783,7 @@ xfs_ialloc( if (error) return error; ASSERT(ip != NULL); + inode = VFS_I(ip); /* * We always convert v1 inodes to v2 now - we only support filesystems @@ -823,10 +825,9 @@ xfs_ialloc( ASSERT(ip->i_d.di_nblocks == 0); tv = current_fs_time(mp->m_super); - ip->i_d.di_mtime.t_sec = (__int32_t)tv.tv_sec; - ip->i_d.di_mtime.t_nsec = (__int32_t)tv.tv_nsec; - ip->i_d.di_atime = ip->i_d.di_mtime; - ip->i_d.di_ctime = ip->i_d.di_mtime; + inode->i_mtime = tv; + inode->i_atime = tv; + inode->i_ctime = tv; /* * di_gen will have been taken care of in xfs_iread. @@ -844,7 +845,8 @@ xfs_ialloc( ip->i_d.di_lsn = 0; ip->i_d.di_flags2 = 0; memset(&(ip->i_d.di_pad2[0]), 0, sizeof(ip->i_d.di_pad2)); - ip->i_d.di_crtime = ip->i_d.di_mtime; + ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec; + ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec; } @@ -3509,12 +3511,11 @@ xfs_iflush_int( ip->i_d.di_flushiter++; /* - * Copy the dirty parts of the inode into the on-disk - * inode. We always copy out the core of the inode, - * because if the inode is dirty at all the core must - * be. + * Copy the dirty parts of the inode into the on-disk inode. We always + * copy out the core of the inode, because if the inode is dirty at all + * the core must be. */ - xfs_dinode_to_disk(dip, &ip->i_d); + xfs_inode_to_disk(ip, dip); /* Wrap, we never let the log put out DI_MAX_FLUSH */ if (ip->i_d.di_flushiter == DI_MAX_FLUSH) diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index 3ad9972..9dcbf58 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -323,10 +323,13 @@ xfs_inode_item_format_attr_fork( } static void -xfs_icdinode_to_log_dinode( - struct xfs_icdinode *from, +xfs_inode_to_log_dinode( + struct xfs_inode *ip, struct xfs_log_dinode *to) { + struct xfs_icdinode *from = &ip->i_d; + struct inode *inode = VFS_I(ip); + to->di_magic = from->di_magic; to->di_mode = from->di_mode; to->di_version = from->di_version; @@ -338,66 +341,14 @@ xfs_icdinode_to_log_dinode( to->di_projid_lo = from->di_projid_lo; to->di_projid_hi = from->di_projid_hi; memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad)); - to->di_atime.t_sec = from->di_atime.t_sec; - to->di_atime.t_nsec = from->di_atime.t_nsec; - to->di_mtime.t_sec = from->di_mtime.t_sec; - to->di_mtime.t_nsec = from->di_mtime.t_nsec; - to->di_ctime.t_sec = from->di_ctime.t_sec; - to->di_ctime.t_nsec = from->di_ctime.t_nsec; - to->di_size = from->di_size; - to->di_nblocks = from->di_nblocks; - to->di_extsize = from->di_extsize; - to->di_nextents = from->di_nextents; - to->di_anextents = from->di_anextents; - to->di_forkoff = from->di_forkoff; - to->di_aformat = from->di_aformat; - to->di_dmevmask = from->di_dmevmask; - to->di_dmstate = from->di_dmstate; - to->di_flags = from->di_flags; - to->di_gen = from->di_gen; - if (from->di_version == 3) { - to->di_changecount = from->di_changecount; - to->di_crtime.t_sec = from->di_crtime.t_sec; - to->di_crtime.t_nsec = from->di_crtime.t_nsec; - to->di_flags2 = from->di_flags2; - to->di_ino = from->di_ino; - to->di_lsn = from->di_lsn; - memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2)); - uuid_copy(&to->di_uuid, &from->di_uuid); - to->di_flushiter = 0; - } else { - to->di_flushiter = from->di_flushiter; - } -} + to->di_atime.t_sec = inode->i_atime.tv_sec; + to->di_atime.t_nsec = inode->i_atime.tv_nsec; + to->di_mtime.t_sec = inode->i_mtime.tv_sec; + to->di_mtime.t_nsec = inode->i_mtime.tv_nsec; + to->di_ctime.t_sec = inode->i_ctime.tv_sec; + to->di_ctime.t_nsec = inode->i_ctime.tv_nsec; -/* - * Recovery needs to be able to convert a log dinode back to a real dinode - * for writeback we do that by converting a log dinode to a icdinode, and - * then passing that to the formatting function. - */ -void -xfs_log_dinode_to_icdinode( - struct xfs_log_dinode *from, - struct xfs_icdinode *to) -{ - to->di_magic = from->di_magic; - to->di_mode = from->di_mode; - to->di_version = from->di_version; - to->di_format = from->di_format; - to->di_onlink = from->di_onlink; - to->di_uid = from->di_uid; - to->di_gid = from->di_gid; - to->di_nlink = from->di_nlink; - to->di_projid_lo = from->di_projid_lo; - to->di_projid_hi = from->di_projid_hi; - memset(to->di_pad, 0, sizeof(to->di_pad)); - to->di_atime.t_sec = from->di_atime.t_sec; - to->di_atime.t_nsec = from->di_atime.t_nsec; - to->di_mtime.t_sec = from->di_mtime.t_sec; - to->di_mtime.t_nsec = from->di_mtime.t_nsec; - to->di_ctime.t_sec = from->di_ctime.t_sec; - to->di_ctime.t_nsec = from->di_ctime.t_nsec; to->di_size = from->di_size; to->di_nblocks = from->di_nblocks; to->di_extsize = from->di_extsize; @@ -439,7 +390,7 @@ xfs_inode_item_format_core( struct xfs_log_dinode *dic; dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE); - xfs_icdinode_to_log_dinode(&ip->i_d, dic); + xfs_inode_to_log_dinode(ip, dic); xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version)); } diff --git a/fs/xfs/xfs_inode_item.h b/fs/xfs/xfs_inode_item.h index 2426118..4c7722e 100644 --- a/fs/xfs/xfs_inode_item.h +++ b/fs/xfs/xfs_inode_item.h @@ -49,8 +49,6 @@ extern void xfs_istale_done(struct xfs_buf *, struct xfs_log_item *); extern void xfs_iflush_abort(struct xfs_inode *, bool); extern int xfs_inode_item_format_convert(xfs_log_iovec_t *, xfs_inode_log_format_t *); -extern void xfs_log_dinode_to_icdinode(struct xfs_log_dinode *from, - struct xfs_icdinode *to); extern struct kmem_zone *xfs_ili_zone; diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c index 245268a..b008677 100644 --- a/fs/xfs/xfs_iops.c +++ b/fs/xfs/xfs_iops.c @@ -517,21 +517,12 @@ xfs_setattr_time( ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL)); - if (iattr->ia_valid & ATTR_ATIME) { + if (iattr->ia_valid & ATTR_ATIME) inode->i_atime = iattr->ia_atime; - ip->i_d.di_atime.t_sec = iattr->ia_atime.tv_sec; - ip->i_d.di_atime.t_nsec = iattr->ia_atime.tv_nsec; - } - if (iattr->ia_valid & ATTR_CTIME) { + if (iattr->ia_valid & ATTR_CTIME) inode->i_ctime = iattr->ia_ctime; - ip->i_d.di_ctime.t_sec = iattr->ia_ctime.tv_sec; - ip->i_d.di_ctime.t_nsec = iattr->ia_ctime.tv_nsec; - } - if (iattr->ia_valid & ATTR_MTIME) { + if (iattr->ia_valid & ATTR_MTIME) inode->i_mtime = iattr->ia_mtime; - ip->i_d.di_mtime.t_sec = iattr->ia_mtime.tv_sec; - ip->i_d.di_mtime.t_nsec = iattr->ia_mtime.tv_nsec; - } } int @@ -986,21 +977,13 @@ xfs_vn_update_time( } xfs_ilock(ip, XFS_ILOCK_EXCL); - if (flags & S_CTIME) { + if (flags & S_CTIME) inode->i_ctime = *now; - ip->i_d.di_ctime.t_sec = (__int32_t)now->tv_sec; - ip->i_d.di_ctime.t_nsec = (__int32_t)now->tv_nsec; - } - if (flags & S_MTIME) { + if (flags & S_MTIME) inode->i_mtime = *now; - ip->i_d.di_mtime.t_sec = (__int32_t)now->tv_sec; - ip->i_d.di_mtime.t_nsec = (__int32_t)now->tv_nsec; - } - if (flags & S_ATIME) { + if (flags & S_ATIME) inode->i_atime = *now; - ip->i_d.di_atime.t_sec = (__int32_t)now->tv_sec; - ip->i_d.di_atime.t_nsec = (__int32_t)now->tv_nsec; - } + xfs_trans_ijoin(tp, ip, XFS_ILOCK_EXCL); xfs_trans_log_inode(tp, ip, XFS_ILOG_TIMESTAMP); return xfs_trans_commit(tp); @@ -1247,12 +1230,6 @@ xfs_setup_inode( inode->i_generation = ip->i_d.di_gen; i_size_write(inode, ip->i_d.di_size); - inode->i_atime.tv_sec = ip->i_d.di_atime.t_sec; - inode->i_atime.tv_nsec = ip->i_d.di_atime.t_nsec; - inode->i_mtime.tv_sec = ip->i_d.di_mtime.t_sec; - inode->i_mtime.tv_nsec = ip->i_d.di_mtime.t_nsec; - inode->i_ctime.tv_sec = ip->i_d.di_ctime.t_sec; - inode->i_ctime.tv_nsec = ip->i_d.di_ctime.t_nsec; xfs_diflags_to_iflags(inode, ip); ip->d_ops = ip->i_mount->m_nondir_inode_ops; diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c index 930ebd8..2acda42 100644 --- a/fs/xfs/xfs_itable.c +++ b/fs/xfs/xfs_itable.c @@ -57,6 +57,7 @@ xfs_bulkstat_one_int( { struct xfs_icdinode *dic; /* dinode core info pointer */ struct xfs_inode *ip; /* incore inode pointer */ + struct inode *inode; struct xfs_bstat *buf; /* return buffer */ int error = 0; /* error value */ @@ -77,6 +78,7 @@ xfs_bulkstat_one_int( ASSERT(ip != NULL); ASSERT(ip->i_imap.im_blkno != 0); + inode = VFS_I(ip); dic = &ip->i_d; @@ -91,12 +93,14 @@ xfs_bulkstat_one_int( buf->bs_uid = dic->di_uid; buf->bs_gid = dic->di_gid; buf->bs_size = dic->di_size; - buf->bs_atime.tv_sec = dic->di_atime.t_sec; - buf->bs_atime.tv_nsec = dic->di_atime.t_nsec; - buf->bs_mtime.tv_sec = dic->di_mtime.t_sec; - buf->bs_mtime.tv_nsec = dic->di_mtime.t_nsec; - buf->bs_ctime.tv_sec = dic->di_ctime.t_sec; - buf->bs_ctime.tv_nsec = dic->di_ctime.t_nsec; + + buf->bs_atime.tv_sec = inode->i_atime.tv_sec; + buf->bs_atime.tv_nsec = inode->i_atime.tv_nsec; + buf->bs_mtime.tv_sec = inode->i_mtime.tv_sec; + buf->bs_mtime.tv_nsec = inode->i_mtime.tv_nsec; + buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec; + buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec; + buf->bs_xflags = xfs_ip2xflags(ip); buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog; buf->bs_extents = dic->di_nextents; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 3120f7b..4b79cf0 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -2793,7 +2793,7 @@ xfs_recover_inode_owner_change( return -ENOMEM; /* instantiate the inode */ - xfs_dinode_from_disk(&ip->i_d, dip); + xfs_inode_from_disk(ip, dip); ASSERT(ip->i_d.di_version >= 3); error = xfs_iformat_fork(ip, dip); @@ -2840,7 +2840,6 @@ xlog_recover_inode_pass2( int attr_index; uint fields; struct xfs_log_dinode *ldip; - struct xfs_icdinode icic; uint isize; int need_free = 0; @@ -3007,9 +3006,8 @@ xlog_recover_inode_pass2( goto out_release; } - /* The core is in in-core format */ - xfs_log_dinode_to_icdinode(ldip, &icic); - xfs_dinode_to_disk(dip, &icic); + /* recover the log dinode inode into the on disk inode */ + xfs_log_dinode_to_disk(ldip, dip); /* the rest is in on-disk format */ if (item->ri_buf[1].i_len > isize) { diff --git a/fs/xfs/xfs_rtalloc.c b/fs/xfs/xfs_rtalloc.c index be02a68..abf4443 100644 --- a/fs/xfs/xfs_rtalloc.c +++ b/fs/xfs/xfs_rtalloc.c @@ -1272,7 +1272,7 @@ xfs_rtpick_extent( ASSERT(xfs_isilocked(mp->m_rbmip, XFS_ILOCK_EXCL)); - seqp = (__uint64_t *)&mp->m_rbmip->i_d.di_atime; + seqp = (__uint64_t *)&VFS_I(mp->m_rbmip)->i_atime; if (!(mp->m_rbmip->i_d.di_flags & XFS_DIFLAG_NEWRTBM)) { mp->m_rbmip->i_d.di_flags |= XFS_DIFLAG_NEWRTBM; *seqp = 0; diff --git a/fs/xfs/xfs_trans_inode.c b/fs/xfs/xfs_trans_inode.c index b97f1df..3f0d466 100644 --- a/fs/xfs/xfs_trans_inode.c +++ b/fs/xfs/xfs_trans_inode.c @@ -75,18 +75,10 @@ xfs_trans_ichgtime( tv = current_fs_time(inode->i_sb); - if ((flags & XFS_ICHGTIME_MOD) && - !timespec_equal(&inode->i_mtime, &tv)) { + if (flags & XFS_ICHGTIME_MOD) inode->i_mtime = tv; - ip->i_d.di_mtime.t_sec = tv.tv_sec; - ip->i_d.di_mtime.t_nsec = tv.tv_nsec; - } - if ((flags & XFS_ICHGTIME_CHG) && - !timespec_equal(&inode->i_ctime, &tv)) { + if (flags & XFS_ICHGTIME_CHG) inode->i_ctime = tv; - ip->i_d.di_ctime.t_sec = tv.tv_sec; - ip->i_d.di_ctime.t_nsec = tv.tv_nsec; - } } /* -- 2.5.0 From dave@fromorbit.com Tue Jan 12 03:01:36 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 691407F56 for ; Tue, 12 Jan 2016 03:01:35 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id E70A9AC002 for ; Tue, 12 Jan 2016 01:01:34 -0800 (PST) X-ASG-Debug-ID: 1452589290-04cb6c75dd8a000002-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id AHrUkljVTWcEIg9R for ; Tue, 12 Jan 2016 01:01:32 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DFDABTwJRWPGu7LXleKAECgw+BP4ZggXmffAaRGooWgSpNAQEBAQEBBwEBAQFBP4Q1AQUnLzMIGBkYOQMHFBmILcAThg6KRCIMhBIFh2SPL483jR6FZYhugi4BCwE3AxyBcSo0hG+BQgEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:24 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005K0-JQ for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008AJ-Ir for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 3/8] xfs; cull unnecessary icdinode fields Date: Tue, 12 Jan 2016 20:01:15 +1100 X-ASG-Orig-Subj: [PATCH 3/8] xfs; cull unnecessary icdinode fields Message-Id: <1452589280-30167-4-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> References: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589291 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner Now that the struct xfs_icdinode is not directly related to the on-disk format, we can cull things in it we really don't need to store: - magic number never changes - padding is not necessary - next_unlinked is never used - inode number is redundant - uuid is redundant - lsn is accessed directly from dinode - inode CRC is only accessed directly from dinode Hence we can remove these from the struct xfs_icdinode and redirect the code that uses them to the xfs_dinode appripriately. This reduces the size of the struct icdinode from 152 bytes to 88 bytes, and removes a fair chunk of unnecessary code, too. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_inode_buf.c | 39 +++++++++++++-------------------------- fs/xfs/libxfs/xfs_inode_buf.h | 27 +++++++-------------------- fs/xfs/xfs_inode.c | 19 +------------------ fs/xfs/xfs_inode_item.c | 19 +++++++++++-------- 4 files changed, 32 insertions(+), 72 deletions(-) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index 0f33265..920aeef 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -202,7 +202,6 @@ xfs_inode_from_disk( struct xfs_icdinode *to = &ip->i_d; struct inode *inode = VFS_I(ip); - to->di_magic = be16_to_cpu(from->di_magic); to->di_mode = be16_to_cpu(from->di_mode); to->di_version = from ->di_version; to->di_format = from->di_format; @@ -212,7 +211,6 @@ xfs_inode_from_disk( to->di_nlink = be32_to_cpu(from->di_nlink); to->di_projid_lo = be16_to_cpu(from->di_projid_lo); to->di_projid_hi = be16_to_cpu(from->di_projid_hi); - memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad)); to->di_flushiter = be16_to_cpu(from->di_flushiter); inode->i_atime.tv_sec = be32_to_cpu(from->di_atime.t_sec); @@ -239,24 +237,22 @@ xfs_inode_from_disk( to->di_crtime.t_sec = be32_to_cpu(from->di_crtime.t_sec); to->di_crtime.t_nsec = be32_to_cpu(from->di_crtime.t_nsec); to->di_flags2 = be64_to_cpu(from->di_flags2); - to->di_ino = be64_to_cpu(from->di_ino); - to->di_lsn = be64_to_cpu(from->di_lsn); - memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2)); - uuid_copy(&to->di_uuid, &from->di_uuid); } } void xfs_inode_to_disk( struct xfs_inode *ip, - struct xfs_dinode *to) + struct xfs_dinode *to, + xfs_lsn_t lsn) { struct xfs_icdinode *from = &ip->i_d; struct inode *inode = VFS_I(ip); - to->di_magic = cpu_to_be16(from->di_magic); + to->di_magic = cpu_to_be16(XFS_DINODE_MAGIC); + to->di_mode = cpu_to_be16(from->di_mode); - to->di_version = from ->di_version; + to->di_version = from->di_version; to->di_format = from->di_format; to->di_onlink = cpu_to_be16(from->di_onlink); to->di_uid = cpu_to_be32(from->di_uid); @@ -264,8 +260,8 @@ xfs_inode_to_disk( to->di_nlink = cpu_to_be32(from->di_nlink); to->di_projid_lo = cpu_to_be16(from->di_projid_lo); to->di_projid_hi = cpu_to_be16(from->di_projid_hi); - memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad)); + memset(to->di_pad, 0, sizeof(to->di_pad)); to->di_atime.t_sec = cpu_to_be32(inode->i_atime.tv_sec); to->di_atime.t_nsec = cpu_to_be32(inode->i_atime.tv_nsec); to->di_mtime.t_sec = cpu_to_be32(inode->i_mtime.tv_sec); @@ -290,10 +286,11 @@ xfs_inode_to_disk( to->di_crtime.t_sec = cpu_to_be32(from->di_crtime.t_sec); to->di_crtime.t_nsec = cpu_to_be32(from->di_crtime.t_nsec); to->di_flags2 = cpu_to_be64(from->di_flags2); - to->di_ino = cpu_to_be64(from->di_ino); - to->di_lsn = cpu_to_be64(from->di_lsn); - memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2)); - uuid_copy(&to->di_uuid, &from->di_uuid); + + to->di_ino = cpu_to_be64(ip->i_ino); + to->di_lsn = cpu_to_be64(lsn); + memset(to->di_pad2, 0, sizeof(to->di_pad2)); + uuid_copy(&to->di_uuid, &ip->i_mount->m_sb.sb_meta_uuid); to->di_flushiter = 0; } else { to->di_flushiter = cpu_to_be16(from->di_flushiter); @@ -428,13 +425,10 @@ xfs_iread( !(mp->m_flags & XFS_MOUNT_IKEEP)) { /* initialise the on-disk inode core */ memset(&ip->i_d, 0, sizeof(ip->i_d)); - ip->i_d.di_magic = XFS_DINODE_MAGIC; ip->i_d.di_gen = prandom_u32(); - if (xfs_sb_version_hascrc(&mp->m_sb)) { + if (xfs_sb_version_hascrc(&mp->m_sb)) ip->i_d.di_version = 3; - ip->i_d.di_ino = ip->i_ino; - uuid_copy(&ip->i_d.di_uuid, &mp->m_sb.sb_meta_uuid); - } else + else ip->i_d.di_version = 2; return 0; } @@ -478,16 +472,10 @@ xfs_iread( * Partial initialisation of the in-core inode. Just the bits * that xfs_ialloc won't overwrite or relies on being correct. */ - ip->i_d.di_magic = be16_to_cpu(dip->di_magic); ip->i_d.di_version = dip->di_version; ip->i_d.di_gen = be32_to_cpu(dip->di_gen); ip->i_d.di_flushiter = be16_to_cpu(dip->di_flushiter); - if (dip->di_version == 3) { - ip->i_d.di_ino = be64_to_cpu(dip->di_ino); - uuid_copy(&ip->i_d.di_uuid, &dip->di_uuid); - } - /* * Make sure to pull in the mode here as well in * case the inode is released without being used. @@ -508,7 +496,6 @@ xfs_iread( */ if (ip->i_d.di_version == 1) { ip->i_d.di_version = 2; - memset(&(ip->i_d.di_pad[0]), 0, sizeof(ip->i_d.di_pad)); ip->i_d.di_nlink = ip->i_d.di_onlink; ip->i_d.di_onlink = 0; xfs_set_projid(ip, 0); diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h index adcc9bf..69d626e 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.h +++ b/fs/xfs/libxfs/xfs_inode_buf.h @@ -22,24 +22,22 @@ struct xfs_inode; struct xfs_dinode; /* - * In memory representation of the XFS inode. This is held in the in-core - * struct xfs_inode to represent the on disk values, but no longer needs to be - * identical to the on-disk structure as it is always translated to on-disk - * format specific structures at the appropriate time. + * In memory representation of the XFS inode. This is held in the in-core struct + * xfs_inode to represent the on disk values, but it's struct is in no way + * related to what is stored on disk. That is, this structure is always + * translated to on-disk format specific structures at the appropriate time. */ struct xfs_icdinode { - __uint16_t di_magic; /* inode magic # = XFS_DINODE_MAGIC */ __uint16_t di_mode; /* mode and type of file */ __int8_t di_version; /* inode version */ __int8_t di_format; /* format of di_c data */ __uint16_t di_onlink; /* old number of links to file */ + __uint16_t di_flushiter; /* incremented on flush */ __uint32_t di_uid; /* owner's user id */ __uint32_t di_gid; /* owner's group id */ __uint32_t di_nlink; /* number of links to file */ __uint16_t di_projid_lo; /* lower part of owner's project id */ __uint16_t di_projid_hi; /* higher part of owner's project id */ - __uint8_t di_pad[6]; /* unused, zeroed space */ - __uint16_t di_flushiter; /* incremented on flush */ xfs_fsize_t di_size; /* number of bytes in file */ xfs_rfsblock_t di_nblocks; /* # of direct & btree blocks used */ xfs_extlen_t di_extsize; /* basic/minimum extent size for file */ @@ -52,22 +50,10 @@ struct xfs_icdinode { __uint16_t di_flags; /* random flags, XFS_DIFLAG_... */ __uint32_t di_gen; /* generation number */ - /* di_next_unlinked is the only non-core field in the old dinode */ - xfs_agino_t di_next_unlinked;/* agi unlinked list ptr */ - - /* start of the extended dinode, writable fields */ - __uint32_t di_crc; /* CRC of the inode */ __uint64_t di_changecount; /* number of attribute changes */ - xfs_lsn_t di_lsn; /* flush sequence */ __uint64_t di_flags2; /* more random flags */ - __uint8_t di_pad2[16]; /* more padding for future expansion */ - /* fields only written to during inode creation */ xfs_ictimestamp_t di_crtime; /* time created */ - xfs_ino_t di_ino; /* inode number */ - uuid_t di_uuid; /* UUID of the filesystem */ - - /* structure must be padded to 64 bit alignment */ }; /* @@ -86,7 +72,8 @@ int xfs_imap_to_bp(struct xfs_mount *, struct xfs_trans *, int xfs_iread(struct xfs_mount *, struct xfs_trans *, struct xfs_inode *, uint); void xfs_dinode_calc_crc(struct xfs_mount *, struct xfs_dinode *); -void xfs_inode_to_disk(struct xfs_inode *ip, struct xfs_dinode *to); +void xfs_inode_to_disk(struct xfs_inode *ip, struct xfs_dinode *to, + xfs_lsn_t lsn); void xfs_inode_from_disk(struct xfs_inode *ip, struct xfs_dinode *from); void xfs_log_dinode_to_disk(struct xfs_log_dinode *from, struct xfs_dinode *to); diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index 3e9b8a0..3f020b5 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -800,7 +800,6 @@ xfs_ialloc( ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid()); ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid()); xfs_set_projid(ip, prid); - memset(&(ip->i_d.di_pad[0]), 0, sizeof(ip->i_d.di_pad)); if (pip && XFS_INHERIT_GID(pip)) { ip->i_d.di_gid = pip->i_d.di_gid; @@ -838,13 +837,8 @@ xfs_ialloc( ip->i_d.di_flags = 0; if (ip->i_d.di_version == 3) { - ASSERT(ip->i_d.di_ino == ino); - ASSERT(uuid_equal(&ip->i_d.di_uuid, &mp->m_sb.sb_meta_uuid)); - ip->i_d.di_crc = 0; ip->i_d.di_changecount = 1; - ip->i_d.di_lsn = 0; ip->i_d.di_flags2 = 0; - memset(&(ip->i_d.di_pad2[0]), 0, sizeof(ip->i_d.di_pad2)); ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec; ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec; } @@ -3450,13 +3444,6 @@ xfs_iflush_int( __func__, ip->i_ino, be16_to_cpu(dip->di_magic), dip); goto corrupt_out; } - if (XFS_TEST_ERROR(ip->i_d.di_magic != XFS_DINODE_MAGIC, - mp, XFS_ERRTAG_IFLUSH_2, XFS_RANDOM_IFLUSH_2)) { - xfs_alert_tag(mp, XFS_PTAG_IFLUSH, - "%s: Bad inode %Lu, ptr 0x%p, magic number 0x%x", - __func__, ip->i_ino, ip, ip->i_d.di_magic); - goto corrupt_out; - } if (S_ISREG(ip->i_d.di_mode)) { if (XFS_TEST_ERROR( (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS) && @@ -3515,7 +3502,7 @@ xfs_iflush_int( * copy out the core of the inode, because if the inode is dirty at all * the core must be. */ - xfs_inode_to_disk(ip, dip); + xfs_inode_to_disk(ip, dip, iip->ili_item.li_lsn); /* Wrap, we never let the log put out DI_MAX_FLUSH */ if (ip->i_d.di_flushiter == DI_MAX_FLUSH) @@ -3567,10 +3554,6 @@ xfs_iflush_int( */ xfs_buf_attach_iodone(bp, xfs_iflush_done, &iip->ili_item); - /* update the lsn in the on disk inode if required */ - if (ip->i_d.di_version == 3) - dip->di_lsn = cpu_to_be64(iip->ili_item.li_lsn); - /* generate the checksum. */ xfs_dinode_calc_crc(mp, dip); diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index 9dcbf58..ae60087 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -325,12 +325,14 @@ xfs_inode_item_format_attr_fork( static void xfs_inode_to_log_dinode( struct xfs_inode *ip, - struct xfs_log_dinode *to) + struct xfs_log_dinode *to, + xfs_lsn_t lsn) { struct xfs_icdinode *from = &ip->i_d; struct inode *inode = VFS_I(ip); - to->di_magic = from->di_magic; + to->di_magic = XFS_DINODE_MAGIC; + to->di_mode = from->di_mode; to->di_version = from->di_version; to->di_format = from->di_format; @@ -340,8 +342,8 @@ xfs_inode_to_log_dinode( to->di_nlink = from->di_nlink; to->di_projid_lo = from->di_projid_lo; to->di_projid_hi = from->di_projid_hi; - memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad)); + memset(to->di_pad, 0, sizeof(to->di_pad)); to->di_atime.t_sec = inode->i_atime.tv_sec; to->di_atime.t_nsec = inode->i_atime.tv_nsec; to->di_mtime.t_sec = inode->i_mtime.tv_sec; @@ -366,10 +368,11 @@ xfs_inode_to_log_dinode( to->di_crtime.t_sec = from->di_crtime.t_sec; to->di_crtime.t_nsec = from->di_crtime.t_nsec; to->di_flags2 = from->di_flags2; - to->di_ino = from->di_ino; - to->di_lsn = from->di_lsn; - memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2)); - uuid_copy(&to->di_uuid, &from->di_uuid); + + to->di_ino = ip->i_ino; + to->di_lsn = lsn; + memset(to->di_pad2, 0, sizeof(to->di_pad2)); + uuid_copy(&to->di_uuid, &ip->i_mount->m_sb.sb_uuid); to->di_flushiter = 0; } else { to->di_flushiter = from->di_flushiter; @@ -390,7 +393,7 @@ xfs_inode_item_format_core( struct xfs_log_dinode *dic; dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE); - xfs_inode_to_log_dinode(ip, dic); + xfs_inode_to_log_dinode(ip, dic, ip->i_itemp->ili_item.li_lsn); xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version)); } -- 2.5.0 From dave@fromorbit.com Tue Jan 12 03:01:38 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0E7DA7F69 for ; Tue, 12 Jan 2016 03:01:38 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id F23868F8049 for ; Tue, 12 Jan 2016 01:01:37 -0800 (PST) X-ASG-Debug-ID: 1452589288-04cbb05f7c11dbe0003-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id q5e1AwzaewmbPiID for ; Tue, 12 Jan 2016 01:01:33 -0800 (PST) X-Barracuda-Envelope-From: dave@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DIDABTwJRWPGu7LXleKAECgw+BP4ZggXmffAaBZo80ihaBKk0BAQEBAQEHAQEBAUE/hDUBBRoNLzMIGDE5AwcUEAmILcAThg6KAkKEQAWHZI8vjzeHbYUxRI4Pgi4BCwE3AxyBcSo0hGeBSgEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 12 Jan 2016 19:31:25 +1030 Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment) by dastard with esmtp (Exim 4.80) (envelope-from ) id 1aIupP-0005K5-Ld for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 Received: from dave by disappointment with local (Exim 4.86) (envelope-from ) id 1aIupP-0008Ai-L3 for xfs@oss.sgi.com; Tue, 12 Jan 2016 20:01:23 +1100 From: Dave Chinner To: xfs@oss.sgi.com Subject: [PATCH 8/8] xfs: mode di_mode to vfs inode Date: Tue, 12 Jan 2016 20:01:20 +1100 X-ASG-Orig-Subj: [PATCH 8/8] xfs: mode di_mode to vfs inode Message-Id: <1452589280-30167-9-git-send-email-david@fromorbit.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> References: <1452589280-30167-1-git-send-email-david@fromorbit.com> X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452589293 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- From: Dave Chinner Move the di_mode value from the xfs_icdinode to the VFS inode, reducing the xfs_icdinode byte another 2 bytes and collapsing another 2 byte hole in the structure. Signed-off-by: Dave Chinner --- fs/xfs/libxfs/xfs_bmap.c | 6 +++--- fs/xfs/libxfs/xfs_dir2.c | 12 +++++------ fs/xfs/libxfs/xfs_inode_buf.c | 8 +++---- fs/xfs/libxfs/xfs_inode_buf.h | 1 - fs/xfs/libxfs/xfs_inode_fork.c | 2 +- fs/xfs/xfs_bmap_util.c | 4 ++-- fs/xfs/xfs_dir2_readdir.c | 2 +- fs/xfs/xfs_file.c | 6 +++--- fs/xfs/xfs_filestream.c | 4 ++-- fs/xfs/xfs_icache.c | 8 +++---- fs/xfs/xfs_inode.c | 48 ++++++++++++++++++++---------------------- fs/xfs/xfs_inode.h | 4 ++-- fs/xfs/xfs_inode_item.c | 2 +- fs/xfs/xfs_ioctl.c | 14 ++++++------ fs/xfs/xfs_iops.c | 12 ++++------- fs/xfs/xfs_itable.c | 2 +- fs/xfs/xfs_log_recover.c | 2 +- fs/xfs/xfs_mount.c | 2 +- 18 files changed, 66 insertions(+), 73 deletions(-) diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c index ef00156..6a05166 100644 --- a/fs/xfs/libxfs/xfs_bmap.c +++ b/fs/xfs/libxfs/xfs_bmap.c @@ -912,7 +912,7 @@ xfs_bmap_local_to_extents( * We don't want to deal with the case of keeping inode data inline yet. * So sending the data fork of a regular inode is invalid. */ - ASSERT(!(S_ISREG(ip->i_d.di_mode) && whichfork == XFS_DATA_FORK)); + ASSERT(!(S_ISREG(VFS_I(ip)->i_mode) && whichfork == XFS_DATA_FORK)); ifp = XFS_IFORK_PTR(ip, whichfork); ASSERT(XFS_IFORK_FORMAT(ip, whichfork) == XFS_DINODE_FMT_LOCAL); @@ -1079,7 +1079,7 @@ xfs_bmap_add_attrfork_local( if (ip->i_df.if_bytes <= XFS_IFORK_DSIZE(ip)) return 0; - if (S_ISDIR(ip->i_d.di_mode)) { + if (S_ISDIR(VFS_I(ip)->i_mode)) { memset(&dargs, 0, sizeof(dargs)); dargs.geo = ip->i_mount->m_dir_geo; dargs.dp = ip; @@ -1091,7 +1091,7 @@ xfs_bmap_add_attrfork_local( return xfs_dir2_sf_to_block(&dargs); } - if (S_ISLNK(ip->i_d.di_mode)) + if (S_ISLNK(VFS_I(ip)->i_mode)) return xfs_bmap_local_to_extents(tp, ip, firstblock, 1, flags, XFS_DATA_FORK, xfs_symlink_local_to_remote); diff --git a/fs/xfs/libxfs/xfs_dir2.c b/fs/xfs/libxfs/xfs_dir2.c index 2fb53a5..af0f9d1 100644 --- a/fs/xfs/libxfs/xfs_dir2.c +++ b/fs/xfs/libxfs/xfs_dir2.c @@ -176,7 +176,7 @@ xfs_dir_isempty( { xfs_dir2_sf_hdr_t *sfp; - ASSERT(S_ISDIR(dp->i_d.di_mode)); + ASSERT(S_ISDIR(VFS_I(dp)->i_mode)); if (dp->i_d.di_size == 0) /* might happen during shutdown. */ return 1; if (dp->i_d.di_size > XFS_IFORK_DSIZE(dp)) @@ -231,7 +231,7 @@ xfs_dir_init( struct xfs_da_args *args; int error; - ASSERT(S_ISDIR(dp->i_d.di_mode)); + ASSERT(S_ISDIR(VFS_I(dp)->i_mode)); error = xfs_dir_ino_validate(tp->t_mountp, pdp->i_ino); if (error) return error; @@ -266,7 +266,7 @@ xfs_dir_createname( int rval; int v; /* type-checking value */ - ASSERT(S_ISDIR(dp->i_d.di_mode)); + ASSERT(S_ISDIR(VFS_I(dp)->i_mode)); if (inum) { rval = xfs_dir_ino_validate(tp->t_mountp, inum); if (rval) @@ -364,7 +364,7 @@ xfs_dir_lookup( int v; /* type-checking value */ int lock_mode; - ASSERT(S_ISDIR(dp->i_d.di_mode)); + ASSERT(S_ISDIR(VFS_I(dp)->i_mode)); XFS_STATS_INC(dp->i_mount, xs_dir_lookup); /* @@ -443,7 +443,7 @@ xfs_dir_removename( int rval; int v; /* type-checking value */ - ASSERT(S_ISDIR(dp->i_d.di_mode)); + ASSERT(S_ISDIR(VFS_I(dp)->i_mode)); XFS_STATS_INC(dp->i_mount, xs_dir_remove); args = kmem_zalloc(sizeof(*args), KM_SLEEP | KM_NOFS); @@ -505,7 +505,7 @@ xfs_dir_replace( int rval; int v; /* type-checking value */ - ASSERT(S_ISDIR(dp->i_d.di_mode)); + ASSERT(S_ISDIR(VFS_I(dp)->i_mode)); rval = xfs_dir_ino_validate(tp->t_mountp, inum); if (rval) diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c index a2b10e3..763eb1f 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.c +++ b/fs/xfs/libxfs/xfs_inode_buf.c @@ -202,13 +202,12 @@ xfs_inode_from_disk( struct xfs_icdinode *to = &ip->i_d; struct inode *inode = VFS_I(ip); - to->di_mode = be16_to_cpu(from->di_mode); - to->di_version = from ->di_version; /* * Convert v1 inodes immediately to v2 inode format as this is the * minimum inode version format we support in the rest of the code. */ + to->di_version = from->di_version; if (to->di_version == 1) { set_nlink(inode, be16_to_cpu(from->di_onlink)); to->di_projid_lo = 0; @@ -232,6 +231,7 @@ xfs_inode_from_disk( inode->i_ctime.tv_sec = be32_to_cpu(from->di_ctime.t_sec); inode->i_ctime.tv_nsec = be32_to_cpu(from->di_ctime.t_nsec); inode->i_generation = be32_to_cpu(from->di_gen); + inode->i_mode = be16_to_cpu(from->di_mode); to->di_size = be64_to_cpu(from->di_size); to->di_nblocks = be64_to_cpu(from->di_nblocks); @@ -264,7 +264,6 @@ xfs_inode_to_disk( to->di_magic = cpu_to_be16(XFS_DINODE_MAGIC); to->di_onlink = 0; - to->di_mode = cpu_to_be16(from->di_mode); to->di_version = from->di_version; to->di_format = from->di_format; to->di_uid = cpu_to_be32(from->di_uid); @@ -281,6 +280,7 @@ xfs_inode_to_disk( to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec); to->di_nlink = cpu_to_be32(inode->i_nlink); to->di_gen = cpu_to_be32(inode->i_generation); + to->di_mode = cpu_to_be16(inode->i_mode); to->di_size = cpu_to_be64(from->di_size); to->di_nblocks = cpu_to_be64(from->di_nblocks); @@ -495,7 +495,7 @@ xfs_iread( * the inode is already free and not try to mess * with the uninitialized part of it. */ - ip->i_d.di_mode = 0; + VFS_I(ip)->i_mode = 0; } ASSERT(ip->i_d.di_version >= 2); diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h index e49c229..5f349a2 100644 --- a/fs/xfs/libxfs/xfs_inode_buf.h +++ b/fs/xfs/libxfs/xfs_inode_buf.h @@ -28,7 +28,6 @@ struct xfs_dinode; * translated to on-disk format specific structures at the appropriate time. */ struct xfs_icdinode { - __uint16_t di_mode; /* mode and type of file */ __int8_t di_version; /* inode version */ __int8_t di_format; /* format of di_c data */ __uint16_t di_flushiter; /* incremented on flush */ diff --git a/fs/xfs/libxfs/xfs_inode_fork.c b/fs/xfs/libxfs/xfs_inode_fork.c index 0defbd0..0bf1c74 100644 --- a/fs/xfs/libxfs/xfs_inode_fork.c +++ b/fs/xfs/libxfs/xfs_inode_fork.c @@ -120,7 +120,7 @@ xfs_iformat_fork( return -EFSCORRUPTED; } - switch (ip->i_d.di_mode & S_IFMT) { + switch (VFS_I(ip)->i_mode & S_IFMT) { case S_IFIFO: case S_IFCHR: case S_IFBLK: diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c index 45ec9e4..7087756 100644 --- a/fs/xfs/xfs_bmap_util.c +++ b/fs/xfs/xfs_bmap_util.c @@ -821,7 +821,7 @@ bool xfs_can_free_eofblocks(struct xfs_inode *ip, bool force) { /* prealloc/delalloc exists only on regular files */ - if (!S_ISREG(ip->i_d.di_mode)) + if (!S_ISREG(VFS_I(ip)->i_mode)) return false; /* @@ -1726,7 +1726,7 @@ xfs_swap_extents( xfs_lock_two_inodes(ip, tip, XFS_MMAPLOCK_EXCL); /* Verify that both files have the same format */ - if ((ip->i_d.di_mode & S_IFMT) != (tip->i_d.di_mode & S_IFMT)) { + if ((VFS_I(ip)->i_mode & S_IFMT) != (VFS_I(tip)->i_mode & S_IFMT)) { error = -EINVAL; goto out_unlock; } diff --git a/fs/xfs/xfs_dir2_readdir.c b/fs/xfs/xfs_dir2_readdir.c index 642d55d..93b3ab0 100644 --- a/fs/xfs/xfs_dir2_readdir.c +++ b/fs/xfs/xfs_dir2_readdir.c @@ -665,7 +665,7 @@ xfs_readdir( if (XFS_FORCED_SHUTDOWN(dp->i_mount)) return -EIO; - ASSERT(S_ISDIR(dp->i_d.di_mode)); + ASSERT(S_ISDIR(VFS_I(dp)->i_mode)); XFS_STATS_INC(dp->i_mount, xs_dir_getdents); args.dp = dp; diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index ebe9b82..20eb6e5 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -156,9 +156,9 @@ xfs_update_prealloc_flags( xfs_trans_ijoin(tp, ip, XFS_ILOCK_EXCL); if (!(flags & XFS_PREALLOC_INVISIBLE)) { - ip->i_d.di_mode &= ~S_ISUID; - if (ip->i_d.di_mode & S_IXGRP) - ip->i_d.di_mode &= ~S_ISGID; + VFS_I(ip)->i_mode &= ~S_ISUID; + if (VFS_I(ip)->i_mode & S_IXGRP) + VFS_I(ip)->i_mode &= ~S_ISGID; xfs_trans_ichgtime(tp, ip, XFS_ICHGTIME_MOD | XFS_ICHGTIME_CHG); } diff --git a/fs/xfs/xfs_filestream.c b/fs/xfs/xfs_filestream.c index c4c130f..a51353a 100644 --- a/fs/xfs/xfs_filestream.c +++ b/fs/xfs/xfs_filestream.c @@ -151,7 +151,7 @@ xfs_filestream_pick_ag( xfs_agnumber_t ag, max_ag = NULLAGNUMBER; int err, trylock, nscan; - ASSERT(S_ISDIR(ip->i_d.di_mode)); + ASSERT(S_ISDIR(VFS_I(ip)->i_mode)); /* 2% of an AG's blocks must be free for it to be chosen. */ minfree = mp->m_sb.sb_agblocks / 50; @@ -319,7 +319,7 @@ xfs_filestream_lookup_ag( xfs_agnumber_t startag, ag = NULLAGNUMBER; struct xfs_mru_cache_elem *mru; - ASSERT(S_ISREG(ip->i_d.di_mode)); + ASSERT(S_ISREG(VFS_I(ip)->i_mode)); pip = xfs_filestream_get_parent(ip); if (!pip) diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index 7c26f86..437cb2e 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -98,7 +98,7 @@ void xfs_inode_free( struct xfs_inode *ip) { - switch (ip->i_d.di_mode & S_IFMT) { + switch (VFS_I(ip)->i_mode & S_IFMT) { case S_IFREG: case S_IFDIR: case S_IFLNK: @@ -185,7 +185,7 @@ xfs_iget_cache_hit( /* * If lookup is racing with unlink return an error immediately. */ - if (ip->i_d.di_mode == 0 && !(flags & XFS_IGET_CREATE)) { + if (VFS_I(ip)->i_mode == 0 && !(flags & XFS_IGET_CREATE)) { error = -ENOENT; goto out_error; } @@ -295,7 +295,7 @@ xfs_iget_cache_miss( trace_xfs_iget_miss(ip); - if ((ip->i_d.di_mode == 0) && !(flags & XFS_IGET_CREATE)) { + if ((VFS_I(ip)->i_mode == 0) && !(flags & XFS_IGET_CREATE)) { error = -ENOENT; goto out_destroy; } @@ -444,7 +444,7 @@ again: * If we have a real type for an on-disk inode, we can setup the inode * now. If it's a new inode being created, xfs_ialloc will handle it. */ - if (xfs_iflags_test(ip, XFS_INEW) && ip->i_d.di_mode != 0) + if (xfs_iflags_test(ip, XFS_INEW) && VFS_I(ip)->i_mode != 0) xfs_setup_existing_inode(ip); return 0; diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index f0b9ccb..1a71633 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -793,7 +793,7 @@ xfs_ialloc( if (ip->i_d.di_version == 1) ip->i_d.di_version = 2; - ip->i_d.di_mode = mode; + inode->i_mode = mode; set_nlink(inode, nlink); ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid()); ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid()); @@ -801,9 +801,8 @@ xfs_ialloc( if (pip && XFS_INHERIT_GID(pip)) { ip->i_d.di_gid = pip->i_d.di_gid; - if ((pip->i_d.di_mode & S_ISGID) && S_ISDIR(mode)) { - ip->i_d.di_mode |= S_ISGID; - } + if ((VFS_I(pip)->i_mode & S_ISGID) && S_ISDIR(mode)) + inode->i_mode |= S_ISGID; } /* @@ -812,10 +811,9 @@ xfs_ialloc( * (and only if the irix_sgid_inherit compatibility variable is set). */ if ((irix_sgid_inherit) && - (ip->i_d.di_mode & S_ISGID) && - (!in_group_p(xfs_gid_to_kgid(ip->i_d.di_gid)))) { - ip->i_d.di_mode &= ~S_ISGID; - } + (inode->i_mode & S_ISGID) && + (!in_group_p(xfs_gid_to_kgid(ip->i_d.di_gid)))) + inode->i_mode &= ~S_ISGID; ip->i_d.di_size = 0; ip->i_d.di_nextents = 0; @@ -1416,7 +1414,7 @@ xfs_link( trace_xfs_link(tdp, target_name); - ASSERT(!S_ISDIR(sip->i_d.di_mode)); + ASSERT(!S_ISDIR(VFS_I(sip)->i_mode)); if (XFS_FORCED_SHUTDOWN(mp)) return -EIO; @@ -1623,7 +1621,7 @@ xfs_release( xfs_mount_t *mp = ip->i_mount; int error; - if (!S_ISREG(ip->i_d.di_mode) || (ip->i_d.di_mode == 0)) + if (!S_ISREG(VFS_I(ip)->i_mode) || (VFS_I(ip)->i_mode == 0)) return 0; /* If this is a read-only mount, don't do this (would generate I/O) */ @@ -1858,7 +1856,7 @@ xfs_inactive( * If the inode is already free, then there can be nothing * to clean up here. */ - if (ip->i_d.di_mode == 0) { + if (VFS_I(ip)->i_mode == 0) { ASSERT(ip->i_df.if_real_bytes == 0); ASSERT(ip->i_df.if_broot_bytes == 0); return; @@ -1882,7 +1880,7 @@ xfs_inactive( return; } - if (S_ISREG(ip->i_d.di_mode) && + if (S_ISREG(VFS_I(ip)->i_mode) && (ip->i_d.di_size != 0 || XFS_ISIZE(ip) != 0 || ip->i_d.di_nextents > 0 || ip->i_delayed_blks > 0)) truncate = 1; @@ -1891,7 +1889,7 @@ xfs_inactive( if (error) return; - if (S_ISLNK(ip->i_d.di_mode)) + if (S_ISLNK(VFS_I(ip)->i_mode)) error = xfs_inactive_symlink(ip); else if (truncate) error = xfs_inactive_truncate(ip); @@ -1947,7 +1945,7 @@ xfs_iunlink( int error; ASSERT(VFS_I(ip)->i_nlink == 0); - ASSERT(ip->i_d.di_mode != 0); + ASSERT(VFS_I(ip)->i_mode != 0); mp = tp->t_mountp; @@ -2390,7 +2388,7 @@ xfs_ifree( ASSERT(VFS_I(ip)->i_nlink == 0); ASSERT(ip->i_d.di_nextents == 0); ASSERT(ip->i_d.di_anextents == 0); - ASSERT(ip->i_d.di_size == 0 || !S_ISREG(ip->i_d.di_mode)); + ASSERT(ip->i_d.di_size == 0 || !S_ISREG(VFS_I(ip)->i_mode)); ASSERT(ip->i_d.di_nblocks == 0); /* @@ -2404,7 +2402,7 @@ xfs_ifree( if (error) return error; - ip->i_d.di_mode = 0; /* mark incore inode as free */ + VFS_I(ip)->i_mode = 0; /* mark incore inode as free */ ip->i_d.di_flags = 0; ip->i_d.di_dmevmask = 0; ip->i_d.di_forkoff = 0; /* mark the attr fork not in use */ @@ -2501,7 +2499,7 @@ xfs_remove( { xfs_mount_t *mp = dp->i_mount; xfs_trans_t *tp = NULL; - int is_dir = S_ISDIR(ip->i_d.di_mode); + int is_dir = S_ISDIR(VFS_I(ip)->i_mode); int error = 0; xfs_bmap_free_t free_list; xfs_fsblock_t first_block; @@ -2746,7 +2744,7 @@ xfs_cross_rename( if (dp1 != dp2) { dp2_flags = XFS_ICHGTIME_MOD | XFS_ICHGTIME_CHG; - if (S_ISDIR(ip2->i_d.di_mode)) { + if (S_ISDIR(VFS_I(ip2)->i_mode)) { error = xfs_dir_replace(tp, ip2, &xfs_name_dotdot, dp1->i_ino, first_block, free_list, spaceres); @@ -2754,7 +2752,7 @@ xfs_cross_rename( goto out_trans_abort; /* transfer ip2 ".." reference to dp1 */ - if (!S_ISDIR(ip1->i_d.di_mode)) { + if (!S_ISDIR(VFS_I(ip1)->i_mode)) { error = xfs_droplink(tp, dp2); if (error) goto out_trans_abort; @@ -2773,7 +2771,7 @@ xfs_cross_rename( ip2_flags |= XFS_ICHGTIME_MOD | XFS_ICHGTIME_CHG; } - if (S_ISDIR(ip1->i_d.di_mode)) { + if (S_ISDIR(VFS_I(ip1)->i_mode)) { error = xfs_dir_replace(tp, ip1, &xfs_name_dotdot, dp2->i_ino, first_block, free_list, spaceres); @@ -2781,7 +2779,7 @@ xfs_cross_rename( goto out_trans_abort; /* transfer ip1 ".." reference to dp2 */ - if (!S_ISDIR(ip2->i_d.di_mode)) { + if (!S_ISDIR(VFS_I(ip2)->i_mode)) { error = xfs_droplink(tp, dp1); if (error) goto out_trans_abort; @@ -2878,7 +2876,7 @@ xfs_rename( struct xfs_inode *inodes[__XFS_SORT_INODES]; int num_inodes = __XFS_SORT_INODES; bool new_parent = (src_dp != target_dp); - bool src_is_directory = S_ISDIR(src_ip->i_d.di_mode); + bool src_is_directory = S_ISDIR(VFS_I(src_ip)->i_mode); int spaceres; int error; @@ -3007,7 +3005,7 @@ xfs_rename( * target and source are directories and that target can be * destroyed, or that neither is a directory. */ - if (S_ISDIR(target_ip->i_d.di_mode)) { + if (S_ISDIR(VFS_I(target_ip)->i_mode)) { /* * Make sure target dir is empty. */ @@ -3437,7 +3435,7 @@ xfs_iflush_int( __func__, ip->i_ino, be16_to_cpu(dip->di_magic), dip); goto corrupt_out; } - if (S_ISREG(ip->i_d.di_mode)) { + if (S_ISREG(VFS_I(ip)->i_mode)) { if (XFS_TEST_ERROR( (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS) && (ip->i_d.di_format != XFS_DINODE_FMT_BTREE), @@ -3447,7 +3445,7 @@ xfs_iflush_int( __func__, ip->i_ino, ip); goto corrupt_out; } - } else if (S_ISDIR(ip->i_d.di_mode)) { + } else if (S_ISDIR(VFS_I(ip)->i_mode)) { if (XFS_TEST_ERROR( (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS) && (ip->i_d.di_format != XFS_DINODE_FMT_BTREE) && diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h index aef5452..447a7fd 100644 --- a/fs/xfs/xfs_inode.h +++ b/fs/xfs/xfs_inode.h @@ -88,7 +88,7 @@ static inline struct inode *VFS_I(struct xfs_inode *ip) */ static inline xfs_fsize_t XFS_ISIZE(struct xfs_inode *ip) { - if (S_ISREG(ip->i_d.di_mode)) + if (S_ISREG(VFS_I(ip)->i_mode)) return i_size_read(VFS_I(ip)); return ip->i_d.di_size; } @@ -369,7 +369,7 @@ static inline int xfs_isiflocked(struct xfs_inode *ip) */ #define XFS_INHERIT_GID(pip) \ (((pip)->i_mount->m_flags & XFS_MOUNT_GRPID) || \ - ((pip)->i_d.di_mode & S_ISGID)) + (VFS_I(pip)->i_mode & S_ISGID)) int xfs_release(struct xfs_inode *ip); void xfs_inactive(struct xfs_inode *ip); diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c index 334d657..bd9808f 100644 --- a/fs/xfs/xfs_inode_item.c +++ b/fs/xfs/xfs_inode_item.c @@ -333,7 +333,6 @@ xfs_inode_to_log_dinode( to->di_magic = XFS_DINODE_MAGIC; - to->di_mode = from->di_mode; to->di_version = from->di_version; to->di_format = from->di_format; to->di_uid = from->di_uid; @@ -351,6 +350,7 @@ xfs_inode_to_log_dinode( to->di_ctime.t_nsec = inode->i_ctime.tv_nsec; to->di_nlink = inode->i_nlink; to->di_gen = inode->i_generation; + to->di_mode = inode->i_mode; to->di_size = from->di_size; to->di_nblocks = from->di_nblocks; diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index 2c40c5f..5069bd8 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -962,7 +962,7 @@ xfs_set_diflags( di_flags |= XFS_DIFLAG_NODEFRAG; if (xflags & XFS_XFLAG_FILESTREAM) di_flags |= XFS_DIFLAG_FILESTREAM; - if (S_ISDIR(ip->i_d.di_mode)) { + if (S_ISDIR(VFS_I(ip)->i_mode)) { if (xflags & XFS_XFLAG_RTINHERIT) di_flags |= XFS_DIFLAG_RTINHERIT; if (xflags & XFS_XFLAG_NOSYMLINKS) @@ -971,7 +971,7 @@ xfs_set_diflags( di_flags |= XFS_DIFLAG_EXTSZINHERIT; if (xflags & XFS_XFLAG_PROJINHERIT) di_flags |= XFS_DIFLAG_PROJINHERIT; - } else if (S_ISREG(ip->i_d.di_mode)) { + } else if (S_ISREG(VFS_I(ip)->i_mode)) { if (xflags & XFS_XFLAG_REALTIME) di_flags |= XFS_DIFLAG_REALTIME; if (xflags & XFS_XFLAG_EXTSIZE) @@ -1112,14 +1112,14 @@ xfs_ioctl_setattr_check_extsize( { struct xfs_mount *mp = ip->i_mount; - if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) && !S_ISREG(ip->i_d.di_mode)) + if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) && !S_ISREG(VFS_I(ip)->i_mode)) return -EINVAL; if ((fa->fsx_xflags & XFS_XFLAG_EXTSZINHERIT) && - !S_ISDIR(ip->i_d.di_mode)) + !S_ISDIR(VFS_I(ip)->i_mode)) return -EINVAL; - if (S_ISREG(ip->i_d.di_mode) && ip->i_d.di_nextents && + if (S_ISREG(VFS_I(ip)->i_mode) && ip->i_d.di_nextents && ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) != fa->fsx_extsize)) return -EINVAL; @@ -1240,9 +1240,9 @@ xfs_ioctl_setattr( * successful return from chown() */ - if ((ip->i_d.di_mode & (S_ISUID|S_ISGID)) && + if ((VFS_I(ip)->i_mode & (S_ISUID|S_ISGID)) && !capable_wrt_inode_uidgid(VFS_I(ip), CAP_FSETID)) - ip->i_d.di_mode &= ~(S_ISUID|S_ISGID); + VFS_I(ip)->i_mode &= ~(S_ISUID|S_ISGID); /* Change the ownerships and register project quota modifications */ if (xfs_get_projid(ip) != fa->fsx_projid) { diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c index 397ce85..34bff1e 100644 --- a/fs/xfs/xfs_iops.c +++ b/fs/xfs/xfs_iops.c @@ -454,7 +454,7 @@ xfs_vn_getattr( stat->size = XFS_ISIZE(ip); stat->dev = inode->i_sb->s_dev; - stat->mode = ip->i_d.di_mode; + stat->mode = inode->i_mode; stat->nlink = inode->i_nlink; stat->uid = inode->i_uid; stat->gid = inode->i_gid; @@ -501,9 +501,6 @@ xfs_setattr_mode( ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL)); - ip->i_d.di_mode &= S_IFMT; - ip->i_d.di_mode |= mode & ~S_IFMT; - inode->i_mode &= S_IFMT; inode->i_mode |= mode & ~S_IFMT; } @@ -647,9 +644,9 @@ xfs_setattr_nonsize( * The set-user-ID and set-group-ID bits of a file will be * cleared upon successful return from chown() */ - if ((ip->i_d.di_mode & (S_ISUID|S_ISGID)) && + if ((inode->i_mode & (S_ISUID|S_ISGID)) && !capable(CAP_FSETID)) - ip->i_d.di_mode &= ~(S_ISUID|S_ISGID); + inode->i_mode &= ~(S_ISUID|S_ISGID); /* * Change the ownerships and register quota modifications @@ -759,7 +756,7 @@ xfs_setattr_size( ASSERT(xfs_isilocked(ip, XFS_IOLOCK_EXCL)); ASSERT(xfs_isilocked(ip, XFS_MMAPLOCK_EXCL)); - ASSERT(S_ISREG(ip->i_d.di_mode)); + ASSERT(S_ISREG(inode->i_mode)); ASSERT((iattr->ia_valid & (ATTR_UID|ATTR_GID|ATTR_ATIME|ATTR_ATIME_SET| ATTR_MTIME_SET|ATTR_KILL_PRIV|ATTR_TIMES_SET)) == 0); @@ -1211,7 +1208,6 @@ xfs_setup_inode( /* make the inode look hashed for the writeback code */ hlist_add_fake(&inode->i_hash); - inode->i_mode = ip->i_d.di_mode; inode->i_uid = xfs_uid_to_kuid(ip->i_d.di_uid); inode->i_gid = xfs_gid_to_kgid(ip->i_d.di_gid); diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c index 6162e65..ce73eb3 100644 --- a/fs/xfs/xfs_itable.c +++ b/fs/xfs/xfs_itable.c @@ -88,7 +88,6 @@ xfs_bulkstat_one_int( buf->bs_projid_lo = dic->di_projid_lo; buf->bs_projid_hi = dic->di_projid_hi; buf->bs_ino = ino; - buf->bs_mode = dic->di_mode; buf->bs_uid = dic->di_uid; buf->bs_gid = dic->di_gid; buf->bs_size = dic->di_size; @@ -101,6 +100,7 @@ xfs_bulkstat_one_int( buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec; buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec; buf->bs_gen = inode->i_generation; + buf->bs_mode = inode->i_mode; buf->bs_xflags = xfs_ip2xflags(ip); buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog; diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 611c25c..bd6f23b 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -4338,7 +4338,7 @@ xlog_recover_process_one_iunlink( goto fail_iput; ASSERT(VFS_I(ip)->i_nlink == 0); - ASSERT(ip->i_d.di_mode != 0); + ASSERT(VFS_I(ip)->i_mode != 0); /* setup for the next pass */ agino = be32_to_cpu(dip->di_next_unlinked); diff --git a/fs/xfs/xfs_mount.c b/fs/xfs/xfs_mount.c index bb753b3..d306105 100644 --- a/fs/xfs/xfs_mount.c +++ b/fs/xfs/xfs_mount.c @@ -865,7 +865,7 @@ xfs_mountfs( ASSERT(rip != NULL); - if (unlikely(!S_ISDIR(rip->i_d.di_mode))) { + if (unlikely(!S_ISDIR(VFS_I(rip)->i_mode))) { xfs_warn(mp, "corrupted root inode %llu: not a directory", (unsigned long long)rip->i_ino); xfs_iunlock(rip, XFS_ILOCK_EXCL); -- 2.5.0 From darrick.wong@oracle.com Tue Jan 12 03:15:14 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id A83C17F47 for ; Tue, 12 Jan 2016 03:15:14 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 7B64330405F for ; Tue, 12 Jan 2016 01:15:14 -0800 (PST) X-ASG-Debug-ID: 1452590109-04bdf06f6b90330001-NocioJ Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id Tz1P4xW6hy0jit5j (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 01:15:10 -0800 (PST) X-Barracuda-Envelope-From: darrick.wong@oracle.com X-Barracuda-Apparent-Source-IP: 156.151.31.81 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0C9EZ2M009798 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 12 Jan 2016 09:14:36 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0C9EZVC010716 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 12 Jan 2016 09:14:35 GMT Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15]) by userv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0C9EYnv015869; Tue, 12 Jan 2016 09:14:34 GMT Received: from localhost (/24.130.224.127) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 12 Jan 2016 01:14:34 -0800 Date: Tue, 12 Jan 2016 01:14:32 -0800 From: "Darrick J. Wong" To: Eric Biggers Cc: david@fromorbit.com, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, xfs@oss.sgi.com, linux-btrfs@vger.kernel.org, hch@infradead.org Subject: Re: [PATCH 8/9] vfs: hoist the btrfs deduplication ioctl to the vfs Message-ID: <20160112091432.GB7832@birch.djwong.org> X-ASG-Orig-Subj: Re: [PATCH 8/9] vfs: hoist the btrfs deduplication ioctl to the vfs References: <20151219085505.12478.71157.stgit@birch.djwong.org> <20151219085559.12478.33700.stgit@birch.djwong.org> <20160112060714.GA4980@zzz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160112060714.GA4980@zzz> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81] X-Barracuda-Start-Time: 1452590110 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26055 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 UNPARSEABLE_RELAY Informational: message has unparseable relay lines [adding btrfs to the cc since we're talking about a whole new dedupe interface] On Tue, Jan 12, 2016 at 12:07:14AM -0600, Eric Biggers wrote: > Some feedback on the VFS portion of the FIDEDUPERANGE ioctl and its man page... > (note: I realize the patch is mostly just moving the code that already existed > in btrfs, but in the VFS it deserves a more thorough review): Wheee. :) Yes, let's discuss the concerns about the btrfs extent same ioctl. I believe Christoph dislikes about the odd return mechanism (i.e. status and bytes_deduped) and doubts that the vectorization is really necessary. There's not a lot of documentation to go on aside from "Do whatever the BTRFS ioctl does". I suspect that will leave my explanations lackng, since I neither designed the btrfs interface nor know all that much about the decisions made to arrive at what we have now. (I agree with both of hch's complaints.) Really, the best argument for keeping this ioctl is to avoid breaking duperemove. Even then, given that current duperemove checks for btrfs before trying to use BTRFS_IOC_EXTENT_SAME, we could very well design a new dedupe ioctl for the VFS, hook the new dedupers (XFS) into the new VFS ioctl leaving the old btrfs ioctl intact, and train duperemove to try the new ioctl and fall back on the btrfs one if the VFS ioctl isn't supported. Frankly, I also wouldn't mind changing the VFS dedupe ioctl that to something that resembles the clone_range interface: int ioctl(int dest_fd, FIDEDUPERANGE, struct file_dedupe_range * arg); struct file_dedupe_range { __s64 src_fd; __u64 src_offset; __u64 length; __u64 dest_offset; __u64 flags; }; "See if the byte range src_offset:length in src_fd matches all of dest_offset:length in dest_fd; if so, share src_fd's physical storage with dest_fd. Both fds must be files; if they are the same file the ranges cannot overlap; src_fd must be readable; dest_fd must be writable or append-only. Offsets and lengths probably need to be block-aligned, but that is filesystem dependent." The error conditions would be superset of the ones we know about today. I'd return EOVERFLOW or something if length is longer than the FS wants to deal with. Now all the vectorization problems go away, and since it's a new VFS interface we can define everything from the start. Christoph, if this new interface solves your complaints I think I'd like to get started on the code/docs soon. > At high level, I am confused about what is meant by the "source" and > "destination" files. I understand that with more than two files, you > effectively have to choose one file to treat specially and dedupe with all > the other files (an NxN comparison isn't realistic). But with just two > files, a deduplication operation should be completely symmetric, should it > not? The end Not sure what you mean by 'symmetric', but in any case the convention seems to be that src_fd's storage is shared with dest_fd if there's a match. > result should be that the data is deduplicated, regardless of the order in > which I gave the file descriptors. So why is there some non-symmetric > behavior? There are several examples but one is that the VFS is checking > !S_ISREG() on the "source" file descriptor but not on the "destination" file > descriptor. The dedupe_range function pointer should only be supplied for regular files. > Another is that different permissions are required on the source versus on > the destination. If there are good reasons for the nonsymmetry then this > needs to be clearly explained in the man page; otherwise it may not be clear > what to use as the "source" and what to use as the "destination". > > It seems odd to be adding "copy" as a system call but then have "dedupe" and > "clone" as ioctls rather than system calls... it seems that they should all > be one or the other (at least, if we put aside the fact that the ioctls > already exist in btrfs). We can't put the clone ioctl aside; coreutils has already started using it. I'm not sure if clone_range or extent_same are all that popular, though. AFAIK duperemove is the only program using extent_same, and I don't know of anything using clone_range. (Well, xfs_io does...) > The range checking in clone_verify_area() appears incomplete. Someone could > provide len=UINT64_MAX and all the checks would still pass even though > 'pos+len' would overflow. Yeah... > Should the ioctl be interruptible? Right now it always goes through *all* > the 'struct file_dedupe_range_info's you passed in --- potentially up to > 65535 of them. There probably ought to be explicit signal checks, or we could just get rid of the vectorization entirely. :) > Why 'info->bytes_deduped += deduped' rather than 'info->bytes_deduped = > deduped'? 'bytes_deduped' is per file descriptor, not for the operation as a > whole. Right, because bytes_deduped is a part of file_dedup_range_info, not file_dedupe_range. (Note the bytes_deduped = 0 earlier in the function.) > What permissions do you need on the destination file descriptors? The man > page implies they must be open for writing and not appending. The > implementation differs: it requires FMODE_WRITE only for non-admin users, and > it doesn't check for O_APPEND at all. I think the result of an earlier discussion was that src_fd must be readable, and dest_fd must be writable or appendable. > The man page also says you get EPERM if "dest_fd is immutable" and ETXTBSY if > "one of the files is a swap file", which I don't see actually happening in > the implementation; it seems those error codes perhaps exist at all for this > ioctl but rather be left to open(..., O_WRONLY). Those could be hoisted to the VFS (from the XFS implementation), I think. > If the filesystem doesn't support deduplication, or I pass in a strange file > descriptor such as one for a named pipe, do I get EINVAL or EOPNOTSUPP? The > man page isn't clear. Should be EOPNOTSUPP if dest_fd isn't a regular file; EISDIR if either are directories; and EINVAL for any other kind of non-file fd. I suspect the clone* manpages don't make this too clear either. > Under what circumstances will 'bytes_deduped' differ from the count that was > passed in? btrfs/xfs will only compare the first 16MB. Not documented anywhere. :( > If short counts are allowed, what will be the 'status' be in that case: > FILE_DEDUP_RANGE_DIFFERS, FILE_DEDUPE_RANGE_SAME, or something else? One of those two. > Can data be deduped even if only a prefix of the data region matches? No. > The man page doesn't mention FILE_DEDUPE_RANGE_SAME at all, instead calling it > 0; it only mentions FILE_DEDUPE_RANGE_DIFFERS. Oops, good catch. :( > The man page isn't clear about whether the ioctl stops early if an error > occurs or always processes all the 'struct file_dedupe_range_info's you pass > in. And if it were, hypothetically, to stop early, how is the user meant to > know on which file it stopped? I don't know if this should be the official behavior, but it stopped at whichever file_dedupe_range_info has both status and bytes_deduped set to zero. > The man page says "logical_offset" but in the struct it is called > "dest_offset". Oops. > There are some variables named "same" which don't really make sense now that > the ioctl is called FIDEDUPERANGE instead of EXTENT_SAME. Perhaps not. I'll later take a look at how many of these issues apply to clone/clone_range. --D > > Eric > -- > To unsubscribe from this list: send the line "unsubscribe linux-api" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html From jack@suse.cz Tue Jan 12 03:34:58 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id BDF3B7F37 for ; Tue, 12 Jan 2016 03:34:58 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id AC71C8F8035 for ; Tue, 12 Jan 2016 01:34:58 -0800 (PST) X-ASG-Debug-ID: 1452591295-04cbb05f7c11ed90001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 9EYBX6Wnvn5WXGhd (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 12 Jan 2016 01:34:56 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 05D26AC67; Tue, 12 Jan 2016 09:34:49 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id F336B823D8; Tue, 12 Jan 2016 10:34:58 +0100 (CET) Date: Tue, 12 Jan 2016 10:34:58 +0100 From: Jan Kara To: Ross Zwisler Cc: linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH v8 1/9] dax: fix NULL pointer dereference in __dax_dbg() Message-ID: <20160112093458.GR6262@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v8 1/9] dax: fix NULL pointer dereference in __dax_dbg() References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> <1452230879-18117-2-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452230879-18117-2-git-send-email-ross.zwisler@linux.intel.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452591296 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26056 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Thu 07-01-16 22:27:51, Ross Zwisler wrote: > In __dax_pmd_fault() we currently assume that get_block() will always set > bh.b_bdev and we unconditionally dereference it in __dax_dbg(). This > assumption isn't always true - when called for reads of holes > ext4_dax_mmap_get_block() returns a buffer head where bh->b_bdev is never > set. I hit this BUG while testing the DAX PMD fault path. > > Instead, initialize bh.b_bdev before passing bh into get_block(). It is > possible that the filesystem's get_block() will update bh.b_bdev, and this > is fine - we just want to initialize bh.b_bdev to something reasonable so > that the calls to __dax_dbg() work and print something useful. > > Signed-off-by: Ross Zwisler > Cc: Dan Williams Looks good. But don't you need to do the same for __dax_fault(), dax_zero_page_range() and similar places passing bh to dax functions? Honza > --- > fs/dax.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/fs/dax.c b/fs/dax.c > index 7af8797..513bba5 100644 > --- a/fs/dax.c > +++ b/fs/dax.c > @@ -624,6 +624,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > } > > memset(&bh, 0, sizeof(bh)); > + bh.b_bdev = inode->i_sb->s_bdev; > block = (sector_t)pgoff << (PAGE_SHIFT - blkbits); > > bh.b_size = PMD_SIZE; > -- > 2.5.0 > > -- Jan Kara SUSE Labs, CR From jack@suse.cz Tue Jan 12 03:44:46 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id E728B7F37 for ; Tue, 12 Jan 2016 03:44:45 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 77B31AC002 for ; Tue, 12 Jan 2016 01:44:45 -0800 (PST) X-ASG-Debug-ID: 1452591881-04cb6c75db8b4c0001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id BUyjoGDybXl1HnK6 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 12 Jan 2016 01:44:43 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id BC3DCABB4; Tue, 12 Jan 2016 09:44:39 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id 9B838823D8; Tue, 12 Jan 2016 10:44:51 +0100 (CET) Date: Tue, 12 Jan 2016 10:44:51 +0100 From: Jan Kara To: Ross Zwisler Cc: linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH v8 2/9] dax: fix conversion of holes to PMDs Message-ID: <20160112094451.GS6262@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v8 2/9] dax: fix conversion of holes to PMDs References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> <1452230879-18117-3-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452230879-18117-3-git-send-email-ross.zwisler@linux.intel.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452591882 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26056 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Thu 07-01-16 22:27:52, Ross Zwisler wrote: > When we get a DAX PMD fault for a write it is possible that there could be > some number of 4k zero pages already present for the same range that were > inserted to service reads from a hole. These 4k zero pages need to be > unmapped from the VMAs and removed from the struct address_space radix tree > before the real DAX PMD entry can be inserted. > > For PTE faults this same use case also exists and is handled by a > combination of unmap_mapping_range() to unmap the VMAs and > delete_from_page_cache() to remove the page from the address_space radix > tree. > > For PMD faults we do have a call to unmap_mapping_range() (protected by a > buffer_new() check), but nothing clears out the radix tree entry. The > buffer_new() check is also incorrect as the current ext4 and XFS filesystem > code will never return a buffer_head with BH_New set, even when allocating > new blocks over a hole. Instead the filesystem will zero the blocks > manually and return a buffer_head with only BH_Mapped set. > > Fix this situation by removing the buffer_new() check and adding a call to > truncate_inode_pages_range() to clear out the radix tree entries before we > insert the DAX PMD. > > Signed-off-by: Ross Zwisler > Reported-by: Dan Williams > Tested-by: Dan Williams Just two nits below. Nothing serious so you can add: Reviewed-by: Jan Kara > --- > fs/dax.c | 20 ++++++++++---------- > 1 file changed, 10 insertions(+), 10 deletions(-) > > diff --git a/fs/dax.c b/fs/dax.c > index 513bba5..5b84a46 100644 > --- a/fs/dax.c > +++ b/fs/dax.c > @@ -589,6 +589,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > bool write = flags & FAULT_FLAG_WRITE; > struct block_device *bdev; > pgoff_t size, pgoff; > + loff_t lstart, lend; > sector_t block; > int result = 0; > > @@ -643,15 +644,13 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, > goto fallback; > } > > - /* > - * If we allocated new storage, make sure no process has any > - * zero pages covering this hole > - */ > - if (buffer_new(&bh)) { > - i_mmap_unlock_read(mapping); > - unmap_mapping_range(mapping, pgoff << PAGE_SHIFT, PMD_SIZE, 0); > - i_mmap_lock_read(mapping); > - } > + /* make sure no process has any zero pages covering this hole */ > + lstart = pgoff << PAGE_SHIFT; > + lend = lstart + PMD_SIZE - 1; /* inclusive */ > + i_mmap_unlock_read(mapping); Just a nit but is there reason why we grab i_mmap_lock_read(mapping) only to release it a few lines below? The bh checks inside the locked region don't seem to rely on i_mmap_lock... > + unmap_mapping_range(mapping, lstart, PMD_SIZE, 0); > + truncate_inode_pages_range(mapping, lstart, lend); These two calls can be shortened as: truncate_pagecache_range(inode, lstart, lend); Honza -- Jan Kara SUSE Labs, CR From jack@suse.cz Tue Jan 12 04:57:11 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 6A6357F37 for ; Tue, 12 Jan 2016 04:57:11 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id 3ED9A30407E for ; Tue, 12 Jan 2016 02:57:11 -0800 (PST) X-ASG-Debug-ID: 1452596227-04bdf06f6b93970001-NocioJ Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id tXZbiqcHMqKqXXTn (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 12 Jan 2016 02:57:08 -0800 (PST) X-Barracuda-Envelope-From: jack@suse.cz X-Barracuda-Apparent-Source-IP: 195.135.220.15 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 72C43AAB6; Tue, 12 Jan 2016 10:57:05 +0000 (UTC) Received: by quack.suse.cz (Postfix, from userid 1000) id 4FCF3823D8; Tue, 12 Jan 2016 11:57:16 +0100 (CET) Date: Tue, 12 Jan 2016 11:57:16 +0100 From: Jan Kara To: Ross Zwisler Cc: linux-kernel@vger.kernel.org, "H. Peter Anvin" , "J. Bruce Fields" , Theodore Ts'o , Alexander Viro , Andreas Dilger , Andrew Morton , Dan Williams , Dave Chinner , Dave Hansen , Ingo Molnar , Jan Kara , Jeff Layton , Matthew Wilcox , Matthew Wilcox , Thomas Gleixner , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com Subject: Re: [PATCH v8 6/9] dax: add support for fsync/msync Message-ID: <20160112105716.GT6262@quack.suse.cz> X-ASG-Orig-Subj: Re: [PATCH v8 6/9] dax: add support for fsync/msync References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com> <1452230879-18117-7-git-send-email-ross.zwisler@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452230879-18117-7-git-send-email-ross.zwisler@linux.intel.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Barracuda-Connect: mx2.suse.de[195.135.220.15] X-Barracuda-Start-Time: 1452596228 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26057 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header On Thu 07-01-16 22:27:56, Ross Zwisler wrote: > To properly handle fsync/msync in an efficient way DAX needs to track dirty > pages so it is able to flush them durably to media on demand. > > The tracking of dirty pages is done via the radix tree in struct > address_space. This radix tree is already used by the page writeback > infrastructure for tracking dirty pages associated with an open file, and > it already has support for exceptional (non struct page*) entries. We > build upon these features to add exceptional entries to the radix tree for > DAX dirty PMD or PTE pages at fault time. > > Signed-off-by: Ross Zwisler Some comments below. > --- > fs/dax.c | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++-- > include/linux/dax.h | 2 + > mm/filemap.c | 6 ++ > 3 files changed, 196 insertions(+), 6 deletions(-) > > diff --git a/fs/dax.c b/fs/dax.c > index 5b84a46..0db21ea 100644 > --- a/fs/dax.c > +++ b/fs/dax.c > @@ -24,6 +24,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -324,6 +325,174 @@ static int copy_user_bh(struct page *to, struct inode *inode, > return 0; > } > > +#define NO_SECTOR -1 > + > +static int dax_radix_entry(struct address_space *mapping, pgoff_t index, > + sector_t sector, bool pmd_entry, bool dirty) > +{ > + struct radix_tree_root *page_tree = &mapping->page_tree; > + int type, error = 0; > + void *entry; > + > + __mark_inode_dirty(mapping->host, I_DIRTY_PAGES); > + > + spin_lock_irq(&mapping->tree_lock); > + entry = radix_tree_lookup(page_tree, index); > + > + if (entry) { > + type = RADIX_DAX_TYPE(entry); > + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && > + type != RADIX_DAX_PMD)) { > + error = -EIO; > + goto unlock; > + } > + > + if (!pmd_entry || type == RADIX_DAX_PMD) > + goto dirty; > + radix_tree_delete(&mapping->page_tree, index); > + mapping->nrexceptional--; In theory, you can delete here DIRTY / TOWRITE PTE entry and insert a clean PMD entry instead of it. That will cause fsync() to miss some flushes. So you should make sure you transfer all the tags to the new entry. > +static int dax_writeback_one(struct block_device *bdev, > + struct address_space *mapping, pgoff_t index, void *entry) > +{ > + struct radix_tree_root *page_tree = &mapping->page_tree; > + int type = RADIX_DAX_TYPE(entry); > + struct radix_tree_node *node; > + struct blk_dax_ctl dax; > + void **slot; > + int ret = 0; > + > + spin_lock_irq(&mapping->tree_lock); > + /* > + * Regular page slots are stabilized by the page lock even > + * without the tree itself locked. These unlocked entries > + * need verification under the tree lock. > + */ > + if (!__radix_tree_lookup(page_tree, index, &node, &slot)) > + goto unlock; > + if (*slot != entry) > + goto unlock; > + > + /* another fsync thread may have already written back this entry */ > + if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE)) > + goto unlock; > + > + radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE); > + > + if (WARN_ON_ONCE(type != RADIX_DAX_PTE && type != RADIX_DAX_PMD)) { > + ret = -EIO; > + goto unlock; > + } > + > + dax.sector = RADIX_DAX_SECTOR(entry); > + dax.size = (type == RADIX_DAX_PMD ? PMD_SIZE : PAGE_SIZE); > + spin_unlock_irq(&mapping->tree_lock); This seems to be somewhat racy as well - if there are two fsyncs running against the same inode, one wins the race and clears TOWRITE tag, the second then bails out and may finish before the skipped page gets flushed. So we should clear the TOWRITE tag only after the range is flushed. This can result in some amount of duplicit flushing but I don't think the race will happen that frequently in practice to be performance relevant. And secondly: You must write-protect all mappings of the flushed range so that you get fault when the sector gets written-to again. We spoke about this in the past already but somehow it got lost and I forgot about it as well. You need something like rmap_walk_file()... > + /* > + * We cannot hold tree_lock while calling dax_map_atomic() because it > + * eventually calls cond_resched(). > + */ > + ret = dax_map_atomic(bdev, &dax); > + if (ret < 0) > + return ret; > + > + if (WARN_ON_ONCE(ret < dax.size)) { > + ret = -EIO; > + goto unmap; > + } > + > + wb_cache_pmem(dax.addr, dax.size); > + unmap: > + dax_unmap_atomic(bdev, &dax); > + return ret; > + > + unlock: > + spin_unlock_irq(&mapping->tree_lock); > + return ret; > +} ... > @@ -791,15 +976,12 @@ EXPORT_SYMBOL_GPL(dax_pmd_fault); > * dax_pfn_mkwrite - handle first write to DAX page > * @vma: The virtual memory area where the fault occurred > * @vmf: The description of the fault > - * > */ > int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf) > { > - struct super_block *sb = file_inode(vma->vm_file)->i_sb; > + struct file *file = vma->vm_file; > > - sb_start_pagefault(sb); > - file_update_time(vma->vm_file); > - sb_end_pagefault(sb); > + dax_radix_entry(file->f_mapping, vmf->pgoff, NO_SECTOR, false, true); Why is NO_SECTOR argument correct here? Honza -- Jan Kara SUSE Labs, CR From bfoster@redhat.com Tue Jan 12 08:01:24 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id D9FD029DF5 for ; Tue, 12 Jan 2016 08:01:24 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id CB29130405F for ; Tue, 12 Jan 2016 06:01:21 -0800 (PST) X-ASG-Debug-ID: 1452607280-04cbb05f7f127db0001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id SRbbtejzf6X3wBla (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 06:01:20 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id 0E4FB1C5F5A; Tue, 12 Jan 2016 14:01:20 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0CE1JBp021610; Tue, 12 Jan 2016 09:01:19 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 94ADA1245A1; Tue, 12 Jan 2016 09:01:18 -0500 (EST) Date: Tue, 12 Jan 2016 09:01:18 -0500 From: Brian Foster To: "Darrick J. Wong" Cc: Dave Chinner , xfs@oss.sgi.com Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile time Message-ID: <20160112140116.GB12156@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile time References: <20160111232657.GA7831@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160111232657.GA7831@birch.djwong.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452607280 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Mon, Jan 11, 2016 at 03:26:57PM -0800, Darrick J. Wong wrote: > Check the sizes of XFS on-disk structures when compiling the kernel. > Use this to catch inadvertent changes in structure size due to padding > and alignment issues, etc. > > Signed-off-by: Darrick J. Wong > --- Seems reasonable to me: Reviewed-by: Brian Foster > fs/xfs/libxfs/xfs_format.h | 4 ++++ > fs/xfs/xfs_super.c | 45 ++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 49 insertions(+) > > diff --git a/fs/xfs/libxfs/xfs_format.h b/fs/xfs/libxfs/xfs_format.h > index e2536bb..a589b47 100644 > --- a/fs/xfs/libxfs/xfs_format.h > +++ b/fs/xfs/libxfs/xfs_format.h > @@ -1509,4 +1509,8 @@ struct xfs_acl { > #define SGI_ACL_FILE_SIZE (sizeof(SGI_ACL_FILE)-1) > #define SGI_ACL_DEFAULT_SIZE (sizeof(SGI_ACL_DEFAULT)-1) > > +#define XFS_CHECK_STRUCT_SIZE(structname, size) \ > + BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(struct " \ > + #structname ") is wrong, expected " #size) > + > #endif /* __XFS_FORMAT_H__ */ > diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c > index 36bd882..31b69d1 100644 > --- a/fs/xfs/xfs_super.c > +++ b/fs/xfs/xfs_super.c > @@ -1812,11 +1812,56 @@ xfs_destroy_workqueues(void) > destroy_workqueue(xfs_alloc_wq); > } > > +static void __init > +xfs_check_ondisk_structs(void) > +{ > + /* on-disk structures */ > + XFS_CHECK_STRUCT_SIZE(struct xfs_dsb, 264); > + XFS_CHECK_STRUCT_SIZE(struct xfs_agf, 224); > + XFS_CHECK_STRUCT_SIZE(struct xfs_agi, 336); > + XFS_CHECK_STRUCT_SIZE(struct xfs_agfl, 36); > + XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp, 8); > + XFS_CHECK_STRUCT_SIZE(struct xfs_dinode, 176); > + XFS_CHECK_STRUCT_SIZE(struct xfs_disk_dquot, 104); > + XFS_CHECK_STRUCT_SIZE(struct xfs_dqblk, 136); > + XFS_CHECK_STRUCT_SIZE(struct xfs_dsymlink_hdr, 56); > + XFS_CHECK_STRUCT_SIZE(xfs_alloc_rec_t, 8); > + XFS_CHECK_STRUCT_SIZE(xfs_alloc_key_t, 8); > + XFS_CHECK_STRUCT_SIZE(xfs_alloc_ptr_t, 4); > + XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_rec, 16); > + XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_key, 4); > + XFS_CHECK_STRUCT_SIZE(xfs_inobt_ptr_t, 4); > + XFS_CHECK_STRUCT_SIZE(struct xfs_bmdr_block, 4); > + XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_rec, 16); > + XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_key, 8); > + XFS_CHECK_STRUCT_SIZE(struct xfs_btree_block, 72); > + XFS_CHECK_STRUCT_SIZE(struct xfs_acl_entry, 12); > + XFS_CHECK_STRUCT_SIZE(struct xfs_acl, 4); > + > + /* log structures */ > + XFS_CHECK_STRUCT_SIZE(struct xfs_trans_header, 16); > + XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_32, 52); > + XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_64, 56); > + XFS_CHECK_STRUCT_SIZE(struct xfs_ictimestamp, 8); > + XFS_CHECK_STRUCT_SIZE(struct xfs_icdinode, 176); > + XFS_CHECK_STRUCT_SIZE(struct xfs_extent_32, 12); > + XFS_CHECK_STRUCT_SIZE(struct xfs_extent_64, 16); > + XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_32, 28); > + XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64, 32); > + XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32, 28); > + XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64, 32); > + XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat, 24); > + XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat, 20); > + XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log, 28); > +} > + > STATIC int __init > init_xfs_fs(void) > { > int error; > > + xfs_check_ondisk_structs(); > + > printk(KERN_INFO XFS_VERSION_STRING " with " > XFS_BUILD_OPTIONS " enabled\n"); > > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Tue Jan 12 08:01:27 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 4756A29DFC for ; Tue, 12 Jan 2016 08:01:27 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 36ED7304053 for ; Tue, 12 Jan 2016 06:01:27 -0800 (PST) X-ASG-Debug-ID: 1452607284-04cb6c75dc94070001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id R1DhAsveOw3M3B2j (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 06:01:25 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 5161BC0293B9; Tue, 12 Jan 2016 14:01:24 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0CE1NK2004544; Tue, 12 Jan 2016 09:01:23 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id E67C41245A1; Tue, 12 Jan 2016 09:01:22 -0500 (EST) Date: Tue, 12 Jan 2016 09:01:22 -0500 From: Brian Foster To: "Darrick J. Wong" Cc: Dave Chinner , xfs@oss.sgi.com Subject: Re: [PATCH] xfs_db: check on-disk structure sizes Message-ID: <20160112140122.GC12156@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH] xfs_db: check on-disk structure sizes References: <20160111234644.GB7831@birch.djwong.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160111234644.GB7831@birch.djwong.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452607284 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Mon, Jan 11, 2016 at 03:46:44PM -0800, Darrick J. Wong wrote: > Check on-disk structure sizes against known values. > Use this to catch inadvertent changes in structure size due to padding > and alignment issues, etc. > > Signed-off-by: Darrick J. Wong > --- What's the need for this in userspace? Not a big deal really, but it seems like it serves the fundamental purpose sufficiently in the kernel. That aside, this does fail if I tweak a structure size, though I don't think I get the intended error. I end up with a linker error instead: ... [LD] xfs_db ondisk.o: In function `xfs_check_ondisk_structs': /home/bfoster/repos/xfsprogs-dev/db/ondisk.c:26: undefined reference to `__compiletime_assert_26' collect2: error: ld returned 1 exit status ../include/buildrules:45: recipe for target 'xfs_db' failed gmake[3]: *** [xfs_db] Error 1 include/buildrules:35: recipe for target 'db' failed gmake[2]: *** [db] Error 2 Makefile:70: recipe for target 'default' failed make[1]: *** [default] Error 2 Makefile:68: recipe for target 'default' failed make: *** [default] Error 2 This is with gcc 5.3.1. It works fine with the kernel patch so I don't _think_ it's my environment, but I could be wrong... Brian > db/Makefile | 2 - > db/init.c | 3 + > db/ondisk.c | 63 +++++++++++++++++++++++ > db/ondisk.h | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++ > libxfs/xfs_format.h | 4 + > 5 files changed, 210 insertions(+), 1 deletion(-) > create mode 100644 db/ondisk.c > create mode 100644 db/ondisk.h > > diff --git a/db/Makefile b/db/Makefile > index 8260da3..ba3e942 100644 > --- a/db/Makefile > +++ b/db/Makefile > @@ -12,7 +12,7 @@ HFILES = addr.h agf.h agfl.h agi.h attr.h attrshort.h bit.h block.h bmap.h \ > dir2.h dir2sf.h dquot.h echo.h faddr.h field.h \ > flist.h fprint.h frag.h freesp.h hash.h help.h init.h inode.h input.h \ > io.h logformat.h malloc.h metadump.h output.h print.h quit.h sb.h \ > - sig.h strvec.h text.h type.h write.h attrset.h symlink.h > + sig.h strvec.h text.h type.h write.h attrset.h symlink.h ondisk.h > CFILES = $(HFILES:.h=.c) > LSRCFILES = xfs_admin.sh xfs_ncheck.sh xfs_metadump.sh > > diff --git a/db/init.c b/db/init.c > index c0472c8..d6df093 100644 > --- a/db/init.c > +++ b/db/init.c > @@ -28,6 +28,7 @@ > #include "output.h" > #include "malloc.h" > #include "type.h" > +#include "ondisk.h" > > static char **cmdline; > static int ncmdline; > @@ -60,6 +61,8 @@ init( > struct xfs_buf *bp; > int c; > > + xfs_check_ondisk_structs(); > + > setlocale(LC_ALL, ""); > bindtextdomain(PACKAGE, LOCALEDIR); > textdomain(PACKAGE); > diff --git a/db/ondisk.c b/db/ondisk.c > new file mode 100644 > index 0000000..532333d > --- /dev/null > +++ b/db/ondisk.c > @@ -0,0 +1,63 @@ > +/* > + * Copyright (c) 2016 Oracle. > + * All Rights Reserved. > + * > + * This program is free software; you can redistribute it and/or > + * modify it under the terms of the GNU General Public License as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it would be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program; if not, write the Free Software Foundation, > + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#include "libxfs.h" > +#include "ondisk.h" > + > +void > +xfs_check_ondisk_structs(void) > +{ > + /* on-disk structures */ > + XFS_CHECK_STRUCT_SIZE(struct xfs_dsb, 264); > + XFS_CHECK_STRUCT_SIZE(struct xfs_agf, 224); > + XFS_CHECK_STRUCT_SIZE(struct xfs_agi, 336); > + XFS_CHECK_STRUCT_SIZE(struct xfs_agfl, 36); > + XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp, 8); > + XFS_CHECK_STRUCT_SIZE(struct xfs_dinode, 176); > + XFS_CHECK_STRUCT_SIZE(struct xfs_disk_dquot, 104); > + XFS_CHECK_STRUCT_SIZE(struct xfs_dqblk, 136); > + XFS_CHECK_STRUCT_SIZE(struct xfs_dsymlink_hdr, 56); > + XFS_CHECK_STRUCT_SIZE(xfs_alloc_rec_t, 8); > + XFS_CHECK_STRUCT_SIZE(xfs_alloc_key_t, 8); > + XFS_CHECK_STRUCT_SIZE(xfs_alloc_ptr_t, 4); > + XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_rec, 16); > + XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_key, 4); > + XFS_CHECK_STRUCT_SIZE(xfs_inobt_ptr_t, 4); > + XFS_CHECK_STRUCT_SIZE(struct xfs_bmdr_block, 4); > + XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_rec, 16); > + XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_key, 8); > + XFS_CHECK_STRUCT_SIZE(struct xfs_btree_block, 72); > + XFS_CHECK_STRUCT_SIZE(struct xfs_acl_entry, 12); > + XFS_CHECK_STRUCT_SIZE(struct xfs_acl, 4); > + > + /* log structures */ > + XFS_CHECK_STRUCT_SIZE(struct xfs_trans_header, 16); > + XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_32, 52); > + XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_64, 56); > + XFS_CHECK_STRUCT_SIZE(struct xfs_ictimestamp, 8); > + XFS_CHECK_STRUCT_SIZE(struct xfs_icdinode, 176); > + XFS_CHECK_STRUCT_SIZE(struct xfs_extent_32, 12); > + XFS_CHECK_STRUCT_SIZE(struct xfs_extent_64, 16); > + XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_32, 28); > + XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64, 32); > + XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32, 28); > + XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64, 32); > + XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat, 24); > + XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat, 20); > + XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log, 28); > +} > diff --git a/db/ondisk.h b/db/ondisk.h > new file mode 100644 > index 0000000..b5784d1 > --- /dev/null > +++ b/db/ondisk.h > @@ -0,0 +1,139 @@ > +/* > + * Copyright (c) 2016 Oracle. > + * All Rights Reserved. > + * > + * This program is free software; you can redistribute it and/or > + * modify it under the terms of the GNU General Public License as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it would be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program; if not, write the Free Software Foundation, > + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > + */ > + > +#ifndef __XFS_DB_ONDISK_H > +#define __XFS_DB_ONDISK_H > + > +/* Compile time object size, -1 for unknown */ > +#ifndef __compiletime_error > +# define __compiletime_error(message) > +/* > + * Sparse complains of variable sized arrays due to the temporary variable in > + * __compiletime_assert. Unfortunately we can't just expand it out to make > + * sparse see a constant array size without breaking compiletime_assert on old > + * versions of GCC (e.g. 4.2.4), so hide the array from sparse altogether. > + */ > +# ifndef __CHECKER__ > +# define __compiletime_error_fallback(condition) \ > + do { ((void)sizeof(char[1 - 2 * condition])); } while (0) > +# endif > +#endif > +#ifndef __compiletime_error_fallback > +# define __compiletime_error_fallback(condition) do { } while (0) > +#endif > + > +#define __compiletime_assert(condition, msg, prefix, suffix) \ > + do { \ > + bool __cond = !(condition); \ > + extern void prefix ## suffix(void) __compiletime_error(msg); \ > + if (__cond) \ > + prefix ## suffix(); \ > + __compiletime_error_fallback(__cond); \ > + } while (0) > + > +#define _compiletime_assert(condition, msg, prefix, suffix) \ > + __compiletime_assert(condition, msg, prefix, suffix) > + > +/** > + * compiletime_assert - break build and emit msg if condition is false > + * @condition: a compile-time constant condition to check > + * @msg: a message to emit if condition is false > + * > + * In tradition of POSIX assert, this macro will break the build if the > + * supplied condition is *false*, emitting the supplied error message if the > + * compiler has support to do so. > + */ > +#define compiletime_assert(condition, msg) \ > + _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__) > + > +#ifdef __CHECKER__ > +#define BUILD_BUG_ON_NOT_POWER_OF_2(n) (0) > +#define BUILD_BUG_ON_ZERO(e) (0) > +#define BUILD_BUG_ON_NULL(e) ((void*)0) > +#define BUILD_BUG_ON_INVALID(e) (0) > +#define BUILD_BUG_ON_MSG(cond, msg) (0) > +#define BUILD_BUG_ON(condition) (0) > +#define BUILD_BUG() (0) > +#else /* __CHECKER__ */ > + > +/* Force a compilation error if a constant expression is not a power of 2 */ > +#define BUILD_BUG_ON_NOT_POWER_OF_2(n) \ > + BUILD_BUG_ON((n) == 0 || (((n) & ((n) - 1)) != 0)) > + > +/* Force a compilation error if condition is true, but also produce a > + result (of value 0 and type size_t), so the expression can be used > + e.g. in a structure initializer (or where-ever else comma expressions > + aren't permitted). */ > +#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); })) > +#define BUILD_BUG_ON_NULL(e) ((void *)sizeof(struct { int:-!!(e); })) > + > +/* > + * BUILD_BUG_ON_INVALID() permits the compiler to check the validity of the > + * expression but avoids the generation of any code, even if that expression > + * has side-effects. > + */ > +#define BUILD_BUG_ON_INVALID(e) ((void)(sizeof((__force long)(e)))) > + > +/** > + * BUILD_BUG_ON_MSG - break compile if a condition is true & emit supplied > + * error message. > + * @condition: the condition which the compiler should know is false. > + * > + * See BUILD_BUG_ON for description. > + */ > +#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg) > + > +/** > + * BUILD_BUG_ON - break compile if a condition is true. > + * @condition: the condition which the compiler should know is false. > + * > + * If you have some code which relies on certain constants being equal, or > + * some other compile-time-evaluated condition, you should use BUILD_BUG_ON to > + * detect if someone changes it. > + * > + * The implementation uses gcc's reluctance to create a negative array, but gcc > + * (as of 4.4) only emits that error for obvious cases (e.g. not arguments to > + * inline functions). Luckily, in 4.3 they added the "error" function > + * attribute just for this type of case. Thus, we use a negative sized array > + * (should always create an error on gcc versions older than 4.4) and then call > + * an undefined function with the error attribute (should always create an > + * error on gcc 4.3 and later). If for some reason, neither creates a > + * compile-time error, we'll still have a link-time error, which is harder to > + * track down. > + */ > +#ifndef __OPTIMIZE__ > +#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)])) > +#else > +#define BUILD_BUG_ON(condition) \ > + BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition) > +#endif > + > +/** > + * BUILD_BUG - break compile if used. > + * > + * If you have some code that you expect the compiler to eliminate at > + * build time, you should use BUILD_BUG to detect if it is > + * unexpectedly used. > + */ > +#define BUILD_BUG() BUILD_BUG_ON_MSG(1, "BUILD_BUG failed") > + > +#endif /* __CHECKER__ */ > + > +void xfs_check_ondisk_structs(void); > + > +#endif /* __XFS_DB_ONDISK_H */ > diff --git a/libxfs/xfs_format.h b/libxfs/xfs_format.h > index a35009a..c4c0c1c 100644 > --- a/libxfs/xfs_format.h > +++ b/libxfs/xfs_format.h > @@ -1505,4 +1505,8 @@ struct xfs_acl { > #define SGI_ACL_FILE_SIZE (sizeof(SGI_ACL_FILE)-1) > #define SGI_ACL_DEFAULT_SIZE (sizeof(SGI_ACL_DEFAULT)-1) > > +#define XFS_CHECK_STRUCT_SIZE(structname, size) \ > + BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(struct " \ > + #structname ") is wrong, expected " #size) > + > #endif /* __XFS_FORMAT_H__ */ > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From bfoster@redhat.com Tue Jan 12 08:15:47 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id CDCA029DF5 for ; Tue, 12 Jan 2016 08:15:47 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay2.corp.sgi.com (Postfix) with ESMTP id B1C02304053 for ; Tue, 12 Jan 2016 06:15:47 -0800 (PST) X-ASG-Debug-ID: 1452608142-04cbb05f7c128660001-NocioJ Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id KyTFCA7jMbcboG6o (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 06:15:43 -0800 (PST) X-Barracuda-Envelope-From: bfoster@redhat.com X-Barracuda-Apparent-Source-IP: 209.132.183.28 X-ASG-Whitelist: Client Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id A2DDB8AE73; Tue, 12 Jan 2016 14:15:42 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0CEFg9H000735; Tue, 12 Jan 2016 09:15:42 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id 4AAF31245A1; Tue, 12 Jan 2016 09:15:41 -0500 (EST) Date: Tue, 12 Jan 2016 09:15:41 -0500 From: Brian Foster To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [PATCH] xfs: log mount failures don't wait for buffers to be released Message-ID: <20160112141540.GD12156@bfoster.bfoster> X-ASG-Orig-Subj: Re: [PATCH] xfs: log mount failures don't wait for buffers to be released References: <1452562536-32145-1-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452562536-32145-1-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Barracuda-Connect: mx1.redhat.com[209.132.183.28] X-Barracuda-Start-Time: 1452608143 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 On Tue, Jan 12, 2016 at 12:35:36PM +1100, Dave Chinner wrote: > From: Dave Chinner > > Recently I've been seeing xfs/051 fail on 1k block size filesystems. > Trying to trace the events during the test lead to the problem going > away, indicating that it was a race condition that lead to this > ASSERT failure: > > XFS: Assertion failed: atomic_read(&pag->pag_ref) == 0, file: fs/xfs/xfs_mount.c, line: 156 > ..... > [] xfs_free_perag+0x87/0xb0 > [] xfs_mountfs+0x4d9/0x900 > [] xfs_fs_fill_super+0x3bf/0x4d0 > [] mount_bdev+0x180/0x1b0 > [] xfs_fs_mount+0x15/0x20 > [] mount_fs+0x38/0x170 > [] vfs_kern_mount+0x67/0x120 > [] do_mount+0x218/0xd60 > [] SyS_mount+0x8b/0xd0 > > When I finally caught it with tracing enabled, I saw that AG 2 had > an elevated reference count and a buffer was responsible for it. I > tracked down the specific buffer, and found that it was missing the > final reference count release that would put it back on the LRU and > hence be found by xfs_wait_buftarg() calls in the log mount failure > handling. > > The last four traces for the buffer before the assert were (trimmed > for relevance) > > kworker/0:1-5259 xfs_buf_iodone: hold 2 lock 0 flags ASYNC > kworker/0:1-5259 xfs_buf_ioerror: hold 2 lock 0 error -5 > mount-7163 xfs_buf_lock_done: hold 2 lock 0 flags ASYNC > mount-7163 xfs_buf_unlock: hold 2 lock 1 flags ASYNC > > This is an async write that is completing, so there's nobody waiting > for it directly. Hence we call xfs_buf_relse() once all the > processing is complete. That does: > > static inline void xfs_buf_relse(xfs_buf_t *bp) > { > xfs_buf_unlock(bp); > xfs_buf_rele(bp); > } > > Now, it's clear that mount is waiting on the buffer lock, and that > it has been released by xfs_buf_relse() and gained by mount. This is > expected, because at this point the mount process is in > xfs_buf_delwri_submit() waiting for all the IO it submitted to > complete. > > The mount process, however, is waiting on the lock for the buffer > because it is in xfs_buf_delwri_submit(). This waits for IO > completion, but it doesn't wait for the buffer reference owned by > the IO to go away. The mount process collects all the completions, > fails the log recovery, and the higher level code then calls > xfs_wait_buftarg() to free all the remaining buffers in the > filesystem. > > The issue is that on unlocking the buffer, the scheduler has decided > that the mount process has higher priority than the the kworker > thread that is running the IO completion, and so immediately > switched contexts to the mount process from the semaphore unlock > code, hence preventing the kworker thread from finishing the IO > completion and releasing the IO reference to the buffer. > > Hence by the time that xfs_wait_buftarg() is run, the buffer still > has an active reference and so isn't on the LRU list that the > function walks to free the remaining buffers. Hence we miss that > buffer and continue onwards to tear down the mount structures, > at which time we get find a stray reference count on the perag > structure. On a non-debug kernel, this will be ignored and the > structure torn down and freed. Hence when the kworker thread is then > rescheduled and the buffer released and freed, it will access a > freed perag structure. > > The problem here is that when the log mount fails, we still need to > quiesce the log to ensure that the IO workqueues have returned to > idle before we run xfs_wait_buftarg(). By synchronising the > workqueues, we ensure that all IO completions are fully processed, > not just to the point where buffers have been unlocked. This ensures > we don't end up in the situation above. > > Signed-off-by: Dave Chinner > --- Reviewed-by: Brian Foster > fs/xfs/xfs_buf.c | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c > index 04d2bdd..08ad701 100644 > --- a/fs/xfs/xfs_buf.c > +++ b/fs/xfs/xfs_buf.c > @@ -1529,6 +1529,16 @@ xfs_wait_buftarg( > LIST_HEAD(dispose); > int loop = 0; > > + /* > + * We need to flush the buffer workqueue to ensure that all IO > + * completion processing is 100% done. Just waiting on buffer locks is > + * not sufficient for async IO as the reference count held over IO is > + * not released until after the buffer lock is dropped. Hence we need to > + * ensure here that all reference counts have been dropped before we > + * start walking the LRU list. > + */ > + drain_workqueue(btp->bt_mount->m_buf_workqueue); > + > /* loop until there is nothing left on the lru list. */ > while (list_lru_count(&btp->bt_lru)) { > list_lru_walk(&btp->bt_lru, xfs_buftarg_wait_rele, > -- > 2.5.0 > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs From BATV+3b4b84fbb96c83bb5ed1+4524+infradead.org+hch@bombadil.srs.infradead.org Tue Jan 12 09:30:16 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id C631029DF5 for ; Tue, 12 Jan 2016 09:30:16 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id A5FFD304053 for ; Tue, 12 Jan 2016 07:30:13 -0800 (PST) X-ASG-Debug-ID: 1452612610-04bdf06f6ba0450001-NocioJ Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id mki5ujrDqqqO8fzP (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 07:30:10 -0800 (PST) X-Barracuda-Envelope-From: BATV+3b4b84fbb96c83bb5ed1+4524+infradead.org+hch@bombadil.srs.infradead.org X-Barracuda-Apparent-Source-IP: 198.137.202.9 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1aJ0te-0001o7-6z; Tue, 12 Jan 2016 15:30:10 +0000 Date: Tue, 12 Jan 2016 07:30:10 -0800 From: Christoph Hellwig To: Dave Chinner Cc: xfs@oss.sgi.com Subject: Re: [RFC PATCH 0/8] xfs: shrink the struct xfs_icdinode Message-ID: <20160112153010.GA29662@infradead.org> X-ASG-Orig-Subj: Re: [RFC PATCH 0/8] xfs: shrink the struct xfs_icdinode References: <1452589280-30167-1-git-send-email-david@fromorbit.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1452589280-30167-1-git-send-email-david@fromorbit.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org See http://www.infradead.org/rpr.html X-Barracuda-Connect: UNKNOWN[198.137.202.9] X-Barracuda-Start-Time: 1452612610 X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.10 X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_SA210e, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26062 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.00 BSF_SC5_SA210e Custom Rule SA210e Nice! I had started on some of this a while ago but never finished it.. > 72 bytes. This means it takes 104 bytes off the size of the struct > xfs_inode, which a 12% reduction in size. This will be a massive win > for systems that cache lots of inodes! How many more inodes can we fit into a slab cache now? Back when I started I noticed it doesn't help us to actually fit more inodes into a 4k page due to the bloated VFS inode. But these days slub actually uses a high order allocations if I remember correctly so it might be more useful. > With this change made, the xfs_icdinode is no long an "in core disk > inode" so I'm wondering whether I should rename it or simply make it > go away altogether and pull the remaining fields straight into the > struct xfs_inode. Any thoughts on new names and/or getting rid of it > woul dbe appreciated. I think it should be merged into the xfs_inode structure soner or later. Another thign I planned but never got to is to move fields into the inode fork that were specific to the attr or data fork. From felix.janda@posteo.de Tue Jan 12 14:02:17 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 8642B7F37 for ; Tue, 12 Jan 2016 14:02:17 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 1997DAC005 for ; Tue, 12 Jan 2016 12:02:13 -0800 (PST) X-ASG-Debug-ID: 1452628924-04bdf06f6caeb50001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id R23D5dXA8Mb3RRfm (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 12:02:05 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id D2D4520A44 for ; Tue, 12 Jan 2016 21:02:03 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg2qb3D6hz5vN3 for ; Tue, 12 Jan 2016 21:02:03 +0100 (CET) Date: Tue, 12 Jan 2016 20:59:28 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 0/6] xfsprogs: Compile with musl libc Message-ID: <20160112195928.GA568@nyan> X-ASG-Orig-Subj: [PATCH 0/6] xfsprogs: Compile with musl libc MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452628925 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- This patch series makes xfsprogs compile on linux systems using the musl c library. I'm sorry for the large size of the first patch... Felix Janda (6): Move from __uint*_t types to uint*_t and likewise for __int*_t libxfs/linux.c: Replace use of ustat by stat fsr/xfs_fsr.c: Include for _PATH_MOUNTED linux.h: Use off64_t instead of loff_t include/linux.h: Include for fprintf and stderr Add configure check for members of dirent for use in io/readdir.c configure.ac | 7 ++ copy/xfs_copy.c | 14 ++-- copy/xfs_copy.h | 2 +- db/attr.c | 8 +- db/attrshort.c | 4 +- db/bit.c | 4 +- db/bit.h | 2 +- db/block.c | 8 +- db/check.c | 18 ++--- db/convert.c | 28 +++---- db/faddr.c | 14 ++-- db/field.c | 44 +++++------ db/fprint.c | 8 +- db/frag.c | 8 +- db/inode.c | 30 ++++---- db/io.c | 4 +- db/io.h | 8 +- db/metadump.c | 12 +-- db/sb.c | 6 +- fsr/xfs_fsr.c | 3 +- include/builddefs.in | 4 + include/darwin.h | 9 --- include/freebsd.h | 6 +- include/gnukfreebsd.h | 6 +- include/irix.h | 28 +++---- include/linux.h | 11 +-- include/xfs_arch.h | 22 +++--- include/xfs_inode.h | 4 +- include/xfs_log_recover.h | 2 +- include/xfs_metadump.h | 4 +- include/xfs_mount.h | 20 ++--- io/Makefile | 10 +++ libhandle/handle.c | 2 +- libhandle/jdm.c | 2 +- libxfs/darwin.c | 2 +- libxfs/freebsd.c | 2 +- libxfs/init.c | 2 +- libxfs/irix.c | 2 +- libxfs/libxfs_priv.h | 4 +- libxfs/linux.c | 31 +++++--- libxfs/logitem.c | 2 +- libxfs/radix-tree.c | 8 +- libxfs/rdwr.c | 6 +- libxfs/util.c | 14 ++-- libxfs/xfs_alloc_btree.c | 10 +-- libxfs/xfs_attr_remote.c | 8 +- libxfs/xfs_attr_sf.h | 18 ++--- libxfs/xfs_bit.h | 24 +++--- libxfs/xfs_bmap_btree.c | 8 +- libxfs/xfs_btree.c | 12 +-- libxfs/xfs_btree.h | 14 ++-- libxfs/xfs_cksum.h | 14 ++-- libxfs/xfs_da_btree.c | 2 +- libxfs/xfs_da_btree.h | 8 +- libxfs/xfs_da_format.c | 24 +++--- libxfs/xfs_da_format.h | 60 +++++++-------- libxfs/xfs_dir2.h | 8 +- libxfs/xfs_dir2_leaf.c | 12 +-- libxfs/xfs_dir2_priv.h | 2 +- libxfs/xfs_format.h | 98 ++++++++++++------------ libxfs/xfs_fs.h | 12 +-- libxfs/xfs_ialloc.c | 6 +- libxfs/xfs_ialloc_btree.c | 4 +- libxfs/xfs_inode_buf.c | 2 +- libxfs/xfs_log_format.h | 190 +++++++++++++++++++++++----------------------- libxfs/xfs_quota_defs.h | 4 +- libxfs/xfs_rtbitmap.c | 2 +- libxfs/xfs_sb.c | 4 +- libxfs/xfs_types.h | 46 +++++------ logprint/log_misc.c | 6 +- mdrestore/xfs_mdrestore.c | 2 +- mkfs/proto.c | 2 +- mkfs/xfs_mkfs.c | 88 ++++++++++----------- quota/edit.c | 64 ++++++++-------- quota/free.c | 44 +++++------ quota/quot.c | 24 +++--- quota/quota.c | 4 +- quota/quota.h | 12 +-- quota/report.c | 2 +- quota/state.c | 2 +- quota/util.c | 66 ++++++++-------- repair/README | 2 +- repair/agheader.h | 14 ++-- repair/attr_repair.h | 26 +++---- repair/avl64.c | 38 +++++----- repair/avl64.h | 18 ++--- repair/dino_chunks.c | 2 +- repair/dinode.c | 38 +++++----- repair/dinode.h | 4 +- repair/dir2.c | 2 +- repair/globals.h | 16 ++-- repair/incore.c | 10 +-- repair/incore.h | 42 +++++----- repair/incore_ext.c | 6 +- repair/incore_ino.c | 72 +++++++++--------- repair/phase2.c | 2 +- repair/phase3.c | 6 +- repair/phase4.c | 4 +- repair/phase5.c | 40 +++++----- repair/phase6.c | 18 ++--- repair/phase7.c | 6 +- repair/progress.c | 30 ++++---- repair/progress.h | 4 +- repair/sb.c | 10 +-- repair/scan.c | 42 +++++----- repair/scan.h | 12 +-- 106 files changed, 914 insertions(+), 893 deletions(-) -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 14:02:32 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id A38927F37 for ; Tue, 12 Jan 2016 14:02:32 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 236D5AC005 for ; Tue, 12 Jan 2016 12:02:32 -0800 (PST) X-ASG-Debug-ID: 1452628940-04cb6c75dea95c0001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id KidPqaoyGx12Rupw (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 12:02:21 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id 31D76207AD for ; Tue, 12 Jan 2016 21:02:20 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg2qw0RVpz5vND for ; Tue, 12 Jan 2016 21:02:20 +0100 (CET) Date: Tue, 12 Jan 2016 20:59:45 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat Message-ID: <20160112195945.GC568@nyan> X-ASG-Orig-Subj: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452628941 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- ustat has been used to check whether a device file is mounted. The function is deprecated and not supported by uclibc and musl. Now do the check using the *mntent functions. Based on patch by Natanael Copa . Signed-off-by: Felix Janda --- libxfs/linux.c | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/libxfs/linux.c b/libxfs/linux.c index c532d65..e7adcf2 100644 --- a/libxfs/linux.c +++ b/libxfs/linux.c @@ -16,11 +16,8 @@ * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ -#define ustat __kernel_ustat #include #include -#undef ustat -#include #include #include #include @@ -51,9 +48,10 @@ static int max_block_alignment; int platform_check_ismounted(char *name, char *block, struct stat64 *s, int verbose) { - /* Pad ust; pre-2.6.28 linux copies out too much in 32bit compat mode */ - struct ustat ust[2]; - struct stat64 st; + FILE *f; + struct stat64 st, mst; + struct mntent *mnt; + char mounts[MAXPATHLEN]; if (!s) { if (stat64(block, &st) < 0) @@ -63,14 +61,27 @@ platform_check_ismounted(char *name, char *block, struct stat64 *s, int verbose) s = &st; } - if (ustat(s->st_rdev, ust) >= 0) { + strcpy(mounts, (!access(PROC_MOUNTED, R_OK)) ? PROC_MOUNTED : MOUNTED); + if ((f = setmntent(mounts, "r")) == NULL) { + fprintf(stderr, + _("%s: %s possibly contains a mounted filesystem\n"), + progname, name); + return 1; + } + while ((mnt = getmntent(f)) != NULL) { + if (stat64(mnt->mnt_dir, &mst) < 0) + continue; + if (mst.st_dev != s->st_rdev) + continue; + if (verbose) fprintf(stderr, _("%s: %s contains a mounted filesystem\n"), progname, name); - return 1; + break; } - return 0; + endmntent(f); + return mnt == NULL; } int -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 14:02:35 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id F3C6F29DF9 for ; Tue, 12 Jan 2016 14:02:34 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id EA01C304059 for ; Tue, 12 Jan 2016 12:02:31 -0800 (PST) X-ASG-Debug-ID: 1452628949-04cb6c75dea95e0001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id 1VVl8KhoyjmHrFDs (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 12:02:30 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id 4517D20A17 for ; Tue, 12 Jan 2016 21:02:29 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg2r50ywDz5vNB for ; Tue, 12 Jan 2016 21:02:29 +0100 (CET) Date: Tue, 12 Jan 2016 20:59:53 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 3/6] fsr/xfs_fsr.c: Include for _PATH_MOUNTED Message-ID: <20160112195953.GD568@nyan> X-ASG-Orig-Subj: [PATCH 3/6] fsr/xfs_fsr.c: Include for _PATH_MOUNTED References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452628950 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Fixes a compilation failure with musl libc Signed-off-by: Felix Janda --- fsr/xfs_fsr.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fsr/xfs_fsr.c b/fsr/xfs_fsr.c index f32aced..8957f93 100644 --- a/fsr/xfs_fsr.c +++ b/fsr/xfs_fsr.c @@ -31,6 +31,7 @@ #include #include #include +#include #ifndef XFS_XFLAG_NODEFRAG #define XFS_XFLAG_NODEFRAG 0x00002000 /* src dependancy, remove later */ -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 14:02:40 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id D0C9D7F37 for ; Tue, 12 Jan 2016 14:02:40 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id B764C304059 for ; Tue, 12 Jan 2016 12:02:40 -0800 (PST) X-ASG-Debug-ID: 1452628957-04bdf06f6caebb0001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id PYmL08KlFLA6Vla1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 12:02:38 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id AFC4B208B1 for ; Tue, 12 Jan 2016 21:02:36 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg2rD45m2z5vN0 for ; Tue, 12 Jan 2016 21:02:36 +0100 (CET) Date: Tue, 12 Jan 2016 21:00:01 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 4/6] linux.h: Use off64_t instead of loff_t Message-ID: <20160112200001.GE568@nyan> X-ASG-Orig-Subj: [PATCH 4/6] linux.h: Use off64_t instead of loff_t References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452628957 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- These are equivalent on glibc, while musl does not know loff_t. In the long run, it would be preferable to enable transparent LFS so that off64_t could be replaced by off_t. Signed-off-by: Felix Janda --- include/linux.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux.h b/include/linux.h index 827dbe1..4872a1e 100644 --- a/include/linux.h +++ b/include/linux.h @@ -141,7 +141,7 @@ platform_discard_blocks(int fd, uint64_t start, uint64_t len) #define EFSCORRUPTED EUCLEAN /* Filesystem is corrupted */ #define EFSBADCRC EBADMSG /* Bad CRC detected */ -typedef loff_t xfs_off_t; +typedef off64_t xfs_off_t; typedef uint64_t xfs_ino_t; typedef uint32_t xfs_dev_t; typedef int64_t xfs_daddr_t; -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 14:02:37 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id D25CE29DFE for ; Tue, 12 Jan 2016 14:02:36 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 10844AC005 for ; Tue, 12 Jan 2016 12:02:36 -0800 (PST) X-ASG-Debug-ID: 1452628931-04cbb05f7f13d040001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id QbrHeWT1eKZwXOmY (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 12:02:12 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id 7327C208A1 for ; Tue, 12 Jan 2016 21:02:11 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg2qk5nF5z5vN3 for ; Tue, 12 Jan 2016 21:02:10 +0100 (CET) Date: Tue, 12 Jan 2016 20:59:35 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise for __int*_t Message-ID: <20160112195935.GB568@nyan> X-ASG-Orig-Subj: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise for __int*_t References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452628932 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.75 X-Barracuda-Spam-Status: No, SCORE=0.75 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_RULE_7580F X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.75 BSF_RULE_7580F Custom Rule 7580F The uint*_t and int*_t are defined by C99 and should be prefered over the less portable __uint*_t and __int*_t variants. The necessary include is in platformdefs.h, which gets included in most places via libxfs.h. In the public headers is included in . Signed-off-by: Felix Janda --- copy/xfs_copy.c | 14 ++-- copy/xfs_copy.h | 2 +- db/attr.c | 8 +- db/attrshort.c | 4 +- db/bit.c | 4 +- db/bit.h | 2 +- db/block.c | 8 +- db/check.c | 18 ++--- db/convert.c | 28 +++---- db/faddr.c | 14 ++-- db/field.c | 44 +++++------ db/fprint.c | 8 +- db/frag.c | 8 +- db/inode.c | 30 ++++---- db/io.c | 4 +- db/io.h | 8 +- db/metadump.c | 12 +-- db/sb.c | 6 +- fsr/xfs_fsr.c | 2 +- include/darwin.h | 9 --- include/freebsd.h | 6 +- include/gnukfreebsd.h | 6 +- include/irix.h | 28 +++---- include/linux.h | 8 +- include/xfs_arch.h | 22 +++--- include/xfs_inode.h | 4 +- include/xfs_log_recover.h | 2 +- include/xfs_metadump.h | 4 +- include/xfs_mount.h | 20 ++--- libhandle/handle.c | 2 +- libhandle/jdm.c | 2 +- libxfs/darwin.c | 2 +- libxfs/freebsd.c | 2 +- libxfs/init.c | 2 +- libxfs/irix.c | 2 +- libxfs/libxfs_priv.h | 4 +- libxfs/linux.c | 2 +- libxfs/logitem.c | 2 +- libxfs/radix-tree.c | 8 +- libxfs/rdwr.c | 6 +- libxfs/util.c | 14 ++-- libxfs/xfs_alloc_btree.c | 10 +-- libxfs/xfs_attr_remote.c | 8 +- libxfs/xfs_attr_sf.h | 18 ++--- libxfs/xfs_bit.h | 24 +++--- libxfs/xfs_bmap_btree.c | 8 +- libxfs/xfs_btree.c | 12 +-- libxfs/xfs_btree.h | 14 ++-- libxfs/xfs_cksum.h | 14 ++-- libxfs/xfs_da_btree.c | 2 +- libxfs/xfs_da_btree.h | 8 +- libxfs/xfs_da_format.c | 24 +++--- libxfs/xfs_da_format.h | 60 +++++++-------- libxfs/xfs_dir2.h | 8 +- libxfs/xfs_dir2_leaf.c | 12 +-- libxfs/xfs_dir2_priv.h | 2 +- libxfs/xfs_format.h | 98 ++++++++++++------------ libxfs/xfs_fs.h | 12 +-- libxfs/xfs_ialloc.c | 6 +- libxfs/xfs_ialloc_btree.c | 4 +- libxfs/xfs_inode_buf.c | 2 +- libxfs/xfs_log_format.h | 190 +++++++++++++++++++++++----------------------- libxfs/xfs_quota_defs.h | 4 +- libxfs/xfs_rtbitmap.c | 2 +- libxfs/xfs_sb.c | 4 +- libxfs/xfs_types.h | 46 +++++------ logprint/log_misc.c | 6 +- mdrestore/xfs_mdrestore.c | 2 +- mkfs/proto.c | 2 +- mkfs/xfs_mkfs.c | 88 ++++++++++----------- quota/edit.c | 64 ++++++++-------- quota/free.c | 44 +++++------ quota/quot.c | 24 +++--- quota/quota.c | 4 +- quota/quota.h | 12 +-- quota/report.c | 2 +- quota/state.c | 2 +- quota/util.c | 66 ++++++++-------- repair/README | 2 +- repair/agheader.h | 14 ++-- repair/attr_repair.h | 26 +++---- repair/avl64.c | 38 +++++----- repair/avl64.h | 18 ++--- repair/dino_chunks.c | 2 +- repair/dinode.c | 38 +++++----- repair/dinode.h | 4 +- repair/dir2.c | 2 +- repair/globals.h | 16 ++-- repair/incore.c | 10 +-- repair/incore.h | 42 +++++----- repair/incore_ext.c | 6 +- repair/incore_ino.c | 72 +++++++++--------- repair/phase2.c | 2 +- repair/phase3.c | 6 +- repair/phase4.c | 4 +- repair/phase5.c | 40 +++++----- repair/phase6.c | 18 ++--- repair/phase7.c | 6 +- repair/progress.c | 30 ++++---- repair/progress.h | 4 +- repair/sb.c | 10 +-- repair/scan.c | 42 +++++----- repair/scan.h | 12 +-- 103 files changed, 870 insertions(+), 883 deletions(-) diff --git a/copy/xfs_copy.c b/copy/xfs_copy.c index 15a3f33..425cd7a 100644 --- a/copy/xfs_copy.c +++ b/copy/xfs_copy.c @@ -43,7 +43,7 @@ unsigned int source_sectorsize; /* source disk sectorsize */ xfs_agblock_t first_agbno; -__uint64_t barcount[11]; +uint64_t barcount[11]; unsigned int num_targets; target_control *target; @@ -303,7 +303,7 @@ usage(void) } void -init_bar(__uint64_t source_blocks) +init_bar(uint64_t source_blocks) { int i; @@ -312,7 +312,7 @@ init_bar(__uint64_t source_blocks) } int -bump_bar(int tenths, __uint64_t numblocks) +bump_bar(int tenths, uint64_t numblocks) { static char *bar[11] = { " 0% ", @@ -522,8 +522,8 @@ main(int argc, char **argv) xfs_off_t pos; size_t length; int c; - __uint64_t size, sizeb; - __uint64_t numblocks = 0; + uint64_t size, sizeb; + uint64_t numblocks = 0; int wblocks = 0; int num_threads = 0; struct dioattr d; @@ -939,8 +939,8 @@ main(int argc, char **argv) num_ags = mp->m_sb.sb_agcount; init_bar(mp->m_sb.sb_blocksize / BBSIZE - * ((__uint64_t)mp->m_sb.sb_dblocks - - (__uint64_t)mp->m_sb.sb_fdblocks + 10 * num_ags)); + * ((uint64_t)mp->m_sb.sb_dblocks + - (uint64_t)mp->m_sb.sb_fdblocks + 10 * num_ags)); kids = num_targets; diff --git a/copy/xfs_copy.h b/copy/xfs_copy.h index 2737068..53c6e42 100644 --- a/copy/xfs_copy.h +++ b/copy/xfs_copy.h @@ -76,7 +76,7 @@ typedef struct { typedef int thread_id; typedef int tm_index; /* index into thread mask array */ -typedef __uint32_t thread_mask; /* a thread mask */ +typedef uint32_t thread_mask; /* a thread mask */ typedef struct { char *name; diff --git a/db/attr.c b/db/attr.c index e26ac67..6f56953 100644 --- a/db/attr.c +++ b/db/attr.c @@ -77,16 +77,16 @@ const field_t attr_leaf_entry_flds[] = { { "nameidx", FLDT_UINT16D, OI(LEOFF(nameidx)), C1, 0, TYP_NONE }, { "flags", FLDT_UINT8X, OI(LEOFF(flags)), C1, FLD_SKIPALL, TYP_NONE }, { "incomplete", FLDT_UINT1, - OI(LEOFF(flags) + bitsz(__uint8_t) - XFS_ATTR_INCOMPLETE_BIT - 1), C1, + OI(LEOFF(flags) + bitsz(uint8_t) - XFS_ATTR_INCOMPLETE_BIT - 1), C1, 0, TYP_NONE }, { "root", FLDT_UINT1, - OI(LEOFF(flags) + bitsz(__uint8_t) - XFS_ATTR_ROOT_BIT - 1), C1, 0, + OI(LEOFF(flags) + bitsz(uint8_t) - XFS_ATTR_ROOT_BIT - 1), C1, 0, TYP_NONE }, { "secure", FLDT_UINT1, - OI(LEOFF(flags) + bitsz(__uint8_t) - XFS_ATTR_SECURE_BIT - 1), C1, 0, + OI(LEOFF(flags) + bitsz(uint8_t) - XFS_ATTR_SECURE_BIT - 1), C1, 0, TYP_NONE }, { "local", FLDT_UINT1, - OI(LEOFF(flags) + bitsz(__uint8_t) - XFS_ATTR_LOCAL_BIT - 1), C1, 0, + OI(LEOFF(flags) + bitsz(uint8_t) - XFS_ATTR_LOCAL_BIT - 1), C1, 0, TYP_NONE }, { "pad2", FLDT_UINT8X, OI(LEOFF(pad2)), C1, FLD_SKIPALL, TYP_NONE }, { NULL } diff --git a/db/attrshort.c b/db/attrshort.c index d82559c..2ef358f 100644 --- a/db/attrshort.c +++ b/db/attrshort.c @@ -51,10 +51,10 @@ const field_t attr_sf_entry_flds[] = { { "valuelen", FLDT_UINT8D, OI(EOFF(valuelen)), C1, 0, TYP_NONE }, { "flags", FLDT_UINT8X, OI(EOFF(flags)), C1, FLD_SKIPALL, TYP_NONE }, { "root", FLDT_UINT1, - OI(EOFF(flags) + bitsz(__uint8_t) - XFS_ATTR_ROOT_BIT - 1), C1, 0, + OI(EOFF(flags) + bitsz(uint8_t) - XFS_ATTR_ROOT_BIT - 1), C1, 0, TYP_NONE }, { "secure", FLDT_UINT1, - OI(EOFF(flags) + bitsz(__uint8_t) - XFS_ATTR_SECURE_BIT - 1), C1, 0, + OI(EOFF(flags) + bitsz(uint8_t) - XFS_ATTR_SECURE_BIT - 1), C1, 0, TYP_NONE }, { "name", FLDT_CHARNS, OI(EOFF(nameval)), attr_sf_entry_name_count, FLD_COUNT, TYP_NONE }, diff --git a/db/bit.c b/db/bit.c index 24872bf..53cbcb3 100644 --- a/db/bit.c +++ b/db/bit.c @@ -60,7 +60,7 @@ setbit( } } -__int64_t +int64_t getbitval( void *obj, int bitoff, @@ -70,7 +70,7 @@ getbitval( int bit; int i; char *p; - __int64_t rval; + int64_t rval; int signext; int z1, z2, z3, z4; diff --git a/db/bit.h b/db/bit.h index 80ba24c..7601066 100644 --- a/db/bit.h +++ b/db/bit.h @@ -25,5 +25,5 @@ #define BVUNSIGNED 0 #define BVSIGNED 1 -extern __int64_t getbitval(void *obj, int bitoff, int nbits, int flags); +extern int64_t getbitval(void *obj, int bitoff, int nbits, int flags); extern void setbitval(void *obuf, int bitoff, int nbits, void *ibuf); diff --git a/db/block.c b/db/block.c index 4a357ce..77b9054 100644 --- a/db/block.c +++ b/db/block.c @@ -98,7 +98,7 @@ ablock_f( } dfsbno = bm.startblock + (bno - bm.startoff); ASSERT(typtab[TYP_ATTR].typnm == TYP_ATTR); - set_cur(&typtab[TYP_ATTR], (__int64_t)XFS_FSB_TO_DADDR(mp, dfsbno), + set_cur(&typtab[TYP_ATTR], (int64_t)XFS_FSB_TO_DADDR(mp, dfsbno), blkbb, DB_RING_ADD, NULL); return 0; } @@ -128,14 +128,14 @@ daddr_f( int argc, char **argv) { - __int64_t d; + int64_t d; char *p; if (argc == 1) { dbprintf(_("current daddr is %lld\n"), iocur_top->off >> BBSHIFT); return 0; } - d = (__int64_t)strtoull(argv[1], &p, 0); + d = (int64_t)strtoull(argv[1], &p, 0); if (*p != '\0' || d >= mp->m_sb.sb_dblocks << (mp->m_sb.sb_blocklog - BBSHIFT)) { dbprintf(_("bad daddr %s\n"), argv[1]); @@ -197,7 +197,7 @@ dblock_f( ASSERT(typtab[type].typnm == type); if (nex > 1) make_bbmap(&bbmap, nex, bmp); - set_cur(&typtab[type], (__int64_t)XFS_FSB_TO_DADDR(mp, dfsbno), + set_cur(&typtab[type], (int64_t)XFS_FSB_TO_DADDR(mp, dfsbno), nb * blkbb, DB_RING_ADD, nex > 1 ? &bbmap : NULL); free(bmp); return 0; diff --git a/db/check.c b/db/check.c index 838db53..8ffa156 100644 --- a/db/check.c +++ b/db/check.c @@ -112,8 +112,8 @@ typedef struct dirhash { static xfs_extlen_t agffreeblks; static xfs_extlen_t agflongest; -static __uint64_t agf_aggr_freeblks; /* aggregate count over all */ -static __uint32_t agfbtreeblks; +static uint64_t agf_aggr_freeblks; /* aggregate count over all */ +static uint32_t agfbtreeblks; static int lazycount; static xfs_agino_t agicount; static xfs_agino_t agifreecount; @@ -122,10 +122,10 @@ static int blist_size; static char **dbmap; /* really dbm_t:8 */ static dirhash_t **dirhash; static int error; -static __uint64_t fdblocks; -static __uint64_t frextents; -static __uint64_t icount; -static __uint64_t ifree; +static uint64_t fdblocks; +static uint64_t frextents; +static uint64_t icount; +static uint64_t ifree; static inodata_t ***inodata; static int inodata_hash_size; static inodata_t ***inomap; @@ -1173,7 +1173,7 @@ blocktrash_f( goto out; } for (i = 0; i < count; i++) { - randb = (xfs_rfsblock_t)((((__int64_t)random() << 32) | + randb = (xfs_rfsblock_t)((((int64_t)random() << 32) | random()) % blocks); for (bi = 0, agno = 0, done = 0; !done && agno < mp->m_sb.sb_agcount; @@ -2991,7 +2991,7 @@ process_leaf_node_dir_v2( } if (v) dbprintf(_("dir inode %lld block %u=%llu\n"), id->ino, - (__uint32_t)dbno, + (uint32_t)dbno, (xfs_fsblock_t)bmp->startblock); push_cur(); if (nex > 1) @@ -3004,7 +3004,7 @@ process_leaf_node_dir_v2( if (!sflag || v) dbprintf(_("can't read block %u for directory " "inode %lld\n"), - (__uint32_t)dbno, id->ino); + (uint32_t)dbno, id->ino); error++; pop_cur(); dbno += mp->m_dir_geo->fsbcount - 1; diff --git a/db/convert.c b/db/convert.c index a337abe..fa4f962 100644 --- a/db/convert.c +++ b/db/convert.c @@ -24,13 +24,13 @@ #define M(A) (1 << CT_ ## A) #define agblock_to_bytes(x) \ - ((__uint64_t)(x) << mp->m_sb.sb_blocklog) + ((uint64_t)(x) << mp->m_sb.sb_blocklog) #define agino_to_bytes(x) \ - ((__uint64_t)(x) << mp->m_sb.sb_inodelog) + ((uint64_t)(x) << mp->m_sb.sb_inodelog) #define agnumber_to_bytes(x) \ - agblock_to_bytes((__uint64_t)(x) * mp->m_sb.sb_agblocks) + agblock_to_bytes((uint64_t)(x) * mp->m_sb.sb_agblocks) #define daddr_to_bytes(x) \ - ((__uint64_t)(x) << BBSHIFT) + ((uint64_t)(x) << BBSHIFT) #define fsblock_to_bytes(x) \ (agnumber_to_bytes(XFS_FSB_TO_AGNO(mp, (x))) + \ agblock_to_bytes(XFS_FSB_TO_AGBNO(mp, (x)))) @@ -38,7 +38,7 @@ (agnumber_to_bytes(XFS_INO_TO_AGNO(mp, (x))) + \ agino_to_bytes(XFS_INO_TO_AGINO(mp, (x)))) #define inoidx_to_bytes(x) \ - ((__uint64_t)(x) << mp->m_sb.sb_inodelog) + ((uint64_t)(x) << mp->m_sb.sb_inodelog) typedef enum { CT_NONE = -1, @@ -68,7 +68,7 @@ typedef union { xfs_agnumber_t agnumber; int bboff; int blkoff; - __uint64_t byte; + uint64_t byte; xfs_daddr_t daddr; xfs_fsblock_t fsblock; xfs_ino_t ino; @@ -76,7 +76,7 @@ typedef union { int inooff; } cval_t; -static __uint64_t bytevalue(ctype_t ctype, cval_t *val); +static uint64_t bytevalue(ctype_t ctype, cval_t *val); static int convert_f(int argc, char **argv); static int getvalue(char *s, ctype_t ctype, cval_t *val); static ctype_t lookupcty(char *ctyname); @@ -118,7 +118,7 @@ static const cmdinfo_t convert_cmd = { "convert", NULL, convert_f, 3, 9, 0, "type num [type num]... type", "convert from one address form to another", NULL }; -static __uint64_t +static uint64_t bytevalue(ctype_t ctype, cval_t *val) { switch (ctype) { @@ -129,9 +129,9 @@ bytevalue(ctype_t ctype, cval_t *val) case CT_AGNUMBER: return agnumber_to_bytes(val->agnumber); case CT_BBOFF: - return (__uint64_t)val->bboff; + return (uint64_t)val->bboff; case CT_BLKOFF: - return (__uint64_t)val->blkoff; + return (uint64_t)val->blkoff; case CT_BYTE: return val->byte; case CT_DADDR: @@ -143,7 +143,7 @@ bytevalue(ctype_t ctype, cval_t *val) case CT_INOIDX: return inoidx_to_bytes(val->inoidx); case CT_INOOFF: - return (__uint64_t)val->inooff; + return (uint64_t)val->inooff; case CT_NONE: case NCTS: break; @@ -160,7 +160,7 @@ convert_f(int argc, char **argv) cval_t cvals[NCTS] = {}; int i; int mask; - __uint64_t v; + uint64_t v; ctype_t wtype; /* move past the "convert" command */ @@ -262,7 +262,7 @@ static int getvalue(char *s, ctype_t ctype, cval_t *val) { char *p; - __uint64_t v; + uint64_t v; v = strtoull(s, &p, 0); if (*p != '\0') { @@ -286,7 +286,7 @@ getvalue(char *s, ctype_t ctype, cval_t *val) val->blkoff = (int)v; break; case CT_BYTE: - val->byte = (__uint64_t)v; + val->byte = (uint64_t)v; break; case CT_DADDR: val->daddr = (xfs_daddr_t)v; diff --git a/db/faddr.c b/db/faddr.c index e368d0d..6ded295 100644 --- a/db/faddr.c +++ b/db/faddr.c @@ -79,11 +79,11 @@ fa_attrblock( typnm_t next) { bmap_ext_t bm; - __uint32_t bno; + uint32_t bno; xfs_fsblock_t dfsbno; int nex; - bno = (__uint32_t)getbitval(obj, bit, bitsz(bno), BVUNSIGNED); + bno = (uint32_t)getbitval(obj, bit, bitsz(bno), BVUNSIGNED); if (bno == 0) { dbprintf(_("null attribute block number, cannot set new addr\n")); return; @@ -96,7 +96,7 @@ fa_attrblock( } dfsbno = bm.startblock + (bno - bm.startoff); ASSERT(typtab[next].typnm == next); - set_cur(&typtab[next], (__int64_t)XFS_FSB_TO_DADDR(mp, dfsbno), blkbb, + set_cur(&typtab[next], (int64_t)XFS_FSB_TO_DADDR(mp, dfsbno), blkbb, DB_RING_ADD, NULL); } @@ -276,11 +276,11 @@ fa_dirblock( { bbmap_t bbmap; bmap_ext_t *bmp; - __uint32_t bno; + uint32_t bno; xfs_fsblock_t dfsbno; int nex; - bno = (__uint32_t)getbitval(obj, bit, bitsz(bno), BVUNSIGNED); + bno = (uint32_t)getbitval(obj, bit, bitsz(bno), BVUNSIGNED); if (bno == 0) { dbprintf(_("null directory block number, cannot set new addr\n")); return; @@ -297,7 +297,7 @@ fa_dirblock( ASSERT(typtab[next].typnm == next); if (nex > 1) make_bbmap(&bbmap, nex, bmp); - set_cur(&typtab[next], (__int64_t)XFS_FSB_TO_DADDR(mp, dfsbno), + set_cur(&typtab[next], (int64_t)XFS_FSB_TO_DADDR(mp, dfsbno), XFS_FSB_TO_BB(mp, mp->m_dir_geo->fsbcount), DB_RING_ADD, nex > 1 ? &bbmap : NULL); free(bmp); @@ -317,7 +317,7 @@ fa_drfsbno( return; } ASSERT(typtab[next].typnm == next); - set_cur(&typtab[next], (__int64_t)XFS_FSB_TO_BB(mp, bno), blkbb, + set_cur(&typtab[next], (int64_t)XFS_FSB_TO_BB(mp, bno), blkbb, DB_RING_ADD, NULL); } diff --git a/db/field.c b/db/field.c index 843c385..2c8d414 100644 --- a/db/field.c +++ b/db/field.c @@ -83,7 +83,7 @@ const ftattr_t ftattrtab[] = { attr_sf_entry_size, FTARG_SIZE, NULL, attr_sf_entry_flds }, { FLDT_ATTR_SF_HDR, "attr_sf_hdr", NULL, (char *)attr_sf_hdr_flds, SI(bitsz(struct xfs_attr_sf_hdr)), 0, NULL, attr_sf_hdr_flds }, - { FLDT_ATTRBLOCK, "attrblock", fp_num, "%u", SI(bitsz(__uint32_t)), 0, + { FLDT_ATTRBLOCK, "attrblock", fp_num, "%u", SI(bitsz(uint32_t)), 0, fa_attrblock, NULL }, { FLDT_ATTRSHORT, "attrshort", NULL, (char *)attr_shortform_flds, attrshort_size, FTARG_SIZE, NULL, attr_shortform_flds }, @@ -165,7 +165,7 @@ const ftattr_t ftattrtab[] = { SI(bitsz(xfs_alloc_rec_t)), 0, NULL, cntbt_rec_flds }, /* CRC field */ - { FLDT_CRC, "crc", fp_crc, "%#x (%s)", SI(bitsz(__uint32_t)), + { FLDT_CRC, "crc", fp_crc, "%#x (%s)", SI(bitsz(uint32_t)), 0, NULL, NULL }, { FLDT_DEV, "dev", fp_num, "%#x", SI(bitsz(xfs_dev_t)), 0, NULL, NULL }, @@ -180,7 +180,7 @@ const ftattr_t ftattrtab[] = { { FLDT_DINODE_CORE, "dinode_core", NULL, (char *)inode_core_flds, SI(bitsz(xfs_dinode_t)), 0, NULL, inode_core_flds }, { FLDT_DINODE_FMT, "dinode_fmt", fp_dinode_fmt, NULL, - SI(bitsz(__int8_t)), 0, NULL, NULL }, + SI(bitsz(int8_t)), 0, NULL, NULL }, { FLDT_DINODE_U, "dinode_u", NULL, (char *)inode_u_flds, inode_u_size, FTARG_SIZE|FTARG_OKEMPTY, NULL, inode_u_flds }, { FLDT_DINODE_V3, "dinode_v3", NULL, (char *)inode_v3_flds, @@ -261,7 +261,7 @@ const ftattr_t ftattrtab[] = { { FLDT_DA3_NODE_HDR, "dir_node_hdr", NULL, (char *)da3_node_hdr_flds, SI(bitsz(struct xfs_da3_node_hdr)), 0, NULL, da3_node_hdr_flds }, - { FLDT_DIRBLOCK, "dirblock", fp_num, "%u", SI(bitsz(__uint32_t)), 0, + { FLDT_DIRBLOCK, "dirblock", fp_num, "%u", SI(bitsz(uint32_t)), 0, fa_dirblock, NULL }, { FLDT_DISK_DQUOT, "disk_dquot", NULL, (char *)disk_dquot_flds, SI(bitsz(xfs_disk_dquot_t)), 0, NULL, disk_dquot_flds }, @@ -301,15 +301,15 @@ const ftattr_t ftattrtab[] = { FTARG_SIZE, NULL, inode_crc_flds }, { FLDT_INOFREE, "inofree", fp_num, "%#llx", SI(bitsz(xfs_inofree_t)), 0, NULL, NULL }, - { FLDT_INT16D, "int16d", fp_num, "%d", SI(bitsz(__int16_t)), + { FLDT_INT16D, "int16d", fp_num, "%d", SI(bitsz(int16_t)), FTARG_SIGNED, NULL, NULL }, - { FLDT_INT32D, "int32d", fp_num, "%d", SI(bitsz(__int32_t)), + { FLDT_INT32D, "int32d", fp_num, "%d", SI(bitsz(int32_t)), FTARG_SIGNED, NULL, NULL }, - { FLDT_INT64D, "int64d", fp_num, "%lld", SI(bitsz(__int64_t)), + { FLDT_INT64D, "int64d", fp_num, "%lld", SI(bitsz(int64_t)), FTARG_SIGNED, NULL, NULL }, - { FLDT_INT8D, "int8d", fp_num, "%d", SI(bitsz(__int8_t)), FTARG_SIGNED, + { FLDT_INT8D, "int8d", fp_num, "%d", SI(bitsz(int8_t)), FTARG_SIGNED, NULL, NULL }, - { FLDT_NSEC, "nsec", fp_num, "%09d", SI(bitsz(__int32_t)), FTARG_SIGNED, + { FLDT_NSEC, "nsec", fp_num, "%09d", SI(bitsz(int32_t)), FTARG_SIGNED, NULL, NULL }, { FLDT_QCNT, "qcnt", fp_num, "%llu", SI(bitsz(xfs_qcnt_t)), 0, NULL, NULL }, @@ -322,34 +322,34 @@ const ftattr_t ftattrtab[] = { { FLDT_SYMLINK_CRC, "symlink", NULL, (char *)symlink_crc_flds, symlink_size, FTARG_SIZE, NULL, symlink_crc_flds }, - { FLDT_TIME, "time", fp_time, NULL, SI(bitsz(__int32_t)), FTARG_SIGNED, + { FLDT_TIME, "time", fp_time, NULL, SI(bitsz(int32_t)), FTARG_SIGNED, NULL, NULL }, { FLDT_TIMESTAMP, "timestamp", NULL, (char *)timestamp_flds, SI(bitsz(xfs_timestamp_t)), 0, NULL, timestamp_flds }, { FLDT_UINT1, "uint1", fp_num, "%u", SI(1), 0, NULL, NULL }, - { FLDT_UINT16D, "uint16d", fp_num, "%u", SI(bitsz(__uint16_t)), 0, NULL, + { FLDT_UINT16D, "uint16d", fp_num, "%u", SI(bitsz(uint16_t)), 0, NULL, NULL }, - { FLDT_UINT16O, "uint16o", fp_num, "%#o", SI(bitsz(__uint16_t)), 0, + { FLDT_UINT16O, "uint16o", fp_num, "%#o", SI(bitsz(uint16_t)), 0, NULL, NULL }, - { FLDT_UINT16X, "uint16x", fp_num, "%#x", SI(bitsz(__uint16_t)), 0, + { FLDT_UINT16X, "uint16x", fp_num, "%#x", SI(bitsz(uint16_t)), 0, NULL, NULL }, - { FLDT_UINT32D, "uint32d", fp_num, "%u", SI(bitsz(__uint32_t)), 0, NULL, + { FLDT_UINT32D, "uint32d", fp_num, "%u", SI(bitsz(uint32_t)), 0, NULL, NULL }, - { FLDT_UINT32O, "uint32o", fp_num, "%#o", SI(bitsz(__uint32_t)), 0, + { FLDT_UINT32O, "uint32o", fp_num, "%#o", SI(bitsz(uint32_t)), 0, NULL, NULL }, - { FLDT_UINT32X, "uint32x", fp_num, "%#x", SI(bitsz(__uint32_t)), 0, + { FLDT_UINT32X, "uint32x", fp_num, "%#x", SI(bitsz(uint32_t)), 0, NULL, NULL }, - { FLDT_UINT64D, "uint64d", fp_num, "%llu", SI(bitsz(__uint64_t)), 0, + { FLDT_UINT64D, "uint64d", fp_num, "%llu", SI(bitsz(uint64_t)), 0, NULL, NULL }, - { FLDT_UINT64O, "uint64o", fp_num, "%#llo", SI(bitsz(__uint64_t)), 0, + { FLDT_UINT64O, "uint64o", fp_num, "%#llo", SI(bitsz(uint64_t)), 0, NULL, NULL }, - { FLDT_UINT64X, "uint64x", fp_num, "%#llx", SI(bitsz(__uint64_t)), 0, + { FLDT_UINT64X, "uint64x", fp_num, "%#llx", SI(bitsz(uint64_t)), 0, NULL, NULL }, - { FLDT_UINT8D, "uint8d", fp_num, "%u", SI(bitsz(__uint8_t)), 0, NULL, + { FLDT_UINT8D, "uint8d", fp_num, "%u", SI(bitsz(uint8_t)), 0, NULL, NULL }, - { FLDT_UINT8O, "uint8o", fp_num, "%#o", SI(bitsz(__uint8_t)), 0, NULL, + { FLDT_UINT8O, "uint8o", fp_num, "%#o", SI(bitsz(uint8_t)), 0, NULL, NULL }, - { FLDT_UINT8X, "uint8x", fp_num, "%#x", SI(bitsz(__uint8_t)), 0, NULL, + { FLDT_UINT8X, "uint8x", fp_num, "%#x", SI(bitsz(uint8_t)), 0, NULL, NULL }, { FLDT_UUID, "uuid", fp_uuid, NULL, SI(bitsz(uuid_t)), 0, NULL, NULL }, { FLDT_ZZZ, NULL } diff --git a/db/fprint.c b/db/fprint.c index f2b3794..12b1574 100644 --- a/db/fprint.c +++ b/db/fprint.c @@ -80,7 +80,7 @@ fp_num( int bitpos; int i; int isnull; - __int64_t val; + int64_t val; for (i = 0, bitpos = bit; i < count && !seenint(); @@ -100,7 +100,7 @@ fp_num( else if (size > 32) dbprintf(fmtstr, val); else - dbprintf(fmtstr, (__int32_t)val); + dbprintf(fmtstr, (int32_t)val); if (i < count - 1) dbprintf(" "); } @@ -203,7 +203,7 @@ fp_crc( { int bitpos; int i; - __int64_t val; + int64_t val; char *ok; switch (iocur_crc_valid()) { @@ -230,7 +230,7 @@ fp_crc( if (size > 32) dbprintf(fmtstr, val, ok); else - dbprintf(fmtstr, (__int32_t)val, ok); + dbprintf(fmtstr, (int32_t)val, ok); if (i < count - 1) dbprintf(" "); } diff --git a/db/frag.c b/db/frag.c index 3beb416..c3d1581 100644 --- a/db/frag.c +++ b/db/frag.c @@ -42,8 +42,8 @@ typedef struct extmap { static int aflag; static int dflag; -static __uint64_t extcount_actual; -static __uint64_t extcount_ideal; +static uint64_t extcount_actual; +static uint64_t extcount_ideal; static int fflag; static int lflag; static int qflag; @@ -306,8 +306,8 @@ process_inode( xfs_agino_t agino, xfs_dinode_t *dip) { - __uint64_t actual; - __uint64_t ideal; + uint64_t actual; + uint64_t ideal; xfs_ino_t ino; int skipa; int skipd; diff --git a/db/inode.c b/db/inode.c index 442e6ea..3b2ee1a 100644 --- a/db/inode.c +++ b/db/inode.c @@ -119,49 +119,49 @@ const field_t inode_core_flds[] = { { "dmstate", FLDT_UINT16D, OI(COFF(dmstate)), C1, 0, TYP_NONE }, { "flags", FLDT_UINT16X, OI(COFF(flags)), C1, FLD_SKIPALL, TYP_NONE }, { "newrtbm", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_NEWRTBM_BIT - 1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_NEWRTBM_BIT - 1), C1, 0, TYP_NONE }, { "prealloc", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_PREALLOC_BIT - 1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_PREALLOC_BIT - 1), C1, 0, TYP_NONE }, { "realtime", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_REALTIME_BIT - 1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_REALTIME_BIT - 1), C1, 0, TYP_NONE }, { "immutable", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_IMMUTABLE_BIT-1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_IMMUTABLE_BIT-1), C1, 0, TYP_NONE }, { "append", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_APPEND_BIT - 1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_APPEND_BIT - 1), C1, 0, TYP_NONE }, { "sync", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_SYNC_BIT - 1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_SYNC_BIT - 1), C1, 0, TYP_NONE }, { "noatime", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_NOATIME_BIT - 1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_NOATIME_BIT - 1), C1, 0, TYP_NONE }, { "nodump", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_NODUMP_BIT - 1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_NODUMP_BIT - 1), C1, 0, TYP_NONE }, { "rtinherit", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_RTINHERIT_BIT-1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_RTINHERIT_BIT-1), C1, 0, TYP_NONE }, { "projinherit", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_PROJINHERIT_BIT-1),C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_PROJINHERIT_BIT-1),C1, 0, TYP_NONE }, { "nosymlinks", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_NOSYMLINKS_BIT-1), C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_NOSYMLINKS_BIT-1), C1, 0, TYP_NONE }, { "extsz", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_EXTSIZE_BIT-1),C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_EXTSIZE_BIT-1),C1, 0, TYP_NONE }, { "extszinherit", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_EXTSZINHERIT_BIT-1),C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_EXTSZINHERIT_BIT-1),C1, 0, TYP_NONE }, { "nodefrag", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_NODEFRAG_BIT-1),C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_NODEFRAG_BIT-1),C1, 0, TYP_NONE }, { "filestream", FLDT_UINT1, - OI(COFF(flags) + bitsz(__uint16_t) - XFS_DIFLAG_FILESTREAM_BIT-1),C1, + OI(COFF(flags) + bitsz(uint16_t) - XFS_DIFLAG_FILESTREAM_BIT-1),C1, 0, TYP_NONE }, { "gen", FLDT_UINT32D, OI(COFF(gen)), C1, 0, TYP_NONE }, { NULL } diff --git a/db/io.c b/db/io.c index 9452e07..8fa2504 100644 --- a/db/io.c +++ b/db/io.c @@ -487,7 +487,7 @@ write_cur(void) void set_cur( const typ_t *t, - __int64_t d, + int64_t d, int c, int ring_flag, bbmap_t *bbmap) @@ -495,7 +495,7 @@ set_cur( struct xfs_buf *bp; xfs_ino_t dirino; xfs_ino_t ino; - __uint16_t mode; + uint16_t mode; const struct xfs_buf_ops *ops = t ? t->bops : NULL; if (iocur_sp < 0) { diff --git a/db/io.h b/db/io.h index 6201d7b..4700db3 100644 --- a/db/io.h +++ b/db/io.h @@ -25,7 +25,7 @@ typedef struct bbmap { } bbmap_t; typedef struct iocur { - __int64_t bb; /* BB number in filesystem of buf */ + int64_t bb; /* BB number in filesystem of buf */ int blen; /* length of "buf", bb's */ int boff; /* data - buf */ void *buf; /* base address of buffer */ @@ -33,7 +33,7 @@ typedef struct iocur { xfs_ino_t dirino; /* current directory inode number */ xfs_ino_t ino; /* current inode number */ int len; /* length of "data", bytes */ - __uint16_t mode; /* current inode's mode */ + uint16_t mode; /* current inode's mode */ xfs_off_t off; /* fs offset of "data" in bytes */ const struct typ *typ; /* type of "data" */ bbmap_t *bbmap; /* map daddr if fragmented */ @@ -57,9 +57,9 @@ extern void off_cur(int off, int len); extern void pop_cur(void); extern void print_iocur(char *tag, iocur_t *ioc); extern void push_cur(void); -extern int read_buf(__int64_t daddr, int count, void *bufp); +extern int read_buf(int64_t daddr, int count, void *bufp); extern void write_cur(void); -extern void set_cur(const struct typ *t, __int64_t d, int c, int ring_add, +extern void set_cur(const struct typ *t, int64_t d, int c, int ring_add, bbmap_t *bbmap); extern void ring_add(void); extern void set_iocur_type(const struct typ *t); diff --git a/db/metadump.c b/db/metadump.c index 8455fdd..0bb6e3d 100644 --- a/db/metadump.c +++ b/db/metadump.c @@ -175,7 +175,7 @@ write_index(void) static int write_buf_segment( char *data, - __int64_t off, + int64_t off, int len) { int i; @@ -1093,7 +1093,7 @@ process_sf_dir( { struct xfs_dir2_sf_hdr *sfp; xfs_dir2_sf_entry_t *sfep; - __uint64_t ino_dir_size; + uint64_t ino_dir_size; int i; sfp = (struct xfs_dir2_sf_hdr *)XFS_DFORK_DPTR(dip); @@ -1159,7 +1159,7 @@ process_sf_dir( static void obfuscate_path_components( char *buf, - __uint64_t len) + uint64_t len) { unsigned char *comp = (unsigned char *)buf; unsigned char *end = comp + len; @@ -1196,7 +1196,7 @@ static void process_sf_symlink( xfs_dinode_t *dip) { - __uint64_t len; + uint64_t len; char *buf; len = be64_to_cpu(dip->di_size); @@ -1468,7 +1468,7 @@ process_attr_block( xfs_attr_leaf_entry_t *entry; xfs_attr_leaf_name_local_t *local; xfs_attr_leaf_name_remote_t *remote; - __uint32_t bs = mp->m_sb.sb_blocksize; + uint32_t bs = mp->m_sb.sb_blocksize; char *first_name; @@ -2106,7 +2106,7 @@ done: return success; } -static __uint32_t inodes_copied = 0; +static uint32_t inodes_copied = 0; static int copy_inode_chunk( diff --git a/db/sb.c b/db/sb.c index 79a3c1d..e870aa6 100644 --- a/db/sb.c +++ b/db/sb.c @@ -606,7 +606,7 @@ version_help(void) } static int -do_version(xfs_agnumber_t agno, __uint16_t version, __uint32_t features) +do_version(xfs_agnumber_t agno, uint16_t version, uint32_t features) { xfs_sb_t tsb; @@ -708,8 +708,8 @@ version_f( int argc, char **argv) { - __uint16_t version = 0; - __uint32_t features = 0; + uint16_t version = 0; + uint32_t features = 0; xfs_agnumber_t ag; if (argc == 2) { /* WRITE VERSION */ diff --git a/fsr/xfs_fsr.c b/fsr/xfs_fsr.c index 424fbce..f32aced 100644 --- a/fsr/xfs_fsr.c +++ b/fsr/xfs_fsr.c @@ -57,7 +57,7 @@ struct getbmap *outmap = NULL; int outmap_size = 0; int RealUid; int tmp_agi; -static __int64_t minimumfree = 2048; +static int64_t minimumfree = 2048; #define MNTTYPE_XFS "xfs" diff --git a/include/darwin.h b/include/darwin.h index dd6132f..8a85878 100644 --- a/include/darwin.h +++ b/include/darwin.h @@ -125,15 +125,6 @@ typedef signed int __s32; typedef unsigned long long int __u64; typedef signed long long int __s64; -#define __int8_t int8_t -#define __int16_t int16_t -#define __int32_t int32_t -#define __int32_t int32_t -#define __int64_t int64_t -#define __uint8_t u_int8_t -#define __uint16_t u_int16_t -#define __uint32_t u_int32_t -#define __uint64_t u_int64_t #define off64_t off_t typedef off_t xfs_off_t; diff --git a/include/freebsd.h b/include/freebsd.h index 65bd60a..e3e995f 100644 --- a/include/freebsd.h +++ b/include/freebsd.h @@ -50,9 +50,9 @@ typedef off_t xfs_off_t; typedef off_t off64_t; -typedef __uint64_t xfs_ino_t; -typedef __uint32_t xfs_dev_t; -typedef __int64_t xfs_daddr_t; +typedef uint64_t xfs_ino_t; +typedef uint32_t xfs_dev_t; +typedef int64_t xfs_daddr_t; typedef unsigned char __u8; typedef signed char __s8; diff --git a/include/gnukfreebsd.h b/include/gnukfreebsd.h index 64167b2..b2d2cf3 100644 --- a/include/gnukfreebsd.h +++ b/include/gnukfreebsd.h @@ -39,9 +39,9 @@ #define EFSBADCRC 991 /* Bad CRC detected */ typedef off_t xfs_off_t; -typedef __uint64_t xfs_ino_t; -typedef __uint32_t xfs_dev_t; -typedef __int64_t xfs_daddr_t; +typedef uint64_t xfs_ino_t; +typedef uint32_t xfs_dev_t; +typedef int64_t xfs_daddr_t; typedef unsigned char __u8; typedef signed char __s8; diff --git a/include/irix.h b/include/irix.h index 293f869..d6aa5e1 100644 --- a/include/irix.h +++ b/include/irix.h @@ -39,14 +39,10 @@ #include #include -#define __int8_t char -#define __int16_t short -#define __uint8_t unsigned char -#define __uint16_t unsigned short typedef off64_t xfs_off_t; -typedef __int64_t xfs_ino_t; -typedef __int32_t xfs_dev_t; -typedef __int64_t xfs_daddr_t; +typedef int64_t xfs_ino_t; +typedef int32_t xfs_dev_t; +typedef int64_t xfs_daddr_t; typedef unsigned char __u8; typedef signed char __s8; @@ -63,17 +59,17 @@ typedef signed long long int __s64; #define EFSBADCRC 991 /* Bad CRC detected */ typedef struct xfs_error_injection { - __int32_t fd; - __int32_t errtag; + int32_t fd; + int32_t errtag; } xfs_error_injection_t; /* --- xfs_fsop_*req - request data structures --- */ typedef struct xfs_fsop_bulkreq { ino64_t *lastip; - __int32_t icount; + int32_t icount; xfs_bstat_t *ubuffer; - __int32_t *ocount; + int32_t *ocount; } xfs_fsop_bulkreq_t; typedef struct xfs_fsop_handlereq { @@ -302,13 +298,13 @@ static __inline__ void platform_getoptreset(void) static __inline__ int platform_uuid_compare(uuid_t *uu1, uuid_t *uu2) { - __uint32_t status; + uint32_t status; return uuid_compare(uu1, uu2, &status); } static __inline__ void platform_uuid_unparse(uuid_t *uu, char *buffer) { - __uint32_t status; + uint32_t status; char *s; uuid_to_string(uu, &s, &status); if (status == uuid_s_ok) @@ -319,7 +315,7 @@ static __inline__ void platform_uuid_unparse(uuid_t *uu, char *buffer) static __inline__ int platform_uuid_parse(char *buffer, uuid_t *uu) { - __uint32_t status; + uint32_t status; uuid_from_string(buffer, uu, &status); return (status == uuid_s_ok); } @@ -332,13 +328,13 @@ static __inline__ int platform_uuid_is_null(uuid_t *uu) static __inline__ void platform_uuid_generate(uuid_t *uu) { - __uint32_t status; + uint32_t status; uuid_create(uu, &status); } static __inline__ void platform_uuid_clear(uuid_t *uu) { - __uint32_t status; + uint32_t status; uuid_create_nil(uu, &status); } diff --git a/include/linux.h b/include/linux.h index 674717c..827dbe1 100644 --- a/include/linux.h +++ b/include/linux.h @@ -124,7 +124,7 @@ static __inline__ void platform_uuid_copy(uuid_t *dst, uuid_t *src) static __inline__ int platform_discard_blocks(int fd, uint64_t start, uint64_t len) { - __uint64_t range[2] = { start, len }; + uint64_t range[2] = { start, len }; if (ioctl(fd, BLKDISCARD, &range) < 0) return errno; @@ -142,9 +142,9 @@ platform_discard_blocks(int fd, uint64_t start, uint64_t len) #define EFSBADCRC EBADMSG /* Bad CRC detected */ typedef loff_t xfs_off_t; -typedef __uint64_t xfs_ino_t; -typedef __uint32_t xfs_dev_t; -typedef __int64_t xfs_daddr_t; +typedef uint64_t xfs_ino_t; +typedef uint32_t xfs_dev_t; +typedef int64_t xfs_daddr_t; /** * Abstraction of mountpoints. diff --git a/include/xfs_arch.h b/include/xfs_arch.h index 87650f1..c44d993 100644 --- a/include/xfs_arch.h +++ b/include/xfs_arch.h @@ -240,39 +240,39 @@ static inline void be64_add_cpu(__be64 *a, __s64 b) *a = cpu_to_be64(be64_to_cpu(*a) + b); } -static inline __uint16_t get_unaligned_be16(void *p) +static inline uint16_t get_unaligned_be16(void *p) { - __uint8_t *__p = p; + uint8_t *__p = p; return __p[0] << 8 | __p[1]; } -static inline __uint32_t get_unaligned_be32(void *p) +static inline uint32_t get_unaligned_be32(void *p) { - __uint8_t *__p = p; + uint8_t *__p = p; return __p[0] << 24 | __p[1] << 16 | __p[2] << 8 | __p[3]; } -static inline __uint64_t get_unaligned_be64(void *p) +static inline uint64_t get_unaligned_be64(void *p) { - return (__uint64_t)get_unaligned_be32(p) << 32 | + return (uint64_t)get_unaligned_be32(p) << 32 | get_unaligned_be32(p + 4); } -static inline void put_unaligned_be16(__uint16_t val, void *p) +static inline void put_unaligned_be16(uint16_t val, void *p) { - __uint8_t *__p = p; + uint8_t *__p = p; *__p++ = val >> 8; *__p++ = val; } -static inline void put_unaligned_be32(__uint32_t val, void *p) +static inline void put_unaligned_be32(uint32_t val, void *p) { - __uint8_t *__p = p; + uint8_t *__p = p; put_unaligned_be16(val >> 16, __p); put_unaligned_be16(val, __p + 2); } -static inline void put_unaligned_be64(__uint64_t val, void *p) +static inline void put_unaligned_be64(uint64_t val, void *p) { put_unaligned_be32(val >> 32, p); put_unaligned_be32(val, p + 4); diff --git a/include/xfs_inode.h b/include/xfs_inode.h index 71c0fb4..708f8c6 100644 --- a/include/xfs_inode.h +++ b/include/xfs_inode.h @@ -77,8 +77,8 @@ xfs_get_projid(struct xfs_icdinode *id) static inline void xfs_set_projid(struct xfs_icdinode *id, prid_t projid) { - id->di_projid_hi = (__uint16_t) (projid >> 16); - id->di_projid_lo = (__uint16_t) (projid & 0xffff); + id->di_projid_hi = (uint16_t) (projid >> 16); + id->di_projid_lo = (uint16_t) (projid & 0xffff); } typedef struct cred { diff --git a/include/xfs_log_recover.h b/include/xfs_log_recover.h index 2fea63d..69f276c 100644 --- a/include/xfs_log_recover.h +++ b/include/xfs_log_recover.h @@ -26,7 +26,7 @@ #define XLOG_RHASH_SIZE 16 #define XLOG_RHASH_SHIFT 2 #define XLOG_RHASH(tid) \ - ((((__uint32_t)tid)>>XLOG_RHASH_SHIFT) & (XLOG_RHASH_SIZE-1)) + ((((uint32_t)tid)>>XLOG_RHASH_SHIFT) & (XLOG_RHASH_SIZE-1)) #define XLOG_MAX_REGIONS_IN_ITEM (XFS_MAX_BLOCKSIZE / XFS_BLF_CHUNK / 2 + 1) diff --git a/include/xfs_metadump.h b/include/xfs_metadump.h index f4be51b..3ce5138 100644 --- a/include/xfs_metadump.h +++ b/include/xfs_metadump.h @@ -24,8 +24,8 @@ typedef struct xfs_metablock { __be32 mb_magic; __be16 mb_count; - __uint8_t mb_blocklog; - __uint8_t mb_reserved; + uint8_t mb_blocklog; + uint8_t mb_reserved; /* followed by an array of xfs_daddr_t */ } xfs_metablock_t; diff --git a/include/xfs_mount.h b/include/xfs_mount.h index 14b05a6..e0d5f48 100644 --- a/include/xfs_mount.h +++ b/include/xfs_mount.h @@ -48,12 +48,12 @@ typedef struct xfs_mount { #define m_dev m_ddev_targp #define m_logdev m_logdev_targp #define m_rtdev m_rtdev_targp - __uint8_t m_dircook_elog; /* log d-cookie entry bits */ - __uint8_t m_blkbit_log; /* blocklog + NBBY */ - __uint8_t m_blkbb_log; /* blocklog - BBSHIFT */ - __uint8_t m_sectbb_log; /* sectorlog - BBSHIFT */ - __uint8_t m_agno_log; /* log #ag's */ - __uint8_t m_agino_log; /* #bits for agino in inum */ + uint8_t m_dircook_elog; /* log d-cookie entry bits */ + uint8_t m_blkbit_log; /* blocklog + NBBY */ + uint8_t m_blkbb_log; /* blocklog - BBSHIFT */ + uint8_t m_sectbb_log; /* sectorlog - BBSHIFT */ + uint8_t m_agno_log; /* log #ag's */ + uint8_t m_agino_log; /* #bits for agino in inum */ uint m_inode_cluster_size;/* min inode buf size */ uint m_blockmask; /* sb_blocksize-1 */ uint m_blockwsize; /* sb_blocksize in words */ @@ -78,7 +78,7 @@ typedef struct xfs_mount { int m_litino; /* size of inode union area */ int m_inoalign_mask;/* mask sb_inoalignmt if used */ struct xfs_trans_resv m_resv; /* precomputed res values */ - __uint64_t m_maxicount; /* maximum inode count */ + uint64_t m_maxicount; /* maximum inode count */ int m_dalign; /* stripe unit */ int m_swidth; /* stripe width */ int m_sinoalign; /* stripe unit inode alignmnt */ @@ -118,12 +118,12 @@ typedef struct xfs_perag { char pagi_init; /* this agi's entry is initialized */ char pagf_metadata; /* the agf is preferred to be metadata */ char pagi_inodeok; /* The agi is ok for inodes */ - __uint8_t pagf_levels[XFS_BTNUM_AGF]; + uint8_t pagf_levels[XFS_BTNUM_AGF]; /* # of levels in bno & cnt btree */ - __uint32_t pagf_flcount; /* count of blocks in freelist */ + uint32_t pagf_flcount; /* count of blocks in freelist */ xfs_extlen_t pagf_freeblks; /* total free blocks */ xfs_extlen_t pagf_longest; /* longest free space */ - __uint32_t pagf_btreeblks; /* # of blocks held in AGF btrees */ + uint32_t pagf_btreeblks; /* # of blocks held in AGF btrees */ xfs_agino_t pagi_freecount; /* number of free inodes */ xfs_agino_t pagi_count; /* number of allocated inodes */ diff --git a/libhandle/handle.c b/libhandle/handle.c index 236ed22..00127b3 100644 --- a/libhandle/handle.c +++ b/libhandle/handle.c @@ -232,7 +232,7 @@ obj_to_handle( { char hbuf [MAXHANSIZ]; int ret; - __uint32_t handlen; + uint32_t handlen; xfs_fsop_handlereq_t hreq; if (opcode == XFS_IOC_FD_TO_HANDLE) { diff --git a/libhandle/jdm.c b/libhandle/jdm.c index e52f5d8..821061b 100644 --- a/libhandle/jdm.c +++ b/libhandle/jdm.c @@ -36,7 +36,7 @@ typedef struct filehandle { fshandle_t fh_fshandle; /* handle of fs containing this inode */ int16_t fh_sz_following; /* bytes in handle after this member */ char fh_pad[FILEHANDLE_SZ_PAD]; /* padding, must be zeroed */ - __uint32_t fh_gen; /* generation count */ + uint32_t fh_gen; /* generation count */ xfs_ino_t fh_ino; /* 64 bit ino */ } filehandle_t; diff --git a/libxfs/darwin.c b/libxfs/darwin.c index 017e190..1bd6c8a 100644 --- a/libxfs/darwin.c +++ b/libxfs/darwin.c @@ -68,7 +68,7 @@ platform_flush_device(int fd, dev_t device) void platform_findsizes(char *path, int fd, long long *sz, int *bsz) { - __uint64_t size; + uint64_t size; struct stat64 st; if (fstat64(fd, &st) < 0) { diff --git a/libxfs/freebsd.c b/libxfs/freebsd.c index 6c9f089..96f2c67 100644 --- a/libxfs/freebsd.c +++ b/libxfs/freebsd.c @@ -109,7 +109,7 @@ void platform_findsizes(char *path, int fd, long long *sz, int *bsz) { struct stat st; - __int64_t size; + int64_t size; u_int ssize; if (fstat(fd, &st) < 0) { diff --git a/libxfs/init.c b/libxfs/init.c index 8d747e8..4e73e13 100644 --- a/libxfs/init.c +++ b/libxfs/init.c @@ -529,7 +529,7 @@ libxfs_initialize_perag( * the max inode percentage. */ if (mp->m_maxicount) { - __uint64_t icount; + uint64_t icount; icount = sbp->sb_dblocks * sbp->sb_imax_pct; do_div(icount, 100); diff --git a/libxfs/irix.c b/libxfs/irix.c index 65aaa7e..f4e7f47 100644 --- a/libxfs/irix.c +++ b/libxfs/irix.c @@ -22,7 +22,7 @@ int platform_has_uuid = 0; extern char *progname; -extern __int64_t findsize(char *); +extern int64_t findsize(char *); int platform_check_ismounted(char *name, char *block, struct stat64 *s, int verbose) diff --git a/libxfs/libxfs_priv.h b/libxfs/libxfs_priv.h index 0f4d3e5..ffd6bf7 100644 --- a/libxfs/libxfs_priv.h +++ b/libxfs/libxfs_priv.h @@ -307,8 +307,8 @@ roundup_pow_of_two(uint v) return 0; } -static inline __uint64_t -roundup_64(__uint64_t x, __uint32_t y) +static inline uint64_t +roundup_64(uint64_t x, uint32_t y) { x += y - 1; do_div(x, y); diff --git a/libxfs/linux.c b/libxfs/linux.c index f6ea1b2..c532d65 100644 --- a/libxfs/linux.c +++ b/libxfs/linux.c @@ -133,7 +133,7 @@ void platform_findsizes(char *path, int fd, long long *sz, int *bsz) { struct stat64 st; - __uint64_t size; + uint64_t size; int error; if (fstat64(fd, &st) < 0) { diff --git a/libxfs/logitem.c b/libxfs/logitem.c index 7e93f60..466b442 100644 --- a/libxfs/logitem.c +++ b/libxfs/logitem.c @@ -121,7 +121,7 @@ xfs_buf_item_init( bip->bli_item.li_mountp = mp; bip->bli_buf = bp; bip->bli_format.blf_type = XFS_LI_BUF; - bip->bli_format.blf_blkno = (__int64_t)XFS_BUF_ADDR(bp); + bip->bli_format.blf_blkno = (int64_t)XFS_BUF_ADDR(bp); bip->bli_format.blf_len = (unsigned short)BTOBB(XFS_BUF_COUNT(bp)); XFS_BUF_SET_FSPRIVATE(bp, bip); } diff --git a/libxfs/radix-tree.c b/libxfs/radix-tree.c index eef9c36..3f0257f 100644 --- a/libxfs/radix-tree.c +++ b/libxfs/radix-tree.c @@ -66,21 +66,21 @@ static unsigned long height_to_maxindex[RADIX_TREE_MAX_PATH]; static inline void tag_set(struct radix_tree_node *node, unsigned int tag, int offset) { - *((__uint32_t *)node->tags[tag] + (offset >> 5)) |= (1 << (offset & 31)); + *((uint32_t *)node->tags[tag] + (offset >> 5)) |= (1 << (offset & 31)); } static inline void tag_clear(struct radix_tree_node *node, unsigned int tag, int offset) { - __uint32_t *p = (__uint32_t*)node->tags[tag] + (offset >> 5); - __uint32_t m = 1 << (offset & 31); + uint32_t *p = (uint32_t*)node->tags[tag] + (offset >> 5); + uint32_t m = 1 << (offset & 31); *p &= ~m; } static inline int tag_get(struct radix_tree_node *node, unsigned int tag, int offset) { - return 1 & (((const __uint32_t *)node->tags[tag])[offset >> 5] >> (offset & 31)); + return 1 & (((const uint32_t *)node->tags[tag])[offset >> 5] >> (offset & 31)); } /* diff --git a/libxfs/rdwr.c b/libxfs/rdwr.c index 7a04985..5f0ac24 100644 --- a/libxfs/rdwr.c +++ b/libxfs/rdwr.c @@ -116,9 +116,9 @@ static void unmount_record(void *p) xlog_op_header_t *op = (xlog_op_header_t *)p; /* the data section must be 32 bit size aligned */ struct { - __uint16_t magic; - __uint16_t pad1; - __uint32_t pad2; /* may as well make it 64 bits */ + uint16_t magic; + uint16_t pad1; + uint32_t pad2; /* may as well make it 64 bits */ } magic = { XLOG_UNMOUNT_TYPE, 0, 0 }; memset(p, 0, BBSIZE); diff --git a/libxfs/util.c b/libxfs/util.c index 90031fd..8d7baa0 100644 --- a/libxfs/util.c +++ b/libxfs/util.c @@ -160,16 +160,16 @@ libxfs_trans_ichgtime( tv.tv_sec = stv.tv_sec; tv.tv_nsec = stv.tv_usec * 1000; if (flags & XFS_ICHGTIME_MOD) { - ip->i_d.di_mtime.t_sec = (__int32_t)tv.tv_sec; - ip->i_d.di_mtime.t_nsec = (__int32_t)tv.tv_nsec; + ip->i_d.di_mtime.t_sec = (int32_t)tv.tv_sec; + ip->i_d.di_mtime.t_nsec = (int32_t)tv.tv_nsec; } if (flags & XFS_ICHGTIME_CHG) { - ip->i_d.di_ctime.t_sec = (__int32_t)tv.tv_sec; - ip->i_d.di_ctime.t_nsec = (__int32_t)tv.tv_nsec; + ip->i_d.di_ctime.t_sec = (int32_t)tv.tv_sec; + ip->i_d.di_ctime.t_nsec = (int32_t)tv.tv_nsec; } if (flags & XFS_ICHGTIME_CREATE) { - ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec; - ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec; + ip->i_d.di_crtime.t_sec = (int32_t)tv.tv_sec; + ip->i_d.di_crtime.t_nsec = (int32_t)tv.tv_nsec; } } @@ -219,7 +219,7 @@ libxfs_ialloc( return error; ASSERT(ip != NULL); - ip->i_d.di_mode = (__uint16_t)mode; + ip->i_d.di_mode = (uint16_t)mode; ip->i_d.di_onlink = 0; ip->i_d.di_nlink = nlink; ASSERT(ip->i_d.di_nlink == nlink); diff --git a/libxfs/xfs_alloc_btree.c b/libxfs/xfs_alloc_btree.c index 6f13572..e3f3338 100644 --- a/libxfs/xfs_alloc_btree.c +++ b/libxfs/xfs_alloc_btree.c @@ -246,25 +246,25 @@ xfs_allocbt_init_ptr_from_cur( ptr->s = agf->agf_roots[cur->bc_btnum]; } -STATIC __int64_t +STATIC int64_t xfs_allocbt_key_diff( struct xfs_btree_cur *cur, union xfs_btree_key *key) { xfs_alloc_rec_incore_t *rec = &cur->bc_rec.a; xfs_alloc_key_t *kp = &key->alloc; - __int64_t diff; + int64_t diff; if (cur->bc_btnum == XFS_BTNUM_BNO) { - return (__int64_t)be32_to_cpu(kp->ar_startblock) - + return (int64_t)be32_to_cpu(kp->ar_startblock) - rec->ar_startblock; } - diff = (__int64_t)be32_to_cpu(kp->ar_blockcount) - rec->ar_blockcount; + diff = (int64_t)be32_to_cpu(kp->ar_blockcount) - rec->ar_blockcount; if (diff) return diff; - return (__int64_t)be32_to_cpu(kp->ar_startblock) - rec->ar_startblock; + return (int64_t)be32_to_cpu(kp->ar_startblock) - rec->ar_startblock; } static bool diff --git a/libxfs/xfs_attr_remote.c b/libxfs/xfs_attr_remote.c index 95383e3..ce55a79 100644 --- a/libxfs/xfs_attr_remote.c +++ b/libxfs/xfs_attr_remote.c @@ -247,7 +247,7 @@ xfs_attr_rmtval_copyout( xfs_ino_t ino, int *offset, int *valuelen, - __uint8_t **dst) + uint8_t **dst) { char *src = bp->b_addr; xfs_daddr_t bno = bp->b_bn; @@ -295,7 +295,7 @@ xfs_attr_rmtval_copyin( xfs_ino_t ino, int *offset, int *valuelen, - __uint8_t **src) + uint8_t **src) { char *dst = bp->b_addr; xfs_daddr_t bno = bp->b_bn; @@ -349,7 +349,7 @@ xfs_attr_rmtval_get( struct xfs_mount *mp = args->dp->i_mount; struct xfs_buf *bp; xfs_dablk_t lblkno = args->rmtblkno; - __uint8_t *dst = args->value; + uint8_ta *dst = args->value; int valuelen; int nmap; int error; @@ -415,7 +415,7 @@ xfs_attr_rmtval_set( struct xfs_bmbt_irec map; xfs_dablk_t lblkno; xfs_fileoff_t lfileoff = 0; - __uint8_t *src = args->value; + uint8_t *src = args->value; int blkcnt; int valuelen; int nmap; diff --git a/libxfs/xfs_attr_sf.h b/libxfs/xfs_attr_sf.h index 919756e..085ff84 100644 --- a/libxfs/xfs_attr_sf.h +++ b/libxfs/xfs_attr_sf.h @@ -34,10 +34,10 @@ typedef struct xfs_attr_shortform { __u8 count; /* count of active entries */ } hdr; struct xfs_attr_sf_entry { - __uint8_t namelen; /* actual length of name (no NULL) */ - __uint8_t valuelen; /* actual length of value (no NULL) */ - __uint8_t flags; /* flags bits (see xfs_attr_leaf.h) */ - __uint8_t nameval[1]; /* name & value bytes concatenated */ + uint8_t namelen; /* actual length of name (no NULL) */ + uint8_t valuelen; /* actual length of value (no NULL) */ + uint8_t flags; /* flags bits (see xfs_attr_leaf.h) */ + uint8_t nameval[1]; /* name & value bytes concatenated */ } list[1]; /* variable sized array */ } xfs_attr_shortform_t; typedef struct xfs_attr_sf_hdr xfs_attr_sf_hdr_t; @@ -47,10 +47,10 @@ typedef struct xfs_attr_sf_entry xfs_attr_sf_entry_t; * We generate this then sort it, attr_list() must return things in hash-order. */ typedef struct xfs_attr_sf_sort { - __uint8_t entno; /* entry number in original list */ - __uint8_t namelen; /* length of name value (no null) */ - __uint8_t valuelen; /* length of value */ - __uint8_t flags; /* flags bits (see xfs_attr_leaf.h) */ + uint8_t entno; /* entry number in original list */ + uint8_t namelen; /* length of name value (no null) */ + uint8_t valuelen; /* length of value */ + uint8_t flags; /* flags bits (see xfs_attr_leaf.h) */ xfs_dahash_t hash; /* this entry's hash value */ unsigned char *name; /* name value, pointer into buffer */ } xfs_attr_sf_sort_t; @@ -58,7 +58,7 @@ typedef struct xfs_attr_sf_sort { #define XFS_ATTR_SF_ENTSIZE_BYNAME(nlen,vlen) /* space name/value uses */ \ (((int)sizeof(xfs_attr_sf_entry_t)-1 + (nlen)+(vlen))) #define XFS_ATTR_SF_ENTSIZE_MAX /* max space for name&value */ \ - ((1 << (NBBY*(int)sizeof(__uint8_t))) - 1) + ((1 << (NBBY*(int)sizeof(uint8_t))) - 1) #define XFS_ATTR_SF_ENTSIZE(sfep) /* space an entry uses */ \ ((int)sizeof(xfs_attr_sf_entry_t)-1 + (sfep)->namelen+(sfep)->valuelen) #define XFS_ATTR_SF_NEXTENTRY(sfep) /* next entry in struct */ \ diff --git a/libxfs/xfs_bit.h b/libxfs/xfs_bit.h index e1649c0..61c6b20 100644 --- a/libxfs/xfs_bit.h +++ b/libxfs/xfs_bit.h @@ -25,47 +25,47 @@ /* * masks with n high/low bits set, 64-bit values */ -static inline __uint64_t xfs_mask64hi(int n) +static inline uint64_t xfs_mask64hi(int n) { - return (__uint64_t)-1 << (64 - (n)); + return (uint64_t)-1 << (64 - (n)); } -static inline __uint32_t xfs_mask32lo(int n) +static inline uint32_t xfs_mask32lo(int n) { - return ((__uint32_t)1 << (n)) - 1; + return ((uint32_t)1 << (n)) - 1; } -static inline __uint64_t xfs_mask64lo(int n) +static inline uint64_t xfs_mask64lo(int n) { - return ((__uint64_t)1 << (n)) - 1; + return ((uint64_t)1 << (n)) - 1; } /* Get high bit set out of 32-bit argument, -1 if none set */ -static inline int xfs_highbit32(__uint32_t v) +static inline int xfs_highbit32(uint32_t v) { return fls(v) - 1; } /* Get high bit set out of 64-bit argument, -1 if none set */ -static inline int xfs_highbit64(__uint64_t v) +static inline int xfs_highbit64(uint64_t v) { return fls64(v) - 1; } /* Get low bit set out of 32-bit argument, -1 if none set */ -static inline int xfs_lowbit32(__uint32_t v) +static inline int xfs_lowbit32(uint32_t v) { return ffs(v) - 1; } /* Get low bit set out of 64-bit argument, -1 if none set */ -static inline int xfs_lowbit64(__uint64_t v) +static inline int xfs_lowbit64(uint64_t v) { - __uint32_t w = (__uint32_t)v; + uint32_t w = (uint32_t)v; int n = 0; if (w) { /* lower bits */ n = ffs(w); } else { /* upper bits */ - w = (__uint32_t)(v >> 32); + w = (uint32_t)(v >> 32); if (w) { n = ffs(w); if (n) diff --git a/libxfs/xfs_bmap_btree.c b/libxfs/xfs_bmap_btree.c index 2ef1836..fb5edca 100644 --- a/libxfs/xfs_bmap_btree.c +++ b/libxfs/xfs_bmap_btree.c @@ -95,8 +95,8 @@ xfs_bmdr_to_bmbt( */ STATIC void __xfs_bmbt_get_all( - __uint64_t l0, - __uint64_t l1, + uint64_t l0, + uint64_t l1, xfs_bmbt_irec_t *s) { int ext_flag; @@ -624,12 +624,12 @@ xfs_bmbt_init_ptr_from_cur( ptr->l = 0; } -STATIC __int64_t +STATIC int64_t xfs_bmbt_key_diff( struct xfs_btree_cur *cur, union xfs_btree_key *key) { - return (__int64_t)be64_to_cpu(key->bmbt.br_startoff) - + return (int64_t)be64_to_cpu(key->bmbt.br_startoff) - cur->bc_rec.b.br_startoff; } diff --git a/libxfs/xfs_btree.c b/libxfs/xfs_btree.c index 658f27e..1c700c0 100644 --- a/libxfs/xfs_btree.c +++ b/libxfs/xfs_btree.c @@ -38,7 +38,7 @@ kmem_zone_t *xfs_btree_cur_zone; /* * Btree magic numbers. */ -static const __uint32_t xfs_magics[2][XFS_BTNUM_MAX] = { +static const uint32_t xfs_magics[2][XFS_BTNUM_MAX] = { { XFS_ABTB_MAGIC, XFS_ABTC_MAGIC, XFS_BMAP_MAGIC, XFS_IBT_MAGIC, XFS_FIBT_MAGIC }, { XFS_ABTB_CRC_MAGIC, XFS_ABTC_CRC_MAGIC, @@ -673,14 +673,14 @@ xfs_btree_lastrec( */ void xfs_btree_offsets( - __int64_t fields, /* bitmask of fields */ + int64_t fields, /* bitmask of fields */ const short *offsets, /* table of field offsets */ int nbits, /* number of bits to inspect */ int *first, /* output: first byte offset */ int *last) /* output: last byte offset */ { int i; /* current bit number */ - __int64_t imask; /* mask for current bit number */ + int64_t imask; /* mask for current bit number */ ASSERT(fields != 0); /* @@ -1711,7 +1711,7 @@ xfs_btree_lookup( int *stat) /* success/failure */ { struct xfs_btree_block *block; /* current btree block */ - __int64_t diff; /* difference for the current key */ + int64_t diff; /* difference for the current key */ int error; /* error return value */ int keyno; /* current key number */ int level; /* level in the btree */ @@ -3992,7 +3992,7 @@ static int xfs_btree_block_change_owner( struct xfs_btree_cur *cur, int level, - __uint64_t new_owner, + uint64_t new_owner, struct list_head *buffer_list) { struct xfs_btree_block *block; @@ -4039,7 +4039,7 @@ xfs_btree_block_change_owner( int xfs_btree_change_owner( struct xfs_btree_cur *cur, - __uint64_t new_owner, + uint64_t new_owner, struct list_head *buffer_list) { union xfs_btree_ptr lptr; diff --git a/libxfs/xfs_btree.h b/libxfs/xfs_btree.h index 48cb251..41376bb 100644 --- a/libxfs/xfs_btree.h +++ b/libxfs/xfs_btree.h @@ -159,7 +159,7 @@ struct xfs_btree_ops { union xfs_btree_ptr *ptr); /* difference between key value and cursor value */ - __int64_t (*key_diff)(struct xfs_btree_cur *cur, + int64_t (*key_diff)(struct xfs_btree_cur *cur, union xfs_btree_key *key); const struct xfs_buf_ops *buf_ops; @@ -202,11 +202,11 @@ typedef struct xfs_btree_cur } bc_rec; /* current insert/search record value */ struct xfs_buf *bc_bufs[XFS_BTREE_MAXLEVELS]; /* buf ptr per level */ int bc_ptrs[XFS_BTREE_MAXLEVELS]; /* key/record # */ - __uint8_t bc_ra[XFS_BTREE_MAXLEVELS]; /* readahead bits */ + uint8_t bc_ra[XFS_BTREE_MAXLEVELS]; /* readahead bits */ #define XFS_BTCUR_LEFTRA 1 /* left sibling has been read-ahead */ #define XFS_BTCUR_RIGHTRA 2 /* right sibling has been read-ahead */ - __uint8_t bc_nlevels; /* number of levels in the tree */ - __uint8_t bc_blocklog; /* log2(blocksize) of btree blocks */ + uint8_t bc_nlevels; /* number of levels in the tree */ + uint8_t bc_blocklog; /* log2(blocksize) of btree blocks */ xfs_btnum_t bc_btnum; /* identifies which btree type */ union { struct { /* needed for BNO, CNT, INO */ @@ -315,7 +315,7 @@ xfs_btree_islastblock( */ void xfs_btree_offsets( - __int64_t fields, /* bitmask of fields */ + int64_t fields, /* bitmask of fields */ const short *offsets,/* table of field offsets */ int nbits, /* number of bits to inspect */ int *first, /* output: first byte offset */ @@ -393,7 +393,7 @@ int xfs_btree_new_iroot(struct xfs_btree_cur *, int *, int *); int xfs_btree_insert(struct xfs_btree_cur *, int *); int xfs_btree_delete(struct xfs_btree_cur *, int *); int xfs_btree_get_rec(struct xfs_btree_cur *, union xfs_btree_rec **, int *); -int xfs_btree_change_owner(struct xfs_btree_cur *cur, __uint64_t new_owner, +int xfs_btree_change_owner(struct xfs_btree_cur *cur, uint64_t new_owner, struct list_head *buffer_list); /* @@ -419,7 +419,7 @@ static inline int xfs_btree_get_numrecs(struct xfs_btree_block *block) } static inline void xfs_btree_set_numrecs(struct xfs_btree_block *block, - __uint16_t numrecs) + uint16_t numrecs) { block->bb_numrecs = cpu_to_be16(numrecs); } diff --git a/libxfs/xfs_cksum.h b/libxfs/xfs_cksum.h index fad1676..3c2bc92 100644 --- a/libxfs/xfs_cksum.h +++ b/libxfs/xfs_cksum.h @@ -1,18 +1,18 @@ #ifndef _XFS_CKSUM_H #define _XFS_CKSUM_H 1 -#define XFS_CRC_SEED (~(__uint32_t)0) +#define XFS_CRC_SEED (~(uint32_t)0) /* * Calculate the intermediate checksum for a buffer that has the CRC field * inside it. The offset of the 32bit crc fields is passed as the * cksum_offset parameter. */ -static inline __uint32_t +static inline uint32_t xfs_start_cksum(char *buffer, size_t length, unsigned long cksum_offset) { - __uint32_t zero = 0; - __uint32_t crc; + uint32_t zero = 0; + uint32_t crc; /* Calculate CRC up to the checksum. */ crc = crc32c(XFS_CRC_SEED, buffer, cksum_offset); @@ -33,7 +33,7 @@ xfs_start_cksum(char *buffer, size_t length, unsigned long cksum_offset) * so that it is consistent on disk. */ static inline __le32 -xfs_end_cksum(__uint32_t crc) +xfs_end_cksum(uint32_t crc) { return ~cpu_to_le32(crc); } @@ -44,7 +44,7 @@ xfs_end_cksum(__uint32_t crc) static inline void xfs_update_cksum(char *buffer, size_t length, unsigned long cksum_offset) { - __uint32_t crc = xfs_start_cksum(buffer, length, cksum_offset); + uint32_t crc = xfs_start_cksum(buffer, length, cksum_offset); *(__le32 *)(buffer + cksum_offset) = xfs_end_cksum(crc); } @@ -55,7 +55,7 @@ xfs_update_cksum(char *buffer, size_t length, unsigned long cksum_offset) static inline int xfs_verify_cksum(char *buffer, size_t length, unsigned long cksum_offset) { - __uint32_t crc = xfs_start_cksum(buffer, length, cksum_offset); + uint32_t crc = xfs_start_cksum(buffer, length, cksum_offset); return *(__le32 *)(buffer + cksum_offset) == xfs_end_cksum(crc); } diff --git a/libxfs/xfs_da_btree.c b/libxfs/xfs_da_btree.c index bf5fe21..cdddca1 100644 --- a/libxfs/xfs_da_btree.c +++ b/libxfs/xfs_da_btree.c @@ -1943,7 +1943,7 @@ xfs_da3_path_shift( * This is implemented with some source-level loop unrolling. */ xfs_dahash_t -xfs_da_hashname(const __uint8_t *name, int namelen) +xfs_da_hashname(const uint8_t *name, int namelen) { xfs_dahash_t hash; diff --git a/libxfs/xfs_da_btree.h b/libxfs/xfs_da_btree.h index 6e153e3..cefbdf6 100644 --- a/libxfs/xfs_da_btree.h +++ b/libxfs/xfs_da_btree.h @@ -60,10 +60,10 @@ enum xfs_dacmp { */ typedef struct xfs_da_args { struct xfs_da_geometry *geo; /* da block geometry */ - const __uint8_t *name; /* string (maybe not NULL terminated) */ + const uint8_t *name; /* string (maybe not NULL terminated) */ int namelen; /* length of string (maybe no NULL) */ - __uint8_t filetype; /* filetype of inode for directories */ - __uint8_t *value; /* set of bytes (maybe contain NULLs) */ + uint8_t filetype; /* filetype of inode for directories */ + uint8_t *value; /* set of bytes (maybe contain NULLs) */ int valuelen; /* length of value */ int flags; /* argument flags (eg: ATTR_NOCREATE) */ xfs_dahash_t hashval; /* hash value of name */ @@ -207,7 +207,7 @@ xfs_daddr_t xfs_da_reada_buf(struct xfs_inode *dp, xfs_dablk_t bno, int xfs_da_shrink_inode(xfs_da_args_t *args, xfs_dablk_t dead_blkno, struct xfs_buf *dead_buf); -uint xfs_da_hashname(const __uint8_t *name_string, int name_length); +uint xfs_da_hashname(const uint8_t *name_string, int name_length); enum xfs_dacmp xfs_da_compname(struct xfs_da_args *args, const unsigned char *name, int len); diff --git a/libxfs/xfs_da_format.c b/libxfs/xfs_da_format.c index e4c37f9..770d1f6 100644 --- a/libxfs/xfs_da_format.c +++ b/libxfs/xfs_da_format.c @@ -50,7 +50,7 @@ xfs_dir3_sf_entsize( struct xfs_dir2_sf_hdr *hdr, int len) { - return xfs_dir2_sf_entsize(hdr, len) + sizeof(__uint8_t); + return xfs_dir2_sf_entsize(hdr, len) + sizeof(uint8_t); } static struct xfs_dir2_sf_entry * @@ -78,7 +78,7 @@ xfs_dir3_sf_nextentry( * not necessary. For non-filetype enable directories, the type is always * unknown and we never store the value. */ -static __uint8_t +static uint8_t xfs_dir2_sfe_get_ftype( struct xfs_dir2_sf_entry *sfep) { @@ -88,16 +88,16 @@ xfs_dir2_sfe_get_ftype( static void xfs_dir2_sfe_put_ftype( struct xfs_dir2_sf_entry *sfep, - __uint8_t ftype) + uint8_t ftype) { ASSERT(ftype < XFS_DIR3_FT_MAX); } -static __uint8_t +static uint8_t xfs_dir3_sfe_get_ftype( struct xfs_dir2_sf_entry *sfep) { - __uint8_t ftype; + uint8_t ftype; ftype = sfep->name[sfep->namelen]; if (ftype >= XFS_DIR3_FT_MAX) @@ -108,7 +108,7 @@ xfs_dir3_sfe_get_ftype( static void xfs_dir3_sfe_put_ftype( struct xfs_dir2_sf_entry *sfep, - __uint8_t ftype) + uint8_t ftype) { ASSERT(ftype < XFS_DIR3_FT_MAX); @@ -230,7 +230,7 @@ xfs_dir3_sfe_put_ino( #define XFS_DIR3_DATA_ENTSIZE(n) \ round_up((offsetof(struct xfs_dir2_data_entry, name[0]) + (n) + \ - sizeof(xfs_dir2_data_off_t) + sizeof(__uint8_t)), \ + sizeof(xfs_dir2_data_off_t) + sizeof(uint8_t)), \ XFS_DIR2_DATA_ALIGN) static int @@ -247,7 +247,7 @@ xfs_dir3_data_entsize( return XFS_DIR3_DATA_ENTSIZE(n); } -static __uint8_t +static uint8_t xfs_dir2_data_get_ftype( struct xfs_dir2_data_entry *dep) { @@ -257,16 +257,16 @@ xfs_dir2_data_get_ftype( static void xfs_dir2_data_put_ftype( struct xfs_dir2_data_entry *dep, - __uint8_t ftype) + uint8_t ftype) { ASSERT(ftype < XFS_DIR3_FT_MAX); } -static __uint8_t +static uint8_t xfs_dir3_data_get_ftype( struct xfs_dir2_data_entry *dep) { - __uint8_t ftype = dep->name[dep->namelen]; + uint8_t ftype = dep->name[dep->namelen]; if (ftype >= XFS_DIR3_FT_MAX) return XFS_DIR3_FT_UNKNOWN; @@ -276,7 +276,7 @@ xfs_dir3_data_get_ftype( static void xfs_dir3_data_put_ftype( struct xfs_dir2_data_entry *dep, - __uint8_t type) + uint8_t type) { ASSERT(type < XFS_DIR3_FT_MAX); ASSERT(dep->namelen != 0); diff --git a/libxfs/xfs_da_format.h b/libxfs/xfs_da_format.h index b14bbd6..6b0f629 100644 --- a/libxfs/xfs_da_format.h +++ b/libxfs/xfs_da_format.h @@ -111,11 +111,11 @@ struct xfs_da3_intnode { * appropriate. */ struct xfs_da3_icnode_hdr { - __uint32_t forw; - __uint32_t back; - __uint16_t magic; - __uint16_t count; - __uint16_t level; + uint32_t forw; + uint32_t back; + uint16_t magic; + uint16_t count; + uint16_t level; }; /* @@ -187,7 +187,7 @@ struct xfs_da3_icnode_hdr { /* * Byte offset in data block and shortform entry. */ -typedef __uint16_t xfs_dir2_data_off_t; +typedef uint16_t xfs_dir2_data_off_t; #define NULLDATAOFF 0xffffU typedef uint xfs_dir2_data_aoff_t; /* argument form */ @@ -195,12 +195,12 @@ typedef uint xfs_dir2_data_aoff_t; /* argument form */ * Normalized offset (in a data block) of the entry, really xfs_dir2_data_off_t. * Only need 16 bits, this is the byte offset into the single block form. */ -typedef struct { __uint8_t i[2]; } __arch_pack xfs_dir2_sf_off_t; +typedef struct { uint8_t i[2]; } __arch_pack xfs_dir2_sf_off_t; /* * Offset in data space of a data entry. */ -typedef __uint32_t xfs_dir2_dataptr_t; +typedef uint32_t xfs_dir2_dataptr_t; #define XFS_DIR2_MAX_DATAPTR ((xfs_dir2_dataptr_t)0xffffffff) #define XFS_DIR2_NULL_DATAPTR ((xfs_dir2_dataptr_t)0) @@ -212,19 +212,19 @@ typedef xfs_off_t xfs_dir2_off_t; /* * Directory block number (logical dirblk in file) */ -typedef __uint32_t xfs_dir2_db_t; +typedef uint32_t xfs_dir2_db_t; /* * Inode number stored as 8 8-bit values. */ -typedef struct { __uint8_t i[8]; } xfs_dir2_ino8_t; +typedef struct { uint8_t i[8]; } xfs_dir2_ino8_t; /* * Inode number stored as 4 8-bit values. * Works a lot of the time, when all the inode numbers in a directory * fit in 32 bits. */ -typedef struct { __uint8_t i[4]; } xfs_dir2_ino4_t; +typedef struct { uint8_t i[4]; } xfs_dir2_ino4_t; typedef union { xfs_dir2_ino8_t i8; @@ -244,8 +244,8 @@ typedef union { * over them. */ typedef struct xfs_dir2_sf_hdr { - __uint8_t count; /* count of entries */ - __uint8_t i8count; /* count of 8-byte inode #s */ + uint8_t count; /* count of entries */ + uint8_t i8count; /* count of 8-byte inode #s */ xfs_dir2_inou_t parent; /* parent dir inode number */ } __arch_pack xfs_dir2_sf_hdr_t; @@ -466,11 +466,11 @@ struct xfs_dir3_leaf_hdr { }; struct xfs_dir3_icleaf_hdr { - __uint32_t forw; - __uint32_t back; - __uint16_t magic; - __uint16_t count; - __uint16_t stale; + uint32_t forw; + uint32_t back; + uint16_t magic; + uint16_t count; + uint16_t stale; }; /* @@ -557,10 +557,10 @@ struct xfs_dir3_free { * xfs_dir3_free_hdr_from_disk/xfs_dir3_free_hdr_to_disk. */ struct xfs_dir3_icfree_hdr { - __uint32_t magic; - __uint32_t firstdb; - __uint32_t nvalid; - __uint32_t nused; + uint32_t magic; + uint32_t firstdb; + uint32_t nvalid; + uint32_t nused; }; @@ -727,22 +727,22 @@ struct xfs_attr3_leafblock { * incore, neutral version of the attribute leaf header */ struct xfs_attr3_icleaf_hdr { - __uint32_t forw; - __uint32_t back; - __uint16_t magic; - __uint16_t count; - __uint16_t usedbytes; + uint32_t forw; + uint32_t back; + uint16_t magic; + uint16_t count; + uint16_t usedbytes; /* * firstused is 32-bit here instead of 16-bit like the on-disk variant * to support maximum fsb size of 64k without overflow issues throughout * the attr code. Instead, the overflow condition is handled on * conversion to/from disk. */ - __uint32_t firstused; + uint32_t firstused; __u8 holes; struct { - __uint16_t base; - __uint16_t size; + uint16_t base; + uint16_t size; } freemap[XFS_ATTR_LEAF_MAPSIZE]; }; diff --git a/libxfs/xfs_dir2.h b/libxfs/xfs_dir2.h index 0129e37..69af9f7 100644 --- a/libxfs/xfs_dir2.h +++ b/libxfs/xfs_dir2.h @@ -45,9 +45,9 @@ struct xfs_dir_ops { struct xfs_dir2_sf_entry * (*sf_nextentry)(struct xfs_dir2_sf_hdr *hdr, struct xfs_dir2_sf_entry *sfep); - __uint8_t (*sf_get_ftype)(struct xfs_dir2_sf_entry *sfep); + uint8_t (*sf_get_ftype)(struct xfs_dir2_sf_entry *sfep); void (*sf_put_ftype)(struct xfs_dir2_sf_entry *sfep, - __uint8_t ftype); + uint8_t ftype); xfs_ino_t (*sf_get_ino)(struct xfs_dir2_sf_hdr *hdr, struct xfs_dir2_sf_entry *sfep); void (*sf_put_ino)(struct xfs_dir2_sf_hdr *hdr, @@ -58,9 +58,9 @@ struct xfs_dir_ops { xfs_ino_t ino); int (*data_entsize)(int len); - __uint8_t (*data_get_ftype)(struct xfs_dir2_data_entry *dep); + uint8_t (*data_get_ftype)(struct xfs_dir2_data_entry *dep); void (*data_put_ftype)(struct xfs_dir2_data_entry *dep, - __uint8_t ftype); + uint8_t ftype); __be16 * (*data_entry_tag_p)(struct xfs_dir2_data_entry *dep); struct xfs_dir2_data_free * (*data_bestfree_p)(struct xfs_dir2_data_hdr *hdr); diff --git a/libxfs/xfs_dir2_leaf.c b/libxfs/xfs_dir2_leaf.c index f80d91f..40a35b8 100644 --- a/libxfs/xfs_dir2_leaf.c +++ b/libxfs/xfs_dir2_leaf.c @@ -142,7 +142,7 @@ xfs_dir3_leaf_check_int( static bool xfs_dir3_leaf_verify( struct xfs_buf *bp, - __uint16_t magic) + uint16_t magic) { struct xfs_mount *mp = bp->b_target->bt_mount; struct xfs_dir2_leaf *leaf = bp->b_addr; @@ -151,7 +151,7 @@ xfs_dir3_leaf_verify( if (xfs_sb_version_hascrc(&mp->m_sb)) { struct xfs_dir3_leaf_hdr *leaf3 = bp->b_addr; - __uint16_t magic3; + uint16_t magic3; magic3 = (magic == XFS_DIR2_LEAF1_MAGIC) ? XFS_DIR3_LEAF1_MAGIC : XFS_DIR3_LEAFN_MAGIC; @@ -175,7 +175,7 @@ xfs_dir3_leaf_verify( static void __read_verify( struct xfs_buf *bp, - __uint16_t magic) + uint16_t magic) { struct xfs_mount *mp = bp->b_target->bt_mount; @@ -192,7 +192,7 @@ __read_verify( static void __write_verify( struct xfs_buf *bp, - __uint16_t magic) + uint16_t magic) { struct xfs_mount *mp = bp->b_target->bt_mount; struct xfs_buf_log_item *bip = bp->b_fspriv; @@ -296,7 +296,7 @@ xfs_dir3_leaf_init( struct xfs_trans *tp, struct xfs_buf *bp, xfs_ino_t owner, - __uint16_t type) + uint16_t type) { struct xfs_dir2_leaf *leaf = bp->b_addr; @@ -340,7 +340,7 @@ xfs_dir3_leaf_get_buf( xfs_da_args_t *args, xfs_dir2_db_t bno, struct xfs_buf **bpp, - __uint16_t magic) + uint16_t magic) { struct xfs_inode *dp = args->dp; struct xfs_trans *tp = args->trans; diff --git a/libxfs/xfs_dir2_priv.h b/libxfs/xfs_dir2_priv.h index d04547f..6cdb465 100644 --- a/libxfs/xfs_dir2_priv.h +++ b/libxfs/xfs_dir2_priv.h @@ -69,7 +69,7 @@ extern void xfs_dir3_leaf_compact_x1(struct xfs_dir3_icleaf_hdr *leafhdr, struct xfs_dir2_leaf_entry *ents, int *indexp, int *lowstalep, int *highstalep, int *lowlogp, int *highlogp); extern int xfs_dir3_leaf_get_buf(struct xfs_da_args *args, xfs_dir2_db_t bno, - struct xfs_buf **bpp, __uint16_t magic); + struct xfs_buf **bpp, uint16_t magic); extern void xfs_dir3_leaf_log_ents(struct xfs_da_args *args, struct xfs_buf *bp, int first, int last); extern void xfs_dir3_leaf_log_header(struct xfs_da_args *args, diff --git a/libxfs/xfs_format.h b/libxfs/xfs_format.h index a35009a..4ebd0da 100644 --- a/libxfs/xfs_format.h +++ b/libxfs/xfs_format.h @@ -103,8 +103,8 @@ struct xfs_ifork; * Must be padded to 64 bit alignment. */ typedef struct xfs_sb { - __uint32_t sb_magicnum; /* magic number == XFS_SB_MAGIC */ - __uint32_t sb_blocksize; /* logical block size, bytes */ + uint32_t sb_magicnum; /* magic number == XFS_SB_MAGIC */ + uint32_t sb_blocksize; /* logical block size, bytes */ xfs_rfsblock_t sb_dblocks; /* number of data blocks */ xfs_rfsblock_t sb_rblocks; /* number of realtime blocks */ xfs_rtblock_t sb_rextents; /* number of realtime extents */ @@ -118,45 +118,45 @@ typedef struct xfs_sb { xfs_agnumber_t sb_agcount; /* number of allocation groups */ xfs_extlen_t sb_rbmblocks; /* number of rt bitmap blocks */ xfs_extlen_t sb_logblocks; /* number of log blocks */ - __uint16_t sb_versionnum; /* header version == XFS_SB_VERSION */ - __uint16_t sb_sectsize; /* volume sector size, bytes */ - __uint16_t sb_inodesize; /* inode size, bytes */ - __uint16_t sb_inopblock; /* inodes per block */ + uint16_t sb_versionnum; /* header version == XFS_SB_VERSION */ + uint16_t sb_sectsize; /* volume sector size, bytes */ + uint16_t sb_inodesize; /* inode size, bytes */ + uint16_t sb_inopblock; /* inodes per block */ char sb_fname[12]; /* file system name */ - __uint8_t sb_blocklog; /* log2 of sb_blocksize */ - __uint8_t sb_sectlog; /* log2 of sb_sectsize */ - __uint8_t sb_inodelog; /* log2 of sb_inodesize */ - __uint8_t sb_inopblog; /* log2 of sb_inopblock */ - __uint8_t sb_agblklog; /* log2 of sb_agblocks (rounded up) */ - __uint8_t sb_rextslog; /* log2 of sb_rextents */ - __uint8_t sb_inprogress; /* mkfs is in progress, don't mount */ - __uint8_t sb_imax_pct; /* max % of fs for inode space */ + uint8_t sb_blocklog; /* log2 of sb_blocksize */ + uint8_t sb_sectlog; /* log2 of sb_sectsize */ + uint8_t sb_inodelog; /* log2 of sb_inodesize */ + uint8_t sb_inopblog; /* log2 of sb_inopblock */ + uint8_t sb_agblklog; /* log2 of sb_agblocks (rounded up) */ + uint8_t sb_rextslog; /* log2 of sb_rextents */ + uint8_t sb_inprogress; /* mkfs is in progress, don't mount */ + uint8_t sb_imax_pct; /* max % of fs for inode space */ /* statistics */ /* * These fields must remain contiguous. If you really * want to change their layout, make sure you fix the * code in xfs_trans_apply_sb_deltas(). */ - __uint64_t sb_icount; /* allocated inodes */ - __uint64_t sb_ifree; /* free inodes */ - __uint64_t sb_fdblocks; /* free data blocks */ - __uint64_t sb_frextents; /* free realtime extents */ + uint64_t sb_icount; /* allocated inodes */ + uint64_t sb_ifree; /* free inodes */ + uint64_t sb_fdblocks; /* free data blocks */ + uint64_t sb_frextents; /* free realtime extents */ /* * End contiguous fields. */ xfs_ino_t sb_uquotino; /* user quota inode */ xfs_ino_t sb_gquotino; /* group quota inode */ - __uint16_t sb_qflags; /* quota flags */ - __uint8_t sb_flags; /* misc. flags */ - __uint8_t sb_shared_vn; /* shared version number */ + uint16_t sb_qflags; /* quota flags */ + uint8_t sb_flags; /* misc. flags */ + uint8_t sb_shared_vn; /* shared version number */ xfs_extlen_t sb_inoalignmt; /* inode chunk alignment, fsblocks */ - __uint32_t sb_unit; /* stripe or raid unit */ - __uint32_t sb_width; /* stripe or raid width */ - __uint8_t sb_dirblklog; /* log2 of dir block size (fsbs) */ - __uint8_t sb_logsectlog; /* log2 of the log sector size */ - __uint16_t sb_logsectsize; /* sector size for the log, bytes */ - __uint32_t sb_logsunit; /* stripe unit size for the log */ - __uint32_t sb_features2; /* additional feature bits */ + uint32_t sb_unit; /* stripe or raid unit */ + uint32_t sb_width; /* stripe or raid width */ + uint8_t sb_dirblklog; /* log2 of dir block size (fsbs) */ + uint8_t sb_logsectlog; /* log2 of the log sector size */ + uint16_t sb_logsectsize; /* sector size for the log, bytes */ + uint32_t sb_logsunit; /* stripe unit size for the log */ + uint32_t sb_features2; /* additional feature bits */ /* * bad features2 field as a result of failing to pad the sb structure to @@ -167,17 +167,17 @@ typedef struct xfs_sb { * the value in sb_features2 when formatting the incore superblock to * the disk buffer. */ - __uint32_t sb_bad_features2; + uint32_t sb_bad_features2; /* version 5 superblock fields start here */ /* feature masks */ - __uint32_t sb_features_compat; - __uint32_t sb_features_ro_compat; - __uint32_t sb_features_incompat; - __uint32_t sb_features_log_incompat; + uint32_t sb_features_compat; + uint32_t sb_features_ro_compat; + uint32_t sb_features_incompat; + uint32_t sb_features_log_incompat; - __uint32_t sb_crc; /* superblock crc */ + uint32_t sb_crc; /* superblock crc */ xfs_extlen_t sb_spino_align; /* sparse inode chunk alignment */ xfs_ino_t sb_pquotino; /* project quota inode */ @@ -449,7 +449,7 @@ static inline void xfs_sb_version_addprojid32bit(struct xfs_sb *sbp) static inline bool xfs_sb_has_compat_feature( struct xfs_sb *sbp, - __uint32_t feature) + uint32_t feature) { return (sbp->sb_features_compat & feature) != 0; } @@ -461,7 +461,7 @@ xfs_sb_has_compat_feature( static inline bool xfs_sb_has_ro_compat_feature( struct xfs_sb *sbp, - __uint32_t feature) + uint32_t feature) { return (sbp->sb_features_ro_compat & feature) != 0; } @@ -479,7 +479,7 @@ xfs_sb_has_ro_compat_feature( static inline bool xfs_sb_has_incompat_feature( struct xfs_sb *sbp, - __uint32_t feature) + uint32_t feature) { return (sbp->sb_features_incompat & feature) != 0; } @@ -489,7 +489,7 @@ xfs_sb_has_incompat_feature( static inline bool xfs_sb_has_incompat_log_feature( struct xfs_sb *sbp, - __uint32_t feature) + uint32_t feature) { return (sbp->sb_features_log_incompat & feature) != 0; } @@ -579,8 +579,8 @@ xfs_is_quota_inode(struct xfs_sb *sbp, xfs_ino_t ino) */ #define XFS_FSB_TO_B(mp,fsbno) ((xfs_fsize_t)(fsbno) << (mp)->m_sb.sb_blocklog) #define XFS_B_TO_FSB(mp,b) \ - ((((__uint64_t)(b)) + (mp)->m_blockmask) >> (mp)->m_sb.sb_blocklog) -#define XFS_B_TO_FSBT(mp,b) (((__uint64_t)(b)) >> (mp)->m_sb.sb_blocklog) + ((((uint64_t)(b)) + (mp)->m_blockmask) >> (mp)->m_sb.sb_blocklog) +#define XFS_B_TO_FSBT(mp,b) (((uint64_t)(b)) >> (mp)->m_sb.sb_blocklog) #define XFS_B_FSB_OFFSET(mp,b) ((b) & (mp)->m_blockmask) /* @@ -1032,7 +1032,7 @@ static inline void xfs_dinode_put_rdev(struct xfs_dinode *dip, xfs_dev_t rdev) * next agno_log bits - ag number * high agno_log-agblklog-inopblog bits - 0 */ -#define XFS_INO_MASK(k) (__uint32_t)((1ULL << (k)) - 1) +#define XFS_INO_MASK(k) (uint32_t)((1ULL << (k)) - 1) #define XFS_INO_OFFSET_BITS(mp) (mp)->m_sb.sb_inopblog #define XFS_INO_AGBNO_BITS(mp) (mp)->m_sb.sb_agblklog #define XFS_INO_AGINO_BITS(mp) (mp)->m_agino_log @@ -1229,16 +1229,16 @@ typedef __be32 xfs_alloc_ptr_t; #define XFS_FIBT_MAGIC 0x46494254 /* 'FIBT' */ #define XFS_FIBT_CRC_MAGIC 0x46494233 /* 'FIB3' */ -typedef __uint64_t xfs_inofree_t; +typedef uint64_t xfs_inofree_t; #define XFS_INODES_PER_CHUNK (NBBY * sizeof(xfs_inofree_t)) #define XFS_INODES_PER_CHUNK_LOG (XFS_NBBYLOG + 3) #define XFS_INOBT_ALL_FREE ((xfs_inofree_t)-1) #define XFS_INOBT_MASK(i) ((xfs_inofree_t)1 << (i)) #define XFS_INOBT_HOLEMASK_FULL 0 /* holemask for full chunk */ -#define XFS_INOBT_HOLEMASK_BITS (NBBY * sizeof(__uint16_t)) +#define XFS_INOBT_HOLEMASK_BITS (NBBY * sizeof(uint16_t)) #define XFS_INODES_PER_HOLEMASK_BIT \ - (XFS_INODES_PER_CHUNK / (NBBY * sizeof(__uint16_t))) + (XFS_INODES_PER_CHUNK / (NBBY * sizeof(uint16_t))) static inline xfs_inofree_t xfs_inobt_maskn(int i, int n) { @@ -1272,9 +1272,9 @@ typedef struct xfs_inobt_rec { typedef struct xfs_inobt_rec_incore { xfs_agino_t ir_startino; /* starting inode number */ - __uint16_t ir_holemask; /* hole mask for sparse chunks */ - __uint8_t ir_count; /* total inode count */ - __uint8_t ir_freecount; /* count of free inodes (set bits) */ + uint16_t ir_holemask; /* hole mask for sparse chunks */ + uint8_t ir_count; /* total inode count */ + uint8_t ir_freecount; /* count of free inodes (set bits) */ xfs_inofree_t ir_free; /* free inode mask */ } xfs_inobt_rec_incore_t; @@ -1345,11 +1345,11 @@ typedef struct xfs_bmbt_rec { __be64 l0, l1; } xfs_bmbt_rec_t; -typedef __uint64_t xfs_bmbt_rec_base_t; /* use this for casts */ +typedef uint64_t xfs_bmbt_rec_base_t; /* use this for casts */ typedef xfs_bmbt_rec_t xfs_bmdr_rec_t; typedef struct xfs_bmbt_rec_host { - __uint64_t l0, l1; + uint64_t l0, l1; } xfs_bmbt_rec_host_t; /* diff --git a/libxfs/xfs_fs.h b/libxfs/xfs_fs.h index d8b733a..8fbbdff 100644 --- a/libxfs/xfs_fs.h +++ b/libxfs/xfs_fs.h @@ -319,10 +319,10 @@ typedef struct xfs_bstat { * and using two 16bit values to hold new 32bit projid was choosen * to retain compatibility with "old" filesystems). */ -static inline __uint32_t +static inline uint32_t bstat_get_projid(struct xfs_bstat *bs) { - return (__uint32_t)bs->bs_projid_hi << 16 | bs->bs_projid_lo; + return (uint32_t)bs->bs_projid_hi << 16 | bs->bs_projid_lo; } /* @@ -472,10 +472,10 @@ typedef struct xfs_handle { */ typedef struct xfs_swapext { - __int64_t sx_version; /* version */ + int64_t sx_version; /* version */ #define XFS_SX_VERSION 0 - __int64_t sx_fdtarget; /* fd of target file */ - __int64_t sx_fdtmp; /* fd of tmp file */ + int64_t sx_fdtarget; /* fd of target file */ + int64_t sx_fdtmp; /* fd of tmp file */ xfs_off_t sx_offset; /* offset into file */ xfs_off_t sx_length; /* leng from offset */ char sx_pad[16]; /* pad space, unused */ @@ -565,7 +565,7 @@ typedef struct xfs_swapext #define XFS_IOC_ATTRLIST_BY_HANDLE _IOW ('X', 122, struct xfs_fsop_attrlist_handlereq) #define XFS_IOC_ATTRMULTI_BY_HANDLE _IOW ('X', 123, struct xfs_fsop_attrmulti_handlereq) #define XFS_IOC_FSGEOMETRY _IOR ('X', 124, struct xfs_fsop_geom) -#define XFS_IOC_GOINGDOWN _IOR ('X', 125, __uint32_t) +#define XFS_IOC_GOINGDOWN _IOR ('X', 125, uint32_t) /* XFS_IOC_GETFSUUID ---------- deprecated 140 */ /* reflink ioctls; these MUST match the btrfs ioctl definitions */ diff --git a/libxfs/xfs_ialloc.c b/libxfs/xfs_ialloc.c index 64c3acf..ed228fb 100644 --- a/libxfs/xfs_ialloc.c +++ b/libxfs/xfs_ialloc.c @@ -133,9 +133,9 @@ xfs_inobt_get_rec( STATIC int xfs_inobt_insert_rec( struct xfs_btree_cur *cur, - __uint16_t holemask, - __uint8_t count, - __int32_t freecount, + uint16_t holemask, + uint8_t count, + int32_t freecount, xfs_inofree_t free, int *stat) { diff --git a/libxfs/xfs_ialloc_btree.c b/libxfs/xfs_ialloc_btree.c index 52939c3..31b5923 100644 --- a/libxfs/xfs_ialloc_btree.c +++ b/libxfs/xfs_ialloc_btree.c @@ -205,12 +205,12 @@ xfs_finobt_init_ptr_from_cur( ptr->s = agi->agi_free_root; } -STATIC __int64_t +STATIC int64_t xfs_inobt_key_diff( struct xfs_btree_cur *cur, union xfs_btree_key *key) { - return (__int64_t)be32_to_cpu(key->inobt.ir_startino) - + return (int64_t)be32_to_cpu(key->inobt.ir_startino) - cur->bc_rec.i.ir_startino; } diff --git a/libxfs/xfs_inode_buf.c b/libxfs/xfs_inode_buf.c index 324715e..8bcb52a 100644 --- a/libxfs/xfs_inode_buf.c +++ b/libxfs/xfs_inode_buf.c @@ -325,7 +325,7 @@ xfs_dinode_calc_crc( struct xfs_mount *mp, struct xfs_dinode *dip) { - __uint32_t crc; + uint32_t crc; if (dip->di_version < 3) return; diff --git a/libxfs/xfs_log_format.h b/libxfs/xfs_log_format.h index db8ceeb..ccc0fdb 100644 --- a/libxfs/xfs_log_format.h +++ b/libxfs/xfs_log_format.h @@ -31,7 +31,7 @@ struct xfs_trans_res; * through all the log items definitions and everything they encode into the * log. */ -typedef __uint32_t xlog_tid_t; +typedef uint32_t xlog_tid_t; #define XLOG_MIN_ICLOGS 2 #define XLOG_MAX_ICLOGS 8 @@ -205,7 +205,7 @@ typedef struct xfs_log_iovec { typedef struct xfs_trans_header { uint th_magic; /* magic number */ uint th_type; /* transaction type */ - __int32_t th_tid; /* transaction id (unused) */ + int32_t th_tid; /* transaction id (unused) */ uint th_num_items; /* num items logged by trans */ } xfs_trans_header_t; @@ -242,52 +242,52 @@ typedef struct xfs_trans_header { * must be added on to the end. */ typedef struct xfs_inode_log_format { - __uint16_t ilf_type; /* inode log item type */ - __uint16_t ilf_size; /* size of this item */ - __uint32_t ilf_fields; /* flags for fields logged */ - __uint16_t ilf_asize; /* size of attr d/ext/root */ - __uint16_t ilf_dsize; /* size of data/ext/root */ - __uint64_t ilf_ino; /* inode number */ + uint16_t ilf_type; /* inode log item type */ + uint16_t ilf_size; /* size of this item */ + uint32_t ilf_fields; /* flags for fields logged */ + uint16_t ilf_asize; /* size of attr d/ext/root */ + uint16_t ilf_dsize; /* size of data/ext/root */ + uint64_t ilf_ino; /* inode number */ union { - __uint32_t ilfu_rdev; /* rdev value for dev inode*/ + uint32_t ilfu_rdev; /* rdev value for dev inode*/ uuid_t ilfu_uuid; /* mount point value */ } ilf_u; - __int64_t ilf_blkno; /* blkno of inode buffer */ - __int32_t ilf_len; /* len of inode buffer */ - __int32_t ilf_boffset; /* off of inode in buffer */ + int64_t ilf_blkno; /* blkno of inode buffer */ + int32_t ilf_len; /* len of inode buffer */ + int32_t ilf_boffset; /* off of inode in buffer */ } xfs_inode_log_format_t; typedef struct xfs_inode_log_format_32 { - __uint16_t ilf_type; /* inode log item type */ - __uint16_t ilf_size; /* size of this item */ - __uint32_t ilf_fields; /* flags for fields logged */ - __uint16_t ilf_asize; /* size of attr d/ext/root */ - __uint16_t ilf_dsize; /* size of data/ext/root */ - __uint64_t ilf_ino; /* inode number */ + uint16_t ilf_type; /* inode log item type */ + uint16_t ilf_size; /* size of this item */ + uint32_t ilf_fields; /* flags for fields logged */ + uint16_t ilf_asize; /* size of attr d/ext/root */ + uint16_t ilf_dsize; /* size of data/ext/root */ + uint64_t ilf_ino; /* inode number */ union { - __uint32_t ilfu_rdev; /* rdev value for dev inode*/ + uint32_t ilfu_rdev; /* rdev value for dev inode*/ uuid_t ilfu_uuid; /* mount point value */ } ilf_u; - __int64_t ilf_blkno; /* blkno of inode buffer */ - __int32_t ilf_len; /* len of inode buffer */ - __int32_t ilf_boffset; /* off of inode in buffer */ + int64_t ilf_blkno; /* blkno of inode buffer */ + int32_t ilf_len; /* len of inode buffer */ + int32_t ilf_boffset; /* off of inode in buffer */ } __attribute__((packed)) xfs_inode_log_format_32_t; typedef struct xfs_inode_log_format_64 { - __uint16_t ilf_type; /* inode log item type */ - __uint16_t ilf_size; /* size of this item */ - __uint32_t ilf_fields; /* flags for fields logged */ - __uint16_t ilf_asize; /* size of attr d/ext/root */ - __uint16_t ilf_dsize; /* size of data/ext/root */ - __uint32_t ilf_pad; /* pad for 64 bit boundary */ - __uint64_t ilf_ino; /* inode number */ + uint16_t ilf_type; /* inode log item type */ + uint16_t ilf_size; /* size of this item */ + uint32_t ilf_fields; /* flags for fields logged */ + uint16_t ilf_asize; /* size of attr d/ext/root */ + uint16_t ilf_dsize; /* size of data/ext/root */ + uint32_t ilf_pad; /* pad for 64 bit boundary */ + uint64_t ilf_ino; /* inode number */ union { - __uint32_t ilfu_rdev; /* rdev value for dev inode*/ + uint32_t ilfu_rdev; /* rdev value for dev inode*/ uuid_t ilfu_uuid; /* mount point value */ } ilf_u; - __int64_t ilf_blkno; /* blkno of inode buffer */ - __int32_t ilf_len; /* len of inode buffer */ - __int32_t ilf_boffset; /* off of inode in buffer */ + int64_t ilf_blkno; /* blkno of inode buffer */ + int32_t ilf_len; /* len of inode buffer */ + int32_t ilf_boffset; /* off of inode in buffer */ } xfs_inode_log_format_64_t; /* @@ -355,8 +355,8 @@ static inline int xfs_ilog_fdata(int w) * information. */ typedef struct xfs_ictimestamp { - __int32_t t_sec; /* timestamp seconds */ - __int32_t t_nsec; /* timestamp nanoseconds */ + int32_t t_sec; /* timestamp seconds */ + int32_t t_nsec; /* timestamp nanoseconds */ } xfs_ictimestamp_t; /* @@ -364,18 +364,18 @@ typedef struct xfs_ictimestamp { * except for the endianness annotations. */ typedef struct xfs_icdinode { - __uint16_t di_magic; /* inode magic # = XFS_DINODE_MAGIC */ - __uint16_t di_mode; /* mode and type of file */ - __int8_t di_version; /* inode version */ - __int8_t di_format; /* format of di_c data */ - __uint16_t di_onlink; /* old number of links to file */ - __uint32_t di_uid; /* owner's user id */ - __uint32_t di_gid; /* owner's group id */ - __uint32_t di_nlink; /* number of links to file */ - __uint16_t di_projid_lo; /* lower part of owner's project id */ - __uint16_t di_projid_hi; /* higher part of owner's project id */ - __uint8_t di_pad[6]; /* unused, zeroed space */ - __uint16_t di_flushiter; /* incremented on flush */ + uint16_t di_magic; /* inode magic # = XFS_DINODE_MAGIC */ + uint16_t di_mode; /* mode and type of file */ + int8_t di_version; /* inode version */ + int8_t di_format; /* format of di_c data */ + uint16_t di_onlink; /* old number of links to file */ + uint32_t di_uid; /* owner's user id */ + uint32_t di_gid; /* owner's group id */ + uint32_t di_nlink; /* number of links to file */ + uint16_t di_projid_lo; /* lower part of owner's project id */ + uint16_t di_projid_hi; /* higher part of owner's project id */ + uint8_t di_pad[6]; /* unused, zeroed space */ + uint16_t di_flushiter; /* incremented on flush */ xfs_ictimestamp_t di_atime; /* time last accessed */ xfs_ictimestamp_t di_mtime; /* time last modified */ xfs_ictimestamp_t di_ctime; /* time created/inode modified */ @@ -384,22 +384,22 @@ typedef struct xfs_icdinode { xfs_extlen_t di_extsize; /* basic/minimum extent size for file */ xfs_extnum_t di_nextents; /* number of extents in data fork */ xfs_aextnum_t di_anextents; /* number of extents in attribute fork*/ - __uint8_t di_forkoff; /* attr fork offs, <<3 for 64b align */ - __int8_t di_aformat; /* format of attr fork's data */ - __uint32_t di_dmevmask; /* DMIG event mask */ - __uint16_t di_dmstate; /* DMIG state info */ - __uint16_t di_flags; /* random flags, XFS_DIFLAG_... */ - __uint32_t di_gen; /* generation number */ + uint8_t di_forkoff; /* attr fork offs, <<3 for 64b align */ + int8_t di_aformat; /* format of attr fork's data */ + uint32_t di_dmevmask; /* DMIG event mask */ + uint16_t di_dmstate; /* DMIG state info */ + uint16_t di_flags; /* random flags, XFS_DIFLAG_... */ + uint32_t di_gen; /* generation number */ /* di_next_unlinked is the only non-core field in the old dinode */ xfs_agino_t di_next_unlinked;/* agi unlinked list ptr */ /* start of the extended dinode, writable fields */ - __uint32_t di_crc; /* CRC of the inode */ - __uint64_t di_changecount; /* number of attribute changes */ + uint32_t di_crc; /* CRC of the inode */ + uint64_t di_changecount; /* number of attribute changes */ xfs_lsn_t di_lsn; /* flush sequence */ - __uint64_t di_flags2; /* more random flags */ - __uint8_t di_pad2[16]; /* more padding for future expansion */ + uint64_t di_flags2; /* more random flags */ + uint8_t di_pad2[16]; /* more padding for future expansion */ /* fields only written to during inode creation */ xfs_ictimestamp_t di_crtime; /* time created */ @@ -458,7 +458,7 @@ typedef struct xfs_buf_log_format { unsigned short blf_size; /* size of this item */ unsigned short blf_flags; /* misc state */ unsigned short blf_len; /* number of blocks in this buf */ - __int64_t blf_blkno; /* starting blkno of this buf */ + int64_t blf_blkno; /* starting blkno of this buf */ unsigned int blf_map_size; /* used size of data bitmap in words */ unsigned int blf_data_map[XFS_BLF_DATAMAP_SIZE]; /* dirty bitmap */ } xfs_buf_log_format_t; @@ -506,7 +506,7 @@ xfs_blft_to_flags(struct xfs_buf_log_format *blf, enum xfs_blft type) blf->blf_flags |= ((type << XFS_BLFT_SHIFT) & XFS_BLFT_MASK); } -static inline __uint16_t +static inline uint16_t xfs_blft_from_flags(struct xfs_buf_log_format *blf) { return (blf->blf_flags & XFS_BLFT_MASK) >> XFS_BLFT_SHIFT; @@ -527,14 +527,14 @@ typedef struct xfs_extent { * conversion routine. */ typedef struct xfs_extent_32 { - __uint64_t ext_start; - __uint32_t ext_len; + uint64_t ext_start; + uint32_t ext_len; } __attribute__((packed)) xfs_extent_32_t; typedef struct xfs_extent_64 { - __uint64_t ext_start; - __uint32_t ext_len; - __uint32_t ext_pad; + uint64_t ext_start; + uint32_t ext_len; + uint32_t ext_pad; } xfs_extent_64_t; /* @@ -543,26 +543,26 @@ typedef struct xfs_extent_64 { * size is given by efi_nextents. */ typedef struct xfs_efi_log_format { - __uint16_t efi_type; /* efi log item type */ - __uint16_t efi_size; /* size of this item */ - __uint32_t efi_nextents; /* # extents to free */ - __uint64_t efi_id; /* efi identifier */ + uint16_t efi_type; /* efi log item type */ + uint16_t efi_size; /* size of this item */ + uint32_t efi_nextents; /* # extents to free */ + uint64_t efi_id; /* efi identifier */ xfs_extent_t efi_extents[1]; /* array of extents to free */ } xfs_efi_log_format_t; typedef struct xfs_efi_log_format_32 { - __uint16_t efi_type; /* efi log item type */ - __uint16_t efi_size; /* size of this item */ - __uint32_t efi_nextents; /* # extents to free */ - __uint64_t efi_id; /* efi identifier */ + uint16_t efi_type; /* efi log item type */ + uint16_t efi_size; /* size of this item */ + uint32_t efi_nextents; /* # extents to free */ + uint64_t efi_id; /* efi identifier */ xfs_extent_32_t efi_extents[1]; /* array of extents to free */ } __attribute__((packed)) xfs_efi_log_format_32_t; typedef struct xfs_efi_log_format_64 { - __uint16_t efi_type; /* efi log item type */ - __uint16_t efi_size; /* size of this item */ - __uint32_t efi_nextents; /* # extents to free */ - __uint64_t efi_id; /* efi identifier */ + uint16_t efi_type; /* efi log item type */ + uint16_t efi_size; /* size of this item */ + uint32_t efi_nextents; /* # extents to free */ + uint64_t efi_id; /* efi identifier */ xfs_extent_64_t efi_extents[1]; /* array of extents to free */ } xfs_efi_log_format_64_t; @@ -572,26 +572,26 @@ typedef struct xfs_efi_log_format_64 { * size is given by efd_nextents; */ typedef struct xfs_efd_log_format { - __uint16_t efd_type; /* efd log item type */ - __uint16_t efd_size; /* size of this item */ - __uint32_t efd_nextents; /* # of extents freed */ - __uint64_t efd_efi_id; /* id of corresponding efi */ + uint16_t efd_type; /* efd log item type */ + uint16_t efd_size; /* size of this item */ + uint32_t efd_nextents; /* # of extents freed */ + uint64_t efd_efi_id; /* id of corresponding efi */ xfs_extent_t efd_extents[1]; /* array of extents freed */ } xfs_efd_log_format_t; typedef struct xfs_efd_log_format_32 { - __uint16_t efd_type; /* efd log item type */ - __uint16_t efd_size; /* size of this item */ - __uint32_t efd_nextents; /* # of extents freed */ - __uint64_t efd_efi_id; /* id of corresponding efi */ + uint16_t efd_type; /* efd log item type */ + uint16_t efd_size; /* size of this item */ + uint32_t efd_nextents; /* # of extents freed */ + uint64_t efd_efi_id; /* id of corresponding efi */ xfs_extent_32_t efd_extents[1]; /* array of extents freed */ } __attribute__((packed)) xfs_efd_log_format_32_t; typedef struct xfs_efd_log_format_64 { - __uint16_t efd_type; /* efd log item type */ - __uint16_t efd_size; /* size of this item */ - __uint32_t efd_nextents; /* # of extents freed */ - __uint64_t efd_efi_id; /* id of corresponding efi */ + uint16_t efd_type; /* efd log item type */ + uint16_t efd_size; /* size of this item */ + uint32_t efd_nextents; /* # of extents freed */ + uint64_t efd_efi_id; /* id of corresponding efi */ xfs_extent_64_t efd_extents[1]; /* array of extents freed */ } xfs_efd_log_format_64_t; @@ -602,12 +602,12 @@ typedef struct xfs_efd_log_format_64 { * 32 bits : log_recovery code assumes that. */ typedef struct xfs_dq_logformat { - __uint16_t qlf_type; /* dquot log item type */ - __uint16_t qlf_size; /* size of this item */ + uint16_t qlf_type; /* dquot log item type */ + uint16_t qlf_size; /* size of this item */ xfs_dqid_t qlf_id; /* usr/grp/proj id : 32 bits */ - __int64_t qlf_blkno; /* blkno of dquot buffer */ - __int32_t qlf_len; /* len of dquot buffer */ - __uint32_t qlf_boffset; /* off of dquot in buffer */ + int64_t qlf_blkno; /* blkno of dquot buffer */ + int32_t qlf_len; /* len of dquot buffer */ + uint32_t qlf_boffset; /* off of dquot in buffer */ } xfs_dq_logformat_t; /* @@ -666,8 +666,8 @@ typedef struct xfs_qoff_logformat { * decoding can be done correctly. */ struct xfs_icreate_log { - __uint16_t icl_type; /* type of log format structure */ - __uint16_t icl_size; /* size of log format structure */ + uint16_t icl_type; /* type of log format structure */ + uint16_t icl_size; /* size of log format structure */ __be32 icl_ag; /* ag being allocated in */ __be32 icl_agbno; /* start block of inode range */ __be32 icl_count; /* number of inodes to initialise */ diff --git a/libxfs/xfs_quota_defs.h b/libxfs/xfs_quota_defs.h index 1b0a083..b6f63e2 100644 --- a/libxfs/xfs_quota_defs.h +++ b/libxfs/xfs_quota_defs.h @@ -27,8 +27,8 @@ * they may need 64-bit accounting. Hence, 64-bit quota-counters, * and quota-limits. This is a waste in the common case, but hey ... */ -typedef __uint64_t xfs_qcnt_t; -typedef __uint16_t xfs_qwarncnt_t; +typedef uint64_t xfs_qcnt_t; +typedef uint16_t xfs_qwarncnt_t; /* * flags for q_flags field in the dquot. diff --git a/libxfs/xfs_rtbitmap.c b/libxfs/xfs_rtbitmap.c index 5648ea5..09237cb 100644 --- a/libxfs/xfs_rtbitmap.c +++ b/libxfs/xfs_rtbitmap.c @@ -978,7 +978,7 @@ xfs_rtfree_extent( mp->m_sb.sb_rextents) { if (!(mp->m_rbmip->i_d.di_flags & XFS_DIFLAG_NEWRTBM)) mp->m_rbmip->i_d.di_flags |= XFS_DIFLAG_NEWRTBM; - *(__uint64_t *)&mp->m_rbmip->i_d.di_atime = 0; + *(uint64_t *)&mp->m_rbmip->i_d.di_atime = 0; xfs_trans_log_inode(tp, mp->m_rbmip, XFS_ILOG_CORE); } return 0; diff --git a/libxfs/xfs_sb.c b/libxfs/xfs_sb.c index 78ad889..28cef20 100644 --- a/libxfs/xfs_sb.c +++ b/libxfs/xfs_sb.c @@ -417,7 +417,7 @@ xfs_sb_quota_to_disk( struct xfs_dsb *to, struct xfs_sb *from) { - __uint16_t qflags = from->sb_qflags; + uint16_t qflags = from->sb_qflags; to->sb_uquotino = cpu_to_be64(from->sb_uquotino); if (xfs_sb_version_has_pquotino(from)) { @@ -712,7 +712,7 @@ xfs_sb_mount_common( mp->m_bmap_dmnr[1] = mp->m_bmap_dmxr[1] / 2; mp->m_bsize = XFS_FSB_TO_BB(mp, 1); - mp->m_ialloc_inos = (int)MAX((__uint16_t)XFS_INODES_PER_CHUNK, + mp->m_ialloc_inos = (int)MAX((uint16_t)XFS_INODES_PER_CHUNK, sbp->sb_inopblock); mp->m_ialloc_blks = mp->m_ialloc_inos >> sbp->sb_inopblog; diff --git a/libxfs/xfs_types.h b/libxfs/xfs_types.h index f0d145a..27b0ed6 100644 --- a/libxfs/xfs_types.h +++ b/libxfs/xfs_types.h @@ -18,34 +18,34 @@ #ifndef __XFS_TYPES_H__ #define __XFS_TYPES_H__ -typedef __uint32_t prid_t; /* project ID */ +typedef uint32_t prid_t; /* project ID */ -typedef __uint32_t xfs_agblock_t; /* blockno in alloc. group */ -typedef __uint32_t xfs_agino_t; /* inode # within allocation grp */ -typedef __uint32_t xfs_extlen_t; /* extent length in blocks */ -typedef __uint32_t xfs_agnumber_t; /* allocation group number */ -typedef __int32_t xfs_extnum_t; /* # of extents in a file */ -typedef __int16_t xfs_aextnum_t; /* # extents in an attribute fork */ -typedef __int64_t xfs_fsize_t; /* bytes in a file */ -typedef __uint64_t xfs_ufsize_t; /* unsigned bytes in a file */ +typedef uint32_t xfs_agblock_t; /* blockno in alloc. group */ +typedef uint32_t xfs_agino_t; /* inode # within allocation grp */ +typedef uint32_t xfs_extlen_t; /* extent length in blocks */ +typedef uint32_t xfs_agnumber_t; /* allocation group number */ +typedef int32_t xfs_extnum_t; /* # of extents in a file */ +typedef int16_t xfs_aextnum_t; /* # extents in an attribute fork */ +typedef int64_t xfs_fsize_t; /* bytes in a file */ +typedef uint64_t xfs_ufsize_t; /* unsigned bytes in a file */ -typedef __int32_t xfs_suminfo_t; /* type of bitmap summary info */ -typedef __int32_t xfs_rtword_t; /* word type for bitmap manipulations */ +typedef int32_t xfs_suminfo_t; /* type of bitmap summary info */ +typedef int32_t xfs_rtword_t; /* word type for bitmap manipulations */ -typedef __int64_t xfs_lsn_t; /* log sequence number */ -typedef __int32_t xfs_tid_t; /* transaction identifier */ +typedef int64_t xfs_lsn_t; /* log sequence number */ +typedef int32_t xfs_tid_t; /* transaction identifier */ -typedef __uint32_t xfs_dablk_t; /* dir/attr block number (in file) */ -typedef __uint32_t xfs_dahash_t; /* dir/attr hash value */ +typedef uint32_t xfs_dablk_t; /* dir/attr block number (in file) */ +typedef uint32_t xfs_dahash_t; /* dir/attr hash value */ -typedef __uint64_t xfs_fsblock_t; /* blockno in filesystem (agno|agbno) */ -typedef __uint64_t xfs_rfsblock_t; /* blockno in filesystem (raw) */ -typedef __uint64_t xfs_rtblock_t; /* extent (block) in realtime area */ -typedef __uint64_t xfs_fileoff_t; /* block number in a file */ -typedef __uint64_t xfs_filblks_t; /* number of blocks in a file */ +typedef uint64_t xfs_fsblock_t; /* blockno in filesystem (agno|agbno) */ +typedef uint64_t xfs_rfsblock_t; /* blockno in filesystem (raw) */ +typedef uint64_t xfs_rtblock_t; /* extent (block) in realtime area */ +typedef uint64_t xfs_fileoff_t; /* block number in a file */ +typedef uint64_t xfs_filblks_t; /* number of blocks in a file */ -typedef __int64_t xfs_srtblock_t; /* signed version of xfs_rtblock_t */ -typedef __int64_t xfs_sfiloff_t; /* signed block number in a file */ +typedef int64_t xfs_srtblock_t; /* signed version of xfs_rtblock_t */ +typedef int64_t xfs_sfiloff_t; /* signed block number in a file */ /* * Null values for the types. @@ -125,7 +125,7 @@ struct xfs_name { * uid_t and gid_t are hard-coded to 32 bits in the inode. * Hence, an 'id' in a dquot is 32 bits.. */ -typedef __uint32_t xfs_dqid_t; +typedef uint32_t xfs_dqid_t; /* * Constants for bit manipulations. diff --git a/logprint/log_misc.c b/logprint/log_misc.c index 7378fe1..52fc504 100644 --- a/logprint/log_misc.c +++ b/logprint/log_misc.c @@ -212,12 +212,12 @@ xlog_print_trans_header(char **ptr, int len) { xfs_trans_header_t *h; char *cptr = *ptr; - __uint32_t magic; + uint32_t magic; char *magic_c = (char *)&magic; *ptr += len; - magic=*(__uint32_t*)cptr; /* XXX be32_to_cpu soon */ + magic=*(uint32_t*)cptr; /* XXX be32_to_cpu soon */ if (len >= 4) { #if __BYTE_ORDER == __LITTLE_ENDIAN @@ -247,7 +247,7 @@ xlog_print_trans_buffer(char **ptr, int len, int *i, int num_ops) int num, skip; int super_block = 0; int bucket, col, buckets; - __int64_t blkno; + int64_t blkno; xfs_buf_log_format_t lbuf; int size, blen, map_size, struct_size; __be64 x, y; diff --git a/mdrestore/xfs_mdrestore.c b/mdrestore/xfs_mdrestore.c index ebc5e54..07614e0 100644 --- a/mdrestore/xfs_mdrestore.c +++ b/mdrestore/xfs_mdrestore.c @@ -65,7 +65,7 @@ perform_restore( int mb_count; xfs_metablock_t tmb; xfs_sb_t sb; - __int64_t bytes_read; + int64_t bytes_read; /* * read in first blocks (superblock 0), set "inprogress" flag for it, diff --git a/mkfs/proto.c b/mkfs/proto.c index cb34b28..f68d691 100644 --- a/mkfs/proto.c +++ b/mkfs/proto.c @@ -655,7 +655,7 @@ rtinit( mp->m_sb.sb_rbmino = rbmip->i_ino; rbmip->i_d.di_size = mp->m_sb.sb_rbmblocks * mp->m_sb.sb_blocksize; rbmip->i_d.di_flags = XFS_DIFLAG_NEWRTBM; - *(__uint64_t *)&rbmip->i_d.di_atime = 0; + *(uint64_t *)&rbmip->i_d.di_atime = 0; libxfs_trans_log_inode(tp, rbmip, XFS_ILOG_CORE); libxfs_log_sb(tp); mp->m_rbmip = rbmip; diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c index 700d12c..42dd69b 100644 --- a/mkfs/xfs_mkfs.c +++ b/mkfs/xfs_mkfs.c @@ -188,9 +188,9 @@ char *mopts[] = { NULL }; -#define TERABYTES(count, blog) ((__uint64_t)(count) << (40 - (blog))) -#define GIGABYTES(count, blog) ((__uint64_t)(count) << (30 - (blog))) -#define MEGABYTES(count, blog) ((__uint64_t)(count) << (20 - (blog))) +#define TERABYTES(count, blog) ((uint64_t)(count) << (40 - (blog))) +#define GIGABYTES(count, blog) ((uint64_t)(count) << (30 - (blog))) +#define MEGABYTES(count, blog) ((uint64_t)(count) << (20 - (blog))) /* * Use this macro before we have superblock and mount structure @@ -540,7 +540,7 @@ fixup_log_stripe_unit( xfs_rfsblock_t *logblocks, int blocklog) { - __uint64_t tmp_logblocks; + uint64_t tmp_logblocks; /* * Make sure that the log size is a multiple of the stripe unit @@ -572,7 +572,7 @@ fixup_internal_log_stripe( xfs_mount_t *mp, int lsflag, xfs_fsblock_t logstart, - __uint64_t agsize, + uint64_t agsize, int sunit, xfs_rfsblock_t *logblocks, int blocklog, @@ -596,7 +596,7 @@ fixup_internal_log_stripe( } void -validate_log_size(__uint64_t logblocks, int blocklog, int min_logblocks) +validate_log_size(uint64_t logblocks, int blocklog, int min_logblocks) { if (logblocks < min_logblocks) { fprintf(stderr, @@ -621,7 +621,7 @@ validate_log_size(__uint64_t logblocks, int blocklog, int min_logblocks) static int calc_default_imaxpct( int blocklog, - __uint64_t dblocks) + uint64_t dblocks) { /* * This returns the % of the disk space that is used for @@ -644,12 +644,12 @@ calc_default_imaxpct( void calc_default_ag_geometry( int blocklog, - __uint64_t dblocks, + uint64_t dblocks, int multidisk, - __uint64_t *agsize, - __uint64_t *agcount) + uint64_t *agsize, + uint64_t *agcount) { - __uint64_t blocks = 0; + uint64_t blocks = 0; int shift = 0; /* @@ -714,9 +714,9 @@ done: static void validate_ag_geometry( int blocklog, - __uint64_t dblocks, - __uint64_t agsize, - __uint64_t agcount) + uint64_t dblocks, + uint64_t agsize, + uint64_t agcount) { if (agsize < XFS_AG_MIN_BLOCKS(blocklog)) { fprintf(stderr, @@ -792,7 +792,7 @@ zero_old_xfs_structures( { void *buf; xfs_sb_t sb; - __uint32_t bsize; + uint32_t bsize; int i; xfs_off_t off; int tmp; @@ -843,8 +843,8 @@ zero_old_xfs_structures( i != sb.sb_blocklog) goto done; - if (sb.sb_dblocks > ((__uint64_t)sb.sb_agcount * sb.sb_agblocks) || - sb.sb_dblocks < ((__uint64_t)(sb.sb_agcount - 1) * + if (sb.sb_dblocks > ((uint64_t)sb.sb_agcount * sb.sb_agblocks) || + sb.sb_dblocks < ((uint64_t)(sb.sb_agcount - 1) * sb.sb_agblocks + XFS_MIN_AG_BLOCKS)) goto done; @@ -864,7 +864,7 @@ done: } static void -discard_blocks(dev_t dev, __uint64_t nsectors) +discard_blocks(dev_t dev, uint64_t nsectors) { int fd; @@ -882,11 +882,11 @@ main( int argc, char **argv) { - __uint64_t agcount; + uint64_t agcount; xfs_agf_t *agf; xfs_agi_t *agi; xfs_agnumber_t agno; - __uint64_t agsize; + uint64_t agsize; xfs_alloc_rec_t *arec; int attrversion; int projid16bit; @@ -971,10 +971,10 @@ main( xfs_sb_t *sbp; int sectorlog; unsigned int sectorsize; - __uint64_t sector_mask; + uint64_t sector_mask; int slflag; int ssflag; - __uint64_t tmp_agsize; + uint64_t tmp_agsize; uuid_t uuid; int worst_freelist; libxfs_init_t xi; @@ -1086,9 +1086,9 @@ main( reqval('d', dopts, D_AGCOUNT); if (daflag) respec('d', dopts, D_AGCOUNT); - agcount = (__uint64_t) + agcount = (uint64_t) strtoul(value, NULL, 10); - if ((__int64_t)agcount <= 0) + if ((int64_t)agcount <= 0) illegal(value, "d agcount"); daflag = 1; break; @@ -1915,7 +1915,7 @@ _("warning: sparse inodes not supported without CRC support, disabled.\n")); usage(); } if (dsize) { - __uint64_t dbytes; + uint64_t dbytes; dbytes = cvtnum(blocksize, sectorsize, dsize); if (dbytes % XFS_MIN_BLOCKSIZE) { @@ -1952,7 +1952,7 @@ _("warning: sparse inodes not supported without CRC support, disabled.\n")); usage(); } if (logsize) { - __uint64_t logbytes; + uint64_t logbytes; logbytes = cvtnum(blocksize, sectorsize, logsize); if (logbytes % XFS_MIN_BLOCKSIZE) { @@ -1974,7 +1974,7 @@ _("warning: sparse inodes not supported without CRC support, disabled.\n")); usage(); } if (rtsize) { - __uint64_t rtbytes; + uint64_t rtbytes; rtbytes = cvtnum(blocksize, sectorsize, rtsize); if (rtbytes % XFS_MIN_BLOCKSIZE) { @@ -1994,7 +1994,7 @@ _("warning: sparse inodes not supported without CRC support, disabled.\n")); * If specified, check rt extent size against its constraints. */ if (rtextsize) { - __uint64_t rtextbytes; + uint64_t rtextbytes; rtextbytes = cvtnum(blocksize, sectorsize, rtextsize); if (rtextbytes % blocksize) { @@ -2022,8 +2022,8 @@ _("warning: sparse inodes not supported without CRC support, disabled.\n")); * and the underlying volume is striped, then set rtextblocks * to the stripe width. */ - __uint64_t rswidth; - __uint64_t rtextbytes; + uint64_t rswidth; + uint64_t rtextbytes; if (!norsflag && !xi.risfile && !(!rtsize && xi.disfile)) rswidth = ft.rtswidth; @@ -2101,10 +2101,10 @@ _("warning: sparse inodes not supported without CRC support, disabled.\n")); * multiple of the sector size, or 1024, whichever is larger. */ - sector_mask = (__uint64_t)-1 << (MAX(sectorlog, 10) - BBSHIFT); + sector_mask = (uint64_t)-1 << (MAX(sectorlog, 10) - BBSHIFT); xi.dsize &= sector_mask; xi.rtsize &= sector_mask; - xi.logBBsize &= (__uint64_t)-1 << (MAX(lsectorlog, 10) - BBSHIFT); + xi.logBBsize &= (uint64_t)-1 << (MAX(lsectorlog, 10) - BBSHIFT); if (!force_overwrite) { if (check_overwrite(dfile) || @@ -2473,9 +2473,9 @@ _("size %s specified for log subvolume is too large, maximum is %lld blocks\n"), mp = &mbuf; sbp = &mp->m_sb; memset(mp, 0, sizeof(xfs_mount_t)); - sbp->sb_blocklog = (__uint8_t)blocklog; - sbp->sb_sectlog = (__uint8_t)sectorlog; - sbp->sb_agblklog = (__uint8_t)libxfs_log2_roundup((unsigned int)agsize); + sbp->sb_blocklog = (uint8_t)blocklog; + sbp->sb_sectlog = (uint8_t)sectorlog; + sbp->sb_agblklog = (uint8_t)libxfs_log2_roundup((unsigned int)agsize); sbp->sb_agblocks = (xfs_agblock_t)agsize; mp->m_blkbb_log = sbp->sb_blocklog - BBSHIFT; mp->m_sectbb_log = sbp->sb_sectlog - BBSHIFT; @@ -2603,14 +2603,14 @@ _("size %s specified for log subvolume is too large, maximum is %lld blocks\n"), sbp->sb_agcount = (xfs_agnumber_t)agcount; sbp->sb_rbmblocks = nbmblocks; sbp->sb_logblocks = (xfs_extlen_t)logblocks; - sbp->sb_sectsize = (__uint16_t)sectorsize; - sbp->sb_inodesize = (__uint16_t)isize; - sbp->sb_inopblock = (__uint16_t)(blocksize / isize); - sbp->sb_sectlog = (__uint8_t)sectorlog; - sbp->sb_inodelog = (__uint8_t)inodelog; - sbp->sb_inopblog = (__uint8_t)(blocklog - inodelog); + sbp->sb_sectsize = (uint16_t)sectorsize; + sbp->sb_inodesize = (uint16_t)isize; + sbp->sb_inopblock = (uint16_t)(blocksize / isize); + sbp->sb_sectlog = (uint8_t)sectorlog; + sbp->sb_inodelog = (uint8_t)inodelog; + sbp->sb_inopblog = (uint8_t)(blocklog - inodelog); sbp->sb_rextslog = - (__uint8_t)(rtextents ? + (uint8_t)(rtextents ? libxfs_highbit32((unsigned int)rtextents) : 0); sbp->sb_inprogress = 1; /* mkfs is in progress */ sbp->sb_imax_pct = imaxpct; @@ -2638,8 +2638,8 @@ _("size %s specified for log subvolume is too large, maximum is %lld blocks\n"), } else sbp->sb_inoalignmt = 0; if (lsectorsize != BBSIZE || sectorsize != BBSIZE) { - sbp->sb_logsectlog = (__uint8_t)lsectorlog; - sbp->sb_logsectsize = (__uint16_t)lsectorsize; + sbp->sb_logsectlog = (uint8_t)lsectorlog; + sbp->sb_logsectsize = (uint16_t)lsectorsize; } else { sbp->sb_logsectlog = 0; sbp->sb_logsectsize = 0; diff --git a/quota/edit.c b/quota/edit.c index c55fa1a..f970e66 100644 --- a/quota/edit.c +++ b/quota/edit.c @@ -115,16 +115,16 @@ warn_help(void) static void set_limits( - __uint32_t id, + uint32_t id, uint type, uint mask, char *dev, - __uint64_t *bsoft, - __uint64_t *bhard, - __uint64_t *isoft, - __uint64_t *ihard, - __uint64_t *rtbsoft, - __uint64_t *rtbhard) + uint64_t *bsoft, + uint64_t *bhard, + uint64_t *isoft, + uint64_t *ihard, + uint64_t *rtbsoft, + uint64_t *rtbhard) { fs_disk_quota_t d; @@ -152,12 +152,12 @@ set_user_limits( char *name, uint type, uint mask, - __uint64_t *bsoft, - __uint64_t *bhard, - __uint64_t *isoft, - __uint64_t *ihard, - __uint64_t *rtbsoft, - __uint64_t *rtbhard) + uint64_t *bsoft, + uint64_t *bhard, + uint64_t *isoft, + uint64_t *ihard, + uint64_t *rtbsoft, + uint64_t *rtbhard) { uid_t uid = uid_from_string(name); @@ -175,12 +175,12 @@ set_group_limits( char *name, uint type, uint mask, - __uint64_t *bsoft, - __uint64_t *bhard, - __uint64_t *isoft, - __uint64_t *ihard, - __uint64_t *rtbsoft, - __uint64_t *rtbhard) + uint64_t *bsoft, + uint64_t *bhard, + uint64_t *isoft, + uint64_t *ihard, + uint64_t *rtbsoft, + uint64_t *rtbhard) { gid_t gid = gid_from_string(name); @@ -198,12 +198,12 @@ set_project_limits( char *name, uint type, uint mask, - __uint64_t *bsoft, - __uint64_t *bhard, - __uint64_t *isoft, - __uint64_t *ihard, - __uint64_t *rtbsoft, - __uint64_t *rtbhard) + uint64_t *bsoft, + uint64_t *bhard, + uint64_t *isoft, + uint64_t *ihard, + uint64_t *rtbsoft, + uint64_t *rtbhard) { prid_t prid = prid_from_string(name); @@ -224,7 +224,7 @@ extractb( int length, uint blocksize, uint sectorsize, - __uint64_t *value) + uint64_t *value) { long long v; char *s = string; @@ -238,7 +238,7 @@ extractb( progname, s); return 0; } - *value = (__uint64_t)v >> 9; /* syscalls use basic blocks */ + *value = (uint64_t)v >> 9; /* syscalls use basic blocks */ if (v > 0 && *value == 0) fprintf(stderr, _("%s: Warning: `%s' in quota blocks is 0 (unlimited).\n"), progname, s); return 1; @@ -252,7 +252,7 @@ extracti( char *string, const char *prefix, int length, - __uint64_t *value) + uint64_t *value) { char *sp, *s = string; @@ -270,7 +270,7 @@ limit_f( char **argv) { char *name; - __uint64_t bsoft, bhard, isoft, ihard, rtbsoft, rtbhard; + uint64_t bsoft, bhard, isoft, ihard, rtbsoft, rtbhard; int c, type = 0, mask = 0, flags = 0; uint bsize, ssize, endoptions; @@ -379,8 +379,8 @@ restore_file( char *dev = NULL; uint mask; int cnt; - __uint32_t id; - __uint64_t bsoft, bhard, isoft, ihard, rtbsoft, rtbhard; + uint32_t id; + uint64_t bsoft, bhard, isoft, ihard, rtbsoft, rtbhard; while (fgets(buffer, sizeof(buffer), fp) != NULL) { if (strncmp("fs = ", buffer, 5) == 0) { @@ -531,7 +531,7 @@ timer_f( static void set_warnings( - __uint32_t id, + uint32_t id, uint type, uint mask, char *dev, diff --git a/quota/free.c b/quota/free.c index dcbe8ce..30ce38b 100644 --- a/quota/free.c +++ b/quota/free.c @@ -48,20 +48,20 @@ free_help(void) static int mount_free_space_data( struct fs_path *mount, - __uint64_t *bcount, - __uint64_t *bused, - __uint64_t *bfree, - __uint64_t *icount, - __uint64_t *iused, - __uint64_t *ifree, - __uint64_t *rcount, - __uint64_t *rused, - __uint64_t *rfree) + uint64_t *bcount, + uint64_t *bused, + uint64_t *bfree, + uint64_t *icount, + uint64_t *iused, + uint64_t *ifree, + uint64_t *rcount, + uint64_t *rused, + uint64_t *rfree) { struct xfs_fsop_counts fscounts; struct xfs_fsop_geom fsgeo; struct statfs st; - __uint64_t logsize, count, free; + uint64_t logsize, count, free; int fd; if ((fd = open(mount->fs_dir, O_RDONLY)) < 0) { @@ -115,15 +115,15 @@ mount_free_space_data( static int projects_free_space_data( struct fs_path *path, - __uint64_t *bcount, - __uint64_t *bused, - __uint64_t *bfree, - __uint64_t *icount, - __uint64_t *iused, - __uint64_t *ifree, - __uint64_t *rcount, - __uint64_t *rused, - __uint64_t *rfree) + uint64_t *bcount, + uint64_t *bused, + uint64_t *bfree, + uint64_t *icount, + uint64_t *iused, + uint64_t *ifree, + uint64_t *rcount, + uint64_t *rused, + uint64_t *rfree) { fs_quota_stat_t qfs; fs_disk_quota_t d; @@ -209,9 +209,9 @@ free_space( fs_path_t *path, uint flags) { - __uint64_t bcount, bused, bfree; - __uint64_t icount, iused, ifree; - __uint64_t rcount, rused, rfree; + uint64_t bcount, bused, bfree; + uint64_t icount, iused, ifree; + uint64_t rcount, rused, rfree; char a[8], s[8], u[8], p[8]; int count; diff --git a/quota/quot.c b/quota/quot.c index 9116e48..19d2253 100644 --- a/quota/quot.c +++ b/quota/quot.c @@ -25,17 +25,17 @@ typedef struct du { struct du *next; - __uint64_t blocks; - __uint64_t blocks30; - __uint64_t blocks60; - __uint64_t blocks90; - __uint64_t nfiles; - __uint32_t id; + uint64_t blocks; + uint64_t blocks30; + uint64_t blocks60; + uint64_t blocks90; + uint64_t nfiles; + uint32_t id; } du_t; #define TSIZE 500 -static __uint64_t sizes[TSIZE]; -static __uint64_t overflow; +static uint64_t sizes[TSIZE]; +static uint64_t overflow; #define NDU 60000 #define DUHASH 8209 @@ -83,8 +83,8 @@ quot_bulkstat_add( { du_t *dp; du_t **hp; - __uint64_t size; - __uint32_t i, id; + uint64_t size; + uint32_t i, id; if ((p->bs_mode & S_IFMT) == 0) return; @@ -202,7 +202,7 @@ qcompare( return 0; } -typedef char *(*idtoname_t)(__uint32_t); +typedef char *(*idtoname_t)(uint32_t); static void quot_report_mount_any_type( @@ -290,7 +290,7 @@ quot_histogram_mount( fs_path_t *mount, uint flags) { - __uint64_t t = 0; + uint64_t t = 0; int i; fprintf(fp, _("%s (%s):\n"), mount->fs_name, mount->fs_dir); diff --git a/quota/quota.c b/quota/quota.c index f6b24c3..d02e54e 100644 --- a/quota/quota.c +++ b/quota/quota.c @@ -51,7 +51,7 @@ quota_help(void) static int quota_mount( FILE *fp, - __uint32_t id, + uint32_t id, char *name, uint form, uint type, @@ -180,7 +180,7 @@ quota_mount( static void quota( FILE *fp, - __uint32_t id, + uint32_t id, char *name, uint form, uint type, diff --git a/quota/quota.h b/quota/quota.h index 416f335..eb7a4fc 100644 --- a/quota/quota.h +++ b/quota/quota.h @@ -51,9 +51,9 @@ enum { extern char *type_to_string(uint __type); extern char *form_to_string(uint __form); extern char *time_to_string(time_t __time, uint __flags); -extern char *bbs_to_string(__uint64_t __v, char *__c, uint __size); -extern char *num_to_string(__uint64_t __v, char *__c, uint __size); -extern char *pct_to_string(__uint64_t __v, __uint64_t __t, char *__c, uint __s); +extern char *bbs_to_string(uint64_t __v, char *__c, uint __size); +extern char *num_to_string(uint64_t __v, char *__c, uint __size); +extern char *pct_to_string(uint64_t __v, uint64_t __t, char *__c, uint __s); extern FILE *fopen_write_secure(char *__filename); @@ -78,7 +78,7 @@ enum { * Identifier (uid/gid/prid) cache routines */ #define NMAX 32 -extern char *uid_to_name(__uint32_t __uid); -extern char *gid_to_name(__uint32_t __gid); -extern char *prid_to_name(__uint32_t __prid); +extern char *uid_to_name(uint32_t __uid); +extern char *gid_to_name(uint32_t __gid); +extern char *prid_to_name(uint32_t __prid); extern bool isdigits_only(const char *); diff --git a/quota/report.c b/quota/report.c index c77b24f..d03f88d 100644 --- a/quota/report.c +++ b/quota/report.c @@ -289,7 +289,7 @@ report_header( static int report_mount( FILE *fp, - __uint32_t id, + uint32_t id, char *name, uint form, uint type, diff --git a/quota/state.c b/quota/state.c index 8186762..82dd26d 100644 --- a/quota/state.c +++ b/quota/state.c @@ -135,7 +135,7 @@ static void state_timelimit( FILE *fp, uint form, - __uint32_t timelimit) + uint32_t timelimit) { fprintf(fp, _("%s grace time: %s\n"), form_to_string(form), diff --git a/quota/util.c b/quota/util.c index 179aafd..f51f375 100644 --- a/quota/util.c +++ b/quota/util.c @@ -83,8 +83,8 @@ round_snprintf( size_t size, const char *fmt_round, const char *fmt_not_round, - __uint64_t value, - __uint64_t divisor) + uint64_t value, + uint64_t divisor) { double v = (double)value / divisor; @@ -96,23 +96,23 @@ round_snprintf( } /* Basic blocks (512) bytes are returned from quotactl */ -#define BBS_TO_EXABYTES(bbs) ((__uint64_t)(bbs)>>51) -#define BBS_TO_PETABYTES(bbs) ((__uint64_t)(bbs)>>41) -#define BBS_TO_TERABYTES(bbs) ((__uint64_t)(bbs)>>31) -#define BBS_TO_GIGABYTES(bbs) ((__uint64_t)(bbs)>>21) -#define BBS_TO_MEGABYTES(bbs) ((__uint64_t)(bbs)>>11) -#define BBS_TO_KILOBYTES(bbs) ((__uint64_t)(bbs)>>1) - -#define BBEXABYTE ((__uint64_t)1<<51) -#define BBPETABYTE ((__uint64_t)1<<41) -#define BBTERABYTE ((__uint64_t)1<<31) -#define BBGIGABYTE ((__uint64_t)1<<21) -#define BBMEGABYTE ((__uint64_t)1<<11) -#define BBKILOBYTE ((__uint64_t)1<< 1) +#define BBS_TO_EXABYTES(bbs) ((uint64_t)(bbs)>>51) +#define BBS_TO_PETABYTES(bbs) ((uint64_t)(bbs)>>41) +#define BBS_TO_TERABYTES(bbs) ((uint64_t)(bbs)>>31) +#define BBS_TO_GIGABYTES(bbs) ((uint64_t)(bbs)>>21) +#define BBS_TO_MEGABYTES(bbs) ((uint64_t)(bbs)>>11) +#define BBS_TO_KILOBYTES(bbs) ((uint64_t)(bbs)>>1) + +#define BBEXABYTE ((uint64_t)1<<51) +#define BBPETABYTE ((uint64_t)1<<41) +#define BBTERABYTE ((uint64_t)1<<31) +#define BBGIGABYTE ((uint64_t)1<<21) +#define BBMEGABYTE ((uint64_t)1<<11) +#define BBKILOBYTE ((uint64_t)1<< 1) char * bbs_to_string( - __uint64_t v, + uint64_t v, char *sp, uint size) { @@ -135,17 +135,17 @@ bbs_to_string( return sp; } -#define THOUSAND ((__uint64_t)1000) -#define MILLION ((__uint64_t)1000*1000) -#define BILLION ((__uint64_t)1000*1000*1000) -#define TRILLION ((__uint64_t)1000*1000*1000*1000) -#define GAZILLION ((__uint64_t)1000*1000*1000*1000*1000) -#define RIDICULOUS ((__uint64_t)1000*1000*1000*1000*1000*1000) -#define STOPALREADY ((__uint64_t)1000*1000*1000*1000*1000*1000*1000) +#define THOUSAND ((uint64_t)1000) +#define MILLION ((uint64_t)1000*1000) +#define BILLION ((uint64_t)1000*1000*1000) +#define TRILLION ((uint64_t)1000*1000*1000*1000) +#define GAZILLION ((uint64_t)1000*1000*1000*1000*1000) +#define RIDICULOUS ((uint64_t)1000*1000*1000*1000*1000*1000) +#define STOPALREADY ((uint64_t)1000*1000*1000*1000*1000*1000*1000) char * num_to_string( - __uint64_t v, + uint64_t v, char *sp, uint size) { @@ -172,8 +172,8 @@ num_to_string( char * pct_to_string( - __uint64_t portion, - __uint64_t whole, + uint64_t portion, + uint64_t whole, char *buf, uint size) { @@ -226,7 +226,7 @@ type_to_string( #define IDMASK (NID-1) typedef struct { - __uint32_t id; + uint32_t id; char name[NMAX+1]; } idcache_t; @@ -239,7 +239,7 @@ static int pentriesleft = NID; static idcache_t * getnextpwent( - __uint32_t id, + uint32_t id, int byid) { struct passwd *pw; @@ -255,7 +255,7 @@ getnextpwent( static idcache_t * getnextgrent( - __uint32_t id, + uint32_t id, int byid) { struct group *gr; @@ -270,7 +270,7 @@ getnextgrent( static idcache_t * getnextprent( - __uint32_t id, + uint32_t id, int byid) { fs_project_t *pr; @@ -285,7 +285,7 @@ getnextprent( char * uid_to_name( - __uint32_t id) + uint32_t id) { idcache_t *ncp, *idp; @@ -322,7 +322,7 @@ uid_to_name( char * gid_to_name( - __uint32_t id) + uint32_t id) { idcache_t *ncp, *idp; @@ -359,7 +359,7 @@ gid_to_name( char * prid_to_name( - __uint32_t id) + uint32_t id) { idcache_t *ncp, *idp; diff --git a/repair/README b/repair/README index 7f168e6..4692463 100644 --- a/repair/README +++ b/repair/README @@ -189,7 +189,7 @@ D - 0) look at usage of XFS_MAKE_IPTR(). It does the right D - 0) look at references to XFS_INODES_PER_CHUNK. change the - ones that really mean sizeof(__uint64_t)*NBBY to + ones that really mean sizeof(uint64_t)*NBBY to something else (like that only defined as a constant INOS_PER_IREC. this isn't as important since XFS_INODES_PER_CHUNK will never chang diff --git a/repair/agheader.h b/repair/agheader.h index 6b2974c..e9075d1 100644 --- a/repair/agheader.h +++ b/repair/agheader.h @@ -20,7 +20,7 @@ typedef struct fs_geometry { /* * these types should match the superblock types */ - __uint32_t sb_blocksize; /* blocksize (bytes) */ + uint32_t sb_blocksize; /* blocksize (bytes) */ xfs_rfsblock_t sb_dblocks; /* # data blocks */ xfs_rfsblock_t sb_rblocks; /* # realtime blocks */ xfs_rtblock_t sb_rextents; /* # realtime extents */ @@ -30,9 +30,9 @@ typedef struct fs_geometry { xfs_agnumber_t sb_agcount; /* # of ags */ xfs_extlen_t sb_rbmblocks; /* # of rt bitmap blocks */ xfs_extlen_t sb_logblocks; /* # of log blocks */ - __uint16_t sb_sectsize; /* volume sector size (bytes) */ - __uint16_t sb_inodesize; /* inode size (bytes) */ - __uint8_t sb_imax_pct; /* max % of fs for inode space */ + uint16_t sb_sectsize; /* volume sector size (bytes) */ + uint16_t sb_inodesize; /* inode size (bytes) */ + uint8_t sb_imax_pct; /* max % of fs for inode space */ /* * these don't have to match the superblock types but are placed @@ -49,10 +49,10 @@ typedef struct fs_geometry { /* * fields after this point have to be checked manually in compare_sb() */ - __uint8_t sb_shared_vn; /* shared version number */ + uint8_t sb_shared_vn; /* shared version number */ xfs_extlen_t sb_inoalignmt; /* inode chunk alignment, fsblocks */ - __uint32_t sb_unit; /* stripe or raid unit */ - __uint32_t sb_width; /* stripe or width unit */ + uint32_t sb_unit; /* stripe or raid unit */ + uint32_t sb_width; /* stripe or width unit */ /* * these don't have to match, they track superblock properties diff --git a/repair/attr_repair.h b/repair/attr_repair.h index 7010e4f..02e056f 100644 --- a/repair/attr_repair.h +++ b/repair/attr_repair.h @@ -32,10 +32,10 @@ #define ACL_WRITE 02 #define ACL_EXECUTE 01 -typedef __uint16_t xfs_acl_perm_t; -typedef __int32_t xfs_acl_type_t; -typedef __int32_t xfs_acl_tag_t; -typedef __int32_t xfs_acl_id_t; +typedef uint16_t xfs_acl_perm_t; +typedef int32_t xfs_acl_type_t; +typedef int32_t xfs_acl_tag_t; +typedef int32_t xfs_acl_id_t; /* * "icacl" = in-core ACL. There is no equivalent in the XFS kernel code, @@ -50,7 +50,7 @@ struct xfs_icacl_entry { }; struct xfs_icacl { - __int32_t acl_cnt; + int32_t acl_cnt; struct xfs_icacl_entry acl_entry[0]; }; @@ -59,14 +59,14 @@ struct xfs_icacl { */ #define XFS_MAC_MAX_SETS 250 typedef struct xfs_mac_label { - __uint8_t ml_msen_type; /* MSEN label type */ - __uint8_t ml_mint_type; /* MINT label type */ - __uint8_t ml_level; /* Hierarchical level */ - __uint8_t ml_grade; /* Hierarchical grade */ - __uint16_t ml_catcount; /* Category count */ - __uint16_t ml_divcount; /* Division count */ + uint8_t ml_msen_type; /* MSEN label type */ + uint8_t ml_mint_type; /* MINT label type */ + uint8_t ml_level; /* Hierarchical level */ + uint8_t ml_grade; /* Hierarchical grade */ + uint16_t ml_catcount; /* Category count */ + uint16_t ml_divcount; /* Division count */ /* Category set, then Division set */ - __uint16_t ml_list[XFS_MAC_MAX_SETS]; + uint16_t ml_list[XFS_MAC_MAX_SETS]; } xfs_mac_label_t; /* MSEN label type names. Choose an upper case ASCII character. */ @@ -93,7 +93,7 @@ typedef struct xfs_mac_label { /* * Capabilities (IRIX) */ -typedef __uint64_t xfs_cap_value_t; +typedef uint64_t xfs_cap_value_t; typedef struct xfs_cap_set { xfs_cap_value_t cap_effective; /* use in capability checks */ diff --git a/repair/avl64.c b/repair/avl64.c index 51cd624..8f4a121 100644 --- a/repair/avl64.c +++ b/repair/avl64.c @@ -70,8 +70,8 @@ avl64_checktree( avl64node_t *root) { avl64node_t *nlast, *nnext, *np; - __uint64_t offset = 0; - __uint64_t end; + uint64_t offset = 0; + uint64_t end; nlast = nnext = root; @@ -583,8 +583,8 @@ attach: avl64node_t * avl64_findanyrange( avl64tree_desc_t *tree, - __uint64_t start, - __uint64_t end, + uint64_t start, + uint64_t end, int checklen) { avl64node_t *np = tree->avl_root; @@ -655,7 +655,7 @@ avl64_findanyrange( avl64node_t * avl64_findrange( avl64tree_desc_t *tree, - __uint64_t value) + uint64_t value) { avl64node_t *np = tree->avl_root; @@ -682,10 +682,10 @@ avl64_findrange( avl64node_t * avl64_find( avl64tree_desc_t *tree, - __uint64_t value) + uint64_t value) { avl64node_t *np = tree->avl_root; - __uint64_t nvalue; + uint64_t nvalue; while (np) { nvalue = AVL_START(tree, np); @@ -909,8 +909,8 @@ static avl64node_t * avl64_insert_find_growth( avl64tree_desc_t *tree, - __uint64_t start, /* range start at start, */ - __uint64_t end, /* exclusive */ + uint64_t start, /* range start at start, */ + uint64_t end, /* exclusive */ int *growthp) /* OUT */ { avl64node_t *root = tree->avl_root; @@ -963,7 +963,7 @@ avl64_insert_grow( int growth) { avl64node_t *nnext; - __uint64_t start = AVL_START(tree, newnode); + uint64_t start = AVL_START(tree, newnode); if (growth == AVL_BACK) { @@ -1005,8 +1005,8 @@ avl64_insert( avl64node_t *newnode) { avl64node_t *np; - __uint64_t start = AVL_START(tree, newnode); - __uint64_t end = AVL_END(tree, newnode); + uint64_t start = AVL_START(tree, newnode); + uint64_t end = AVL_END(tree, newnode); int growth; ASSERT(newnode); @@ -1159,16 +1159,16 @@ avl64ops_t avl_debug_ops = { avl_debug_end, } -static __uint64_t +static uint64_t avl64_debug_start(avl64node_t *node) { - return (__uint64_t)(struct avl_debug_node *)node->avl_start; + return (uint64_t)(struct avl_debug_node *)node->avl_start; } -static __uint64_t +static uint64_t avl64_debug_end(avl64node_t *node) { - return (__uint64_t) + return (uint64_t) ((struct avl_debug_node *)node->avl_start + (struct avl_debug_node *)node->avl_size); } @@ -1304,7 +1304,7 @@ main() avl64node_t * avl64_findadjacent( avl64tree_desc_t *tree, - __uint64_t value, + uint64_t value, int dir) { avl64node_t *np = tree->avl_root; @@ -1378,8 +1378,8 @@ avl64_findadjacent( void avl64_findranges( avl64tree_desc_t *tree, - __uint64_t start, - __uint64_t end, + uint64_t start, + uint64_t end, avl64node_t **startp, avl64node_t **endp) { diff --git a/repair/avl64.h b/repair/avl64.h index fd19321..cd079a0 100644 --- a/repair/avl64.h +++ b/repair/avl64.h @@ -32,8 +32,8 @@ typedef struct avl64node { * avl-tree operations */ typedef struct avl64ops { - __uint64_t (*avl_start)(avl64node_t *); - __uint64_t (*avl_end)(avl64node_t *); + uint64_t (*avl_start)(avl64node_t *); + uint64_t (*avl_end)(avl64node_t *); } avl64ops_t; /* @@ -89,32 +89,32 @@ avl64_init_tree( avl64node_t * avl64_findrange( avl64tree_desc_t *tree, - __uint64_t value); + uint64_t value); avl64node_t * avl64_find( avl64tree_desc_t *tree, - __uint64_t value); + uint64_t value); avl64node_t * avl64_findanyrange( avl64tree_desc_t *tree, - __uint64_t start, - __uint64_t end, + uint64_t start, + uint64_t end, int checklen); avl64node_t * avl64_findadjacent( avl64tree_desc_t *tree, - __uint64_t value, + uint64_t value, int dir); void avl64_findranges( avl64tree_desc_t *tree, - __uint64_t start, - __uint64_t end, + uint64_t start, + uint64_t end, avl64node_t **startp, avl64node_t **endp); diff --git a/repair/dino_chunks.c b/repair/dino_chunks.c index 7dbaca6..ecfd0f1 100644 --- a/repair/dino_chunks.c +++ b/repair/dino_chunks.c @@ -826,7 +826,7 @@ next_readbuf: * we do now, this is where to start. */ if (is_used) { - __uint16_t di_mode; + uint16_t di_mode; if (is_inode_free(ino_rec, irec_offset)) { if (verbose || no_modify) { diff --git a/repair/dinode.c b/repair/dinode.c index df28e9e..3f3bf08 100644 --- a/repair/dinode.c +++ b/repair/dinode.c @@ -742,7 +742,7 @@ _("%s fork in ino %" PRIu64 " claims dup extent, " case XR_E_FREE1: do_warn( _("%s fork in ino %" PRIu64 " claims free block %" PRIu64 "\n"), - forkname, ino, (__uint64_t) b); + forkname, ino, (uint64_t) b); /* fall through ... */ case XR_E_UNKNOWN: set_bmap_ext(agno, agbno, blen, XR_E_INUSE); @@ -888,7 +888,7 @@ process_btinode( int type, int *dirty, xfs_rfsblock_t *tot, - __uint64_t *nex, + uint64_t *nex, blkmap_t **blkmapp, int whichfork, int check_dups) @@ -904,7 +904,7 @@ process_btinode( int level; int numrecs; bmap_cursor_t cursor; - __uint64_t magic; + uint64_t magic; dib = (xfs_bmdr_block_t *)XFS_DFORK_PTR(dip, whichfork); lino = XFS_AGINO_TO_INO(mp, agno, ino); @@ -1063,7 +1063,7 @@ process_exinode( int type, int *dirty, xfs_rfsblock_t *tot, - __uint64_t *nex, + uint64_t *nex, blkmap_t **blkmapp, int whichfork, int check_dups) @@ -1165,14 +1165,14 @@ process_symlink_extlist(xfs_mount_t *mp, xfs_ino_t lino, xfs_dinode_t *dino) do_warn( _("mismatch between format (%d) and size (%" PRId64 ") in symlink ino %" PRIu64 "\n"), dino->di_format, - (__int64_t)be64_to_cpu(dino->di_size), lino); + (int64_t)be64_to_cpu(dino->di_size), lino); return 1; } if (dino->di_format == XFS_DINODE_FMT_LOCAL) { do_warn( _("mismatch between format (%d) and size (%" PRId64 ") in symlink inode %" PRIu64 "\n"), dino->di_format, - (__int64_t)be64_to_cpu(dino->di_size), lino); + (int64_t)be64_to_cpu(dino->di_size), lino); return 1; } @@ -1414,22 +1414,22 @@ _("inode %" PRIu64 " has bad inode type (IFMNT)\n"), lino); case XR_INO_CHRDEV: do_warn( _("size of character device inode %" PRIu64 " != 0 (%" PRId64 " bytes)\n"), lino, - (__int64_t)be64_to_cpu(dino->di_size)); + (int64_t)be64_to_cpu(dino->di_size)); break; case XR_INO_BLKDEV: do_warn( _("size of block device inode %" PRIu64 " != 0 (%" PRId64 " bytes)\n"), lino, - (__int64_t)be64_to_cpu(dino->di_size)); + (int64_t)be64_to_cpu(dino->di_size)); break; case XR_INO_SOCK: do_warn( _("size of socket inode %" PRIu64 " != 0 (%" PRId64 " bytes)\n"), lino, - (__int64_t)be64_to_cpu(dino->di_size)); + (int64_t)be64_to_cpu(dino->di_size)); break; case XR_INO_FIFO: do_warn( _("size of fifo inode %" PRIu64 " != 0 (%" PRId64 " bytes)\n"), lino, - (__int64_t)be64_to_cpu(dino->di_size)); + (int64_t)be64_to_cpu(dino->di_size)); break; default: do_warn(_("Internal error - process_misc_ino_types, " @@ -1703,12 +1703,12 @@ _("found inode %" PRIu64 " claiming to be a real-time file\n"), lino); break; case XR_INO_RTBITMAP: - if (size != (__int64_t)mp->m_sb.sb_rbmblocks * + if (size != (int64_t)mp->m_sb.sb_rbmblocks * mp->m_sb.sb_blocksize) { do_warn( _("realtime bitmap inode %" PRIu64 " has bad size %" PRId64 " (should be %" PRIu64 ")\n"), lino, size, - (__int64_t) mp->m_sb.sb_rbmblocks * + (int64_t) mp->m_sb.sb_rbmblocks * mp->m_sb.sb_blocksize); return 1; } @@ -1777,8 +1777,8 @@ static int process_inode_blocks_and_extents( xfs_dinode_t *dino, xfs_rfsblock_t nblocks, - __uint64_t nextents, - __uint64_t anextents, + uint64_t nextents, + uint64_t anextents, xfs_ino_t lino, int *dirty) { @@ -1868,7 +1868,7 @@ process_inode_data_fork( int type, int *dirty, xfs_rfsblock_t *totblocks, - __uint64_t *nextents, + uint64_t *nextents, blkmap_t **dblkmap, int check_dups) { @@ -1977,7 +1977,7 @@ process_inode_attr_fork( int type, int *dirty, xfs_rfsblock_t *atotblocks, - __uint64_t *anextents, + uint64_t *anextents, int check_dups, int extra_attr_check, int *retval) @@ -2188,8 +2188,8 @@ process_dinode_int(xfs_mount_t *mp, int di_mode; int type; int retval = 0; - __uint64_t nextents; - __uint64_t anextents; + uint64_t nextents; + uint64_t anextents; xfs_ino_t lino; const int is_free = 0; const int is_used = 1; @@ -2306,7 +2306,7 @@ _("inode identifier %llu mismatch on inode %" PRIu64 "\n"), if (!uncertain) do_warn( _("bad (negative) size %" PRId64 " on inode %" PRIu64 "\n"), - (__int64_t)be64_to_cpu(dino->di_size), + (int64_t)be64_to_cpu(dino->di_size), lino); if (verify_mode) return 1; diff --git a/repair/dinode.h b/repair/dinode.h index 5aebf5b..42a526b 100644 --- a/repair/dinode.h +++ b/repair/dinode.h @@ -46,8 +46,8 @@ process_bmbt_reclist(xfs_mount_t *mp, xfs_ino_t ino, xfs_rfsblock_t *tot, struct blkmap **blkmapp, - __uint64_t *first_key, - __uint64_t *last_key, + uint64_t *first_key, + uint64_t *last_key, int whichfork); int diff --git a/repair/dir2.c b/repair/dir2.c index 61912d1..3bff438 100644 --- a/repair/dir2.c +++ b/repair/dir2.c @@ -160,7 +160,7 @@ process_sf_dir2( int bad_sfnamelen; int i; int i8; - __int64_t ino_dir_size; + int64_t ino_dir_size; int ino_off; ino_tree_node_t *irec_p; int junkit; diff --git a/repair/globals.h b/repair/globals.h index efd1d03..1a55d78 100644 --- a/repair/globals.h +++ b/repair/globals.h @@ -140,7 +140,7 @@ EXTERN int inodes_per_cluster; EXTERN unsigned int glob_agcount; EXTERN int chunks_pblock; /* # of 64-ino chunks per allocation */ EXTERN int max_symlink_blocks; -EXTERN __int64_t fs_max_file_offset; +EXTERN int64_t fs_max_file_offset; /* realtime info */ @@ -160,16 +160,16 @@ EXTERN int full_ino_ex_data;/* /* superblock counters */ -EXTERN __uint64_t sb_icount; /* allocated (made) inodes */ -EXTERN __uint64_t sb_ifree; /* free inodes */ -EXTERN __uint64_t sb_fdblocks; /* free data blocks */ -EXTERN __uint64_t sb_frextents; /* free realtime extents */ +EXTERN uint64_t sb_icount; /* allocated (made) inodes */ +EXTERN uint64_t sb_ifree; /* free inodes */ +EXTERN uint64_t sb_fdblocks; /* free data blocks */ +EXTERN uint64_t sb_frextents; /* free realtime extents */ /* superblock geometry info */ EXTERN xfs_extlen_t sb_inoalignmt; -EXTERN __uint32_t sb_unit; -EXTERN __uint32_t sb_width; +EXTERN uint32_t sb_unit; +EXTERN uint32_t sb_width; struct aglock { pthread_mutex_t lock __attribute__((__aligned__(64))); @@ -177,7 +177,7 @@ struct aglock { EXTERN struct aglock *ag_locks; EXTERN int report_interval; -EXTERN __uint64_t *prog_rpt_done; +EXTERN uint64_t *prog_rpt_done; EXTERN int ag_stride; EXTERN int thread_count; diff --git a/repair/incore.c b/repair/incore.c index cb57316..a9191ae 100644 --- a/repair/incore.c +++ b/repair/incore.c @@ -179,7 +179,7 @@ get_bmap_ext( static uint64_t *rt_bmap; static size_t rt_bmap_size; -/* block records fit into __uint64_t's units */ +/* block records fit into uint64_t's units */ #define XR_BB_UNIT 64 /* number of bits/unit */ #define XR_BB 4 /* bits per block record */ #define XR_BB_NUM (XR_BB_UNIT/XR_BB) /* number of records per unit */ @@ -203,8 +203,8 @@ set_rtbmap( { *(rt_bmap + bno / XR_BB_NUM) = ((*(rt_bmap + bno / XR_BB_NUM) & - (~((__uint64_t) XR_BB_MASK << ((bno % XR_BB_NUM) * XR_BB)))) | - (((__uint64_t) state) << ((bno % XR_BB_NUM) * XR_BB))); + (~((uint64_t) XR_BB_MASK << ((bno % XR_BB_NUM) * XR_BB)))) | + (((uint64_t) state) << ((bno % XR_BB_NUM) * XR_BB))); } static void @@ -222,9 +222,9 @@ init_rt_bmap( return; rt_bmap_size = roundup(mp->m_sb.sb_rextents / (NBBY / XR_BB), - sizeof(__uint64_t)); + sizeof(uint64_t)); - rt_bmap = memalign(sizeof(__uint64_t), rt_bmap_size); + rt_bmap = memalign(sizeof(uint64_t), rt_bmap_size); if (!rt_bmap) { do_error( _("couldn't allocate realtime block map, size = %" PRIu64 "\n"), diff --git a/repair/incore.h b/repair/incore.h index c92475e..c0b593d 100644 --- a/repair/incore.h +++ b/repair/incore.h @@ -261,7 +261,7 @@ typedef xfs_ino_t parent_entry_t; struct nlink_ops; typedef struct parent_list { - __uint64_t pmask; + uint64_t pmask; parent_entry_t *pentries; #ifdef DEBUG short cnt; @@ -269,14 +269,14 @@ typedef struct parent_list { } parent_list_t; union ino_nlink { - __uint8_t *un8; - __uint16_t *un16; - __uint32_t *un32; + uint8_t *un8; + uint16_t *un16; + uint32_t *un32; }; typedef struct ino_ex_data { - __uint64_t ino_reached; /* bit == 1 if reached */ - __uint64_t ino_processed; /* reference checked bit mask */ + uint64_t ino_reached; /* bit == 1 if reached */ + uint64_t ino_processed; /* reference checked bit mask */ parent_list_t *parents; union ino_nlink counted_nlinks;/* counted nlinks in P6 */ } ino_ex_data_t; @@ -285,20 +285,20 @@ typedef struct ino_tree_node { avlnode_t avl_node; xfs_agino_t ino_startnum; /* starting inode # */ xfs_inofree_t ir_free; /* inode free bit mask */ - __uint64_t ir_sparse; /* sparse inode bitmask */ - __uint64_t ino_confirmed; /* confirmed bitmask */ - __uint64_t ino_isa_dir; /* bit == 1 if a directory */ - __uint8_t nlink_size; + uint64_t ir_sparse; /* sparse inode bitmask */ + uint64_t ino_confirmed; /* confirmed bitmask */ + uint64_t ino_isa_dir; /* bit == 1 if a directory */ + uint8_t nlink_size; union ino_nlink disk_nlinks; /* on-disk nlinks, set in P3 */ union { ino_ex_data_t *ex_data; /* phases 6,7 */ parent_list_t *plist; /* phases 2-5 */ } ino_un; - __uint8_t *ftypes; /* phases 3,6 */ + uint8_t *ftypes; /* phases 3,6 */ } ino_tree_node_t; -#define INOS_PER_IREC (sizeof(__uint64_t) * NBBY) -#define IREC_MASK(i) ((__uint64_t)1 << (i)) +#define INOS_PER_IREC (sizeof(uint64_t) * NBBY) +#define IREC_MASK(i) ((uint64_t)1 << (i)) void add_ino_ex_data(xfs_mount_t *mp); @@ -509,10 +509,10 @@ static inline bool is_inode_sparse(struct ino_tree_node *irec, int offset) */ void add_inode_ref(struct ino_tree_node *irec, int offset); void drop_inode_ref(struct ino_tree_node *irec, int offset); -__uint32_t num_inode_references(struct ino_tree_node *irec, int offset); +uint32_t num_inode_references(struct ino_tree_node *irec, int offset); -void set_inode_disk_nlinks(struct ino_tree_node *irec, int offset, __uint32_t nlinks); -__uint32_t get_inode_disk_nlinks(struct ino_tree_node *irec, int offset); +void set_inode_disk_nlinks(struct ino_tree_node *irec, int offset, uint32_t nlinks); +uint32_t get_inode_disk_nlinks(struct ino_tree_node *irec, int offset); static inline int is_inode_reached(struct ino_tree_node *irec, int offset) { @@ -533,13 +533,13 @@ static inline void add_inode_reached(struct ino_tree_node *irec, int offset) static inline void set_inode_ftype(struct ino_tree_node *irec, int ino_offset, - __uint8_t ftype) + uint8_t ftype) { if (irec->ftypes) irec->ftypes[ino_offset] = ftype; } -static inline __uint8_t +static inline uint8_t get_inode_ftype( struct ino_tree_node *irec, int ino_offset) @@ -572,11 +572,11 @@ typedef struct bm_level_state { xfs_fsblock_t fsbno; xfs_fsblock_t left_fsbno; xfs_fsblock_t right_fsbno; - __uint64_t first_key; - __uint64_t last_key; + uint64_t first_key; + uint64_t last_key; /* int level; - __uint64_t prev_last_key; + uint64_t prev_last_key; xfs_buf_t *bp; xfs_bmbt_block_t *block; */ diff --git a/repair/incore_ext.c b/repair/incore_ext.c index 7e6786c..0a8138a 100644 --- a/repair/incore_ext.c +++ b/repair/incore_ext.c @@ -721,13 +721,13 @@ search_rt_dup_extent(xfs_mount_t *mp, xfs_rtblock_t bno) return(ret); } -static __uint64_t +static uint64_t avl64_rt_ext_start(avl64node_t *node) { return(((rt_extent_tree_node_t *) node)->rt_startblock); } -static __uint64_t +static uint64_t avl64_ext_end(avl64node_t *node) { return(((rt_extent_tree_node_t *) node)->rt_startblock + @@ -834,7 +834,7 @@ count_extents(xfs_agnumber_t agno, avltree_desc_t *tree, int whichtree) int count_bno_extents_blocks(xfs_agnumber_t agno, uint *numblocks) { - __uint64_t nblocks; + uint64_t nblocks; extent_tree_node_t *node; int i = 0; diff --git a/repair/incore_ino.c b/repair/incore_ino.c index 1898257..6ffe08e 100644 --- a/repair/incore_ino.c +++ b/repair/incore_ino.c @@ -38,7 +38,7 @@ static avltree_desc_t **inode_uncertain_tree_ptrs; /* memory optimised nlink counting for all inodes */ static void * -alloc_nlink_array(__uint8_t nlink_size) +alloc_nlink_array(uint8_t nlink_size) { void *ptr; @@ -51,10 +51,10 @@ alloc_nlink_array(__uint8_t nlink_size) static void nlink_grow_8_to_16(ino_tree_node_t *irec) { - __uint16_t *new_nlinks; + uint16_t *new_nlinks; int i; - irec->nlink_size = sizeof(__uint16_t); + irec->nlink_size = sizeof(uint16_t); new_nlinks = alloc_nlink_array(irec->nlink_size); for (i = 0; i < XFS_INODES_PER_CHUNK; i++) @@ -76,10 +76,10 @@ nlink_grow_8_to_16(ino_tree_node_t *irec) static void nlink_grow_16_to_32(ino_tree_node_t *irec) { - __uint32_t *new_nlinks; + uint32_t *new_nlinks; int i; - irec->nlink_size = sizeof(__uint32_t); + irec->nlink_size = sizeof(uint32_t); new_nlinks = alloc_nlink_array(irec->nlink_size); for (i = 0; i < XFS_INODES_PER_CHUNK; i++) @@ -104,21 +104,21 @@ void add_inode_ref(struct ino_tree_node *irec, int ino_offset) ASSERT(irec->ino_un.ex_data != NULL); switch (irec->nlink_size) { - case sizeof(__uint8_t): + case sizeof(uint8_t): if (irec->ino_un.ex_data->counted_nlinks.un8[ino_offset] < 0xff) { irec->ino_un.ex_data->counted_nlinks.un8[ino_offset]++; break; } nlink_grow_8_to_16(irec); /*FALLTHRU*/ - case sizeof(__uint16_t): + case sizeof(uint16_t): if (irec->ino_un.ex_data->counted_nlinks.un16[ino_offset] < 0xffff) { irec->ino_un.ex_data->counted_nlinks.un16[ino_offset]++; break; } nlink_grow_16_to_32(irec); /*FALLTHRU*/ - case sizeof(__uint32_t): + case sizeof(uint32_t): irec->ino_un.ex_data->counted_nlinks.un32[ino_offset]++; break; default: @@ -128,20 +128,20 @@ void add_inode_ref(struct ino_tree_node *irec, int ino_offset) void drop_inode_ref(struct ino_tree_node *irec, int ino_offset) { - __uint32_t refs = 0; + uint32_t refs = 0; ASSERT(irec->ino_un.ex_data != NULL); switch (irec->nlink_size) { - case sizeof(__uint8_t): + case sizeof(uint8_t): ASSERT(irec->ino_un.ex_data->counted_nlinks.un8[ino_offset] > 0); refs = --irec->ino_un.ex_data->counted_nlinks.un8[ino_offset]; break; - case sizeof(__uint16_t): + case sizeof(uint16_t): ASSERT(irec->ino_un.ex_data->counted_nlinks.un16[ino_offset] > 0); refs = --irec->ino_un.ex_data->counted_nlinks.un16[ino_offset]; break; - case sizeof(__uint32_t): + case sizeof(uint32_t): ASSERT(irec->ino_un.ex_data->counted_nlinks.un32[ino_offset] > 0); refs = --irec->ino_un.ex_data->counted_nlinks.un32[ino_offset]; break; @@ -153,16 +153,16 @@ void drop_inode_ref(struct ino_tree_node *irec, int ino_offset) irec->ino_un.ex_data->ino_reached &= ~IREC_MASK(ino_offset); } -__uint32_t num_inode_references(struct ino_tree_node *irec, int ino_offset) +uint32_t num_inode_references(struct ino_tree_node *irec, int ino_offset) { ASSERT(irec->ino_un.ex_data != NULL); switch (irec->nlink_size) { - case sizeof(__uint8_t): + case sizeof(uint8_t): return irec->ino_un.ex_data->counted_nlinks.un8[ino_offset]; - case sizeof(__uint16_t): + case sizeof(uint16_t): return irec->ino_un.ex_data->counted_nlinks.un16[ino_offset]; - case sizeof(__uint32_t): + case sizeof(uint32_t): return irec->ino_un.ex_data->counted_nlinks.un32[ino_offset]; default: ASSERT(0); @@ -171,24 +171,24 @@ __uint32_t num_inode_references(struct ino_tree_node *irec, int ino_offset) } void set_inode_disk_nlinks(struct ino_tree_node *irec, int ino_offset, - __uint32_t nlinks) + uint32_t nlinks) { switch (irec->nlink_size) { - case sizeof(__uint8_t): + case sizeof(uint8_t): if (nlinks < 0xff) { irec->disk_nlinks.un8[ino_offset] = nlinks; break; } nlink_grow_8_to_16(irec); /*FALLTHRU*/ - case sizeof(__uint16_t): + case sizeof(uint16_t): if (nlinks < 0xffff) { irec->disk_nlinks.un16[ino_offset] = nlinks; break; } nlink_grow_16_to_32(irec); /*FALLTHRU*/ - case sizeof(__uint32_t): + case sizeof(uint32_t): irec->disk_nlinks.un32[ino_offset] = nlinks; break; default: @@ -196,14 +196,14 @@ void set_inode_disk_nlinks(struct ino_tree_node *irec, int ino_offset, } } -__uint32_t get_inode_disk_nlinks(struct ino_tree_node *irec, int ino_offset) +uint32_t get_inode_disk_nlinks(struct ino_tree_node *irec, int ino_offset) { switch (irec->nlink_size) { - case sizeof(__uint8_t): + case sizeof(uint8_t): return irec->disk_nlinks.un8[ino_offset]; - case sizeof(__uint16_t): + case sizeof(uint16_t): return irec->disk_nlinks.un16[ino_offset]; - case sizeof(__uint32_t): + case sizeof(uint32_t): return irec->disk_nlinks.un32[ino_offset]; default: ASSERT(0); @@ -211,11 +211,11 @@ __uint32_t get_inode_disk_nlinks(struct ino_tree_node *irec, int ino_offset) return 0; } -static __uint8_t * +static uint8_t * alloc_ftypes_array( struct xfs_mount *mp) { - __uint8_t *ptr; + uint8_t *ptr; if (!xfs_sb_version_hasftype(&mp->m_sb)) return NULL; @@ -260,23 +260,23 @@ alloc_ino_node( irec->ir_free = (xfs_inofree_t) - 1; irec->ir_sparse = 0; irec->ino_un.ex_data = NULL; - irec->nlink_size = sizeof(__uint8_t); + irec->nlink_size = sizeof(uint8_t); irec->disk_nlinks.un8 = alloc_nlink_array(irec->nlink_size); irec->ftypes = alloc_ftypes_array(mp); return irec; } static void -free_nlink_array(union ino_nlink nlinks, __uint8_t nlink_size) +free_nlink_array(union ino_nlink nlinks, uint8_t nlink_size) { switch (nlink_size) { - case sizeof(__uint8_t): + case sizeof(uint8_t): free(nlinks.un8); break; - case sizeof(__uint16_t): + case sizeof(uint16_t): free(nlinks.un16); break; - case sizeof(__uint32_t): + case sizeof(uint32_t): free(nlinks.un32); break; default: @@ -607,7 +607,7 @@ set_inode_parent( int i; int cnt; int target; - __uint64_t bitmask; + uint64_t bitmask; parent_entry_t *tmp; if (full_ino_ex_data) @@ -697,7 +697,7 @@ set_inode_parent( xfs_ino_t get_inode_parent(ino_tree_node_t *irec, int offset) { - __uint64_t bitmask; + uint64_t bitmask; parent_list_t *ptbl; int i; int target; @@ -738,15 +738,15 @@ alloc_ex_data(ino_tree_node_t *irec) irec->ino_un.ex_data->parents = ptbl; switch (irec->nlink_size) { - case sizeof(__uint8_t): + case sizeof(uint8_t): irec->ino_un.ex_data->counted_nlinks.un8 = alloc_nlink_array(irec->nlink_size); break; - case sizeof(__uint16_t): + case sizeof(uint16_t): irec->ino_un.ex_data->counted_nlinks.un16 = alloc_nlink_array(irec->nlink_size); break; - case sizeof(__uint32_t): + case sizeof(uint32_t): irec->ino_un.ex_data->counted_nlinks.un32 = alloc_nlink_array(irec->nlink_size); break; diff --git a/repair/phase2.c b/repair/phase2.c index e21ffa6..a2dfa0b 100644 --- a/repair/phase2.c +++ b/repair/phase2.c @@ -173,7 +173,7 @@ phase2( bad_ino_btree = 0; - set_progress_msg(PROG_FMT_SCAN_AG, (__uint64_t) glob_agcount); + set_progress_msg(PROG_FMT_SCAN_AG, (uint64_t) glob_agcount); scan_ags(mp, scan_threads); diff --git a/repair/phase3.c b/repair/phase3.c index 76c9440..398c60c 100644 --- a/repair/phase3.c +++ b/repair/phase3.c @@ -98,7 +98,7 @@ phase3(xfs_mount_t *mp) else do_log(_(" - scan (but don't clear) agi unlinked lists...\n")); - set_progress_msg(PROG_FMT_AGI_UNLINKED, (__uint64_t) glob_agcount); + set_progress_msg(PROG_FMT_AGI_UNLINKED, (uint64_t) glob_agcount); /* first clear the agi unlinked AGI list */ if (!no_modify) { @@ -118,7 +118,7 @@ phase3(xfs_mount_t *mp) do_log(_( " - process known inodes and perform inode discovery...\n")); - set_progress_msg(PROG_FMT_PROCESS_INO, (__uint64_t) mp->m_sb.sb_icount); + set_progress_msg(PROG_FMT_PROCESS_INO, (uint64_t) mp->m_sb.sb_icount); process_ags(mp); @@ -128,7 +128,7 @@ phase3(xfs_mount_t *mp) * process newly discovered inode chunks */ do_log(_(" - process newly discovered inodes...\n")); - set_progress_msg(PROG_FMT_NEW_INODES, (__uint64_t) glob_agcount); + set_progress_msg(PROG_FMT_NEW_INODES, (uint64_t) glob_agcount); do { /* * have to loop until no ag has any uncertain diff --git a/repair/phase4.c b/repair/phase4.c index 1a7d7b5..90386c2 100644 --- a/repair/phase4.c +++ b/repair/phase4.c @@ -175,7 +175,7 @@ phase4(xfs_mount_t *mp) do_log(_("Phase 4 - check for duplicate blocks...\n")); do_log(_(" - setting up duplicate extent list...\n")); - set_progress_msg(PROG_FMT_DUP_EXTENT, (__uint64_t) glob_agcount); + set_progress_msg(PROG_FMT_DUP_EXTENT, (uint64_t) glob_agcount); irec = find_inode_rec(mp, XFS_INO_TO_AGNO(mp, mp->m_sb.sb_rootino), XFS_INO_TO_AGINO(mp, mp->m_sb.sb_rootino)); @@ -289,7 +289,7 @@ phase4(xfs_mount_t *mp) reset_bmaps(mp); do_log(_(" - check for inodes claiming duplicate blocks...\n")); - set_progress_msg(PROG_FMT_DUP_BLOCKS, (__uint64_t) mp->m_sb.sb_icount); + set_progress_msg(PROG_FMT_DUP_BLOCKS, (uint64_t) mp->m_sb.sb_icount); /* * ok, now process the inodes -- signal 2-pass check per inode. diff --git a/repair/phase5.c b/repair/phase5.c index 5d48848..c280292 100644 --- a/repair/phase5.c +++ b/repair/phase5.c @@ -83,9 +83,9 @@ struct agi_stat { xfs_agino_t freecount; }; -static __uint64_t *sb_icount_ag; /* allocated inodes per ag */ -static __uint64_t *sb_ifree_ag; /* free inodes per ag */ -static __uint64_t *sb_fdblocks_ag; /* free data blocks per ag */ +static uint64_t *sb_icount_ag; /* allocated inodes per ag */ +static uint64_t *sb_ifree_ag; /* free inodes per ag */ +static uint64_t *sb_fdblocks_ag; /* free data blocks per ag */ static int mk_incore_fstree(xfs_mount_t *mp, xfs_agnumber_t agno) @@ -618,14 +618,14 @@ calculate_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, static void prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs, xfs_agblock_t startblock, - xfs_extlen_t blockcount, int level, __uint32_t magic) + xfs_extlen_t blockcount, int level, uint32_t magic) { struct xfs_btree_block *bt_hdr; xfs_alloc_key_t *bt_key; xfs_alloc_ptr_t *bt_ptr; xfs_agblock_t agbno; bt_stat_level_t *lptr; - __uint32_t crc_magic; + uint32_t crc_magic; if (magic == XFS_ABTB_MAGIC) crc_magic = XFS_ABTB_CRC_MAGIC; @@ -720,7 +720,7 @@ prop_freespace_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, */ static xfs_extlen_t build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno, - bt_status_t *btree_curs, __uint32_t magic) + bt_status_t *btree_curs, uint32_t magic) { xfs_agnumber_t i; xfs_agblock_t j; @@ -731,7 +731,7 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno, extent_tree_node_t *ext_ptr; bt_stat_level_t *lptr; xfs_extlen_t freeblks; - __uint32_t crc_magic; + uint32_t crc_magic; #ifdef XR_BLD_FREE_TRACE fprintf(stderr, "in build_freespace_tree, agno = %d\n", agno); @@ -895,10 +895,10 @@ build_freespace_tree(xfs_mount_t *mp, xfs_agnumber_t agno, */ static void init_ino_cursor(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs, - __uint64_t *num_inos, __uint64_t *num_free_inos, int finobt) + uint64_t *num_inos, uint64_t *num_free_inos, int finobt) { - __uint64_t ninos; - __uint64_t nfinos; + uint64_t ninos; + uint64_t nfinos; int rec_nfinos; int rec_ninos; ino_tree_node_t *ino_rec; @@ -1144,7 +1144,7 @@ build_agi(xfs_mount_t *mp, xfs_agnumber_t agno, bt_status_t *btree_curs, */ static void build_ino_tree(xfs_mount_t *mp, xfs_agnumber_t agno, - bt_status_t *btree_curs, __uint32_t magic, + bt_status_t *btree_curs, uint32_t magic, struct agi_stat *agi_stat, int finobt) { xfs_agnumber_t i; @@ -1573,10 +1573,10 @@ phase5_func( xfs_mount_t *mp, xfs_agnumber_t agno) { - __uint64_t num_inos; - __uint64_t num_free_inos; - __uint64_t finobt_num_inos; - __uint64_t finobt_num_free_inos; + uint64_t num_inos; + uint64_t num_free_inos; + uint64_t finobt_num_inos; + uint64_t finobt_num_free_inos; bt_status_t bno_btree_curs; bt_status_t bcnt_btree_curs; bt_status_t ino_btree_curs; @@ -1588,7 +1588,7 @@ phase5_func( xfs_extlen_t freeblks2; #endif xfs_agblock_t num_extents; - __uint32_t magic; + uint32_t magic; struct agi_stat agi_stat = {0,}; if (verbose) @@ -1772,7 +1772,7 @@ phase5(xfs_mount_t *mp) xfs_agnumber_t agno; do_log(_("Phase 5 - rebuild AG headers and trees...\n")); - set_progress_msg(PROG_FMT_REBUILD_AG, (__uint64_t )glob_agcount); + set_progress_msg(PROG_FMT_REBUILD_AG, (uint64_t )glob_agcount); #ifdef XR_BLD_FREE_TRACE fprintf(stderr, "inobt level 1, maxrec = %d, minrec = %d\n", @@ -1798,15 +1798,15 @@ phase5(xfs_mount_t *mp) keep_fsinos(mp); /* allocate per ag counters */ - sb_icount_ag = calloc(mp->m_sb.sb_agcount, sizeof(__uint64_t)); + sb_icount_ag = calloc(mp->m_sb.sb_agcount, sizeof(uint64_t)); if (sb_icount_ag == NULL) do_error(_("cannot alloc sb_icount_ag buffers\n")); - sb_ifree_ag = calloc(mp->m_sb.sb_agcount, sizeof(__uint64_t)); + sb_ifree_ag = calloc(mp->m_sb.sb_agcount, sizeof(uint64_t)); if (sb_ifree_ag == NULL) do_error(_("cannot alloc sb_ifree_ag buffers\n")); - sb_fdblocks_ag = calloc(mp->m_sb.sb_agcount, sizeof(__uint64_t)); + sb_fdblocks_ag = calloc(mp->m_sb.sb_agcount, sizeof(uint64_t)); if (sb_fdblocks_ag == NULL) do_error(_("cannot alloc sb_fdblocks_ag buffers\n")); diff --git a/repair/phase6.c b/repair/phase6.c index 7680deb..437142a 100644 --- a/repair/phase6.c +++ b/repair/phase6.c @@ -82,7 +82,7 @@ typedef struct dir_hash_ent { struct dir_hash_ent *nextbyhash; /* next in name bucket */ struct dir_hash_ent *nextbyorder; /* next in order added */ xfs_dahash_t hashval; /* hash value of name */ - __uint32_t address; /* offset of data entry */ + uint32_t address; /* offset of data entry */ xfs_ino_t inum; /* inode num of entry */ short junkit; /* name starts with / */ short seen; /* have seen leaf entry */ @@ -170,11 +170,11 @@ static int dir_hash_add( xfs_mount_t *mp, dir_hash_tab_t *hashtab, - __uint32_t addr, + uint32_t addr, xfs_ino_t inum, int namelen, unsigned char *name, - __uint8_t ftype) + uint8_t ftype) { xfs_dahash_t hash = 0; int byaddr; @@ -357,7 +357,7 @@ static void dir_hash_update_ftype( dir_hash_tab_t *hashtab, xfs_dir2_dataptr_t addr, - __uint8_t ftype) + uint8_t ftype) { int i; dir_hash_ent_t *p; @@ -887,7 +887,7 @@ mk_root_dir(xfs_mount_t *mp) memset(&ip->i_d, 0, xfs_icdinode_size(vers)); ip->i_d.di_magic = XFS_DINODE_MAGIC; - ip->i_d.di_mode = (__uint16_t) mode|S_IFDIR; + ip->i_d.di_mode = (uint16_t) mode|S_IFDIR; ip->i_d.di_version = vers; ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS; ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS; @@ -1839,8 +1839,8 @@ longform_dir2_entry_check_data( /* validate ftype field if supported */ if (xfs_sb_version_hasftype(&mp->m_sb)) { - __uint8_t dir_ftype; - __uint8_t ino_ftype; + uint8_t dir_ftype; + uint8_t ino_ftype; dir_ftype = M_DIROPS(mp)->data_get_ftype(dep); ino_ftype = get_inode_ftype(irec, ino_offset); @@ -2750,8 +2750,8 @@ _("entry \"%s\" (ino %" PRIu64 ") in dir %" PRIu64 " is a duplicate name"), /* validate ftype field if supported */ if (xfs_sb_version_hasftype(&mp->m_sb)) { - __uint8_t dir_ftype; - __uint8_t ino_ftype; + uint8_t dir_ftype; + uint8_t ino_ftype; dir_ftype = M_DIROPS(mp)->sf_get_ftype(sfep); ino_ftype = get_inode_ftype(irec, ino_offset); diff --git a/repair/phase7.c b/repair/phase7.c index b1e3a55..d8e788e 100644 --- a/repair/phase7.c +++ b/repair/phase7.c @@ -32,7 +32,7 @@ static void set_nlinks( xfs_icdinode_t *dinoc, xfs_ino_t ino, - __uint32_t nrefs, + uint32_t nrefs, int *dirty) { if (dinoc->di_nlink == nrefs) @@ -56,7 +56,7 @@ static void update_inode_nlinks( xfs_mount_t *mp, xfs_ino_t ino, - __uint32_t nlinks) + uint32_t nlinks) { xfs_trans_t *tp; xfs_inode_t *ip; @@ -114,7 +114,7 @@ phase7(xfs_mount_t *mp) ino_tree_node_t *irec; int i; int j; - __uint32_t nrefs; + uint32_t nrefs; if (!no_modify) do_log(_("Phase 7 - verify and correct link counts...\n")); diff --git a/repair/progress.c b/repair/progress.c index 418b803..eae14e7 100644 --- a/repair/progress.c +++ b/repair/progress.c @@ -85,8 +85,8 @@ pthread_t report_thread; typedef struct msg_block_s { pthread_mutex_t mutex; progress_rpt_t *format; - __uint64_t *done; - __uint64_t *total; + uint64_t *done; + uint64_t *total; int count; int interval; } msg_block_t; @@ -96,14 +96,14 @@ typedef struct phase_times_s { time_t start; time_t end; time_t duration; - __uint64_t item_counts[4]; + uint64_t item_counts[4]; } phase_times_t; static phase_times_t phase_times[8]; static void *progress_rpt_thread(void *); static int current_phase; static int running; -static __uint64_t prog_rpt_total; +static uint64_t prog_rpt_total; void init_progress_rpt (void) @@ -113,11 +113,11 @@ init_progress_rpt (void) * allocate the done vector */ - if ((prog_rpt_done = (__uint64_t *) - malloc(sizeof(__uint64_t)*glob_agcount)) == NULL ) { + if ((prog_rpt_done = (uint64_t *) + malloc(sizeof(uint64_t)*glob_agcount)) == NULL ) { do_error(_("cannot malloc pointer to done vector\n")); } - bzero(prog_rpt_done, sizeof(__uint64_t)*glob_agcount); + bzero(prog_rpt_done, sizeof(uint64_t)*glob_agcount); /* * Setup comm block, start the thread @@ -165,10 +165,10 @@ progress_rpt_thread (void *p) timer_t timerid; struct itimerspec timespec; char *msgbuf; - __uint64_t *donep; - __uint64_t sum; + uint64_t *donep; + uint64_t sum; msg_block_t *msgp = (msg_block_t *)p; - __uint64_t percent; + uint64_t percent; /* It's possible to get here very early w/ no progress msg set */ if (!msgp->format) @@ -286,7 +286,7 @@ progress_rpt_thread (void *p) } int -set_progress_msg (int report, __uint64_t total) +set_progress_msg (int report, uint64_t total) { if (!ag_stride) @@ -300,7 +300,7 @@ set_progress_msg (int report, __uint64_t total) /* reset all the accumulative totals */ if (prog_rpt_done) - bzero(prog_rpt_done, sizeof(__uint64_t)*glob_agcount); + bzero(prog_rpt_done, sizeof(uint64_t)*glob_agcount); if (pthread_mutex_unlock(&global_msgs.mutex)) do_error(_("set_progress_msg: cannot unlock progress mutex\n")); @@ -308,14 +308,14 @@ set_progress_msg (int report, __uint64_t total) return (0); } -__uint64_t +uint64_t print_final_rpt(void) { int i; struct tm *tmp; time_t now; - __uint64_t *donep; - __uint64_t sum; + uint64_t *donep; + uint64_t sum; msg_block_t *msgp = &global_msgs; char msgbuf[DURATION_BUF_SIZE]; diff --git a/repair/progress.h b/repair/progress.h index 33db834..5152648 100644 --- a/repair/progress.h +++ b/repair/progress.h @@ -32,8 +32,8 @@ extern void init_progress_rpt(void); extern void stop_progress_rpt(void); extern void summary_report(void); -extern int set_progress_msg(int report, __uint64_t total); -extern __uint64_t print_final_rpt(void); +extern int set_progress_msg(int report, uint64_t total); +extern uint64_t print_final_rpt(void); extern char *timestamp(int end, int phase, char *buf); extern char *duration(int val, char *buf); extern int do_parallel; diff --git a/repair/sb.c b/repair/sb.c index 4eef14a..ea89ca9 100644 --- a/repair/sb.c +++ b/repair/sb.c @@ -41,7 +41,7 @@ copy_sb(xfs_sb_t *source, xfs_sb_t *dest) xfs_ino_t uquotino; xfs_ino_t gquotino; xfs_ino_t pquotino; - __uint16_t versionnum; + uint16_t versionnum; rootino = dest->sb_rootino; rbmino = dest->sb_rbmino; @@ -264,7 +264,7 @@ sb_validate_ino_align(struct xfs_sb *sb) int verify_sb(char *sb_buf, xfs_sb_t *sb, int is_primary_sb) { - __uint32_t bsize; + uint32_t bsize; int i; /* check magic number and version number */ @@ -329,13 +329,13 @@ verify_sb(char *sb_buf, xfs_sb_t *sb, int is_primary_sb) if (sb->sb_dblocks == 0 || sb->sb_dblocks > - ((__uint64_t)sb->sb_agcount * sb->sb_agblocks) || + ((uint64_t)sb->sb_agcount * sb->sb_agblocks) || sb->sb_dblocks < - ((__uint64_t)(sb->sb_agcount - 1) * sb->sb_agblocks + ((uint64_t)(sb->sb_agcount - 1) * sb->sb_agblocks + XFS_MIN_AG_BLOCKS)) return(XR_BAD_FS_SIZE_DATA); - if (sb->sb_agblklog != (__uint8_t)libxfs_log2_roundup(sb->sb_agblocks)) + if (sb->sb_agblklog != (uint8_t)libxfs_log2_roundup(sb->sb_agblocks)) return(XR_BAD_FS_SIZE_DATA); if (sb->sb_inodesize < XFS_DINODE_MIN_SIZE || diff --git a/repair/scan.c b/repair/scan.c index 964ff06..5a79325 100644 --- a/repair/scan.c +++ b/repair/scan.c @@ -40,12 +40,12 @@ struct aghdr_cnts { xfs_agnumber_t agno; xfs_extlen_t agffreeblks; xfs_extlen_t agflongest; - __uint64_t agfbtreeblks; - __uint32_t agicount; - __uint32_t agifreecount; - __uint64_t fdblocks; - __uint64_t ifreecount; - __uint32_t fibtfreecount; + uint64_t agfbtreeblks; + uint32_t agicount; + uint32_t agifreecount; + uint64_t fdblocks; + uint64_t ifreecount; + uint32_t fibtfreecount; }; void @@ -67,10 +67,10 @@ scan_sbtree( xfs_agnumber_t agno, int suspect, int isroot, - __uint32_t magic, + uint32_t magic, void *priv), int isroot, - __uint32_t magic, + uint32_t magic, void *priv, const struct xfs_buf_ops *ops) { @@ -107,23 +107,23 @@ scan_lbtree( xfs_fsblock_t bno, xfs_ino_t ino, xfs_rfsblock_t *tot, - __uint64_t *nex, + uint64_t *nex, blkmap_t **blkmapp, bmap_cursor_t *bm_cursor, int isroot, int check_dups, int *dirty, - __uint64_t magic), + uint64_t magic), int type, int whichfork, xfs_ino_t ino, xfs_rfsblock_t *tot, - __uint64_t *nex, + uint64_t *nex, blkmap_t **blkmapp, bmap_cursor_t *bm_cursor, int isroot, int check_dups, - __uint64_t magic, + uint64_t magic, const struct xfs_buf_ops *ops) { xfs_buf_t *bp; @@ -176,13 +176,13 @@ scan_bmapbt( xfs_fsblock_t bno, xfs_ino_t ino, xfs_rfsblock_t *tot, - __uint64_t *nex, + uint64_t *nex, blkmap_t **blkmapp, bmap_cursor_t *bm_cursor, int isroot, int check_dups, int *dirty, - __uint64_t magic) + uint64_t magic) { int i; int err; @@ -528,7 +528,7 @@ scan_allocbt( xfs_agnumber_t agno, int suspect, int isroot, - __uint32_t magic, + uint32_t magic, void *priv) { struct aghdr_cnts *agcnts = priv; @@ -1243,7 +1243,7 @@ scan_inobt( xfs_agnumber_t agno, int suspect, int isroot, - __uint32_t magic, + uint32_t magic, void *priv) { struct aghdr_cnts *agcnts = priv; @@ -1479,7 +1479,7 @@ validate_agf( struct aghdr_cnts *agcnts) { xfs_agblock_t bno; - __uint32_t magic; + uint32_t magic; bno = be32_to_cpu(agf->agf_roots[XFS_BTNUM_BNO]); if (bno != 0 && verify_agbno(mp, agno, bno)) { @@ -1530,7 +1530,7 @@ validate_agi( { xfs_agblock_t bno; int i; - __uint32_t magic; + uint32_t magic; bno = be32_to_cpu(agi->agi_root); if (bno != 0 && verify_agbno(mp, agno, bno)) { @@ -1756,9 +1756,9 @@ scan_ags( int scan_threads) { struct aghdr_cnts *agcnts; - __uint64_t fdblocks = 0; - __uint64_t icount = 0; - __uint64_t ifreecount = 0; + uint64_t fdblocks = 0; + uint64_t icount = 0; + uint64_t ifreecount = 0; xfs_agnumber_t i; work_queue_t wq; diff --git a/repair/scan.h b/repair/scan.h index ea8c0bf..9bbe1e6 100644 --- a/repair/scan.h +++ b/repair/scan.h @@ -30,23 +30,23 @@ int scan_lbtree( xfs_fsblock_t bno, xfs_ino_t ino, xfs_rfsblock_t *tot, - __uint64_t *nex, + uint64_t *nex, struct blkmap **blkmapp, bmap_cursor_t *bm_cursor, int isroot, int check_dups, int *dirty, - __uint64_t magic), + uint64_t magic), int type, int whichfork, xfs_ino_t ino, xfs_rfsblock_t *tot, - __uint64_t *nex, + uint64_t *nex, struct blkmap **blkmapp, bmap_cursor_t *bm_cursor, int isroot, int check_dups, - __uint64_t magic, + uint64_t magic, const struct xfs_buf_ops *ops); int scan_bmapbt( @@ -57,13 +57,13 @@ int scan_bmapbt( xfs_fsblock_t bno, xfs_ino_t ino, xfs_rfsblock_t *tot, - __uint64_t *nex, + uint64_t *nex, struct blkmap **blkmapp, bmap_cursor_t *bm_cursor, int isroot, int check_dups, int *dirty, - __uint64_t magic); + uint64_t magic); void scan_ags( -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 14:02:48 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id EABF929E06 for ; Tue, 12 Jan 2016 14:02:48 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay1.corp.sgi.com (Postfix) with ESMTP id E59C98F8049 for ; Tue, 12 Jan 2016 12:02:45 -0800 (PST) X-ASG-Debug-ID: 1452628963-04cbb05f7f13d0b0001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id 4nGarKBgFzUdo6Ze (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 12:02:44 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id 28E6420A88 for ; Tue, 12 Jan 2016 21:02:43 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg2rM0C5jz5vMH for ; Tue, 12 Jan 2016 21:02:43 +0100 (CET) Date: Tue, 12 Jan 2016 21:00:08 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 5/6] include/linux.h: Include for fprintf and stderr Message-ID: <20160112200008.GF568@nyan> X-ASG-Orig-Subj: [PATCH 5/6] include/linux.h: Include for fprintf and stderr References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452628963 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Signed-off-by: Felix Janda --- include/linux.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux.h b/include/linux.h index 4872a1e..a4acedc 100644 --- a/include/linux.h +++ b/include/linux.h @@ -29,6 +29,7 @@ #include #include #include +#include #include #include -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 14:03:00 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 9460A29DF9 for ; Tue, 12 Jan 2016 14:03:00 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay3.corp.sgi.com (Postfix) with ESMTP id 280FBAC001 for ; Tue, 12 Jan 2016 12:03:00 -0800 (PST) X-ASG-Debug-ID: 1452628978-04bdf06f6baebd0001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id KFGzB8vIobwSYUJg (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 12:02:59 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id B281B20B41 for ; Tue, 12 Jan 2016 21:02:57 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg2rd466fz5vN3 for ; Tue, 12 Jan 2016 21:02:57 +0100 (CET) Date: Tue, 12 Jan 2016 21:00:22 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 6/6] Add configure check for members of dirent for use in io/readdir.c Message-ID: <20160112200022.GG568@nyan> X-ASG-Orig-Subj: [PATCH 6/6] Add configure check for members of dirent for use in io/readdir.c References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452628978 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Signed-off-by: Felix Janda --- configure.ac | 7 +++++++ include/builddefs.in | 4 ++++ io/Makefile | 10 ++++++++++ 3 files changed, 21 insertions(+) diff --git a/configure.ac b/configure.ac index feee50a..3d6443a 100644 --- a/configure.ac +++ b/configure.ac @@ -135,5 +135,12 @@ AC_CHECK_SIZEOF([char *]) AC_TYPE_UMODE_T AC_MANUAL_FORMAT +AC_CHECK_MEMBERS([struct dirent.d_reclen],[],[],[[#include ]]) +AC_SUBST(ac_cv_member_struct_dirent_d_reclen) +AC_CHECK_MEMBERS([struct dirent.d_off],[],[],[[#include ]]) +AC_SUBST(ac_cv_member_struct_dirent_d_off) +AC_CHECK_MEMBERS([struct dirent.d_type],[],[],[[#include ]]) +AC_SUBST(ac_cv_member_struct_dirent_d_type) + AC_CONFIG_FILES([include/builddefs]) AC_OUTPUT diff --git a/include/builddefs.in b/include/builddefs.in index b5ce336..67d963c 100644 --- a/include/builddefs.in +++ b/include/builddefs.in @@ -109,6 +109,10 @@ HAVE_FLS = @have_fls@ HAVE_FSETXATTR = @have_fsetxattr@ HAVE_MREMAP = @have_mremap@ +HAVE_STRUCT_DIRENT_D_RECLEN = @ac_cv_member_struct_dirent_d_reclen@ +HAVE_STRUCT_DIRENT_D_OFF = @ac_cv_member_struct_dirent_d_off@ +HAVE_STRUCT_DIRENT_D_TYPE = @ac_cv_member_struct_dirent_d_type@ + GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall # -Wbitwise -Wno-transparent-union -Wno-old-initializer -Wno-decl diff --git a/io/Makefile b/io/Makefile index 0b53f41..9574168 100644 --- a/io/Makefile +++ b/io/Makefile @@ -94,6 +94,16 @@ ifeq ($(HAVE_MREMAP),yes) LCFLAGS += -DHAVE_MREMAP endif +ifeq ($(HAVE_STRUCT_DIRENT_D_RECLEN),yes) +LCFLAGS += -D_DIRENT_HAVE_D_RECLEN +endif +ifeq ($(HAVE_STRUCT_DIRENT_D_OFF),yes) +LCFLAGS += -D_DIRENT_HAVE_D_OFF +endif +ifeq ($(HAVE_STRUCT_DIRENT_D_TYPE),yes) +LCFLAGS += -D_DIRENT_HAVE_D_TYPE +endif + default: depend $(LTCOMMAND) include $(BUILDRULES) -- 2.4.10 From david@fromorbit.com Tue Jan 12 15:07:22 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id DF2477F37 for ; Tue, 12 Jan 2016 15:07:22 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id B75A7304032 for ; Tue, 12 Jan 2016 13:07:19 -0800 (PST) X-ASG-Debug-ID: 1452632836-04bdf06f6eb0f50001-NocioJ Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id vJAVlKg2LqitNVl7 for ; Tue, 12 Jan 2016 13:07:16 -0800 (PST) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Apparent-Source-IP: 150.101.137.141 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DjDACgaZVWPGu7LXleKAECgw+BP4Jig36BeZ9qAQEBAQEBBotdhT2EBYYJBAICgSxNAQEBAQEBBwEBAQFBP0ESAYNgAQEBAwE6HCMFCwgDDgoJJQ8FJQMHGhMeiAgHwAQBAQgCIRmFdYVHiT0FkxCEA4g2hRqPBUSOD4JyAxyBcSo0hDKCEgEBAQ Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107]) by ipmail04.adl6.internode.on.net with ESMTP; 13 Jan 2016 07:35:18 +1030 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1aJ67x-00074l-EN; Wed, 13 Jan 2016 08:05:17 +1100 Date: Wed, 13 Jan 2016 08:05:17 +1100 From: Dave Chinner To: Christoph Hellwig Cc: xfs@oss.sgi.com Subject: Re: [RFC PATCH 0/8] xfs: shrink the struct xfs_icdinode Message-ID: <20160112210517.GK10456@dastard> X-ASG-Orig-Subj: Re: [RFC PATCH 0/8] xfs: shrink the struct xfs_icdinode References: <1452589280-30167-1-git-send-email-david@fromorbit.com> <20160112153010.GA29662@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160112153010.GA29662@infradead.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141] X-Barracuda-Start-Time: 1452632836 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_SA210e X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header 0.00 BSF_SC5_SA210e Custom Rule SA210e On Tue, Jan 12, 2016 at 07:30:10AM -0800, Christoph Hellwig wrote: > Nice! > > I had started on some of this a while ago but never finished it.. *nod* > > 72 bytes. This means it takes 104 bytes off the size of the struct > > xfs_inode, which a 12% reduction in size. This will be a massive win > > for systems that cache lots of inodes! > > How many more inodes can we fit into a slab cache now? Back when I > started I noticed it doesn't help us to actually fit more inodes into > a 4k page due to the bloated VFS inode. But these days slub actually > uses a high order allocations if I remember correctly so it might be > more useful. On a debug build (because that's the only numbers I have at hand), the current code is 1280 bytes/inode, 25 inodes/slab, 8 pages/slab. This patch set makes it 1152 bytes/inode, 28 inodes/slab, 8 pages/slab, so that's a 12% increase in the number of cached inodes for a given amount of slab memory. IIRC, a production build is around 960 bytes/inode, so that would put it at 34 inodes/slab for the current code and 38 inodes/slab for the current patchset. That, again, is roughly 12% increase in inodes per slab.... > > With this change made, the xfs_icdinode is no long an "in core disk > > inode" so I'm wondering whether I should rename it or simply make it > > go away altogether and pull the remaining fields straight into the > > struct xfs_inode. Any thoughts on new names and/or getting rid of it > > woul dbe appreciated. > > I think it should be merged into the xfs_inode structure soner or > later. Ok, I'll look at doing that. > Another thign I planned but never got to is to move fields > into the inode fork that were specific to the attr or data fork. That's a whole different problem ;) Cheers, Dave. -- Dave Chinner david@fromorbit.com From felix.janda@posteo.de Tue Jan 12 15:10:45 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 3A4287F37 for ; Tue, 12 Jan 2016 15:10:45 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id B9552AC004 for ; Tue, 12 Jan 2016 13:10:44 -0800 (PST) X-ASG-Debug-ID: 1452633037-04cb6c75dcab430001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id vWbxwNGb6yxQbSzM (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:10:41 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id 0DD1920970 for ; Tue, 12 Jan 2016 22:10:36 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg4Lh4sJhz5vND for ; Tue, 12 Jan 2016 22:10:36 +0100 (CET) Resent-From: Felix Janda Resent-Date: Tue, 12 Jan 2016 22:08:01 +0100 Resent-Message-ID: <20160112210801.GA10044@nyan> Resent-To: xfs@oss.sgi.com Message-Id: From: Felix Janda Date: Tue, 12 Jan 2016 22:04:07 +0100 Subject: [PATCH 0/6] xfsdump: Compile with musl libc To: xfs@oss.sgi.com X-ASG-Orig-Subj: [PATCH 0/6] xfsdump: Compile with musl libc X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452633041 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Six simple patches fixing compilation on linux systems using the musl c library. Felix Janda (6): invutil/invidx: include for O_CREAT dump/content.c: Include for XATTR_LIST_MAX common/main.c: Replace nonstandard SIGCLD by SIGCHLD librmt/rmtopen: Prefer over dump/var.c: Include for PATH_MAX Prefer stdint types over __uint*_t and __int*_t common/content_inode.h | 4 ++-- common/hsmapi.c | 2 +- common/main.c | 2 +- doc/xfsdump.html | 8 ++++---- dump/content.c | 1 + dump/var.c | 1 + inventory/inv_oref.h | 2 +- inventory/inv_priv.h | 4 ++-- inventory/inventory.h | 2 +- invutil/invidx.c | 1 + librmt/rmtopen.c | 2 +- 11 files changed, 16 insertions(+), 13 deletions(-) -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 15:10:54 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id 0B3817F37 for ; Tue, 12 Jan 2016 15:10:54 -0600 (CST) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay1.corp.sgi.com (Postfix) with ESMTP id D83CA8F8040 for ; Tue, 12 Jan 2016 13:10:50 -0800 (PST) X-ASG-Debug-ID: 1452633048-04bdf06f6db1190001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id 6M5PXp0ciuEQtz6p (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:10:49 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id C72392097C for ; Tue, 12 Jan 2016 22:10:47 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg4Lv4YyDz5vND for ; Tue, 12 Jan 2016 22:10:47 +0100 (CET) Date: Tue, 12 Jan 2016 22:08:10 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 1/6] invutil/invidx: include for O_CREAT Message-ID: <20160112210810.GB10044@nyan> X-ASG-Orig-Subj: [PATCH 1/6] invutil/invidx: include for O_CREAT References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452633048 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Signed-off-by: Felix Janda --- invutil/invidx.c | 1 + 1 file changed, 1 insertion(+) diff --git a/invutil/invidx.c b/invutil/invidx.c index 67efdf7..82d89d9 100644 --- a/invutil/invidx.c +++ b/invutil/invidx.c @@ -20,6 +20,7 @@ #include #include +#include #include #include #include -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 15:10:58 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 070F729DF9 for ; Tue, 12 Jan 2016 15:10:58 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id EEF48304059 for ; Tue, 12 Jan 2016 13:10:57 -0800 (PST) X-ASG-Debug-ID: 1452633055-04cb6c75ddab460001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id sLtZv48GmEk3AR4Z (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:10:56 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id 8D624209AB for ; Tue, 12 Jan 2016 22:10:55 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg4M336V1z5vN2 for ; Tue, 12 Jan 2016 22:10:55 +0100 (CET) Date: Tue, 12 Jan 2016 22:08:20 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 2/6] dump/content.c: Include for XATTR_LIST_MAX Message-ID: <20160112210820.GC10044@nyan> X-ASG-Orig-Subj: [PATCH 2/6] dump/content.c: Include for XATTR_LIST_MAX References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452633056 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Signed-off-by: Felix Janda --- dump/content.c | 1 + 1 file changed, 1 insertion(+) diff --git a/dump/content.c b/dump/content.c index 1e86292..97e42ab 100644 --- a/dump/content.c +++ b/dump/content.c @@ -36,6 +36,7 @@ #include #include #ifdef linux +#include #include #endif -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 15:11:06 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id AA40C7F37 for ; Tue, 12 Jan 2016 15:11:06 -0600 (CST) Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by relay3.corp.sgi.com (Postfix) with ESMTP id 36AFCAC001 for ; Tue, 12 Jan 2016 13:11:06 -0800 (PST) X-ASG-Debug-ID: 1452633064-04cbb05f7c13f250001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id BP3BNvRjt8qT39eW (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:11:04 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id BD445209CA for ; Tue, 12 Jan 2016 22:11:03 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg4MC4Y8Lz5vMK for ; Tue, 12 Jan 2016 22:11:03 +0100 (CET) Date: Tue, 12 Jan 2016 22:08:28 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 3/6] common/main.c: Replace nonstandard SIGCLD by SIGCHLD Message-ID: <20160112210828.GD10044@nyan> X-ASG-Orig-Subj: [PATCH 3/6] common/main.c: Replace nonstandard SIGCLD by SIGCHLD References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452633064 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Signed-off-by: Felix Janda --- common/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/common/main.c b/common/main.c index 3848499..faa66ee 100644 --- a/common/main.c +++ b/common/main.c @@ -2225,7 +2225,7 @@ static sig_printmap_t sig_printmap[ ] = { {SIGTERM, "SIGTERM"}, {SIGUSR1, "SIGUSR1"}, {SIGUSR2, "SIGUSR2"}, - {SIGCLD, "SIGCLD"}, + {SIGCHLD, "SIGCHLD"}, {SIGPWR, "SIGPWR"}, {SIGURG, "SIGURG"}, {SIGPOLL, "SIGPOLL"}, -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 15:11:13 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id AA4BA29E00 for ; Tue, 12 Jan 2016 15:11:13 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id 3CC0AAC004 for ; Tue, 12 Jan 2016 13:11:13 -0800 (PST) X-ASG-Debug-ID: 1452633071-04cb6c75dbab490001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id zBoB6uoTWQC3Cv38 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:11:12 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id DAAF4209B3 for ; Tue, 12 Jan 2016 22:11:10 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg4ML55fgz5vN3 for ; Tue, 12 Jan 2016 22:11:10 +0100 (CET) Date: Tue, 12 Jan 2016 22:08:35 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 4/6] librmt/rmtopen: Prefer over Message-ID: <20160112210835.GE10044@nyan> X-ASG-Orig-Subj: [PATCH 4/6] librmt/rmtopen: Prefer over References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452633071 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Signed-off-by: Felix Janda --- librmt/rmtopen.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/librmt/rmtopen.c b/librmt/rmtopen.c index 4bf9383..09ba3e4 100644 --- a/librmt/rmtopen.c +++ b/librmt/rmtopen.c @@ -27,7 +27,7 @@ #include #include #include -#include +#include #include #include #include -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 15:11:20 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15]) by oss.sgi.com (Postfix) with ESMTP id 4069929E06 for ; Tue, 12 Jan 2016 15:11:20 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay3.corp.sgi.com (Postfix) with ESMTP id BDAFAAC004 for ; Tue, 12 Jan 2016 13:11:19 -0800 (PST) X-ASG-Debug-ID: 1452633077-04cb6c75ddab4c0001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id ab8pY1YyZrayAx5N (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:11:18 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id 210A920963 for ; Tue, 12 Jan 2016 22:11:17 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg4MS75glz5vN3 for ; Tue, 12 Jan 2016 22:11:16 +0100 (CET) Date: Tue, 12 Jan 2016 22:08:41 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 5/6] dump/var.c: Include for PATH_MAX Message-ID: <20160112210841.GF10044@nyan> X-ASG-Orig-Subj: [PATCH 5/6] dump/var.c: Include for PATH_MAX References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452633077 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Signed-off-by: Felix Janda --- dump/var.c | 1 + 1 file changed, 1 insertion(+) diff --git a/dump/var.c b/dump/var.c index 645caab..6e3ead0 100644 --- a/dump/var.c +++ b/dump/var.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include "config.h" -- 2.4.10 From felix.janda@posteo.de Tue Jan 12 15:11:27 2016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham version=3.3.1 X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111]) by oss.sgi.com (Postfix) with ESMTP id BFCFE29E00 for ; Tue, 12 Jan 2016 15:11:27 -0600 (CST) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay1.corp.sgi.com (Postfix) with ESMTP id B23818F8040 for ; Tue, 12 Jan 2016 13:11:27 -0800 (PST) X-ASG-Debug-ID: 1452633085-04cb6c75dbab4e0001-NocioJ Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id 91xEBIHO0Vrm1IeM (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:11:26 -0800 (PST) X-Barracuda-Envelope-From: felix.janda@posteo.de X-Barracuda-Apparent-Source-IP: 185.67.36.65 Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13]) by mout01.posteo.de (Postfix) with ESMTPS id EE1502089D for ; Tue, 12 Jan 2016 22:11:24 +0100 (CET) Received: from mail.posteo.de (localhost [127.0.0.1]) by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg4Mc5xQGz5vNH for ; Tue, 12 Jan 2016 22:11:24 +0100 (CET) Date: Tue, 12 Jan 2016 22:08:49 +0100 From: Felix Janda To: xfs@oss.sgi.com Subject: [PATCH 6/6] Prefer stdint types over __uint*_t and __int*_t Message-ID: <20160112210849.GG10044@nyan> X-ASG-Orig-Subj: [PATCH 6/6] Prefer stdint types over __uint*_t and __int*_t References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Barracuda-Connect: mout01.posteo.de[185.67.36.65] X-Barracuda-Start-Time: 1452633085 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Signed-off-by: Felix Janda --- common/content_inode.h | 4 ++-- common/hsmapi.c | 2 +- doc/xfsdump.html | 8 ++++---- inventory/inv_oref.h | 2 +- inventory/inv_priv.h | 4 ++-- inventory/inventory.h | 2 +- 6 files changed, 11 insertions(+), 11 deletions(-) diff --git a/common/content_inode.h b/common/content_inode.h index a69a9a0..6936e05 100644 --- a/common/content_inode.h +++ b/common/content_inode.h @@ -191,10 +191,10 @@ typedef struct bstat bstat_t; * and using two 16bit values to hold new 32bit projid was choosen * to retain compatibility with "old" filesystems). */ -static inline __uint32_t +static inline uint32_t bstat_projid(struct bstat *bs) { - return (__uint32_t)bs->bs_projid_hi << 16 | bs->bs_projid_lo; + return (uint32_t)bs->bs_projid_hi << 16 | bs->bs_projid_lo; } diff --git a/common/hsmapi.c b/common/hsmapi.c index 6054773..515d392 100644 --- a/common/hsmapi.c +++ b/common/hsmapi.c @@ -585,7 +585,7 @@ HsmModifyExtentMap( struct getbmapx *bmap) { dmf_f_ctxt_t *dmf_f_ctxtp = (dmf_f_ctxt_t *)contextp; - __int64_t length; + int64_t length; if (bmap[0].bmv_entries <= 0) { return 1; /* caller must already be at EOF */ diff --git a/doc/xfsdump.html b/doc/xfsdump.html index 9c6722e..f85128f 100644 --- a/doc/xfsdump.html +++ b/doc/xfsdump.html @@ -1935,7 +1935,7 @@ The files are constructed like so: 
 typedef struct invt_counter {
     INVT_COUNTER_FIELDS
-        __uint32_t    ic_vernum;/* on disk version number for posterity */\
+        uint32_t    ic_vernum;/* on disk version number for posterity */\
         u_int         ic_curnum;/* number of sessions/invindices recorded \
                                    so far */                              \
         u_int         ic_maxnum;/* maximum number of sessions/inv_indices \
@@ -1975,7 +1975,7 @@ typedef struct invt_fstab {
 
 typedef struct invt_counter {
     INVT_COUNTER_FIELDS
-        __uint32_t    ic_vernum;/* on disk version number for posterity */\
+        uint32_t    ic_vernum;/* on disk version number for posterity */\
         u_int         ic_curnum;/* number of sessions/invindices recorded \
                                    so far */                              \
         u_int         ic_maxnum;/* maximum number of sessions/inv_indices \
@@ -2012,7 +2012,7 @@ typedef struct invt_entry {
 
 typedef struct invt_sescounter {
     INVT_COUNTER_FIELDS
-        __uint32_t    ic_vernum;/* on disk version number for posterity */\
+        uint32_t    ic_vernum;/* on disk version number for posterity */\
         u_int         ic_curnum;/* number of sessions/invindices recorded \
                                    so far */                              \
         u_int         ic_maxnum;/* maximum number of sessions/inv_indices \
@@ -2034,7 +2034,7 @@ typedef struct invt_seshdr {
     off64_t    sh_streams_off; /* offset to start of the set of
                                   stream hdrs */
     time_t     sh_time;        /* time of the dump */
-    __uint32_t sh_flag;        /* for misc flags */
+    uint32_t sh_flag;        /* for misc flags */
     u_char     sh_level;       /* dump level */
     u_char     sh_pruned;      /* pruned by invutil flag */
     char       sh_padding[22];
diff --git a/inventory/inv_oref.h b/inventory/inv_oref.h
index e16684d..38303a4 100644
--- a/inventory/inv_oref.h
+++ b/inventory/inv_oref.h
@@ -46,7 +46,7 @@
 
 
 
-typedef __uint32_t	invt_objtype_t;
+typedef uint32_t	invt_objtype_t;
 
 #define INVT_LOCKED		0x0001
 
diff --git a/inventory/inv_priv.h b/inventory/inv_priv.h
index 1690271..364ffbc 100644
--- a/inventory/inv_priv.h
+++ b/inventory/inv_priv.h
@@ -120,7 +120,7 @@ typedef struct invt_seshdr {
 	off64_t		sh_streams_off; /* offset to start of the set of 
 					   stream hdrs */
 	time32_t	sh_time;        /* time of the dump */
-	__uint32_t	sh_flag;        /* for misc flags */
+	uint32_t	sh_flag;        /* for misc flags */
 	u_char		sh_level;       /* dump level */
 	u_char		sh_pruned;      /* pruned by invutil flag */
 	char		sh_padding[22];
@@ -184,7 +184,7 @@ typedef struct invt_entry {
 
 /* Cheap Inheritance, and an attempt to avoid a nested type */
 #define INVT_COUNTER_FIELDS  \
-        __uint32_t    ic_vernum;/* on disk version number for posterity */\
+        uint32_t    ic_vernum;/* on disk version number for posterity */\
 	uint	      ic_curnum;/* number of sessions/invindices recorded \
 				   so far */				  \
 	uint	      ic_maxnum;/* maximum number of sessions/inv_indices \
diff --git a/inventory/inventory.h b/inventory/inventory.h
index 134b9ba..c1d7403 100644
--- a/inventory/inventory.h
+++ b/inventory/inventory.h
@@ -43,7 +43,7 @@
 /* length of labels, mntpts, etc */
 #define INV_STRLEN              GLOBAL_HDR_STRING_SZ
 
-typedef __uint32_t		inv_version_t;
+typedef uint32_t		inv_version_t;
 
 /* This is the general inventory version. 
  */
-- 
2.4.10

From david@fromorbit.com  Tue Jan 12 15:24:14 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id CB1467F37
	for ; Tue, 12 Jan 2016 15:24:14 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id AFE6D8F8052
	for ; Tue, 12 Jan 2016 13:24:14 -0800 (PST)
X-ASG-Debug-ID: 1452633851-04bdf06f6eb18f0001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id B0IDUeh1onL0EWVE for ; Tue, 12 Jan 2016 13:24:12 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2BkDQCybpVWPGu7LXleKAECgw+BP4Jig36BeZ9qAQEBAQEBBotdhT2EBYYJAgIBAQKBLE0BAQEBAQEHAQEBAUE/QRIBg2EBAQQ6HCMQCAMYCSUPBSUDBxoTiC3AAgELASAZhXWFR4k9BZcTiDaFGo8FRI4PhQIqNIQyghIBAQE
Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107])
  by ipmail04.adl6.internode.on.net with ESMTP; 13 Jan 2016 07:54:06 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aJ6QA-00076z-1c; Wed, 13 Jan 2016 08:24:06 +1100
Date: Wed, 13 Jan 2016 08:24:06 +1100
From: Dave Chinner 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
Message-ID: <20160112212405.GL10456@dastard>
X-ASG-Orig-Subj: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
References: 
 <20160112195935.GB568@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112195935.GB568@nyan>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1452633851
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue, Jan 12, 2016 at 08:59:35PM +0100, Felix Janda wrote:
> The uint*_t and int*_t are defined by C99 and should be prefered
> over the less portable __uint*_t and __int*_t variants. The
> necessary  include is in platformdefs.h, which gets
> included in most places via libxfs.h. In the public headers
>  is included in .
> 
> Signed-off-by: Felix Janda 

I can't apply this straight off. Most of the libxfs code that is
changed is shared with the kernel code, and so the definitions of
the variables need to be the same as the kernel code. There are
reasons for the kernel code using __[u]int*_t type variants (e.g.  I
think the endian conversion static checker requires the __ variants
for host order variables), and so before making sweeping changes
like this we need to ensure that we can make the equivalent changes
to the kernel code as well...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Tue Jan 12 15:24:58 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 9EC2F7F37
	for ; Tue, 12 Jan 2016 15:24:58 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 84E2C8F804B
	for ; Tue, 12 Jan 2016 13:24:58 -0800 (PST)
X-ASG-Debug-ID: 1452633895-04cb6c75ddabb20001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id BOLS9affOiFUQE8U for ; Tue, 12 Jan 2016 13:24:55 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DAFQCybpVWPGu7LXleKAECgw+BP4Jig36BeZ9qAQEBAQEBBotdhT2EBYYJBAICgSxNAQEBAQEBBwEBAQFBP0EBBA0Bg2EBAQQ6HCMQCAMYCSUPBSUDBxoTiC3AAgEBCAIhGYV1hUeJPQWXE41QjwWOU4UCKjSCHYIVghIBAQE
Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107])
  by ipmail04.adl6.internode.on.net with ESMTP; 13 Jan 2016 07:54:54 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aJ6Qw-000778-Bj; Wed, 13 Jan 2016 08:24:54 +1100
Date: Wed, 13 Jan 2016 08:24:54 +1100
From: Dave Chinner 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
Message-ID: <20160112212454.GM10456@dastard>
X-ASG-Orig-Subj: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
References: 
 <20160112195945.GC568@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112195945.GC568@nyan>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1452633895
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue, Jan 12, 2016 at 08:59:45PM +0100, Felix Janda wrote:
> ustat has been used to check whether a device file is mounted.
> The function is deprecated and not supported by uclibc and musl.
> Now do the check using the *mntent functions.

Which will cause problems with builds on other platforms. Why not
just use statfs() instead?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Tue Jan 12 15:30:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id E5C097F37
	for ; Tue, 12 Jan 2016 15:30:12 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 6368BAC001
	for ; Tue, 12 Jan 2016 13:30:12 -0800 (PST)
X-ASG-Debug-ID: 1452634208-04cb6c75deabcd0001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id 0A5N4AMNHcLCpuAg for ; Tue, 12 Jan 2016 13:30:08 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2CzDQDib5VWPGu7LXleKAECgw9SbYJig36BeZ9eDAEBAQEBAQaLXYU9hAUahW8CAgEBAoEsTQEBAQEBAQcBAQEBQT9BEgGDYQEBBDocIxAIAxgJJQ8FJQMHGhOILcAEAQEIAgEgGYV1hUeJPQWXE4VDiA2CMYxUjlODLoFUKjSEMoISAQEB
Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107])
  by ipmail04.adl6.internode.on.net with ESMTP; 13 Jan 2016 08:00:07 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aJ6Vz-00077w-04; Wed, 13 Jan 2016 08:30:07 +1100
Date: Wed, 13 Jan 2016 08:30:06 +1100
From: Dave Chinner 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
Message-ID: <20160112213006.GN10456@dastard>
X-ASG-Orig-Subj: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
References: 
 <20160112200022.GG568@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112200022.GG568@nyan>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1452634208
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue, Jan 12, 2016 at 09:00:22PM +0100, Felix Janda wrote:
> Signed-off-by: Felix Janda 
> ---
>  configure.ac         |  7 +++++++
>  include/builddefs.in |  4 ++++
>  io/Makefile          | 10 ++++++++++
>  3 files changed, 21 insertions(+)
> 
> diff --git a/configure.ac b/configure.ac
> index feee50a..3d6443a 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -135,5 +135,12 @@ AC_CHECK_SIZEOF([char *])
>  AC_TYPE_UMODE_T
>  AC_MANUAL_FORMAT
>  
> +AC_CHECK_MEMBERS([struct dirent.d_reclen],[],[],[[#include ]])
> +AC_SUBST(ac_cv_member_struct_dirent_d_reclen)
> +AC_CHECK_MEMBERS([struct dirent.d_off],[],[],[[#include ]])
> +AC_SUBST(ac_cv_member_struct_dirent_d_off)
> +AC_CHECK_MEMBERS([struct dirent.d_type],[],[],[[#include ]])
> +AC_SUBST(ac_cv_member_struct_dirent_d_type)
> +
>  AC_CONFIG_FILES([include/builddefs])
>  AC_OUTPUT
> diff --git a/include/builddefs.in b/include/builddefs.in
> index b5ce336..67d963c 100644
> --- a/include/builddefs.in
> +++ b/include/builddefs.in
> @@ -109,6 +109,10 @@ HAVE_FLS = @have_fls@
>  HAVE_FSETXATTR = @have_fsetxattr@
>  HAVE_MREMAP = @have_mremap@
>  
> +HAVE_STRUCT_DIRENT_D_RECLEN = @ac_cv_member_struct_dirent_d_reclen@
> +HAVE_STRUCT_DIRENT_D_OFF = @ac_cv_member_struct_dirent_d_off@
> +HAVE_STRUCT_DIRENT_D_TYPE = @ac_cv_member_struct_dirent_d_type@
> +
>  GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall
>  #	   -Wbitwise -Wno-transparent-union -Wno-old-initializer -Wno-decl
>  
> diff --git a/io/Makefile b/io/Makefile
> index 0b53f41..9574168 100644
> --- a/io/Makefile
> +++ b/io/Makefile
> @@ -94,6 +94,16 @@ ifeq ($(HAVE_MREMAP),yes)
>  LCFLAGS += -DHAVE_MREMAP
>  endif
>  
> +ifeq ($(HAVE_STRUCT_DIRENT_D_RECLEN),yes)
> +LCFLAGS += -D_DIRENT_HAVE_D_RECLEN
> +endif

The C library should be defining these if those fields are
supported, right?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From felix.janda@posteo.de  Tue Jan 12 15:36:01 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 54B917F37
	for ; Tue, 12 Jan 2016 15:36:01 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 43CC2304032
	for ; Tue, 12 Jan 2016 13:36:01 -0800 (PST)
X-ASG-Debug-ID: 1452634557-04cb6c75deabf10001-NocioJ
Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id Tqil2ipCMgzViZVT (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:35:58 -0800 (PST)
X-Barracuda-Envelope-From: felix.janda@posteo.de
X-Barracuda-Apparent-Source-IP: 185.67.36.65
Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13])
	by mout01.posteo.de (Postfix) with ESMTPS id A4F1D20B03
	for ; Tue, 12 Jan 2016 22:35:57 +0100 (CET)
Received: from mail.posteo.de (localhost [127.0.0.1])
	by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg4vx3ntLz5vND;
	Tue, 12 Jan 2016 22:35:57 +0100 (CET)
Date: Tue, 12 Jan 2016 22:33:22 +0100
From: Felix Janda 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
Message-ID: <20160112213322.GA10558@nyan>
X-ASG-Orig-Subj: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
References: 
 <20160112195945.GC568@nyan>
 <20160112212454.GM10456@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112212454.GM10456@dastard>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Barracuda-Connect: mout01.posteo.de[185.67.36.65]
X-Barracuda-Start-Time: 1452634558
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Dave Chinner wrote:
> On Tue, Jan 12, 2016 at 08:59:45PM +0100, Felix Janda wrote:
> > ustat has been used to check whether a device file is mounted.
> > The function is deprecated and not supported by uclibc and musl.
> > Now do the check using the *mntent functions.
> 
> Which will cause problems with builds on other platforms.

This is the linux specific code...

>  Why not
> just use statfs() instead?

Since it does not have the functionality which ustat is used for.

Felix

From felix.janda@posteo.de  Tue Jan 12 15:39:59 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 486FF7F3F
	for ; Tue, 12 Jan 2016 15:39:59 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 3A26F304032
	for ; Tue, 12 Jan 2016 13:39:59 -0800 (PST)
X-ASG-Debug-ID: 1452634796-04cb6c75deac0d0001-NocioJ
Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id vobvtfH8IBro5mj1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:39:57 -0800 (PST)
X-Barracuda-Envelope-From: felix.janda@posteo.de
X-Barracuda-Apparent-Source-IP: 185.67.36.65
Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13])
	by mout01.posteo.de (Postfix) with ESMTPS id A312C20AB3
	for ; Tue, 12 Jan 2016 22:39:56 +0100 (CET)
Received: from mail.posteo.de (localhost [127.0.0.1])
	by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg50X3P2jz5vNG;
	Tue, 12 Jan 2016 22:39:56 +0100 (CET)
Date: Tue, 12 Jan 2016 22:37:11 +0100
From: Felix Janda 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
Message-ID: <20160112213711.GB10558@nyan>
X-ASG-Orig-Subj: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
References: 
 <20160112200022.GG568@nyan>
 <20160112213006.GN10456@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112213006.GN10456@dastard>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Barracuda-Connect: mout01.posteo.de[185.67.36.65]
X-Barracuda-Start-Time: 1452634797
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Dave Chinner wrote:
> On Tue, Jan 12, 2016 at 09:00:22PM +0100, Felix Janda wrote:
> > Signed-off-by: Felix Janda 
> > ---
> >  configure.ac         |  7 +++++++
> >  include/builddefs.in |  4 ++++
> >  io/Makefile          | 10 ++++++++++
> >  3 files changed, 21 insertions(+)
> > 
> > diff --git a/configure.ac b/configure.ac
> > index feee50a..3d6443a 100644
> > --- a/configure.ac
> > +++ b/configure.ac
> > @@ -135,5 +135,12 @@ AC_CHECK_SIZEOF([char *])
> >  AC_TYPE_UMODE_T
> >  AC_MANUAL_FORMAT
> >  
> > +AC_CHECK_MEMBERS([struct dirent.d_reclen],[],[],[[#include ]])
> > +AC_SUBST(ac_cv_member_struct_dirent_d_reclen)
> > +AC_CHECK_MEMBERS([struct dirent.d_off],[],[],[[#include ]])
> > +AC_SUBST(ac_cv_member_struct_dirent_d_off)
> > +AC_CHECK_MEMBERS([struct dirent.d_type],[],[],[[#include ]])
> > +AC_SUBST(ac_cv_member_struct_dirent_d_type)
> > +
> >  AC_CONFIG_FILES([include/builddefs])
> >  AC_OUTPUT
> > diff --git a/include/builddefs.in b/include/builddefs.in
> > index b5ce336..67d963c 100644
> > --- a/include/builddefs.in
> > +++ b/include/builddefs.in
> > @@ -109,6 +109,10 @@ HAVE_FLS = @have_fls@
> >  HAVE_FSETXATTR = @have_fsetxattr@
> >  HAVE_MREMAP = @have_mremap@
> >  
> > +HAVE_STRUCT_DIRENT_D_RECLEN = @ac_cv_member_struct_dirent_d_reclen@
> > +HAVE_STRUCT_DIRENT_D_OFF = @ac_cv_member_struct_dirent_d_off@
> > +HAVE_STRUCT_DIRENT_D_TYPE = @ac_cv_member_struct_dirent_d_type@
> > +
> >  GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall
> >  #	   -Wbitwise -Wno-transparent-union -Wno-old-initializer -Wno-decl
> >  
> > diff --git a/io/Makefile b/io/Makefile
> > index 0b53f41..9574168 100644
> > --- a/io/Makefile
> > +++ b/io/Makefile
> > @@ -94,6 +94,16 @@ ifeq ($(HAVE_MREMAP),yes)
> >  LCFLAGS += -DHAVE_MREMAP
> >  endif
> >  
> > +ifeq ($(HAVE_STRUCT_DIRENT_D_RECLEN),yes)
> > +LCFLAGS += -D_DIRENT_HAVE_D_RECLEN
> > +endif
> 
> The C library should be defining these if those fields are
> supported, right?

It is strongly advised not to use these non-standard fields of dirent.
These macros are a feature of glibc. See also

https://seiscode.iris.washington.edu/issues/676

Felix

From felix.janda@posteo.de  Tue Jan 12 15:49:27 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 0CA3D7F3F
	for ; Tue, 12 Jan 2016 15:49:27 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E5C2C8F8049
	for ; Tue, 12 Jan 2016 13:49:26 -0800 (PST)
X-ASG-Debug-ID: 1452635364-04cb6c75deac4f0001-NocioJ
Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id GdnwhXZkJHx7JVTb (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 13:49:24 -0800 (PST)
X-Barracuda-Envelope-From: felix.janda@posteo.de
X-Barracuda-Apparent-Source-IP: 185.67.36.65
Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13])
	by mout01.posteo.de (Postfix) with ESMTPS id D6CBD20AAC
	for ; Tue, 12 Jan 2016 22:49:23 +0100 (CET)
Received: from mail.posteo.de (localhost [127.0.0.1])
	by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pg5CN3wX9z5vNB;
	Tue, 12 Jan 2016 22:49:20 +0100 (CET)
Date: Tue, 12 Jan 2016 22:46:34 +0100
From: Felix Janda 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
Message-ID: <20160112214634.GC10558@nyan>
X-ASG-Orig-Subj: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
References: 
 <20160112195935.GB568@nyan>
 <20160112212405.GL10456@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112212405.GL10456@dastard>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Barracuda-Connect: mout01.posteo.de[185.67.36.65]
X-Barracuda-Start-Time: 1452635364
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Dave Chinner wrote:
> On Tue, Jan 12, 2016 at 08:59:35PM +0100, Felix Janda wrote:
> > The uint*_t and int*_t are defined by C99 and should be prefered
> > over the less portable __uint*_t and __int*_t variants. The
> > necessary  include is in platformdefs.h, which gets
> > included in most places via libxfs.h. In the public headers
> >  is included in .
> > 
> > Signed-off-by: Felix Janda 
> 
> I can't apply this straight off. Most of the libxfs code that is
> changed is shared with the kernel code, and so the definitions of
> the variables need to be the same as the kernel code. There are
> reasons for the kernel code using __[u]int*_t type variants (e.g.  I
> think the endian conversion static checker requires the __ variants
> for host order variables), and so before making sweeping changes
> like this we need to ensure that we can make the equivalent changes
> to the kernel code as well...

Thanks for the review!

Sorry, I was not aware about this difference between the types.

The simplest fix for musl would be to add defines or something similar
to linux.h.

On the other hand, on the long run it would be preferable to use
the stdint types (consistently).

Felix

From david@fromorbit.com  Tue Jan 12 16:03:04 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 4D6577F37
	for ; Tue, 12 Jan 2016 16:03:04 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id DE638AC004
	for ; Tue, 12 Jan 2016 14:03:00 -0800 (PST)
X-ASG-Debug-ID: 1452636177-04bdf06f6db2b40001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id sYU08B1OBpdESfK2 for ; Tue, 12 Jan 2016 14:02:57 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2BkDQDpdpVWPGu7LXleKAECgw+BP4Jig36BeZ9qAQEBAQEBBotdhT2EBYYJAgIBAQKBLE0BAQEBAQEHAQEBAUE/QRIBg2EBAQQ6HCMQCAMYCSUPBSUDBxoTG4gSwAQBCwEgGYV1hUeJPQWXE4g2hRqPBUSOD4UCKjSEMoISAQEB
Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107])
  by ipmail04.adl6.internode.on.net with ESMTP; 13 Jan 2016 08:32:41 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aJ71U-0007CQ-D3; Wed, 13 Jan 2016 09:02:40 +1100
Date: Wed, 13 Jan 2016 09:02:40 +1100
From: Dave Chinner 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
Message-ID: <20160112220240.GO10456@dastard>
X-ASG-Orig-Subj: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
References: 
 <20160112195935.GB568@nyan>
 <20160112212405.GL10456@dastard>
 <20160112214634.GC10558@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112214634.GC10558@nyan>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1452636177
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue, Jan 12, 2016 at 10:46:34PM +0100, Felix Janda wrote:
> Dave Chinner wrote:
> > On Tue, Jan 12, 2016 at 08:59:35PM +0100, Felix Janda wrote:
> > > The uint*_t and int*_t are defined by C99 and should be prefered
> > > over the less portable __uint*_t and __int*_t variants. The
> > > necessary  include is in platformdefs.h, which gets
> > > included in most places via libxfs.h. In the public headers
> > >  is included in .
> > > 
> > > Signed-off-by: Felix Janda 
> > 
> > I can't apply this straight off. Most of the libxfs code that is
> > changed is shared with the kernel code, and so the definitions of
> > the variables need to be the same as the kernel code. There are
> > reasons for the kernel code using __[u]int*_t type variants (e.g.  I
> > think the endian conversion static checker requires the __ variants
> > for host order variables), and so before making sweeping changes
> > like this we need to ensure that we can make the equivalent changes
> > to the kernel code as well...
> 
> Thanks for the review!
> 
> Sorry, I was not aware about this difference between the types.
> 
> The simplest fix for musl would be to add defines or something similar
> to linux.h.
> 
> On the other hand, on the long run it would be preferable to use
> the stdint types (consistently).

Right, that's definitely the better solution. We can't just jump
there in one go. :/

FWIW, changing everything outside libxfs/ and include/ should be ok
to do straight away; it's just the libxfs stuff that we have other
constraints on.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Tue Jan 12 16:37:11 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 557227F37
	for ; Tue, 12 Jan 2016 16:37:11 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 2673730405F
	for ; Tue, 12 Jan 2016 14:37:07 -0800 (PST)
X-ASG-Debug-ID: 1452638224-04bdf06f6cb3c00001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id TAHoGJSzcpqYyl04 for ; Tue, 12 Jan 2016 14:37:05 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2C0DQAff5VWPGu7LXleKAECgw9SbYJig36BeZ9eDAEBAQEBAQaLXYU9hAUahW8CAgEBAoEsTQEBAQEBAQcBAQEBQT9BEgGDYAEBAQMBOhwjBQsIAxgJJQ8FJQMHGhOIJgfAFgEBCAIBIBmFdYVHiT0FlxOFQ4gNgjGMVI5Tgy6BVCo0hDKCEgEBAQ
Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107])
  by ipmail04.adl6.internode.on.net with ESMTP; 13 Jan 2016 09:07:04 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aJ7Yl-0007H7-QA; Wed, 13 Jan 2016 09:37:03 +1100
Date: Wed, 13 Jan 2016 09:37:03 +1100
From: Dave Chinner 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
Message-ID: <20160112223703.GP10456@dastard>
X-ASG-Orig-Subj: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
References: 
 <20160112200022.GG568@nyan>
 <20160112213006.GN10456@dastard>
 <20160112213711.GB10558@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112213711.GB10558@nyan>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1452638224
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26070
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue, Jan 12, 2016 at 10:37:11PM +0100, Felix Janda wrote:
> Dave Chinner wrote:
> > On Tue, Jan 12, 2016 at 09:00:22PM +0100, Felix Janda wrote:
> > > Signed-off-by: Felix Janda 
> > > ---
> > >  configure.ac         |  7 +++++++
> > >  include/builddefs.in |  4 ++++
> > >  io/Makefile          | 10 ++++++++++
> > >  3 files changed, 21 insertions(+)
> > > 
> > > diff --git a/configure.ac b/configure.ac
> > > index feee50a..3d6443a 100644
> > > --- a/configure.ac
> > > +++ b/configure.ac
> > > @@ -135,5 +135,12 @@ AC_CHECK_SIZEOF([char *])
> > >  AC_TYPE_UMODE_T
> > >  AC_MANUAL_FORMAT
> > >  
> > > +AC_CHECK_MEMBERS([struct dirent.d_reclen],[],[],[[#include ]])
> > > +AC_SUBST(ac_cv_member_struct_dirent_d_reclen)
> > > +AC_CHECK_MEMBERS([struct dirent.d_off],[],[],[[#include ]])
> > > +AC_SUBST(ac_cv_member_struct_dirent_d_off)
> > > +AC_CHECK_MEMBERS([struct dirent.d_type],[],[],[[#include ]])
> > > +AC_SUBST(ac_cv_member_struct_dirent_d_type)
> > > +
> > >  AC_CONFIG_FILES([include/builddefs])
> > >  AC_OUTPUT
> > > diff --git a/include/builddefs.in b/include/builddefs.in
> > > index b5ce336..67d963c 100644
> > > --- a/include/builddefs.in
> > > +++ b/include/builddefs.in
> > > @@ -109,6 +109,10 @@ HAVE_FLS = @have_fls@
> > >  HAVE_FSETXATTR = @have_fsetxattr@
> > >  HAVE_MREMAP = @have_mremap@
> > >  
> > > +HAVE_STRUCT_DIRENT_D_RECLEN = @ac_cv_member_struct_dirent_d_reclen@
> > > +HAVE_STRUCT_DIRENT_D_OFF = @ac_cv_member_struct_dirent_d_off@
> > > +HAVE_STRUCT_DIRENT_D_TYPE = @ac_cv_member_struct_dirent_d_type@
> > > +
> > >  GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall
> > >  #	   -Wbitwise -Wno-transparent-union -Wno-old-initializer -Wno-decl
> > >  
> > > diff --git a/io/Makefile b/io/Makefile
> > > index 0b53f41..9574168 100644
> > > --- a/io/Makefile
> > > +++ b/io/Makefile
> > > @@ -94,6 +94,16 @@ ifeq ($(HAVE_MREMAP),yes)
> > >  LCFLAGS += -DHAVE_MREMAP
> > >  endif
> > >  
> > > +ifeq ($(HAVE_STRUCT_DIRENT_D_RECLEN),yes)
> > > +LCFLAGS += -D_DIRENT_HAVE_D_RECLEN
> > > +endif
> > 
> > The C library should be defining these if those fields are
> > supported, right?
> 
> It is strongly advised not to use these non-standard fields of dirent.

We're not writing POSIX applications here - this is a diagnostic
tool so if we are running on linux we need to expose them.

If we've got linux C libraries that don't provide feature macros for
the fields in the linux_dirent eposed by readdir(3), then I guess
we're stuck with having to play autoconf games, eh?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From darrick.wong@oracle.com  Tue Jan 12 19:30:00 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 6485F7F37
	for ; Tue, 12 Jan 2016 19:30:00 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 54596304053
	for ; Tue, 12 Jan 2016 17:29:57 -0800 (PST)
X-ASG-Debug-ID: 1452648593-04cb6c75dcb22c0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id AugVC0sE0Hy4vkHj (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 12 Jan 2016 17:29:53 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0D1TmoT031981
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 13 Jan 2016 01:29:48 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0D1Tme2012482
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 13 Jan 2016 01:29:48 GMT
Received: from abhmp0017.oracle.com (abhmp0017.oracle.com [141.146.116.23])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0D1TloE016147;
	Wed, 13 Jan 2016 01:29:47 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Tue, 12 Jan 2016 17:29:46 -0800
Date: Tue, 12 Jan 2016 17:29:45 -0800
From: "Darrick J. Wong" 
To: Brian Foster 
Cc: Dave Chinner , xfs@oss.sgi.com
Subject: Re: [PATCH] xfs_db: check on-disk structure sizes
Message-ID: <20160113012945.GC2455@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH] xfs_db: check on-disk structure sizes
References: <20160111234644.GB7831@birch.djwong.org>
 <20160112140122.GC12156@bfoster.bfoster>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112140122.GC12156@bfoster.bfoster>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1452648593
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26075
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Tue, Jan 12, 2016 at 09:01:22AM -0500, Brian Foster wrote:
> On Mon, Jan 11, 2016 at 03:46:44PM -0800, Darrick J. Wong wrote:
> > Check on-disk structure sizes against known values.
> > Use this to catch inadvertent changes in structure size due to padding
> > and alignment issues, etc.
> > 
> > Signed-off-by: Darrick J. Wong 
> > ---
> 
> What's the need for this in userspace? Not a big deal really, but it
> seems like it serves the fundamental purpose sufficiently in the kernel.

The primary point is to make sure that we didn't make any errors with the
on-disk structures when porting libxfs changes.  The kernel build is the first
line of defense since it tends to big get changes first, but I figure a
defensive build check for xfsprogs won't harm anyone...

> That aside, this does fail if I tweak a structure size, though I don't
> think I get the intended error. I end up with a linker error instead:

...except for that.  Should I dig deeper into figuring out why this doesn't
work in userland, or is a filename and line number sufficient to figure out
which structure is misbehaving?

--D

> 
> ...
>     [LD]     xfs_db
> ondisk.o: In function `xfs_check_ondisk_structs':
> /home/bfoster/repos/xfsprogs-dev/db/ondisk.c:26: undefined reference to `__compiletime_assert_26'
> collect2: error: ld returned 1 exit status
> ../include/buildrules:45: recipe for target 'xfs_db' failed
> gmake[3]: *** [xfs_db] Error 1
> include/buildrules:35: recipe for target 'db' failed
> gmake[2]: *** [db] Error 2
> Makefile:70: recipe for target 'default' failed
> make[1]: *** [default] Error 2
> Makefile:68: recipe for target 'default' failed
> make: *** [default] Error 2
> 
> This is with gcc 5.3.1. It works fine with the kernel patch so I don't
> _think_ it's my environment, but I could be wrong...
> 
> Brian
> 
> >  db/Makefile         |    2 -
> >  db/init.c           |    3 +
> >  db/ondisk.c         |   63 +++++++++++++++++++++++
> >  db/ondisk.h         |  139 +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  libxfs/xfs_format.h |    4 +
> >  5 files changed, 210 insertions(+), 1 deletion(-)
> >  create mode 100644 db/ondisk.c
> >  create mode 100644 db/ondisk.h
> > 
> > diff --git a/db/Makefile b/db/Makefile
> > index 8260da3..ba3e942 100644
> > --- a/db/Makefile
> > +++ b/db/Makefile
> > @@ -12,7 +12,7 @@ HFILES = addr.h agf.h agfl.h agi.h attr.h attrshort.h bit.h block.h bmap.h \
> >  	dir2.h dir2sf.h dquot.h echo.h faddr.h field.h \
> >  	flist.h fprint.h frag.h freesp.h hash.h help.h init.h inode.h input.h \
> >  	io.h logformat.h malloc.h metadump.h output.h print.h quit.h sb.h \
> > -	 sig.h strvec.h text.h type.h write.h attrset.h symlink.h
> > +	 sig.h strvec.h text.h type.h write.h attrset.h symlink.h ondisk.h
> >  CFILES = $(HFILES:.h=.c)
> >  LSRCFILES = xfs_admin.sh xfs_ncheck.sh xfs_metadump.sh
> >  
> > diff --git a/db/init.c b/db/init.c
> > index c0472c8..d6df093 100644
> > --- a/db/init.c
> > +++ b/db/init.c
> > @@ -28,6 +28,7 @@
> >  #include "output.h"
> >  #include "malloc.h"
> >  #include "type.h"
> > +#include "ondisk.h"
> >  
> >  static char		**cmdline;
> >  static int		ncmdline;
> > @@ -60,6 +61,8 @@ init(
> >  	struct xfs_buf	*bp;
> >  	int		c;
> >  
> > +	xfs_check_ondisk_structs();
> > +
> >  	setlocale(LC_ALL, "");
> >  	bindtextdomain(PACKAGE, LOCALEDIR);
> >  	textdomain(PACKAGE);
> > diff --git a/db/ondisk.c b/db/ondisk.c
> > new file mode 100644
> > index 0000000..532333d
> > --- /dev/null
> > +++ b/db/ondisk.c
> > @@ -0,0 +1,63 @@
> > +/*
> > + * Copyright (c) 2016 Oracle.
> > + * All Rights Reserved.
> > + *
> > + * This program is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU General Public License as
> > + * published by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it would be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > + * GNU General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU General Public License
> > + * along with this program; if not, write the Free Software Foundation,
> > + * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > + */
> > +
> > +#include "libxfs.h"
> > +#include "ondisk.h"
> > +
> > +void
> > +xfs_check_ondisk_structs(void)
> > +{
> > +	/* on-disk structures */
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dsb,			264);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_agf,			224);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_agi,			336);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_agfl,			36);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp,		8);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dinode,		176);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_disk_dquot,		104);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dqblk,			136);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dsymlink_hdr,		56);
> > +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_rec_t,			8);
> > +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_key_t,			8);
> > +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_ptr_t,			4);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_rec,		16);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_key,		4);
> > +	XFS_CHECK_STRUCT_SIZE(xfs_inobt_ptr_t,			4);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmdr_block,		4);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_rec,		16);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_key,		8);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_btree_block,		72);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_acl_entry,		12);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_acl,			4);
> > +
> > +	/* log structures */
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_trans_header,		16);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_32,	52);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_64,	56);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_ictimestamp,		8);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_icdinode,		176);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_extent_32,		12);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_extent_64,		16);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_32,	28);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
> > +}
> > diff --git a/db/ondisk.h b/db/ondisk.h
> > new file mode 100644
> > index 0000000..b5784d1
> > --- /dev/null
> > +++ b/db/ondisk.h
> > @@ -0,0 +1,139 @@
> > +/*
> > + * Copyright (c) 2016 Oracle.
> > + * All Rights Reserved.
> > + *
> > + * This program is free software; you can redistribute it and/or
> > + * modify it under the terms of the GNU General Public License as
> > + * published by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it would be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > + * GNU General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU General Public License
> > + * along with this program; if not, write the Free Software Foundation,
> > + * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > + */
> > +
> > +#ifndef __XFS_DB_ONDISK_H
> > +#define __XFS_DB_ONDISK_H
> > +
> > +/* Compile time object size, -1 for unknown */
> > +#ifndef __compiletime_error
> > +# define __compiletime_error(message)
> > +/*
> > + * Sparse complains of variable sized arrays due to the temporary variable in
> > + * __compiletime_assert. Unfortunately we can't just expand it out to make
> > + * sparse see a constant array size without breaking compiletime_assert on old
> > + * versions of GCC (e.g. 4.2.4), so hide the array from sparse altogether.
> > + */
> > +# ifndef __CHECKER__
> > +#  define __compiletime_error_fallback(condition) \
> > +	do { ((void)sizeof(char[1 - 2 * condition])); } while (0)
> > +# endif
> > +#endif
> > +#ifndef __compiletime_error_fallback
> > +# define __compiletime_error_fallback(condition) do { } while (0)
> > +#endif
> > +
> > +#define __compiletime_assert(condition, msg, prefix, suffix)		\
> > +	do {								\
> > +		bool __cond = !(condition);				\
> > +		extern void prefix ## suffix(void) __compiletime_error(msg); \
> > +		if (__cond)						\
> > +			prefix ## suffix();				\
> > +		__compiletime_error_fallback(__cond);			\
> > +	} while (0)
> > +
> > +#define _compiletime_assert(condition, msg, prefix, suffix) \
> > +	__compiletime_assert(condition, msg, prefix, suffix)
> > +
> > +/**
> > + * compiletime_assert - break build and emit msg if condition is false
> > + * @condition: a compile-time constant condition to check
> > + * @msg:       a message to emit if condition is false
> > + *
> > + * In tradition of POSIX assert, this macro will break the build if the
> > + * supplied condition is *false*, emitting the supplied error message if the
> > + * compiler has support to do so.
> > + */
> > +#define compiletime_assert(condition, msg) \
> > +	_compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
> > +
> > +#ifdef __CHECKER__
> > +#define BUILD_BUG_ON_NOT_POWER_OF_2(n) (0)
> > +#define BUILD_BUG_ON_ZERO(e) (0)
> > +#define BUILD_BUG_ON_NULL(e) ((void*)0)
> > +#define BUILD_BUG_ON_INVALID(e) (0)
> > +#define BUILD_BUG_ON_MSG(cond, msg) (0)
> > +#define BUILD_BUG_ON(condition) (0)
> > +#define BUILD_BUG() (0)
> > +#else /* __CHECKER__ */
> > +
> > +/* Force a compilation error if a constant expression is not a power of 2 */
> > +#define BUILD_BUG_ON_NOT_POWER_OF_2(n)			\
> > +	BUILD_BUG_ON((n) == 0 || (((n) & ((n) - 1)) != 0))
> > +
> > +/* Force a compilation error if condition is true, but also produce a
> > +   result (of value 0 and type size_t), so the expression can be used
> > +   e.g. in a structure initializer (or where-ever else comma expressions
> > +   aren't permitted). */
> > +#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
> > +#define BUILD_BUG_ON_NULL(e) ((void *)sizeof(struct { int:-!!(e); }))
> > +
> > +/*
> > + * BUILD_BUG_ON_INVALID() permits the compiler to check the validity of the
> > + * expression but avoids the generation of any code, even if that expression
> > + * has side-effects.
> > + */
> > +#define BUILD_BUG_ON_INVALID(e) ((void)(sizeof((__force long)(e))))
> > +
> > +/**
> > + * BUILD_BUG_ON_MSG - break compile if a condition is true & emit supplied
> > + *		      error message.
> > + * @condition: the condition which the compiler should know is false.
> > + *
> > + * See BUILD_BUG_ON for description.
> > + */
> > +#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
> > +
> > +/**
> > + * BUILD_BUG_ON - break compile if a condition is true.
> > + * @condition: the condition which the compiler should know is false.
> > + *
> > + * If you have some code which relies on certain constants being equal, or
> > + * some other compile-time-evaluated condition, you should use BUILD_BUG_ON to
> > + * detect if someone changes it.
> > + *
> > + * The implementation uses gcc's reluctance to create a negative array, but gcc
> > + * (as of 4.4) only emits that error for obvious cases (e.g. not arguments to
> > + * inline functions).  Luckily, in 4.3 they added the "error" function
> > + * attribute just for this type of case.  Thus, we use a negative sized array
> > + * (should always create an error on gcc versions older than 4.4) and then call
> > + * an undefined function with the error attribute (should always create an
> > + * error on gcc 4.3 and later).  If for some reason, neither creates a
> > + * compile-time error, we'll still have a link-time error, which is harder to
> > + * track down.
> > + */
> > +#ifndef __OPTIMIZE__
> > +#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)]))
> > +#else
> > +#define BUILD_BUG_ON(condition) \
> > +	BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
> > +#endif
> > +
> > +/**
> > + * BUILD_BUG - break compile if used.
> > + *
> > + * If you have some code that you expect the compiler to eliminate at
> > + * build time, you should use BUILD_BUG to detect if it is
> > + * unexpectedly used.
> > + */
> > +#define BUILD_BUG() BUILD_BUG_ON_MSG(1, "BUILD_BUG failed")
> > +
> > +#endif	/* __CHECKER__ */
> > +
> > +void xfs_check_ondisk_structs(void);
> > +
> > +#endif /* __XFS_DB_ONDISK_H */
> > diff --git a/libxfs/xfs_format.h b/libxfs/xfs_format.h
> > index a35009a..c4c0c1c 100644
> > --- a/libxfs/xfs_format.h
> > +++ b/libxfs/xfs_format.h
> > @@ -1505,4 +1505,8 @@ struct xfs_acl {
> >  #define SGI_ACL_FILE_SIZE	(sizeof(SGI_ACL_FILE)-1)
> >  #define SGI_ACL_DEFAULT_SIZE	(sizeof(SGI_ACL_DEFAULT)-1)
> >  
> > +#define XFS_CHECK_STRUCT_SIZE(structname, size) \
> > +	BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(struct " \
> > +		#structname ") is wrong, expected " #size)
> > +
> >  #endif /* __XFS_FORMAT_H__ */
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs

From ebiggers3@gmail.com  Tue Jan 12 20:37:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.3 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,T_DKIM_INVALID autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id A16E129DF5
	for ; Tue, 12 Jan 2016 20:37:05 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 908FD304062
	for ; Tue, 12 Jan 2016 18:37:02 -0800 (PST)
X-ASG-Debug-ID: 1452652620-04cb6c75deb3e70001-NocioJ
Received: from mail-io0-f194.google.com (mail-io0-f194.google.com [209.85.223.194]) by cuda.sgi.com with ESMTP id X6Hg7HUg0PCBMoRp (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 18:37:01 -0800 (PST)
X-Barracuda-Envelope-From: ebiggers3@gmail.com
X-Barracuda-RBL-Trusted-Forwarder: 209.85.223.194
Received: by mail-io0-f194.google.com with SMTP id k127so46384723iok.1
        for ; Tue, 12 Jan 2016 18:37:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=date:from:to:cc:subject:message-id:references:mime-version
         :content-type:content-disposition:in-reply-to:user-agent;
        bh=ZJzjZilyj5CBwshqEPCn8u2N6z1StQNtDowzpiiwvkc=;
        b=ufWkyoeYd/AVklWyI+N9ggnQ52ukYwEOXBvEqaV8EYSPhCGt+2pbsmbkK9UL+7aYdk
         Lc/pKIgHJcBXGmvkpVu0iAv3u/BKaA1/r6aJUNc+oTnDwhMEftNu0HGkh18kR5GHOC2U
         twhp6OGC45m/iLE7MgqI4xKpiPOxHL5vZdhO1Qrd5LhiBLilE4MDY8zmymv9KtnKZMPd
         FRocUxxVM7zZMMiBeHwzP7d4oSTCoYHo7LefX6dPTcotvQHB98FpuTUYsraY8ZF34ghP
         epgi+6Q0+WFnHhaUGAOkAfkvWHCPBjKbYs4vRZsC3wMIKaQ43LGVtRlwh8eFIqYReoeu
         X0Eg==
X-Received: by 10.50.142.7 with SMTP id rs7mr21588319igb.90.1452652620703;
        Tue, 12 Jan 2016 18:37:00 -0800 (PST)
Received: from zzz (c-24-7-245-123.hsd1.mn.comcast.net. [24.7.245.123])
        by smtp.gmail.com with ESMTPSA id ax5sm242024igc.13.2016.01.12.18.36.59
        (version=TLSv1/SSLv3 cipher=OTHER);
        Tue, 12 Jan 2016 18:37:00 -0800 (PST)
Date: Tue, 12 Jan 2016 20:36:58 -0600
X-Barracuda-Apparent-Source-IP: 24.7.245.123
X-Barracuda-BBL-IP: 24.7.245.123
From: Eric Biggers 
To: "Darrick J. Wong" 
Cc: david@fromorbit.com, linux-fsdevel@vger.kernel.org,
	linux-api@vger.kernel.org, xfs@oss.sgi.com,
	linux-btrfs@vger.kernel.org, hch@infradead.org
Subject: Re: [PATCH 8/9] vfs: hoist the btrfs deduplication ioctl to the vfs
Message-ID: <20160113023658.GA1551@zzz>
X-ASG-Orig-Subj: Re: [PATCH 8/9] vfs: hoist the btrfs deduplication ioctl to the vfs
References: <20151219085505.12478.71157.stgit@birch.djwong.org>
 <20151219085559.12478.33700.stgit@birch.djwong.org>
 <20160112060714.GA4980@zzz>
 <20160112091432.GB7832@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112091432.GB7832@birch.djwong.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mail-io0-f194.google.com[209.85.223.194]
X-Barracuda-Start-Time: 1452652621
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26076
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

On Tue, Jan 12, 2016 at 01:14:32AM -0800, Darrick J. Wong wrote:
> 
> Frankly, I also wouldn't mind changing the VFS dedupe ioctl that to something
> that resembles the clone_range interface:
> 
> int ioctl(int dest_fd, FIDEDUPERANGE, struct file_dedupe_range * arg);

Getting rid of the vectorization certainly avoids a lot of API complexity.  I
don't know precisely why it was made vectorized in the first place, but to me it
seems it shouldn't be vectorized unless there's a strong reason for it to be.

> Not sure what you mean by 'symmetric', but in any case the convention seems
> to be that src_fd's storage is shared with dest_fd if there's a match.
> ...
> I think the result of an earlier discussion was that src_fd must be readable,
> and dest_fd must be writable or appendable.

Deduplication is a strange operation since you're not "reading" or "writing" in
the traditional sense.  All file data stays exactly the same from a user
perspective.

So rather than arbitrarily say that you're "reading" from one file and "writing"
to the other, one possibility is to have an API where you submit two files and
the implementation decides how to best deduplicate them.  That could involve
sharing existing extents in file 1 with file 2; or sharing existing extents in
file 2 with file 1; or creating a new extent and referencing it from each file.
It would be up to the implementation to choose the most efficient approach.

But I take it that the existing btrfs ioctl doesn't work that way, and it always
will share the existing extents in file 1 with file 2.

That's probably a perfectly fine way to do it, but I was wondering whether it
really makes sense to hard-code that detail in the API, since an API could be
defined more generally.

And from a practical perpective, people using the ioctl to deduplicate two files
need to know which one to submit to the API as the "source" and which as the
"destination", if it matters.

Related to this are the permissions you need on each file descriptor.  Since
deduplication isn't "reading" or "writing" in the traditional sense it could,
theoretically, be argued that no permissions at all should be required: neither
FMODE_READ nor FMODE_WRITE.

Most likely are concerns that would make that a bad idea, though.  Perhaps
someone could create mischief by causing fragmentation in system files.

If this was previously discussed, can you point me to that discussion?

The proposed man page is also very brief on permissions, only mentioning them
indirectly in the error codes section.

> > The man page also says you get EPERM if "dest_fd is immutable" and ETXTBSY if
> > "one of the files is a swap file", which I don't see actually happening in
> > the implementation; it seems those error codes perhaps exist at all for this
> > ioctl but rather be left to open(..., O_WRONLY).
> 
> Those could be hoisted to the VFS (from the XFS implementation), I think.

If the destination file has to already be open for writing then it can't be
immutable.  So I don't think that error code is needed.  Checking for an active
swapfile, on the other hand, may be valid, although I don't see such a check
anywhere in the version of the code I'm looking at.

> I'll later take a look at how many of these issues apply to clone/clone_range.

clone and clone_range seem to avoid the two biggest issues I saw: they aren't
vectorized, and there's a natural distinction between the "source" and the
"destination" of the operation.  They definitely need careful consideration as
well, though.

From sandeen@sandeen.net  Tue Jan 12 21:10:32 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id D5EAE29DF5
	for ; Tue, 12 Jan 2016 21:10:32 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 82169AC002
	for ; Tue, 12 Jan 2016 19:10:29 -0800 (PST)
X-ASG-Debug-ID: 1452654624-04bdf06f6bbbfd0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 3N26QUUbojEDJCUs for ; Tue, 12 Jan 2016 19:10:25 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 8AC8861AC50F
	for ; Tue, 12 Jan 2016 21:10:24 -0600 (CST)
Subject: Re: [PATCH] xfs_db: check on-disk structure sizes
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] xfs_db: check on-disk structure sizes
References: <20160111234644.GB7831@birch.djwong.org>
 <20160112140122.GC12156@bfoster.bfoster>
 <20160113012945.GC2455@birch.djwong.org>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <5695C01F.7030807@sandeen.net>
Date: Tue, 12 Jan 2016 21:10:23 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160113012945.GC2455@birch.djwong.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1452654625
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26076
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------



On 1/12/16 7:29 PM, Darrick J. Wong wrote:
> On Tue, Jan 12, 2016 at 09:01:22AM -0500, Brian Foster wrote:
>> On Mon, Jan 11, 2016 at 03:46:44PM -0800, Darrick J. Wong wrote:
>>> Check on-disk structure sizes against known values.
>>> Use this to catch inadvertent changes in structure size due to padding
>>> and alignment issues, etc.
>>>
>>> Signed-off-by: Darrick J. Wong 
>>> ---
>>
>> What's the need for this in userspace? Not a big deal really, but it
>> seems like it serves the fundamental purpose sufficiently in the kernel.
> 
> The primary point is to make sure that we didn't make any errors with the
> on-disk structures when porting libxfs changes.  The kernel build is the first
> line of defense since it tends to big get changes first, but I figure a
> defensive build check for xfsprogs won't harm anyone...

Does it need to actually be in the code?

$ pahole -s fs/xfs/xfs.ko | grep -w "xfs_dsb\|xfs_agf\|xfs_agi\|xfs_agfl"
xfs_agf	224	0
xfs_agfl	40	0
xfs_agi	336	0
xfs_dsb	264	0

pahole needs a binary w/ debuginfo, but maybe this could just be hooked up
in the Makefiles?

-Eric

From dave@fromorbit.com  Tue Jan 12 23:12:56 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 6728D7F37
	for ; Tue, 12 Jan 2016 23:12:56 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 49B058F8040
	for ; Tue, 12 Jan 2016 21:12:53 -0800 (PST)
X-ASG-Debug-ID: 1452661970-04bdf06f6dbf9c0001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id OGVj8IlwwwNbt6HZ for ; Tue, 12 Jan 2016 21:12:50 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DaCwA63JVWPGu7LXleKAECgw+BP4ZggXmfagEBAQEBAQaRG4s8TQEBAQEBAQcBAQEBQT+EYi8jGIECAwctiC3AEYYOiWt/hBoFh1uHGoggnFgCRI4Qgi4BAQgBAQEBgkYqNIRRBB+BKAEBAQ
Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107])
  by ipmail04.adl6.internode.on.net with ESMTP; 13 Jan 2016 15:42:49 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJDjk-0007sc-Fr; Wed, 13 Jan 2016 16:12:48 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJDjk-0002zs-Er; Wed, 13 Jan 2016 16:12:48 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Cc: jkosina@suse.cz
Subject: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Date: Wed, 13 Jan 2016 16:12:48 +1100
X-ASG-Orig-Subj: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Message-Id: <1452661968-11482-1-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1452661970
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26078
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

This reverts commit 24ba16bb3d499c49974669cd8429c3e4138ab102 as it
prevents machines from suspending. This regression occurs when the
xfsaild is idle on entry to suspend, and so there s no activity to
wake it from it's idle sleep and hence see that it is supposed to
freeze. Hence the freezer times out waiting for it and suspend is
cancelled.

There is no obvious fix for this short of freezing the filesystem
properly, so revert this change for now.

Signed-off-by: Dave Chinner 
---
 fs/xfs/xfs_trans_ail.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
index aa67339..4f18fd9 100644
--- a/fs/xfs/xfs_trans_ail.c
+++ b/fs/xfs/xfs_trans_ail.c
@@ -497,7 +497,6 @@ xfsaild(
 	long		tout = 0;	/* milliseconds */
 
 	current->flags |= PF_MEMALLOC;
-	set_freezable();
 
 	while (!kthread_should_stop()) {
 		if (tout && tout <= 20)
-- 
2.5.0

From david@fromorbit.com  Tue Jan 12 23:47:35 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 57A9B7F37
	for ; Tue, 12 Jan 2016 23:47:35 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 1B769304059
	for ; Tue, 12 Jan 2016 21:47:35 -0800 (PST)
X-ASG-Debug-ID: 1452664052-04cb6c75dcb8480001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id WQQAxyTqWzKST9Ny for ; Tue, 12 Jan 2016 21:47:32 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DJEABy5JVWPGu7LXleKAECgw+BP4Jig36BeZ9qAQEBAQEBBotdhT6EBYYJAgIBAQKBJk0BAQEBAQEHAQEBAUE/QRIBg2EBAQQ6HCMQCAMYCSUPBSUDBxoTiC2/cgEBCAIBIBmFdYVHiT0FlxWNUYFnjSJEhSKIboUCKjSEFYIHAQEB
Received: from ppp121-45-187-107.lns20.syd7.internode.on.net (HELO dastard) ([121.45.187.107])
  by ipmail04.adl6.internode.on.net with ESMTP; 13 Jan 2016 16:17:31 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aJEHK-0007vi-TV; Wed, 13 Jan 2016 16:47:30 +1100
Date: Wed, 13 Jan 2016 16:47:30 +1100
From: Dave Chinner 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs_db: check on-disk structure sizes
Message-ID: <20160113054730.GQ10456@dastard>
X-ASG-Orig-Subj: Re: [PATCH] xfs_db: check on-disk structure sizes
References: <20160111234644.GB7831@birch.djwong.org>
 <20160112140122.GC12156@bfoster.bfoster>
 <20160113012945.GC2455@birch.djwong.org>
 <5695C01F.7030807@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5695C01F.7030807@sandeen.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1452664052
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26079
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue, Jan 12, 2016 at 09:10:23PM -0600, Eric Sandeen wrote:
> 
> 
> On 1/12/16 7:29 PM, Darrick J. Wong wrote:
> > On Tue, Jan 12, 2016 at 09:01:22AM -0500, Brian Foster wrote:
> >> On Mon, Jan 11, 2016 at 03:46:44PM -0800, Darrick J. Wong wrote:
> >>> Check on-disk structure sizes against known values.
> >>> Use this to catch inadvertent changes in structure size due to padding
> >>> and alignment issues, etc.
> >>>
> >>> Signed-off-by: Darrick J. Wong 
> >>> ---
> >>
> >> What's the need for this in userspace? Not a big deal really, but it
> >> seems like it serves the fundamental purpose sufficiently in the kernel.
> > 
> > The primary point is to make sure that we didn't make any errors with the
> > on-disk structures when porting libxfs changes.  The kernel build is the first
> > line of defense since it tends to big get changes first, but I figure a
> > defensive build check for xfsprogs won't harm anyone...
> 
> Does it need to actually be in the code?
> 
> $ pahole -s fs/xfs/xfs.ko | grep -w "xfs_dsb\|xfs_agf\|xfs_agi\|xfs_agfl"
> xfs_agf	224	0
> xfs_agfl	40	0
> xfs_agi	336	0
> xfs_dsb	264	0
> 
> pahole needs a binary w/ debuginfo, but maybe this could just be hooked up
> in the Makefiles?

As I've pointed out previously to Darrick: xfstests:/tests/xfs/122

Make that build again, update it.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

From sandeen@sandeen.net  Wed Jan 13 00:03:20 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=MIME_QP_LONG_LINE
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 5F3537F37
	for ; Wed, 13 Jan 2016 00:03:20 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 340B78F8040
	for ; Tue, 12 Jan 2016 22:03:17 -0800 (PST)
X-ASG-Debug-ID: 1452664990-04bdf06f6ec0e40001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id KH9YlBk4P6Ma4eSy for ; Tue, 12 Jan 2016 22:03:10 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from [10.0.0.194] (unknown [10.0.0.194])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id CACF963737DF;
	Wed, 13 Jan 2016 00:03:07 -0600 (CST)
Content-Type: text/plain;
	charset=us-ascii
Mime-Version: 1.0 (1.0)
Subject: Re: [PATCH] xfs_db: check on-disk structure sizes
From: Eric Sandeen 
X-ASG-Orig-Subj: Re: [PATCH] xfs_db: check on-disk structure sizes
X-Mailer: iPhone Mail (13C75)
In-Reply-To: <20160113054730.GQ10456@dastard>
Date: Wed, 13 Jan 2016 00:02:36 -0600
Cc: xfs@oss.sgi.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <10021348-43A2-465E-BC00-E0A0882FFD22@sandeen.net>
References: <20160111234644.GB7831@birch.djwong.org> <20160112140122.GC12156@bfoster.bfoster> <20160113012945.GC2455@birch.djwong.org> <5695C01F.7030807@sandeen.net> <20160113054730.GQ10456@dastard>
To: Dave Chinner 
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1452664990
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, MIME_QP_LONG_LINE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26079
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 MIME_QP_LONG_LINE      RAW: Quoted-printable line longer than 76 chars

LOn Jan 12, 2016, at 11:47 PM, Dave Chinner  wrote:
>=20
>> On Tue, Jan 12, 2016 at 09:10:23PM -0600, Eric Sandeen wrote:
>>=20
>>=20
>>> On 1/12/16 7:29 PM, Darrick J. Wong wrote:
>>>> On Tue, Jan 12, 2016 at 09:01:22AM -0500, Brian Foster wrote:
>>>>> On Mon, Jan 11, 2016 at 03:46:44PM -0800, Darrick J. Wong wrote:
>>>>> Check on-disk structure sizes against known values.
>>>>> Use this to catch inadvertent changes in structure size due to padding=

>>>>> and alignment issues, etc.
>>>>>=20
>>>>> Signed-off-by: Darrick J. Wong 
>>>>> ---
>>>>=20
>>>> What's the need for this in userspace? Not a big deal really, but it
>>>> seems like it serves the fundamental purpose sufficiently in the kernel=
.
>>>=20
>>> The primary point is to make sure that we didn't make any errors with th=
e
>>> on-disk structures when porting libxfs changes.  The kernel build is the=
 first
>>> line of defense since it tends to big get changes first, but I figure a
>>> defensive build check for xfsprogs won't harm anyone...
>>=20
>> Does it need to actually be in the code?
>>=20
>> $ pahole -s fs/xfs/xfs.ko | grep -w "xfs_dsb\|xfs_agf\|xfs_agi\|xfs_agfl"=

>> xfs_agf    224    0
>> xfs_agfl    40    0
>> xfs_agi    336    0
>> xfs_dsb    264    0
>>=20
>> pahole needs a binary w/ debuginfo, but maybe this could just be hooked u=
p
>> in the Makefiles?
>=20
> As I've pointed out previously to Darrick: xfstests:/tests/xfs/122
>=20
> Make that build again, update it.
>=20
Oh, I went looking for that and missed it somehow, thought it had been remov=
ed.   Ok then!

-Eric

> -Dave.
> --=20
> Dave Chinner
> david@fromorbit.com
>=20

From ross.zwisler@linux.intel.com  Wed Jan 13 01:08:36 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 638277F37
	for ; Wed, 13 Jan 2016 01:08:36 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 2C0A08F8054
	for ; Tue, 12 Jan 2016 23:08:36 -0800 (PST)
X-ASG-Debug-ID: 1452668911-04bdf06f6ec26a0001-NocioJ
Received: from mga01.intel.com ([192.55.52.88]) by cuda.sgi.com with ESMTP id ROgHYDcUBaNtutUw for ; Tue, 12 Jan 2016 23:08:31 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.88
X-ASG-Whitelist: EmailCat (corporate)
Received: from orsmga002.jf.intel.com ([10.7.209.21])
  by fmsmga101.fm.intel.com with ESMTP; 12 Jan 2016 23:08:32 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,287,1449561600"; 
   d="scan'208";a="889460697"
Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.206.118])
  by orsmga002.jf.intel.com with ESMTP; 12 Jan 2016 23:08:30 -0800
Date: Wed, 13 Jan 2016 00:08:29 -0700
From: Ross Zwisler 
To: Jan Kara 
Cc: Ross Zwisler ,
	linux-kernel@vger.kernel.org, "H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 1/9] dax: fix NULL pointer dereference in __dax_dbg()
Message-ID: <20160113070829.GA30496@linux.intel.com>
X-ASG-Orig-Subj: Re: [PATCH v8 1/9] dax: fix NULL pointer dereference in __dax_dbg()
Mail-Followup-To: Ross Zwisler ,
	Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-2-git-send-email-ross.zwisler@linux.intel.com>
 <20160112093458.GR6262@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112093458.GR6262@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.88]
X-Barracuda-Start-Time: 1452668911
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Tue, Jan 12, 2016 at 10:34:58AM +0100, Jan Kara wrote:
> On Thu 07-01-16 22:27:51, Ross Zwisler wrote:
> > In __dax_pmd_fault() we currently assume that get_block() will always set
> > bh.b_bdev and we unconditionally dereference it in __dax_dbg().  This
> > assumption isn't always true - when called for reads of holes
> > ext4_dax_mmap_get_block() returns a buffer head where bh->b_bdev is never
> > set.  I hit this BUG while testing the DAX PMD fault path.
> > 
> > Instead, initialize bh.b_bdev before passing bh into get_block().  It is
> > possible that the filesystem's get_block() will update bh.b_bdev, and this
> > is fine - we just want to initialize bh.b_bdev to something reasonable so
> > that the calls to __dax_dbg() work and print something useful.
> > 
> > Signed-off-by: Ross Zwisler 
> > Cc: Dan Williams 
> 
> Looks good. But don't you need to do the same for __dax_fault(),
> dax_zero_page_range() and similar places passing bh to dax functions?
> 
> 								Honza

I don't think we need it anywhere else.  The only reason that we need to
initialize the bh.b_bdev manually in the __dax_pmd_fault() path is that if the
get_block() call ends up finding a hole (so doesn't fill out b_bdev) we still
go through the dax_pmd_dbg() path to print an error message, which uses
b_bdev.  I believe that in the other paths where we hit a hole, such as
__dax_fault(), we don't use b_bdev because we don't have the same error path
prints, and the regular code for handling holes doesn't use b_bdev.

That being said, if you feel like it's cleaner to initialize it everywhere so
everything is consistent and we don't have to worry about it, I'm fine to make
the change.

> > ---
> >  fs/dax.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/fs/dax.c b/fs/dax.c
> > index 7af8797..513bba5 100644
> > --- a/fs/dax.c
> > +++ b/fs/dax.c
> > @@ -624,6 +624,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
> >  	}
> >  
> >  	memset(&bh, 0, sizeof(bh));
> > +	bh.b_bdev = inode->i_sb->s_bdev;
> >  	block = (sector_t)pgoff << (PAGE_SHIFT - blkbits);
> >  
> >  	bh.b_size = PMD_SIZE;
> > -- 
> > 2.5.0
> > 
> > 
> -- 
> Jan Kara 
> SUSE Labs, CR

From ross.zwisler@linux.intel.com  Wed Jan 13 01:30:34 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 5F9577F37
	for ; Wed, 13 Jan 2016 01:30:34 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id F2B61AC001
	for ; Tue, 12 Jan 2016 23:30:30 -0800 (PST)
X-ASG-Debug-ID: 1452670228-04bdf06f6ec2d80001-NocioJ
Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by cuda.sgi.com with ESMTP id CeADhvTUG18giXmz for ; Tue, 12 Jan 2016 23:30:28 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 134.134.136.24
X-ASG-Whitelist: EmailCat (corporate)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
  by orsmga102.jf.intel.com with ESMTP; 12 Jan 2016 23:30:27 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,288,1449561600"; 
   d="scan'208";a="880327780"
Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.206.118])
  by fmsmga001.fm.intel.com with ESMTP; 12 Jan 2016 23:30:20 -0800
Date: Wed, 13 Jan 2016 00:30:19 -0700
From: Ross Zwisler 
To: Jan Kara 
Cc: Ross Zwisler ,
	linux-kernel@vger.kernel.org, "H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 6/9] dax: add support for fsync/msync
Message-ID: <20160113073019.GB30496@linux.intel.com>
X-ASG-Orig-Subj: Re: [PATCH v8 6/9] dax: add support for fsync/msync
Mail-Followup-To: Ross Zwisler ,
	Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-7-git-send-email-ross.zwisler@linux.intel.com>
 <20160112105716.GT6262@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112105716.GT6262@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mga09.intel.com[134.134.136.24]
X-Barracuda-Start-Time: 1452670228
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Tue, Jan 12, 2016 at 11:57:16AM +0100, Jan Kara wrote:
> On Thu 07-01-16 22:27:56, Ross Zwisler wrote:
> > To properly handle fsync/msync in an efficient way DAX needs to track dirty
> > pages so it is able to flush them durably to media on demand.
> > 
> > The tracking of dirty pages is done via the radix tree in struct
> > address_space.  This radix tree is already used by the page writeback
> > infrastructure for tracking dirty pages associated with an open file, and
> > it already has support for exceptional (non struct page*) entries.  We
> > build upon these features to add exceptional entries to the radix tree for
> > DAX dirty PMD or PTE pages at fault time.
> > 
> > Signed-off-by: Ross Zwisler 
> 
> Some comments below.
> 
> > ---
> >  fs/dax.c            | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++--
> >  include/linux/dax.h |   2 +
> >  mm/filemap.c        |   6 ++
> >  3 files changed, 196 insertions(+), 6 deletions(-)
> > 
> > diff --git a/fs/dax.c b/fs/dax.c
> > index 5b84a46..0db21ea 100644
> > --- a/fs/dax.c
> > +++ b/fs/dax.c
> > @@ -24,6 +24,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  #include 
> >  #include 
> >  #include 
> > @@ -324,6 +325,174 @@ static int copy_user_bh(struct page *to, struct inode *inode,
> >  	return 0;
> >  }
> >  
> > +#define NO_SECTOR -1
> > +
> > +static int dax_radix_entry(struct address_space *mapping, pgoff_t index,
> > +		sector_t sector, bool pmd_entry, bool dirty)
> > +{
> > +	struct radix_tree_root *page_tree = &mapping->page_tree;
> > +	int type, error = 0;
> > +	void *entry;
> > +
> > +	__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
> > +
> > +	spin_lock_irq(&mapping->tree_lock);
> > +	entry = radix_tree_lookup(page_tree, index);
> > +
> > +	if (entry) {
> > +		type = RADIX_DAX_TYPE(entry);
> > +		if (WARN_ON_ONCE(type != RADIX_DAX_PTE &&
> > +					type != RADIX_DAX_PMD)) {
> > +			error = -EIO;
> > +			goto unlock;
> > +		}
> > +
> > +		if (!pmd_entry || type == RADIX_DAX_PMD)
> > +			goto dirty;
> > +		radix_tree_delete(&mapping->page_tree, index);
> > +		mapping->nrexceptional--;
> 
> In theory, you can delete here DIRTY / TOWRITE PTE entry and insert a clean
> PMD entry instead of it. That will cause fsync() to miss some flushes. So
> you should make sure you transfer all the tags to the new entry.

Ah, great catch, I'll address it in v9 which I'll send out tomorrow.

> > +static int dax_writeback_one(struct block_device *bdev,
> > +		struct address_space *mapping, pgoff_t index, void *entry)
> > +{
> > +	struct radix_tree_root *page_tree = &mapping->page_tree;
> > +	int type = RADIX_DAX_TYPE(entry);
> > +	struct radix_tree_node *node;
> > +	struct blk_dax_ctl dax;
> > +	void **slot;
> > +	int ret = 0;
> > +
> > +	spin_lock_irq(&mapping->tree_lock);
> > +	/*
> > +	 * Regular page slots are stabilized by the page lock even
> > +	 * without the tree itself locked.  These unlocked entries
> > +	 * need verification under the tree lock.
> > +	 */
> > +	if (!__radix_tree_lookup(page_tree, index, &node, &slot))
> > +		goto unlock;
> > +	if (*slot != entry)
> > +		goto unlock;
> > +
> > +	/* another fsync thread may have already written back this entry */
> > +	if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE))
> > +		goto unlock;
> > +
> > +	radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE);
> > +
> > +	if (WARN_ON_ONCE(type != RADIX_DAX_PTE && type != RADIX_DAX_PMD)) {
> > +		ret = -EIO;
> > +		goto unlock;
> > +	}
> > +
> > +	dax.sector = RADIX_DAX_SECTOR(entry);
> > +	dax.size = (type == RADIX_DAX_PMD ? PMD_SIZE : PAGE_SIZE);
> > +	spin_unlock_irq(&mapping->tree_lock);
> 
> This seems to be somewhat racy as well - if there are two fsyncs running
> against the same inode, one wins the race and clears TOWRITE tag, the
> second then bails out and may finish before the skipped page gets flushed.
> 
> So we should clear the TOWRITE tag only after the range is flushed.  This
> can result in some amount of duplicit flushing but I don't think the race
> will happen that frequently in practice to be performance relevant.

Yep, this make sense.  I'll also fix that in v9.

> And secondly: You must write-protect all mappings of the flushed range so
> that you get fault when the sector gets written-to again. We spoke about
> this in the past already but somehow it got lost and I forgot about it as
> well. You need something like rmap_walk_file()...

The code that write protected mappings and then cleaned the radix tree entries
did get written, and was part of v2:

https://lkml.org/lkml/2015/11/13/759

I removed all the code that cleaned PTE entries and radix tree entries for v3.
The reason behind this was that there was a race that I couldn't figure out
how to solve between the cleaning of the PTEs and the cleaning of the radix
tree entries.

The race goes like this:

Thread 1 (write)			Thread 2 (fsync)
================			================
wp_pfn_shared()
pfn_mkwrite()
dax_radix_entry()
radix_tree_tag_set(DIRTY)
					dax_writeback_mapping_range()
					dax_writeback_one()
					radix_tag_clear(DIRTY)
					pgoff_mkclean()
... return up to wp_pfn_shared()
wp_page_reuse()
pte_mkdirty()

After this sequence we end up with a dirty PTE that is writeable, but with a
clean radix tree entry.  This means that users can write to the page, but that
a follow-up fsync or msync won't flush this dirty data to media.

The overall issue is that in the write path that goes through wp_pfn_shared(),
the DAX code has control over when the radix tree entry is dirtied but not
when the PTE is made dirty and writeable.  This happens up in wp_page_reuse().
This means that we can't easily add locking, etc. to protect ourselves.

I spoke a bit about this with Dave Chinner and with Dave Hansen, but no really
easy solutions presented themselves in the absence of a page lock.  I do have
one idea, but I think it's pretty invasive and will need to wait for another
kernel cycle.

The current code that leaves the radix tree entry will give us correct
behavior - it'll just be less efficient because we will have an ever-growing
dirty set to flush.

> > +	/*
> > +	 * We cannot hold tree_lock while calling dax_map_atomic() because it
> > +	 * eventually calls cond_resched().
> > +	 */
> > +	ret = dax_map_atomic(bdev, &dax);
> > +	if (ret < 0)
> > +		return ret;
> > +
> > +	if (WARN_ON_ONCE(ret < dax.size)) {
> > +		ret = -EIO;
> > +		goto unmap;
> > +	}
> > +
> > +	wb_cache_pmem(dax.addr, dax.size);
> > + unmap:
> > +	dax_unmap_atomic(bdev, &dax);
> > +	return ret;
> > +
> > + unlock:
> > +	spin_unlock_irq(&mapping->tree_lock);
> > +	return ret;
> > +}
> 
> ...
> 
> > @@ -791,15 +976,12 @@ EXPORT_SYMBOL_GPL(dax_pmd_fault);
> >   * dax_pfn_mkwrite - handle first write to DAX page
> >   * @vma: The virtual memory area where the fault occurred
> >   * @vmf: The description of the fault
> > - *
> >   */
> >  int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
> >  {
> > -	struct super_block *sb = file_inode(vma->vm_file)->i_sb;
> > +	struct file *file = vma->vm_file;
> >  
> > -	sb_start_pagefault(sb);
> > -	file_update_time(vma->vm_file);
> > -	sb_end_pagefault(sb);
> > +	dax_radix_entry(file->f_mapping, vmf->pgoff, NO_SECTOR, false, true);
> 
> Why is NO_SECTOR argument correct here?

Right - so NO_SECTOR means "I expect there to already be an entry in the radix
tree - just make that entry dirty".  This works because pfn_mkwrite() always
follows a normal __dax_fault() or __dax_pmd_fault() call.  These fault calls
will insert the radix tree entry, regardless of whether the fault was for a
read or a write.  If the fault was for a write, the radix tree entry will also
be made dirty.

For reads the radix tree entry will be inserted but left clean.  When the
first write happens we will get a pfn_mkwrite() call, which will call
dax_radix_entry() with the NO_SECTOR argument.  This will look up the radix
tree entry & set the dirty tag.

This also has the added benefit that the pfn_mkwrite() path can remain minimal
- if we needed to actually insert a radix tree entry with sector information
we'd have to duplicate a bunch of the fault path code so that we could call
get_block().

From ross.zwisler@linux.intel.com  Wed Jan 13 01:37:20 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 874457F37
	for ; Wed, 13 Jan 2016 01:37:20 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 5438A8F8040
	for ; Tue, 12 Jan 2016 23:37:20 -0800 (PST)
X-ASG-Debug-ID: 1452670638-04bdf06f6dc2fe0001-NocioJ
Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id fqKlGNFweetyZlgw for ; Tue, 12 Jan 2016 23:37:18 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 134.134.136.20
X-ASG-Whitelist: EmailCat (corporate)
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
  by orsmga101.jf.intel.com with ESMTP; 12 Jan 2016 23:37:18 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,288,1449561600"; 
   d="scan'208";a="892156104"
Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.206.118])
  by fmsmga002.fm.intel.com with ESMTP; 12 Jan 2016 23:37:16 -0800
Date: Wed, 13 Jan 2016 00:37:16 -0700
From: Ross Zwisler 
To: Jan Kara 
Cc: Ross Zwisler ,
	linux-kernel@vger.kernel.org, "H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 2/9] dax: fix conversion of holes to PMDs
Message-ID: <20160113073716.GC30496@linux.intel.com>
X-ASG-Orig-Subj: Re: [PATCH v8 2/9] dax: fix conversion of holes to PMDs
Mail-Followup-To: Ross Zwisler ,
	Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-3-git-send-email-ross.zwisler@linux.intel.com>
 <20160112094451.GS6262@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112094451.GS6262@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mga02.intel.com[134.134.136.20]
X-Barracuda-Start-Time: 1452670638
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Tue, Jan 12, 2016 at 10:44:51AM +0100, Jan Kara wrote:
> On Thu 07-01-16 22:27:52, Ross Zwisler wrote:
> > When we get a DAX PMD fault for a write it is possible that there could be
> > some number of 4k zero pages already present for the same range that were
> > inserted to service reads from a hole.  These 4k zero pages need to be
> > unmapped from the VMAs and removed from the struct address_space radix tree
> > before the real DAX PMD entry can be inserted.
> > 
> > For PTE faults this same use case also exists and is handled by a
> > combination of unmap_mapping_range() to unmap the VMAs and
> > delete_from_page_cache() to remove the page from the address_space radix
> > tree.
> > 
> > For PMD faults we do have a call to unmap_mapping_range() (protected by a
> > buffer_new() check), but nothing clears out the radix tree entry.  The
> > buffer_new() check is also incorrect as the current ext4 and XFS filesystem
> > code will never return a buffer_head with BH_New set, even when allocating
> > new blocks over a hole.  Instead the filesystem will zero the blocks
> > manually and return a buffer_head with only BH_Mapped set.
> > 
> > Fix this situation by removing the buffer_new() check and adding a call to
> > truncate_inode_pages_range() to clear out the radix tree entries before we
> > insert the DAX PMD.
> > 
> > Signed-off-by: Ross Zwisler 
> > Reported-by: Dan Williams 
> > Tested-by: Dan Williams 
> 
> Just two nits below. Nothing serious so you can add:
> 
> Reviewed-by: Jan Kara 

Cool, thank you for the review!

> > ---
> >  fs/dax.c | 20 ++++++++++----------
> >  1 file changed, 10 insertions(+), 10 deletions(-)
> > 
> > diff --git a/fs/dax.c b/fs/dax.c
> > index 513bba5..5b84a46 100644
> > --- a/fs/dax.c
> > +++ b/fs/dax.c
> > @@ -589,6 +589,7 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
> >  	bool write = flags & FAULT_FLAG_WRITE;
> >  	struct block_device *bdev;
> >  	pgoff_t size, pgoff;
> > +	loff_t lstart, lend;
> >  	sector_t block;
> >  	int result = 0;
> >  
> > @@ -643,15 +644,13 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
> >  		goto fallback;
> >  	}
> >  
> > -	/*
> > -	 * If we allocated new storage, make sure no process has any
> > -	 * zero pages covering this hole
> > -	 */
> > -	if (buffer_new(&bh)) {
> > -		i_mmap_unlock_read(mapping);
> > -		unmap_mapping_range(mapping, pgoff << PAGE_SHIFT, PMD_SIZE, 0);
> > -		i_mmap_lock_read(mapping);
> > -	}
> > +	/* make sure no process has any zero pages covering this hole */
> > +	lstart = pgoff << PAGE_SHIFT;
> > +	lend = lstart + PMD_SIZE - 1; /* inclusive */
> > +	i_mmap_unlock_read(mapping);
> 
> Just a nit but is there reason why we grab i_mmap_lock_read(mapping) only
> to release it a few lines below? The bh checks inside the locked region
> don't seem to rely on i_mmap_lock...

I think we can probably just take it when we're done with the truncate() -
I'll fix for v9.

> > +	unmap_mapping_range(mapping, lstart, PMD_SIZE, 0);
> > +	truncate_inode_pages_range(mapping, lstart, lend);
> 
> These two calls can be shortened as:
> 
> truncate_pagecache_range(inode, lstart, lend);

Nice.  I'll change it for v9.

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:46:44 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 04B137F37
	for ; Wed, 13 Jan 2016 01:46:44 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id E6AB630405F
	for ; Tue, 12 Jan 2016 23:46:43 -0800 (PST)
X-ASG-Debug-ID: 1452671201-04cbb05f7e151fd0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id bIc3SqUI3NbKzoLg (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:46:42 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJG8e-00078D-8v; Wed, 13 Jan 2016 07:46:40 +0000
Date: Tue, 12 Jan 2016 23:46:40 -0800
From: Christoph Hellwig 
To: Eric Sandeen 
Cc: Dave Chinner , xfs@oss.sgi.com
Subject: Re: [PATCH] xfs_db: check on-disk structure sizes
Message-ID: <20160113074640.GA21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH] xfs_db: check on-disk structure sizes
References: <20160111234644.GB7831@birch.djwong.org>
 <20160112140122.GC12156@bfoster.bfoster>
 <20160113012945.GC2455@birch.djwong.org>
 <5695C01F.7030807@sandeen.net>
 <20160113054730.GQ10456@dastard>
 <10021348-43A2-465E-BC00-E0A0882FFD22@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <10021348-43A2-465E-BC00-E0A0882FFD22@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671201
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Wed, Jan 13, 2016 at 12:02:36AM -0600, Eric Sandeen wrote:
> > As I've pointed out previously to Darrick: xfstests:/tests/xfs/122
> > 
> > Make that build again, update it.
> > 
> Oh, I went looking for that and missed it somehow, thought it had been removed.   Ok then!

The real issue is that build environment change alignments.  32 vs 64
bit builds are obvious, but on some architectures different ABIs have
different alignments (we had some fun with ARM in that regard),
nevermind the equivalents to IRIX n32 popping up everywhere these days
that make things complicated.

I'd really love to have Darrick's check in xfs_format.h as an opt-in
if a build time assert is provided - that way every user can check it
doesn't screw up the structures.

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:49:03 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id A611F7F37
	for ; Wed, 13 Jan 2016 01:49:03 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 924A1304053
	for ; Tue, 12 Jan 2016 23:49:03 -0800 (PST)
X-ASG-Debug-ID: 1452671341-04cb6c75debae60001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id A23bdTznK8am8iiU (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:49:02 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGAt-0007fn-OZ; Wed, 13 Jan 2016 07:48:59 +0000
Date: Tue, 12 Jan 2016 23:48:59 -0800
From: Christoph Hellwig 
To: Dave Chinner 
Cc: Felix Janda , xfs@oss.sgi.com
Subject: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
Message-ID: <20160113074859.GB21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
References: 
 <20160112195935.GB568@nyan>
 <20160112212405.GL10456@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112212405.GL10456@dastard>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671341
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Wed, Jan 13, 2016 at 08:24:06AM +1100, Dave Chinner wrote:
> I can't apply this straight off. Most of the libxfs code that is
> changed is shared with the kernel code, and so the definitions of
> the variables need to be the same as the kernel code. There are
> reasons for the kernel code using __[u]int*_t type variants (e.g.  I
> think the endian conversion static checker requires the __ variants
> for host order variables), and so before making sweeping changes
> like this we need to ensure that we can make the equivalent changes
> to the kernel code as well...

There is no functional requirement for these types.  I'm not sure if
they are an IRIXism, or something that was added during the early
Linux port, but they certainly aren't nessecary.

I'd love to kill them in favour or either the kernel __u/s types or the
C99 types which are like the urrent ones just without the __ prefix.

Maybe we should get an agreement which ones we want and ask Felix for
a patch to the kernel tree?

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:55:04 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 176C97F37
	for ; Wed, 13 Jan 2016 01:55:04 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E917D8F804B
	for ; Tue, 12 Jan 2016 23:55:03 -0800 (PST)
X-ASG-Debug-ID: 1452671701-04cb6c75ddbb350001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id lisMjiJANYFX0vyb (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:55:02 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGGj-0002GK-2u; Wed, 13 Jan 2016 07:55:01 +0000
Date: Tue, 12 Jan 2016 23:55:01 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
Message-ID: <20160113075501.GC21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
References: 
 <20160112195945.GC568@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112195945.GC568@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671702
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Tue, Jan 12, 2016 at 08:59:45PM +0100, Felix Janda wrote:
> ustat has been used to check whether a device file is mounted.
> The function is deprecated and not supported by uclibc and musl.
> Now do the check using the *mntent functions.
> 
> Based on patch by Natanael Copa .
> 
> Signed-off-by: Felix Janda 

ustat is a lot more efficient in checking for the mountedness
than this loop, so I'd really prefer not to do this unconditionally.

As far as Linux is concerned the best thing we could do is to use
O_EXCL for any open of a device file, as Linux overloadds O_EXCL
behavior for block devices so that any other O_EXCL open will
be rejected if one has the file open.

But that might be a major pain to find into the cross platform
framework..

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:55:26 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id EC30C7F37
	for ; Wed, 13 Jan 2016 01:55:26 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id DA888304059
	for ; Tue, 12 Jan 2016 23:55:26 -0800 (PST)
X-ASG-Debug-ID: 1452671725-04cbb05f7c152470001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id UiZcfu0k2vjiZBqd (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:55:25 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGH7-0003SK-4T; Wed, 13 Jan 2016 07:55:25 +0000
Date: Tue, 12 Jan 2016 23:55:25 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 3/6] fsr/xfs_fsr.c: Include  for _PATH_MOUNTED
Message-ID: <20160113075525.GD21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 3/6] fsr/xfs_fsr.c: Include  for _PATH_MOUNTED
References: 
 <20160112195953.GD568@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112195953.GD568@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671725
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Tue, Jan 12, 2016 at 08:59:53PM +0100, Felix Janda wrote:
> Fixes a compilation failure with musl libc
> 
> Signed-off-by: Felix Janda 

Looks fine,

Reviewed-by: Christoph Hellwig 

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:55:44 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 1E8107F37
	for ; Wed, 13 Jan 2016 01:55:44 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id F1DC9304053
	for ; Tue, 12 Jan 2016 23:55:43 -0800 (PST)
X-ASG-Debug-ID: 1452671740-04bdf06f6bc3d70001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id cwDWbg7qce9kCUtU (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:55:40 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGHM-0003UU-7Q; Wed, 13 Jan 2016 07:55:40 +0000
Date: Tue, 12 Jan 2016 23:55:40 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 4/6] linux.h: Use off64_t instead of loff_t
Message-ID: <20160113075540.GE21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 4/6] linux.h: Use off64_t instead of loff_t
References: 
 <20160112200001.GE568@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112200001.GE568@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671740
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Tue, Jan 12, 2016 at 09:00:01PM +0100, Felix Janda wrote:
> These are equivalent on glibc, while musl does not know loff_t.
> 
> In the long run, it would be preferable to enable transparent LFS so
> that off64_t could be replaced by off_t.
> 
> Signed-off-by: Felix Janda 

Looks fine,

Reviewed-by: Christoph Hellwig 

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:56:01 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 603017F37
	for ; Wed, 13 Jan 2016 01:56:01 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 3AA8130405F
	for ; Tue, 12 Jan 2016 23:56:01 -0800 (PST)
X-ASG-Debug-ID: 1452671755-04bdf06f6ec3da0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id uY4Ul60FaAiE0I4n (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:55:59 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGHb-0003Wi-FT; Wed, 13 Jan 2016 07:55:55 +0000
Date: Tue, 12 Jan 2016 23:55:55 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 5/6] include/linux.h: Include  for fprintf and
 stderr
Message-ID: <20160113075555.GF21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 5/6] include/linux.h: Include  for fprintf and
 stderr
References: 
 <20160112200008.GF568@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112200008.GF568@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671757
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Tue, Jan 12, 2016 at 09:00:08PM +0100, Felix Janda wrote:
> Signed-off-by: Felix Janda 

Looks fine,

Reviewed-by: Christoph Hellwig 

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:58:58 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 0C88E7F37
	for ; Wed, 13 Jan 2016 01:58:58 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id DC663304062
	for ; Tue, 12 Jan 2016 23:58:57 -0800 (PST)
X-ASG-Debug-ID: 1452671935-04bdf06f6cc3e80001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id pzQhdEGUx07nc3z5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:58:56 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGKV-0003q4-Iz; Wed, 13 Jan 2016 07:58:55 +0000
Date: Tue, 12 Jan 2016 23:58:55 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
Message-ID: <20160113075855.GG21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
References: 
 <20160112200022.GG568@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112200022.GG568@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671935
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

I think we're having a deeper issue here.  For an LFS build we
absolutely should have these values as the kernel dirent64 has them.

Either we manage to not pull in the LFS version of readdir/getdents
or musl is doing something inredibly stupid.  Can you investigate if
and why we're not using the LFS getdents64 here?

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:59:17 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id D46E97F37
	for ; Wed, 13 Jan 2016 01:59:17 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id B3D118F8040
	for ; Tue, 12 Jan 2016 23:59:17 -0800 (PST)
X-ASG-Debug-ID: 1452671955-04cb6c75dbbb4e0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id ltk8vdo3IjXXXNZI (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:59:15 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGKp-0003uP-5E; Wed, 13 Jan 2016 07:59:15 +0000
Date: Tue, 12 Jan 2016 23:59:15 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/6] invutil/invidx: include  for O_CREAT
Message-ID: <20160113075915.GH21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 1/6] invutil/invidx: include  for O_CREAT
References: 
 <20160112210810.GB10044@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112210810.GB10044@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671955
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

Looks fine,

Reviewed-by: Christoph Hellwig 

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:59:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 06D727F37
	for ; Wed, 13 Jan 2016 01:59:40 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id EB3368F8040
	for ; Tue, 12 Jan 2016 23:59:39 -0800 (PST)
X-ASG-Debug-ID: 1452671978-04cb6c75debb510001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 4YGLnm0R6HNLENTI (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:59:39 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGLC-0003ym-Cl; Wed, 13 Jan 2016 07:59:38 +0000
Date: Tue, 12 Jan 2016 23:59:38 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/6] dump/content.c: Include  for
 XATTR_LIST_MAX
Message-ID: <20160113075938.GI21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 2/6] dump/content.c: Include  for
 XATTR_LIST_MAX
References: 
 <20160112210820.GC10044@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112210820.GC10044@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671978
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

Looks fine,

Reviewed-by: Christoph Hellwig 

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 01:59:54 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 24A227F37
	for ; Wed, 13 Jan 2016 01:59:54 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E1C9F8F8040
	for ; Tue, 12 Jan 2016 23:59:53 -0800 (PST)
X-ASG-Debug-ID: 1452671992-04cb6c75dcbb520001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id Gf0gkWiRaWjezwTD (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 12 Jan 2016 23:59:52 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGLQ-00041m-5B; Wed, 13 Jan 2016 07:59:52 +0000
Date: Tue, 12 Jan 2016 23:59:52 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 3/6] common/main.c: Replace nonstandard SIGCLD by SIGCHLD
Message-ID: <20160113075952.GJ21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 3/6] common/main.c: Replace nonstandard SIGCLD by SIGCHLD
References: 
 <20160112210828.GD10044@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112210828.GD10044@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452671992
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Tue, Jan 12, 2016 at 10:08:28PM +0100, Felix Janda wrote:
> Signed-off-by: Felix Janda 

Looks fine,

Reviewed-by: Christoph Hellwig 

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 02:00:10 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id DB4837F37
	for ; Wed, 13 Jan 2016 02:00:10 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 69A9CAC001
	for ; Wed, 13 Jan 2016 00:00:10 -0800 (PST)
X-ASG-Debug-ID: 1452672008-04cb6c75dcbb540001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id WXBnorYgjYucHbQC (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 00:00:09 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGLf-0005BV-Rs; Wed, 13 Jan 2016 08:00:07 +0000
Date: Wed, 13 Jan 2016 00:00:07 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 4/6] librmt/rmtopen: Prefer  over 
Message-ID: <20160113080007.GK21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 4/6] librmt/rmtopen: Prefer  over 
References: 
 <20160112210835.GE10044@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112210835.GE10044@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452672008
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

Looks fine,

Reviewed-by: Christoph Hellwig 

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 02:00:22 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 3366E7F37
	for ; Wed, 13 Jan 2016 02:00:22 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 11B2D30405F
	for ; Wed, 13 Jan 2016 00:00:22 -0800 (PST)
X-ASG-Debug-ID: 1452672020-04cbb05f7d152630001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id Nsp6HGL10YwvWgpt (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 00:00:20 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGLs-0005Dp-A4; Wed, 13 Jan 2016 08:00:20 +0000
Date: Wed, 13 Jan 2016 00:00:20 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 5/6] dump/var.c: Include  for PATH_MAX
Message-ID: <20160113080020.GL21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 5/6] dump/var.c: Include  for PATH_MAX
References: 
 <20160112210841.GF10044@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112210841.GF10044@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452672020
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

Looks fine,

Reviewed-by: Christoph Hellwig 

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 02:01:39 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 9B96E7F37
	for ; Wed, 13 Jan 2016 02:01:39 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 31E3AAC002
	for ; Wed, 13 Jan 2016 00:01:36 -0800 (PST)
X-ASG-Debug-ID: 1452672092-04bdf06f6ec3fa0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id GkjR6ayjCK6cVuxT (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 00:01:33 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJGN2-0005PM-NE; Wed, 13 Jan 2016 08:01:32 +0000
Date: Wed, 13 Jan 2016 00:01:32 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 6/6] Prefer stdint types over __uint*_t and __int*_t
Message-ID: <20160113080132.GM21939@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 6/6] Prefer stdint types over __uint*_t and __int*_t
References: 
 <20160112210849.GG10044@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160112210849.GG10044@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452672093
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26081
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

>  	dmf_f_ctxt_t	*dmf_f_ctxtp = (dmf_f_ctxt_t *)contextp;
> -	__int64_t	length;
> +	int64_t	length;
>  
>  	if (bmap[0].bmv_entries <= 0) {
>  		return 1;	/* caller must already be at EOF */
> diff --git a/doc/xfsdump.html b/doc/xfsdump.html
> index 9c6722e..f85128f 100644
> --- a/doc/xfsdump.html
> +++ b/doc/xfsdump.html
> @@ -1935,7 +1935,7 @@ The files are constructed like so:
>  
>  typedef struct invt_counter {
>      INVT_COUNTER_FIELDS
> -        __uint32_t    ic_vernum;/* on disk version number for posterity */\
> +        uint32_t    ic_vernum;/* on disk version number for posterity */\

please keep the existing alignments for variable declarations or
structure members.

Otherwise this looks fine:

Reviewed-by: Christoph Hellwig 

From jkosina@suse.cz  Wed Jan 13 02:37:37 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 33E917F37
	for ; Wed, 13 Jan 2016 02:37:37 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 14C768F8040
	for ; Wed, 13 Jan 2016 00:37:33 -0800 (PST)
X-ASG-Debug-ID: 1452674250-04cbb05f7d153590001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id QvjKaGUGNBm8iPhu (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 13 Jan 2016 00:37:31 -0800 (PST)
X-Barracuda-Envelope-From: jkosina@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id BCCFEABE5;
	Wed, 13 Jan 2016 08:37:29 +0000 (UTC)
Date: Wed, 13 Jan 2016 09:37:29 +0100 (CET)
From: Jiri Kosina 
To: Dave Chinner 
cc: xfs@oss.sgi.com
Subject: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
In-Reply-To: <1452661968-11482-1-git-send-email-david@fromorbit.com>
X-ASG-Orig-Subj: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Message-ID: 
References: <1452661968-11482-1-git-send-email-david@fromorbit.com>
User-Agent: Alpine 2.00 (LNX 1167 2008-08-23)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1452674251
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26082
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed, 13 Jan 2016, Dave Chinner wrote:

> This reverts commit 24ba16bb3d499c49974669cd8429c3e4138ab102 as it
> prevents machines from suspending. This regression occurs when the
> xfsaild is idle on entry to suspend, and so there s no activity to
> wake it from it's idle sleep and hence see that it is supposed to
> freeze. Hence the freezer times out waiting for it and suspend is
> cancelled.
> 
> There is no obvious fix for this short of freezing the filesystem
> properly, so revert this change for now.
> 
> Signed-off-by: Dave Chinner 

I hope I'll find time in the future and make some progress on making the 
fs freezing happen during suspend properly. Unfortunately, before that 
happens

	Acked-by: Jiri Kosina 

Thanks,

-- 
Jiri Kosina
SUSE Labs

From jack@suse.cz  Wed Jan 13 03:07:42 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 8B6E57F37
	for ; Wed, 13 Jan 2016 03:07:42 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E322E8F8040
	for ; Wed, 13 Jan 2016 01:07:38 -0800 (PST)
X-ASG-Debug-ID: 1452676053-04bdf06f6ec5b20001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id kltlrckFuF8HNBgE (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 13 Jan 2016 01:07:34 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 382D0AAB2;
	Wed, 13 Jan 2016 09:07:27 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id CC9EC823D8; Wed, 13 Jan 2016 10:07:34 +0100 (CET)
Date: Wed, 13 Jan 2016 10:07:34 +0100
From: Jan Kara 
To: Ross Zwisler 
Cc: Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 1/9] dax: fix NULL pointer dereference in __dax_dbg()
Message-ID: <20160113090734.GC14630@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH v8 1/9] dax: fix NULL pointer dereference in __dax_dbg()
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-2-git-send-email-ross.zwisler@linux.intel.com>
 <20160112093458.GR6262@quack.suse.cz>
 <20160113070829.GA30496@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113070829.GA30496@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1452676054
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26082
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed 13-01-16 00:08:29, Ross Zwisler wrote:
> On Tue, Jan 12, 2016 at 10:34:58AM +0100, Jan Kara wrote:
> > On Thu 07-01-16 22:27:51, Ross Zwisler wrote:
> > > In __dax_pmd_fault() we currently assume that get_block() will always set
> > > bh.b_bdev and we unconditionally dereference it in __dax_dbg().  This
> > > assumption isn't always true - when called for reads of holes
> > > ext4_dax_mmap_get_block() returns a buffer head where bh->b_bdev is never
> > > set.  I hit this BUG while testing the DAX PMD fault path.
> > > 
> > > Instead, initialize bh.b_bdev before passing bh into get_block().  It is
> > > possible that the filesystem's get_block() will update bh.b_bdev, and this
> > > is fine - we just want to initialize bh.b_bdev to something reasonable so
> > > that the calls to __dax_dbg() work and print something useful.
> > > 
> > > Signed-off-by: Ross Zwisler 
> > > Cc: Dan Williams 
> > 
> > Looks good. But don't you need to do the same for __dax_fault(),
> > dax_zero_page_range() and similar places passing bh to dax functions?
> > 
> > 								Honza
> 
> I don't think we need it anywhere else.  The only reason that we need to
> initialize the bh.b_bdev manually in the __dax_pmd_fault() path is that if the
> get_block() call ends up finding a hole (so doesn't fill out b_bdev) we still
> go through the dax_pmd_dbg() path to print an error message, which uses
> b_bdev.  I believe that in the other paths where we hit a hole, such as
> __dax_fault(), we don't use b_bdev because we don't have the same error path
> prints, and the regular code for handling holes doesn't use b_bdev.
> 
> That being said, if you feel like it's cleaner to initialize it
> everywhere so everything is consistent and we don't have to worry about
> it, I'm fine to make the change.

Well, it seems more futureproof to me. In case someone decides to add some
debug message later on...

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Wed Jan 13 03:35:22 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 081857F37
	for ; Wed, 13 Jan 2016 03:35:22 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E9FC88F804B
	for ; Wed, 13 Jan 2016 01:35:21 -0800 (PST)
X-ASG-Debug-ID: 1452677717-04cb6c75debd6c0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id dhbHfu6LZpVcN3LG (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 13 Jan 2016 01:35:18 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 95520AAB2;
	Wed, 13 Jan 2016 09:35:15 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id A334D823D8; Wed, 13 Jan 2016 10:35:25 +0100 (CET)
Date: Wed, 13 Jan 2016 10:35:25 +0100
From: Jan Kara 
To: Ross Zwisler 
Cc: Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 6/9] dax: add support for fsync/msync
Message-ID: <20160113093525.GD14630@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH v8 6/9] dax: add support for fsync/msync
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-7-git-send-email-ross.zwisler@linux.intel.com>
 <20160112105716.GT6262@quack.suse.cz>
 <20160113073019.GB30496@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113073019.GB30496@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1452677718
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26083
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed 13-01-16 00:30:19, Ross Zwisler wrote:
> > And secondly: You must write-protect all mappings of the flushed range so
> > that you get fault when the sector gets written-to again. We spoke about
> > this in the past already but somehow it got lost and I forgot about it as
> > well. You need something like rmap_walk_file()...
> 
> The code that write protected mappings and then cleaned the radix tree entries
> did get written, and was part of v2:
> 
> https://lkml.org/lkml/2015/11/13/759
> 
> I removed all the code that cleaned PTE entries and radix tree entries for v3.
> The reason behind this was that there was a race that I couldn't figure out
> how to solve between the cleaning of the PTEs and the cleaning of the radix
> tree entries.
> 
> The race goes like this:
> 
> Thread 1 (write)			Thread 2 (fsync)
> ================			================
> wp_pfn_shared()
> pfn_mkwrite()
> dax_radix_entry()
> radix_tree_tag_set(DIRTY)
> 					dax_writeback_mapping_range()
> 					dax_writeback_one()
> 					radix_tag_clear(DIRTY)
> 					pgoff_mkclean()
> ... return up to wp_pfn_shared()
> wp_page_reuse()
> pte_mkdirty()
> 
> After this sequence we end up with a dirty PTE that is writeable, but with a
> clean radix tree entry.  This means that users can write to the page, but that
> a follow-up fsync or msync won't flush this dirty data to media.
> 
> The overall issue is that in the write path that goes through wp_pfn_shared(),
> the DAX code has control over when the radix tree entry is dirtied but not
> when the PTE is made dirty and writeable.  This happens up in wp_page_reuse().
> This means that we can't easily add locking, etc. to protect ourselves.
> 
> I spoke a bit about this with Dave Chinner and with Dave Hansen, but no really
> easy solutions presented themselves in the absence of a page lock.  I do have
> one idea, but I think it's pretty invasive and will need to wait for another
> kernel cycle.
> 
> The current code that leaves the radix tree entry will give us correct
> behavior - it'll just be less efficient because we will have an ever-growing
> dirty set to flush.

Ahaa! Somehow I imagined tag_pages_for_writeback() clears DIRTY radix tree
tags but it does not (I should have known, I have written that functions
few years ago ;). Makes sense. Thanks for clarification.

> > > @@ -791,15 +976,12 @@ EXPORT_SYMBOL_GPL(dax_pmd_fault);
> > >   * dax_pfn_mkwrite - handle first write to DAX page
> > >   * @vma: The virtual memory area where the fault occurred
> > >   * @vmf: The description of the fault
> > > - *
> > >   */
> > >  int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
> > >  {
> > > -	struct super_block *sb = file_inode(vma->vm_file)->i_sb;
> > > +	struct file *file = vma->vm_file;
> > >  
> > > -	sb_start_pagefault(sb);
> > > -	file_update_time(vma->vm_file);
> > > -	sb_end_pagefault(sb);
> > > +	dax_radix_entry(file->f_mapping, vmf->pgoff, NO_SECTOR, false, true);
> > 
> > Why is NO_SECTOR argument correct here?
> 
> Right - so NO_SECTOR means "I expect there to already be an entry in the radix
> tree - just make that entry dirty".  This works because pfn_mkwrite() always
> follows a normal __dax_fault() or __dax_pmd_fault() call.  These fault calls
> will insert the radix tree entry, regardless of whether the fault was for a
> read or a write.  If the fault was for a write, the radix tree entry will also
> be made dirty.
>
> For reads the radix tree entry will be inserted but left clean.  When the
> first write happens we will get a pfn_mkwrite() call, which will call
> dax_radix_entry() with the NO_SECTOR argument.  This will look up the radix
> tree entry & set the dirty tag.

So the explanation of this should be somewhere so that everyone knows that
we must have radix tree entries even for clean mapped blocks. Because upto
know that was not clear to me.  Also __dax_pmd_fault() seems to insert
entries only for write fault so the assumption doesn't seem to hold there?

I'm somewhat uneasy that a bug in this logic can be hidden as a simple race
with hole punching. But I guess I can live with that.
 
								Honza
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Wed Jan 13 03:44:06 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 11EEF7F3F
	for ; Wed, 13 Jan 2016 03:44:06 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D82328F8040
	for ; Wed, 13 Jan 2016 01:44:05 -0800 (PST)
X-ASG-Debug-ID: 1452678242-04bdf06f6cc6900001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id NUObHW4eRtm2VpKG (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 13 Jan 2016 01:44:03 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 32FBDAAB2;
	Wed, 13 Jan 2016 09:44:01 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 081DE823D8; Wed, 13 Jan 2016 10:44:12 +0100 (CET)
Date: Wed, 13 Jan 2016 10:44:11 +0100
From: Jan Kara 
To: Ross Zwisler 
Cc: linux-kernel@vger.kernel.org, "H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree
Message-ID: <20160113094411.GA17057@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-5-git-send-email-ross.zwisler@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452230879-18117-5-git-send-email-ross.zwisler@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1452678243
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26083
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu 07-01-16 22:27:54, Ross Zwisler wrote:
> Add support for tracking dirty DAX entries in the struct address_space
> radix tree.  This tree is already used for dirty page writeback, and it
> already supports the use of exceptional (non struct page*) entries.
> 
> In order to properly track dirty DAX pages we will insert new exceptional
> entries into the radix tree that represent dirty DAX PTE or PMD pages.
> These exceptional entries will also contain the writeback sectors for the
> PTE or PMD faults that we can use at fsync/msync time.
> 
> There are currently two types of exceptional entries (shmem and shadow)
> that can be placed into the radix tree, and this adds a third.  We rely on
> the fact that only one type of exceptional entry can be found in a given
> radix tree based on its usage.  This happens for free with DAX vs shmem but
> we explicitly prevent shadow entries from being added to radix trees for
> DAX mappings.
> 
> The only shadow entries that would be generated for DAX radix trees would
> be to track zero page mappings that were created for holes.  These pages
> would receive minimal benefit from having shadow entries, and the choice
> to have only one type of exceptional entry in a given radix tree makes the
> logic simpler both in clear_exceptional_entry() and in the rest of DAX.
> 
> Signed-off-by: Ross Zwisler 
> Reviewed-by: Jan Kara 

I have realized there's one issue with this code. See below:

> @@ -34,31 +35,39 @@ static void clear_exceptional_entry(struct address_space *mapping,
>  		return;
>  
>  	spin_lock_irq(&mapping->tree_lock);
> -	/*
> -	 * Regular page slots are stabilized by the page lock even
> -	 * without the tree itself locked.  These unlocked entries
> -	 * need verification under the tree lock.
> -	 */
> -	if (!__radix_tree_lookup(&mapping->page_tree, index, &node, &slot))
> -		goto unlock;
> -	if (*slot != entry)
> -		goto unlock;
> -	radix_tree_replace_slot(slot, NULL);
> -	mapping->nrshadows--;
> -	if (!node)
> -		goto unlock;
> -	workingset_node_shadows_dec(node);
> -	/*
> -	 * Don't track node without shadow entries.
> -	 *
> -	 * Avoid acquiring the list_lru lock if already untracked.
> -	 * The list_empty() test is safe as node->private_list is
> -	 * protected by mapping->tree_lock.
> -	 */
> -	if (!workingset_node_shadows(node) &&
> -	    !list_empty(&node->private_list))
> -		list_lru_del(&workingset_shadow_nodes, &node->private_list);
> -	__radix_tree_delete_node(&mapping->page_tree, node);
> +
> +	if (dax_mapping(mapping)) {
> +		if (radix_tree_delete_item(&mapping->page_tree, index, entry))
> +			mapping->nrexceptional--;

So when you punch hole in a file, you can delete a PMD entry from a radix
tree which covers part of the file which still stays. So in this case you
have to split the PMD entry into PTE entries (probably that needs to happen
up in truncate_inode_pages_range()) or something similar...

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From miwwherq@xunu.com  Wed Jan 13 05:05:25 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: ****
X-Spam-Status: No, score=4.2 required=5.0 tests=HK_RANDOM_FROM,
	HK_RANDOM_REPLYTO,HTML_MESSAGE,MIME_HTML_ONLY autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 948FC7F37
	for ; Wed, 13 Jan 2016 05:05:25 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 08189AC001
	for ; Wed, 13 Jan 2016 03:05:21 -0800 (PST)
X-ASG-Debug-ID: 1452683117-04cbb05f7f158dc0001-NocioJ
Received: from xunu.com ([220.179.8.207]) by cuda.sgi.com with ESMTP id MbKpGkBnEZm8d6Vb for ; Wed, 13 Jan 2016 03:05:18 -0800 (PST)
X-Barracuda-Envelope-From: miwwherq@xunu.com
X-Barracuda-Apparent-Source-IP: 220.179.8.207
Received: from PC-20150903UGRM ([127.0.0.1]) by localhost via TCP with ESMTPA; Wed, 13 Jan 2016 19:05:10 +0800
MIME-Version: 1.0
From: Appliances 
Sender: Appliances 
To: xfs@oss.sgi.com
Reply-To: Appliances 
Date: 13 Jan 2016 19:05:10 +0800
Subject: =?utf-8?B?SGF2aW5nIHRvcGNoaW5hIEFwcGxpYW5jZXMgcHJpY2VsaXN0Pw==?=
Content-Type: text/html; charset=utf-8
X-ASG-Orig-Subj: =?utf-8?B?SGF2aW5nIHRvcGNoaW5hIEFwcGxpYW5jZXMgcHJpY2VsaXN0Pw==?=
Content-Transfer-Encoding: base64
X-Barracuda-Connect: UNKNOWN[220.179.8.207]
X-Barracuda-Start-Time: 1452683117
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.74
X-Barracuda-Spam-Status: No, SCORE=0.74 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, HTML_MESSAGE, MIME_HTML_ONLY, MISSING_MID, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26084
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.14 MISSING_MID            Missing Message-Id: header
	0.00 MIME_HTML_ONLY         BODY: Message only has text/html MIME parts
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963
Message-Id: <20160113110521.71534106C0A0@cuda.sgi.com>

PGh0bWw+PGJvZHk+PERJViBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9V
TkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IEZPTlQ6IDE0
cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IExFVFRFUi1TUEFD
SU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQQUNJTkc6IDBweDsgLXdl
YmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48U1RST05HPjxTUEFOIHN0eWxlPSJURVhU
LVRSQU5TRk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsg
VEVYVC1JTkRFTlQ6IDBweDsgRElTUExBWTogaW5saW5lICFpbXBvcnRhbnQ7IEZPTlQ6IDE0
cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IEZMT0FUOiBub25l
OyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgV09SRC1TUEFD
SU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+SGVsbG8sPC9TUEFO
PjxCUiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJn
YigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IEZPTlQ6IDE0cHgvMjFweCDlvq7o
va/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7
IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQQUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0
cm9rZS13aWR0aDogMHB4Ij48QlIgc3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25lOyBCQUNL
R1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBURVhULUlOREVOVDogMHB4OyBGT05U
OiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTogbm9ybWFsOyBMRVRURVIt
U1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgV09SRC1TUEFDSU5HOiAwcHg7
IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PFNQQU4gc3R5bGU9IlRFWFQtVFJB
TlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBURVhU
LUlOREVOVDogMHB4OyBESVNQTEFZOiBpbmxpbmUgIWltcG9ydGFudDsgRk9OVDogMTRweC8y
MXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsgRkxPQVQ6IG5vbmU7IExF
VFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQQUNJTkc6
IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij5OaWNlIHRvIGtub3cgeW91
LCBJIGFtIEplZmZyZXkgZnJvbSBIT05HS09ORyBBSkwgQVBQTElBTkNFIENvLixMdGQuLjwv
U1BBTj48QlIgc3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9S
OiByZ2IoMjU1LDI1NSwyNTUpOyBURVhULUlOREVOVDogMHB4OyBGT05UOiAxNHB4LzIxcHgg
5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTogbm9ybWFsOyBMRVRURVItU1BBQ0lORzogbm9y
bWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgV09SRC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4
dC1zdHJva2Utd2lkdGg6IDBweCI+PEJSIHN0eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsg
QkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsgVEVYVC1JTkRFTlQ6IDBweDsg
Rk9OVDogMTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsgTEVU
VEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IFdPUkQtU1BBQ0lORzog
MHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPjxTUEFOIHN0eWxlPSJURVhU
LVRSQU5TRk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsg
VEVYVC1JTkRFTlQ6IDBweDsgRElTUExBWTogaW5saW5lICFpbXBvcnRhbnQ7IEZPTlQ6IDE0
cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IEZMT0FUOiBub25l
OyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgV09SRC1TUEFD
SU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+T3VyIG1haW5seSBp
bmNsdWRpbmcgSG91c2Vob2xkIGFwcGxpYW5jZXMsIGVzcGVjaWFsbHkgYWJvdXQgdmFjdXVt
IGNsZWFuZXJzLjwvU1BBTj48QlIgc3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25lOyBCQUNL
R1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBURVhULUlOREVOVDogMHB4OyBGT05U
OiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTogbm9ybWFsOyBMRVRURVIt
U1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgV09SRC1TUEFDSU5HOiAwcHg7
IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PEJSIHN0eWxlPSJURVhULVRSQU5T
Rk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsgVEVYVC1J
TkRFTlQ6IDBweDsgRk9OVDogMTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6
IG5vcm1hbDsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IFdP
UkQtU1BBQ0lORzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPjxTUEFO
IHN0eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1
NSwyNTUsMjU1KTsgVEVYVC1JTkRFTlQ6IDBweDsgRElTUExBWTogaW5saW5lICFpbXBvcnRh
bnQ7IEZPTlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7
IEZMT0FUOiBub25lOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCww
KTsgV09SRC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+
VGhlIGNvbXBhbnkgaXMgY2FwYWJsZSBvZiBwcm9kdWNpbmcgbmV3IHByb2R1Y3RzIGFjY29y
ZGluZyB0byBjdXN0b21lciBzYW1wbGVzIGFuZCByZXF1aXJlbWVudHMuIEFsbCB0aGUgcHJv
ZHVjdHMgaGF2ZSBiZWVuIGV4cG9ydGVkIGFsbCBvdmVyIHRoZSB3b3JsZC48L1NQQU4+PEJS
IHN0eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1
NSwyNTUsMjU1KTsgVEVYVC1JTkRFTlQ6IDBweDsgRk9OVDogMTRweC8yMXB4IOW+rui9r+mb
hem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09M
T1I6IHJnYigwLDAsMCk7IFdPUkQtU1BBQ0lORzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tl
LXdpZHRoOiAwcHgiPjxCUiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9V
TkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IEZPTlQ6IDE0
cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IExFVFRFUi1TUEFD
SU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQQUNJTkc6IDBweDsgLXdl
YmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48U1BBTiBzdHlsZT0iVEVYVC1UUkFOU0ZP
Uk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5E
RU5UOiAwcHg7IERJU1BMQVk6IGlubGluZSAhaW1wb3J0YW50OyBGT05UOiAxNHB4LzIxcHgg
5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTogbm9ybWFsOyBGTE9BVDogbm9uZTsgTEVUVEVS
LVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IFdPUkQtU1BBQ0lORzogMHB4
OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPkZhY3RvcnkgY2VydGlmaWNhdGlv
bjombmJzcDsgSVNPIDkwMDEuPC9TUEFOPjxCUiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5v
bmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAw
cHg7IEZPTlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7
IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQQUNJ
Tkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48QlIgc3R5bGU9IlRF
WFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUp
OyBURVhULUlOREVOVDogMHB4OyBGT05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElU
RS1TUEFDRTogbm9ybWFsOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAs
MCwwKTsgV09SRC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBw
eCI+PFNQQU4gc3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9S
OiByZ2IoMjU1LDI1NSwyNTUpOyBURVhULUlOREVOVDogMHB4OyBESVNQTEFZOiBpbmxpbmUg
IWltcG9ydGFudDsgRk9OVDogMTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6
IG5vcm1hbDsgRkxPQVQ6IG5vbmU7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiBy
Z2IoMCwwLDApOyBXT1JELVNQQUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0
aDogMHB4Ij5Gb3IgbW9yZSBpbmZvcm1hdGlvbiwgcGxlYXNlIGZlZWwgZnJlZSBjb250YWN0
IHdpdGggbWUuPC9TUEFOPjxCUiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tH
Uk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IEZPTlQ6
IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IExFVFRFUi1T
UEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQQUNJTkc6IDBweDsg
LXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48QlIgc3R5bGU9IlRFWFQtVFJBTlNG
T1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBURVhULUlO
REVOVDogMHB4OyBGT05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTog
bm9ybWFsOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgV09S
RC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PFNQQU4g
c3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1
LDI1NSwyNTUpOyBURVhULUlOREVOVDogMHB4OyBESVNQTEFZOiBpbmxpbmUgIWltcG9ydGFu
dDsgRk9OVDogMTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsg
RkxPQVQ6IG5vbmU7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDAp
OyBXT1JELVNQQUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij5X
ZSB3aXNoIHlvdSBkZXZlbG9wIGZ1cnRoZXIgYnVzaW5lc3Mgd2l0aCB1cyBpbiB0aGUgY29t
aW5nIDIwMTYgYW5kIGhhcHB5IG5ldyB5ZWFyLjwvU1BBTj48QlIgc3R5bGU9IlRFWFQtVFJB
TlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBURVhU
LUlOREVOVDogMHB4OyBGT05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFD
RTogbm9ybWFsOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsg
V09SRC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PEJS
IHN0eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1
NSwyNTUsMjU1KTsgVEVYVC1JTkRFTlQ6IDBweDsgRk9OVDogMTRweC8yMXB4IOW+rui9r+mb
hem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09M
T1I6IHJnYigwLDAsMCk7IFdPUkQtU1BBQ0lORzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tl
LXdpZHRoOiAwcHgiPjxTUEFOIHN0eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsgQkFDS0dS
T1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsgVEVYVC1JTkRFTlQ6IDBweDsgRElTUExB
WTogaW5saW5lICFpbXBvcnRhbnQ7IEZPTlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdI
SVRFLVNQQUNFOiBub3JtYWw7IEZMT0FUOiBub25lOyBMRVRURVItU1BBQ0lORzogbm9ybWFs
OyBDT0xPUjogcmdiKDAsMCwwKTsgV09SRC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4dC1z
dHJva2Utd2lkdGg6IDBweCI+SmVmZnJleTwvU1BBTj48QlIgc3R5bGU9IlRFWFQtVFJBTlNG
T1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBURVhULUlO
REVOVDogMHB4OyBGT05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTog
bm9ybWFsOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgV09S
RC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PFNQQU4g
c3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1
LDI1NSwyNTUpOyBURVhULUlOREVOVDogMHB4OyBESVNQTEFZOiBpbmxpbmUgIWltcG9ydGFu
dDsgRk9OVDogMTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsg
RkxPQVQ6IG5vbmU7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDAp
OyBXT1JELVNQQUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij5H
ZXJuYWwgTWFuZ2VyPC9TUEFOPjxCUiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJB
Q0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IEZP
TlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IExFVFRF
Ui1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQQUNJTkc6IDBw
eDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48QlIgc3R5bGU9IlRFWFQtVFJB
TlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBURVhU
LUlOREVOVDogMHB4OyBGT05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFD
RTogbm9ybWFsOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsg
V09SRC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PFNQ
QU4gc3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2Io
MjU1LDI1NSwyNTUpOyBURVhULUlOREVOVDogMHB4OyBESVNQTEFZOiBpbmxpbmUgIWltcG9y
dGFudDsgRk9OVDogMTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1h
bDsgRkxPQVQ6IG5vbmU7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCww
LDApOyBXT1JELVNQQUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4
Ij5IT05HUk9ORyBHUk9VUDwvU1BBTj48QlIgc3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25l
OyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBURVhULUlOREVOVDogMHB4
OyBGT05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTogbm9ybWFsOyBM
RVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgV09SRC1TUEFDSU5H
OiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PFNQQU4gc3R5bGU9IlRF
WFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUp
OyBURVhULUlOREVOVDogMHB4OyBESVNQTEFZOiBpbmxpbmUgIWltcG9ydGFudDsgRk9OVDog
MTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsgRkxPQVQ6IG5v
bmU7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQ
QUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij5IT05HS09ORyBB
SkwgQVBQTElBTkNFIENPLExJTUlURUQ8L1NQQU4+PEJSIHN0eWxlPSJURVhULVRSQU5TRk9S
TTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsgVEVYVC1JTkRF
TlQ6IDBweDsgRk9OVDogMTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6IG5v
cm1hbDsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IFdPUkQt
U1BBQ0lORzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPjxTUEFOIHN0
eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwy
NTUsMjU1KTsgVEVYVC1JTkRFTlQ6IDBweDsgRElTUExBWTogaW5saW5lICFpbXBvcnRhbnQ7
IEZPTlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IEZM
T0FUOiBub25lOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsg
V09SRC1TUEFDSU5HOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+QWRk
cmVzczogIzEsIFdlc3QgSmlhb3hpbmcgUm9hZCwgSmlhb3RvdSBJbmR1c3RyaWFsIFpvbmUs
IEppYW5nbWVuIENpdHksIEd1YW5nZG9uZywgQ2hpbmEgNTI5MDAwPC9TUEFOPjxCUiBzdHls
ZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1
LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IEZPTlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7
IFdISVRFLVNQQUNFOiBub3JtYWw7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiBy
Z2IoMCwwLDApOyBXT1JELVNQQUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0
aDogMHB4Ij48U1BBTiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQt
Q09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IERJU1BMQVk6IGlu
bGluZSAhaW1wb3J0YW50OyBGT05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1T
UEFDRTogbm9ybWFsOyBGTE9BVDogbm9uZTsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09M
T1I6IHJnYigwLDAsMCk7IFdPUkQtU1BBQ0lORzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tl
LXdpZHRoOiAwcHgiPlFROiAzMjg4MDgwMDE4PC9TUEFOPjxCUiBzdHlsZT0iVEVYVC1UUkFO
U0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQt
SU5ERU5UOiAwcHg7IEZPTlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNF
OiBub3JtYWw7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBX
T1JELVNQQUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48U1BB
TiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigy
NTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IERJU1BMQVk6IGlubGluZSAhaW1wb3J0
YW50OyBGT05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTogbm9ybWFs
OyBGTE9BVDogbm9uZTsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAs
MCk7IFdPUkQtU1BBQ0lORzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgi
PlRlbDogKzg2IDc1MCAzMzM2NTM5PC9TUEFOPjxCUiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06
IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5U
OiAwcHg7IEZPTlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3Jt
YWw7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQ
QUNJTkc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48U1BBTiBzdHls
ZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1
LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7IERJU1BMQVk6IGlubGluZSAhaW1wb3J0YW50OyBG
T05UOiAxNHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTogbm9ybWFsOyBGTE9B
VDogbm9uZTsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IFdP
UkQtU1BBQ0lORzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPkZheDog
Kzg2IDc1MCAzMzM2NTM5PC9TUEFOPjxCUiBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7
IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IFRFWFQtSU5ERU5UOiAwcHg7
IEZPTlQ6IDE0cHgvMjFweCDlvq7ova/pm4Xpu5E7IFdISVRFLVNQQUNFOiBub3JtYWw7IExF
VFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBXT1JELVNQQUNJTkc6
IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48U1BBTiBzdHlsZT0iVEVY
VC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7
IFRFWFQtSU5ERU5UOiAwcHg7IERJU1BMQVk6IGlubGluZSAhaW1wb3J0YW50OyBGT05UOiAx
NHB4LzIxcHgg5b6u6L2v6ZuF6buROyBXSElURS1TUEFDRTogbm9ybWFsOyBGTE9BVDogbm9u
ZTsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IFdPUkQtU1BB
Q0lORzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPk1vYmlsZSBwaG9u
ZTogMTg2NjYxOTU3MDc8L1NQQU4+PEJSIHN0eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsg
QkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsgVEVYVC1JTkRFTlQ6IDBweDsg
Rk9OVDogMTRweC8yMXB4IOW+rui9r+mbhem7kTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsgTEVU
VEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IFdPUkQtU1BBQ0lORzog
MHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPjwvU1RST05HPjwvRElWPjwv
Ym9keT48L2h0bWw+

From bfoster@redhat.com  Wed Jan 13 07:22:30 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 1EA217F37
	for ; Wed, 13 Jan 2016 07:22:30 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id BBD9EAC001
	for ; Wed, 13 Jan 2016 05:22:29 -0800 (PST)
X-ASG-Debug-ID: 1452691348-04bdf06f6dcdd90001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 9zCjPWRXBhthxfbS (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 13 Jan 2016 05:22:28 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id 1D4AE8E90B;
	Wed, 13 Jan 2016 13:22:28 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0DDMRIm020140;
	Wed, 13 Jan 2016 08:22:27 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id BC8751245A1; Wed, 13 Jan 2016 08:22:26 -0500 (EST)
Date: Wed, 13 Jan 2016 08:22:26 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, jkosina@suse.cz
Subject: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Message-ID: <20160113132226.GA37138@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
References: <1452661968-11482-1-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452661968-11482-1-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452691348
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 13, 2016 at 04:12:48PM +1100, Dave Chinner wrote:
> This reverts commit 24ba16bb3d499c49974669cd8429c3e4138ab102 as it
> prevents machines from suspending. This regression occurs when the
> xfsaild is idle on entry to suspend, and so there s no activity to
> wake it from it's idle sleep and hence see that it is supposed to
> freeze. Hence the freezer times out waiting for it and suspend is
> cancelled.
> 
> There is no obvious fix for this short of freezing the filesystem
> properly, so revert this change for now.
> 
> Signed-off-by: Dave Chinner 
> ---

Reviewed-by: Brian Foster 

>  fs/xfs/xfs_trans_ail.c | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> index aa67339..4f18fd9 100644
> --- a/fs/xfs/xfs_trans_ail.c
> +++ b/fs/xfs/xfs_trans_ail.c
> @@ -497,7 +497,6 @@ xfsaild(
>  	long		tout = 0;	/* milliseconds */
>  
>  	current->flags |= PF_MEMALLOC;
> -	set_freezable();
>  
>  	while (!kthread_should_stop()) {
>  		if (tout && tout <= 20)
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From jason.reed@globalcontactspro.com  Wed Jan 13 09:12:27 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,
	T_FILL_THIS_FORM_SHORT autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id D566B7F37
	for ; Wed, 13 Jan 2016 09:12:26 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 83BA4AC001
	for ; Wed, 13 Jan 2016 07:12:23 -0800 (PST)
X-ASG-Debug-ID: 1452697935-04bdf06f6ed27f0001-NocioJ
Received: from mail.globalmarketics.com (dmcsquare.net [182.18.136.235]) by cuda.sgi.com with SMTP id MTX4A1zDkEWpw4vN for ; Wed, 13 Jan 2016 07:12:15 -0800 (PST)
X-Barracuda-Envelope-From: jason.reed@globalcontactspro.com
X-Barracuda-Apparent-Source-IP: 182.18.136.235
Received: from localhost (localhost [127.0.0.1])
	by mail.globalmarketics.com (Postfix) with ESMTP id 4EDBC12216F2
	for ; Wed, 13 Jan 2016 20:54:43 +0530 (IST)
Received: from mail.globalmarketics.com ([127.0.0.1])
	by localhost (mail.globalmarketics.com [127.0.0.1]) (amavisd-new, port 10032)
	with ESMTP id PLAfN2CmwAfG for ;
	Wed, 13 Jan 2016 20:54:41 +0530 (IST)
Received: from localhost (localhost [127.0.0.1])
	by mail.globalmarketics.com (Postfix) with ESMTP id C110412216C6
	for ; Wed, 13 Jan 2016 20:54:41 +0530 (IST)
X-Virus-Scanned: amavisd-new at mail.globalmarketics.com
Received: from mail.globalmarketics.com ([127.0.0.1])
	by localhost (mail.globalmarketics.com [127.0.0.1]) (amavisd-new, port 10026)
	with ESMTP id FMZ42tHKIyGb for ;
	Wed, 13 Jan 2016 20:54:41 +0530 (IST)
Received: from sphere097 (unknown [106.51.31.192])
	by mail.globalmarketics.com (Postfix) with ESMTPSA id BBDDB12216F0
	for ; Wed, 13 Jan 2016 20:54:40 +0530 (IST)
Return-Receipt-To: "Jason Reed" 
From: "Jason Reed" 
To: 
Subject: Hadoop
Date: Wed, 13 Jan 2016 20:48:58 +0530
X-ASG-Orig-Subj: Hadoop
Message-ID: 
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0DF3_01D14E43.D3874CD0"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AdFMa6v3sXKeF1E5QBiqiRXN977rQwAAvGvgAABI7tAAAAAUsABmU7IgAABg+UAAAH6W8AACQ/YwAAADQDAAAAAUYAAAAAbAAAAAB/AAAAAHIAAAAAeAAAAAB/AAAAAHUAAAAAeAAAAAB+AAAAAIQAAAAAewAAAAB0AAAAAI4AAAAAcQAAAAB3AAAAAG0AAAAAhAAAAAB3AAAAAI0AAAAAdgAAAAB9AAAAAIYAAAAAfwAAAACJAAAAAI8AAAAAiQ
Content-Language: en-in
Disposition-Notification-To: "Jason Reed" 
X-Barracuda-Connect: dmcsquare.net[182.18.136.235]
X-Barracuda-Start-Time: 1452697935
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.01
X-Barracuda-Spam-Status: No, SCORE=0.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=HTML_MESSAGE, THREAD_INDEX
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26088
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.01 THREAD_INDEX           thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig==
	0.00 HTML_MESSAGE           BODY: HTML included in message

This is a multipart message in MIME format.

------=_NextPart_000_0DF3_01D14E43.D3874CD0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,

=20

I was reviewing your website and thought you=92ll be interested in the
database of companies using Hadoop.

=20

We are Email list providers and offer our services globally. We also =
have
the data of Apache Hadoop Users, Splunk Users, Horton Works Users, =
Cloudera
Users, Python Users, R Users, Apache Spark Users, AWS Users, MS Lync =
Users,
HP Cloud Users, Oracle Cloud Users, VMWare Users, Salesforce Users, =
Cisco
Users, VMware Users, EMC Users, Citrix Users, Magento Users, NoSQL Users =
and
many more.=20

=20

Database Features:

=20

=D8  Data fields: Contact Name, Job title, Phone Number, Fax Number, =
SIC,
Email address, Company Name, Website, Address, Industry, Revenue, =
Employees
Size, LinkedIn profile.

=D8  Once you procure the list from us, you=92ll have the complete =
ownership of
the data and can use it for your unlimited marketing usage.=20

=20

Kindly let me know the target audience you would like to target, I shall
send across more information accordingly.

               =20

Jason Reed

Business Development Coordinator

Phone: +1 832 644 3131=20

=20

=20

=20

=20

=20

We respect your privacy, If you don=92t wish to receive further emails =
from
us, please reply with the subject =93Remove=94

=20


------=_NextPart_000_0DF3_01D14E43.D3874CD0
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
















Hi,



 



I was =
reviewing your
website and thought you’ll be interested in the database of =
companies
using Hadoop.



 



We are Email list =
providers and
offer our services globally. We also have the data of Apache Hadoop =
Users,
Splunk Users, Horton Works Users, Cloudera Users, Python Users, R Users, =
Apache
Spark Users, AWS Users, MS Lync Users, HP Cloud Users, Oracle Cloud =
Users,
VMWare Users, Salesforce Users, Cisco Users, VMware Users, EMC Users, =
Citrix
Users, Magento Users, NoSQL Users and many more. =




 



Database =
Features:



 



=D8  Data fields: Contact Name, Job title, Phone =
Number, Fax
Number, SIC, Email address, Company Name, Website, Address, Industry, =
Revenue,
Employees Size, LinkedIn profile.



=D8  Once you procure the list from us, you’ll =
have the
complete ownership of the data and can use it for your unlimited =
marketing
usage. 



 



Kindly let =
me know the
target audience you would like to target, I shall send across more =
information
accordingly.



        &nb=
sp;      




Jason =
Reed



Business Development
Coordinator



Phone: +1 832 644 =
3131 



 



 



 



 



 



We =
respect your
privacy, If you don’t wish to receive further emails from us, =
please
reply with the subject “Remove”



 









------=_NextPart_000_0DF3_01D14E43.D3874CD0--

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 10:15:45 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 7F3E37F37
	for ; Wed, 13 Jan 2016 10:15:45 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 5F02F30407F
	for ; Wed, 13 Jan 2016 08:15:45 -0800 (PST)
X-ASG-Debug-ID: 1452701740-04cb6c75decb5f0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id Ins8VJN4MkbxpOSO (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 08:15:41 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from [83.175.99.196] (helo=localhost)
	by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJO5D-0000fm-VV; Wed, 13 Jan 2016 16:15:40 +0000
From: Christoph Hellwig 
To: darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Subject: a reflink fix
Date: Wed, 13 Jan 2016 17:13:48 +0100
X-ASG-Orig-Subj: a reflink fix
Message-Id: <1452701629-11537-1-git-send-email-hch@lst.de>
X-Mailer: git-send-email 1.9.1
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452701740
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26089
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

This fixes the issues I saw with generic/168 over NFS.  Turns out it
wasn't anything specific to that particular test cases, we just needed
a sufficiently aged refcount btree to trigger it.

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 10:15:51 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id D53E57F50
	for ; Wed, 13 Jan 2016 10:15:50 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 54ECDAC00C
	for ; Wed, 13 Jan 2016 08:15:47 -0800 (PST)
X-ASG-Debug-ID: 1452701744-04cbb05f7f1641d0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id vnMMUsOp9WR8G7Oy (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 08:15:45 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from [83.175.99.196] (helo=localhost)
	by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJO5H-0000gn-T1; Wed, 13 Jan 2016 16:15:44 +0000
From: Christoph Hellwig 
To: darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Subject: [PATCH] xfs: always invalidate buffer in xfs_refcountbt_free_block
Date: Wed, 13 Jan 2016 17:13:49 +0100
X-ASG-Orig-Subj: [PATCH] xfs: always invalidate buffer in xfs_refcountbt_free_block
Message-Id: <1452701629-11537-2-git-send-email-hch@lst.de>
X-Mailer: git-send-email 1.9.1
In-Reply-To: <1452701629-11537-1-git-send-email-hch@lst.de>
References: <1452701629-11537-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452701745
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26089
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

Simplify the return value from xfs_perag_pool_free_block to a bool so that
we can easily call xfs_trans_binval for both the per-AG pool and the real
freeing case.  Without this we fail to invalidate the btree buffer and
will trip over the write verifier on a shrinking refcount btree.

Signed-off-by: Christoph Hellwig 
---
 fs/xfs/libxfs/xfs_perag_pool.c     | 22 ++++++++++------------
 fs/xfs/libxfs/xfs_perag_pool.h     |  2 +-
 fs/xfs/libxfs/xfs_refcount_btree.c | 20 +++++++-------------
 3 files changed, 18 insertions(+), 26 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_perag_pool.c b/fs/xfs/libxfs/xfs_perag_pool.c
index b49ffd2..41c4e5e 100644
--- a/fs/xfs/libxfs/xfs_perag_pool.c
+++ b/fs/xfs/libxfs/xfs_perag_pool.c
@@ -338,7 +338,7 @@ err:
  * @tp: Transaction to record the free operation.
  * @bno: Block to put back.
  */
-int
+bool
 xfs_perag_pool_free_block(
 	struct xfs_perag_pool		*p,
 	struct xfs_trans		*tp,
@@ -348,7 +348,7 @@ xfs_perag_pool_free_block(
 	struct xfs_perag_pool_entry	*ppe;
 
 	if (p == NULL)
-		return -EINVAL;
+		return false;
 
 	mp = p->pp_mount;
 	mp = mp;
@@ -356,18 +356,12 @@ xfs_perag_pool_free_block(
 
 	list_for_each_entry(ppe, &p->pp_entries, ppe_list) {
 		if (ppe->ppe_bno - 1 == bno) {
-
 			/* Adjust bookkeeping. */
-			p->pp_inuse--;
 			ppe->ppe_bno--;
-			ppe->ppe_len++;
-			return 0;
-		}
-		if (ppe->ppe_bno + ppe->ppe_len == bno) {
-			p->pp_inuse--;
-			ppe->ppe_len++;
-			return 0;
+			goto adjust;
 		}
+		if (ppe->ppe_bno + ppe->ppe_len == bno)
+			goto adjust;
 	}
 	ppe = kmem_alloc(sizeof(struct xfs_perag_pool_entry), KM_SLEEP);
 	ppe->ppe_bno = bno;
@@ -375,5 +369,9 @@ xfs_perag_pool_free_block(
 	p->pp_inuse--;
 
 	list_add_tail(&ppe->ppe_list, &p->pp_entries);
-	return 0;
+	return true;
+adjust:
+	p->pp_inuse--;
+	ppe->ppe_len++;
+	return true;
 }
diff --git a/fs/xfs/libxfs/xfs_perag_pool.h b/fs/xfs/libxfs/xfs_perag_pool.h
index ecdcd2a..665ea16 100644
--- a/fs/xfs/libxfs/xfs_perag_pool.h
+++ b/fs/xfs/libxfs/xfs_perag_pool.h
@@ -43,5 +43,5 @@ int xfs_perag_pool_ensure_capacity(struct xfs_perag_pool *p, xfs_extlen_t sz);
 
 int xfs_perag_pool_alloc_block(struct xfs_perag_pool *p, struct xfs_trans *tp,
 		xfs_agblock_t *bno);
-int xfs_perag_pool_free_block(struct xfs_perag_pool *p, struct xfs_trans *tp,
+bool xfs_perag_pool_free_block(struct xfs_perag_pool *p, struct xfs_trans *tp,
 		xfs_agblock_t bno);
diff --git a/fs/xfs/libxfs/xfs_refcount_btree.c b/fs/xfs/libxfs/xfs_refcount_btree.c
index 2379488..4a31164 100644
--- a/fs/xfs/libxfs/xfs_refcount_btree.c
+++ b/fs/xfs/libxfs/xfs_refcount_btree.c
@@ -141,27 +141,21 @@ xfs_refcountbt_free_block(
 	struct xfs_perag	*pag;
 	xfs_fsblock_t		fsbno = XFS_DADDR_TO_FSB(mp, XFS_BUF_ADDR(bp));
 	struct xfs_owner_info	oinfo;
-	int			error;
+	bool			freed;
 
 	/* Try to give it back to the pool. */
 	pag = xfs_perag_get(cur->bc_mp, cur->bc_private.a.agno);
-	error = xfs_perag_pool_free_block(pag->pagf_refcountbt_pool, cur->bc_tp,
+	freed = xfs_perag_pool_free_block(pag->pagf_refcountbt_pool, cur->bc_tp,
 			XFS_FSB_TO_AGBNO(cur->bc_mp, fsbno));
 	xfs_perag_put(pag);
 
-	switch (error) {
-	case 0:
-		return 0;
-	case -EINVAL:
-		break;
-	default:
-		return error;
+	if (!freed) {
+		/* Return it to the AG. */
+		XFS_RMAP_AG_OWNER(&oinfo, XFS_RMAP_OWN_REFC);
+		xfs_bmap_add_free(mp, cur->bc_private.a.flist, fsbno, 1,
+				&oinfo);
 	}
 
-	/* Return it to the AG. */
-	XFS_RMAP_AG_OWNER(&oinfo, XFS_RMAP_OWN_REFC);
-	xfs_bmap_add_free(mp, cur->bc_private.a.flist, fsbno, 1,
-			&oinfo);
 	xfs_trans_binval(tp, bp);
 	return 0;
 }
-- 
1.9.1

From felix.janda@posteo.de  Wed Jan 13 11:12:38 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id D32767F37
	for ; Wed, 13 Jan 2016 11:12:38 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id B335C30405F
	for ; Wed, 13 Jan 2016 09:12:35 -0800 (PST)
X-ASG-Debug-ID: 1452705147-04cb6c75dbcf150001-NocioJ
Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id m3E1W6b5CSssElcA (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 13 Jan 2016 09:12:29 -0800 (PST)
X-Barracuda-Envelope-From: felix.janda@posteo.de
X-Barracuda-Apparent-Source-IP: 185.67.36.65
Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13])
	by mout01.posteo.de (Postfix) with ESMTPS id 5B1D320BBC
	for ; Wed, 13 Jan 2016 18:12:23 +0100 (CET)
Received: from mail.posteo.de (localhost [127.0.0.1])
	by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pgb1B3FZcz5vNx;
	Wed, 13 Jan 2016 18:12:14 +0100 (CET)
Date: Wed, 13 Jan 2016 18:09:38 +0100
From: Felix Janda 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
Message-ID: <20160113170938.GA10942@nyan>
X-ASG-Orig-Subj: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
References: 
 <20160112200022.GG568@nyan>
 <20160113075855.GG21939@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113075855.GG21939@infradead.org>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Barracuda-Connect: mout01.posteo.de[185.67.36.65]
X-Barracuda-Start-Time: 1452705148
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26090
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Christoph Hellwig wrote:
> I think we're having a deeper issue here.  For an LFS build we
> absolutely should have these values as the kernel dirent64 has them.
> 
> Either we manage to not pull in the LFS version of readdir/getdents
> or musl is doing something inredibly stupid.  Can you investigate if
> and why we're not using the LFS getdents64 here?

With musl you always have LFS. However still musl does not define the
_DIRENT_HAVE_*. You can see the full dirent.h at:

http://git.musl-libc.org/cgit/musl/tree/include/dirent.h

I will ask on the musl list whether these constants can be added. So
you can disregard this patch.


However notice that these constants are not defined when not on linux
or with glibc, e.g. freebsd (without k). On these systems the code in
io/readdir.c just happens to have worked because either
dirent.d_namelen exists or nobody has tested it on these systems.

Felix

From felix.janda@posteo.de  Wed Jan 13 11:45:27 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 6E1E27F37
	for ; Wed, 13 Jan 2016 11:45:27 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 4D50A8F8040
	for ; Wed, 13 Jan 2016 09:45:24 -0800 (PST)
X-ASG-Debug-ID: 1452707116-04bdf06f6cdac00001-NocioJ
Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id 0MedDD743JM9PWsZ (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 13 Jan 2016 09:45:18 -0800 (PST)
X-Barracuda-Envelope-From: felix.janda@posteo.de
X-Barracuda-Apparent-Source-IP: 185.67.36.65
Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13])
	by mout01.posteo.de (Postfix) with ESMTPS id A6FAF20AD9
	for ; Wed, 13 Jan 2016 18:45:16 +0100 (CET)
Received: from mail.posteo.de (localhost [127.0.0.1])
	by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pgblJ3pqfz5vML;
	Wed, 13 Jan 2016 18:45:16 +0100 (CET)
Date: Wed, 13 Jan 2016 18:42:41 +0100
From: Felix Janda 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
Message-ID: <20160113174241.GC10942@nyan>
X-ASG-Orig-Subj: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
References: 
 <20160112195945.GC568@nyan>
 <20160113075501.GC21939@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113075501.GC21939@infradead.org>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Barracuda-Connect: mout01.posteo.de[185.67.36.65]
X-Barracuda-Start-Time: 1452707118
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26091
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Christoph Hellwig wrote:
> On Tue, Jan 12, 2016 at 08:59:45PM +0100, Felix Janda wrote:
> > ustat has been used to check whether a device file is mounted.
> > The function is deprecated and not supported by uclibc and musl.
> > Now do the check using the *mntent functions.
> > 
> > Based on patch by Natanael Copa .
> > 
> > Signed-off-by: Felix Janda 
> 
> ustat is a lot more efficient in checking for the mountedness
> than this loop, so I'd really prefer not to do this unconditionally.

Is this really performance relevant?

Notice that linux's platform_check_iswritable does something very
similar. I've tried to make it consistent with the version from
freebsd.c.

> As far as Linux is concerned the best thing we could do is to use
> O_EXCL for any open of a device file, as Linux overloadds O_EXCL
> behavior for block devices so that any other O_EXCL open will
> be rejected if one has the file open.
> 
> But that might be a major pain to find into the cross platform
> framework..

Hmm, so we couldn't we use for linux just something like

int
platform_check_ismounted(char *name, char *block, struct stat64 *s, int verbose)
{
	int fd = open(block, O_EXCL);
	int e = errno;
	
	if (fd != -1) close(fd);
	if (e == EBUSY) {
		if (verbose)
			fprintf(stderr, _("%s: %s contains a mounted filesystem\n"), progname, name);
		return 1;
	}

	return 0;
}


Felix

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 11:52:49 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id DE93D7F37
	for ; Wed, 13 Jan 2016 11:52:49 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 8ADA7AC00C
	for ; Wed, 13 Jan 2016 09:52:46 -0800 (PST)
X-ASG-Debug-ID: 1452707564-04bdf06f6ddb210001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id UWCP9p35ZAJURWLR (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 09:52:45 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from [83.175.99.196] (helo=localhost)
	by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJPbA-0003Lc-4h
	for xfs@oss.sgi.com; Wed, 13 Jan 2016 17:52:44 +0000
From: Christoph Hellwig 
To: xfs@oss.sgi.com
Subject: a couple minor btree free_block patches
Date: Wed, 13 Jan 2016 18:50:51 +0100
X-ASG-Orig-Subj: a couple minor btree free_block patches
Message-Id: <1452707454-15229-1-git-send-email-hch@lst.de>
X-Mailer: git-send-email 1.9.1
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452707565
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26092
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

The intent of this series is to invalidate buffers for freed btree
blocks in common code instead of the methods after I found a reflink
bug due to improper buffer invalidation.  The first two patches are
things I noticed while doing that move.

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 11:52:50 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 3B32E7F3F
	for ; Wed, 13 Jan 2016 11:52:50 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id D101AAC00E
	for ; Wed, 13 Jan 2016 09:52:49 -0800 (PST)
X-ASG-Debug-ID: 1452707567-04bdf06f6edb220001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id EaKvMg6QHCGC2VJH (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 09:52:48 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from [83.175.99.196] (helo=localhost)
	by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJPbD-0003NR-1g
	for xfs@oss.sgi.com; Wed, 13 Jan 2016 17:52:47 +0000
From: Christoph Hellwig 
To: xfs@oss.sgi.com
Subject: [PATCH 1/3] xfs: handle errors from ->free_blocks in xfs_btree_kill_iroot
Date: Wed, 13 Jan 2016 18:50:52 +0100
X-ASG-Orig-Subj: [PATCH 1/3] xfs: handle errors from ->free_blocks in xfs_btree_kill_iroot
Message-Id: <1452707454-15229-2-git-send-email-hch@lst.de>
X-Mailer: git-send-email 1.9.1
In-Reply-To: <1452707454-15229-1-git-send-email-hch@lst.de>
References: <1452707454-15229-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452707567
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26092
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

Signed-off-by: Christoph Hellwig 
---
 fs/xfs/libxfs/xfs_btree.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
index a0eb18c..3143577 100644
--- a/fs/xfs/libxfs/xfs_btree.c
+++ b/fs/xfs/libxfs/xfs_btree.c
@@ -3209,6 +3209,7 @@ xfs_btree_kill_iroot(
 	int			level;
 	int			index;
 	int			numrecs;
+	int			error;
 #ifdef DEBUG
 	union xfs_btree_ptr	ptr;
 	int			i;
@@ -3272,8 +3273,6 @@ xfs_btree_kill_iroot(
 	cpp = xfs_btree_ptr_addr(cur, 1, cblock);
 #ifdef DEBUG
 	for (i = 0; i < numrecs; i++) {
-		int		error;
-
 		error = xfs_btree_check_ptr(cur, cpp, i, level - 1);
 		if (error) {
 			XFS_BTREE_TRACE_CURSOR(cur, XBT_ERROR);
@@ -3283,7 +3282,11 @@ xfs_btree_kill_iroot(
 #endif
 	xfs_btree_copy_ptrs(cur, pp, cpp, numrecs);
 
-	cur->bc_ops->free_block(cur, cbp);
+	error = cur->bc_ops->free_block(cur, cbp);
+	if (error) {
+		XFS_BTREE_TRACE_CURSOR(cur, XBT_ERROR);
+		return error;
+	}
 	XFS_BTREE_STATS_INC(cur, free);
 
 	cur->bc_bufs[level - 1] = NULL;
-- 
1.9.1

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 11:52:53 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 738D67F54
	for ; Wed, 13 Jan 2016 11:52:53 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 15275AC012
	for ; Wed, 13 Jan 2016 09:52:53 -0800 (PST)
X-ASG-Debug-ID: 1452707570-04cbb05f7e16b9a0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 5rE3gVvn8uvlGK8m (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 09:52:51 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from [83.175.99.196] (helo=localhost)
	by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJPbF-0003Oa-Vo
	for xfs@oss.sgi.com; Wed, 13 Jan 2016 17:52:50 +0000
From: Christoph Hellwig 
To: xfs@oss.sgi.com
Subject: [PATCH 2/3] xfs: factor btree block freeing into a helper
Date: Wed, 13 Jan 2016 18:50:53 +0100
X-ASG-Orig-Subj: [PATCH 2/3] xfs: factor btree block freeing into a helper
Message-Id: <1452707454-15229-3-git-send-email-hch@lst.de>
X-Mailer: git-send-email 1.9.1
In-Reply-To: <1452707454-15229-1-git-send-email-hch@lst.de>
References: <1452707454-15229-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452707570
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26092
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

Signed-off-by: Christoph Hellwig 
---
 fs/xfs/libxfs/xfs_btree.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
index 3143577..77afb4a 100644
--- a/fs/xfs/libxfs/xfs_btree.c
+++ b/fs/xfs/libxfs/xfs_btree.c
@@ -294,6 +294,19 @@ xfs_btree_sblock_verify_crc(
 	return true;
 }
 
+static int
+xfs_btree_free_block(
+	struct xfs_btree_cur	*cur,
+	struct xfs_buf		*bp)
+{
+	int			error;
+
+	error = cur->bc_ops->free_block(cur, bp);
+	if (!error)
+		XFS_BTREE_STATS_INC(cur, free);
+	return error;
+}
+
 /*
  * Delete the btree cursor.
  */
@@ -3282,12 +3295,11 @@ xfs_btree_kill_iroot(
 #endif
 	xfs_btree_copy_ptrs(cur, pp, cpp, numrecs);
 
-	error = cur->bc_ops->free_block(cur, cbp);
+	error = xfs_btree_free_block(cur, cbp);
 	if (error) {
 		XFS_BTREE_TRACE_CURSOR(cur, XBT_ERROR);
 		return error;
 	}
-	XFS_BTREE_STATS_INC(cur, free);
 
 	cur->bc_bufs[level - 1] = NULL;
 	be16_add_cpu(&block->bb_level, -1);
@@ -3320,14 +3332,12 @@ xfs_btree_kill_root(
 	 */
 	cur->bc_ops->set_root(cur, newroot, -1);
 
-	error = cur->bc_ops->free_block(cur, bp);
+	error = xfs_btree_free_block(cur, bp);
 	if (error) {
 		XFS_BTREE_TRACE_CURSOR(cur, XBT_ERROR);
 		return error;
 	}
 
-	XFS_BTREE_STATS_INC(cur, free);
-
 	cur->bc_bufs[level] = NULL;
 	cur->bc_ra[level] = 0;
 	cur->bc_nlevels--;
@@ -3833,10 +3843,9 @@ xfs_btree_delrec(
 	}
 
 	/* Free the deleted block. */
-	error = cur->bc_ops->free_block(cur, rbp);
+	error = xfs_btree_free_block(cur, rbp);
 	if (error)
 		goto error0;
-	XFS_BTREE_STATS_INC(cur, free);
 
 	/*
 	 * If we joined with the left neighbor, set the buffer in the
-- 
1.9.1

From BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org  Wed Jan 13 11:52:59 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 0EBB77F50
	for ; Wed, 13 Jan 2016 11:52:59 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D62138F8052
	for ; Wed, 13 Jan 2016 09:52:55 -0800 (PST)
X-ASG-Debug-ID: 1452707573-04bdf06f6bdb230001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 5eGIKVdXGYvaqug5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 13 Jan 2016 09:52:54 -0800 (PST)
X-Barracuda-Envelope-From: BATV+8dd500979b0e5f094249+4525+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from [83.175.99.196] (helo=localhost)
	by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJPbI-0003Q1-Ro
	for xfs@oss.sgi.com; Wed, 13 Jan 2016 17:52:53 +0000
From: Christoph Hellwig 
To: xfs@oss.sgi.com
Subject: [PATCH 3/3] xfs: move buffer invalidation to xfs_btree_free_block
Date: Wed, 13 Jan 2016 18:50:54 +0100
X-ASG-Orig-Subj: [PATCH 3/3] xfs: move buffer invalidation to xfs_btree_free_block
Message-Id: <1452707454-15229-4-git-send-email-hch@lst.de>
X-Mailer: git-send-email 1.9.1
In-Reply-To: <1452707454-15229-1-git-send-email-hch@lst.de>
References: <1452707454-15229-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452707573
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26092
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

... instead of leaving it in the methods.

Signed-off-by: Christoph Hellwig 
---
 fs/xfs/libxfs/xfs_alloc_btree.c  |  2 --
 fs/xfs/libxfs/xfs_bmap_btree.c   |  1 -
 fs/xfs/libxfs/xfs_btree.c        |  4 +++-
 fs/xfs/libxfs/xfs_ialloc_btree.c | 12 ++----------
 4 files changed, 5 insertions(+), 14 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_alloc_btree.c b/fs/xfs/libxfs/xfs_alloc_btree.c
index 444626d..d9b42425 100644
--- a/fs/xfs/libxfs/xfs_alloc_btree.c
+++ b/fs/xfs/libxfs/xfs_alloc_btree.c
@@ -118,8 +118,6 @@ xfs_allocbt_free_block(
 	xfs_extent_busy_insert(cur->bc_tp, be32_to_cpu(agf->agf_seqno), bno, 1,
 			      XFS_EXTENT_BUSY_SKIP_DISCARD);
 	xfs_trans_agbtree_delta(cur->bc_tp, -1);
-
-	xfs_trans_binval(cur->bc_tp, bp);
 	return 0;
 }
 
diff --git a/fs/xfs/libxfs/xfs_bmap_btree.c b/fs/xfs/libxfs/xfs_bmap_btree.c
index 1637c37..e37508a 100644
--- a/fs/xfs/libxfs/xfs_bmap_btree.c
+++ b/fs/xfs/libxfs/xfs_bmap_btree.c
@@ -531,7 +531,6 @@ xfs_bmbt_free_block(
 
 	xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 	xfs_trans_mod_dquot_byino(tp, ip, XFS_TRANS_DQ_BCOUNT, -1L);
-	xfs_trans_binval(tp, bp);
 	return 0;
 }
 
diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
index 77afb4a..1f88e1c 100644
--- a/fs/xfs/libxfs/xfs_btree.c
+++ b/fs/xfs/libxfs/xfs_btree.c
@@ -302,8 +302,10 @@ xfs_btree_free_block(
 	int			error;
 
 	error = cur->bc_ops->free_block(cur, bp);
-	if (!error)
+	if (!error) {
+		xfs_trans_binval(cur->bc_tp, bp);
 		XFS_BTREE_STATS_INC(cur, free);
+	}
 	return error;
 }
 
diff --git a/fs/xfs/libxfs/xfs_ialloc_btree.c b/fs/xfs/libxfs/xfs_ialloc_btree.c
index c679f3c..89c21d7 100644
--- a/fs/xfs/libxfs/xfs_ialloc_btree.c
+++ b/fs/xfs/libxfs/xfs_ialloc_btree.c
@@ -125,16 +125,8 @@ xfs_inobt_free_block(
 	struct xfs_btree_cur	*cur,
 	struct xfs_buf		*bp)
 {
-	xfs_fsblock_t		fsbno;
-	int			error;
-
-	fsbno = XFS_DADDR_TO_FSB(cur->bc_mp, XFS_BUF_ADDR(bp));
-	error = xfs_free_extent(cur->bc_tp, fsbno, 1);
-	if (error)
-		return error;
-
-	xfs_trans_binval(cur->bc_tp, bp);
-	return error;
+	return xfs_free_extent(cur->bc_tp,
+			XFS_DADDR_TO_FSB(cur->bc_mp, XFS_BUF_ADDR(bp)), 1);
 }
 
 STATIC int
-- 
1.9.1

From felix.janda@posteo.de  Wed Jan 13 11:54:04 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id F11027F37
	for ; Wed, 13 Jan 2016 11:54:03 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id C33E08F8040
	for ; Wed, 13 Jan 2016 09:54:02 -0800 (PST)
X-ASG-Debug-ID: 1452707638-04bdf06f6bdb2e0001-NocioJ
Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id HmJM1ivGfFAUoelC (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 13 Jan 2016 09:53:59 -0800 (PST)
X-Barracuda-Envelope-From: felix.janda@posteo.de
X-Barracuda-Apparent-Source-IP: 185.67.36.65
Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13])
	by mout01.posteo.de (Postfix) with ESMTPS id 4F09E20AA1
	for ; Wed, 13 Jan 2016 18:53:58 +0100 (CET)
Received: from mail.posteo.de (localhost [127.0.0.1])
	by dovecot03.posteo.de (Postfix) with ESMTPSA id 3pgbxK2ScPz5vMH;
	Wed, 13 Jan 2016 18:53:57 +0100 (CET)
Date: Wed, 13 Jan 2016 18:51:21 +0100
From: Felix Janda 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: [PATCH v2] cleanup: Kill __[u]int*_t types
Message-ID: <20160113175121.GA11266@nyan>
X-ASG-Orig-Subj: [PATCH v2] cleanup: Kill __[u]int*_t types
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113080132.GM21939@infradead.org>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Barracuda-Connect: mout01.posteo.de[185.67.36.65]
X-Barracuda-Start-Time: 1452707639
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26092
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Replace them by the stdint [u]int*_t types.

Signed-off-by: Felix Janda 
Reviewed-by: Christoph Hellwig 
---
v2: fix alignmentation
---
 common/content_inode.h | 4 ++--
 common/hsmapi.c        | 2 +-
 doc/xfsdump.html       | 8 ++++----
 inventory/inv_oref.h   | 2 +-
 inventory/inv_priv.h   | 4 ++--
 inventory/inventory.h  | 2 +-
 6 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/common/content_inode.h b/common/content_inode.h
index a69a9a0..6936e05 100644
--- a/common/content_inode.h
+++ b/common/content_inode.h
@@ -191,10 +191,10 @@ typedef struct bstat bstat_t;
  * and using two 16bit values to hold new 32bit projid was choosen
  * to retain compatibility with "old" filesystems).
  */
-static inline __uint32_t
+static inline uint32_t
 bstat_projid(struct bstat *bs)
 {
-        return (__uint32_t)bs->bs_projid_hi << 16 | bs->bs_projid_lo;
+        return (uint32_t)bs->bs_projid_hi << 16 | bs->bs_projid_lo;
 }
 
 
diff --git a/common/hsmapi.c b/common/hsmapi.c
index 6054773..0709dbe 100644
--- a/common/hsmapi.c
+++ b/common/hsmapi.c
@@ -585,7 +585,7 @@ HsmModifyExtentMap(
 	struct getbmapx	*bmap)
 {
 	dmf_f_ctxt_t	*dmf_f_ctxtp = (dmf_f_ctxt_t *)contextp;
-	__int64_t	length;
+	int64_t		length;
 
 	if (bmap[0].bmv_entries <= 0) {
 		return 1;	/* caller must already be at EOF */
diff --git a/doc/xfsdump.html b/doc/xfsdump.html
index 9c6722e..8cc02d4 100644
--- a/doc/xfsdump.html
+++ b/doc/xfsdump.html
@@ -1935,7 +1935,7 @@ The files are constructed like so:
 
 typedef struct invt_counter {
     INVT_COUNTER_FIELDS
-        __uint32_t    ic_vernum;/* on disk version number for posterity */\
+        uint32_t      ic_vernum;/* on disk version number for posterity */\
         u_int         ic_curnum;/* number of sessions/invindices recorded \
                                    so far */                              \
         u_int         ic_maxnum;/* maximum number of sessions/inv_indices \
@@ -1975,7 +1975,7 @@ typedef struct invt_fstab {
 
 typedef struct invt_counter {
     INVT_COUNTER_FIELDS
-        __uint32_t    ic_vernum;/* on disk version number for posterity */\
+        uint32_t      ic_vernum;/* on disk version number for posterity */\
         u_int         ic_curnum;/* number of sessions/invindices recorded \
                                    so far */                              \
         u_int         ic_maxnum;/* maximum number of sessions/inv_indices \
@@ -2012,7 +2012,7 @@ typedef struct invt_entry {
 
 typedef struct invt_sescounter {
     INVT_COUNTER_FIELDS
-        __uint32_t    ic_vernum;/* on disk version number for posterity */\
+        uint32_t      ic_vernum;/* on disk version number for posterity */\
         u_int         ic_curnum;/* number of sessions/invindices recorded \
                                    so far */                              \
         u_int         ic_maxnum;/* maximum number of sessions/inv_indices \
@@ -2034,7 +2034,7 @@ typedef struct invt_seshdr {
     off64_t    sh_streams_off; /* offset to start of the set of
                                   stream hdrs */
     time_t     sh_time;        /* time of the dump */
-    __uint32_t sh_flag;        /* for misc flags */
+    uint32_t   sh_flag;        /* for misc flags */
     u_char     sh_level;       /* dump level */
     u_char     sh_pruned;      /* pruned by invutil flag */
     char       sh_padding[22];
diff --git a/inventory/inv_oref.h b/inventory/inv_oref.h
index e16684d..38303a4 100644
--- a/inventory/inv_oref.h
+++ b/inventory/inv_oref.h
@@ -46,7 +46,7 @@
 
 
 
-typedef __uint32_t	invt_objtype_t;
+typedef uint32_t	invt_objtype_t;
 
 #define INVT_LOCKED		0x0001
 
diff --git a/inventory/inv_priv.h b/inventory/inv_priv.h
index 1690271..157e1dd 100644
--- a/inventory/inv_priv.h
+++ b/inventory/inv_priv.h
@@ -120,7 +120,7 @@ typedef struct invt_seshdr {
 	off64_t		sh_streams_off; /* offset to start of the set of 
 					   stream hdrs */
 	time32_t	sh_time;        /* time of the dump */
-	__uint32_t	sh_flag;        /* for misc flags */
+	uint32_t	sh_flag;        /* for misc flags */
 	u_char		sh_level;       /* dump level */
 	u_char		sh_pruned;      /* pruned by invutil flag */
 	char		sh_padding[22];
@@ -184,7 +184,7 @@ typedef struct invt_entry {
 
 /* Cheap Inheritance, and an attempt to avoid a nested type */
 #define INVT_COUNTER_FIELDS  \
-        __uint32_t    ic_vernum;/* on disk version number for posterity */\
+        uint32_t      ic_vernum;/* on disk version number for posterity */\
 	uint	      ic_curnum;/* number of sessions/invindices recorded \
 				   so far */				  \
 	uint	      ic_maxnum;/* maximum number of sessions/inv_indices \
diff --git a/inventory/inventory.h b/inventory/inventory.h
index 134b9ba..c1d7403 100644
--- a/inventory/inventory.h
+++ b/inventory/inventory.h
@@ -43,7 +43,7 @@
 /* length of labels, mntpts, etc */
 #define INV_STRLEN              GLOBAL_HDR_STRING_SZ
 
-typedef __uint32_t		inv_version_t;
+typedef uint32_t		inv_version_t;
 
 /* This is the general inventory version. 
  */
-- 
2.4.10

From sandeen@sandeen.net  Wed Jan 13 12:25:34 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 1AD357F37
	for ; Wed, 13 Jan 2016 12:25:34 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 9E8CBAC00E
	for ; Wed, 13 Jan 2016 10:25:33 -0800 (PST)
X-ASG-Debug-ID: 1452709527-04cbb05f7e16d790001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id GVPDArU5tWHP27oj for ; Wed, 13 Jan 2016 10:25:27 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 7A9E563737DF
	for ; Wed, 13 Jan 2016 12:25:27 -0600 (CST)
Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
References: <20160111232657.GA7831@birch.djwong.org>
From: Eric Sandeen 
Message-ID: <56969696.2040206@sandeen.net>
Date: Wed, 13 Jan 2016 12:25:26 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160111232657.GA7831@birch.djwong.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1452709527
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26093
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/11/16 5:26 PM, Darrick J. Wong wrote:

> diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c
> index 36bd882..31b69d1 100644
> --- a/fs/xfs/xfs_super.c
> +++ b/fs/xfs/xfs_super.c
> @@ -1812,11 +1812,56 @@ xfs_destroy_workqueues(void)
>  	destroy_workqueue(xfs_alloc_wq);
>  }
>  
> +static void __init
> +xfs_check_ondisk_structs(void)
> +{
> +	/* on-disk structures */
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dsb,			264);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agf,			224);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agi,			336);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agfl,			36);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp,		8);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dinode,		176);

I'm not sure why, but all this hard-coded stuff in the codebase bugs me.

What if we did it as part of the build, but outside the code, something
like (pardon my bad Make-fu):

diff --git a/fs/xfs/Makefile b/fs/xfs/Makefile
index f646391..da9ec6d 100644
--- a/fs/xfs/Makefile
+++ b/fs/xfs/Makefile
@@ -122,3 +122,8 @@ xfs-$(CONFIG_XFS_POSIX_ACL) += xfs_acl.o
 xfs-$(CONFIG_SYSCTL)           += xfs_sysctl.o
 xfs-$(CONFIG_COMPAT)           += xfs_ioctl32.o
 xfs-$(CONFIG_NFSD_PNFS)                += xfs_pnfs.o
+
+check_structures: $(obj)/xfs.o
+       $(src)/scripts/check_structures.sh $(obj)/xfs.o
+
+$(obj)/built-in.o: check_structures

where the script executes pahole or whatever we want, to check the
structure sizes and possibly even alignment explicitly?  Presumably
we could make running the check a Kconfig option as well...

similar tricks could surely be done for userspace as well.

-Eric

From ross.zwisler@linux.intel.com  Wed Jan 13 12:49:25 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id B5F397F37
	for ; Wed, 13 Jan 2016 12:49:25 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id A2C1830407B
	for ; Wed, 13 Jan 2016 10:49:25 -0800 (PST)
X-ASG-Debug-ID: 1452710963-04cbb05f7f16ead0001-NocioJ
Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by cuda.sgi.com with ESMTP id NeqfdVNt783Bvm9L for ; Wed, 13 Jan 2016 10:49:23 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 134.134.136.65
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
  by orsmga103.jf.intel.com with ESMTP; 13 Jan 2016 10:48:34 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,290,1449561600"; 
   d="scan'208";a="880719918"
Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.206.118])
  by fmsmga001.fm.intel.com with ESMTP; 13 Jan 2016 10:48:33 -0800
Date: Wed, 13 Jan 2016 11:48:32 -0700
From: Ross Zwisler 
To: Jan Kara 
Cc: Ross Zwisler ,
	linux-kernel@vger.kernel.org, "H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree
Message-ID: <20160113184832.GA5904@linux.intel.com>
X-ASG-Orig-Subj: Re: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree
Mail-Followup-To: Ross Zwisler ,
	Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-5-git-send-email-ross.zwisler@linux.intel.com>
 <20160113094411.GA17057@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113094411.GA17057@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mga03.intel.com[134.134.136.65]
X-Barracuda-Start-Time: 1452710963
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26093
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed, Jan 13, 2016 at 10:44:11AM +0100, Jan Kara wrote:
> On Thu 07-01-16 22:27:54, Ross Zwisler wrote:
> > Add support for tracking dirty DAX entries in the struct address_space
> > radix tree.  This tree is already used for dirty page writeback, and it
> > already supports the use of exceptional (non struct page*) entries.
> > 
> > In order to properly track dirty DAX pages we will insert new exceptional
> > entries into the radix tree that represent dirty DAX PTE or PMD pages.
> > These exceptional entries will also contain the writeback sectors for the
> > PTE or PMD faults that we can use at fsync/msync time.
> > 
> > There are currently two types of exceptional entries (shmem and shadow)
> > that can be placed into the radix tree, and this adds a third.  We rely on
> > the fact that only one type of exceptional entry can be found in a given
> > radix tree based on its usage.  This happens for free with DAX vs shmem but
> > we explicitly prevent shadow entries from being added to radix trees for
> > DAX mappings.
> > 
> > The only shadow entries that would be generated for DAX radix trees would
> > be to track zero page mappings that were created for holes.  These pages
> > would receive minimal benefit from having shadow entries, and the choice
> > to have only one type of exceptional entry in a given radix tree makes the
> > logic simpler both in clear_exceptional_entry() and in the rest of DAX.
> > 
> > Signed-off-by: Ross Zwisler 
> > Reviewed-by: Jan Kara 
> 
> I have realized there's one issue with this code. See below:
> 
> > @@ -34,31 +35,39 @@ static void clear_exceptional_entry(struct address_space *mapping,
> >  		return;
> >  
> >  	spin_lock_irq(&mapping->tree_lock);
> > -	/*
> > -	 * Regular page slots are stabilized by the page lock even
> > -	 * without the tree itself locked.  These unlocked entries
> > -	 * need verification under the tree lock.
> > -	 */
> > -	if (!__radix_tree_lookup(&mapping->page_tree, index, &node, &slot))
> > -		goto unlock;
> > -	if (*slot != entry)
> > -		goto unlock;
> > -	radix_tree_replace_slot(slot, NULL);
> > -	mapping->nrshadows--;
> > -	if (!node)
> > -		goto unlock;
> > -	workingset_node_shadows_dec(node);
> > -	/*
> > -	 * Don't track node without shadow entries.
> > -	 *
> > -	 * Avoid acquiring the list_lru lock if already untracked.
> > -	 * The list_empty() test is safe as node->private_list is
> > -	 * protected by mapping->tree_lock.
> > -	 */
> > -	if (!workingset_node_shadows(node) &&
> > -	    !list_empty(&node->private_list))
> > -		list_lru_del(&workingset_shadow_nodes, &node->private_list);
> > -	__radix_tree_delete_node(&mapping->page_tree, node);
> > +
> > +	if (dax_mapping(mapping)) {
> > +		if (radix_tree_delete_item(&mapping->page_tree, index, entry))
> > +			mapping->nrexceptional--;
> 
> So when you punch hole in a file, you can delete a PMD entry from a radix
> tree which covers part of the file which still stays. So in this case you
> have to split the PMD entry into PTE entries (probably that needs to happen
> up in truncate_inode_pages_range()) or something similar...

I think (and will verify) that the DAX code just unmaps the entire PMD range
when we receive a hole punch request inside of the PMD.  If this is true then
I think the radix tree code should behave the same way and just remove the PMD
entry in the radix tree.

This will cause new accesses that used to land in the PMD range to get new
page faults.  These faults will call get_blocks(), where presumably the
filesystem will tell us that we don't have a contiguous 2MiB range anymore, so
we will fall back to PTE faults.  These PTEs will fill in both the radix tree
and the page tables.

So, I think the work here is to verify the behavior of DAX wrt hole punches
for PMD ranges, and make the radix tree code match that behavior.  Sound good?

From ross.zwisler@linux.intel.com  Wed Jan 13 12:58:06 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 76ECE7F37
	for ; Wed, 13 Jan 2016 12:58:06 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 39CE630407B
	for ; Wed, 13 Jan 2016 10:58:06 -0800 (PST)
X-ASG-Debug-ID: 1452711484-04bdf06f6eddb70001-NocioJ
Received: from mga11.intel.com ([192.55.52.93]) by cuda.sgi.com with ESMTP id wKvaxg3RuEGBJwHP for ; Wed, 13 Jan 2016 10:58:04 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.93
X-ASG-Whitelist: EmailCat (corporate)
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
  by fmsmga102.fm.intel.com with ESMTP; 13 Jan 2016 10:58:04 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,290,1449561600"; 
   d="scan'208";a="880725285"
Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.206.118])
  by fmsmga001.fm.intel.com with ESMTP; 13 Jan 2016 10:58:03 -0800
Date: Wed, 13 Jan 2016 11:58:02 -0700
From: Ross Zwisler 
To: Jan Kara 
Cc: Ross Zwisler ,
	linux-kernel@vger.kernel.org, "H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 6/9] dax: add support for fsync/msync
Message-ID: <20160113185802.GB5904@linux.intel.com>
X-ASG-Orig-Subj: Re: [PATCH v8 6/9] dax: add support for fsync/msync
Mail-Followup-To: Ross Zwisler ,
	Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-7-git-send-email-ross.zwisler@linux.intel.com>
 <20160112105716.GT6262@quack.suse.cz>
 <20160113073019.GB30496@linux.intel.com>
 <20160113093525.GD14630@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113093525.GD14630@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.93]
X-Barracuda-Start-Time: 1452711484
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 13, 2016 at 10:35:25AM +0100, Jan Kara wrote:
> On Wed 13-01-16 00:30:19, Ross Zwisler wrote:
> > > And secondly: You must write-protect all mappings of the flushed range so
> > > that you get fault when the sector gets written-to again. We spoke about
> > > this in the past already but somehow it got lost and I forgot about it as
> > > well. You need something like rmap_walk_file()...
> > 
> > The code that write protected mappings and then cleaned the radix tree entries
> > did get written, and was part of v2:
> > 
> > https://lkml.org/lkml/2015/11/13/759
> > 
> > I removed all the code that cleaned PTE entries and radix tree entries for v3.
> > The reason behind this was that there was a race that I couldn't figure out
> > how to solve between the cleaning of the PTEs and the cleaning of the radix
> > tree entries.
> > 
> > The race goes like this:
> > 
> > Thread 1 (write)			Thread 2 (fsync)
> > ================			================
> > wp_pfn_shared()
> > pfn_mkwrite()
> > dax_radix_entry()
> > radix_tree_tag_set(DIRTY)
> > 					dax_writeback_mapping_range()
> > 					dax_writeback_one()
> > 					radix_tag_clear(DIRTY)
> > 					pgoff_mkclean()
> > ... return up to wp_pfn_shared()
> > wp_page_reuse()
> > pte_mkdirty()
> > 
> > After this sequence we end up with a dirty PTE that is writeable, but with a
> > clean radix tree entry.  This means that users can write to the page, but that
> > a follow-up fsync or msync won't flush this dirty data to media.
> > 
> > The overall issue is that in the write path that goes through wp_pfn_shared(),
> > the DAX code has control over when the radix tree entry is dirtied but not
> > when the PTE is made dirty and writeable.  This happens up in wp_page_reuse().
> > This means that we can't easily add locking, etc. to protect ourselves.
> > 
> > I spoke a bit about this with Dave Chinner and with Dave Hansen, but no really
> > easy solutions presented themselves in the absence of a page lock.  I do have
> > one idea, but I think it's pretty invasive and will need to wait for another
> > kernel cycle.
> > 
> > The current code that leaves the radix tree entry will give us correct
> > behavior - it'll just be less efficient because we will have an ever-growing
> > dirty set to flush.
> 
> Ahaa! Somehow I imagined tag_pages_for_writeback() clears DIRTY radix tree
> tags but it does not (I should have known, I have written that functions
> few years ago ;). Makes sense. Thanks for clarification.
> 
> > > > @@ -791,15 +976,12 @@ EXPORT_SYMBOL_GPL(dax_pmd_fault);
> > > >   * dax_pfn_mkwrite - handle first write to DAX page
> > > >   * @vma: The virtual memory area where the fault occurred
> > > >   * @vmf: The description of the fault
> > > > - *
> > > >   */
> > > >  int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
> > > >  {
> > > > -	struct super_block *sb = file_inode(vma->vm_file)->i_sb;
> > > > +	struct file *file = vma->vm_file;
> > > >  
> > > > -	sb_start_pagefault(sb);
> > > > -	file_update_time(vma->vm_file);
> > > > -	sb_end_pagefault(sb);
> > > > +	dax_radix_entry(file->f_mapping, vmf->pgoff, NO_SECTOR, false, true);
> > > 
> > > Why is NO_SECTOR argument correct here?
> > 
> > Right - so NO_SECTOR means "I expect there to already be an entry in the radix
> > tree - just make that entry dirty".  This works because pfn_mkwrite() always
> > follows a normal __dax_fault() or __dax_pmd_fault() call.  These fault calls
> > will insert the radix tree entry, regardless of whether the fault was for a
> > read or a write.  If the fault was for a write, the radix tree entry will also
> > be made dirty.
> >
> > For reads the radix tree entry will be inserted but left clean.  When the
> > first write happens we will get a pfn_mkwrite() call, which will call
> > dax_radix_entry() with the NO_SECTOR argument.  This will look up the radix
> > tree entry & set the dirty tag.
> 
> So the explanation of this should be somewhere so that everyone knows that
> we must have radix tree entries even for clean mapped blocks. Because upto
> know that was not clear to me.  Also __dax_pmd_fault() seems to insert
> entries only for write fault so the assumption doesn't seem to hold there?

Ah, right, sorry, the read fault() -> pfn_mkwrite() sequence only happens for
4k pages.  You are right about our handling of 2MiB pages - for a read
followed by a write we will just call into the normal __dax_pmd_fault() code
again, which will do the get_block() call and insert a dirty radix tree entry.
Because we have to go all the way through the fault handler again at write
time there isn't a benefit to inserting a clean radix tree entry on read, so
we just skip it.

> I'm somewhat uneasy that a bug in this logic can be hidden as a simple race
> with hole punching. But I guess I can live with that.
>  
> 								Honza
> -- 
> Jan Kara 
> SUSE Labs, CR

From darrick.wong@oracle.com  Wed Jan 13 14:52:11 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id CFFE07F37
	for ; Wed, 13 Jan 2016 14:52:11 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 6DD00AC004
	for ; Wed, 13 Jan 2016 12:52:11 -0800 (PST)
X-ASG-Debug-ID: 1452718329-04cbb05f7e173f90001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id 6jpQFCVF9vjM37gm (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 13 Jan 2016 12:52:09 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0DKq6Mf019910
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 13 Jan 2016 20:52:06 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0DKq6Ll026199
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 13 Jan 2016 20:52:06 GMT
Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0DKq5q6030131;
	Wed, 13 Jan 2016 20:52:06 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Wed, 13 Jan 2016 12:52:05 -0800
Date: Wed, 13 Jan 2016 12:52:04 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: a reflink fix
Message-ID: <20160113205204.GA5892@birch.djwong.org>
X-ASG-Orig-Subj: Re: a reflink fix
References: <1452701629-11537-1-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452701629-11537-1-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1452718329
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26098
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Wed, Jan 13, 2016 at 05:13:48PM +0100, Christoph Hellwig wrote:
> This fixes the issues I saw with generic/168 over NFS.  Turns out it
> wasn't anything specific to that particular test cases, we just needed
> a sufficiently aged refcount btree to trigger it.

Thanks for the patch; I've worked the fixes into the overall patchset.

I also wrote a quick reproducer which will appear as xfs/807 in my next
xfstests drop.

--D

From sandeen@redhat.com  Wed Jan 13 16:20:39 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 069C17F51
	for ; Wed, 13 Jan 2016 16:20:39 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id EAB878F8059
	for ; Wed, 13 Jan 2016 14:20:35 -0800 (PST)
X-ASG-Debug-ID: 1452723634-04cbb05f7e1782a0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id iEC6gy8yFw3XUiVl (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 13 Jan 2016 14:20:35 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id 540E01836
	for ; Wed, 13 Jan 2016 22:20:34 +0000 (UTC)
Received: from Liberator.local (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0DMKWNm010894
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for ; Wed, 13 Jan 2016 17:20:34 -0500
To: xfs@oss.sgi.com
From: Eric Sandeen 
Subject: [PATCH] xfs: remove unused function definitions
Message-ID: <5696CDB1.3060404@redhat.com>
X-ASG-Orig-Subj: [PATCH] xfs: remove unused function definitions
Date: Wed, 13 Jan 2016 16:20:33 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452723634
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

Old leftovers.

Signed-off-by: Eric Sandeen 
---

diff --git a/fs/xfs/libxfs/xfs_sb.h b/fs/xfs/libxfs/xfs_sb.h
index b25bb9a..961e647 100644
--- a/fs/xfs/libxfs/xfs_sb.h
+++ b/fs/xfs/libxfs/xfs_sb.h
@@ -27,7 +27,6 @@ extern struct xfs_perag *xfs_perag_get_tag(struct xfs_mount *, xfs_agnumber_t,
 extern void	xfs_perag_put(struct xfs_perag *pag);
 extern int	xfs_initialize_perag_data(struct xfs_mount *, xfs_agnumber_t);
 
-extern void	xfs_sb_calc_crc(struct xfs_buf *bp);
 extern void	xfs_log_sb(struct xfs_trans *tp);
 extern int	xfs_sync_sb(struct xfs_mount *mp, bool wait);
 extern void	xfs_sb_mount_common(struct xfs_mount *mp, struct xfs_sb *sbp);
diff --git a/fs/xfs/xfs_fsops.h b/fs/xfs/xfs_fsops.h
index 1b6a98b..f32713f 100644
--- a/fs/xfs/xfs_fsops.h
+++ b/fs/xfs/xfs_fsops.h
@@ -25,6 +25,5 @@ extern int xfs_fs_counts(xfs_mount_t *mp, xfs_fsop_counts_t *cnt);
 extern int xfs_reserve_blocks(xfs_mount_t *mp, __uint64_t *inval,
 				xfs_fsop_resblks_t *outval);
 extern int xfs_fs_goingdown(xfs_mount_t *mp, __uint32_t inflags);
-extern int xfs_fs_log_dummy(struct xfs_mount *mp);
 
 #endif	/* __XFS_FSOPS_H__ */
diff --git a/fs/xfs/xfs_mount.h b/fs/xfs/xfs_mount.h
index b570984..a4e03ab 100644
--- a/fs/xfs/xfs_mount.h
+++ b/fs/xfs/xfs_mount.h
@@ -327,7 +327,6 @@ extern int	xfs_mod_fdblocks(struct xfs_mount *mp, int64_t delta,
 				 bool reserved);
 extern int	xfs_mod_frextents(struct xfs_mount *mp, int64_t delta);
 
-extern int	xfs_mount_log_sb(xfs_mount_t *);
 extern struct xfs_buf *xfs_getsb(xfs_mount_t *, int);
 extern int	xfs_readsb(xfs_mount_t *, int);
 extern void	xfs_freesb(xfs_mount_t *);

From sandeen@sandeen.net  Wed Jan 13 16:40:12 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 9C0B27F37
	for ; Wed, 13 Jan 2016 16:40:12 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 8BF2F304053
	for ; Wed, 13 Jan 2016 14:40:09 -0800 (PST)
X-ASG-Debug-ID: 1452724807-04cb6c75dddd390001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id K5cF392yordEnfh7 for ; Wed, 13 Jan 2016 14:40:07 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from Liberator.local (70-90-76-85-BusName-mn.hfc.comcastbusiness.net [70.90.76.85])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 5814A60003C3
	for ; Wed, 13 Jan 2016 16:40:07 -0600 (CST)
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org> <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net> <20160111162807.GK6262@quack.suse.cz>
From: Eric Sandeen 
Message-ID: <5696D247.30809@sandeen.net>
Date: Wed, 13 Jan 2016 16:40:07 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160111162807.GK6262@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1452724807
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26101
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------



On 1/11/16 10:28 AM, Jan Kara wrote:
> On Mon 11-01-16 10:07:22, Eric Sandeen wrote:
>> On 1/11/16 7:26 AM, Jan Kara wrote:
>>> On Fri 08-01-16 23:26:00, Christoph Hellwig wrote:
>>>> On Fri, Jan 08, 2016 at 10:56:12AM -0600, Eric Sandeen wrote:
>>>>> This adds a new quotactl, Q_XGETQUOTA2.
>>>>>
>>>>> Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will
>>>>> return quota information for the id equal to or greater than
>>>>> the id requested.  In other words, if the specified id has
>>>>> no quota, the command will return quota information for the
>>>>> next higher id which does have a quota set.  If no higher id
>>>>> has an active quota, -ESRCH is returned.
>>>>
>>>> Please add a flags argument to Q_XGETQUOTA2, and then make the
>>>> new behavior the first flag.  Keep Q_XGETQUOTA behavior for the
>>>> flag-less case.  That way we get future etensibility for free.
>>>
>>> So this is what I wanted to suggest at first as well. What I somewhat
>>> dislike is that 'addr' must now point to something like:
>>>
>>> struct getquota_args {
>>> 	__u64 flags;
>>> 	struct fs_disk_quota ret;
>>> };
>>
>> Ok...
>>
>>> which is not as nice as passing pointer to fs_disk_quota directly. But
>>> probably still OK. So I agree with the flags idea.
>>
>> I understand that flags are for future-proofing, for possibly-not-yet-imagined
>> scenarios, but on the other hand, I wonder how many different flavors of
>> "get me a disk quota" we really expect to have?
>>
>>> Another issue is that OCFS2 and ext4 with quota in hidden inodes would need
>>> call with these capabilities as well (they have the same problem as xfs).
>>> For reporting VFS quota we use bytes for space (since some filesystems need
>>> it) and we don't need RT fields (easy so zero-fill) and warning fields
>>> (there zeros may be confusing). So we would need something like struct
>>> qc_dqblk (currently only internal) extended with ID and maybe flags field
>>> XFS is using to return quota type which can contain all the information and
>>> tell which info is actually valid as return structure.
>>
>> Hohum, yeah - I forgot that ext4 has (can have?) a hidden quota inode.
>> And I didn't know about OCFS2.
>>
>> And TBH I did the xfs-specific one because it was trivial to extend, with the
>> id already in the returned quota struct.
>>
>> So ok, it sounds like this needs a more significant overhaul at the vfs
>> quota level...
> 
> Actually, what I want from you is just an interface which is usable for VFS
> quotas as well since I'd like to avoid adding GETQUOTA2 quotactl shortly
> after XGETQUOTA2 :).

Actually, that's exactly what I thought would *need* to happen ... we already
have this weird 15-year-old split-brain quota interface, so if xfs and ext4
both need the same functionality, then we'd probably add both GETQUOTA2 and
XGETQUOTA2.  If we were doing this all from scratch, sure, but adding a new
handles-both-quota-types interface when every other operation is already split
between the two almost seems to make matters worse.

-Eric

From sandeen@sandeen.net  Wed Jan 13 16:41:00 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 67C757F37
	for ; Wed, 13 Jan 2016 16:41:00 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 5607A304053
	for ; Wed, 13 Jan 2016 14:41:00 -0800 (PST)
X-ASG-Debug-ID: 1452724858-04cb6c75dbdd3c0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id hBXTzHnBfP6NLy9p for ; Wed, 13 Jan 2016 14:40:59 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from Liberator.local (70-90-76-85-BusName-mn.hfc.comcastbusiness.net [70.90.76.85])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 9A8E363737DF;
	Wed, 13 Jan 2016 16:40:58 -0600 (CST)
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
To: Jan Kara 
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org> <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net> <20160111162807.GK6262@quack.suse.cz>
Cc: Christoph Hellwig ,
 fsdevel , xfs@oss.sgi.com,
 Eric Sandeen 
From: Eric Sandeen 
Message-ID: <5696D27A.9070700@sandeen.net>
Date: Wed, 13 Jan 2016 16:40:58 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160111162807.GK6262@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1452724858
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26101
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header



On 1/11/16 10:28 AM, Jan Kara wrote:
> On Mon 11-01-16 10:07:22, Eric Sandeen wrote:
>> On 1/11/16 7:26 AM, Jan Kara wrote:
>>> On Fri 08-01-16 23:26:00, Christoph Hellwig wrote:
>>>> On Fri, Jan 08, 2016 at 10:56:12AM -0600, Eric Sandeen wrote:
>>>>> This adds a new quotactl, Q_XGETQUOTA2.
>>>>>
>>>>> Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will
>>>>> return quota information for the id equal to or greater than
>>>>> the id requested.  In other words, if the specified id has
>>>>> no quota, the command will return quota information for the
>>>>> next higher id which does have a quota set.  If no higher id
>>>>> has an active quota, -ESRCH is returned.
>>>>
>>>> Please add a flags argument to Q_XGETQUOTA2, and then make the
>>>> new behavior the first flag.  Keep Q_XGETQUOTA behavior for the
>>>> flag-less case.  That way we get future etensibility for free.
>>>
>>> So this is what I wanted to suggest at first as well. What I somewhat
>>> dislike is that 'addr' must now point to something like:
>>>
>>> struct getquota_args {
>>> 	__u64 flags;
>>> 	struct fs_disk_quota ret;
>>> };
>>
>> Ok...
>>
>>> which is not as nice as passing pointer to fs_disk_quota directly. But
>>> probably still OK. So I agree with the flags idea.
>>
>> I understand that flags are for future-proofing, for possibly-not-yet-imagined
>> scenarios, but on the other hand, I wonder how many different flavors of
>> "get me a disk quota" we really expect to have?
>>
>>> Another issue is that OCFS2 and ext4 with quota in hidden inodes would need
>>> call with these capabilities as well (they have the same problem as xfs).
>>> For reporting VFS quota we use bytes for space (since some filesystems need
>>> it) and we don't need RT fields (easy so zero-fill) and warning fields
>>> (there zeros may be confusing). So we would need something like struct
>>> qc_dqblk (currently only internal) extended with ID and maybe flags field
>>> XFS is using to return quota type which can contain all the information and
>>> tell which info is actually valid as return structure.
>>
>> Hohum, yeah - I forgot that ext4 has (can have?) a hidden quota inode.
>> And I didn't know about OCFS2.
>>
>> And TBH I did the xfs-specific one because it was trivial to extend, with the
>> id already in the returned quota struct.
>>
>> So ok, it sounds like this needs a more significant overhaul at the vfs
>> quota level...
> 
> Actually, what I want from you is just an interface which is usable for VFS
> quotas as well since I'd like to avoid adding GETQUOTA2 quotactl shortly
> after XGETQUOTA2 :).

Actually, that's exactly what I thought would *need* to happen ... we already
have this weird 15-year-old split-brain quota interface, so if xfs and ext4
both need the same functionality, then we'd probably add both GETQUOTA2 and
XGETQUOTA2.  If we were doing this all from scratch, sure, but adding a new
handles-both-quota-types interface when every other operation is already split
between the two almost seems to make matters worse.

-Eric

From dave@fromorbit.com  Wed Jan 13 18:04:09 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id BB7167F37
	for ; Wed, 13 Jan 2016 18:04:09 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id AAD36304053
	for ; Wed, 13 Jan 2016 16:04:06 -0800 (PST)
X-ASG-Debug-ID: 1452729841-04cb6c75dcdf470001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id XzUFfA0EkpErDdkl for ; Wed, 13 Jan 2016 16:04:02 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2C7FQCN5ZZWPGwtLHleKAECgw9SbYZigXmfVAIWAQEBAQEBBlsBATCQDoQjhyxNAQEBAQEBBwEBAQFAAT+EYi8VDhgYagMHLYgtwGKGDolkHoUCBY1BiVSFQ4tmiy8CRI4Tgi4BCzscgXEqNIROgUoBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 14 Jan 2016 10:33:10 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJVNd-0001nK-0X; Thu, 14 Jan 2016 11:03:09 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJVNc-0001Ph-Vf; Thu, 14 Jan 2016 11:03:08 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Cc: fstests@vger.kernel.org,
	linux-ext4@vger.kernel.org
Subject: [PATCH] xfs_quota: allow operation on foreign filesystem types
Date: Thu, 14 Jan 2016 11:03:08 +1100
X-ASG-Orig-Subj: [PATCH] xfs_quota: allow operation on foreign filesystem types
Message-Id: <1452729788-5396-1-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1452729841
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.50
X-Barracuda-Spam-Status: No, SCORE=0.50 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MV0713
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26105
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.50 BSF_SC0_MV0713         Custom rule MV0713

From: Dave Chinner 

This allows xfs_quota to be used on ext4 for project quota testing
in xfstests.

Signed-off-by: Dave Chinner 
---
 include/command.h |  3 ++-
 include/path.h    |  1 +
 io/init.h         |  2 +-
 libxcmd/paths.c   |  7 +++----
 quota/free.c      |  1 +
 quota/init.c      | 16 ++++++++++++++++
 quota/path.c      |  5 +++--
 quota/project.c   |  1 +
 quota/quota.c     |  1 +
 quota/report.c    |  2 ++
 quota/state.c     |  2 ++
 11 files changed, 33 insertions(+), 8 deletions(-)

diff --git a/include/command.h b/include/command.h
index 7b9fc28..81d5a4d 100644
--- a/include/command.h
+++ b/include/command.h
@@ -20,7 +20,8 @@
 
 #include 
 
-#define CMD_FLAG_GLOBAL	((int)0x80000000)	/* don't iterate "args" */
+#define CMD_FLAG_GLOBAL		(1<<31)	/* don't iterate "args" */
+#define CMD_FLAG_FOREIGN_OK	(1<<30)	/* command not restricted to XFS */
 
 typedef int (*cfunc_t)(int argc, char **argv);
 typedef void (*helpfunc_t)(void);
diff --git a/include/path.h b/include/path.h
index 46a887e..39c1a95 100644
--- a/include/path.h
+++ b/include/path.h
@@ -29,6 +29,7 @@
 
 #define FS_MOUNT_POINT	(1<<0)
 #define FS_PROJECT_PATH	(1<<1)
+#define FS_FOREIGN	(1<<2)
 
 typedef struct fs_path {
 	char		*fs_name;	/* Data device for filesystem 	*/
diff --git a/io/init.h b/io/init.h
index d773b1b..bb25242 100644
--- a/io/init.h
+++ b/io/init.h
@@ -18,7 +18,7 @@
 
 #define CMD_NOFILE_OK	(1<<0)	/* command doesn't need an open file	*/
 #define CMD_NOMAP_OK	(1<<1)	/* command doesn't need a mapped region	*/
-#define CMD_FOREIGN_OK	(1<<2)	/* command not restricted to XFS files	*/
+#define CMD_FOREIGN_OK	CMD_FLAG_FOREIGN_OK
 
 extern char	*progname;
 extern int	exitcode;
diff --git a/libxcmd/paths.c b/libxcmd/paths.c
index 71af25f..7c8c673 100644
--- a/libxcmd/paths.c
+++ b/libxcmd/paths.c
@@ -113,6 +113,9 @@ fs_table_insert(
 			goto out_nodev;
 	}
 
+	if (!platform_test_xfs_path(dir))
+		flags |= FS_FOREIGN;
+
 	/*
 	 * Make copies of the directory and data device path.
 	 * The log device and real-time device, if non-null,
@@ -301,8 +304,6 @@ fs_table_initialise_mounts(
 			return errno;
 
 	while ((mnt = getmntent(mtp)) != NULL) {
-		if (strcmp(mnt->mnt_type, "xfs") != 0)
-			continue;
 		if (!realpath(mnt->mnt_dir, rmnt_dir))
 			continue;
 		if (!realpath(mnt->mnt_fsname, rmnt_fsname))
@@ -360,8 +361,6 @@ fs_table_initialise_mounts(
 			return errno;
 
 	for (i = 0; i < count; i++) {
-		if (strcmp(stats[i].f_fstypename, "xfs") != 0)
-			continue;
 		if (!realpath(stats[i].f_mntfromname, rmntfromname))
 			continue;
 		if (!realpath(stats[i].f_mntonname, rmntonname))
diff --git a/quota/free.c b/quota/free.c
index dcbe8ce..c7a1ec5 100644
--- a/quota/free.c
+++ b/quota/free.c
@@ -371,6 +371,7 @@ free_init(void)
 	free_cmd.args = _("[-bir] [-hn] [-f file]");
 	free_cmd.oneline = _("show free and used counts for blocks and inodes");
 	free_cmd.help = free_help;
+	free_cmd.flags = CMD_FLAG_FOREIGN_OK;
 
 	add_command(&free_cmd);
 }
diff --git a/quota/init.c b/quota/init.c
index 52f7941..f1d3328 100644
--- a/quota/init.c
+++ b/quota/init.c
@@ -92,6 +92,21 @@ init_args_command(
 	return index;
 }
 
+static int
+init_check_command(
+	const cmdinfo_t	*ct)
+{
+	if (fs_path &&
+	    !(ct->flags & CMD_FLAG_FOREIGN_OK) &&
+	     (fs_path->fs_flags & FS_FOREIGN)) {
+		fprintf(stderr,
+	_("foreign mount active, %s command is for XFS filesystems only\n"),
+			ct->name);
+		return 0;
+	}
+	return 1;
+}
+
 static void
 init(
 	int		argc,
@@ -140,6 +155,7 @@ init(
 
 	init_commands();
 	add_args_command(init_args_command);
+	add_check_command(init_check_command);
 
 	/*
 	 * Ensure that global commands don't end up with an invalid path pointer
diff --git a/quota/path.c b/quota/path.c
index bdb8c98..a623d25 100644
--- a/quota/path.c
+++ b/quota/path.c
@@ -42,6 +42,7 @@ printpath(
 	if (number) {
 		printf(_("%c%03d%c "), braces? '[':' ', index, braces? ']':' ');
 	}
+	printf("%s ", (path->fs_flags & FS_FOREIGN) ? "(F)" : "   ");
 	printf(_("%-19s %s"), path->fs_dir, path->fs_name);
 	if (path->fs_flags & FS_PROJECT_PATH) {
 		prj = getprprid(path->fs_prid);
@@ -127,7 +128,7 @@ path_init(void)
 	path_cmd.cfunc = path_f;
 	path_cmd.argmin = 0;
 	path_cmd.argmax = 1;
-	path_cmd.flags = CMD_FLAG_GLOBAL;
+	path_cmd.flags = CMD_FLAG_GLOBAL | CMD_FLAG_FOREIGN_OK;
 	path_cmd.oneline = _("set current path, or show the list of paths");
 
 	print_cmd.name = "print";
@@ -135,7 +136,7 @@ path_init(void)
 	print_cmd.cfunc = print_f;
 	print_cmd.argmin = 0;
 	print_cmd.argmax = 0;
-	print_cmd.flags = CMD_FLAG_GLOBAL;
+	print_cmd.flags = CMD_FLAG_GLOBAL | CMD_FLAG_FOREIGN_OK;
 	print_cmd.oneline = _("list known mount points and projects");
 
 	if (expert)
diff --git a/quota/project.c b/quota/project.c
index 17a83b0..79a0a8e 100644
--- a/quota/project.c
+++ b/quota/project.c
@@ -355,6 +355,7 @@ project_init(void)
 	project_cmd.argmax = -1;
 	project_cmd.oneline = _("check, setup or clear project quota trees");
 	project_cmd.help = project_help;
+	project_cmd.flags = CMD_FLAG_FOREIGN_OK;
 
 	if (expert)
 		add_command(&project_cmd);
diff --git a/quota/quota.c b/quota/quota.c
index f6b24c3..555ffd7 100644
--- a/quota/quota.c
+++ b/quota/quota.c
@@ -464,6 +464,7 @@ quota_init(void)
 	quota_cmd.args = _("[-bir] [-gpu] [-hnNv] [-f file] [id|name]...");
 	quota_cmd.oneline = _("show usage and limits");
 	quota_cmd.help = quota_help;
+	quota_cmd.flags = CMD_FLAG_FOREIGN_OK;
 
 	add_command("a_cmd);
 }
diff --git a/quota/report.c b/quota/report.c
index c77b24f..eaf3a67 100644
--- a/quota/report.c
+++ b/quota/report.c
@@ -658,6 +658,7 @@ report_init(void)
 	dump_cmd.args = _("[-gpu] [-f file]");
 	dump_cmd.oneline = _("dump quota information for backup utilities");
 	dump_cmd.help = dump_help;
+	dump_cmd.flags = CMD_FLAG_FOREIGN_OK;
 
 	report_cmd.name = "report";
 	report_cmd.altname = "repquota";
@@ -668,6 +669,7 @@ report_init(void)
 	report_cmd.args = _("[-bir] [-gpu] [-ahnt] [-f file]");
 	report_cmd.oneline = _("report filesystem quota information");
 	report_cmd.help = report_help;
+	report_cmd.flags = CMD_FLAG_FOREIGN_OK;
 
 	if (expert) {
 		add_command(&dump_cmd);
diff --git a/quota/state.c b/quota/state.c
index 8186762..9d24d74 100644
--- a/quota/state.c
+++ b/quota/state.c
@@ -527,6 +527,7 @@ state_init(void)
 	off_cmd.args = _("[-gpu] [-v]");
 	off_cmd.oneline = _("permanently switch quota off for a path");
 	off_cmd.help = off_help;
+	off_cmd.flags = CMD_FLAG_FOREIGN_OK;
 
 	state_cmd.name = "state";
 	state_cmd.cfunc = state_f;
@@ -535,6 +536,7 @@ state_init(void)
 	state_cmd.args = _("[-gpu] [-a] [-v] [-f file]");
 	state_cmd.oneline = _("get overall quota state information");
 	state_cmd.help = state_help;
+	state_cmd.flags = CMD_FLAG_FOREIGN_OK;
 
 	enable_cmd.name = "enable";
 	enable_cmd.cfunc = enable_f;
-- 
2.5.0

From david@fromorbit.com  Wed Jan 13 22:12:39 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 550F37F37
	for ; Wed, 13 Jan 2016 22:12:39 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 2671630405F
	for ; Wed, 13 Jan 2016 20:12:36 -0800 (PST)
X-ASG-Debug-ID: 1452744752-04bdf06f6ced980001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id KZPDuer0TjzUG3N6 for ; Wed, 13 Jan 2016 20:12:32 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DlCgBZH5dWPGwtLHleKAECgw9SbYJkg36BeZ9AAhwGgQ2KUIU+hAUahW8EAoE4TQEBAQEBAQcBAQEBQAE/QRIBhA4THCMYJDQFJQMHLYgtwFkZhXWKAgmEeQWHZYVciVSFQ4gOgWeERIMqhTIChWmIb4UCKjSEDz8BHgeBJAEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 14 Jan 2016 14:41:23 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aJZFr-0002Ay-91; Thu, 14 Jan 2016 15:11:23 +1100
Date: Thu, 14 Jan 2016 15:11:23 +1100
From: Dave Chinner 
To: torvalds@linux-foundation.org
Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org,
	xfs@oss.sgi.com
Subject: [GIT PULL] xfs: updates for 4.5-rc1
Message-ID: <20160114041123.GS10456@dastard>
X-ASG-Orig-Subj: [GIT PULL] xfs: updates for 4.5-rc1
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1452744752
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26110
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Hi Linus,

Can you please pull the XFS update from the location below? There's
not a lot in this - the main addition is the CRC validation of the
entire region of the log that the will be recovered, along with
several log recovery fixes. Most of the rest is small bug fixes and
cleanups.

I have 3 bug fixes still pending, all that address recently
fixed regressions that I will send to next week after they've had
some time in for-next.

-Dave.

The following changes since commit 168309855a7d1e16db751e9c647119fe2d2dc878:

  Linux 4.4-rc8 (2016-01-03 15:15:37 -0800)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git tags/xfs-for-linus-4.5

for you to fetch changes up to dde7f55bd000696acc38296c21241971e1840142:

  Merge branch 'xfs-misc-fixes-for-4.5-2' into for-next (2016-01-12 07:04:30 +1100)

----------------------------------------------------------------

xfs: updates for 4.5-rc1

This update contains:
o extensive CRC validation during log recovery
o several log recovery bug fixes
o Various DAX support fixes
o AGFL size calculation fix
o various cleanups in preparation for new functionality
o project quota ENOSPC notification via netlink
o tracing and debug improvements

----------------------------------------------------------------
Alexander Kuleshov (1):
      xfs: make xfs_buf_ioend_async() static

Brian Foster (8):
      xfs: detect and handle invalid iclog size set by mkfs
      xfs: refactor log record unpack and data processing
      xfs: refactor and open code log record crc check
      xfs: return start block of first bad log record during recovery
      xfs: support a crc verification only log record pass
      xfs: refactor log record start detection into a new helper
      xfs: detect and trim torn writes during log recovery
      xfs: debug mode log record crc error injection

Darrick J. Wong (5):
      libxfs: make xfs_alloc_fix_freelist non-static
      xfs: fix log ticket type printing
      libxfs: use a convenience variable instead of open-coding the fork
      libxfs: pack the agfl header structure so XFS_AGFL_SIZE is correct
      libxfs: refactor short btree block verification

Dave Chinner (9):
      xfs: Don't use reserved blocks for data blocks with DAX
      xfs: fix recursive splice read locking with DAX
      Merge branch 'xfs-misc-fixes-for-4.5' into for-next
      Merge branch 'xfs-dax-fixes-for-4.5' into for-next
      xfs: add tracepoints to readpage calls
      xfs: bmapbt checking on debug kernels too expensive
      xfs: inode recovery readahead can race with inode buffer creation
      xfs: handle dquot buffer readahead in log recovery correctly
      Merge branch 'xfs-misc-fixes-for-4.5-2' into for-next

Eric Sandeen (3):
      xfs: print name of verifier if it fails
      xfs: get mp from bma->ip in xfs_bmap code
      xfs: eliminate committed arg from xfs_bmap_finish

Jia He (1):
      libxfs: Optimize the loop for xfs_bitmap_empty

Markus Elfring (1):
      XFS: Use a signed return type for suffix_kstrtoint()

Masatake YAMATO (1):
      xfs: send warning of project quota to userspace via netlink

 fs/xfs/libxfs/xfs_alloc.c          |    4 +-
 fs/xfs/libxfs/xfs_alloc.h          |    1 +
 fs/xfs/libxfs/xfs_alloc_btree.c    |   35 +--
 fs/xfs/libxfs/xfs_attr.c           |  141 ++-------
 fs/xfs/libxfs/xfs_attr_leaf.c      |    1 +
 fs/xfs/libxfs/xfs_attr_remote.c    |   32 +-
 fs/xfs/libxfs/xfs_bit.c            |    6 +-
 fs/xfs/libxfs/xfs_bmap.c           |   43 +--
 fs/xfs/libxfs/xfs_bmap.h           |    2 +-
 fs/xfs/libxfs/xfs_bmap_btree.c     |    1 +
 fs/xfs/libxfs/xfs_btree.c          |   58 ++++
 fs/xfs/libxfs/xfs_btree.h          |    3 +
 fs/xfs/libxfs/xfs_da_btree.c       |    1 +
 fs/xfs/libxfs/xfs_dir2_block.c     |    1 +
 fs/xfs/libxfs/xfs_dir2_data.c      |    2 +
 fs/xfs/libxfs/xfs_dir2_leaf.c      |    2 +
 fs/xfs/libxfs/xfs_dir2_node.c      |    1 +
 fs/xfs/libxfs/xfs_dquot_buf.c      |   37 ++-
 fs/xfs/libxfs/xfs_format.h         |    2 +-
 fs/xfs/libxfs/xfs_ialloc.c         |    1 +
 fs/xfs/libxfs/xfs_ialloc_btree.c   |   27 +-
 fs/xfs/libxfs/xfs_inode_buf.c      |   16 +-
 fs/xfs/libxfs/xfs_log_recover.h    |    1 +
 fs/xfs/libxfs/xfs_quota_defs.h     |    2 +-
 fs/xfs/libxfs/xfs_sb.c             |    2 +
 fs/xfs/libxfs/xfs_shared.h         |    1 +
 fs/xfs/libxfs/xfs_symlink_remote.c |    1 +
 fs/xfs/xfs_aops.c                  |    2 +
 fs/xfs/xfs_bmap_util.c             |   43 +--
 fs/xfs/xfs_buf.c                   |    9 +-
 fs/xfs/xfs_buf.h                   |    1 +
 fs/xfs/xfs_dquot.c                 |   13 +-
 fs/xfs/xfs_error.c                 |    4 +-
 fs/xfs/xfs_file.c                  |   25 +-
 fs/xfs/xfs_inode.c                 |   25 +-
 fs/xfs/xfs_iomap.c                 |   21 +-
 fs/xfs/xfs_log.c                   |   51 +++-
 fs/xfs/xfs_log_priv.h              |    3 +
 fs/xfs/xfs_log_recover.c           |  581 +++++++++++++++++++++++++++++-------
 fs/xfs/xfs_rtalloc.c               |    3 +-
 fs/xfs/xfs_super.c                 |    2 +-
 fs/xfs/xfs_symlink.c               |   12 +-
 fs/xfs/xfs_sysfs.c                 |   36 +++
 fs/xfs/xfs_trace.h                 |   26 ++
 fs/xfs/xfs_trans_dquot.c           |   14 +-
 45 files changed, 852 insertions(+), 443 deletions(-)

-- 
Dave Chinner
david@fromorbit.com

From dave@fromorbit.com  Thu Jan 14 00:10:26 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 83A197F47
	for ; Thu, 14 Jan 2016 00:10:26 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 491278F804B
	for ; Wed, 13 Jan 2016 22:10:26 -0800 (PST)
X-ASG-Debug-ID: 1452751821-04bdf06f6eefa50001-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id 2XjZSHjzYxlfD4z6 for ; Wed, 13 Jan 2016 22:10:21 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DsBwA8O5dWPGwtLHleKAECgw+BP4ZigXmfXgaBZo81ihSBO00BAQEBAQEHAQEBAUABP4Q1AQUnLzMIGBkYOQMHFBmILcA/hg6KRCIMhBIFh2WPMJxajliCLgELATcDHIFxKjSETwcXB4EkAQEB
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:29 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OW-NQ
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001VB-Mf
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH 5/8] xfs: use vfs inode nlink field everywhere
Date: Thu, 14 Jan 2016 17:09:22 +1100
X-ASG-Orig-Subj: [PATCH 5/8] xfs: use vfs inode nlink field everywhere
Message-Id: <1452751765-4420-6-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
In-Reply-To: <1452751765-4420-1-git-send-email-david@fromorbit.com>
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751821
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

The Vfs tracks the inodenlink just like the xfs_icdinode. We can
remove the variable from the icdinode and use the vfs inode variable
everywhere, reducing the size of the xfs_icdinode by a further 4
bytes.

Signed-off-by: Dave Chinner 
---
 fs/xfs/libxfs/xfs_inode_buf.c |  6 ++--
 fs/xfs/libxfs/xfs_inode_buf.h |  1 -
 fs/xfs/xfs_inode.c            | 73 ++++++++++++++++++++-----------------------
 fs/xfs/xfs_inode.h            |  2 --
 fs/xfs/xfs_inode_item.c       |  2 +-
 fs/xfs/xfs_iops.c             |  3 +-
 fs/xfs/xfs_itable.c           |  2 +-
 fs/xfs/xfs_log_recover.c      |  2 +-
 8 files changed, 41 insertions(+), 50 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
index 8a1b460..0dcaa9a 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.c
+++ b/fs/xfs/libxfs/xfs_inode_buf.c
@@ -210,12 +210,12 @@ xfs_inode_from_disk(
 	 * minimum inode version format we support in the rest of the code.
 	 */
 	if (to->di_version == 1) {
-		to->di_nlink = be16_to_cpu(from->di_onlink);
+		set_nlink(inode, be16_to_cpu(from->di_onlink));
 		to->di_projid_lo = 0;
 		to->di_projid_hi = 0;
 		to->di_version = 2;
 	} else {
-		to->di_nlink = be32_to_cpu(from->di_nlink);
+		set_nlink(inode, be32_to_cpu(from->di_nlink));
 		to->di_projid_lo = be16_to_cpu(from->di_projid_lo);
 		to->di_projid_hi = be16_to_cpu(from->di_projid_hi);
 	}
@@ -275,7 +275,6 @@ xfs_inode_to_disk(
 	to->di_format = from->di_format;
 	to->di_uid = cpu_to_be32(from->di_uid);
 	to->di_gid = cpu_to_be32(from->di_gid);
-	to->di_nlink = cpu_to_be32(from->di_nlink);
 	to->di_projid_lo = cpu_to_be16(from->di_projid_lo);
 	to->di_projid_hi = cpu_to_be16(from->di_projid_hi);
 
@@ -286,6 +285,7 @@ xfs_inode_to_disk(
 	to->di_mtime.t_nsec = cpu_to_be32(inode->i_mtime.tv_nsec);
 	to->di_ctime.t_sec = cpu_to_be32(inode->i_ctime.tv_sec);
 	to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec);
+	to->di_nlink = cpu_to_be32(inode->i_nlink);
 
 	to->di_size = cpu_to_be64(from->di_size);
 	to->di_nblocks = cpu_to_be64(from->di_nblocks);
diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index feb04e6..774d71f 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -34,7 +34,6 @@ struct xfs_icdinode {
 	__uint16_t	di_flushiter;	/* incremented on flush */
 	__uint32_t	di_uid;		/* owner's user id */
 	__uint32_t	di_gid;		/* owner's group id */
-	__uint32_t	di_nlink;	/* number of links to file */
 	__uint16_t	di_projid_lo;	/* lower part of owner's project id */
 	__uint16_t	di_projid_hi;	/* higher part of owner's project id */
 	xfs_fsize_t	di_size;	/* number of bytes in file */
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 914ec41..9ee766b 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -58,7 +58,7 @@ kmem_zone_t *xfs_inode_zone;
 #define	XFS_ITRUNC_MAX_EXTENTS	2
 
 STATIC int xfs_iflush_int(xfs_inode_t *, xfs_buf_t *);
-
+STATIC int xfs_iunlink(xfs_trans_t *, xfs_inode_t *, bool);
 STATIC int xfs_iunlink_remove(xfs_trans_t *, xfs_inode_t *);
 
 /*
@@ -794,7 +794,7 @@ xfs_ialloc(
 		ip->i_d.di_version = 2;
 
 	ip->i_d.di_mode = mode;
-	ip->i_d.di_nlink = nlink;
+	set_nlink(inode, nlink);
 	ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid());
 	ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid());
 	xfs_set_projid(ip, prid);
@@ -1072,35 +1072,24 @@ xfs_dir_ialloc(
 }
 
 /*
- * Decrement the link count on an inode & log the change.
- * If this causes the link count to go to zero, initiate the
- * logging activity required to truncate a file.
+ * Decrement the link count on an inode & log the change.  If this causes the
+ * link count to go to zero, move the inode to AGI unlinked list so that it can
+ * be freed when the last active reference goes away via xfs_inactive().
  */
 int				/* error */
 xfs_droplink(
 	xfs_trans_t *tp,
 	xfs_inode_t *ip)
 {
-	int	error;
-
 	xfs_trans_ichgtime(tp, ip, XFS_ICHGTIME_CHG);
 
-	ASSERT (ip->i_d.di_nlink > 0);
-	ip->i_d.di_nlink--;
 	drop_nlink(VFS_I(ip));
 	xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 
-	error = 0;
-	if (ip->i_d.di_nlink == 0) {
-		/*
-		 * We're dropping the last link to this file.
-		 * Move the on-disk inode to the AGI unlinked list.
-		 * From xfs_inactive() we will pull the inode from
-		 * the list and free it.
-		 */
-		error = xfs_iunlink(tp, ip);
-	}
-	return error;
+	if (VFS_I(ip)->i_nlink)
+		return 0;
+
+	return xfs_iunlink(tp, ip, false);
 }
 
 /*
@@ -1114,8 +1103,6 @@ xfs_bumplink(
 	xfs_trans_ichgtime(tp, ip, XFS_ICHGTIME_CHG);
 
 	ASSERT(ip->i_d.di_version > 1);
-	ASSERT(ip->i_d.di_nlink > 0 || (VFS_I(ip)->i_state & I_LINKABLE));
-	ip->i_d.di_nlink++;
 	inc_nlink(VFS_I(ip));
 	xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 	return 0;
@@ -1373,8 +1360,7 @@ xfs_create_tmpfile(
 	 */
 	xfs_qm_vop_create_dqattach(tp, ip, udqp, gdqp, pdqp);
 
-	ip->i_d.di_nlink--;
-	error = xfs_iunlink(tp, ip);
+	error = xfs_iunlink(tp, ip, true);
 	if (error)
 		goto out_trans_cancel;
 
@@ -1472,7 +1458,10 @@ xfs_link(
 
 	xfs_bmap_init(&free_list, &first_block);
 
-	if (sip->i_d.di_nlink == 0) {
+	/*
+	 * Handle initial link state of O_TMPFILE inode
+	 */
+	if (VFS_I(sip)->i_nlink == 0) {
 		error = xfs_iunlink_remove(tp, sip);
 		if (error)
 			goto error_return;
@@ -1659,7 +1648,7 @@ xfs_release(
 		}
 	}
 
-	if (ip->i_d.di_nlink == 0)
+	if (VFS_I(ip)->i_nlink == 0)
 		return 0;
 
 	if (xfs_can_free_eofblocks(ip, false)) {
@@ -1875,7 +1864,7 @@ xfs_inactive(
 	if (mp->m_flags & XFS_MOUNT_RDONLY)
 		return;
 
-	if (ip->i_d.di_nlink != 0) {
+	if (VFS_I(ip)->i_nlink != 0) {
 		/*
 		 * force is true because we are evicting an inode from the
 		 * cache. Post-eof blocks must be freed, lest we end up with
@@ -1932,14 +1921,20 @@ xfs_inactive(
 }
 
 /*
- * This is called when the inode's link count goes to 0.
- * We place the on-disk inode on a list in the AGI.  It
- * will be pulled from this list when the inode is freed.
+ * This is called when the inode's link count goes to 0 or we are creating a
+ * tmpfile via O_TMPFILE. In the case of a tmpfile, @ignore_linkcount will be
+ * set to true as the link count is dropped to zero by the VFS after we've
+ * created the file successfully, so we have to add it to the unlinked list
+ * while the link count is non-zero.
+ *
+ * We place the on-disk inode on a list in the AGI.  It will be pulled from this
+ * list when the inode is freed.
  */
-int
+STATIC int
 xfs_iunlink(
-	xfs_trans_t	*tp,
-	xfs_inode_t	*ip)
+	struct xfs_trans *tp,
+	struct xfs_inode *ip,
+	bool		ignore_linkcount)
 {
 	xfs_mount_t	*mp;
 	xfs_agi_t	*agi;
@@ -1951,7 +1946,7 @@ xfs_iunlink(
 	int		offset;
 	int		error;
 
-	ASSERT(ip->i_d.di_nlink == 0);
+	ASSERT(VFS_I(ip)->i_nlink == 0 || ignore_linkcount);
 	ASSERT(ip->i_d.di_mode != 0);
 
 	mp = tp->t_mountp;
@@ -2392,7 +2387,7 @@ xfs_ifree(
 	struct xfs_icluster	xic = { 0 };
 
 	ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL));
-	ASSERT(ip->i_d.di_nlink == 0);
+	ASSERT(VFS_I(ip)->i_nlink == 0);
 	ASSERT(ip->i_d.di_nextents == 0);
 	ASSERT(ip->i_d.di_anextents == 0);
 	ASSERT(ip->i_d.di_size == 0 || !S_ISREG(ip->i_d.di_mode));
@@ -2560,8 +2555,8 @@ xfs_remove(
 	 * If we're removing a directory perform some additional validation.
 	 */
 	if (is_dir) {
-		ASSERT(ip->i_d.di_nlink >= 2);
-		if (ip->i_d.di_nlink != 2) {
+		ASSERT(VFS_I(ip)->i_nlink >= 2);
+		if (VFS_I(ip)->i_nlink != 2) {
 			error = -ENOTEMPTY;
 			goto out_trans_cancel;
 		}
@@ -3017,7 +3012,7 @@ xfs_rename(
 			 * Make sure target dir is empty.
 			 */
 			if (!(xfs_dir_isempty(target_ip)) ||
-			    (target_ip->i_d.di_nlink > 2)) {
+			    (VFS_I(target_ip)->i_nlink > 2)) {
 				error = -EEXIST;
 				goto out_trans_cancel;
 			}
@@ -3124,7 +3119,7 @@ xfs_rename(
 	 * intermediate state on disk.
 	 */
 	if (wip) {
-		ASSERT(VFS_I(wip)->i_nlink == 0 && wip->i_d.di_nlink == 0);
+		ASSERT(VFS_I(wip)->i_nlink == 0);
 		error = xfs_bumplink(tp, wip);
 		if (error)
 			goto out_bmap_cancel;
diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
index aef5452..e74d13d 100644
--- a/fs/xfs/xfs_inode.h
+++ b/fs/xfs/xfs_inode.h
@@ -405,8 +405,6 @@ int		xfs_ifree(struct xfs_trans *, xfs_inode_t *,
 			   struct xfs_bmap_free *);
 int		xfs_itruncate_extents(struct xfs_trans **, struct xfs_inode *,
 				      int, xfs_fsize_t);
-int		xfs_iunlink(struct xfs_trans *, xfs_inode_t *);
-
 void		xfs_iext_realloc(xfs_inode_t *, int, int);
 
 void		xfs_iunpin_wait(xfs_inode_t *);
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index d777b7a..4e3611d 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -338,7 +338,6 @@ xfs_inode_to_log_dinode(
 	to->di_format = from->di_format;
 	to->di_uid = from->di_uid;
 	to->di_gid = from->di_gid;
-	to->di_nlink = from->di_nlink;
 	to->di_projid_lo = from->di_projid_lo;
 	to->di_projid_hi = from->di_projid_hi;
 
@@ -350,6 +349,7 @@ xfs_inode_to_log_dinode(
 	to->di_mtime.t_nsec = inode->i_mtime.tv_nsec;
 	to->di_ctime.t_sec = inode->i_ctime.tv_sec;
 	to->di_ctime.t_nsec = inode->i_ctime.tv_nsec;
+	to->di_nlink = inode->i_nlink;
 
 	to->di_size = from->di_size;
 	to->di_nblocks = from->di_nblocks;
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index b008677..c424d4b 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -455,7 +455,7 @@ xfs_vn_getattr(
 	stat->size = XFS_ISIZE(ip);
 	stat->dev = inode->i_sb->s_dev;
 	stat->mode = ip->i_d.di_mode;
-	stat->nlink = ip->i_d.di_nlink;
+	stat->nlink = inode->i_nlink;
 	stat->uid = inode->i_uid;
 	stat->gid = inode->i_gid;
 	stat->ino = ip->i_ino;
@@ -1212,7 +1212,6 @@ xfs_setup_inode(
 	hlist_add_fake(&inode->i_hash);
 
 	inode->i_mode	= ip->i_d.di_mode;
-	set_nlink(inode, ip->i_d.di_nlink);
 	inode->i_uid    = xfs_uid_to_kuid(ip->i_d.di_uid);
 	inode->i_gid    = xfs_gid_to_kgid(ip->i_d.di_gid);
 
diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
index 2acda42..cfb6527 100644
--- a/fs/xfs/xfs_itable.c
+++ b/fs/xfs/xfs_itable.c
@@ -85,7 +85,6 @@ xfs_bulkstat_one_int(
 	/* xfs_iget returns the following without needing
 	 * further change.
 	 */
-	buf->bs_nlink = dic->di_nlink;
 	buf->bs_projid_lo = dic->di_projid_lo;
 	buf->bs_projid_hi = dic->di_projid_hi;
 	buf->bs_ino = ino;
@@ -94,6 +93,7 @@ xfs_bulkstat_one_int(
 	buf->bs_gid = dic->di_gid;
 	buf->bs_size = dic->di_size;
 
+	buf->bs_nlink = inode->i_nlink;
 	buf->bs_atime.tv_sec = inode->i_atime.tv_sec;
 	buf->bs_atime.tv_nsec = inode->i_atime.tv_nsec;
 	buf->bs_mtime.tv_sec = inode->i_mtime.tv_sec;
diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
index 4b79cf0..611c25c 100644
--- a/fs/xfs/xfs_log_recover.c
+++ b/fs/xfs/xfs_log_recover.c
@@ -4337,7 +4337,7 @@ xlog_recover_process_one_iunlink(
 	if (error)
 		goto fail_iput;
 
-	ASSERT(ip->i_d.di_nlink == 0);
+	ASSERT(VFS_I(ip)->i_nlink == 0);
 	ASSERT(ip->i_d.di_mode != 0);
 
 	/* setup for the next pass */
-- 
2.5.0

From dave@fromorbit.com  Thu Jan 14 00:10:28 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 75BB07F50
	for ; Thu, 14 Jan 2016 00:10:27 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 416688F8054
	for ; Wed, 13 Jan 2016 22:10:27 -0800 (PST)
X-ASG-Debug-ID: 1452751821-04bdf06f6eefa50002-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id VjDMG9E28HAketNn for ; Wed, 13 Jan 2016 22:10:24 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DrBwA8O5dWPGwtLHleKAECgw+BP4ZigXmfXgaRG4oUgTtNAQEBAQEBBwEBAQFAAT+ENQEFJy8zCBgxOQMHFBmILcA/hg6KRIRABZcVnFpEjhSCLgELOB+BcSo0hhgBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:30 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OY-OS
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001VL-NX
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH 7/8] xfs: move di_changecount to VFS inode
Date: Thu, 14 Jan 2016 17:09:24 +1100
X-ASG-Orig-Subj: [PATCH 7/8] xfs: move di_changecount to VFS inode
Message-Id: <1452751765-4420-8-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
In-Reply-To: <1452751765-4420-1-git-send-email-david@fromorbit.com>
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751823
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

We can store the di_changecount in the i_version field of the VFS
inode and remove another 8 bytes from the xfs_icdinode.

Signed-off-by: Dave Chinner 
---
 fs/xfs/libxfs/xfs_inode_buf.c | 4 ++--
 fs/xfs/libxfs/xfs_inode_buf.h | 1 -
 fs/xfs/xfs_inode.c            | 2 +-
 fs/xfs/xfs_inode_item.c       | 2 +-
 fs/xfs/xfs_trans_inode.c      | 2 +-
 5 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
index 0a3f10b..9807c21 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.c
+++ b/fs/xfs/libxfs/xfs_inode_buf.c
@@ -251,7 +251,7 @@ xfs_inode_from_disk(
 	to->di_flags	= be16_to_cpu(from->di_flags);
 
 	if (to->di_version == 3) {
-		to->di_changecount = be64_to_cpu(from->di_changecount);
+		inode->i_version = be64_to_cpu(from->di_changecount);
 		to->di_crtime.t_sec = be32_to_cpu(from->di_crtime.t_sec);
 		to->di_crtime.t_nsec = be32_to_cpu(from->di_crtime.t_nsec);
 		to->di_flags2 = be64_to_cpu(from->di_flags2);
@@ -300,7 +300,7 @@ xfs_inode_to_disk(
 	to->di_flags = cpu_to_be16(from->di_flags);
 
 	if (from->di_version == 3) {
-		to->di_changecount = cpu_to_be64(from->di_changecount);
+		to->di_changecount = cpu_to_be64(inode->i_version);
 		to->di_crtime.t_sec = cpu_to_be32(from->di_crtime.t_sec);
 		to->di_crtime.t_nsec = cpu_to_be32(from->di_crtime.t_nsec);
 		to->di_flags2 = cpu_to_be64(from->di_flags2);
diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index 68da576..e49c229 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -47,7 +47,6 @@ struct xfs_icdinode {
 	__uint16_t	di_dmstate;	/* DMIG state info */
 	__uint16_t	di_flags;	/* random flags, XFS_DIFLAG_... */
 
-	__uint64_t	di_changecount;	/* number of attribute changes */
 	__uint64_t	di_flags2;	/* more random flags */
 
 	xfs_ictimestamp_t di_crtime;	/* time created */
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 391d676..96aee23 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -832,7 +832,7 @@ xfs_ialloc(
 	ip->i_d.di_flags = 0;
 
 	if (ip->i_d.di_version == 3) {
-		ip->i_d.di_changecount = 1;
+		inode->i_version = 1;
 		ip->i_d.di_flags2 = 0;
 		ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec;
 		ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec;
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index 5cdf5ef..334d657 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -364,7 +364,7 @@ xfs_inode_to_log_dinode(
 	to->di_flags = from->di_flags;
 
 	if (from->di_version == 3) {
-		to->di_changecount = from->di_changecount;
+		to->di_changecount = inode->i_version;
 		to->di_crtime.t_sec = from->di_crtime.t_sec;
 		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
 		to->di_flags2 = from->di_flags2;
diff --git a/fs/xfs/xfs_trans_inode.c b/fs/xfs/xfs_trans_inode.c
index 3f0d466..11a3af0 100644
--- a/fs/xfs/xfs_trans_inode.c
+++ b/fs/xfs/xfs_trans_inode.c
@@ -117,7 +117,7 @@ xfs_trans_log_inode(
 	 */
 	if (!(ip->i_itemp->ili_item.li_desc->lid_flags & XFS_LID_DIRTY) &&
 	    IS_I_VERSION(VFS_I(ip))) {
-		ip->i_d.di_changecount = ++VFS_I(ip)->i_version;
+		VFS_I(ip)->i_version++;
 		flags |= XFS_ILOG_CORE;
 	}
 
-- 
2.5.0

From dave@fromorbit.com  Thu Jan 14 00:10:28 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 8AFB37F51
	for ; Thu, 14 Jan 2016 00:10:27 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 1D55DAC003
	for ; Wed, 13 Jan 2016 22:10:23 -0800 (PST)
X-ASG-Debug-ID: 1452751818-04bdf06f6eefa40002-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id AVrF98SkvWJjGrbD for ; Wed, 13 Jan 2016 22:10:20 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DrBwA8O5dWPGwtLHleKAECgw+BP4ZigXmfXgaRG4oUgTtNAQEBAQEBBwEBAQFAAT+ENQEFJy8zCBgxOQMHFBmILcA/hg6KRIRABZcVnFqOWIIuAQs4H4FxKjSGGAEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:29 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OX-Nm
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001VG-N8
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH 6/8] xfs: move inode generation count to VFS inode
Date: Thu, 14 Jan 2016 17:09:23 +1100
X-ASG-Orig-Subj: [PATCH 6/8] xfs: move inode generation count to VFS inode
Message-Id: <1452751765-4420-7-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
In-Reply-To: <1452751765-4420-1-git-send-email-david@fromorbit.com>
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751820
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

Pull another 4 bytes out of the xfs_icdinode.

Signed-off-by: Dave Chinner 
---
 fs/xfs/libxfs/xfs_inode_buf.c | 8 ++++----
 fs/xfs/libxfs/xfs_inode_buf.h | 1 -
 fs/xfs/xfs_export.c           | 2 +-
 fs/xfs/xfs_inode.c            | 5 +----
 fs/xfs/xfs_inode_item.c       | 2 +-
 fs/xfs/xfs_ioctl.c            | 2 +-
 fs/xfs/xfs_iops.c             | 1 -
 fs/xfs/xfs_itable.c           | 2 +-
 8 files changed, 9 insertions(+), 14 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
index 0dcaa9a..0a3f10b 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.c
+++ b/fs/xfs/libxfs/xfs_inode_buf.c
@@ -237,6 +237,7 @@ xfs_inode_from_disk(
 	inode->i_mtime.tv_nsec = (int)be32_to_cpu(from->di_mtime.t_nsec);
 	inode->i_ctime.tv_sec = (int)be32_to_cpu(from->di_ctime.t_sec);
 	inode->i_ctime.tv_nsec = (int)be32_to_cpu(from->di_ctime.t_nsec);
+	inode->i_generation = be32_to_cpu(from->di_gen);
 
 	to->di_size = be64_to_cpu(from->di_size);
 	to->di_nblocks = be64_to_cpu(from->di_nblocks);
@@ -248,7 +249,6 @@ xfs_inode_from_disk(
 	to->di_dmevmask	= be32_to_cpu(from->di_dmevmask);
 	to->di_dmstate	= be16_to_cpu(from->di_dmstate);
 	to->di_flags	= be16_to_cpu(from->di_flags);
-	to->di_gen	= be32_to_cpu(from->di_gen);
 
 	if (to->di_version == 3) {
 		to->di_changecount = be64_to_cpu(from->di_changecount);
@@ -286,6 +286,7 @@ xfs_inode_to_disk(
 	to->di_ctime.t_sec = cpu_to_be32(inode->i_ctime.tv_sec);
 	to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec);
 	to->di_nlink = cpu_to_be32(inode->i_nlink);
+	to->di_gen = cpu_to_be32(inode->i_generation);
 
 	to->di_size = cpu_to_be64(from->di_size);
 	to->di_nblocks = cpu_to_be64(from->di_nblocks);
@@ -297,7 +298,6 @@ xfs_inode_to_disk(
 	to->di_dmevmask = cpu_to_be32(from->di_dmevmask);
 	to->di_dmstate = cpu_to_be16(from->di_dmstate);
 	to->di_flags = cpu_to_be16(from->di_flags);
-	to->di_gen = cpu_to_be32(from->di_gen);
 
 	if (from->di_version == 3) {
 		to->di_changecount = cpu_to_be64(from->di_changecount);
@@ -443,7 +443,7 @@ xfs_iread(
 	    !(mp->m_flags & XFS_MOUNT_IKEEP)) {
 		/* initialise the on-disk inode core */
 		memset(&ip->i_d, 0, sizeof(ip->i_d));
-		ip->i_d.di_gen = prandom_u32();
+		VFS_I(ip)->i_generation = prandom_u32();
 		if (xfs_sb_version_hascrc(&mp->m_sb))
 			ip->i_d.di_version = 3;
 		else
@@ -491,7 +491,7 @@ xfs_iread(
 		 * that xfs_ialloc won't overwrite or relies on being correct.
 		 */
 		ip->i_d.di_version = dip->di_version;
-		ip->i_d.di_gen = be32_to_cpu(dip->di_gen);
+		VFS_I(ip)->i_generation = be32_to_cpu(dip->di_gen);
 		ip->i_d.di_flushiter = be16_to_cpu(dip->di_flushiter);
 
 		/*
diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index 774d71f..68da576 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -46,7 +46,6 @@ struct xfs_icdinode {
 	__uint32_t	di_dmevmask;	/* DMIG event mask */
 	__uint16_t	di_dmstate;	/* DMIG state info */
 	__uint16_t	di_flags;	/* random flags, XFS_DIFLAG_... */
-	__uint32_t	di_gen;		/* generation number */
 
 	__uint64_t	di_changecount;	/* number of attribute changes */
 	__uint64_t	di_flags2;	/* more random flags */
diff --git a/fs/xfs/xfs_export.c b/fs/xfs/xfs_export.c
index 652cd3c..2816d42 100644
--- a/fs/xfs/xfs_export.c
+++ b/fs/xfs/xfs_export.c
@@ -152,7 +152,7 @@ xfs_nfs_get_inode(
 		return ERR_PTR(error);
 	}
 
-	if (ip->i_d.di_gen != generation) {
+	if (VFS_I(ip)->i_generation != generation) {
 		IRELE(ip);
 		return ERR_PTR(-ESTALE);
 	}
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 9ee766b..391d676 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -826,9 +826,6 @@ xfs_ialloc(
 	inode->i_atime = tv;
 	inode->i_ctime = tv;
 
-	/*
-	 * di_gen will have been taken care of in xfs_iread.
-	 */
 	ip->i_d.di_extsize = 0;
 	ip->i_d.di_dmevmask = 0;
 	ip->i_d.di_dmstate = 0;
@@ -2414,7 +2411,7 @@ xfs_ifree(
 	 * Bump the generation count so no one will be confused
 	 * by reincarnations of this inode.
 	 */
-	ip->i_d.di_gen++;
+	VFS_I(ip)->i_generation++;
 	xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 
 	if (xic.deleted)
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index 4e3611d..5cdf5ef 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -350,6 +350,7 @@ xfs_inode_to_log_dinode(
 	to->di_ctime.t_sec = inode->i_ctime.tv_sec;
 	to->di_ctime.t_nsec = inode->i_ctime.tv_nsec;
 	to->di_nlink = inode->i_nlink;
+	to->di_gen = inode->i_generation;
 
 	to->di_size = from->di_size;
 	to->di_nblocks = from->di_nblocks;
@@ -361,7 +362,6 @@ xfs_inode_to_log_dinode(
 	to->di_dmevmask = from->di_dmevmask;
 	to->di_dmstate = from->di_dmstate;
 	to->di_flags = from->di_flags;
-	to->di_gen = from->di_gen;
 
 	if (from->di_version == 3) {
 		to->di_changecount = from->di_changecount;
diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
index d42738d..2c40c5f 100644
--- a/fs/xfs/xfs_ioctl.c
+++ b/fs/xfs/xfs_ioctl.c
@@ -114,7 +114,7 @@ xfs_find_handle(
 		handle.ha_fid.fid_len = sizeof(xfs_fid_t) -
 					sizeof(handle.ha_fid.fid_len);
 		handle.ha_fid.fid_pad = 0;
-		handle.ha_fid.fid_gen = ip->i_d.di_gen;
+		handle.ha_fid.fid_gen = inode->i_generation;
 		handle.ha_fid.fid_ino = ip->i_ino;
 
 		hsize = XFS_HSIZE(handle);
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index c424d4b..397ce85 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -1227,7 +1227,6 @@ xfs_setup_inode(
 		break;
 	}
 
-	inode->i_generation = ip->i_d.di_gen;
 	i_size_write(inode, ip->i_d.di_size);
 	xfs_diflags_to_iflags(inode, ip);
 
diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
index cfb6527..6162e65 100644
--- a/fs/xfs/xfs_itable.c
+++ b/fs/xfs/xfs_itable.c
@@ -100,11 +100,11 @@ xfs_bulkstat_one_int(
 	buf->bs_mtime.tv_nsec = inode->i_mtime.tv_nsec;
 	buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec;
 	buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec;
+	buf->bs_gen = inode->i_generation;
 
 	buf->bs_xflags = xfs_ip2xflags(ip);
 	buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog;
 	buf->bs_extents = dic->di_nextents;
-	buf->bs_gen = dic->di_gen;
 	memset(buf->bs_pad, 0, sizeof(buf->bs_pad));
 	buf->bs_dmevmask = dic->di_dmevmask;
 	buf->bs_dmstate = dic->di_dmstate;
-- 
2.5.0

From dave@fromorbit.com  Thu Jan 14 00:10:26 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id D5FC77F4E
	for ; Thu, 14 Jan 2016 00:10:26 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id A9AB6304059
	for ; Wed, 13 Jan 2016 22:10:23 -0800 (PST)
X-ASG-Debug-ID: 1452751819-04bdf06f6cefa40001-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id yDCGU5Siq7H1usVP for ; Wed, 13 Jan 2016 22:10:20 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DrBwA8O5dWPGwtLHleKAECgw+BP4ZigXmfXgaRG4oUgTtNAQEBAQEBBwEBAQFAAT+ENQEFJy8zCBgZGDkDBxQZiC3AP4YOikQiDIQSBYdljzCcWkSOFIIuAQs7HIFxKjSEVoFCAQEB
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:29 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OV-Mu
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001V6-MA
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH 4/8] xfs: move v1 inode conversion to xfs_inode_from_disk
Date: Thu, 14 Jan 2016 17:09:21 +1100
X-ASG-Orig-Subj: [PATCH 4/8] xfs: move v1 inode conversion to xfs_inode_from_disk
Message-Id: <1452751765-4420-5-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
In-Reply-To: <1452751765-4420-1-git-send-email-david@fromorbit.com>
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751819
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

So we don't have to carry an di_onlink variable around anymore, move
the inode conversion from v1 inode format to v2 inode format into
xfs_inode_from_disk(). This means we can remove the di_onlink fields
from the struct xfs_icdinode.

Signed-off-by: Dave Chinner 
---
 fs/xfs/libxfs/xfs_inode_buf.c  | 42 ++++++++++++++++++++----------------------
 fs/xfs/libxfs/xfs_inode_buf.h  |  1 -
 fs/xfs/libxfs/xfs_log_format.h |  2 +-
 fs/xfs/xfs_inode.c             |  2 --
 fs/xfs/xfs_inode_item.c        |  2 +-
 5 files changed, 22 insertions(+), 27 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
index 17ff4e8..8a1b460 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.c
+++ b/fs/xfs/libxfs/xfs_inode_buf.c
@@ -204,13 +204,25 @@ xfs_inode_from_disk(
 
 	to->di_mode = be16_to_cpu(from->di_mode);
 	to->di_version = from ->di_version;
+
+	/*
+	 * Convert v1 inodes immediately to v2 inode format as this is the
+	 * minimum inode version format we support in the rest of the code.
+	 */
+	if (to->di_version == 1) {
+		to->di_nlink = be16_to_cpu(from->di_onlink);
+		to->di_projid_lo = 0;
+		to->di_projid_hi = 0;
+		to->di_version = 2;
+	} else {
+		to->di_nlink = be32_to_cpu(from->di_nlink);
+		to->di_projid_lo = be16_to_cpu(from->di_projid_lo);
+		to->di_projid_hi = be16_to_cpu(from->di_projid_hi);
+	}
+
 	to->di_format = from->di_format;
-	to->di_onlink = be16_to_cpu(from->di_onlink);
 	to->di_uid = be32_to_cpu(from->di_uid);
 	to->di_gid = be32_to_cpu(from->di_gid);
-	to->di_nlink = be32_to_cpu(from->di_nlink);
-	to->di_projid_lo = be16_to_cpu(from->di_projid_lo);
-	to->di_projid_hi = be16_to_cpu(from->di_projid_hi);
 	to->di_flushiter = be16_to_cpu(from->di_flushiter);
 
 	/*
@@ -256,11 +268,11 @@ xfs_inode_to_disk(
 	struct inode		*inode = VFS_I(ip);
 
 	to->di_magic = cpu_to_be16(XFS_DINODE_MAGIC);
+	to->di_onlink = 0;
 
 	to->di_mode = cpu_to_be16(from->di_mode);
 	to->di_version = from->di_version;
 	to->di_format = from->di_format;
-	to->di_onlink = cpu_to_be16(from->di_onlink);
 	to->di_uid = cpu_to_be32(from->di_uid);
 	to->di_gid = cpu_to_be32(from->di_gid);
 	to->di_nlink = cpu_to_be32(from->di_nlink);
@@ -310,9 +322,9 @@ xfs_log_dinode_to_disk(
 {
 	to->di_magic = cpu_to_be16(from->di_magic);
 	to->di_mode = cpu_to_be16(from->di_mode);
-	to->di_version = from ->di_version;
+	to->di_version = from->di_version;
 	to->di_format = from->di_format;
-	to->di_onlink = cpu_to_be16(from->di_onlink);
+	to->di_onlink = 0;
 	to->di_uid = cpu_to_be32(from->di_uid);
 	to->di_gid = cpu_to_be32(from->di_gid);
 	to->di_nlink = cpu_to_be32(from->di_nlink);
@@ -492,21 +504,7 @@ xfs_iread(
 		ip->i_d.di_mode = 0;
 	}
 
-	/*
-	 * Automatically convert version 1 inode formats in memory to version 2
-	 * inode format. If the inode is modified, it will get logged and
-	 * rewritten as a version 2 inode. We can do this because we set the
-	 * superblock feature bit for v2 inodes unconditionally during mount
-	 * and it means the reast of the code can assume the inode version is 2
-	 * or higher.
-	 */
-	if (ip->i_d.di_version == 1) {
-		ip->i_d.di_version = 2;
-		ip->i_d.di_nlink = ip->i_d.di_onlink;
-		ip->i_d.di_onlink = 0;
-		xfs_set_projid(ip, 0);
-	}
-
+	ASSERT(ip->i_d.di_version >= 2);
 	ip->i_delayed_blks = 0;
 
 	/*
diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index 69d626e..feb04e6 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -31,7 +31,6 @@ struct xfs_icdinode {
 	__uint16_t	di_mode;	/* mode and type of file */
 	__int8_t	di_version;	/* inode version */
 	__int8_t	di_format;	/* format of di_c data */
-	__uint16_t	di_onlink;	/* old number of links to file */
 	__uint16_t	di_flushiter;	/* incremented on flush */
 	__uint32_t	di_uid;		/* owner's user id */
 	__uint32_t	di_gid;		/* owner's group id */
diff --git a/fs/xfs/libxfs/xfs_log_format.h b/fs/xfs/libxfs/xfs_log_format.h
index d00ed63..03f90b9 100644
--- a/fs/xfs/libxfs/xfs_log_format.h
+++ b/fs/xfs/libxfs/xfs_log_format.h
@@ -369,7 +369,7 @@ struct xfs_log_dinode {
 	__uint16_t	di_mode;	/* mode and type of file */
 	__int8_t	di_version;	/* inode version */
 	__int8_t	di_format;	/* format of di_c data */
-	__uint16_t	di_onlink;	/* old number of links to file */
+	__uint8_t	di_pad3[2];	/* unused in v2/3 inodes */
 	__uint32_t	di_uid;		/* owner's user id */
 	__uint32_t	di_gid;		/* owner's group id */
 	__uint32_t	di_nlink;	/* number of links to file */
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 3f020b5..914ec41 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -794,9 +794,7 @@ xfs_ialloc(
 		ip->i_d.di_version = 2;
 
 	ip->i_d.di_mode = mode;
-	ip->i_d.di_onlink = 0;
 	ip->i_d.di_nlink = nlink;
-	ASSERT(ip->i_d.di_nlink == nlink);
 	ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid());
 	ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid());
 	xfs_set_projid(ip, prid);
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index ae60087..d777b7a 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -336,7 +336,6 @@ xfs_inode_to_log_dinode(
 	to->di_mode = from->di_mode;
 	to->di_version = from->di_version;
 	to->di_format = from->di_format;
-	to->di_onlink = from->di_onlink;
 	to->di_uid = from->di_uid;
 	to->di_gid = from->di_gid;
 	to->di_nlink = from->di_nlink;
@@ -344,6 +343,7 @@ xfs_inode_to_log_dinode(
 	to->di_projid_hi = from->di_projid_hi;
 
 	memset(to->di_pad, 0, sizeof(to->di_pad));
+	memset(to->di_pad3, 0, sizeof(to->di_pad3));
 	to->di_atime.t_sec = inode->i_atime.tv_sec;
 	to->di_atime.t_nsec = inode->i_atime.tv_nsec;
 	to->di_mtime.t_sec = inode->i_mtime.tv_sec;
-- 
2.5.0

From dave@fromorbit.com  Thu Jan 14 00:10:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 001767F52
	for ; Thu, 14 Jan 2016 00:10:28 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id B81D48F804B
	for ; Wed, 13 Jan 2016 22:10:27 -0800 (PST)
X-ASG-Debug-ID: 1452751818-04bdf06f6eefa40003-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id WbJsbwPJVul3a90D for ; Wed, 13 Jan 2016 22:10:22 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DrBwA8O5dWPGwtLHleKAECgw+BP4ZigXmfXgaRG4oUgTtNAQEBAQEBBwEBAQFAAT+ENQEFJy8zCBgxOQMHFBmILcA/hg6KRIRABYdljzCcWo5Ygi4BCwE3AxyBcSo0hhgBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:29 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OT-Ln
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001Uw-L4
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH 2/8] xfs: remove timestamps from incore inode
Date: Thu, 14 Jan 2016 17:09:19 +1100
X-ASG-Orig-Subj: [PATCH 2/8] xfs: remove timestamps from incore inode
Message-Id: <1452751765-4420-3-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
In-Reply-To: <1452751765-4420-1-git-send-email-david@fromorbit.com>
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751822
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

The struct xfs_inode has two copies of the current timestamps in it,
one in the vfs inode and one in the struct xfs_icdinode. Now that we
no longer log the struct xfs_icdinode directly, we don't need to
keep the timestamps in this structure. instead we can copy them
straight out of the VFS inode when formatting the inode log item or
the on-disk inode.

This reduces the struct xfs_inode in size by 24 bytes.

Signed-off-by: Dave Chinner 
---
 fs/xfs/libxfs/xfs_inode_buf.c | 91 +++++++++++++++++++++++++++++++++++++------
 fs/xfs/libxfs/xfs_inode_buf.h |  9 ++---
 fs/xfs/libxfs/xfs_rtbitmap.c  |  2 +-
 fs/xfs/xfs_inode.c            | 21 +++++-----
 fs/xfs/xfs_inode_item.c       | 73 ++++++----------------------------
 fs/xfs/xfs_inode_item.h       |  2 -
 fs/xfs/xfs_iops.c             | 37 ++++--------------
 fs/xfs/xfs_itable.c           | 16 +++++---
 fs/xfs/xfs_log_recover.c      |  8 ++--
 fs/xfs/xfs_rtalloc.c          |  2 +-
 fs/xfs/xfs_trans_inode.c      | 12 +-----
 11 files changed, 130 insertions(+), 143 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
index 63d46bf..26e3c46 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.c
+++ b/fs/xfs/libxfs/xfs_inode_buf.c
@@ -195,10 +195,13 @@ xfs_imap_to_bp(
 }
 
 void
-xfs_dinode_from_disk(
-	struct xfs_icdinode	*to,
+xfs_inode_from_disk(
+	struct xfs_inode	*ip,
 	struct xfs_dinode	*from)
 {
+	struct xfs_icdinode	*to = &ip->i_d;
+	struct inode		*inode = VFS_I(ip);
+
 	to->di_magic = be16_to_cpu(from->di_magic);
 	to->di_mode = be16_to_cpu(from->di_mode);
 	to->di_version = from ->di_version;
@@ -211,12 +214,20 @@ xfs_dinode_from_disk(
 	to->di_projid_hi = be16_to_cpu(from->di_projid_hi);
 	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
 	to->di_flushiter = be16_to_cpu(from->di_flushiter);
-	to->di_atime.t_sec = be32_to_cpu(from->di_atime.t_sec);
-	to->di_atime.t_nsec = be32_to_cpu(from->di_atime.t_nsec);
-	to->di_mtime.t_sec = be32_to_cpu(from->di_mtime.t_sec);
-	to->di_mtime.t_nsec = be32_to_cpu(from->di_mtime.t_nsec);
-	to->di_ctime.t_sec = be32_to_cpu(from->di_ctime.t_sec);
-	to->di_ctime.t_nsec = be32_to_cpu(from->di_ctime.t_nsec);
+
+	/*
+	 * time is signed, so need to convert to signed 32 bit before
+	 * storing in inode timestamp which may be 64 bit. Otherwise
+	 * a time before epoch is converted to a time long after epoch
+	 * on 64 bit systems.
+	 */
+	inode->i_atime.tv_sec = (int)be32_to_cpu(from->di_atime.t_sec);
+	inode->i_atime.tv_nsec = (int)be32_to_cpu(from->di_atime.t_nsec);
+	inode->i_mtime.tv_sec = (int)be32_to_cpu(from->di_mtime.t_sec);
+	inode->i_mtime.tv_nsec = (int)be32_to_cpu(from->di_mtime.t_nsec);
+	inode->i_ctime.tv_sec = (int)be32_to_cpu(from->di_ctime.t_sec);
+	inode->i_ctime.tv_nsec = (int)be32_to_cpu(from->di_ctime.t_nsec);
+
 	to->di_size = be64_to_cpu(from->di_size);
 	to->di_nblocks = be64_to_cpu(from->di_nblocks);
 	to->di_extsize = be32_to_cpu(from->di_extsize);
@@ -242,9 +253,63 @@ xfs_dinode_from_disk(
 }
 
 void
-xfs_dinode_to_disk(
-	struct xfs_dinode	*to,
-	struct xfs_icdinode	*from)
+xfs_inode_to_disk(
+	struct xfs_inode	*ip,
+	struct xfs_dinode	*to)
+{
+	struct xfs_icdinode	*from = &ip->i_d;
+	struct inode		*inode = VFS_I(ip);
+
+	to->di_magic = cpu_to_be16(from->di_magic);
+	to->di_mode = cpu_to_be16(from->di_mode);
+	to->di_version = from ->di_version;
+	to->di_format = from->di_format;
+	to->di_onlink = cpu_to_be16(from->di_onlink);
+	to->di_uid = cpu_to_be32(from->di_uid);
+	to->di_gid = cpu_to_be32(from->di_gid);
+	to->di_nlink = cpu_to_be32(from->di_nlink);
+	to->di_projid_lo = cpu_to_be16(from->di_projid_lo);
+	to->di_projid_hi = cpu_to_be16(from->di_projid_hi);
+	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
+
+	to->di_atime.t_sec = cpu_to_be32(inode->i_atime.tv_sec);
+	to->di_atime.t_nsec = cpu_to_be32(inode->i_atime.tv_nsec);
+	to->di_mtime.t_sec = cpu_to_be32(inode->i_mtime.tv_sec);
+	to->di_mtime.t_nsec = cpu_to_be32(inode->i_mtime.tv_nsec);
+	to->di_ctime.t_sec = cpu_to_be32(inode->i_ctime.tv_sec);
+	to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec);
+
+	to->di_size = cpu_to_be64(from->di_size);
+	to->di_nblocks = cpu_to_be64(from->di_nblocks);
+	to->di_extsize = cpu_to_be32(from->di_extsize);
+	to->di_nextents = cpu_to_be32(from->di_nextents);
+	to->di_anextents = cpu_to_be16(from->di_anextents);
+	to->di_forkoff = from->di_forkoff;
+	to->di_aformat = from->di_aformat;
+	to->di_dmevmask = cpu_to_be32(from->di_dmevmask);
+	to->di_dmstate = cpu_to_be16(from->di_dmstate);
+	to->di_flags = cpu_to_be16(from->di_flags);
+	to->di_gen = cpu_to_be32(from->di_gen);
+
+	if (from->di_version == 3) {
+		to->di_changecount = cpu_to_be64(from->di_changecount);
+		to->di_crtime.t_sec = cpu_to_be32(from->di_crtime.t_sec);
+		to->di_crtime.t_nsec = cpu_to_be32(from->di_crtime.t_nsec);
+		to->di_flags2 = cpu_to_be64(from->di_flags2);
+		to->di_ino = cpu_to_be64(from->di_ino);
+		to->di_lsn = cpu_to_be64(from->di_lsn);
+		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
+		uuid_copy(&to->di_uuid, &from->di_uuid);
+		to->di_flushiter = 0;
+	} else {
+		to->di_flushiter = cpu_to_be16(from->di_flushiter);
+	}
+}
+
+void
+xfs_log_dinode_to_disk(
+	struct xfs_log_dinode	*from,
+	struct xfs_dinode	*to)
 {
 	to->di_magic = cpu_to_be16(from->di_magic);
 	to->di_mode = cpu_to_be16(from->di_mode);
@@ -257,12 +322,14 @@ xfs_dinode_to_disk(
 	to->di_projid_lo = cpu_to_be16(from->di_projid_lo);
 	to->di_projid_hi = cpu_to_be16(from->di_projid_hi);
 	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
+
 	to->di_atime.t_sec = cpu_to_be32(from->di_atime.t_sec);
 	to->di_atime.t_nsec = cpu_to_be32(from->di_atime.t_nsec);
 	to->di_mtime.t_sec = cpu_to_be32(from->di_mtime.t_sec);
 	to->di_mtime.t_nsec = cpu_to_be32(from->di_mtime.t_nsec);
 	to->di_ctime.t_sec = cpu_to_be32(from->di_ctime.t_sec);
 	to->di_ctime.t_nsec = cpu_to_be32(from->di_ctime.t_nsec);
+
 	to->di_size = cpu_to_be64(from->di_size);
 	to->di_nblocks = cpu_to_be64(from->di_nblocks);
 	to->di_extsize = cpu_to_be32(from->di_extsize);
@@ -403,7 +470,7 @@ xfs_iread(
 	 * Otherwise, just get the truly permanent information.
 	 */
 	if (dip->di_mode) {
-		xfs_dinode_from_disk(&ip->i_d, dip);
+		xfs_inode_from_disk(ip, dip);
 		error = xfs_iformat_fork(ip, dip);
 		if (error)  {
 #ifdef DEBUG
diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index 642f2a2..adcc9bf 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -40,9 +40,6 @@ struct xfs_icdinode {
 	__uint16_t	di_projid_hi;	/* higher part of owner's project id */
 	__uint8_t	di_pad[6];	/* unused, zeroed space */
 	__uint16_t	di_flushiter;	/* incremented on flush */
-	xfs_ictimestamp_t di_atime;	/* time last accessed */
-	xfs_ictimestamp_t di_mtime;	/* time last modified */
-	xfs_ictimestamp_t di_ctime;	/* time created/inode modified */
 	xfs_fsize_t	di_size;	/* number of bytes in file */
 	xfs_rfsblock_t	di_nblocks;	/* # of direct & btree blocks used */
 	xfs_extlen_t	di_extsize;	/* basic/minimum extent size for file */
@@ -89,8 +86,10 @@ int	xfs_imap_to_bp(struct xfs_mount *, struct xfs_trans *,
 int	xfs_iread(struct xfs_mount *, struct xfs_trans *,
 		  struct xfs_inode *, uint);
 void	xfs_dinode_calc_crc(struct xfs_mount *, struct xfs_dinode *);
-void	xfs_dinode_to_disk(struct xfs_dinode *to, struct xfs_icdinode *from);
-void	xfs_dinode_from_disk(struct xfs_icdinode *to, struct xfs_dinode *from);
+void	xfs_inode_to_disk(struct xfs_inode *ip, struct xfs_dinode *to);
+void	xfs_inode_from_disk(struct xfs_inode *ip, struct xfs_dinode *from);
+void	xfs_log_dinode_to_disk(struct xfs_log_dinode *from,
+			       struct xfs_dinode *to);
 
 #if defined(DEBUG)
 void	xfs_inobp_check(struct xfs_mount *, struct xfs_buf *);
diff --git a/fs/xfs/libxfs/xfs_rtbitmap.c b/fs/xfs/libxfs/xfs_rtbitmap.c
index 9b59ffa..acc71dd 100644
--- a/fs/xfs/libxfs/xfs_rtbitmap.c
+++ b/fs/xfs/libxfs/xfs_rtbitmap.c
@@ -983,7 +983,7 @@ xfs_rtfree_extent(
 	    mp->m_sb.sb_rextents) {
 		if (!(mp->m_rbmip->i_d.di_flags & XFS_DIFLAG_NEWRTBM))
 			mp->m_rbmip->i_d.di_flags |= XFS_DIFLAG_NEWRTBM;
-		*(__uint64_t *)&mp->m_rbmip->i_d.di_atime = 0;
+		*(__uint64_t *)&VFS_I(mp->m_rbmip)->i_atime = 0;
 		xfs_trans_log_inode(tp, mp->m_rbmip, XFS_ILOG_CORE);
 	}
 	return 0;
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 7e24232..3e9b8a0 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -757,6 +757,7 @@ xfs_ialloc(
 	uint		flags;
 	int		error;
 	struct timespec	tv;
+	struct inode	*inode;
 
 	/*
 	 * Call the space management code to pick
@@ -782,6 +783,7 @@ xfs_ialloc(
 	if (error)
 		return error;
 	ASSERT(ip != NULL);
+	inode = VFS_I(ip);
 
 	/*
 	 * We always convert v1 inodes to v2 now - we only support filesystems
@@ -823,10 +825,9 @@ xfs_ialloc(
 	ASSERT(ip->i_d.di_nblocks == 0);
 
 	tv = current_fs_time(mp->m_super);
-	ip->i_d.di_mtime.t_sec = (__int32_t)tv.tv_sec;
-	ip->i_d.di_mtime.t_nsec = (__int32_t)tv.tv_nsec;
-	ip->i_d.di_atime = ip->i_d.di_mtime;
-	ip->i_d.di_ctime = ip->i_d.di_mtime;
+	inode->i_mtime = tv;
+	inode->i_atime = tv;
+	inode->i_ctime = tv;
 
 	/*
 	 * di_gen will have been taken care of in xfs_iread.
@@ -844,7 +845,8 @@ xfs_ialloc(
 		ip->i_d.di_lsn = 0;
 		ip->i_d.di_flags2 = 0;
 		memset(&(ip->i_d.di_pad2[0]), 0, sizeof(ip->i_d.di_pad2));
-		ip->i_d.di_crtime = ip->i_d.di_mtime;
+		ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec;
+		ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec;
 	}
 
 
@@ -3509,12 +3511,11 @@ xfs_iflush_int(
 		ip->i_d.di_flushiter++;
 
 	/*
-	 * Copy the dirty parts of the inode into the on-disk
-	 * inode.  We always copy out the core of the inode,
-	 * because if the inode is dirty at all the core must
-	 * be.
+	 * Copy the dirty parts of the inode into the on-disk inode.  We always
+	 * copy out the core of the inode, because if the inode is dirty at all
+	 * the core must be.
 	 */
-	xfs_dinode_to_disk(dip, &ip->i_d);
+	xfs_inode_to_disk(ip, dip);
 
 	/* Wrap, we never let the log put out DI_MAX_FLUSH */
 	if (ip->i_d.di_flushiter == DI_MAX_FLUSH)
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index 3ad9972..9dcbf58 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -323,10 +323,13 @@ xfs_inode_item_format_attr_fork(
 }
 
 static void
-xfs_icdinode_to_log_dinode(
-	struct xfs_icdinode	*from,
+xfs_inode_to_log_dinode(
+	struct xfs_inode	*ip,
 	struct xfs_log_dinode	*to)
 {
+	struct xfs_icdinode	*from = &ip->i_d;
+	struct inode		*inode = VFS_I(ip);
+
 	to->di_magic = from->di_magic;
 	to->di_mode = from->di_mode;
 	to->di_version = from->di_version;
@@ -338,66 +341,14 @@ xfs_icdinode_to_log_dinode(
 	to->di_projid_lo = from->di_projid_lo;
 	to->di_projid_hi = from->di_projid_hi;
 	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
-	to->di_atime.t_sec = from->di_atime.t_sec;
-	to->di_atime.t_nsec = from->di_atime.t_nsec;
-	to->di_mtime.t_sec = from->di_mtime.t_sec;
-	to->di_mtime.t_nsec = from->di_mtime.t_nsec;
-	to->di_ctime.t_sec = from->di_ctime.t_sec;
-	to->di_ctime.t_nsec = from->di_ctime.t_nsec;
-	to->di_size = from->di_size;
-	to->di_nblocks = from->di_nblocks;
-	to->di_extsize = from->di_extsize;
-	to->di_nextents = from->di_nextents;
-	to->di_anextents = from->di_anextents;
-	to->di_forkoff = from->di_forkoff;
-	to->di_aformat = from->di_aformat;
-	to->di_dmevmask = from->di_dmevmask;
-	to->di_dmstate = from->di_dmstate;
-	to->di_flags = from->di_flags;
-	to->di_gen = from->di_gen;
 
-	if (from->di_version == 3) {
-		to->di_changecount = from->di_changecount;
-		to->di_crtime.t_sec = from->di_crtime.t_sec;
-		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
-		to->di_flags2 = from->di_flags2;
-		to->di_ino = from->di_ino;
-		to->di_lsn = from->di_lsn;
-		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
-		uuid_copy(&to->di_uuid, &from->di_uuid);
-		to->di_flushiter = 0;
-	} else {
-		to->di_flushiter = from->di_flushiter;
-	}
-}
+	to->di_atime.t_sec = inode->i_atime.tv_sec;
+	to->di_atime.t_nsec = inode->i_atime.tv_nsec;
+	to->di_mtime.t_sec = inode->i_mtime.tv_sec;
+	to->di_mtime.t_nsec = inode->i_mtime.tv_nsec;
+	to->di_ctime.t_sec = inode->i_ctime.tv_sec;
+	to->di_ctime.t_nsec = inode->i_ctime.tv_nsec;
 
-/*
- * Recovery needs to be able to convert a log dinode back to a real dinode
- * for writeback we do that by converting a log dinode to a icdinode, and
- * then passing that to the formatting function.
- */
-void
-xfs_log_dinode_to_icdinode(
-	struct xfs_log_dinode	*from,
-	struct xfs_icdinode	*to)
-{
-	to->di_magic = from->di_magic;
-	to->di_mode = from->di_mode;
-	to->di_version = from->di_version;
-	to->di_format = from->di_format;
-	to->di_onlink = from->di_onlink;
-	to->di_uid = from->di_uid;
-	to->di_gid = from->di_gid;
-	to->di_nlink = from->di_nlink;
-	to->di_projid_lo = from->di_projid_lo;
-	to->di_projid_hi = from->di_projid_hi;
-	memset(to->di_pad, 0, sizeof(to->di_pad));
-	to->di_atime.t_sec = from->di_atime.t_sec;
-	to->di_atime.t_nsec = from->di_atime.t_nsec;
-	to->di_mtime.t_sec = from->di_mtime.t_sec;
-	to->di_mtime.t_nsec = from->di_mtime.t_nsec;
-	to->di_ctime.t_sec = from->di_ctime.t_sec;
-	to->di_ctime.t_nsec = from->di_ctime.t_nsec;
 	to->di_size = from->di_size;
 	to->di_nblocks = from->di_nblocks;
 	to->di_extsize = from->di_extsize;
@@ -439,7 +390,7 @@ xfs_inode_item_format_core(
 	struct xfs_log_dinode	*dic;
 
 	dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE);
-	xfs_icdinode_to_log_dinode(&ip->i_d, dic);
+	xfs_inode_to_log_dinode(ip, dic);
 	xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version));
 }
 
diff --git a/fs/xfs/xfs_inode_item.h b/fs/xfs/xfs_inode_item.h
index 2426118..4c7722e 100644
--- a/fs/xfs/xfs_inode_item.h
+++ b/fs/xfs/xfs_inode_item.h
@@ -49,8 +49,6 @@ extern void xfs_istale_done(struct xfs_buf *, struct xfs_log_item *);
 extern void xfs_iflush_abort(struct xfs_inode *, bool);
 extern int xfs_inode_item_format_convert(xfs_log_iovec_t *,
 					 xfs_inode_log_format_t *);
-extern void xfs_log_dinode_to_icdinode(struct xfs_log_dinode *from,
-				       struct xfs_icdinode *to);
 
 extern struct kmem_zone	*xfs_ili_zone;
 
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index 245268a..b008677 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -517,21 +517,12 @@ xfs_setattr_time(
 
 	ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL));
 
-	if (iattr->ia_valid & ATTR_ATIME) {
+	if (iattr->ia_valid & ATTR_ATIME)
 		inode->i_atime = iattr->ia_atime;
-		ip->i_d.di_atime.t_sec = iattr->ia_atime.tv_sec;
-		ip->i_d.di_atime.t_nsec = iattr->ia_atime.tv_nsec;
-	}
-	if (iattr->ia_valid & ATTR_CTIME) {
+	if (iattr->ia_valid & ATTR_CTIME)
 		inode->i_ctime = iattr->ia_ctime;
-		ip->i_d.di_ctime.t_sec = iattr->ia_ctime.tv_sec;
-		ip->i_d.di_ctime.t_nsec = iattr->ia_ctime.tv_nsec;
-	}
-	if (iattr->ia_valid & ATTR_MTIME) {
+	if (iattr->ia_valid & ATTR_MTIME)
 		inode->i_mtime = iattr->ia_mtime;
-		ip->i_d.di_mtime.t_sec = iattr->ia_mtime.tv_sec;
-		ip->i_d.di_mtime.t_nsec = iattr->ia_mtime.tv_nsec;
-	}
 }
 
 int
@@ -986,21 +977,13 @@ xfs_vn_update_time(
 	}
 
 	xfs_ilock(ip, XFS_ILOCK_EXCL);
-	if (flags & S_CTIME) {
+	if (flags & S_CTIME)
 		inode->i_ctime = *now;
-		ip->i_d.di_ctime.t_sec = (__int32_t)now->tv_sec;
-		ip->i_d.di_ctime.t_nsec = (__int32_t)now->tv_nsec;
-	}
-	if (flags & S_MTIME) {
+	if (flags & S_MTIME)
 		inode->i_mtime = *now;
-		ip->i_d.di_mtime.t_sec = (__int32_t)now->tv_sec;
-		ip->i_d.di_mtime.t_nsec = (__int32_t)now->tv_nsec;
-	}
-	if (flags & S_ATIME) {
+	if (flags & S_ATIME)
 		inode->i_atime = *now;
-		ip->i_d.di_atime.t_sec = (__int32_t)now->tv_sec;
-		ip->i_d.di_atime.t_nsec = (__int32_t)now->tv_nsec;
-	}
+
 	xfs_trans_ijoin(tp, ip, XFS_ILOCK_EXCL);
 	xfs_trans_log_inode(tp, ip, XFS_ILOG_TIMESTAMP);
 	return xfs_trans_commit(tp);
@@ -1247,12 +1230,6 @@ xfs_setup_inode(
 
 	inode->i_generation = ip->i_d.di_gen;
 	i_size_write(inode, ip->i_d.di_size);
-	inode->i_atime.tv_sec	= ip->i_d.di_atime.t_sec;
-	inode->i_atime.tv_nsec	= ip->i_d.di_atime.t_nsec;
-	inode->i_mtime.tv_sec	= ip->i_d.di_mtime.t_sec;
-	inode->i_mtime.tv_nsec	= ip->i_d.di_mtime.t_nsec;
-	inode->i_ctime.tv_sec	= ip->i_d.di_ctime.t_sec;
-	inode->i_ctime.tv_nsec	= ip->i_d.di_ctime.t_nsec;
 	xfs_diflags_to_iflags(inode, ip);
 
 	ip->d_ops = ip->i_mount->m_nondir_inode_ops;
diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
index 930ebd8..2acda42 100644
--- a/fs/xfs/xfs_itable.c
+++ b/fs/xfs/xfs_itable.c
@@ -57,6 +57,7 @@ xfs_bulkstat_one_int(
 {
 	struct xfs_icdinode	*dic;		/* dinode core info pointer */
 	struct xfs_inode	*ip;		/* incore inode pointer */
+	struct inode		*inode;
 	struct xfs_bstat	*buf;		/* return buffer */
 	int			error = 0;	/* error value */
 
@@ -77,6 +78,7 @@ xfs_bulkstat_one_int(
 
 	ASSERT(ip != NULL);
 	ASSERT(ip->i_imap.im_blkno != 0);
+	inode = VFS_I(ip);
 
 	dic = &ip->i_d;
 
@@ -91,12 +93,14 @@ xfs_bulkstat_one_int(
 	buf->bs_uid = dic->di_uid;
 	buf->bs_gid = dic->di_gid;
 	buf->bs_size = dic->di_size;
-	buf->bs_atime.tv_sec = dic->di_atime.t_sec;
-	buf->bs_atime.tv_nsec = dic->di_atime.t_nsec;
-	buf->bs_mtime.tv_sec = dic->di_mtime.t_sec;
-	buf->bs_mtime.tv_nsec = dic->di_mtime.t_nsec;
-	buf->bs_ctime.tv_sec = dic->di_ctime.t_sec;
-	buf->bs_ctime.tv_nsec = dic->di_ctime.t_nsec;
+
+	buf->bs_atime.tv_sec = inode->i_atime.tv_sec;
+	buf->bs_atime.tv_nsec = inode->i_atime.tv_nsec;
+	buf->bs_mtime.tv_sec = inode->i_mtime.tv_sec;
+	buf->bs_mtime.tv_nsec = inode->i_mtime.tv_nsec;
+	buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec;
+	buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec;
+
 	buf->bs_xflags = xfs_ip2xflags(ip);
 	buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog;
 	buf->bs_extents = dic->di_nextents;
diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
index 3120f7b..4b79cf0 100644
--- a/fs/xfs/xfs_log_recover.c
+++ b/fs/xfs/xfs_log_recover.c
@@ -2793,7 +2793,7 @@ xfs_recover_inode_owner_change(
 		return -ENOMEM;
 
 	/* instantiate the inode */
-	xfs_dinode_from_disk(&ip->i_d, dip);
+	xfs_inode_from_disk(ip, dip);
 	ASSERT(ip->i_d.di_version >= 3);
 
 	error = xfs_iformat_fork(ip, dip);
@@ -2840,7 +2840,6 @@ xlog_recover_inode_pass2(
 	int			attr_index;
 	uint			fields;
 	struct xfs_log_dinode	*ldip;
-	struct xfs_icdinode	icic;
 	uint			isize;
 	int			need_free = 0;
 
@@ -3007,9 +3006,8 @@ xlog_recover_inode_pass2(
 		goto out_release;
 	}
 
-	/* The core is in in-core format */
-	xfs_log_dinode_to_icdinode(ldip, &icic);
-	xfs_dinode_to_disk(dip, &icic);
+	/* recover the log dinode inode into the on disk inode */
+	xfs_log_dinode_to_disk(ldip, dip);
 
 	/* the rest is in on-disk format */
 	if (item->ri_buf[1].i_len > isize) {
diff --git a/fs/xfs/xfs_rtalloc.c b/fs/xfs/xfs_rtalloc.c
index be02a68..abf4443 100644
--- a/fs/xfs/xfs_rtalloc.c
+++ b/fs/xfs/xfs_rtalloc.c
@@ -1272,7 +1272,7 @@ xfs_rtpick_extent(
 
 	ASSERT(xfs_isilocked(mp->m_rbmip, XFS_ILOCK_EXCL));
 
-	seqp = (__uint64_t *)&mp->m_rbmip->i_d.di_atime;
+	seqp = (__uint64_t *)&VFS_I(mp->m_rbmip)->i_atime;
 	if (!(mp->m_rbmip->i_d.di_flags & XFS_DIFLAG_NEWRTBM)) {
 		mp->m_rbmip->i_d.di_flags |= XFS_DIFLAG_NEWRTBM;
 		*seqp = 0;
diff --git a/fs/xfs/xfs_trans_inode.c b/fs/xfs/xfs_trans_inode.c
index b97f1df..3f0d466 100644
--- a/fs/xfs/xfs_trans_inode.c
+++ b/fs/xfs/xfs_trans_inode.c
@@ -75,18 +75,10 @@ xfs_trans_ichgtime(
 
 	tv = current_fs_time(inode->i_sb);
 
-	if ((flags & XFS_ICHGTIME_MOD) &&
-	    !timespec_equal(&inode->i_mtime, &tv)) {
+	if (flags & XFS_ICHGTIME_MOD)
 		inode->i_mtime = tv;
-		ip->i_d.di_mtime.t_sec = tv.tv_sec;
-		ip->i_d.di_mtime.t_nsec = tv.tv_nsec;
-	}
-	if ((flags & XFS_ICHGTIME_CHG) &&
-	    !timespec_equal(&inode->i_ctime, &tv)) {
+	if (flags & XFS_ICHGTIME_CHG)
 		inode->i_ctime = tv;
-		ip->i_d.di_ctime.t_sec = tv.tv_sec;
-		ip->i_d.di_ctime.t_nsec = tv.tv_nsec;
-	}
 }
 
 /*
-- 
2.5.0

From dave@fromorbit.com  Thu Jan 14 00:10:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 8F0247F56
	for ; Thu, 14 Jan 2016 00:10:28 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 519858F8052
	for ; Wed, 13 Jan 2016 22:10:28 -0800 (PST)
X-ASG-Debug-ID: 1452751819-04bdf06f6cefa40002-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id cVgmg1FMCACcpIU1 for ; Wed, 13 Jan 2016 22:10:21 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DtBwA8O5dWPGwtLHleKAECgw+BP4ZigXmfXgaBZo81ihSBO00BAQEBAQEHAQEBAUABP4Q1AQUaDS8WHQgYMTkDBxQQCYgtwD+GDooCQoRABYdiA4cQiCCIN4cBh26FNESOFIIuAQsBNwMcgXEqNIROgUoBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:29 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OZ-Ou
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001VQ-OE
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH 8/8] xfs: mode di_mode to vfs inode
Date: Thu, 14 Jan 2016 17:09:25 +1100
X-ASG-Orig-Subj: [PATCH 8/8] xfs: mode di_mode to vfs inode
Message-Id: <1452751765-4420-9-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
In-Reply-To: <1452751765-4420-1-git-send-email-david@fromorbit.com>
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751821
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

Move the di_mode value from the xfs_icdinode to the VFS inode, reducing
the xfs_icdinode byte another 2 bytes and collapsing another 2 byte hole
in the structure.

Signed-off-by: Dave Chinner 
---
 fs/fs-writeback.c              |  1 +
 fs/xfs/libxfs/xfs_bmap.c       |  6 +++---
 fs/xfs/libxfs/xfs_dir2.c       | 12 +++++------
 fs/xfs/libxfs/xfs_inode_buf.c  |  8 +++----
 fs/xfs/libxfs/xfs_inode_buf.h  |  1 -
 fs/xfs/libxfs/xfs_inode_fork.c |  2 +-
 fs/xfs/xfs_bmap_util.c         |  4 ++--
 fs/xfs/xfs_dir2_readdir.c      |  2 +-
 fs/xfs/xfs_file.c              |  6 +++---
 fs/xfs/xfs_filestream.c        |  4 ++--
 fs/xfs/xfs_icache.c            | 42 +++++++++++++++++++++++++++++++-----
 fs/xfs/xfs_inode.c             | 48 ++++++++++++++++++++----------------------
 fs/xfs/xfs_inode.h             |  4 ++--
 fs/xfs/xfs_inode_item.c        |  2 +-
 fs/xfs/xfs_ioctl.c             | 14 ++++++------
 fs/xfs/xfs_iops.c              | 12 ++++-------
 fs/xfs/xfs_itable.c            |  2 +-
 fs/xfs/xfs_log_recover.c       |  2 +-
 fs/xfs/xfs_mount.c             |  2 +-
 19 files changed, 100 insertions(+), 74 deletions(-)

diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 023f6a1..0858457 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -2106,6 +2106,7 @@ EXPORT_SYMBOL(__mark_inode_dirty);
  * completed by the time we have gained the lock and waited for all IO that is
  * in progress regardless of the order callers are granted the lock.
  */
+#include "linux/delay.h"
 static void wait_sb_inodes(struct super_block *sb)
 {
 	struct inode *inode, *old_inode = NULL;
diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
index ef00156..6a05166 100644
--- a/fs/xfs/libxfs/xfs_bmap.c
+++ b/fs/xfs/libxfs/xfs_bmap.c
@@ -912,7 +912,7 @@ xfs_bmap_local_to_extents(
 	 * We don't want to deal with the case of keeping inode data inline yet.
 	 * So sending the data fork of a regular inode is invalid.
 	 */
-	ASSERT(!(S_ISREG(ip->i_d.di_mode) && whichfork == XFS_DATA_FORK));
+	ASSERT(!(S_ISREG(VFS_I(ip)->i_mode) && whichfork == XFS_DATA_FORK));
 	ifp = XFS_IFORK_PTR(ip, whichfork);
 	ASSERT(XFS_IFORK_FORMAT(ip, whichfork) == XFS_DINODE_FMT_LOCAL);
 
@@ -1079,7 +1079,7 @@ xfs_bmap_add_attrfork_local(
 	if (ip->i_df.if_bytes <= XFS_IFORK_DSIZE(ip))
 		return 0;
 
-	if (S_ISDIR(ip->i_d.di_mode)) {
+	if (S_ISDIR(VFS_I(ip)->i_mode)) {
 		memset(&dargs, 0, sizeof(dargs));
 		dargs.geo = ip->i_mount->m_dir_geo;
 		dargs.dp = ip;
@@ -1091,7 +1091,7 @@ xfs_bmap_add_attrfork_local(
 		return xfs_dir2_sf_to_block(&dargs);
 	}
 
-	if (S_ISLNK(ip->i_d.di_mode))
+	if (S_ISLNK(VFS_I(ip)->i_mode))
 		return xfs_bmap_local_to_extents(tp, ip, firstblock, 1,
 						 flags, XFS_DATA_FORK,
 						 xfs_symlink_local_to_remote);
diff --git a/fs/xfs/libxfs/xfs_dir2.c b/fs/xfs/libxfs/xfs_dir2.c
index 2fb53a5..af0f9d1 100644
--- a/fs/xfs/libxfs/xfs_dir2.c
+++ b/fs/xfs/libxfs/xfs_dir2.c
@@ -176,7 +176,7 @@ xfs_dir_isempty(
 {
 	xfs_dir2_sf_hdr_t	*sfp;
 
-	ASSERT(S_ISDIR(dp->i_d.di_mode));
+	ASSERT(S_ISDIR(VFS_I(dp)->i_mode));
 	if (dp->i_d.di_size == 0)	/* might happen during shutdown. */
 		return 1;
 	if (dp->i_d.di_size > XFS_IFORK_DSIZE(dp))
@@ -231,7 +231,7 @@ xfs_dir_init(
 	struct xfs_da_args *args;
 	int		error;
 
-	ASSERT(S_ISDIR(dp->i_d.di_mode));
+	ASSERT(S_ISDIR(VFS_I(dp)->i_mode));
 	error = xfs_dir_ino_validate(tp->t_mountp, pdp->i_ino);
 	if (error)
 		return error;
@@ -266,7 +266,7 @@ xfs_dir_createname(
 	int			rval;
 	int			v;		/* type-checking value */
 
-	ASSERT(S_ISDIR(dp->i_d.di_mode));
+	ASSERT(S_ISDIR(VFS_I(dp)->i_mode));
 	if (inum) {
 		rval = xfs_dir_ino_validate(tp->t_mountp, inum);
 		if (rval)
@@ -364,7 +364,7 @@ xfs_dir_lookup(
 	int		v;		/* type-checking value */
 	int		lock_mode;
 
-	ASSERT(S_ISDIR(dp->i_d.di_mode));
+	ASSERT(S_ISDIR(VFS_I(dp)->i_mode));
 	XFS_STATS_INC(dp->i_mount, xs_dir_lookup);
 
 	/*
@@ -443,7 +443,7 @@ xfs_dir_removename(
 	int		rval;
 	int		v;		/* type-checking value */
 
-	ASSERT(S_ISDIR(dp->i_d.di_mode));
+	ASSERT(S_ISDIR(VFS_I(dp)->i_mode));
 	XFS_STATS_INC(dp->i_mount, xs_dir_remove);
 
 	args = kmem_zalloc(sizeof(*args), KM_SLEEP | KM_NOFS);
@@ -505,7 +505,7 @@ xfs_dir_replace(
 	int		rval;
 	int		v;		/* type-checking value */
 
-	ASSERT(S_ISDIR(dp->i_d.di_mode));
+	ASSERT(S_ISDIR(VFS_I(dp)->i_mode));
 
 	rval = xfs_dir_ino_validate(tp->t_mountp, inum);
 	if (rval)
diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
index 9807c21..4a388b0 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.c
+++ b/fs/xfs/libxfs/xfs_inode_buf.c
@@ -202,13 +202,12 @@ xfs_inode_from_disk(
 	struct xfs_icdinode	*to = &ip->i_d;
 	struct inode		*inode = VFS_I(ip);
 
-	to->di_mode = be16_to_cpu(from->di_mode);
-	to->di_version = from ->di_version;
 
 	/*
 	 * Convert v1 inodes immediately to v2 inode format as this is the
 	 * minimum inode version format we support in the rest of the code.
 	 */
+	to->di_version = from->di_version;
 	if (to->di_version == 1) {
 		set_nlink(inode, be16_to_cpu(from->di_onlink));
 		to->di_projid_lo = 0;
@@ -238,6 +237,7 @@ xfs_inode_from_disk(
 	inode->i_ctime.tv_sec = (int)be32_to_cpu(from->di_ctime.t_sec);
 	inode->i_ctime.tv_nsec = (int)be32_to_cpu(from->di_ctime.t_nsec);
 	inode->i_generation = be32_to_cpu(from->di_gen);
+	inode->i_mode = be16_to_cpu(from->di_mode);
 
 	to->di_size = be64_to_cpu(from->di_size);
 	to->di_nblocks = be64_to_cpu(from->di_nblocks);
@@ -270,7 +270,6 @@ xfs_inode_to_disk(
 	to->di_magic = cpu_to_be16(XFS_DINODE_MAGIC);
 	to->di_onlink = 0;
 
-	to->di_mode = cpu_to_be16(from->di_mode);
 	to->di_version = from->di_version;
 	to->di_format = from->di_format;
 	to->di_uid = cpu_to_be32(from->di_uid);
@@ -287,6 +286,7 @@ xfs_inode_to_disk(
 	to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec);
 	to->di_nlink = cpu_to_be32(inode->i_nlink);
 	to->di_gen = cpu_to_be32(inode->i_generation);
+	to->di_mode = cpu_to_be16(inode->i_mode);
 
 	to->di_size = cpu_to_be64(from->di_size);
 	to->di_nblocks = cpu_to_be64(from->di_nblocks);
@@ -501,7 +501,7 @@ xfs_iread(
 		 * the inode is already free and not try to mess
 		 * with the uninitialized part of it.
 		 */
-		ip->i_d.di_mode = 0;
+		VFS_I(ip)->i_mode = 0;
 	}
 
 	ASSERT(ip->i_d.di_version >= 2);
diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index e49c229..5f349a2 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -28,7 +28,6 @@ struct xfs_dinode;
  * translated to on-disk format specific structures at the appropriate time.
  */
 struct xfs_icdinode {
-	__uint16_t	di_mode;	/* mode and type of file */
 	__int8_t	di_version;	/* inode version */
 	__int8_t	di_format;	/* format of di_c data */
 	__uint16_t	di_flushiter;	/* incremented on flush */
diff --git a/fs/xfs/libxfs/xfs_inode_fork.c b/fs/xfs/libxfs/xfs_inode_fork.c
index 0defbd0..0bf1c74 100644
--- a/fs/xfs/libxfs/xfs_inode_fork.c
+++ b/fs/xfs/libxfs/xfs_inode_fork.c
@@ -120,7 +120,7 @@ xfs_iformat_fork(
 		return -EFSCORRUPTED;
 	}
 
-	switch (ip->i_d.di_mode & S_IFMT) {
+	switch (VFS_I(ip)->i_mode & S_IFMT) {
 	case S_IFIFO:
 	case S_IFCHR:
 	case S_IFBLK:
diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c
index 45ec9e4..7087756 100644
--- a/fs/xfs/xfs_bmap_util.c
+++ b/fs/xfs/xfs_bmap_util.c
@@ -821,7 +821,7 @@ bool
 xfs_can_free_eofblocks(struct xfs_inode *ip, bool force)
 {
 	/* prealloc/delalloc exists only on regular files */
-	if (!S_ISREG(ip->i_d.di_mode))
+	if (!S_ISREG(VFS_I(ip)->i_mode))
 		return false;
 
 	/*
@@ -1726,7 +1726,7 @@ xfs_swap_extents(
 	xfs_lock_two_inodes(ip, tip, XFS_MMAPLOCK_EXCL);
 
 	/* Verify that both files have the same format */
-	if ((ip->i_d.di_mode & S_IFMT) != (tip->i_d.di_mode & S_IFMT)) {
+	if ((VFS_I(ip)->i_mode & S_IFMT) != (VFS_I(tip)->i_mode & S_IFMT)) {
 		error = -EINVAL;
 		goto out_unlock;
 	}
diff --git a/fs/xfs/xfs_dir2_readdir.c b/fs/xfs/xfs_dir2_readdir.c
index 642d55d..93b3ab0 100644
--- a/fs/xfs/xfs_dir2_readdir.c
+++ b/fs/xfs/xfs_dir2_readdir.c
@@ -665,7 +665,7 @@ xfs_readdir(
 	if (XFS_FORCED_SHUTDOWN(dp->i_mount))
 		return -EIO;
 
-	ASSERT(S_ISDIR(dp->i_d.di_mode));
+	ASSERT(S_ISDIR(VFS_I(dp)->i_mode));
 	XFS_STATS_INC(dp->i_mount, xs_dir_getdents);
 
 	args.dp = dp;
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index ebe9b82..20eb6e5 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -156,9 +156,9 @@ xfs_update_prealloc_flags(
 	xfs_trans_ijoin(tp, ip, XFS_ILOCK_EXCL);
 
 	if (!(flags & XFS_PREALLOC_INVISIBLE)) {
-		ip->i_d.di_mode &= ~S_ISUID;
-		if (ip->i_d.di_mode & S_IXGRP)
-			ip->i_d.di_mode &= ~S_ISGID;
+		VFS_I(ip)->i_mode &= ~S_ISUID;
+		if (VFS_I(ip)->i_mode & S_IXGRP)
+			VFS_I(ip)->i_mode &= ~S_ISGID;
 		xfs_trans_ichgtime(tp, ip, XFS_ICHGTIME_MOD | XFS_ICHGTIME_CHG);
 	}
 
diff --git a/fs/xfs/xfs_filestream.c b/fs/xfs/xfs_filestream.c
index c4c130f..a51353a 100644
--- a/fs/xfs/xfs_filestream.c
+++ b/fs/xfs/xfs_filestream.c
@@ -151,7 +151,7 @@ xfs_filestream_pick_ag(
 	xfs_agnumber_t		ag, max_ag = NULLAGNUMBER;
 	int			err, trylock, nscan;
 
-	ASSERT(S_ISDIR(ip->i_d.di_mode));
+	ASSERT(S_ISDIR(VFS_I(ip)->i_mode));
 
 	/* 2% of an AG's blocks must be free for it to be chosen. */
 	minfree = mp->m_sb.sb_agblocks / 50;
@@ -319,7 +319,7 @@ xfs_filestream_lookup_ag(
 	xfs_agnumber_t		startag, ag = NULLAGNUMBER;
 	struct xfs_mru_cache_elem *mru;
 
-	ASSERT(S_ISREG(ip->i_d.di_mode));
+	ASSERT(S_ISREG(VFS_I(ip)->i_mode));
 
 	pip = xfs_filestream_get_parent(ip);
 	if (!pip)
diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
index 7c26f86..c1edb0f 100644
--- a/fs/xfs/xfs_icache.c
+++ b/fs/xfs/xfs_icache.c
@@ -63,6 +63,9 @@ xfs_inode_alloc(
 		return NULL;
 	}
 
+	/* VFS doesn't initialise i_mode! */
+	VFS_I(ip)->i_mode = 0;
+
 	XFS_STATS_INC(mp, vn_active);
 	ASSERT(atomic_read(&ip->i_pincount) == 0);
 	ASSERT(!spin_is_locked(&ip->i_flags_lock));
@@ -98,7 +101,7 @@ void
 xfs_inode_free(
 	struct xfs_inode	*ip)
 {
-	switch (ip->i_d.di_mode & S_IFMT) {
+	switch (VFS_I(ip)->i_mode & S_IFMT) {
 	case S_IFREG:
 	case S_IFDIR:
 	case S_IFLNK:
@@ -135,6 +138,35 @@ xfs_inode_free(
 }
 
 /*
+ * When we recycle a reclaimable inode, we need to re-initialise the VFS inode
+ * part of the structure. This is made more complex by the fact we store
+ * information about the on-disk values in the VFS inode and so we can't just
+ * overwrite it's values unconditionally. Hence we save the parameters we
+ * need to retain across reinitialisation, and rewrite them into the VFS inode
+ * after resetting it's state even if resetting fails.
+ */
+static int
+xfs_reinit_inode(
+	struct xfs_mount	*mp,
+	struct inode		*inode)
+{
+	uint32_t	nlink = inode->i_nlink;
+	umode_t		mode = inode->i_mode;
+	uint32_t	generation = inode->i_generation;
+	uint64_t	version = inode->i_version;
+	int		error;
+
+	error = inode_init_always(mp->m_super, inode);
+
+	set_nlink(inode, nlink);
+	inode->i_mode = mode;
+	inode->i_generation = generation;
+	inode->i_version = version;
+
+	return error;
+}
+
+/*
  * Check the validity of the inode we just found it the cache
  */
 static int
@@ -185,7 +217,7 @@ xfs_iget_cache_hit(
 	/*
 	 * If lookup is racing with unlink return an error immediately.
 	 */
-	if (ip->i_d.di_mode == 0 && !(flags & XFS_IGET_CREATE)) {
+	if (VFS_I(ip)->i_mode == 0 && !(flags & XFS_IGET_CREATE)) {
 		error = -ENOENT;
 		goto out_error;
 	}
@@ -208,7 +240,7 @@ xfs_iget_cache_hit(
 		spin_unlock(&ip->i_flags_lock);
 		rcu_read_unlock();
 
-		error = inode_init_always(mp->m_super, inode);
+		error = xfs_reinit_inode(mp, inode);
 		if (error) {
 			/*
 			 * Re-initializing the inode failed, and we are in deep
@@ -295,7 +327,7 @@ xfs_iget_cache_miss(
 
 	trace_xfs_iget_miss(ip);
 
-	if ((ip->i_d.di_mode == 0) && !(flags & XFS_IGET_CREATE)) {
+	if ((VFS_I(ip)->i_mode == 0) && !(flags & XFS_IGET_CREATE)) {
 		error = -ENOENT;
 		goto out_destroy;
 	}
@@ -444,7 +476,7 @@ again:
 	 * If we have a real type for an on-disk inode, we can setup the inode
 	 * now.	 If it's a new inode being created, xfs_ialloc will handle it.
 	 */
-	if (xfs_iflags_test(ip, XFS_INEW) && ip->i_d.di_mode != 0)
+	if (xfs_iflags_test(ip, XFS_INEW) && VFS_I(ip)->i_mode != 0)
 		xfs_setup_existing_inode(ip);
 	return 0;
 
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 96aee23..a0e7cdf 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -793,7 +793,7 @@ xfs_ialloc(
 	if (ip->i_d.di_version == 1)
 		ip->i_d.di_version = 2;
 
-	ip->i_d.di_mode = mode;
+	inode->i_mode = mode;
 	set_nlink(inode, nlink);
 	ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid());
 	ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid());
@@ -801,9 +801,8 @@ xfs_ialloc(
 
 	if (pip && XFS_INHERIT_GID(pip)) {
 		ip->i_d.di_gid = pip->i_d.di_gid;
-		if ((pip->i_d.di_mode & S_ISGID) && S_ISDIR(mode)) {
-			ip->i_d.di_mode |= S_ISGID;
-		}
+		if ((VFS_I(pip)->i_mode & S_ISGID) && S_ISDIR(mode))
+			inode->i_mode |= S_ISGID;
 	}
 
 	/*
@@ -812,10 +811,9 @@ xfs_ialloc(
 	 * (and only if the irix_sgid_inherit compatibility variable is set).
 	 */
 	if ((irix_sgid_inherit) &&
-	    (ip->i_d.di_mode & S_ISGID) &&
-	    (!in_group_p(xfs_gid_to_kgid(ip->i_d.di_gid)))) {
-		ip->i_d.di_mode &= ~S_ISGID;
-	}
+	    (inode->i_mode & S_ISGID) &&
+	    (!in_group_p(xfs_gid_to_kgid(ip->i_d.di_gid))))
+		inode->i_mode &= ~S_ISGID;
 
 	ip->i_d.di_size = 0;
 	ip->i_d.di_nextents = 0;
@@ -1407,7 +1405,7 @@ xfs_link(
 
 	trace_xfs_link(tdp, target_name);
 
-	ASSERT(!S_ISDIR(sip->i_d.di_mode));
+	ASSERT(!S_ISDIR(VFS_I(sip)->i_mode));
 
 	if (XFS_FORCED_SHUTDOWN(mp))
 		return -EIO;
@@ -1614,7 +1612,7 @@ xfs_release(
 	xfs_mount_t	*mp = ip->i_mount;
 	int		error;
 
-	if (!S_ISREG(ip->i_d.di_mode) || (ip->i_d.di_mode == 0))
+	if (!S_ISREG(VFS_I(ip)->i_mode) || (VFS_I(ip)->i_mode == 0))
 		return 0;
 
 	/* If this is a read-only mount, don't do this (would generate I/O) */
@@ -1849,7 +1847,7 @@ xfs_inactive(
 	 * If the inode is already free, then there can be nothing
 	 * to clean up here.
 	 */
-	if (ip->i_d.di_mode == 0) {
+	if (VFS_I(ip)->i_mode == 0) {
 		ASSERT(ip->i_df.if_real_bytes == 0);
 		ASSERT(ip->i_df.if_broot_bytes == 0);
 		return;
@@ -1873,7 +1871,7 @@ xfs_inactive(
 		return;
 	}
 
-	if (S_ISREG(ip->i_d.di_mode) &&
+	if (S_ISREG(VFS_I(ip)->i_mode) &&
 	    (ip->i_d.di_size != 0 || XFS_ISIZE(ip) != 0 ||
 	     ip->i_d.di_nextents > 0 || ip->i_delayed_blks > 0))
 		truncate = 1;
@@ -1882,7 +1880,7 @@ xfs_inactive(
 	if (error)
 		return;
 
-	if (S_ISLNK(ip->i_d.di_mode))
+	if (S_ISLNK(VFS_I(ip)->i_mode))
 		error = xfs_inactive_symlink(ip);
 	else if (truncate)
 		error = xfs_inactive_truncate(ip);
@@ -1944,7 +1942,7 @@ xfs_iunlink(
 	int		error;
 
 	ASSERT(VFS_I(ip)->i_nlink == 0 || ignore_linkcount);
-	ASSERT(ip->i_d.di_mode != 0);
+	ASSERT(VFS_I(ip)->i_mode != 0);
 
 	mp = tp->t_mountp;
 
@@ -2387,7 +2385,7 @@ xfs_ifree(
 	ASSERT(VFS_I(ip)->i_nlink == 0);
 	ASSERT(ip->i_d.di_nextents == 0);
 	ASSERT(ip->i_d.di_anextents == 0);
-	ASSERT(ip->i_d.di_size == 0 || !S_ISREG(ip->i_d.di_mode));
+	ASSERT(ip->i_d.di_size == 0 || !S_ISREG(VFS_I(ip)->i_mode));
 	ASSERT(ip->i_d.di_nblocks == 0);
 
 	/*
@@ -2401,7 +2399,7 @@ xfs_ifree(
 	if (error)
 		return error;
 
-	ip->i_d.di_mode = 0;		/* mark incore inode as free */
+	VFS_I(ip)->i_mode = 0;		/* mark incore inode as free */
 	ip->i_d.di_flags = 0;
 	ip->i_d.di_dmevmask = 0;
 	ip->i_d.di_forkoff = 0;		/* mark the attr fork not in use */
@@ -2498,7 +2496,7 @@ xfs_remove(
 {
 	xfs_mount_t		*mp = dp->i_mount;
 	xfs_trans_t             *tp = NULL;
-	int			is_dir = S_ISDIR(ip->i_d.di_mode);
+	int			is_dir = S_ISDIR(VFS_I(ip)->i_mode);
 	int                     error = 0;
 	xfs_bmap_free_t         free_list;
 	xfs_fsblock_t           first_block;
@@ -2743,7 +2741,7 @@ xfs_cross_rename(
 	if (dp1 != dp2) {
 		dp2_flags = XFS_ICHGTIME_MOD | XFS_ICHGTIME_CHG;
 
-		if (S_ISDIR(ip2->i_d.di_mode)) {
+		if (S_ISDIR(VFS_I(ip2)->i_mode)) {
 			error = xfs_dir_replace(tp, ip2, &xfs_name_dotdot,
 						dp1->i_ino, first_block,
 						free_list, spaceres);
@@ -2751,7 +2749,7 @@ xfs_cross_rename(
 				goto out_trans_abort;
 
 			/* transfer ip2 ".." reference to dp1 */
-			if (!S_ISDIR(ip1->i_d.di_mode)) {
+			if (!S_ISDIR(VFS_I(ip1)->i_mode)) {
 				error = xfs_droplink(tp, dp2);
 				if (error)
 					goto out_trans_abort;
@@ -2770,7 +2768,7 @@ xfs_cross_rename(
 			ip2_flags |= XFS_ICHGTIME_MOD | XFS_ICHGTIME_CHG;
 		}
 
-		if (S_ISDIR(ip1->i_d.di_mode)) {
+		if (S_ISDIR(VFS_I(ip1)->i_mode)) {
 			error = xfs_dir_replace(tp, ip1, &xfs_name_dotdot,
 						dp2->i_ino, first_block,
 						free_list, spaceres);
@@ -2778,7 +2776,7 @@ xfs_cross_rename(
 				goto out_trans_abort;
 
 			/* transfer ip1 ".." reference to dp2 */
-			if (!S_ISDIR(ip2->i_d.di_mode)) {
+			if (!S_ISDIR(VFS_I(ip2)->i_mode)) {
 				error = xfs_droplink(tp, dp1);
 				if (error)
 					goto out_trans_abort;
@@ -2875,7 +2873,7 @@ xfs_rename(
 	struct xfs_inode	*inodes[__XFS_SORT_INODES];
 	int			num_inodes = __XFS_SORT_INODES;
 	bool			new_parent = (src_dp != target_dp);
-	bool			src_is_directory = S_ISDIR(src_ip->i_d.di_mode);
+	bool			src_is_directory = S_ISDIR(VFS_I(src_ip)->i_mode);
 	int			spaceres;
 	int			error;
 
@@ -3004,7 +3002,7 @@ xfs_rename(
 		 * target and source are directories and that target can be
 		 * destroyed, or that neither is a directory.
 		 */
-		if (S_ISDIR(target_ip->i_d.di_mode)) {
+		if (S_ISDIR(VFS_I(target_ip)->i_mode)) {
 			/*
 			 * Make sure target dir is empty.
 			 */
@@ -3434,7 +3432,7 @@ xfs_iflush_int(
 			__func__, ip->i_ino, be16_to_cpu(dip->di_magic), dip);
 		goto corrupt_out;
 	}
-	if (S_ISREG(ip->i_d.di_mode)) {
+	if (S_ISREG(VFS_I(ip)->i_mode)) {
 		if (XFS_TEST_ERROR(
 		    (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS) &&
 		    (ip->i_d.di_format != XFS_DINODE_FMT_BTREE),
@@ -3444,7 +3442,7 @@ xfs_iflush_int(
 				__func__, ip->i_ino, ip);
 			goto corrupt_out;
 		}
-	} else if (S_ISDIR(ip->i_d.di_mode)) {
+	} else if (S_ISDIR(VFS_I(ip)->i_mode)) {
 		if (XFS_TEST_ERROR(
 		    (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS) &&
 		    (ip->i_d.di_format != XFS_DINODE_FMT_BTREE) &&
diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
index e74d13d..d627749 100644
--- a/fs/xfs/xfs_inode.h
+++ b/fs/xfs/xfs_inode.h
@@ -88,7 +88,7 @@ static inline struct inode *VFS_I(struct xfs_inode *ip)
  */
 static inline xfs_fsize_t XFS_ISIZE(struct xfs_inode *ip)
 {
-	if (S_ISREG(ip->i_d.di_mode))
+	if (S_ISREG(VFS_I(ip)->i_mode))
 		return i_size_read(VFS_I(ip));
 	return ip->i_d.di_size;
 }
@@ -369,7 +369,7 @@ static inline int xfs_isiflocked(struct xfs_inode *ip)
  */
 #define XFS_INHERIT_GID(pip)	\
 	(((pip)->i_mount->m_flags & XFS_MOUNT_GRPID) || \
-	 ((pip)->i_d.di_mode & S_ISGID))
+	 (VFS_I(pip)->i_mode & S_ISGID))
 
 int		xfs_release(struct xfs_inode *ip);
 void		xfs_inactive(struct xfs_inode *ip);
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index 334d657..bd9808f 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -333,7 +333,6 @@ xfs_inode_to_log_dinode(
 
 	to->di_magic = XFS_DINODE_MAGIC;
 
-	to->di_mode = from->di_mode;
 	to->di_version = from->di_version;
 	to->di_format = from->di_format;
 	to->di_uid = from->di_uid;
@@ -351,6 +350,7 @@ xfs_inode_to_log_dinode(
 	to->di_ctime.t_nsec = inode->i_ctime.tv_nsec;
 	to->di_nlink = inode->i_nlink;
 	to->di_gen = inode->i_generation;
+	to->di_mode = inode->i_mode;
 
 	to->di_size = from->di_size;
 	to->di_nblocks = from->di_nblocks;
diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
index 2c40c5f..5069bd8 100644
--- a/fs/xfs/xfs_ioctl.c
+++ b/fs/xfs/xfs_ioctl.c
@@ -962,7 +962,7 @@ xfs_set_diflags(
 		di_flags |= XFS_DIFLAG_NODEFRAG;
 	if (xflags & XFS_XFLAG_FILESTREAM)
 		di_flags |= XFS_DIFLAG_FILESTREAM;
-	if (S_ISDIR(ip->i_d.di_mode)) {
+	if (S_ISDIR(VFS_I(ip)->i_mode)) {
 		if (xflags & XFS_XFLAG_RTINHERIT)
 			di_flags |= XFS_DIFLAG_RTINHERIT;
 		if (xflags & XFS_XFLAG_NOSYMLINKS)
@@ -971,7 +971,7 @@ xfs_set_diflags(
 			di_flags |= XFS_DIFLAG_EXTSZINHERIT;
 		if (xflags & XFS_XFLAG_PROJINHERIT)
 			di_flags |= XFS_DIFLAG_PROJINHERIT;
-	} else if (S_ISREG(ip->i_d.di_mode)) {
+	} else if (S_ISREG(VFS_I(ip)->i_mode)) {
 		if (xflags & XFS_XFLAG_REALTIME)
 			di_flags |= XFS_DIFLAG_REALTIME;
 		if (xflags & XFS_XFLAG_EXTSIZE)
@@ -1112,14 +1112,14 @@ xfs_ioctl_setattr_check_extsize(
 {
 	struct xfs_mount	*mp = ip->i_mount;
 
-	if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) && !S_ISREG(ip->i_d.di_mode))
+	if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) && !S_ISREG(VFS_I(ip)->i_mode))
 		return -EINVAL;
 
 	if ((fa->fsx_xflags & XFS_XFLAG_EXTSZINHERIT) &&
-	    !S_ISDIR(ip->i_d.di_mode))
+	    !S_ISDIR(VFS_I(ip)->i_mode))
 		return -EINVAL;
 
-	if (S_ISREG(ip->i_d.di_mode) && ip->i_d.di_nextents &&
+	if (S_ISREG(VFS_I(ip)->i_mode) && ip->i_d.di_nextents &&
 	    ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) != fa->fsx_extsize))
 		return -EINVAL;
 
@@ -1240,9 +1240,9 @@ xfs_ioctl_setattr(
 	 * successful return from chown()
 	 */
 
-	if ((ip->i_d.di_mode & (S_ISUID|S_ISGID)) &&
+	if ((VFS_I(ip)->i_mode & (S_ISUID|S_ISGID)) &&
 	    !capable_wrt_inode_uidgid(VFS_I(ip), CAP_FSETID))
-		ip->i_d.di_mode &= ~(S_ISUID|S_ISGID);
+		VFS_I(ip)->i_mode &= ~(S_ISUID|S_ISGID);
 
 	/* Change the ownerships and register project quota modifications */
 	if (xfs_get_projid(ip) != fa->fsx_projid) {
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index 397ce85..34bff1e 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -454,7 +454,7 @@ xfs_vn_getattr(
 
 	stat->size = XFS_ISIZE(ip);
 	stat->dev = inode->i_sb->s_dev;
-	stat->mode = ip->i_d.di_mode;
+	stat->mode = inode->i_mode;
 	stat->nlink = inode->i_nlink;
 	stat->uid = inode->i_uid;
 	stat->gid = inode->i_gid;
@@ -501,9 +501,6 @@ xfs_setattr_mode(
 
 	ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL));
 
-	ip->i_d.di_mode &= S_IFMT;
-	ip->i_d.di_mode |= mode & ~S_IFMT;
-
 	inode->i_mode &= S_IFMT;
 	inode->i_mode |= mode & ~S_IFMT;
 }
@@ -647,9 +644,9 @@ xfs_setattr_nonsize(
 		 * The set-user-ID and set-group-ID bits of a file will be
 		 * cleared upon successful return from chown()
 		 */
-		if ((ip->i_d.di_mode & (S_ISUID|S_ISGID)) &&
+		if ((inode->i_mode & (S_ISUID|S_ISGID)) &&
 		    !capable(CAP_FSETID))
-			ip->i_d.di_mode &= ~(S_ISUID|S_ISGID);
+			inode->i_mode &= ~(S_ISUID|S_ISGID);
 
 		/*
 		 * Change the ownerships and register quota modifications
@@ -759,7 +756,7 @@ xfs_setattr_size(
 
 	ASSERT(xfs_isilocked(ip, XFS_IOLOCK_EXCL));
 	ASSERT(xfs_isilocked(ip, XFS_MMAPLOCK_EXCL));
-	ASSERT(S_ISREG(ip->i_d.di_mode));
+	ASSERT(S_ISREG(inode->i_mode));
 	ASSERT((iattr->ia_valid & (ATTR_UID|ATTR_GID|ATTR_ATIME|ATTR_ATIME_SET|
 		ATTR_MTIME_SET|ATTR_KILL_PRIV|ATTR_TIMES_SET)) == 0);
 
@@ -1211,7 +1208,6 @@ xfs_setup_inode(
 	/* make the inode look hashed for the writeback code */
 	hlist_add_fake(&inode->i_hash);
 
-	inode->i_mode	= ip->i_d.di_mode;
 	inode->i_uid    = xfs_uid_to_kuid(ip->i_d.di_uid);
 	inode->i_gid    = xfs_gid_to_kgid(ip->i_d.di_gid);
 
diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
index 6162e65..ce73eb3 100644
--- a/fs/xfs/xfs_itable.c
+++ b/fs/xfs/xfs_itable.c
@@ -88,7 +88,6 @@ xfs_bulkstat_one_int(
 	buf->bs_projid_lo = dic->di_projid_lo;
 	buf->bs_projid_hi = dic->di_projid_hi;
 	buf->bs_ino = ino;
-	buf->bs_mode = dic->di_mode;
 	buf->bs_uid = dic->di_uid;
 	buf->bs_gid = dic->di_gid;
 	buf->bs_size = dic->di_size;
@@ -101,6 +100,7 @@ xfs_bulkstat_one_int(
 	buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec;
 	buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec;
 	buf->bs_gen = inode->i_generation;
+	buf->bs_mode = inode->i_mode;
 
 	buf->bs_xflags = xfs_ip2xflags(ip);
 	buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog;
diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
index 611c25c..bd6f23b 100644
--- a/fs/xfs/xfs_log_recover.c
+++ b/fs/xfs/xfs_log_recover.c
@@ -4338,7 +4338,7 @@ xlog_recover_process_one_iunlink(
 		goto fail_iput;
 
 	ASSERT(VFS_I(ip)->i_nlink == 0);
-	ASSERT(ip->i_d.di_mode != 0);
+	ASSERT(VFS_I(ip)->i_mode != 0);
 
 	/* setup for the next pass */
 	agino = be32_to_cpu(dip->di_next_unlinked);
diff --git a/fs/xfs/xfs_mount.c b/fs/xfs/xfs_mount.c
index bb753b3..d306105 100644
--- a/fs/xfs/xfs_mount.c
+++ b/fs/xfs/xfs_mount.c
@@ -865,7 +865,7 @@ xfs_mountfs(
 
 	ASSERT(rip != NULL);
 
-	if (unlikely(!S_ISDIR(rip->i_d.di_mode))) {
+	if (unlikely(!S_ISDIR(VFS_I(rip)->i_mode))) {
 		xfs_warn(mp, "corrupted root inode %llu: not a directory",
 			(unsigned long long)rip->i_ino);
 		xfs_iunlock(rip, XFS_ILOCK_EXCL);
-- 
2.5.0

From dave@fromorbit.com  Thu Jan 14 00:10:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 9B05D7F58
	for ; Thu, 14 Jan 2016 00:10:29 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 554EC8F804B
	for ; Wed, 13 Jan 2016 22:10:29 -0800 (PST)
X-ASG-Debug-ID: 1452751819-04bdf06f6cefa40003-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id r6jrj4HjO8DeZenD for ; Wed, 13 Jan 2016 22:10:25 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DrBwA8O5dWPGwtLHleKAECgw+BP4ZigXmfXgaRG4oUgTtNAQEBAQEBBwEBAQFAAT+ENQEFJy8zCBgZGDkDBxQZiC3AP4YOikQiDIQSBYdljzCPOI0ihWmIb4IuAQsBNwMcgXEqNIRWgUIBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:29 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OU-MV
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001V1-LX
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH 3/8] xfs; cull unnecessary icdinode fields
Date: Thu, 14 Jan 2016 17:09:20 +1100
X-ASG-Orig-Subj: [PATCH 3/8] xfs; cull unnecessary icdinode fields
Message-Id: <1452751765-4420-4-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
In-Reply-To: <1452751765-4420-1-git-send-email-david@fromorbit.com>
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751825
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

Now that the struct xfs_icdinode is not directly related to the
on-disk format, we can cull things in it we really don't need to
store:

	- magic number never changes
	- padding is not necessary
	- next_unlinked is never used
	- inode number is redundant
	- uuid is redundant
	- lsn is accessed directly from dinode
	- inode CRC is only accessed directly from dinode

Hence we can remove these from the struct xfs_icdinode and redirect
the code that uses them to the xfs_dinode appripriately.  This
reduces the size of the struct icdinode from 152 bytes to 88 bytes,
and removes a fair chunk of unnecessary code, too.

Signed-off-by: Dave Chinner 
---
 fs/xfs/libxfs/xfs_inode_buf.c | 39 +++++++++++++--------------------------
 fs/xfs/libxfs/xfs_inode_buf.h | 27 +++++++--------------------
 fs/xfs/xfs_inode.c            | 19 +------------------
 fs/xfs/xfs_inode_item.c       | 19 +++++++++++--------
 4 files changed, 32 insertions(+), 72 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
index 26e3c46..17ff4e8 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.c
+++ b/fs/xfs/libxfs/xfs_inode_buf.c
@@ -202,7 +202,6 @@ xfs_inode_from_disk(
 	struct xfs_icdinode	*to = &ip->i_d;
 	struct inode		*inode = VFS_I(ip);
 
-	to->di_magic = be16_to_cpu(from->di_magic);
 	to->di_mode = be16_to_cpu(from->di_mode);
 	to->di_version = from ->di_version;
 	to->di_format = from->di_format;
@@ -212,7 +211,6 @@ xfs_inode_from_disk(
 	to->di_nlink = be32_to_cpu(from->di_nlink);
 	to->di_projid_lo = be16_to_cpu(from->di_projid_lo);
 	to->di_projid_hi = be16_to_cpu(from->di_projid_hi);
-	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
 	to->di_flushiter = be16_to_cpu(from->di_flushiter);
 
 	/*
@@ -245,24 +243,22 @@ xfs_inode_from_disk(
 		to->di_crtime.t_sec = be32_to_cpu(from->di_crtime.t_sec);
 		to->di_crtime.t_nsec = be32_to_cpu(from->di_crtime.t_nsec);
 		to->di_flags2 = be64_to_cpu(from->di_flags2);
-		to->di_ino = be64_to_cpu(from->di_ino);
-		to->di_lsn = be64_to_cpu(from->di_lsn);
-		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
-		uuid_copy(&to->di_uuid, &from->di_uuid);
 	}
 }
 
 void
 xfs_inode_to_disk(
 	struct xfs_inode	*ip,
-	struct xfs_dinode	*to)
+	struct xfs_dinode	*to,
+	xfs_lsn_t		lsn)
 {
 	struct xfs_icdinode	*from = &ip->i_d;
 	struct inode		*inode = VFS_I(ip);
 
-	to->di_magic = cpu_to_be16(from->di_magic);
+	to->di_magic = cpu_to_be16(XFS_DINODE_MAGIC);
+
 	to->di_mode = cpu_to_be16(from->di_mode);
-	to->di_version = from ->di_version;
+	to->di_version = from->di_version;
 	to->di_format = from->di_format;
 	to->di_onlink = cpu_to_be16(from->di_onlink);
 	to->di_uid = cpu_to_be32(from->di_uid);
@@ -270,8 +266,8 @@ xfs_inode_to_disk(
 	to->di_nlink = cpu_to_be32(from->di_nlink);
 	to->di_projid_lo = cpu_to_be16(from->di_projid_lo);
 	to->di_projid_hi = cpu_to_be16(from->di_projid_hi);
-	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
 
+	memset(to->di_pad, 0, sizeof(to->di_pad));
 	to->di_atime.t_sec = cpu_to_be32(inode->i_atime.tv_sec);
 	to->di_atime.t_nsec = cpu_to_be32(inode->i_atime.tv_nsec);
 	to->di_mtime.t_sec = cpu_to_be32(inode->i_mtime.tv_sec);
@@ -296,10 +292,11 @@ xfs_inode_to_disk(
 		to->di_crtime.t_sec = cpu_to_be32(from->di_crtime.t_sec);
 		to->di_crtime.t_nsec = cpu_to_be32(from->di_crtime.t_nsec);
 		to->di_flags2 = cpu_to_be64(from->di_flags2);
-		to->di_ino = cpu_to_be64(from->di_ino);
-		to->di_lsn = cpu_to_be64(from->di_lsn);
-		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
-		uuid_copy(&to->di_uuid, &from->di_uuid);
+
+		to->di_ino = cpu_to_be64(ip->i_ino);
+		to->di_lsn = cpu_to_be64(lsn);
+		memset(to->di_pad2, 0, sizeof(to->di_pad2));
+		uuid_copy(&to->di_uuid, &ip->i_mount->m_sb.sb_meta_uuid);
 		to->di_flushiter = 0;
 	} else {
 		to->di_flushiter = cpu_to_be16(from->di_flushiter);
@@ -434,13 +431,10 @@ xfs_iread(
 	    !(mp->m_flags & XFS_MOUNT_IKEEP)) {
 		/* initialise the on-disk inode core */
 		memset(&ip->i_d, 0, sizeof(ip->i_d));
-		ip->i_d.di_magic = XFS_DINODE_MAGIC;
 		ip->i_d.di_gen = prandom_u32();
-		if (xfs_sb_version_hascrc(&mp->m_sb)) {
+		if (xfs_sb_version_hascrc(&mp->m_sb))
 			ip->i_d.di_version = 3;
-			ip->i_d.di_ino = ip->i_ino;
-			uuid_copy(&ip->i_d.di_uuid, &mp->m_sb.sb_meta_uuid);
-		} else
+		else
 			ip->i_d.di_version = 2;
 		return 0;
 	}
@@ -484,16 +478,10 @@ xfs_iread(
 		 * Partial initialisation of the in-core inode. Just the bits
 		 * that xfs_ialloc won't overwrite or relies on being correct.
 		 */
-		ip->i_d.di_magic = be16_to_cpu(dip->di_magic);
 		ip->i_d.di_version = dip->di_version;
 		ip->i_d.di_gen = be32_to_cpu(dip->di_gen);
 		ip->i_d.di_flushiter = be16_to_cpu(dip->di_flushiter);
 
-		if (dip->di_version == 3) {
-			ip->i_d.di_ino = be64_to_cpu(dip->di_ino);
-			uuid_copy(&ip->i_d.di_uuid, &dip->di_uuid);
-		}
-
 		/*
 		 * Make sure to pull in the mode here as well in
 		 * case the inode is released without being used.
@@ -514,7 +502,6 @@ xfs_iread(
 	 */
 	if (ip->i_d.di_version == 1) {
 		ip->i_d.di_version = 2;
-		memset(&(ip->i_d.di_pad[0]), 0, sizeof(ip->i_d.di_pad));
 		ip->i_d.di_nlink = ip->i_d.di_onlink;
 		ip->i_d.di_onlink = 0;
 		xfs_set_projid(ip, 0);
diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index adcc9bf..69d626e 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -22,24 +22,22 @@ struct xfs_inode;
 struct xfs_dinode;
 
 /*
- * In memory representation of the XFS inode. This is held in the in-core
- * struct xfs_inode to represent the on disk values, but no longer needs to be
- * identical to the on-disk structure as it is always translated to on-disk
- * format specific structures at the appropriate time.
+ * In memory representation of the XFS inode. This is held in the in-core struct
+ * xfs_inode to represent the on disk values, but it's struct is in no way
+ * related to what is stored on disk. That is, this structure is always
+ * translated to on-disk format specific structures at the appropriate time.
  */
 struct xfs_icdinode {
-	__uint16_t	di_magic;	/* inode magic # = XFS_DINODE_MAGIC */
 	__uint16_t	di_mode;	/* mode and type of file */
 	__int8_t	di_version;	/* inode version */
 	__int8_t	di_format;	/* format of di_c data */
 	__uint16_t	di_onlink;	/* old number of links to file */
+	__uint16_t	di_flushiter;	/* incremented on flush */
 	__uint32_t	di_uid;		/* owner's user id */
 	__uint32_t	di_gid;		/* owner's group id */
 	__uint32_t	di_nlink;	/* number of links to file */
 	__uint16_t	di_projid_lo;	/* lower part of owner's project id */
 	__uint16_t	di_projid_hi;	/* higher part of owner's project id */
-	__uint8_t	di_pad[6];	/* unused, zeroed space */
-	__uint16_t	di_flushiter;	/* incremented on flush */
 	xfs_fsize_t	di_size;	/* number of bytes in file */
 	xfs_rfsblock_t	di_nblocks;	/* # of direct & btree blocks used */
 	xfs_extlen_t	di_extsize;	/* basic/minimum extent size for file */
@@ -52,22 +50,10 @@ struct xfs_icdinode {
 	__uint16_t	di_flags;	/* random flags, XFS_DIFLAG_... */
 	__uint32_t	di_gen;		/* generation number */
 
-	/* di_next_unlinked is the only non-core field in the old dinode */
-	xfs_agino_t	di_next_unlinked;/* agi unlinked list ptr */
-
-	/* start of the extended dinode, writable fields */
-	__uint32_t	di_crc;		/* CRC of the inode */
 	__uint64_t	di_changecount;	/* number of attribute changes */
-	xfs_lsn_t	di_lsn;		/* flush sequence */
 	__uint64_t	di_flags2;	/* more random flags */
-	__uint8_t	di_pad2[16];	/* more padding for future expansion */
 
-	/* fields only written to during inode creation */
 	xfs_ictimestamp_t di_crtime;	/* time created */
-	xfs_ino_t	di_ino;		/* inode number */
-	uuid_t		di_uuid;	/* UUID of the filesystem */
-
-	/* structure must be padded to 64 bit alignment */
 };
 
 /*
@@ -86,7 +72,8 @@ int	xfs_imap_to_bp(struct xfs_mount *, struct xfs_trans *,
 int	xfs_iread(struct xfs_mount *, struct xfs_trans *,
 		  struct xfs_inode *, uint);
 void	xfs_dinode_calc_crc(struct xfs_mount *, struct xfs_dinode *);
-void	xfs_inode_to_disk(struct xfs_inode *ip, struct xfs_dinode *to);
+void	xfs_inode_to_disk(struct xfs_inode *ip, struct xfs_dinode *to,
+			  xfs_lsn_t lsn);
 void	xfs_inode_from_disk(struct xfs_inode *ip, struct xfs_dinode *from);
 void	xfs_log_dinode_to_disk(struct xfs_log_dinode *from,
 			       struct xfs_dinode *to);
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 3e9b8a0..3f020b5 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -800,7 +800,6 @@ xfs_ialloc(
 	ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid());
 	ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid());
 	xfs_set_projid(ip, prid);
-	memset(&(ip->i_d.di_pad[0]), 0, sizeof(ip->i_d.di_pad));
 
 	if (pip && XFS_INHERIT_GID(pip)) {
 		ip->i_d.di_gid = pip->i_d.di_gid;
@@ -838,13 +837,8 @@ xfs_ialloc(
 	ip->i_d.di_flags = 0;
 
 	if (ip->i_d.di_version == 3) {
-		ASSERT(ip->i_d.di_ino == ino);
-		ASSERT(uuid_equal(&ip->i_d.di_uuid, &mp->m_sb.sb_meta_uuid));
-		ip->i_d.di_crc = 0;
 		ip->i_d.di_changecount = 1;
-		ip->i_d.di_lsn = 0;
 		ip->i_d.di_flags2 = 0;
-		memset(&(ip->i_d.di_pad2[0]), 0, sizeof(ip->i_d.di_pad2));
 		ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec;
 		ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec;
 	}
@@ -3450,13 +3444,6 @@ xfs_iflush_int(
 			__func__, ip->i_ino, be16_to_cpu(dip->di_magic), dip);
 		goto corrupt_out;
 	}
-	if (XFS_TEST_ERROR(ip->i_d.di_magic != XFS_DINODE_MAGIC,
-				mp, XFS_ERRTAG_IFLUSH_2, XFS_RANDOM_IFLUSH_2)) {
-		xfs_alert_tag(mp, XFS_PTAG_IFLUSH,
-			"%s: Bad inode %Lu, ptr 0x%p, magic number 0x%x",
-			__func__, ip->i_ino, ip, ip->i_d.di_magic);
-		goto corrupt_out;
-	}
 	if (S_ISREG(ip->i_d.di_mode)) {
 		if (XFS_TEST_ERROR(
 		    (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS) &&
@@ -3515,7 +3502,7 @@ xfs_iflush_int(
 	 * copy out the core of the inode, because if the inode is dirty at all
 	 * the core must be.
 	 */
-	xfs_inode_to_disk(ip, dip);
+	xfs_inode_to_disk(ip, dip, iip->ili_item.li_lsn);
 
 	/* Wrap, we never let the log put out DI_MAX_FLUSH */
 	if (ip->i_d.di_flushiter == DI_MAX_FLUSH)
@@ -3567,10 +3554,6 @@ xfs_iflush_int(
 	 */
 	xfs_buf_attach_iodone(bp, xfs_iflush_done, &iip->ili_item);
 
-	/* update the lsn in the on disk inode if required */
-	if (ip->i_d.di_version == 3)
-		dip->di_lsn = cpu_to_be64(iip->ili_item.li_lsn);
-
 	/* generate the checksum. */
 	xfs_dinode_calc_crc(mp, dip);
 
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index 9dcbf58..ae60087 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -325,12 +325,14 @@ xfs_inode_item_format_attr_fork(
 static void
 xfs_inode_to_log_dinode(
 	struct xfs_inode	*ip,
-	struct xfs_log_dinode	*to)
+	struct xfs_log_dinode	*to,
+	xfs_lsn_t		lsn)
 {
 	struct xfs_icdinode	*from = &ip->i_d;
 	struct inode		*inode = VFS_I(ip);
 
-	to->di_magic = from->di_magic;
+	to->di_magic = XFS_DINODE_MAGIC;
+
 	to->di_mode = from->di_mode;
 	to->di_version = from->di_version;
 	to->di_format = from->di_format;
@@ -340,8 +342,8 @@ xfs_inode_to_log_dinode(
 	to->di_nlink = from->di_nlink;
 	to->di_projid_lo = from->di_projid_lo;
 	to->di_projid_hi = from->di_projid_hi;
-	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
 
+	memset(to->di_pad, 0, sizeof(to->di_pad));
 	to->di_atime.t_sec = inode->i_atime.tv_sec;
 	to->di_atime.t_nsec = inode->i_atime.tv_nsec;
 	to->di_mtime.t_sec = inode->i_mtime.tv_sec;
@@ -366,10 +368,11 @@ xfs_inode_to_log_dinode(
 		to->di_crtime.t_sec = from->di_crtime.t_sec;
 		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
 		to->di_flags2 = from->di_flags2;
-		to->di_ino = from->di_ino;
-		to->di_lsn = from->di_lsn;
-		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
-		uuid_copy(&to->di_uuid, &from->di_uuid);
+
+		to->di_ino = ip->i_ino;
+		to->di_lsn = lsn;
+		memset(to->di_pad2, 0, sizeof(to->di_pad2));
+		uuid_copy(&to->di_uuid, &ip->i_mount->m_sb.sb_uuid);
 		to->di_flushiter = 0;
 	} else {
 		to->di_flushiter = from->di_flushiter;
@@ -390,7 +393,7 @@ xfs_inode_item_format_core(
 	struct xfs_log_dinode	*dic;
 
 	dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE);
-	xfs_inode_to_log_dinode(ip, dic);
+	xfs_inode_to_log_dinode(ip, dic, ip->i_itemp->ili_item.li_lsn);
 	xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version));
 }
 
-- 
2.5.0

From dave@fromorbit.com  Thu Jan 14 00:10:30 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 7C66E7F50
	for ; Thu, 14 Jan 2016 00:10:29 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 1E813AC003
	for ; Wed, 13 Jan 2016 22:10:29 -0800 (PST)
X-ASG-Debug-ID: 1452751818-04bdf06f6eefa40001-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id di6Adl9VBZ4ZOxGF for ; Wed, 13 Jan 2016 22:10:19 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DuBwA8O5dWPGwtLHleKAECgw9SbYhbn0ACHAaBDZAOhCeHKE0BAQEBAQEHAQEBAUABP4URO4ECAweIWg+fVqBahg6INII2hBoFlxWFQ4l1h26FNESOFIIuAQsBgkcqNAGGFwEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:29 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OR-Kr
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001Uo-Jt
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH v2 0/8] xfs: shrink the xfs_icdinode
Date: Thu, 14 Jan 2016 17:09:17 +1100
X-ASG-Orig-Subj: [PATCH v2 0/8] xfs: shrink the xfs_icdinode
Message-Id: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751818
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Hi folks,

This is a followup to the original RFC patchset posted here:

http://oss.sgi.com/archives/xfs/2016-01/msg00366.html

This patchset actually works, and passes through xfstests without
any regressions at all. So feel free to test and comment on this
version.

-Dave.

Version 2
- timestamps are signed, so need casting to int before being stored
  in a 64 bit variable when read from disk. Fixes generic/258
  failure (patch 2).
- nlink wasn't being accounted correctly for O_TMPFILE files and
  rename whiteouts. Fixes generic/078 and others. (patch 5)
- VFS does not initialise i_mode, ever, so we need to initialise it
  directly after slab allocation to ensure that we correctly detect
  new file creates and hence don't add the inode the sb list twice,
  corrupting it. (busy inodes after unmount, self destruct in 5
  seconds, have a nice day!) (patch 8)
- when reallocating a reclaimable inode, we need to save the
  variables that hold on-disk values across the inode_init_always()
  call, otherwise we lose them. This results in corruption of these
  fields. (patch 8)

From dave@fromorbit.com  Thu Jan 14 00:10:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 4AA447F4E
	for ; Thu, 14 Jan 2016 00:10:30 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 12287304059
	for ; Wed, 13 Jan 2016 22:10:30 -0800 (PST)
X-ASG-Debug-ID: 1452751818-04bdf06f6eefa40004-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id ZK22wYS6gngXUsdX for ; Wed, 13 Jan 2016 22:10:25 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DpBwA8O5dWPGwtLHleKAECgw+BP4ZigXmfXgaRG4tPTQEBAQEBAQcBAQEBQAE/hDUBBScvMwgYGRg5AwcUGYgtwD+GDopmDIQSBZcVnFqOWIIuAQsBNwMcgXEqNIRWgUIBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 14 Jan 2016 16:39:29 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aJb68-0002OS-LL
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aJb68-0001Ur-KW
	for xfs@oss.sgi.com; Thu, 14 Jan 2016 17:09:28 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH 1/8] xfs: introduce inode log format object
Date: Thu, 14 Jan 2016 17:09:18 +1100
X-ASG-Orig-Subj: [PATCH 1/8] xfs: introduce inode log format object
Message-Id: <1452751765-4420-2-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
In-Reply-To: <1452751765-4420-1-git-send-email-david@fromorbit.com>
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452751825
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26112
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

We currently carry around and log an entire inode core in the
struct xfs_inode. A lot of the information in the inode core is
duplicated in the VFS inode, but we cannot remove this duplication
of infomration because the inode core is logged directly in
xfs_inode_item_format().

Add a new function xfs_inode_item_format_core() that copies the
inode core data into a struct xfs_icdinode that is pulled directly
from the log vector buffer. This means we no longer directly
copy the inode core, but copy the structures one member at a time.
This will be slightly less efficient than copying, but will allow us
to remove duplicate and unnecessary items from the struct xfs_inode.

To enable us to do this, call the new structure a xfs_log_dinode,
so that we know it's different to the physical xfs_dinode and the
in-core xfs_icdinode.

Signed-off-by: Dave Chinner 
---
 fs/xfs/libxfs/xfs_inode_buf.c  |   8 +--
 fs/xfs/libxfs/xfs_inode_buf.h  |  53 ++++++++++++++++-
 fs/xfs/libxfs/xfs_log_format.h |  15 ++---
 fs/xfs/xfs_icache.c            |   2 +-
 fs/xfs/xfs_inode.c             |   6 +-
 fs/xfs/xfs_inode.h             |   2 +-
 fs/xfs/xfs_inode_item.c        | 128 +++++++++++++++++++++++++++++++++++++++--
 fs/xfs/xfs_inode_item.h        |   2 +
 fs/xfs/xfs_log_recover.c       |  52 +++++++++--------
 9 files changed, 221 insertions(+), 47 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
index 1aabfda..63d46bf 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.c
+++ b/fs/xfs/libxfs/xfs_inode_buf.c
@@ -196,8 +196,8 @@ xfs_imap_to_bp(
 
 void
 xfs_dinode_from_disk(
-	xfs_icdinode_t		*to,
-	xfs_dinode_t		*from)
+	struct xfs_icdinode	*to,
+	struct xfs_dinode	*from)
 {
 	to->di_magic = be16_to_cpu(from->di_magic);
 	to->di_mode = be16_to_cpu(from->di_mode);
@@ -243,8 +243,8 @@ xfs_dinode_from_disk(
 
 void
 xfs_dinode_to_disk(
-	xfs_dinode_t		*to,
-	xfs_icdinode_t		*from)
+	struct xfs_dinode	*to,
+	struct xfs_icdinode	*from)
 {
 	to->di_magic = cpu_to_be16(from->di_magic);
 	to->di_mode = cpu_to_be16(from->di_mode);
diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
index 9308c47..642f2a2 100644
--- a/fs/xfs/libxfs/xfs_inode_buf.h
+++ b/fs/xfs/libxfs/xfs_inode_buf.h
@@ -20,7 +20,58 @@
 
 struct xfs_inode;
 struct xfs_dinode;
-struct xfs_icdinode;
+
+/*
+ * In memory representation of the XFS inode. This is held in the in-core
+ * struct xfs_inode to represent the on disk values, but no longer needs to be
+ * identical to the on-disk structure as it is always translated to on-disk
+ * format specific structures at the appropriate time.
+ */
+struct xfs_icdinode {
+	__uint16_t	di_magic;	/* inode magic # = XFS_DINODE_MAGIC */
+	__uint16_t	di_mode;	/* mode and type of file */
+	__int8_t	di_version;	/* inode version */
+	__int8_t	di_format;	/* format of di_c data */
+	__uint16_t	di_onlink;	/* old number of links to file */
+	__uint32_t	di_uid;		/* owner's user id */
+	__uint32_t	di_gid;		/* owner's group id */
+	__uint32_t	di_nlink;	/* number of links to file */
+	__uint16_t	di_projid_lo;	/* lower part of owner's project id */
+	__uint16_t	di_projid_hi;	/* higher part of owner's project id */
+	__uint8_t	di_pad[6];	/* unused, zeroed space */
+	__uint16_t	di_flushiter;	/* incremented on flush */
+	xfs_ictimestamp_t di_atime;	/* time last accessed */
+	xfs_ictimestamp_t di_mtime;	/* time last modified */
+	xfs_ictimestamp_t di_ctime;	/* time created/inode modified */
+	xfs_fsize_t	di_size;	/* number of bytes in file */
+	xfs_rfsblock_t	di_nblocks;	/* # of direct & btree blocks used */
+	xfs_extlen_t	di_extsize;	/* basic/minimum extent size for file */
+	xfs_extnum_t	di_nextents;	/* number of extents in data fork */
+	xfs_aextnum_t	di_anextents;	/* number of extents in attribute fork*/
+	__uint8_t	di_forkoff;	/* attr fork offs, <<3 for 64b align */
+	__int8_t	di_aformat;	/* format of attr fork's data */
+	__uint32_t	di_dmevmask;	/* DMIG event mask */
+	__uint16_t	di_dmstate;	/* DMIG state info */
+	__uint16_t	di_flags;	/* random flags, XFS_DIFLAG_... */
+	__uint32_t	di_gen;		/* generation number */
+
+	/* di_next_unlinked is the only non-core field in the old dinode */
+	xfs_agino_t	di_next_unlinked;/* agi unlinked list ptr */
+
+	/* start of the extended dinode, writable fields */
+	__uint32_t	di_crc;		/* CRC of the inode */
+	__uint64_t	di_changecount;	/* number of attribute changes */
+	xfs_lsn_t	di_lsn;		/* flush sequence */
+	__uint64_t	di_flags2;	/* more random flags */
+	__uint8_t	di_pad2[16];	/* more padding for future expansion */
+
+	/* fields only written to during inode creation */
+	xfs_ictimestamp_t di_crtime;	/* time created */
+	xfs_ino_t	di_ino;		/* inode number */
+	uuid_t		di_uuid;	/* UUID of the filesystem */
+
+	/* structure must be padded to 64 bit alignment */
+};
 
 /*
  * Inode location information.  Stored in the inode and passed to
diff --git a/fs/xfs/libxfs/xfs_log_format.h b/fs/xfs/libxfs/xfs_log_format.h
index 2653146..d00ed63 100644
--- a/fs/xfs/libxfs/xfs_log_format.h
+++ b/fs/xfs/libxfs/xfs_log_format.h
@@ -290,6 +290,7 @@ typedef struct xfs_inode_log_format_64 {
 	__int32_t		ilf_boffset;	/* off of inode in buffer */
 } xfs_inode_log_format_64_t;
 
+
 /*
  * Flags for xfs_trans_log_inode flags field.
  */
@@ -360,10 +361,10 @@ typedef struct xfs_ictimestamp {
 } xfs_ictimestamp_t;
 
 /*
- * NOTE:  This structure must be kept identical to struct xfs_dinode
- *	  except for the endianness annotations.
+ * Define the format of the inode core that is logged. This structure must be
+ * kept identical to struct xfs_dinode except for the endianness annotations.
  */
-typedef struct xfs_icdinode {
+struct xfs_log_dinode {
 	__uint16_t	di_magic;	/* inode magic # = XFS_DINODE_MAGIC */
 	__uint16_t	di_mode;	/* mode and type of file */
 	__int8_t	di_version;	/* inode version */
@@ -407,13 +408,13 @@ typedef struct xfs_icdinode {
 	uuid_t		di_uuid;	/* UUID of the filesystem */
 
 	/* structure must be padded to 64 bit alignment */
-} xfs_icdinode_t;
+};
 
-static inline uint xfs_icdinode_size(int version)
+static inline uint xfs_log_dinode_size(int version)
 {
 	if (version == 3)
-		return sizeof(struct xfs_icdinode);
-	return offsetof(struct xfs_icdinode, di_next_unlinked);
+		return sizeof(struct xfs_log_dinode);
+	return offsetof(struct xfs_log_dinode, di_next_unlinked);
 }
 
 /*
diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
index d7a490f..7c26f86 100644
--- a/fs/xfs/xfs_icache.c
+++ b/fs/xfs/xfs_icache.c
@@ -79,7 +79,7 @@ xfs_inode_alloc(
 	memset(&ip->i_df, 0, sizeof(xfs_ifork_t));
 	ip->i_flags = 0;
 	ip->i_delayed_blks = 0;
-	memset(&ip->i_d, 0, sizeof(xfs_icdinode_t));
+	memset(&ip->i_d, 0, sizeof(ip->i_d));
 
 	return ip;
 }
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index ae3758a..7e24232 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -650,9 +650,9 @@ _xfs_dic2xflags(
 
 uint
 xfs_ip2xflags(
-	xfs_inode_t		*ip)
+	struct xfs_inode	*ip)
 {
-	xfs_icdinode_t		*dic = &ip->i_d;
+	struct xfs_icdinode	*dic = &ip->i_d;
 
 	return _xfs_dic2xflags(dic->di_flags) |
 				(XFS_IFORK_Q(ip) ? XFS_XFLAG_HASATTR : 0);
@@ -660,7 +660,7 @@ xfs_ip2xflags(
 
 uint
 xfs_dic2xflags(
-	xfs_dinode_t		*dip)
+	struct xfs_dinode	*dip)
 {
 	return _xfs_dic2xflags(be16_to_cpu(dip->di_flags)) |
 				(XFS_DFORK_Q(dip) ? XFS_XFLAG_HASATTR : 0);
diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
index ca9e119..aef5452 100644
--- a/fs/xfs/xfs_inode.h
+++ b/fs/xfs/xfs_inode.h
@@ -63,7 +63,7 @@ typedef struct xfs_inode {
 	unsigned long		i_flags;	/* see defined flags below */
 	unsigned int		i_delayed_blks;	/* count of delay alloc blks */
 
-	xfs_icdinode_t		i_d;		/* most of ondisk inode */
+	struct xfs_icdinode	i_d;		/* most of ondisk inode */
 
 	/* VFS inode */
 	struct inode		i_vnode;	/* embedded VFS inode */
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index d14b12b..3ad9972 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -135,7 +135,7 @@ xfs_inode_item_size(
 
 	*nvecs += 2;
 	*nbytes += sizeof(struct xfs_inode_log_format) +
-		   xfs_icdinode_size(ip->i_d.di_version);
+		   xfs_log_dinode_size(ip->i_d.di_version);
 
 	xfs_inode_item_data_fork_size(iip, nvecs, nbytes);
 	if (XFS_IFORK_Q(ip))
@@ -322,6 +322,127 @@ xfs_inode_item_format_attr_fork(
 	}
 }
 
+static void
+xfs_icdinode_to_log_dinode(
+	struct xfs_icdinode	*from,
+	struct xfs_log_dinode	*to)
+{
+	to->di_magic = from->di_magic;
+	to->di_mode = from->di_mode;
+	to->di_version = from->di_version;
+	to->di_format = from->di_format;
+	to->di_onlink = from->di_onlink;
+	to->di_uid = from->di_uid;
+	to->di_gid = from->di_gid;
+	to->di_nlink = from->di_nlink;
+	to->di_projid_lo = from->di_projid_lo;
+	to->di_projid_hi = from->di_projid_hi;
+	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
+	to->di_atime.t_sec = from->di_atime.t_sec;
+	to->di_atime.t_nsec = from->di_atime.t_nsec;
+	to->di_mtime.t_sec = from->di_mtime.t_sec;
+	to->di_mtime.t_nsec = from->di_mtime.t_nsec;
+	to->di_ctime.t_sec = from->di_ctime.t_sec;
+	to->di_ctime.t_nsec = from->di_ctime.t_nsec;
+	to->di_size = from->di_size;
+	to->di_nblocks = from->di_nblocks;
+	to->di_extsize = from->di_extsize;
+	to->di_nextents = from->di_nextents;
+	to->di_anextents = from->di_anextents;
+	to->di_forkoff = from->di_forkoff;
+	to->di_aformat = from->di_aformat;
+	to->di_dmevmask = from->di_dmevmask;
+	to->di_dmstate = from->di_dmstate;
+	to->di_flags = from->di_flags;
+	to->di_gen = from->di_gen;
+
+	if (from->di_version == 3) {
+		to->di_changecount = from->di_changecount;
+		to->di_crtime.t_sec = from->di_crtime.t_sec;
+		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
+		to->di_flags2 = from->di_flags2;
+		to->di_ino = from->di_ino;
+		to->di_lsn = from->di_lsn;
+		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
+		uuid_copy(&to->di_uuid, &from->di_uuid);
+		to->di_flushiter = 0;
+	} else {
+		to->di_flushiter = from->di_flushiter;
+	}
+}
+
+/*
+ * Recovery needs to be able to convert a log dinode back to a real dinode
+ * for writeback we do that by converting a log dinode to a icdinode, and
+ * then passing that to the formatting function.
+ */
+void
+xfs_log_dinode_to_icdinode(
+	struct xfs_log_dinode	*from,
+	struct xfs_icdinode	*to)
+{
+	to->di_magic = from->di_magic;
+	to->di_mode = from->di_mode;
+	to->di_version = from->di_version;
+	to->di_format = from->di_format;
+	to->di_onlink = from->di_onlink;
+	to->di_uid = from->di_uid;
+	to->di_gid = from->di_gid;
+	to->di_nlink = from->di_nlink;
+	to->di_projid_lo = from->di_projid_lo;
+	to->di_projid_hi = from->di_projid_hi;
+	memset(to->di_pad, 0, sizeof(to->di_pad));
+	to->di_atime.t_sec = from->di_atime.t_sec;
+	to->di_atime.t_nsec = from->di_atime.t_nsec;
+	to->di_mtime.t_sec = from->di_mtime.t_sec;
+	to->di_mtime.t_nsec = from->di_mtime.t_nsec;
+	to->di_ctime.t_sec = from->di_ctime.t_sec;
+	to->di_ctime.t_nsec = from->di_ctime.t_nsec;
+	to->di_size = from->di_size;
+	to->di_nblocks = from->di_nblocks;
+	to->di_extsize = from->di_extsize;
+	to->di_nextents = from->di_nextents;
+	to->di_anextents = from->di_anextents;
+	to->di_forkoff = from->di_forkoff;
+	to->di_aformat = from->di_aformat;
+	to->di_dmevmask = from->di_dmevmask;
+	to->di_dmstate = from->di_dmstate;
+	to->di_flags = from->di_flags;
+	to->di_gen = from->di_gen;
+
+	if (from->di_version == 3) {
+		to->di_changecount = from->di_changecount;
+		to->di_crtime.t_sec = from->di_crtime.t_sec;
+		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
+		to->di_flags2 = from->di_flags2;
+		to->di_ino = from->di_ino;
+		to->di_lsn = from->di_lsn;
+		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
+		uuid_copy(&to->di_uuid, &from->di_uuid);
+		to->di_flushiter = 0;
+	} else {
+		to->di_flushiter = from->di_flushiter;
+	}
+}
+
+/*
+ * Format the inode core. Current timestamp data is only in the VFS inode
+ * fields, so we need to grab them from there. Hence rather than just copying
+ * the XFS inode core structure, format the fields directly into the iovec.
+ */
+static void
+xfs_inode_item_format_core(
+	struct xfs_inode	*ip,
+	struct xfs_log_vec	*lv,
+	struct xfs_log_iovec	**vecp)
+{
+	struct xfs_log_dinode	*dic;
+
+	dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE);
+	xfs_icdinode_to_log_dinode(&ip->i_d, dic);
+	xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version));
+}
+
 /*
  * This is called to fill in the vector of log iovecs for the given inode
  * log item.  It fills the first item with an inode log format structure,
@@ -351,10 +472,7 @@ xfs_inode_item_format(
 	ilf->ilf_size = 2; /* format + core */
 	xlog_finish_iovec(lv, vecp, sizeof(struct xfs_inode_log_format));
 
-	xlog_copy_iovec(lv, &vecp, XLOG_REG_TYPE_ICORE,
-			&ip->i_d,
-			xfs_icdinode_size(ip->i_d.di_version));
-
+	xfs_inode_item_format_core(ip, lv, &vecp);
 	xfs_inode_item_format_data_fork(iip, ilf, lv, &vecp);
 	if (XFS_IFORK_Q(ip)) {
 		xfs_inode_item_format_attr_fork(iip, ilf, lv, &vecp);
diff --git a/fs/xfs/xfs_inode_item.h b/fs/xfs/xfs_inode_item.h
index 4c7722e..2426118 100644
--- a/fs/xfs/xfs_inode_item.h
+++ b/fs/xfs/xfs_inode_item.h
@@ -49,6 +49,8 @@ extern void xfs_istale_done(struct xfs_buf *, struct xfs_log_item *);
 extern void xfs_iflush_abort(struct xfs_inode *, bool);
 extern int xfs_inode_item_format_convert(xfs_log_iovec_t *,
 					 xfs_inode_log_format_t *);
+extern void xfs_log_dinode_to_icdinode(struct xfs_log_dinode *from,
+				       struct xfs_icdinode *to);
 
 extern struct kmem_zone	*xfs_ili_zone;
 
diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
index da37beb..3120f7b 100644
--- a/fs/xfs/xfs_log_recover.c
+++ b/fs/xfs/xfs_log_recover.c
@@ -2839,7 +2839,8 @@ xlog_recover_inode_pass2(
 	int			error;
 	int			attr_index;
 	uint			fields;
-	xfs_icdinode_t		*dicp;
+	struct xfs_log_dinode	*ldip;
+	struct xfs_icdinode	icic;
 	uint			isize;
 	int			need_free = 0;
 
@@ -2892,8 +2893,8 @@ xlog_recover_inode_pass2(
 		error = -EFSCORRUPTED;
 		goto out_release;
 	}
-	dicp = item->ri_buf[1].i_addr;
-	if (unlikely(dicp->di_magic != XFS_DINODE_MAGIC)) {
+	ldip = item->ri_buf[1].i_addr;
+	if (unlikely(ldip->di_magic != XFS_DINODE_MAGIC)) {
 		xfs_alert(mp,
 			"%s: Bad inode log record, rec ptr 0x%p, ino %Ld",
 			__func__, item, in_f->ilf_ino);
@@ -2929,13 +2930,13 @@ xlog_recover_inode_pass2(
 	 * to skip replay when the on disk inode is newer than the log one
 	 */
 	if (!xfs_sb_version_hascrc(&mp->m_sb) &&
-	    dicp->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
+	    ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
 		/*
 		 * Deal with the wrap case, DI_MAX_FLUSH is less
 		 * than smaller numbers
 		 */
 		if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH &&
-		    dicp->di_flushiter < (DI_MAX_FLUSH >> 1)) {
+		    ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) {
 			/* do nothing */
 		} else {
 			trace_xfs_log_recover_inode_skip(log, in_f);
@@ -2945,13 +2946,13 @@ xlog_recover_inode_pass2(
 	}
 
 	/* Take the opportunity to reset the flush iteration count */
-	dicp->di_flushiter = 0;
+	ldip->di_flushiter = 0;
 
-	if (unlikely(S_ISREG(dicp->di_mode))) {
-		if ((dicp->di_format != XFS_DINODE_FMT_EXTENTS) &&
-		    (dicp->di_format != XFS_DINODE_FMT_BTREE)) {
+	if (unlikely(S_ISREG(ldip->di_mode))) {
+		if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) &&
+		    (ldip->di_format != XFS_DINODE_FMT_BTREE)) {
 			XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(3)",
-					 XFS_ERRLEVEL_LOW, mp, dicp);
+					 XFS_ERRLEVEL_LOW, mp, ldip);
 			xfs_alert(mp,
 		"%s: Bad regular inode log record, rec ptr 0x%p, "
 		"ino ptr = 0x%p, ino bp = 0x%p, ino %Ld",
@@ -2959,12 +2960,12 @@ xlog_recover_inode_pass2(
 			error = -EFSCORRUPTED;
 			goto out_release;
 		}
-	} else if (unlikely(S_ISDIR(dicp->di_mode))) {
-		if ((dicp->di_format != XFS_DINODE_FMT_EXTENTS) &&
-		    (dicp->di_format != XFS_DINODE_FMT_BTREE) &&
-		    (dicp->di_format != XFS_DINODE_FMT_LOCAL)) {
+	} else if (unlikely(S_ISDIR(ldip->di_mode))) {
+		if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) &&
+		    (ldip->di_format != XFS_DINODE_FMT_BTREE) &&
+		    (ldip->di_format != XFS_DINODE_FMT_LOCAL)) {
 			XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(4)",
-					     XFS_ERRLEVEL_LOW, mp, dicp);
+					     XFS_ERRLEVEL_LOW, mp, ldip);
 			xfs_alert(mp,
 		"%s: Bad dir inode log record, rec ptr 0x%p, "
 		"ino ptr = 0x%p, ino bp = 0x%p, ino %Ld",
@@ -2973,32 +2974,32 @@ xlog_recover_inode_pass2(
 			goto out_release;
 		}
 	}
-	if (unlikely(dicp->di_nextents + dicp->di_anextents > dicp->di_nblocks)){
+	if (unlikely(ldip->di_nextents + ldip->di_anextents > ldip->di_nblocks)){
 		XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(5)",
-				     XFS_ERRLEVEL_LOW, mp, dicp);
+				     XFS_ERRLEVEL_LOW, mp, ldip);
 		xfs_alert(mp,
 	"%s: Bad inode log record, rec ptr 0x%p, dino ptr 0x%p, "
 	"dino bp 0x%p, ino %Ld, total extents = %d, nblocks = %Ld",
 			__func__, item, dip, bp, in_f->ilf_ino,
-			dicp->di_nextents + dicp->di_anextents,
-			dicp->di_nblocks);
+			ldip->di_nextents + ldip->di_anextents,
+			ldip->di_nblocks);
 		error = -EFSCORRUPTED;
 		goto out_release;
 	}
-	if (unlikely(dicp->di_forkoff > mp->m_sb.sb_inodesize)) {
+	if (unlikely(ldip->di_forkoff > mp->m_sb.sb_inodesize)) {
 		XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(6)",
-				     XFS_ERRLEVEL_LOW, mp, dicp);
+				     XFS_ERRLEVEL_LOW, mp, ldip);
 		xfs_alert(mp,
 	"%s: Bad inode log record, rec ptr 0x%p, dino ptr 0x%p, "
 	"dino bp 0x%p, ino %Ld, forkoff 0x%x", __func__,
-			item, dip, bp, in_f->ilf_ino, dicp->di_forkoff);
+			item, dip, bp, in_f->ilf_ino, ldip->di_forkoff);
 		error = -EFSCORRUPTED;
 		goto out_release;
 	}
-	isize = xfs_icdinode_size(dicp->di_version);
+	isize = xfs_log_dinode_size(ldip->di_version);
 	if (unlikely(item->ri_buf[1].i_len > isize)) {
 		XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(7)",
-				     XFS_ERRLEVEL_LOW, mp, dicp);
+				     XFS_ERRLEVEL_LOW, mp, ldip);
 		xfs_alert(mp,
 			"%s: Bad inode log record length %d, rec ptr 0x%p",
 			__func__, item->ri_buf[1].i_len, item);
@@ -3007,7 +3008,8 @@ xlog_recover_inode_pass2(
 	}
 
 	/* The core is in in-core format */
-	xfs_dinode_to_disk(dip, dicp);
+	xfs_log_dinode_to_icdinode(ldip, &icic);
+	xfs_dinode_to_disk(dip, &icic);
 
 	/* the rest is in on-disk format */
 	if (item->ri_buf[1].i_len > isize) {
-- 
2.5.0

From BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org  Thu Jan 14 04:12:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id DD2DF7F47
	for ; Thu, 14 Jan 2016 04:12:40 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id CD4658F804B
	for ; Thu, 14 Jan 2016 02:12:37 -0800 (PST)
X-ASG-Debug-ID: 1452766351-04cbb05f7c18a8e0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id SeiDmcoNsnQdAXA0 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 14 Jan 2016 02:12:32 -0800 (PST)
X-Barracuda-Envelope-From: BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from 212095007047.public.telering.at ([212.95.7.47] helo=localhost)
	by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJetK-00078N-Kp; Thu, 14 Jan 2016 10:12:31 +0000
From: Christoph Hellwig 
To: xfs@oss.sgi.com
Cc: darrick.wong@oracle.com
Subject: stop using ioends for direct write completions
Date: Thu, 14 Jan 2016 11:10:36 +0100
X-ASG-Orig-Subj: stop using ioends for direct write completions
Message-Id: <1452766237-2314-1-git-send-email-hch@lst.de>
X-Mailer: git-send-email 1.9.1
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452766351
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26116
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

The patch below rewrites the direct write I/O completion path to avoid
the use of ioends.  Besides simplifying the code this will also allow
us to make COW I/O a type of it's own for the buffer I/O path.

From BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org  Thu Jan 14 04:12:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 862CC7F47
	for ; Thu, 14 Jan 2016 04:12:41 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 5AB548F804B
	for ; Thu, 14 Jan 2016 02:12:41 -0800 (PST)
X-ASG-Debug-ID: 1452766355-04bdf06f6cf5470001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id iId9H0yrQby56gv3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 14 Jan 2016 02:12:36 -0800 (PST)
X-Barracuda-Envelope-From: BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from 212095007047.public.telering.at ([212.95.7.47] helo=localhost)
	by bombadil.infradead.org with esmtpsa (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJetO-00079A-Gp; Thu, 14 Jan 2016 10:12:35 +0000
From: Christoph Hellwig 
To: xfs@oss.sgi.com
Cc: darrick.wong@oracle.com
Subject: [PATCH] xfs: don't use ioends for direct write completions
Date: Thu, 14 Jan 2016 11:10:37 +0100
X-ASG-Orig-Subj: [PATCH] xfs: don't use ioends for direct write completions
Message-Id: <1452766237-2314-2-git-send-email-hch@lst.de>
X-Mailer: git-send-email 1.9.1
In-Reply-To: <1452766237-2314-1-git-send-email-hch@lst.de>
References: <1452766237-2314-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452766356
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26116
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

We only need to communicate two bits of information to the direct I/O
completion handler:

 (1) do we need to convert any unwritten extents in the range
 (2) do we need to check if we need to update the inode size based
     on the range passed to the completion handler

We can use the private data passed to the get_block handler and the
completion handler as a simple bitmask to communicate this information
instead of the current complicated infrastructure reusing the ioends
from the buffer I/O path, and thus avoiding a memory allocation and
a context switch for any non-trivial direct write.  As a nice side
effect we also decouple the direct I/O path implementation from that
of the buffered I/O path.

Signed-off-by: Christoph Hellwig 
---
 fs/xfs/xfs_aops.c  | 210 +++++++++++++++++++----------------------------------
 fs/xfs/xfs_trace.h |   9 +--
 2 files changed, 77 insertions(+), 142 deletions(-)

diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 379c089..ff2c571 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -36,6 +36,10 @@
 #include 
 #include 
 
+/* flags for direct write completions */
+#define XFS_DIO_FLAG_UNWRITTEN	(1 << 0)
+#define XFS_DIO_FLAG_APPEND	(1 << 1)
+
 void
 xfs_count_page_state(
 	struct page		*page,
@@ -1238,27 +1242,8 @@ xfs_vm_releasepage(
 }
 
 /*
- * When we map a DIO buffer, we may need to attach an ioend that describes the
- * type of write IO we are doing. This passes to the completion function the
- * operations it needs to perform. If the mapping is for an overwrite wholly
- * within the EOF then we don't need an ioend and so we don't allocate one.
- * This avoids the unnecessary overhead of allocating and freeing ioends for
- * workloads that don't require transactions on IO completion.
- *
- * If we get multiple mappings in a single IO, we might be mapping different
- * types. But because the direct IO can only have a single private pointer, we
- * need to ensure that:
- *
- * a) i) the ioend spans the entire region of unwritten mappings; or
- *    ii) the ioend spans all the mappings that cross or are beyond EOF; and
- * b) if it contains unwritten extents, it is *permanently* marked as such
- *
- * We could do this by chaining ioends like buffered IO does, but we only
- * actually get one IO completion callback from the direct IO, and that spans
- * the entire IO regardless of how many mappings and IOs are needed to complete
- * the DIO. There is only going to be one reference to the ioend and its life
- * cycle is constrained by the DIO completion code. hence we don't need
- * reference counting here.
+ * When we map a DIO buffer, we may need to pass flags to
+ * xfs_end_io_direct_write to tell it what kind of write IO we are doing.
  *
  * Note that for DIO, an IO to the highest supported file block offset (i.e.
  * 2^63 - 1FSB bytes) will result in the offset + count overflowing a signed 64
@@ -1266,68 +1251,26 @@ xfs_vm_releasepage(
  * extending the file size. We won't know for sure until IO completion is run
  * and the actual max write offset is communicated to the IO completion
  * routine.
- *
- * For DAX page faults, we are preparing to never see unwritten extents here,
- * nor should we ever extend the inode size. Hence we will soon have nothing to
- * do here for this case, ensuring we don't have to provide an IO completion
- * callback to free an ioend that we don't actually need for a fault into the
- * page at offset (2^63 - 1FSB) bytes.
  */
-
 static void
 xfs_map_direct(
 	struct inode		*inode,
 	struct buffer_head	*bh_result,
 	struct xfs_bmbt_irec	*imap,
-	xfs_off_t		offset,
-	bool			dax_fault)
+	xfs_off_t		offset)
 {
-	struct xfs_ioend	*ioend;
+	uintptr_t		*flags = (uintptr_t *)&bh_result->b_private;
 	xfs_off_t		size = bh_result->b_size;
-	int			type;
 
-	if (ISUNWRITTEN(imap))
-		type = XFS_IO_UNWRITTEN;
-	else
-		type = XFS_IO_OVERWRITE;
+	trace_xfs_get_blocks_map_direct(XFS_I(inode), offset, size,
+		ISUNWRITTEN(imap) ? XFS_IO_UNWRITTEN : XFS_IO_OVERWRITE, imap);
 
-	trace_xfs_gbmap_direct(XFS_I(inode), offset, size, type, imap);
-
-	if (dax_fault) {
-		ASSERT(type == XFS_IO_OVERWRITE);
-		trace_xfs_gbmap_direct_none(XFS_I(inode), offset, size, type,
-					    imap);
-		return;
-	}
-
-	if (bh_result->b_private) {
-		ioend = bh_result->b_private;
-		ASSERT(ioend->io_size > 0);
-		ASSERT(offset >= ioend->io_offset);
-		if (offset + size > ioend->io_offset + ioend->io_size)
-			ioend->io_size = offset - ioend->io_offset + size;
-
-		if (type == XFS_IO_UNWRITTEN && type != ioend->io_type)
-			ioend->io_type = XFS_IO_UNWRITTEN;
-
-		trace_xfs_gbmap_direct_update(XFS_I(inode), ioend->io_offset,
-					      ioend->io_size, ioend->io_type,
-					      imap);
-	} else if (type == XFS_IO_UNWRITTEN ||
-		   offset + size > i_size_read(inode) ||
-		   offset + size < 0) {
-		ioend = xfs_alloc_ioend(inode, type);
-		ioend->io_offset = offset;
-		ioend->io_size = size;
-
-		bh_result->b_private = ioend;
+	if (ISUNWRITTEN(imap)) {
+		*flags |= XFS_DIO_FLAG_UNWRITTEN;
+		set_buffer_defer_completion(bh_result);
+	} else if (offset + size > i_size_read(inode) || offset + size < 0) {
+		*flags |= XFS_DIO_FLAG_APPEND;
 		set_buffer_defer_completion(bh_result);
-
-		trace_xfs_gbmap_direct_new(XFS_I(inode), offset, size, type,
-					   imap);
-	} else {
-		trace_xfs_gbmap_direct_none(XFS_I(inode), offset, size, type,
-					    imap);
 	}
 }
 
@@ -1498,9 +1441,12 @@ __xfs_get_blocks(
 		if (ISUNWRITTEN(&imap))
 			set_buffer_unwritten(bh_result);
 		/* direct IO needs special help */
-		if (create && direct)
-			xfs_map_direct(inode, bh_result, &imap, offset,
-				       dax_fault);
+		if (create && direct) {
+			if (dax_fault)
+				ASSERT(!ISUNWRITTEN(&imap));
+			else
+				xfs_map_direct(inode, bh_result, &imap, offset);
+		}
 	}
 
 	/*
@@ -1570,17 +1516,34 @@ xfs_get_blocks_dax_fault(
 	return __xfs_get_blocks(inode, iblock, bh_result, create, true, true);
 }
 
-static void
-__xfs_end_io_direct_write(
-	struct inode		*inode,
-	struct xfs_ioend	*ioend,
+/*
+ * Complete a direct I/O write request.
+ *
+ * xfs_map_direct passes us some flags in the private data to tell us what to
+ * do.  If not flags are set, then the write IO is an overwrite wholly within
+ * the existing allocated file size and so there is nothing for us to do.
+ *
+ * Note that in this case the completion can be called in interrupt context,
+ * whereas if we have flags set we will always be called in task context
+ * (i.e. from a workqueue).
+ */
+STATIC void
+xfs_end_io_direct_write(
+	struct kiocb		*iocb,
 	loff_t			offset,
-	ssize_t			size)
+	ssize_t			size,
+	void			*private)
 {
-	struct xfs_mount	*mp = XFS_I(inode)->i_mount;
+	struct inode		*inode = file_inode(iocb->ki_filp);
+	struct xfs_inode	*ip = XFS_I(inode);
+	struct xfs_mount	*mp = ip->i_mount;
+	uintptr_t		flags = (uintptr_t)private;
+	int			error = 0;
+
+	trace_xfs_end_io_direct_write(ip, offset, size);
 
-	if (XFS_FORCED_SHUTDOWN(mp) || ioend->io_error)
-		goto out_end_io;
+	if (XFS_FORCED_SHUTDOWN(mp))
+		return;
 
 	/*
 	 * dio completion end_io functions are only called on writes if more
@@ -1589,23 +1552,17 @@ __xfs_end_io_direct_write(
 	ASSERT(size > 0);
 
 	/*
-	 * The ioend only maps whole blocks, while the IO may be sector aligned.
-	 * Hence the ioend offset/size may not match the IO offset/size exactly.
-	 * Because we don't map overwrites within EOF into the ioend, the offset
-	 * may not match, but only if the endio spans EOF.  Either way, write
-	 * the IO sizes into the ioend so that completion processing does the
-	 * right thing.
-	 */
-	ASSERT(offset + size <= ioend->io_offset + ioend->io_size);
-	ioend->io_size = size;
-	ioend->io_offset = offset;
-
-	/*
-	 * The ioend tells us whether we are doing unwritten extent conversion
+	 * The flags tell us whether we are doing unwritten extent conversion
 	 * or an append transaction that updates the on-disk file size. These
 	 * cases are the only cases where we should *potentially* be needing
 	 * to update the VFS inode size.
-	 *
+	 */
+	if (flags == 0) {
+		ASSERT(offset + size <= i_size_read(inode));
+		return;
+	}
+
+	/*
 	 * We need to update the in-core inode size here so that we don't end up
 	 * with the on-disk inode size being outside the in-core inode size. We
 	 * have no other method of updating EOF for AIO, so always do it here
@@ -1616,54 +1573,33 @@ __xfs_end_io_direct_write(
 	 * here can result in EOF moving backwards and Bad Things Happen when
 	 * that occurs.
 	 */
-	spin_lock(&XFS_I(inode)->i_flags_lock);
+	spin_lock(&ip->i_flags_lock);
 	if (offset + size > i_size_read(inode))
 		i_size_write(inode, offset + size);
-	spin_unlock(&XFS_I(inode)->i_flags_lock);
+	spin_unlock(&ip->i_flags_lock);
 
-	/*
-	 * If we are doing an append IO that needs to update the EOF on disk,
-	 * do the transaction reserve now so we can use common end io
-	 * processing. Stashing the error (if there is one) in the ioend will
-	 * result in the ioend processing passing on the error if it is
-	 * possible as we can't return it from here.
-	 */
-	if (ioend->io_type == XFS_IO_OVERWRITE)
-		ioend->io_error = xfs_setfilesize_trans_alloc(ioend);
+	if (flags & XFS_DIO_FLAG_UNWRITTEN) {
+		trace_xfs_end_io_direct_write_unwritten(ip, offset, size);
 
-out_end_io:
-	xfs_end_io(&ioend->io_work);
-	return;
-}
+		error = xfs_iomap_write_unwritten(ip, offset, size);
 
-/*
- * Complete a direct I/O write request.
- *
- * The ioend structure is passed from __xfs_get_blocks() to tell us what to do.
- * If no ioend exists (i.e. @private == NULL) then the write IO is an overwrite
- * wholly within the EOF and so there is nothing for us to do. Note that in this
- * case the completion can be called in interrupt context, whereas if we have an
- * ioend we will always be called in task context (i.e. from a workqueue).
- */
-STATIC void
-xfs_end_io_direct_write(
-	struct kiocb		*iocb,
-	loff_t			offset,
-	ssize_t			size,
-	void			*private)
-{
-	struct inode		*inode = file_inode(iocb->ki_filp);
-	struct xfs_ioend	*ioend = private;
+		WARN_ON_ONCE(error);
+	} else if (flags & XFS_DIO_FLAG_APPEND) {
+		struct xfs_trans *tp;
 
-	trace_xfs_gbmap_direct_endio(XFS_I(inode), offset, size,
-				     ioend ? ioend->io_type : 0, NULL);
+		trace_xfs_end_io_direct_write_append(ip, offset, size);
 
-	if (!ioend) {
-		ASSERT(offset + size <= i_size_read(inode));
-		return;
-	}
+		tp = xfs_trans_alloc(mp, XFS_TRANS_FSYNC_TS);
 
-	__xfs_end_io_direct_write(inode, ioend, offset, size);
+		error = xfs_trans_reserve(tp, &M_RES(mp)->tr_fsyncts, 0, 0);
+		if (error) {
+			xfs_trans_cancel(tp);
+			WARN_ON_ONCE(error);
+			return;
+		}
+		error = xfs_setfilesize(ip, tp, offset, size);
+		WARN_ON_ONCE(error);
+	}
 }
 
 static inline ssize_t
diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h
index 391d797..c8d5842 100644
--- a/fs/xfs/xfs_trace.h
+++ b/fs/xfs/xfs_trace.h
@@ -1296,11 +1296,7 @@ DEFINE_IOMAP_EVENT(xfs_map_blocks_found);
 DEFINE_IOMAP_EVENT(xfs_map_blocks_alloc);
 DEFINE_IOMAP_EVENT(xfs_get_blocks_found);
 DEFINE_IOMAP_EVENT(xfs_get_blocks_alloc);
-DEFINE_IOMAP_EVENT(xfs_gbmap_direct);
-DEFINE_IOMAP_EVENT(xfs_gbmap_direct_new);
-DEFINE_IOMAP_EVENT(xfs_gbmap_direct_update);
-DEFINE_IOMAP_EVENT(xfs_gbmap_direct_none);
-DEFINE_IOMAP_EVENT(xfs_gbmap_direct_endio);
+DEFINE_IOMAP_EVENT(xfs_get_blocks_map_direct);
 
 DECLARE_EVENT_CLASS(xfs_simple_io_class,
 	TP_PROTO(struct xfs_inode *ip, xfs_off_t offset, ssize_t count),
@@ -1340,6 +1336,9 @@ DEFINE_SIMPLE_IO_EVENT(xfs_unwritten_convert);
 DEFINE_SIMPLE_IO_EVENT(xfs_get_blocks_notfound);
 DEFINE_SIMPLE_IO_EVENT(xfs_setfilesize);
 DEFINE_SIMPLE_IO_EVENT(xfs_zero_eof);
+DEFINE_SIMPLE_IO_EVENT(xfs_end_io_direct_write);
+DEFINE_SIMPLE_IO_EVENT(xfs_end_io_direct_write_unwritten);
+DEFINE_SIMPLE_IO_EVENT(xfs_end_io_direct_write_append);
 
 DECLARE_EVENT_CLASS(xfs_itrunc_class,
 	TP_PROTO(struct xfs_inode *ip, xfs_fsize_t new_size),
-- 
1.9.1

From BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org  Thu Jan 14 04:17:36 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 14E437F50
	for ; Thu, 14 Jan 2016 04:17:36 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D85C58F8059
	for ; Thu, 14 Jan 2016 02:17:35 -0800 (PST)
X-ASG-Debug-ID: 1452766651-04bdf06f6df5620001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id VdpzQ6dy4LRycVlM (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 14 Jan 2016 02:17:31 -0800 (PST)
X-Barracuda-Envelope-From: BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJey9-00016g-Vh; Thu, 14 Jan 2016 10:17:29 +0000
Date: Thu, 14 Jan 2016 02:17:29 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: Christoph Hellwig , xfs@oss.sgi.com
Subject: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
Message-ID: <20160114101729.GC27644@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
References: 
 <20160112200022.GG568@nyan>
 <20160113075855.GG21939@infradead.org>
 <20160113170938.GA10942@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113170938.GA10942@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452766651
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 2.60
X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC7_SA_HREF_FROM_MISMATCH_TEXT_URIx1_HL, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26116
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	2.50 BSF_SC7_SA_HREF_FROM_MISMATCH_TEXT_URIx1_HL Custom Rule
	                           HREF_FROM_MISMATCH_TEXT_URIx1_HL

On Wed, Jan 13, 2016 at 06:09:38PM +0100, Felix Janda wrote:
> Christoph Hellwig wrote:
> > I think we're having a deeper issue here.  For an LFS build we
> > absolutely should have these values as the kernel dirent64 has them.
> > 
> > Either we manage to not pull in the LFS version of readdir/getdents
> > or musl is doing something inredibly stupid.  Can you investigate if
> > and why we're not using the LFS getdents64 here?
> 
> With musl you always have LFS. However still musl does not define the
> _DIRENT_HAVE_*. You can see the full dirent.h at:
> 
> http://git.musl-libc.org/cgit/musl/tree/include/dirent.h
> 
> I will ask on the musl list whether these constants can be added. So
> you can disregard this patch.

Oh, ok.  That's a little weird, but I guess working around that
with a new autoconf test is fine.

From BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org  Thu Jan 14 04:20:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id F23267F50
	for ; Thu, 14 Jan 2016 04:20:12 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 85D6EAC002
	for ; Thu, 14 Jan 2016 02:20:12 -0800 (PST)
X-ASG-Debug-ID: 1452766810-04cbb05f7d18ac10001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id HtVF5v0tPt32MnvH (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 14 Jan 2016 02:20:10 -0800 (PST)
X-Barracuda-Envelope-From: BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJf0j-0004Gs-Tk; Thu, 14 Jan 2016 10:20:09 +0000
Date: Thu, 14 Jan 2016 02:20:09 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: Christoph Hellwig , xfs@oss.sgi.com
Subject: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
Message-ID: <20160114102009.GD27644@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
References: 
 <20160112195945.GC568@nyan>
 <20160113075501.GC21939@infradead.org>
 <20160113174241.GC10942@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113174241.GC10942@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452766810
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26117
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Wed, Jan 13, 2016 at 06:42:41PM +0100, Felix Janda wrote:
> Is this really performance relevant?

I'm not entirely sure.  There are systems with giant amounts of mounted
file systems, in which case iterating each of them and requiring
multiple syscalls each might not be a good idea.

> Hmm, so we couldn't we use for linux just something like
> 
> int
> platform_check_ismounted(char *name, char *block, struct stat64 *s, int verbose)
> {
> 	int fd = open(block, O_EXCL);
> 	int e = errno;
> 	
> 	if (fd != -1) close(fd);
> 	if (e == EBUSY) {
> 		if (verbose)
> 			fprintf(stderr, _("%s: %s contains a mounted filesystem\n"), progname, name);
> 		return 1;
> 	}
> 
> 	return 0;

Please give me a couple of days to come up with an idea to use O_EXCL
on Linux and work around it on other platforms in a more integrated way.

From BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org  Thu Jan 14 04:20:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 1766F7F50
	for ; Thu, 14 Jan 2016 04:20:40 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 067618F804B
	for ; Thu, 14 Jan 2016 02:20:40 -0800 (PST)
X-ASG-Debug-ID: 1452766838-04cbb05f7f18ac40001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id SZQ8nwhlK7BYKxuA (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 14 Jan 2016 02:20:38 -0800 (PST)
X-Barracuda-Envelope-From: BATV+5eeb91f72ff4ba78a643+4526+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aJf1C-0004nM-0N; Thu, 14 Jan 2016 10:20:38 +0000
Date: Thu, 14 Jan 2016 02:20:37 -0800
From: Christoph Hellwig 
To: Felix Janda 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH v2] cleanup: Kill __[u]int*_t types
Message-ID: <20160114102037.GE27644@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH v2] cleanup: Kill __[u]int*_t types
References: <20160113080132.GM21939@infradead.org>
 <20160113175121.GA11266@nyan>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113175121.GA11266@nyan>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1452766838
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26117
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

Looks good,

Reviewed-by: Christoph Hellwig 

From pieterconradie47@gmail.com  Thu Jan 14 07:49:43 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: *
X-Spam-Status: No, score=1.3 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,FREEMAIL_REPLYTO,T_DKIM_INVALID autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id BD4507F47
	for ; Thu, 14 Jan 2016 07:49:43 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 7D928304059
	for ; Thu, 14 Jan 2016 05:49:39 -0800 (PST)
X-ASG-Debug-ID: 1452779377-04bdf06f6cfb900001-NocioJ
Received: from mail-io0-f194.google.com (mail-io0-f194.google.com [209.85.223.194]) by cuda.sgi.com with ESMTP id EznFfIHE9GT5lLFK (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 14 Jan 2016 05:49:37 -0800 (PST)
X-Barracuda-Envelope-From: pieterconradie47@gmail.com
X-Barracuda-RBL-Trusted-Forwarder: 209.85.223.194
Received: by mail-io0-f194.google.com with SMTP id q21so33086182iod.3
        for ; Thu, 14 Jan 2016 05:49:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:reply-to:in-reply-to:references:date:message-id
         :subject:from:to:content-type;
        bh=QzfoaPRnXVsABSbEFfcm7JiTfiv46MxlrNM0UZMaHQM=;
        b=iSbsypt5oSu/qEzXDaGKMBolOF1AziNNfCvV1RopFd/Btvi/qWIqb5mrziIwsK+Uus
         v/OOe2Eg0lcHLdEXkc6/gVlgVDidiqLHa6QFHOY6P3tvfYLrhwpzQwVI3nALrVXushFa
         ad7AEox+y6wTSCf1RuCo1aEy/eynoj1WWtytqOMVgkr5faS267maxHEsjqnwXVLE3Fjv
         FN+78hU+Sj0kdzoM2Tf4G6L+8S+fbNhySZ8O+z1YTGxdGLyyjz8wd0EAR8C9UvzVa7YK
         NDpExvF5uGIojYLZpwicSAlEtWuzQJ6bKHPwPpYeH/81Sdebju2S/uD6W0PN8jw8ELMm
         5ZtQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:reply-to:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=QzfoaPRnXVsABSbEFfcm7JiTfiv46MxlrNM0UZMaHQM=;
        b=ZbfeeCgxJZxieMX3eVQWEaZnOhPOmYk+3rJ7mI3evx3bNakRvDJjgY8vV1E9z3ZK6F
         hwTsZWZVTLvVg7XOgGSiEJMiT69c13ENXgkmoGlhhzXYJhZ4tytD2OshO4EwHdFL2Pg0
         09gQXvZ0E330XYQKaPO/o+BL7JjW/zuppa6pAcCw1MIQpLznNXzV7vPCIjzVSb+heNTe
         uiB2hN9OZEHT6gcIZPMoCLXq5vCqjzwElcGU0XvkHcqgaUxfm6ZNLrgKR+uPU0dLuFud
         4iTMJRiWW/YnkSR1O/Arux3JLOpWMqFmLkSrje5icU/z5l0NltF0SEDWm9ea0EKd7kHg
         gIlQ==
X-Gm-Message-State: ALoCoQnjMB+HbNZupqWyfrVRrM1XrX6zcmObOZwsAOCHRb9zBV9BpGWextZHAD5ZsN1ihn4kfBvQFlPFJDJlIl7VEzC1NoA4bg==
MIME-Version: 1.0
X-Received: by 10.107.8.135 with SMTP id h7mr4833162ioi.93.1452779377365; Thu,
 14 Jan 2016 05:49:37 -0800 (PST)
Received: by 10.79.96.65 with HTTP; Thu, 14 Jan 2016 05:49:36 -0800 (PST)
Reply-To: sandrajohntg@gmail.com
In-Reply-To: 
References: 
	
	
	
	
	
	
	
Date: Thu, 14 Jan 2016 13:49:36 +0000
Message-ID: 
Subject: From western union office
From: Miss Sandra John 
X-ASG-Orig-Subj: From western union office
To: chrispine722@gmail.com
Content-Type: multipart/mixed; boundary=001a113fb8a4b9e53905294b8e8a
X-Barracuda-Connect: mail-io0-f194.google.com[209.85.223.194]
X-Barracuda-Start-Time: 1452779377
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 2.01
X-Barracuda-Spam-Status: No, SCORE=2.01 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_MV0646, BSF_SC0_MV0646_4, BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26120
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	0.50 BSF_SC0_MV0646         Custom rule MV0646
	1.50 BSF_SC0_MV0646_4       BSF_SC0_MV0646_4
	0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient
	                           Domain

--001a113fb8a4b9e53905294b8e8a
Content-Type: text/plain; charset=UTF-8



--001a113fb8a4b9e53905294b8e8a
Content-Type: image/png; name="SANRA JOHN WESTERN UNION..png"
Content-Disposition: attachment; filename="SANRA JOHN WESTERN UNION..png"
Content-Transfer-Encoding: base64
X-Attachment-Id: file0

iVBORw0KGgoAAAANSUhEUgAACzQAAAV2CAIAAAB5k484AAAAAXNSR0IArs4c6QAAAARnQU1BAACx
jwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAP+lSURBVHhe7N3pgeOqtkDhjqsCqng6mk7mBHNv
2dbAsEeEZLu8zp/3bpcl4GMzCLD853/8hwACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIHCawJ/T7syNEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBP7H
4QyCAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQOFGAwxkn4nJrBBBAAAEE
EEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEECAwxnEAAIIIIAAAggggAACCCCAAAIIIIAA
AggggAACCCCAAAIIIIAAAgggcKIAhzNOxOXWCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAhzOIAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA4EQBDmec
iMutEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABDmcQAwgggAACCCCAAAII
IIAAAggggAACCCCAAAIIIIAAAggggAACCCCAwIkCHM44EZdbI4AAAggggAACCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggggAACHM4gBhBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEE
EEAAgRMFOJxxIi63RgABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEOJxBDCCA
AAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAicKcDjjRFxujQACCCCAAAIIIIAA
AggggAACCCCAAAIIIIAAAggggAACCCCAAAIIcDiDGEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQAABBE4U4HDGibjcGgEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAA
AQQQ4HAGMYAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCJwokDic8e/7z/rf
9z8/S/++5U9p/+7f8cgn/vv7teVe/H8iRTqSgVOvLapGLuW7lq4t2E851n8aL9LxO5xamdwcAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQOC3CYQPZ1SnG77+/mdBPD7bf0j79wtQ3eML
90MNTrkuyOdQEu7Jk8eJjfHjDEO5OnxRX2c/9bOWdbyqjt/hcMm4AQIIIIAAAggggAACCCCAAAII
IIAAAggggAACCCCAAAIIIIDARwlED2c0G+XWPr/2YoJnvrAgeHzhHY8w/IRr7OTJmx0+kaqMF2d8
VOdEYRFAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAIHfIhA7nLFtlH99PX4dxHhvgfZi
gqe+sGA9vqDluzre8G5vmNhqR8t48gdpXiK295Cr3tJyPIqO3+ElfMgEAggggAACCCCAAAIIIIAA
AggggAACCCCAAAIIIIAAAggggMD7CIQOZ2yb+9///NdfvPSLM6xzF8WrGt7reIZ38uQWjUUVvkdw
Rgr1HiUhlwgggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACny4QOZxRbuy7hzPe8sUZ
SxTsxzPe6XSG++KMe+ne7LBDrFCf3nopPwIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IIDAWwj4hzPq35dwfxTipV+cYfway6O23u4FE+E8uxX3WuH6Ztl9LTxygwACCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggggAACCCDwWgLu4Yz2uIL66oztg3+q/76/v+t/WP7X8m6KdRN+f1VFdR/l
DRbJ9yokjlx4L5gofvvkVhD3tMettjsZ+aqWYr/MfI9HfXhGjS6nYNFyjdXXmqtAKs1H1th5kDlH
NtprBTf30Mfxyvr79+ueazU2ktH7Wh0GuUEAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAA
AQQQyAs4hzP6H/rwfrakOYnx9f392Kpu//2+2b4dXKj33pvjHf/aYgXPI6yXZT5ubJwrp0+sExrK
SYN7+bqTA8v97xR1WubhjNjJE+NsRqpcZSa3sxJefRUVrUfCUltKbh4Bov+ojirdHJEwfpZnWmVV
Rn2T9M7/5BsxVyCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIvLaAfThj2ynft7i1
0wuHX5yh7413ryCInUfY4DMf1w5yqCcYHqcNxOMTzjXNqxX2d1IEXgCxB1Xo5IlQj8sdkuUqXpyR
qK/+3SHqcQ7trnfg5MGgJZEyfPQXZ0ysrP7tImUfYBwPee2ugtwhgAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCIwKmIczxFMN5hf/ve3z9pzF+pKB7f0a5SkH7VBF6DxCAZJ5VYF47/LE
QHUMY9/R73/Dotjtr49uFHcTSru+amS74X//Pd4xovznnjwp8tFmMl+ukfrKp6KdwpArsjxWUYCK
zFooTK0s66dT3J9VGW3IXIcAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggg8LoCxuGM
/idN7sUwfvhD/90J5XUB1XsS9PMN5s96uLZmhturpcMZ5vkHZbvfTlTgqCjiBbZ+i6N+PYVxfERK
TizXSH2N6CnhIv2zEqX3eu2Tlm88u7LWIyzLT/cUMZY5J+QGNh9AAAEEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQAABBBBAAAEEEHgTAf1whvpTGMbvMmRfnKG8RGKxm/SWgdSGeL9P7+VC/LudpnQYwHoJ
hxFLocMZ/bmMnzuOlCtfXyOpaFmT7mW/OKQNVTk30ytrTaY99JI6JvQmHQjZRAABBBBAAAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBDwBbTDGcYLCYz9du3chvfvf8TjA6lTFWpRczvi3Wa/cRTl
kaR3/KDNmPIrHPYpA7V05d3qN2W0/6shHiqXemDnnj+hvoZS2W7UHG4QbpYMETc7kypLfnVGNlL8
1ssnEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQTeQkA5nGHtwuunHYZfnGG+2uFP
/Ec+JPLU/n33qybDhx9uWfFea1EUO3eEZCtn4LKiBEV6Q+XqdCpvIS9DqSRenJGqW+cczczKEl+d
kT4Z8hYdCJlEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEPAF5MMZ6TckPBLyXpDR
nrKw99aTO+9KWQPHF/Yru/eFeDv2xesp6gMmoWMJhcfgzn3MqH8Nyli5svU1lkr8xRnZV1F48Wm+
eiRVWUtKxTWpOPTbLZ9AAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEHgjAelwRnhP
vTlsMfriDPG9GftrJ657cUZ/iCF0xuKxpx97LcXPp6Rt+uwxgyXCojv+XcGGymVn8tCLM0o9JRnh
n5MnWuQbGxTjldX9rknsFM0bdRxkFQEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAg
LiAczohv3DenKrwXE2gvzpBPXwyeV2iKHj2+cLusP5uhvgvEAq7Otghlk/bpk8cM1uS3qvIOsLQf
HErPvuj44YmlVEoy/T9n6vZ2b+nGJ1VW+7smQ97xVswnEUAAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQReWqA7nBF+bcbtbRHVmYC5L84YOhfRUyfeWCD8lkt2+/+W/naN/EKQrVwl3uBB
FC+pnaM5nDFSrq1gTrn2mBhKRXtjikCUVJM+7glKp19CyVavzghd8dLdBJlDAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQQQQAABBBBAAAEEEDgi0B7O8HarH2mJLwJ47xdn7O8LKU+cDLzxwLlEOAKiePrV
Gn5xRlerA+VyTstIBxBGUtmS8V/L4px5aFOXcnNaZS15uxcicUDIr3M+gQACCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggggAACCCDwfgLN4Qzx6EBfKuslBO2LFbIv1FhSG9rY73IaPb6wH82os++9
/KHfd7evKN5LUpwBGXyzQuwgzY9Jf/AkX678izOKl4gov7oinVpQciYSWTHS3cd8cYaYwUOVteTt
+59H/X59BjlGAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEgKVIcz9t1oZTd9vbm0
4fzSL85QforjXp5iD/5P9znrtIrMZbwnYT8k8adKafAgSvDkiXjwZLxccmjI50vyqWhnQEQi/XSK
kLB4h/Mqa311xvf31+0HgPrISjZUPo4AAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggg
8L4C5eGM4GszboWNvzFCe3FA8jcpHsLhd0UsFeK8taA8lqFtoBefKU9uaOcsqrMe0g+k3Hfqf/67
5MUZdQGrYxXZcg3VV3nwJaan/HiKkrpYiLLQe5HlO8gvxyheNTJcWU1s2UdaOLrxvj0oOUcAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQCAvvhjPBrM2537U89aAcdtH+33xahbMYH3xWx
lbs8QrGeitD+r/pyDecm/bZ7KNXisqMvzvBK1h0wePjkyjVUX+lUvBdn6G82kRCkwzHtHU6rrOrG
zu+6cDgj0FXxEQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQOB9BfbDGX8fP79QvdNB
L1f/6oxml3vbjpb/3XmnhXwUYPjFGd7xBetXT34M2gMm2+2065Qd/9vH+5I576TQ6kDNlFBYJZ+J
cg3V1yPriVT2EyPNaQbjaIh2+/AdTqos5dUkZXWmzkO9bx9DzhFAAAEEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQAABBBBAAIGPF9gPZyx7+s45hRVM3Kwvtrmr2wj/bp9IuOzFGcHS3krd7OG7V9af344K
9EU798UZyXxq73AYqq+idQX1nJ8vCb7dRHhPRfBXWZrjSYcqy3/NS+J3hD6+owIAAQQQQAABBBBA
AAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQeGeB/XDGO5eCvCPwagLBF6LcP6b86smrFYn8IIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgiMCXA4Y8yNqxAwBfrf/ZE/fvscZzMIJgQQQAAB
BBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQOB3C3A443fXL6V7ioD4qz9CTnhvxlOqh0QRQAAB
BBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQACBiwU4nHExOMn9foH1rRl/vv7+Z5WWoxm/PxYo
IQIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIHAT4HAGcYDAFIHtbRl/lv/soxm3jzuH
N6Zki5sggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCDxdgMMZT68CMvA7BLb3ZTzO
Znz/+x3FohQIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAocFOJxxmJAbIHATKA5n
8EYMQgIBBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAoBDgcAbhgAACCCCAAAIIIIAA
AggggAACCCCAAAIIIIAAAggggAACCCCAAAIInCjA4YwTcbk1AggggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCCDA4QxiAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAA
AQQQOFGAwxkn4nJrBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEECAwxnEAAII
IIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgggcKIAhzNOxOXWCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAhzOIAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQAABBBBA4EQBDmeciMutEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQ
QAABDmcQAwgggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAwIkCHM44EZdbI4AA
AggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACHM4gBhBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQAABBBBAAAEEEEAAgRMFOJxxIi63RgABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAA
AQQQQAABBBBAAAEEOJxBDCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAicK
cDjjRFxujQACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIcDiDGEAAAQQQQAAB
BBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBE4U4HDGibjcGgEEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQAABBBBAAAEEEEAAAQQQ4HAGMYAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAA
AggggAACCJwowOGME3G5NQIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgggwOEM
YgABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEDhRgMMZJ+JyawQQQAABBBBA
AAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAgMMZxAACCCCAAAIIIIAAAggggAACCCCAAAII
IIAAAggggAACCCCAAAIIIHCiAIczTsTl1ggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggg
gAACCCCAAIcziAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQOBEAQ5nnIjL
rRFAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQ5nEAMIIIAAAggggAACCCCA
AAIIIIAAAggggAACCCCAAAIIIIAAAggggMCJAhzOOBGXWyOAAAIIIIAAAggggAACCCCAAAIIIIAA
AggggAACCCCAAAIIIIAAAhzOIAYQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
AIETBTiccSIut0YAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBDicQQwggAAC
CCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAInCnA440Rcbo0AAggggAACCCCAAAII
IIAAAggggAACCCCAAAIIIIAAAggggAACCHA4gxhAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQROFOBwxom43BoBBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEE
EOBwBjGAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgicKMDhjBNxuTUCCCCA
AAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIMDhDGIAAQQQQAABBBBAAAEEEEAAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBA4UYDDGSficmsEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQIDDGcQAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCBwogCH
M07E5dYIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggACHM4gBBBBAAAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDgRAEOZ5yIy60RQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQQQQAABBBBAAAEOZxADCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCA
AAIIIIDAiQIczjgRl1sjgAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIcziAG
EEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQACBEwU4nHEiLrdGAAEEEEAAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQ4nEEMIIAAAggggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACJwpwOONEXG6NAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IIAAAghwOIMYQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEThTgcMaJuNwa
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBDgcAYxgAACCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIInCjA4YwTcbk1AggggAACCCCAAAIIIIAAAggggAACCCCA
AAIIIIAAAggggAACCCDA4QxiAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQ
OFGAwxkn4nJrBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEECAwxnEAAIIIIAA
AggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgggcKKAejjj3/cf+7/vfydma731mospibVF
mnLTCxBmJfHf3697lX79/S96y3/fLdLUGnGy0acezfdTP3cl0VML+h6Jv2B1DLTE97Aml6MChMSo
3KHrkkOMMIUQu5dL+pzfMp9ZQ9+ZcD7+XM5HUhc28x73Wnd26N6hLk816wpf6+Zij/4iHOyrtg8m
bq42sj3V6LRSu0LtAXcsN8Prveu8BPpW4XHHKE+4+ooQWG8XvlYq7JbNGPaeVuzzZS0L+QzcJFy6
ti1rATZ8w/PCqc9quOn973+Hg7lLvQ9dp55SpLHHxUvGu0MD/eyLA33K7CSP3O+5D/JHcp661ojD
cwVmh0NyXppCesKHz8U/v0DJ/m3epOj8oh1J4ZdF6REKrkUAAQQQQAABBBBA4KiAdjgjuIDjrpYe
y9/ER970EtaxnL/i1bknzId9u843sUZMIjn1V1Tt8nQV0VtgPD+Tr1gduZb4fENycLoAIXE6cZNA
eogRpxBi93JBn/OL5jPuOWBhi/tel5kL27lM7FprmzN2hy3v1a1S1wb2xMtNX+/47Ta1D93XaZXl
c0LoYUC/QO0B41vf283rrJh9q/OkIxYqVX1LBKw3Sl3bVVFib784B5Cr6RGRNUwypQvlaviG54VT
2yQqL69Qh4O5SN22UZtjhtTrTB65uWC8u3p24KX3PvO15z7Ie45T/67E4QUCE8MhPS+dSjj9Zhfg
T8+z1sN7nfty3cRJ0elFG03gl0XpKAPXIYAAAggggAACCCAwTUA7nBFewAk+r4xleNojr7TeGFpL
Hsv3S16VWkHT5KfViEl0TSon1dJbZ/4kkyfe9gWrI9USn0hH0pcJEBKXUT8SSncLyhRCvE/65tnC
/6b5TPAc8OEXZ4y+dEOZJ6ayfeyVH/X7QsRYaebr1tx2++iMCXCZbuRRQP+82gMm9r7Xu4dfnBGq
xM4pdFXzHpj0izPW65vUw6czikwmKjpWNLWmY5c3x1Wszm/4hueFU5vdRNPLnKHaRigZO/SALl6a
Im06LqWqTh/vsuPj6Z9XjoGdnm46gec+yKeze+yC1GRsZtDOm77PzNUxzClX/47wS1bKvEnRlDo4
4SZJkBNywC0RQAABBBBAAAEEEPhtAsrhDHftIf4FoANi0x55t+JE1o8PZPilL808UGny02rElLom
lZMq660zf5LJE2/7itWRaYlPpCPp6wQIieusbymluwVlCiHeJ33zbNl/1Xxm+LCAvBUfs/SurfY+
zbcnjEwpo0Xec2Hvr+/x8OX+ct3U2Kk3ev1DAAar2gPW29AWt/LUot65zH17Xz8A6kDzAmr99PGq
D91h6GhGs+HfkNR17RwZ8EMh106rk1WRS88Lpzr1TNP7ufJwMN9Tt+/i7A1GIzWi/PjM6eNdPCtX
fXI+4jk5f+6D/DllUu+amoxNDdpp0/epubqYX0jud4RfslImToqeX4NiDpIgL1oKsoUAAggggAAC
CCCAwEsJKIczImsPoUXC1yhspDivkdPzcpF6oNIWG6YtQpjlvCaVk6jfOvMnmXDbUiDVEqH7BAFC
4uJaTvfSmSlE+ubZwmcyk7331Z8fPiygbMWH8h+61tzgDt1ByUuiyKFkipm4H3pTp+3xveYbhbW9
r/aA2x/8cydys3Dv3P1231pte37dEziheioJ3DvuWu1Bhw098gKL8CmJqnLUq4pPCaknAjvUTHcD
n6u+oVvpX6PhVKeTanrFMYYDqZcVpbLsDVP7Ycr0cZdYjfGplxLwx4OXyu4JmbliKWPe9P2X1dfv
KE6yFPMmRSc0hym3TIJMSZObIIAAAggggAACCCDwywXkwxmxpca32SKIFeeX13Tmgeq5X3iYt9Tx
hCp968w/wesDk8y0xA/k+cQiExLX1nq6l85MIdI3z5Y9k5nsvS///PBhgSPTz+C1+z5ot1sdvIPI
mSlyIJjK27kdydT98y1z39/3N3bYJwGWvH1/f99/16L+sJrxHfrv4zr/SELszpFWFNaKhsOUqndy
ZR4qUtq3feiiumjfe1IbxbSN/wyXmEmj3Q6Gk2zxk47b9H6uPBzM5QGn6Nt0/v5X5jkaqZcPBCQ4
XyAwesxP9KXueMlSRqTph1R+WX39juIkSzFxUhSKmes/lAS5PoOkiAACCCCAAAIIIIDAGwrIhzNi
a2JvM0d/m4yeGEApgyu+baKXddpSx4mc6q3fOvPPAPu4NFMt8eN0PrLAhMTF1Z7upTM1lL55tvCZ
zGTvffXnw9vfbcYie+taYeKJajPha1L/yb9b13VZ3I/HpvbBKNg2epfDGZHfHPn++/fx0yvV3rLq
Wf5hSU5LRd52du9sv5Qh1pij4RAPPLPq96MU/Qb9yNGM4ncyIm/acBtF9j0Xx9tpcwe30m/RNxRO
ZULJple8OmY4deNkTKyDjEZqsAfgYy8tEOu9XroIBzN3xVLGvDb1y+rrdxQnWYp5k6KDoX/a5UmQ
0/LBjRFAAAEEEEAAAQQQ+E0C4uGM4BKi8x2cYpXQ/MKbtzJXrfStN93XEPflqv6re00W7mvC0nfv
2s9pi4tt6vVSWbswXt61WfOsErSXMvOMpch279gDVYW5cm3r6N3Kf503b123u7n5O99d6lKcWKUy
/qZvYeS162ydTRTYrTH2c2aVbq1pdak4n1DbBINfC1Qa9FopanXobW5mbBxoiX20u83ncUmuhwyP
ZYOpiwJyk+mDSv2WtN0PF7ePdOx6hatRtKWQj/PDIaGX3fhS+U4SqUdXpFmNfu7Alwp4e4ATWoM5
hRChPL18zGjhVs9npjaKsidZ+8o585nGuAmlcH9UfAm9+mJ46AaJEwrKJNeZ+5q5SKS+f9deGw/b
m3nzvCMZb0tV1N3jtsZMdvns199/y9mM+rNqo6nyaxZOiST1zrHI81rzwySqOq3qtRsVbTR+QmK/
yJvCLxHgUE97cUasioxeuzM4HE5FYtmm14TJQDDnKkoM3WikhvrRasJ54Dk9mFg7FqeeEdwfL4ot
KfhFVtqdVN2pqYva+y7TgEoj+SCvDcragknTDyTopKqe8xTfhHtSoI6tYD/YMMhDWggnPC+Nzh7t
2eChwNv7sW2NJreS442pbTuPLsc1GfMqMfIoVDxQB8fTiZMidd5/DogfWuEoLU57ml2IEaXrGd7s
i9KCYwkfQwABBBBAAAEEEEDg1QTEwxmxJURjmUeZxId2jzYh8ZF9+cfHjF3eOCmeyZR8lPN9NavS
CmOZen3hLdEyw0LW1nxJuZafQPKMosgtYe95+KEug27LWlWNyJmTH4iV+zZfoHRSl5qOUawtf72t
HLl5bSFHJxPdUqyagJGFutwzS3e/cx//j7zkE1KjQ2wVqfv3fchVegdbYlWvseazXJLrIQPj0XDq
Uk0JVap3qurJG7sfduKw69jfKyQGuMzOXfiJAa+LafvP5w58mYDPDzHmFMKaohjzh/3coXpcVG6X
kcyMdc5Wbsv5Q5H1/HxGKpQ8Fvvd0laTwTXz8o6Za+X94cwd2rJkrvU+u8dzWxsKi1wcn1v8RFF3
y/+r7ofsjVRsM+pvQjT5lS+uO/ym5PolxgwxCRJV9aozEaRi5ovuLdEsMrlacijPu0fbsqo9fEOt
0o+H05bXdNOrX5xRzJSlRiOWfKCijnQ+wUZgDYLB53Q/pUnPCN5Dvb+kUD3jiOch7Efgql1mpi5i
39DOJKRzlMVn1j8b3aj6eDWJTqrqOU/xdaHOWsqw5jB1zWcqNzQvVatG6uvtR6RM3gYeRTP4Neg1
4TfySBs9cThxUmQup0xtj7HQCkXpiK34zFLFaN/shucG/mDDJxBAAAEEEEAAAQQQeIaAdDgjtAak
r2zqz1fSPpBaaPGJff3Hn8dg/SFge1bVPrI9RDtZbR9699TbOzfnH+T73m6n5rpb0skz6iLf/6xV
7b0KlDSXvBU1omdO333tt6SaQ/V26mKgqMs61jK1ZJHX9vJzCtEt0aINCHkQ9wkml+77nxT/9lKC
1vatZ/n+MFeyIEJ4XKJ3vCUWNeuUuflGYK6H9IecsdS//i6/6O7vQ+91lL5kOA69r7iJC+FlD52M
w6LV9iDRzvmRqQGu+3WperQbSdt/uuPCqQNfKuDTQ4w5hTCnKNoZIG0Y9L7nZ8TQ/VJ7chIIAW22
8/PvM+YzclcjDld+r1TEc8Ctvd9WnMC18mQ4c4fx1Hd2ZatdmIQ7oDOXtMuklv/fORJSTNvlLri7
vsmvUTrlT+oMse0TA7GgxmVUNR42btXvTxK7mX+RVIKhJijNoYduZLX14RuqlX44nLbs5pte936V
bDDLXVGksxSe8I6Ee5WkOQjGntO9Ikx9RtAf6iNLCs0crHhpk9XRFH1N6Z6aumxIEY3wg3xN70wS
J9CJVa3SFUVVRwblVGJYIPWcnsh+qnL9eWnyicOeDabyVpQ5+hARxn9K+EVLUT9wBQ9nTJwUuQ9t
k9pjOLT8KM09ZtrPLLmHUG8M4e8IIIAAAggggAACCLy6gHQ4w11CLCbp2vPBbQtC/F5J+O0Z4oLn
8o9fy29c12ko+dYe/atFDmXnrS5ek/p+COS/2yLNtl75fX/Fc1H8rShrtvfUlAW3Mm9BRvmSNctf
j9/5Dn2fTgPbirHcrKpgNWiKYKkXBIsMSytWoaxq37hU61ZcJhvQVtr1JURrnd7fHtP8J9Td9NKt
ra+J/2ptMxi05eN+cYkcGvmCSH3I+XrzWmIRrW1/WmoLnVeqh9QHqWzjbZZv9jas1l27nJO4JBKH
ch/VdITvFBIDwvkosjYa+r89d+AbmBI4+yihXvX2IWuKopyZOj4v0l6CZU9O6o2nzGznz/eE+Yza
w4iEgUnzoY3KTKJiQuenXjdzZSdVnHOZhSu3DgLK9keqlJb/Iea07F+3zuJ2oGj7T/7X7l0D4uxt
uYlSbu3Oj6ua3jQ2P1YnXe7+TSjwQlX/k4c2CPfrYlPnpRhjISGxhkqXibrhG54XTmv2803veDAP
c+zmhzouueqsQXDKLLRsD6lnhNDcb3tLWGJJQQwvu6MRpxwDU5eqx/I03Af56gny8HpIaDVGDCLF
LvUUL8ahK5BaylD6LnNIyzQBdV46/OSrPCKNBF41WB5ayZH7savCr+hOQqVI9rrzJkXXgORDy3h6
GrSVozT3EJqZV/BZBBBAAAEEEEAAAQReUUA4nFE9FKivO5C3+sU1q3ZBK7Z2KD3yVlnrb5NZ+JX3
f7u1tzIRa62g+UmQJm/K+lK5PCzvZktr3eYqhHD+wkjcXm5rEh/Ct5d+rSXe2Fe71vvXn14XHx5L
L9XfhEe+eUF7DZFc5mqzoSjySaXrqyef0M4VuVv+/uLRnbP19kKpX/bqglIdGdLNZygCteQPpd6V
Xq7tMsMDl/T7YQc6dqHLEXqofBxODIkBLmlXqKxxoxM2XoMkH6JpdzevHPjEs4cD+yhCa1A2D+I3
z8eMPlmUMxOenBhBXtWe1ZNkq9Xv38x55vJH5UhJ4FJlFT4yxdgYyrZQlt9PXt0D8C+1DtQqo6fZ
aSd3Gswnljqh5X9Jc/sl0TuDvOKursML+VWKoBTbWuF/lE6oy0hcSD2o92ATf7Jq3inn7AreBkFj
jLGfO8dCQuiEUqWLEA/f8LxwWihHmp54li8TzGMVVVf+kY7LCULpgL3yhQBlOBUTSD0jHJj7JbIq
hpfT0UhFHpirpzTU34pyMjNjhlB0rIGmvhZrxlO8uJAyYynDe0qS3wWViCu3vsTDf85yUHeovxiE
M3nbBxjr/KV4Hqq94Jnhl36kVStFDodpk6JrurJ99hNeadRB0rb2M4vx45ozhsJXXI0nTwgggAAC
CCCAAAIfLTB6OENch/RWQr2/lzVhroBIT5y5hV9xzb1IX/r7vrJlLEFLG6/Gwqn0QOMxCX/fsibl
LLluqyVvPy7KV9lPUSOFb1qrmOy6G/D3b3c4o//8gHZkr0l42D2dSPCcWToz/gcSMncs2zWAgfvL
O0HqjebozWyJ8lfzt+izmk+mh9TCOd947R5SLM7sS0Y69jcKiQGugShSfzpJCrliw6ft9M4f+LQX
yDS7aNWw6HUlXXtQLjCnKNKpTm3zNpUh5cOhzll7M4A92+n2BJLVakywM5uF6u6fe8ZBOSrr7aXf
8+10sm7a3WnZTJGtNzmoA42x9SkVZvjxpxke1klX9UaMAvBRfXLmtCyL+ZWHJWWwiu0DS3USio46
RLxLZlX9VmX7DOr78bY+edi3qngwJATXTOkCG7buS92rphfZkpX3F1Ph9JAcaXrKEclE6rFQNptz
8QiX77jkO5uD4PFZ6NAzQmaYG3iqFSvCrp1JSikN7RVb4r8fmTYfr+XjT/G5Z72BStfblVzzA0lk
ppnCUpV8etReEJKOe5y9TCHd/6rwyz/Spubm3QG84UnRNSBe4ayqUr96Is+C7CdHczlZfaoMzSKG
57dciAACCCCAAAIIIIDAtQLC4Yz4Ipuy6K3Omb1ngbLo5gqIvPpiL3SJy3f6i4iFvJpLI/vDeOQt
AFtJjWQSjPaijX08vos3bZlJXZKsFizDD0xlmBUXZZcgl88XobiQ/vzL/v+thezv7qY3ErR2fB4l
0r5SMCeW/FUoqdEMMNoLJU02Bu7v7AS16wFz9Ka2RGsocJpPpoccGHG81JUQt7Z05lzi+P8UVWrP
fSdyN3nBkPDK57aTurKVetyaTtNIzHVVoVswcjvH1tgoq0alqhxJI60fkTeajRBPDOhGk1RybwWG
FzRSpM+bzxhlyWwWiudrAluM3daRtEycG/Qy2W5e3FX/vIyRfWeSYFSPMWFJDbnOsNAy9pOtsgUu
tSfnTa0Re0Yv7kbVbpm6lp57QlM1v3k9KKNhE0pUuWXi2voO3sGSOhoE12jprJfBVIkM3/C8cGqq
scM2nxUOBnPmOURruhnStuNS7ikOR7FHxUDIpTosvx1aD/VibqT05YqwJxVzlFIa6rRFmrdcRKcO
Kf0EPPcUrxRWq5RYfMb6UqW3iSURmJe6s1XjgUZpwwN5O/AoGnhxxrPD71Y65VHI5ZfHrKXQo5Oi
a0DcslmhFWsdDx3HVluF7XsF5ancmaryZwQQQAABBBBAAAEE3kGgP5wRWM8s5trF06W0uCmvPgdW
ZcQVkC1r5kFr92cslB+L95YfbRjjccq6sH8+GmD0Fm1S63rah2P4ds0omxH9yYr4j393gMXzcPdo
PEVbb9bXEO2r/PXzsfCoPRBLgdIJj+UDCXlBW2Vk4P7am3SUtwLM0fMKlWqJq4C3pC40H7n9BLr2
vvpHUtcWbnpkN0u5Szz/n+JJKa7/dkaD8rKUCokBroEoKpT8byQ/c+ArajM+JUiBG6uB5hTlrHmR
9nXYyBzDmHb1l0+r1siyfmBCWN/Ga1TTEhWH9JNSt/dM5PFQkNNDwZ6fJB9XWgSxD23/Ud4O0Dy1
kvSf10qWr6l2puGHZljVyEyi6pXHlJG3ZoxvdQhFyVM7ATd8w/PC6ZZjq6qsXvhoMKeHLYF3mFSt
KnMQPDwLTeU38OG+DmKPbP6vNdi1I2YtnXSggGU9aVkS/j1w5yl0ahh148KS3K33Lf7fx/XSIBKd
jN1vEJP3e/4qN/XHY0lUcSXXVzse6ec5pcdA8UFsIG8DDxEvHH4bwPT1qEmTomva46HQ0lrHgK36
DTlxSilPIZOTVz6OAAIIIIAAAggggMDrCfSHMwIPBuXz7Taz9mblxYNQ4LFXnILbWZP/aj3J6+/N
kPbwzKcCa1nOyHb/BDvAaKWtrmbE10mWT8bwta9JmF9xPfDijG6rfcnm45bV/5C+Dj6gbbTha4j2
cpXtSFoLOad0woLPQEKpteaB+yuLeEVUnKA3tyUWRcg2H7mHjXXte4CHVm8yjbfvQt2lltQlrn85
cpVxvKTy8iExwDUQRULXqcGZP0V99sBXFC0R8K5h28MrF5hTlD24xvqu7PBsFGqoUUybz1hT3kjO
xOuHL1TDWMnmXnlFb3Fi6luPa38vcftY+Ove1TzQmvdGH1F6BL0P3XIpD/qqp9p/dH9Q3EZrqhr4
vKeVYJ2ZPWXZSWe+klq9jiN34VLRqXnYGhxCkUep1XgbvuF54VSMi92vFT3KYYx4R4N5qKJq3WFS
vVMwB8HEoGx2814LvF8cKZzaaSlfBpDqTK6HgcWB2NPiXvZsAGhZ6v/9Kjo9jNYcrF1Y7im+aHj+
YeLwpDHWnSp22crVijA2e7SnuAN5G3iIeM3wq0Z27ak680hbxfSkSdE17XF6aM23rbqBcC8fncLy
OQQQQAABBBBAAAEEXkqgO5wReTCo1kLWld7Q1PzxOOQvtogLEfbqhJxx8ZrI9kx3oZ16ah9qj4A+
JwOMbnGEZXM1CLVijuAbZfkJAanCsgtQ3esR6se56n9J9x7Q9leY5PCW4zNPVDa+fguwSvuy0g0k
5AZt6Txwf/XFGdKLR8xeIvDq/FXdLVSmJVYvA20yMdR8ol37Qz4fme0Ka9dSeuYDl0itzPUvFvGr
6/t8vGRIDHAN1GNZ/buSMsI9c+Az2ri+kpceYpQLolOUob4rOzxbhRpqFNOq1ZrtbjSxfZB+2uRP
JPvk7a2J+vP7+rVwkiswjc2nLqeo5coZmhrW7d4jbG1Resbl9sXNu9FOttdqxBivmnjXSpapa62A
3uNKWNXLTKTqhdbk3dZ53Ex3h+XUQNjFmnHwp55/zOocjodT/JdpeoXDqY9UVFP3B0NFiKToIFhc
mpiFRkau7c6RD3fZtVHFrIrp2PcR3dNJRwooOLedvZBu5M4z6MKPztVze/0VC5ktFYdp+cgcRn5x
RmIpQM7V0OwxVsBE3gYeIrQ8PDP8xh9pgzOmSZOia9rj3NA6xTY4gXSmOPwZAQQQQAABBBBAAIG3
EOgOZ4QXzNsPRp4owiTizewU7Ifb0DcqzCUkM3Vrxcm4cPBJVVlzi/+YgFEPWm5tfGn5aV9s7n72
vVp9LR98B4KofIBbk1wlyv2C9m9VHrILwCLgNUS3pLuyyEkPaOqRYd0sn1BiibbYDUhVk5qp8/Q8
B+/vpf785pNZ3j+QurqQJdR5t/rSxJ9+ibgJFPCVA+89QmJdI00Ij9TjvQ5aEYX2qQOf/rPqjzAS
Az4QI3UQKhdEpyjp9MxJUiYzy40CGehqcV61GqXZEgkufW+3yg0edQ4y1+6Npxx4MndoSx8o8jav
1lSMZfD+CF+ZgfDUPjBNlxCW+zeTr6KjtvverkeXJpVr1uqmrZTMTs+NuphX7FPeu/Sr2Wjm2M+R
aBycAkslDgR2IKqkBzC3mvwpw+MTh8PJPLBqN70JqWcruu/2s3cIVFh0EBQq1q/XXH5HhjnxFzL2
vErzB3kSbSYul8POb59MTkOfGgnpXkRnxVP5INBOPZf/fQ+YbqJet+3QMo9dXKub6Eug1Eq2covu
yf1pRb9ZxgooP0xbMa/9dJZEpuXheeE38igUaBnSmNUfWkxNigKp9nGXDrlAKn2oKRedZNu29qEs
++2FTyCAAAIIIIAAAggg8AoC7eGM+CJbs0SXXTswCy+ugMjLItt9xIm7mKtAVnsGO3Xjid66cMoC
mlccx62uCO3D9k2kLHiRZC3x+gt3e66LZZ1yFef+geIfxFUdjy7VQq8hqhajFicl5ctKN5DQUFCm
vpVppFCuBKrfvj9QKC2fmUKf0HwSSxsnpC7tT7ggfb9qXRKoMq1g7xAS8mGDso/quEbqcblhJaKq
P3Pg816cEf9xcqufV4JKFBH+MRCTiVEmk5nltoEM9FEyrVqtskkzgJBFbv+kvmXi2i1/9XQkcYeu
NJEib7UhDiP7XwNvdKr2X7y+IES/fkhCqKdf3WRM26DWPO3ArZqaMrLJd3YHnbWMgZbz89FoOByv
erGCosmrtRsrZTffbl4oEindUICl5ny3FM4Lp/Gm5/3oRiSYw4FWPvn8KfkOh0pXgdFBUJqj+Aet
ww21eNSrCtzkt+8AnRSkbsVcZhCfW+V+N510SkOfGo1NUabQBeYCN8Bu4Cj+Yc2H8k4o8W14gVeH
FDlLdodDQ4wUQ3L1JnPzKIcdKenA238wSFmWyazkPC/8vOlPphRKJE+aFAUqfUJ7DKTSF1MJntNs
Yw+hqYkFH0YAAQQQQAABBBBA4EUF2sMZ4UW2bj6e2PpzLcR72QnIjw3ys7ObVeFhw7zGetBJ7nO4
eevx7EvCNXq/sXavWBqJl2Bs2UpcI8bNvl7TreoU6zzVU95+nwFtNXivIarXX+6LVGrCl5Uun1Bu
iSh/f/NL9dUS30y9WAiE9hucIg80n8zq7pHUlSU8aQHH6BurBff+VfmyoRsn6irSO4SEtz2THbWK
bwILdVb2pmo9PXfgG/jiq3OJ0LkrZQ9PUdyYdGdDxQcymVkvczOgr/RKrSw5n9FL563oqlcOLSsv
d4tfu/WwDUL8Dvratv1qhC0FgT+GZgZnaPhxglJGWJItJyX+K0dUz+Dg8NN1eVsGzVae2yCWwodm
ztFwiFWcVfXm7PdApcZy1prUCabuEenwhm94QTiZR8eV4DoczMU8P3B0XUouGqmR6mmiIfTCgvrJ
JVCIyc8I6emRlL6cJ6tHUZq03QkJyeQ0Ug/yboc4hc6Mq33+Xc47q5D5iZglo13shCdj1hLHIzGn
oTaFUNpUunLVx1W3aowZq3z8KZ+3Iw+DbR6ke7llnBJ+M0shh/K0SdFLgIhldEY35cDdNpFKr/WV
nUGuZSZGMT6KAAIIIIAAAggggMBLCDSHM8JrYv3Stbf2k5hbi+sQycWJu67z7KysaO7LKfvDhJ26
8VeLRXrUGWC0ZKWyGIGnFSSPb5djz1ZZB04q9iPx99+/X7cvkwoLhV9//37fv2fareoMaGt61xCt
qRf7IPqj/GWlG0gotJ7ZvgxU24IQWoBZHWfpzWuJJzQfd8lnj+wjqQutrFz0LNug0zlJazp2K9uu
sF/ZK2Xx9UNiH83kvV2Ba6QetyjYRIy7PHfgy3e5/stH2u5dKXx8ijLQN6oDdCYzbT1q3ylOznay
8xl9suHBqFcmZpLdPZweokfr2toFqe9Vou1tOD98IUXndtPAtqj7aCIjLEncMl10p51pnQEtDNzw
WD/w/U8Z2Ybv/MixuKWgBpSnGgw8verlOgne1qxQZxgurt2f+priupXlRlTzgeEbDle6F06xcFAG
GLc4XurV06z3szdyUBzpuOxHr9ALC5ZbJGah5kHKTnRg7jcwfxCzH5kFNXPOWUkvqp2Gdn/x36+h
C/VBh57iQ3E4IJ+dwwwk4dXXrCdfbwIsxLfdd+2DR2Ql53nhN7MUSjxMmxRd0x69MSm+qHKibeQh
NDu34PMIIIAAAggggAACCLykQHM4I7jWJy7SWatXuTVHcQXEXtWRH261R14rq3vZpFfCpr+NYCyJ
ybnLM+q4RVm8Bb16QTryhYc9ns1HbmHVWstWZuFuT3256uurP5uxLO8tf5II8tpaMx6ITyM2vJpb
yL/+/nscSQnv127Zn9AkuzoQsyEntP2r+jWoskj5ajKr4yy9E1ritObjrBjWYT0QmUXE9nne/1j2
LAcuyXxHp9kWaA9wPf78RiEhtnZbOBFFbQfx9f2t9zBGK7tg4DvQ5fovVW+2PRrBxBQl33epM0Wl
DTvj5tTZTrpa1cIYdwrt44y8LEAfdYoki/2G/HFOK+uh1IvOqO2pwkO2tFrurcVnHk+0ez2qdJuU
hE5haGEQCI965hc/RhNx3PtT89hFVPVo1Su1E76tXbvmQNx0hOJkM1BZmfjKfoG9uPdJ4RQJmXIe
IR4SV2bp1aOX8axSnBfSf71D3izdJzjW737kqkj5tv/AoKylO+8ZoQjwYuTPZ1Vu7urUunyKk741
4B2YFE8b9GOSMMBrRZP/feoMoYg178Ra326HnuITkzHzjYr5nMu9TT6u/LdfSg/4cqdkp34gb4mH
iBcMP2OIKptp+uUO/YKK+KKu1KTomvaYfzDxqnVChDRjwfpYbj2EZoctPo8AAggggAACCCCAwEsK
1IczAmt95cp1s7ZQ/El6xFG3kFsYca3D2Vu0Fx26rRhlNVJdTLFTN/5qLd4qjzp5RvmKqjCxNTkt
tyP4yiphkyvp6xbJXRft0bre5ZC3ZIuXq9Qrp9ZNxYZ8DdGe9COD286pstmYjyW5l3JKN8IoZq3s
XaoiZQsSqo75etNaYvGahKo5BJqPsiJqr8rJayLtm2gCqd9PClWNLVKn2UvUVd9sx35+g5oWEvXI
W3boqvBAFO0gTXpSD/PcgS/UxuuBzxqO5Z5PbjbmFKUdv7J9lz5RTGSmvEm2UcytVqU4+apYbrSV
JnzARohp79U6j07MWmyen3oBtYeMsJcYOGcr7EQYmxPpJxPtXvWkRPk1kdCRDe2ld3VW6/GoSc8I
sLpra2vS/qucA28bMh7tYtVrNXSkLVT3NCXdJ4l46YKhNnxD7wnGfLbQEax9rKZIUtsIFcesgiWR
OjTboFMfX29XTwuVrbzmIDhjFlpOX/qXWCqvSew67sGHenG/XB4ZpSa7fVLcVhyYuijPWPLcz2sG
RuRoSzdVyxnKv9n2DzzFD03G5san+KyaSCI2VoVW1eyqGam4IsSUo/XN45s6eGupT5ygll2dtoJw
sBRKHE+cFO2dtb4qNaE9Zh9M1Cgdj5DwvKldVQhOJPgYAggggAACCCCAAAJvI1Afzmg339adNuX/
WmcepEuC69niU469t2gvmkhPAE5RI1+U3SrZyJuxJK4/Jzt5Exi1K5w3K7RhWq/97Q+GI/jV97z0
QCoKo6XuNCblEfN+lbx0Za3wdzlNBq2962Mup2pI4i2rGjcecfOxJHHbAXC/Ip+QecXBvsXZijpP
b1JL9EG71YqxRqo1rtBQIH4LsW3GW1hrwf/9b+AStVPIdez7bV4+JPb6DXMNRFHhau71VG3eGJGE
rmnWwDcQ8OkhRsmrOUWxXhwj9vHBIUZ7JfXxzjk+2xmaz4ht1emh1fatBr82fKq7Tc7sVhxUp6Tu
rUhXE5c1OKQdQB9pC61Ext3sRRbpuxMkykX2lob5roGmd9OGlyP16FEYm2ryJNO7oVj1fjUHbus9
kgYDxJyMTsjGI5ujnYN6YeyG5YhXlOVg04sWR0m9qbjsxGy5PFi9e5/o16U5COaeg8Zmc9lnhPgw
VzTD6iJlqFXr5CeH4iUDUxf3GavU8B7k01OU43RO/3PgKV6MQ09gSnzKvcpA5Zrz0tyTtZ36QN4G
HiI8/OeEX7bnjI0Za1zPnBT55JPaYy60jCgdtXUfffyHUG9iw98RQAABBBBAAAEEEHgPgepwRmYB
R1m7yewbaULSA6SzCio/c8aeRPs1+q5s+nbSrQzGX61sW7nLM0pX3J58Ajs3RT00z1jLo9MQvrGY
dPOV8iWn7jUka+l0/5u11pjX7vJ0FdGWcJlnex11YulmJ6RlLfa9ZnXXv1jbUB7+z9Sb0xLTzWc4
Ar1uuOkf5cZb9YGSQB86Ry9R+wV9GIt2AlfEebZzHuB6CCnrZlonvLNa6+X3Tz134BsK+OwQ4+zG
VN3L3AFdCG85AXtyst4m0ShOqNa+ME7t6YN+aBm47LOM45/q8Qy1A5iSursiXbSt9c0v1izHCJWt
IImMu9nTdy32VDpB+SItDKLhURSrybZ/B4/EdYjuuheHhAP3LE4Ue5/2y+hNnuu/2yJqo5idjdhr
U6SinRJOR5teuDh6MNdl1fvy7m0S24VesA8cSp/2nG4G6aRnhPQPVFmli20uP1rvcp/qkqGpSzXl
aqur7SmUec7QFGUKndcRWY1s/5vYBdmLP4vU0aWMeG8zVLn2vDTxCG/PBofyNvAQ8aLhl3sUGls6
s/qG4KToEWsDM3blvVBDrV5qalaUpmxjzyyNgn9c0Otk+DsCCCCAAAIIIIAAAi8sUB3OCC7guJPk
5j7u5+WlJ3E5Q76XPNePPAG0T0Dycqj9mGb8VV/Gtna2Fo80Y3lBsxgRr4PiJutFsQd+JYm6FN03
OZWvAMjvFJcbklU7RgU0N0trV9dfSNRFh7eELyyuxMPhnlpimSLPqASI0mnG7u82/f02p+hNaYkV
ffOijL6AhyJQG6HCjXdNfdMUBJpEBi5JxGGsYy+z9OIhMcC1ly5cj1UdbVcpbeS5A99owAsDnDpB
U9IQ/9ntc9pjMslO+MCLM9byxRrFWdUqxtYogvPaC/kb4P4E18mOvS0q5Km8X3wucpOqd6u2nAe9
2l2YRMbdFIwdHv1Pyl80k3BPr8JEtaWYiMwJqjryLohm5tFErI3KKY3IfiDtRZzi5UoXeBoevuEZ
4XS06SWmz6mk+iZtVFOi/T96MbcXkIej0UHZCYrcM0JsmBvIqtkrlanuet20rWzgmdWMQiimIcxz
/CnKaXRuq7dszf5AL1RawPcJdL8DcdU+6ZlnUJZZhrPcMvJmEKf4uYeINH498lk/YzEqPPOBuo3n
mZOi4t7XtMdmxLf6fvPpKRwh4endGT/H5XZFfAABBBBAAAEEEEAAgWcI1D9r8owckCYCCCCAwDsL
LKst/pL+XsiBS95Z6Gjer+dKLlUfLSDXI4AAAggggAACryfAhOj16oQcIYDA7xWgz/29dUvJEEAA
AQQQQAABBGoBDmcQEQgggAACBwTE7wfa9xu45EAG3/7S67mGv7789tYUAAEEEEAAAQQQWASML8dj
hAACCCAwWYCH0Mmg3A4BBBBAAAEEEEDgdQU4nPG6dUPOEEAAgTcQGHitw8AlbwBxWhav5mIn4rSq
5MYIIIAAAggg8DYCzIjepqrIKAIIvL8AXe771yElQAABBBBAAAEEEAgLcDgjTMUHEUAAAQQ6gYF3
jw5c8snwF3NtPx6c+Z2aT64fyo4AAggggAACv1GAKdFvrFXKhAACrylAj/ua9UKuEEAAAQQQQAAB
BE4S4HDGSbDcFgEEEPgIgXUZ5ftfuLgDl4Tv/Qs/eAnX9kWlP8t/HM34haFEkRBAAAEEEEDAE9h2
CNc5UWKO692bvyOAAAIIFAI8hBIOCCCAAAIIIIAAAh8qwOGMD614io0AAgjMEBh4rcPAJTNy+q73
uIar2YhgG+Jdw4V8I4AAAggggMARgXarkOOqRzS5FgEEELAEeAglPhBAAAEEEEAAAQQ+VIDDGR9a
8RQbAQQQmCEw8FqHgUtm5PRd73ENV7EuxibEu4YK+UYAAQQQQACBowLVViGToqOcXI8AAggYAjyE
Eh4IIIAAAggggAACHyrA4YwPrXiKjQACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCA
AALXCHA44xpnUkEAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQACBDxXgcMaHVjzF
RgABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAIFrBDiccY0zqSCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIIIIDAhwpwOONDK55iI4AAAggggAACCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggggMA1AhzOuMaZVBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEDgQwU4nPGhFU+xEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQOAaAQ5n
XONMKggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCDwoQIczvjQiqfYCCCAAAII
IIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIHCNAIczrnEmFQQQQAABBBBAAAEEEEAAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBD4UAH1cMZ/f7/+hP/7/vdSfGvev/7+l87Xv+9bqfcCqfd6fPD2
XzQZ7Yoh6jaf6YJywasL/Ptu29UaQS/W3v63R/ajRbxa/l69pk/L36yAeZ9QPI1y9MZHBqPRNLku
LSC2lFnNJ52b+wUDkTNwyVjeLr7quRVxcWFfP7lXGw5Obbx9YadX0HnNdlbDubzG94ciJpPT440b
IoAAAggggAACCCCAAAIIIIAAAgi8hIB6OKPdb7UOakRPJ1xV4vEVyf7Ig3av8kRFaPlQv2CIerld
KOmr3ElnlsCjdtt2dd4a/pF89+H7av3BkdK99bUzAuadQvEVK2t8MHrF0vzWPIktZUbzOQA2EDkD
lxzI4GWXPrkiLivnGyT0isPBaY1XLuz8Wjqt2c5oOM+p8W1WyWRyfrxxRwQQQAABBBBAAAEEEEAA
AQQQQACBlxDQDmcMvc3hJUo09I3TR86Ft1EEXpwRe3VGuYVdrzcOUnM640XCbX42tLX609bwDxRB
il5ODB0AnXnp8YB5p1CcKTftXjO2x6ZlhhtpAqd+936MfSByBi4Zy9vFVx3vxy7O8G9N7jWHg5Ma
72VRd16zPV6EJ9U4L874rV0I5UIAAQQQQAABBBBAAAEEEEAAAQQQ2AS0wxnbWYI33GgdW5FclgOb
L2pFXpwR+RmH6vUCteko9XLdG1YQ7c8U0Nbqz1vDH6+QbRGdLziOI5515fGAeadQPEvx2H3HBqNj
aXJ1VuC0795nM1J8fiByBi45kMHLLj3ej12W1d+d0GsOByc13uui7rRme7wIz6pxXpzxu3sSSocA
AggggAACCCCAAAIIIIAAAggg8COgHM545z3XsRXJx2pg+Ick6t9y8Ham69cLKC/O8G7Sxut6U05n
/LKWfNpa/QlOa16zwXtCVrjlfIF3CsX5pZ9wx7HBaELC3OKwwHODfyByBi45jMQNPkjgSa9RcIRP
aqcn3bYvzCs328sQapZ3fv78oP6AoiKAAAIIIIAAAggggAACCCCAAAIIHBJQDmeMvs3hUF4mXTyy
nqi9hcJ7ccbX9/fX7VCHfUBivfv3/QBI8+Fx6uVK9sUnBc6L3OaV1+pbom0RnRNCLxI9U7PxTqE4
teDTbjYyGE1LnBsdEXhy8A9EzsAlR4C49sMEnvUaBZv5pHZ60m2Fsrxws70OoXZ5Vrof1qApLgII
IIAAAggggAACCCCAAAIIIIDAUwXkwxnv/MWlgXU99R0U6t7z9sKA5XCGdUBiu/vfv/eDHPXhjAPU
vDrjqU3nrMRfeK2+K/JAYzuLjfvOF3inUJxf+gl3pH1MQHzSLZ4b/AORM3DJk2hJ9i0FeHHGGdX2
ys32WX3gYsK58zMCjnsigAACCCCAAAIIIIAAAggggAACCLyIgHM4I/eFeGeZUVrp6w8YVD8YEnoj
xf3Ew7aOl19P1N9A4X1V8OechfxzKEXtbuuM/5azGcqvmuSo7wkceHXGdibkjue9+qP9tLZo2nqV
1zXlq24p3K8NjDIs/DXbJsPqBePhFyXZI6EKbOk3dLoPVFXjreFHMzRe5LLLalNbwqiDHs3VbhFo
F9FE1gLkgn9tZksRhR8/etx3YvBLg4NX/10GnAtMhIFQnF/8bLXqQ6rb9Ooa3EPussHIrd0No2sP
UaYJo3OqVZZtQuse2ha59h8HBo4yCpL1XvVebo1o4RZMVB4YgtMYcTSMzK/UGoyOmEu2o0FX9Yxr
D+qP39JQI12VyoY55Q7W2vxeTmgm3a/rtc3kcAe13WAptjPIJoeDulaceycDr6tEsZ1ajdeLGbuw
/cw+EdXhZvvpE4BHHfUt3qu7F3mqJhsIIIAAAggggAACCCCAAAIIIIAAAggEBOTDGduLIf7+F7jH
9hH7XIS4YFqdL5B3fMXFXfGjt0+m91T6zep+26TNQuHjrW7vBZR1BqnvecyfQ5F2ArZNZ3HPRFmq
ls9zlBkSamhllCqvTbwMDCkP6gaPmmHpirHwU5NQ9iHkuG5PxSifWrNtVPZIHT1um2hxdUegJFka
D+eqvnBs30g7bqRWhRhPsYpbZCYGv9jpVrEqfKLtSfSAcRFGQnFu8bNNTBumRmow3TRmDEZO7crD
xEgTU36Cyx2dU62yGJz2M03GCY25waN2bMpBRLGlDAyuqWC7R+xA5OQu0YdRZYO/rC5ljE21zdSH
rX6vi9vjd34kl6q16YE60oSndFD3m1iT370yMsOBXB55EE9N1bQePtV4AzHjFPaRjXzOc8320ycA
d9+m+wnUXeZhlc8igAACCCCAAAIIIIAAAggggAACCCDwZAHxcMa2kBj41nqRf+dchLSKWiwO62vk
7tfx9w2F9U0W3qsg9lxbezBagUqf5f9X9jGE8s17cUZwab2KMGsnov+qmr5zUL+tZEmi8JKvvJVd
zUEVbPut/t7WaaX/jMMWyhXd/s5I+OUErYX8ZrNSwV5Y1MY1WEc/t020uLaX0i5tv5GrVURbd3tF
tHc2u6BkXSQ/7sg2mwcTg18eE+yNtK7Pdjsvo1ENhOLM4ifrSR9Cx2ow1zT0zKYGI7t2pVFqsOE7
O+5lextslX6f1zbqmcFj7p2K50PElpKfA+WC7SebA5GTvcSvQSfX7nta6l6kCa4ZDVnux2bc+d5v
5GptcqAONuF5HdRSHudVJuHhQC9Pn0I28JReXmynR2LGLmwgZIQZS7bZfvoEoC//tPb+5OdtkkcA
AQQQQAABBBBAAAEEEEAAAQQQQGATEA9nrOuTuddPOy9yEBdMl5S+vr9vr7FtXsewLZPWy53lOl33
9c+vr8eNolm3zmaoBap8lv8hbiKvxbt9SVFOaZD6UX/a/qsa3+XCc5HhQlT+1/qsS3GXmnkr4vfj
QEX/5dytovc/bYlX92qWyPe/ybVfcbRxVGzAyF/Gy4TfnjFpG7N/oUhREDWMpTspbxJRTgTobadt
CiMtToun0P5/WTgvctaK2Ar/33/Wq3tydZEK/mrXLlRxE4Nf8S57k/YjfVXIPUsGQate8c4Ti5+r
Vn0wzza9kaYxbTCy+nLhb9VmVaaJjbw4Yxmdg62yOnRQNZ29QpQvRX8fHzh+wmGw3uuuNTswZxNV
lNYYFKcx+WBrQrqtQXUwdWuqnhHIE4iiImJjpdiUzd4mlA1jup+ttYm9XH02J9OEM9OV6vCJOlsO
HcJ2h4MlZqseZgOOzOGrthucwZvhoUwpA9Godsf5JjPS0uWQ/ZQJQF/6WVMCnvwRQAABBBBAAAEE
EEAAAQQQQAABBBB4HQHpcIb9LaXmC9f9LvvAV3OV8xTbUvjPz5Ws/+2Z69JRNh4tbCmFLinta/6P
VWjjy4fL3cuPWZvr2msG1n9Xv8UaXMfe6frF+G4RXdlvWXC2j5dJV3FjvJO4Sd2uZeGcjVIOZR/g
kWNxDd/KcHFN5OxE8T1ocenfOrsj7ma0FzhKwv3TdaQUWW8/cus5GjmhnaJbpvT97C0LwhkgYXda
iBzhDoWDUPCJwa+Br0kYdV38yQ6YQA/gnk1T9rjM95En2n7fq9mV0rBNrUGnNxA6qfRgpO+89X3X
gSY2OjqHW+W+HSxdku6HDUen+znW5Zozg66FHgi28DRmYOajHuGpB0XxwI5ZU6G2meqf1VHG6sdC
2dDHrwO1Zv/qQqKX649zltOVarp4YLoivEmuaFix+aPSIgZyNdBFpMJDHLly0ag1/yOTzHBL//AJ
QF/8XN29ztM1OUEAAQQQQAABBBBAAAEEEEAAAQQQQMAQkA5nlPshiQMD9naGuL5mbzJJl4g7zmv5
jO0LmcDOsrdAuyy2lkcwymTqvVT5ZoPUSzL6mqVUXHNduV3P3iiVhXvp70Vh2o0xo2bEHZL9VmL6
wsaRZyH9PR9++7cg+60/IQVxg2urG6nkWjHM7CfqaKDIau8h5/VY5MT2iR5ZMvbWuqylgl85y2NW
3MTg18DVEI8GUg4hE4r62wrKX2/oKleswUy16iPbeNOTt0tPH4zq4WIvmMBxpInJhyzEunY6Ydl+
pB+e2XbG672KTq8YdeHTiQ5MYwYu0V8ZVXafWpdrDjmR4W9KQxYrYsqd5QObF3XyI004P3abHf5+
u9DBK284iHebXtvy/i5Nsv3Gm4oZJQtezoS/jzTbD58A9MVP1R3P+wgggAACCCCAAAIIIIAAAggg
gAACCLyHgHA4o1gD9o5mVF/sGzjpYK5cSovn9vai/Y28vkLSeyr3W7Qrhcv/Vn6HYvlneVl3lHop
irdWXJXYLmyD4zgXCnuxrcV+4372vqCyb9DFmh18Yn6L999HT4D83Gdox1LoC8pjOc77Dm5XC+Ub
qaM9+4kiaz2ZaH4sV6Ftoi0/8bpIBb/dc8sVNzP41fSXpLU3+YgvcRdiK3gARmtS0r9PLX68WsfG
WKfpyT+JJUTQ3MFobZpN7Vj7fWprMS6SrxlsyvqQmsna1OAxQiLT5brjSTT0xEQHImfgkmKwir1s
py7T8QM7MxqyXBEz7mzV4Nmd/LFRMjp2O6Oen4mSyBsO7G4zMMXZEstMa82Oq476TMzYUZfo3Iaa
7adPAPRhJf57ldH+mc8hgAACCCCAAAIIIIAAAggggAACCCDwLAHhcMbgRqa9pCre1P76oPB1KS9r
mWVd62cR7rWhJdb++5potWTb/qO92hvcLm2DxOOQ1tZDSQVu3NeOsQ5tfPPNfAW1+L71sm5W83Iz
xT5RVJQ/HX5VWCzJBDy9IzhClgL770N1NFZkuXMyN+8Mlj4YrPAwO8am2tVEA1ZSOomKmxj8eonF
nkb+9RFjez4QsbcsaH1p9jBVtu0XPW+8iWlmiRpMNw0vqHKD0Q5ejSPCuOGlW9Td3n8eGZ0TJ6aG
+uET2s5AvfvfvffmaOFEvRocaLxi/Uaannv+ttl0j3a5g2Nla6zFbSobZsWFa8086xKh3tuRFwBm
E5ZPQZw9W/aGAztXReMa6iK8/j3UeOMxIxd2IOdeRSfHiE+ZAPTVHa87r5fm7wgggAACCCCAAAII
IIAAAggggAACCLyIQH84Y3SnVD5+sBRTvqm9dNn/1c2amYdO3P60llj/78t9ygXi5UPbP8mLsG55
nCBJFDe1ChzJV/eZyPaEsC9s7vWq24JNyb0NlmIHStgmkHer9eAUUtM2vEPr+YFvlfY1PVRHzpvc
va2EOhyl6DuWq8Qu8JqVUF2kgv9x52TFzQx+o9HrPU27N6YHTPBshnzmo5ApK2t68UPVaveOyRpM
Nw031BO9cxVzBayUhpvuz83UMUo+7WaPzolWOdQPTw6e0Xr3zsSY0ZZK1K3Bgd5eDDYjAodq6kGQ
apupDwvEehGO3vmJnbwbAGYTjk5X3FFv/UCogWsVEZvD70kcCLxoeByPGfEOAzl3Kzo7RnzMBKCv
6wnt/UWeuckGAggggAACCCCAAAIIIIAAAggggAACd4H+cMa2zxBas90Y7aVgcQnXvkRY2HSXMoW1
S72encVrbdW5//flRv22//4v8s0GqdciuavvRdldutIp8uEucWOV3sqotcUffn10aG/scUKj/xkW
ebPDWVePLBUbufpJ09p4Tbw4w2qlnXu6xRm9pHivocjJxLGUIbcuIrmS4l/4VrlScROD3xqY+u00
xU745ySCVilmtUcPXy1ltLPkVqshNa/pPRI5fTC6JdKNXmJUReox2fAHRmfZfqgfntl2JtV7rkfK
JurWYD+NGbjEfjPYUE3tlZ5qm6kP14HlVMSBO1vH787u5N3aLHoc53xvOxcsTl+5qWRmy6nhoHso
EHLlv7FFfhdHFR9iro7HjHyHgSYztQrKcag/7BL/Nazgc12qxmf24YE5p3zYmkd6BBBAAAEEEEAA
AQQQQAABBBBAAAEE3kqgO5yxrXdHv2G9FNdch5R3umN7c8KrDrTlRXcttKoaY0GxWBxvv2ssFWW5
05otfQu1vtko9VoKuwDiKrb6OyGijLGM2zFYhxmMerH2etUI7NLKVXwkYtVfdZBZl9Vi8W0Ct78J
klLtaeUQ/j1Q5L5K0i3O6MvEe52Qq2h32m7UCV/WjW1LVDeKVdzM4LfK2/Us8YCxsiglGb+zeHRh
u2W27bd50atVhxqoQfVFIY9UhG7KC3Xv733229qV7xC4b7Lhj4zOMn4gb92FE9vOtHrPlCOfqHf3
gd4+3Rl7eQj1uqm2mfpwO8Uxe+6RO+drTTqgNdjLBfCTTXikg+qnqdnR/ufzdlGEOU6g7KHYu38o
Hfbbrc2YGe5425x7hfX+rnaWW3uID9PvOgHQx8n1iEZsWhePKj6JAAIIIIAAAggggAACCCCAAAII
IIDAJQLd4Yx1tS+2jd8tdoob6vIpBHFffi90v+zoLS86N8ytnEp75+V6sP4jJks2yg/INxukXsuR
KW7ms+YuxJJ4X6Oa188FVuLS4rKb2TYtLzDElpQOP7U9Fgv9wokA5YiJVPlaloR/DxS5r6N5RVY2
5M/IVa4blOrCjacqCbmz6vulsoOcGPx2cZeElphSC2YETPDMXSIU9z2yqS/OkCtFPOfUmI3UYLpp
eKGeC7pHCaraVW7gpVv0t/tXz83MDI3OZpeamjvMazvT6j1Td/lEvRoc6O3FDFul8PKQ6XTF4U+7
QerDsROS7ZTkrC5iWqAG8GeM3c7OfybI1YqwbyIUNFD2eOylw769tRyNcqEGcu5dkqoCYYjQ28ev
mwD0QZHrSeJBxScRQAABBBBAAAEEEEAAAQQQQAABBBC4SKA9nOHtNWjZslaC92W0xO+6SwuXsa/p
BbeG7JVT7a/yvy8Zuxev/P/bfYOq/KPUaxVkvnbnLAM3t3LvrO8dRN8R0chU27rGLsjtMsHNzbAQ
tvY1PZj5+f6ohZOl7QLpDE/7VUDpXm6R00y5rQL7vJH7gzTFO8vdkvSVl6uLqcEvVZzVlQxu6Sn9
7JpS2dMITU7wySGo34i2DlMdbvu5alVHyJlN75HI6YNRlcy9S1Djxm0uyYY/NDor9m7e+usmtp1p
9Z4pxkCiA9OYgUvslxpkinivtFTbTH3YmOn295l054Fauy5QB6Y4+Q4qN/vUvGJxGZnjjDzviKkf
jxmlUOkmE32vSPCBpRqJfucE4OhMbySIuAYBBBBAAAEEEEAAAQQQQAABBBBAAIEnCaiHM3LvijW2
/opvOFVb8MndwnKDQPr2957Mwe+GV/tk3bcw5T2zJfGb2fIB8RhKnTP7dIgbD7m1YuvTXUa2DWg5
CCTqSAAYPxEh/x6IWJPZ3XElcNLhZ+yyC3tXwaM/VXxpWRL/fW4dudtvbTgq5ZufK7Ed5OrC3Fhs
CzKz4n7yHtnSy3S1RfeiN2kzYKTUulymQnFa209Wq9JFTq5BrWlYmU0PRktRtto1yjC3iQ2OzkP0
4omTacHjvPCpKGjRAsTUnaGhKvtIsA1EzsAldim8yUeXYKptpj6sz3SEIsy580itzQvUbTxSduWT
8yutg7IKKbcGtSZSw8F+F2mESgdeKjzMH6CSHwy6THqF1c5SCKE50Gy9ef+vngD0hZ/T3j1U/o4A
AggggAACCCCAAAIIIIAAAggggMATBNrDGfZiWHr1dl+Hvv1A8MEXZ+zbL9366LZj9ZPKlMMZioO2
tPz4+Nfff3+/Hv/Pz3eft/+cjexgfhv7ZEVteehSkw47SP+2pl9QF/Wp7xTr3wFXfvCkSKCn2f8o
BpNU+ftGRHmJkeGfkgoL9PJtFpW+Noz60YJVy5L871PrSC6y0R9pOxizcyVnIVkXqeCfWHE/mTfu
ltkDbvuSracRezsxYIYQIu9wsc6+5IqfrFYtPOfWoNY09MwODEZLUZZ7fn1/34cReSjLNzE10sqc
ZkZntWOw8pbuh3PBY7a1VJdrDw3xcVhLdCByBi5xSpGtqVTbTH1YH2WEIky+szD3SoXKI+/ZQN1n
xXb6sfmVOnZvJWl78eJoRmy2nJuZrBUqdzvZwEuFh/iyimTMqA0nn/OBZus9A65DxPqo8asmAH3h
k3Xn6fF3BBBAAAEEEEAAAQQQQAABBBBAAAEEXkegOZzhfa9Ny7i0hratzYpbTc7WpLxCWqwqF8va
1RZT5BfH76UwM6A5aMvwj39fN9WaxXD5olHqRxWsV4e/dS/Slav08mGHeoNQ3cwzMK1ySrVc5qk+
0aPmtvBodjTVfdKB8Itg7YjFp6UzIfe91/q8kgalZbUom/Te8Pb2dsB7r+BuG76/g3E8coxeMlcX
xamuYi9Ducd4xQk7bdng9weGuqcRv8WrBEyiB0iF4kltX+uP/B5vag3eauT8wWip96bvU4qabfji
ocZDo7MepvIIXZ3RjL4/K912JtV7bmDOJ1q8TqfqI81pTHrm4wxwWib8mlJP2ZajWbJ/luNJLMKU
O5cDQnJ0ntPJZ5vwwHSl6LeqecbAbDk1HOx1qcwRsl2E+8whvXZN7GRyodv3vvmcp5vth08A+uIn
2/v28bEj7z4/n0AAAQQQQAABBBBAAAEEEEAAAQQQQGCWQHM4w/gOoJ1is+S77j3fj0qIi7T2FzvV
xWgtHfur5GLerRwoDuq+TbmE2C4LKhcNU+vbhYNV1K1ZlynsNVn9f00hDcvs10r3WzVblXv69s+d
yDlWDszYP9yilFIxaXbK1SZRXi68Yn/985q4DuikEK+jnwp3N/Oq4LI+PTNXekR7ukp9i1WX+axw
qmZi8PvdelVu+0Xt+ktyJAX5l4UenzRDcWrxk9WqeHl3aQ8uPX8wWgpSZVw/hpJrYvvpkq7ex0Zn
M0qdvMX74ezA8ZOr0XqvWkp2YM4mao2uxjQmOfOxQ/peg7macj8vv3olNlaaE7RDo7AWq9lam9rL
+fhzxm59BmW+m6c1a2/TDgepSZRfdv/4XRW+kcbr1XZgGjbUZPRGM/DA8qiWXz4B6Ntrou4KnWAQ
+RMuPoEAAggggAACCCCAAAIIIIAAAggggMBJAvXhjNyXRussSWtojxWy5S/yl9nkRTRjKVxacL7d
JLAbIWdYyIHmENknV7dEQ791EqvmdFkft9XW6u3F9X57pSuisVef/f5zdSspw/pX4hKHOZxAV33V
JMSfIFCWlW8lkFJoPr7WilXZen7UOlJ+Mj0XUrHd7GOR4zWFXF2kgj9VcROD3yty04aVppDsPKWz
WYlQnF38ZLUqZEM1mG8a0wajpRjKd7P7QiYa/v3isdF57OvHA/3wnPcRKOVcThcJzUJsKbmeUMPV
E9XHYmcakwm22Fm7RE2ZE4ji/NYeqocb8qRRWOtUh7qIaYGqz8aEqcTodEVv+Lkgl4eDA7nKBp5U
h7nGG49GZezre+h6eqP2lZlmywRAEojX3fbJsZErwM9HEEAAAQQQQAABBBBAAAEEEEAAAQQQmCVQ
H87QDx+E0isX0fbVsfVfqyMA9uaBt7VQrp8ut/Uu6QtgXKE4GKvR+p+Uvxyhlk67hCro9qF67dn7
glm7Lpo+S2OUU6yALlqEurbK2iytK4u0h8KvWyu2VoIVbiUDxafXm/qBHaujQ0VuxP089XtP6cgJ
RXSqLlLBH644Y58rG/yRIu/5UtquWzmxHiAaiqcUP1mtslu4Bo81jSmD0VqC7WZex3y7INbw11sn
Rufc1q2oH+uHTwmeY/XuNp+DwVZcPhA50UsSNRirqT3bqbaZ+nAt61TEgTvv6YRD5ZRADTfhYx1U
NeV79CvGTFYZg6LDwXa5346ygRcIj0kxIxS2Sjyd82izZQKgCoTae2r4jGDzGQQQQAABBBBAAAEE
EEAAAQQQQAABBE4TaH7W5LR0XvHGy0reu33Jqju+8Iq2w3l600oZLi8XIoAAAg8Bf0cTKQQQQOCY
QOLszrGEuBoBBBBAAAEEEEAAAQQQQAABBBBAAAEEEEBAEPjkwxnvecphWVePfLP6DSP+PevkDaHJ
MgIIvJiA8aaVF8sp2UEAgXcVoJ9515oj3wgggAACCCCAAAIIIIAAAggggAACCCDwSwQ++XDG+j3l
d3p1xnJ44Z2ynGopvDgjxcWHEUDglwjkf2vglxScYiCAwFQB6xU8Wz/za2eRUym5GQIIIIAAAggg
gAACCCCAAAIIIIAAAgggMF/gow9nrD/G/T5voXicXfi9i+q81n9+E+eOCCDw8gLrt9l/ce/+8nVA
BhH4HQLbAYy2O9m6mT9/3mfa+zvqhFIggAACCCCAAAIIIIAAAggggAACCCCAAAKbwGcfzniz0xm/
/WjG/9b6YN+ALgoBBH6/wL6Jejt196sP3v3+yqSECLyKQNexLP3L9n84mvEqVUU+EEAAAQQQQAAB
BBBAAAEEEEAAAQQQQOADBT78cMZyGuAtXkXxWG5/i6yONiRenDEqx3UIIPB+AsUX2W/bpuyYvl8V
kmMEXlHAOp7xq2eRr1gZ5AkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEKgEPv1wxtuczviAoxm8OIPO
CQEEPkmgOJzBhuknVTxlReACgf6EBt3MBewkgQACCCCAAAIIIIAAAggggAACCCCAAAII2AIffziD
AEEAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBM4U4HDGmbrcGwEEEEAAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQ+XoDDGR8fAgAggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCCCAwJkCHM44U5d7I4AAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IIAAAggggMDHC3A44+NDAAAEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQOFOA
wxln6nJvBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEPh4AQ5nfHwIAIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACZwpwOONMXe6NAAIIIIAAAggggAACCCCA
AAIIIIAAAggggAACCCCAAAIIIIAAAh8vwOGMjw8BABBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQ
QAABBBBAAAEEEEDgTAEOZ5ypy70RQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
4OMFOJzx8SEAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgicKcDhjDN1uTcC
CCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIfLwAhzM+PgQAQAABBBBAAAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAIEzBTiccaYu90YAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQQQQACBjxfgcMbHhwAACCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCA
AAIIIHCmAIczztTl3ggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCDw8QIczvj4
EAAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBM4U4HDGmbrcGwEEEEAAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQ+XoDDGR8fAgAggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCCCAwJkCHM44U5d7I4AAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IIAAAggggMDHC3A44+NDAAAEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQOFOA
wxln6nJvBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEPh4AQ5nfHwIAIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACZwpwOONMXe6NAAIIIIAAAggggAACCCCA
AAIIIIAAAggggAACCCCAAAIIIIAAAh8vwOGMjw8BABBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQ
QAABBBBAAAEEEEDgTAEOZ5ypy70RQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
4OMFOJzx8SEAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgicKcDhjDN1uTcC
CCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIfLwAhzM+PgQAQAABBBBAAAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAIEzBTiccaYu90YAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQQQQACBjxfgcMbHhwAACCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCA
AAIIIHCmAIczztTl3ggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCDw8QIczvj4
EAAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBM4U4HDGmbrcGwEEEEAAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQ+XoDDGR8fAgAggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCCCAwJkCHM44U5d7I4AAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IIAAAggggMDHC3A44+NDAAAEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQOFOA
wxln6nJvBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEPh4AQ5nfHwIAIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACZwpwOONMXe6NAAIIIIAAAggggAACCCCA
AAIIIIAAAggggAACCCCAAAIIIIAAAh8vwOGMjw8BABBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQ
QAABBBBAAAEEEEDgTAEOZ5ypy70RQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
4OMFOJzx8SEAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgicKcDhjDN1uTcC
CCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIfLwAhzM+PgQAQAABBBBAAAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAIEzBTiccaYu90YAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQQQQACBjxfgcMbHhwAACCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCA
AAIIIHCmAIczztTl3ggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCDw8QIczvj4
EAAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBM4U4HDGmbrcGwEEEEAAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQ+XoDDGR8fAgAggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCCCAwJkCHM44U5d7I4AAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IIAAAggggMDHC3A44+NDAAAEEEAAgdcW+O/v158/3/9eO5PkDgEEEEAAAQQQQAABBBBAAAEEEEAA
AQQQQAABBBBAwBBQD2fct4J+/vv6+98rAP77vudG2pz69336jtWpSRhFy8uvN3tosZmXFxy7oo+Q
qdXqZOrU+BwDueYqETn+j1omr6y7a6BOTUUcKzCcY9706N543A4A60Cw/F/98rUSE+PGNZdojCem
vt3anVkEPinUiF2JdcGaCjxvUJ814zve/eYbDtOevNlFV2hVI/z7Sw0ZL5WZi+rqtyfzu+v0d5du
ODZ/B8tzHzCH8d/lwlmTn0d5L3ge/x1JXBIeTA4vYSYRBBBAAAEEEEAAAQQQOCKgHs54rSUN5dn5
8c/eftURn/+dnsTEZYF+G+hUmkOuv+diOUImVqtJdXp8vnJFicjxf/S2fWk9scqXxoqr4j+Ww/f8
lHrQQg9M52yG8uYN+SrzZMI1l2j1dnLq6+39DsD8pHPKQvP16vBxXMPPWzLoJ834jne/yXz/bIW0
B1im22SzxOfXXSq5asQqe6kh46UyQzhNEfjddfq7SzccAO/P8twHzGH4t7pw0uTnf+evF/2WJC4K
DyaHF0GTDAIIIIAAAggggAACCBwS0A5nvNiSxjO+jXlznfbMrlfStCSkrSD3m7+HgoeL9QiZVq0m
8jWpvGw9x7uFFFTqwy+Lc1nGeHHGGdTOFr3cs3svXZB2rY2EtE3uay7RVE9PPfA6jCVvxie9qlBf
bBW6MPF+k2BwzprxxfvkYMacjzHtmeN4wl20qlH+/aWG3ZfKzAl184m3/N11+rtLNxyv786i5f/d
yzVcoWdceOrkZ26GL6j3C5KYa6LejcnhRdAkgwACCCCAAAIIIIAAAgcFtMMZr/V8dv23MR+ss57Z
jUqalsT2HMrXRg+2iczlWvVNq1YzM9ekkvG49LPxbiEFlfrwpQV+zcSksQLDY3VVHEB4HMO4g37/
XX5sTPlxiwH1LqFbYvs/Oqc59iMi0y/R/C7I8JqEP5KqnyxXhdvbVIdL+kM2XurO5aNxN2nGF++T
RzNaX8e0Z47jCXfRqkb594HO64RML7d8qcycV8yPuvPvrtPfXbrhQH13luc+YA6zv9mFZ05+5lJc
EM8XJDHXRL0bk8OLoEkGAQQQQAABBBBAAAEEjgoohzPe4fls0gO1JXhBEkcrcLve29OZlhA3KgSe
+72md4rPq8Lm+De3aUmpunqHsSJVoOd/WFhUvAXlY5d/25xX9/X9QwVrEbd7dZeojeCaS7Q6uCL1
Td9975T6SXdVeD+80cqHUi/OfriZjIXzqa34xHGKzjpWv0/4lFY171BlJ0bsE2qCJG8Cv7tO36FV
EYdpgec+YKaz+54XzJr8XNDD/I4kLgoT+sSLoEkGAQQQQAABBBBAAAEEjgoohzMueAQ8mvNZD9RG
Pi5I4qjCdn1oT2daatzoIfDc7zW9U3xeFTHHv7lNS8rV1RuMFbkCPf3TQgTe/2nZh1cDNFsT5hEC
+WbXXKLVwDWpx5d0tU9GuhCtLMHU9+MZc05nZGMn00hOHKci0pms8tlpAlrVvEOVnRix03y5UU7g
d9fpO7SqXH3x6ac/YH5KFUya/FzQw/yOJC6KK/rEi6BJBgEEEEAAAQQQQAABBA4LyIcz3uGpZtID
tSV4QRKHK3C9wQVP7dPy+ntu9NzvNb1TfF5V57w44yrpJR06nungAun+4oyf1JT9+/Wy6G698QoO
bVvgmks00GtSN0+AVFlTp0mxNiF2VfHUbY1kUMZynLzp+vETx6lT8z1YXC67C2hV8w5VdmLEEh1P
EvjddRo80/cke5IdFHjuA+Zgpt/tslkj0gU9zO9I4qIAmVWvF2WXZBBAAAEEEEAAAQQQQOCDBezD
Gf770a2HReNvzWNTu6207TvcvyzcfHj/45/yv35Lav9y6f1zfmG2OBhNIpOGvoDdb7JV+WkK2pRy
JelK235O40jVxVY3t2StjEXx3edp/dxQtoBKlmLLH3aEdKWo8+btnnY3b7Majs8uoqWk4+206CjH
G5f9ful4ZsRQif+j1um3AVZaexW391VLM4yGfZuZaCxv10WrI9e3NNkqKbaSSRVm1sKOaHVrRtLb
DZbr47WiVseL9CpLoXu++7845XS7ToVUQxEq1tsBmnOJ1jIvSj1+6EH9pB5OZdnECounrp3R8Waz
4VZc3sgdkh4fDna/bqj6gOa0x57D6KOYOn9rM1ymPmPaU9dZtPd3GK2eec10PdP2QqeoYXWe2Y2B
2ye/v386sf6/R//jlKUlsfvC6GjozQG2rjE1f4umPhqlo5OKe2mDDbkZiYzaVgznPBhOmnCKXVOq
TiNubq9mBHk0ZryIXScJp89aoxk+NOH0ZgLVU6c0G+yq5Io6mlLkMx8wo3XnBdvaEU0aDYNd0yHe
cyY/4efxcfnRJBJLUrGQ82cOp9ejOz8yJ4cDY0p4pA5MoPgIAggggAACCCCAAAIIICAIyIczrJU1
eR1ZWDDcntH6v7XbLMv/vn+wfrS7PQrWuQk9eCnPmcETGseS8DbRSkARutSoTj/sC9vF2rRS1BJd
1ZCyGq8LgWnNlyQYW1OuCt8HrLxBN1JApZr81btypbfZaliLWFWrnDl5d0GJvHsy+xWh+DyjnT7u
eaxx/dzAMM53GnU9Wg0q1jDLCJNKqobxYZalxtT7KBtSqXRzfcseQ2LICSfn6gipcjwz6Xsl9Ku0
6hD/Hr3Klv02iu8l9frPooyRfSe5Jy0E+6Z0zSVaJV6U+mbnee9Dsz7BcSutLWw89WIT3D+dtCST
a8WPi2JDUtN3SQ2/7H6H2mNlZU577DmMM4oZk6LqrHAx9h+f9qxlO3smE5HxGmB/vqIOQK1qvsSz
GUsTEQduOwDFBpoaDb2S3suVmb+lUo/UReqG6vCXb8jWNC/U3cyZ402acJaz1nSdxjtAp1dTTtNN
qeIrZ62pDI/O+rRYVp8sxdmgOodxZ/H1By4vsvOIV5XrtA4q0j1OWQSIN7G9X6rP9NVjkthDnTf5
CT2Pp0Kokz+WROjJNxZyyhrd+FwxWY8qY1nn5uRwYEyJjNT24MtfEUAAAQQQQAABBBBAAAFTQDyc
Edugrlb8rBc1dH8T957u28/tA+LQizP0Be5mk9tbfGhWof0TEdkvmInQxSqTvmihfKdPOL7haLSV
s6fu1IV839vt1FxHVnXtHVdpCX+wgNe8OEPPnHWcSfqCafHuF+WmFq/apIvKcttpucatZDFSxfrh
jExmzLZTrUZtdw1lbgux72+nGsru43Cfs9zMWqaU9ueT6eb6lrWAeqa+/0ktckK3Fkl6ScbdRP+5
13v0KntAbdX6CNnb//SKaUdCF/rlbo44Dvat5ppLvEFZdZiS4WIv1u0tmkqqMl7VhnsnteLNudtW
YC80vJVzuRXfr3L6lyYqw31yvj02ElqXdKe25zCBYmmTop9/P2naE8E2ciXFidUzyzNtK9risyyt
ar6UwxlVlWlllIfi9tPJ0VArbxHGmflbMvXDUZroVrIN2TyS5Y9FRRs89GAYaKpRhME6TXWAdq8m
TkOTMaO20MtmrckMj004vYYpHQ+TZoPCeHR+HU0psv2INxjMybrzamHWaDg2x/hp+PpzidXrdKs6
0kAZGY+3yY//PH5Y/nAS/hwxFnLGzOH8egw+HZuTw4ExxR+pzXk6f0QAAQQQQAABBBBAAAEEXAHx
cIb6tCrcTvls9XzULKB1lyz/8LW893g/ePDf7YXH4p6QuA1wy16ZsPj9zdDSYnkn+0FfObDvPwuX
D4nVpxuN+nX2yoaQxqFWQ/EgLX5HyKmL7+/HzvVe+q2W1pdXCy96iKCoFSsv+A4X8NCLM5Z2oOV1
s1i3I8QgaZeViyqp/1SUsfyDJdU31BntdFrjmpEZsVuI/6O38rgs4QlRXIX9/Tbz+pw9BqSablPO
p3ukbxHb+7rjJvVg9c7tSNLlcl/X1axJR7Zn3qlXaUPq6++/xI+aKAeK2lFPGUeKVtGRXXOJ1i6v
Sn1rU+5g5XyyXap271ftB0Q+Hc+q2ketjXIZqIzN7tCQFO5+k+3R66ybfNvzyXpPJzR/28ar06Y9
zRtKcrm6H+Bq/xOJHRnPuTvcrE4j1QOYSs2LkVPtjBck8oQoPxp6fc12nCQwf8unnojSYw8yxTNU
N3dR5pa5eYjoOGOOd8LMKlOn1ZGWSAe4Vun9h3qa/4yDAvXMqqquyDjQ9O7qS/aaiVI+YgeqY2TW
5zVLcyIqNpVS8fQ6mljkmQ+YA9WtVMTc0TD72DvCK5d98uTndyxJOSGnrAuNDzHrDSPLXLlRKbIm
FhtYB2dNWj/GvyOAAAIIIIAAAggggAACnYB0OCO167t+uF75qZ97q7/1d1c32B+5FVf5lKW/fTlX
2rKTV4C9tSB9TUtIY3t+DK2qSaWoNPq7yAWX/1VZeV0KLGbVrgsrb+VmlHcax2qI+uJZX30HCnjo
xRmNoLHqKXzh3akrca9ZukZrAjLuhHY6r3HNzIy4vuL/o9fq+yMY5f6ZvKhzuM/Re94t0os0thYX
Tnegb9kvUV+sou4tKAdMrDeAyJd0BdQ3Be0FZaHVv1yvspSgqiv3RKHWbyu98vZxfZhqXa65RGuW
l6VuNqoqd/4nS/zusJdYUqmZq4NlwGS9dqQV71txx4Yka/4Wao/6bMGd9tRL/vcbHZgz3OtQPggi
vZMtM0AfyFViJuPMtBXogbwps3ZtNi9/fE+3D8C+8eVHQ7evEarbex4Rz/yKzx3Bp574DfVmYvcq
Qpjm5iFvMOGsD2RFn6qG3YwHw0PzN7WKi5GmS1tpRgPtJX/JwITTb5Xx2aD1fH1WHU0ssrz00Zzj
CwZzvu78ipgwGg43MWksVjrnZ05+iixN6MmVGcXQAG0/KLWZ9WYOV9ZjH/Fi6gpWviF4ZbdWs/gb
AggggAACCCCAAAIIIBASkA5nZBaV5W/ILXf4/nv7ym+9QC6s++0rS9I6s7hQqKwe6ouKDwzv7yWZ
k4T2mvnEjomcHftJW86V/a+ZrNp1YawBGisg9oN7G6brp9vFM+EunrX0dzu4MwHifjlU2t/XYlBc
v99oJMFUVpVkU+3US9D7e1HR4kdTmbHbTtWPJPKlnr9Y8y5Uk3d77+9SnyOs43W38e4r/T3ft2wt
3uqamx1Ls8eONwpzFW0vibhz3I1979OrlFnvdvfVwhpYEpXd3ZRD5RaK11yiTVouS91PyOgLhNx3
VdjvaBRXxVMvZjP+L4qPtGL5jWX5IcmemwRGeX0eq3SB5hzmyJxB6LsmTXuO5EruE6wh1gxBtevM
TCPduVEznEi5Nfv/dt9yZDTUAis/So6kHopS7eycl2BVtPG5ZWQe4ilK79SKPRh6hfT+Ls6shONa
F8zJhQm8l3vv73LpzHN0RVV4dxf+PnCJcxLOu6M4FZJKqMwGzYmo/kLMPQUvg3Pm2OoApyV/TQfl
TcVmjIYHuqZwQ37m5MdfcvJC7BWWpH7yYK8LHZgrxp8HrcPC4e7K4zYnIu5PS4aWXPkQAggggAAC
CCCAAAIIINALCIczhIUkS255bOsXfn7+ZblV8Tdpbzy1CvvIirzF7p4q8R7NpNUg8ZUg4srA4+pE
GmJ+zbUE5SnYvpG6oydkdXhHNLA7GHqZyFa3/vK9HTdKXdgx4kaQHyHFQoZY4FQSZaw3x5yy9zna
Tt30EoG/hvGETsN/R4abcalOtVjtbubePcPirYEVOR1KN923OE1M/vOcbs1et7Xf4iKMV334Ox2E
vuVtdJtyV+vWlTkz2ddFH29fMN6T4nyJfrs0MMS3/NdcokFclnogoSWP8U9Wb8fffnbG3GQKnTjy
x78lq0OtODDna8PRbPhNkcLt0ciH0q6s4vpmfetWtv6qieiRbewiPkamaok+xy99r+1fY21otJlz
qkx4r0Bo1uj2sJlRODZKClnN1J45gE0tzkBDTsxD9Lu/4oRTeQOV692WshyTy1qXezXpmdBNciRi
z5y1DmU41pQCTXxoNihn+eQ6mlZk/+1D8maxUOqhulNa9vmjYb3Eozz22mUXVpyU+bw/vAXbvhls
qYcJtUu1k5iShhYpeaa1FEpXOdJM9lslegx/gaD0Ng9nhGbl5rMcf0QAAQQQQAABBBBAAAEEZAHh
cIazDqI9q25PLsWJjO5whvTsZ+9yJL79Uj4Gbnsg4v8TeLaT18YCNvFNG7No8uKDfHPxRkNZtTNv
PKFbF7pLQ3VIrbfynqoPFVCOgMAt97xqq6cbhZiGuea6L3zJ8Zv46lvX3LtayLXTqY2rP1yVy4xy
BCrRVzgrj2oH0SrOZem2cd182P1c8172WGQWi3xee4iDX5O0Mcq/S6/SF+Ge8/W7xvr5jMge3Nql
evVavEHm2ku0QlyT4Z/U/YTaRefAXGK5pO0r9BeSh+5pN6lC0ivTqUOSN0p6o3ygSSvHOKXNCo/i
J7V+JnP6tGcoV/bmrXjL+NR0Vx/K24QXZwTS7TOZGg21yIoNVdL2n5u8MH+zotS9X+arvIm5ZdEP
LlkIdUgN6CtNONN12k/ztcqoaMRZhvRVgqnzRrsrKGqzHcwTATaU4Ri7/+Ynd1iOT0T3Dl7+1oVw
5iqhVAwf5tNfoMj61zxiqkc7KG8qZhxHtPo0+8Un4cfe8PKIN46cOvkpDrl5QRToYeWsegUUpzTe
2NfmJjZzSAwxsRhuA6nphCwzEetYJ8bZDFbREUAAAQQQQAABBBBA4DSB/nBG7DmsX7xqFn4e/3N5
Gmr+Vj9U2Xv34l/Ff/QeDYvH48CTsJ2E8ZwW9zOLljk9IN0okg19H0Isn3VLY4nAXTpsg7sJGmV1
4eIC9i1Qi1t7tcS7ylzEyXxbs81wu25cKbvtdG7j2jZv5nYa8b7CWXnU+4cmickst2wJt+yzM5Zu
LDL31NwmFgcfTVqtCXkbxBgn36RXkfuZhWFbXAwMYOWN2nqyh9xi4N7XwK+5xNsR0MfdLsAGMizt
zHvL2KFtluIm1fpw4lhBl4+9C3AWjcda8T55s7cVvCFJrYZYe7QmvsqtjYp3KaS5xunTnqFcKS+P
W7zkW/pNQo+y3IzXm+XIr1Ep2lJq0jg2Gnp9jT0D30swlnokSr0NPeU9EF3Bqj5Hu2lbwaF5iNU8
X2nCGZt+aK95ic7J9z6zjB0plMdi5omz1rEMx9j9yYzbG4izQacTOqeOZhW5GH+1sDy3g1KCbfJo
mO2asrzu2OqNVNG2L95nrNV4A1MVD24BleUTORFPQ50FnF2Pa3bDo5JUkrHqGJg1WcMif0MAAQQQ
QAABBBBAAAEEBIH+cIb9+Nvfolk4qdfcq/8lL7HYCy/iX+VLQs+Hj2ft0eWgyHOau5DUPmeKX72S
cyg/iYspDmXVzrsRGdaFkZxUYbVcsBOICUdu22VruIBu3G8fiMVzvadnhO6PglTr4SjT8pVsp1Mb
V/elsGRmTn9xRvj95JNZwktQQ+mmI9NtYl1LLVYCj3VrQ0mbI/x79CpKP7MFZHZ0ftyv0fQXdLtk
rrlEq8DLUg/zbjnypxJ9ofbWKx9TjdxzX2j2vtA31pQmDUl6nxNrj0aTVm59dCrSXX/6tMetIKlX
HZjJ5OcM+pv1y3rp76ulFK+yiMmWh6HR0OtrwjPwodQjUeqfzQg8yFhf3lbmlruMsJsV6ZseNzj2
YDikOq1OizFTqAXDbS10f3angjuldGfOWocynJ5wavXn9gbhiWgZmGfU0bQi+y/OOLeD8qpCSD3S
p4kvyoo3sTTvUOQM9JlyxoZaTa4TcwuoPZJJySTH7fUWk+aK3VKFcvA5MiqJJRmqjpFZk/kgyh8R
QAABBBBAAAEEEEAAgV6gO5zh74O0NykXRtq1qeV/35/k+3Wr+53sp0vxr/IlkcfUcAgMJxH3SxRt
zba9BJB9Q3ifVZPQKplxYf7htg2UiXUxXMA+cLRbxeJZ+trWfY1KWGGVdocGgv1QOx1Iz2xrhzKj
dBoDDarJo7TGW32kTWI2S5VYuwjlfUnd69zszArt1CtdHHx60spgYhC8R68i9zNFb+HVibXmunYu
bpe8rWJuIXfNJVr1XZV6fvCu9yjcfHbL2WV3H0+9fL+OdzbDmWDJfWnV9RwZkvRwjbVHo0Ert7Ya
SKDxdJVw/rRnJFcXvTjDDZ6f+hGAknOj/uOZpuA9QnjDojwHUDaGhLlYoPris0f/iShXnJGG3KZg
zEPOm+MNqWrZuWZOfku9m5cMP7zE6/mKWetQdaRnfWP1pywtqKmfWUfTiqz3aulgHqo7pSomjoYj
XVOa1yu79Pd5GfNSj7dxdaEskER8ME2O24/sz+NaOULTaHtUij+aenUQAPZuwd8RQAABBBBAAAEE
EEAAAU+gO5whrD469yjWhpYHpu7F+Ld/ULbT7Acx8a/yJfFHUM+keOCU916MV5lvuXC/YZYo2pZh
8UFRLHhAo8+qXRdGZFgXDjzcVquNyr3nFzBedeWSRFvPsXguI8hLdwWUrnGjrIj1I+00oB1oVZMy
c/qLM1TVtglMZxEQi0Wobqs89ZsKzpJXfocs3oOlk/Y2BUPLdw3mG/QqQhd7L2mxPd6V3Kfoo9Tp
k6Ue55pLtE7kotTD7VkSUqcNVqMuDz4kpl5b+v4A4JVJip9pQ1JgYvAogR/FraJSLutGHkW507DB
nj7tGcqV6SVXX5449DM/QmJaieJVlsprADA+OUnP34ZSPxil4eKMNGTl5tI8xM7Hy0w403W6bzgq
/avS//94VLMMrVsbihkN220rE2atQxl2MpZ4MHQ+KiZkpH5eHc0rsjoipoN5qO68idjxF2eMdE1p
Xq/s0g3nZcxLPdyN6wsOgSS88uy50HgHJhv7TQeWLxI9gzIqiSwBq75G3N41VYl8GAEEEEAAAQQQ
QAABBBCQBdrDGQMPMOsl3/+W/694cN//pV4Q6Z/d5C+AZg7AJx7p3GhQ7uUmEX8Qlrcf7QTkB0V5
92Akq+Y1VmQMbmA4y433OFJvPbuA+2O++13kW7615O1sSaWJbjyW62Fu4QXaQ+10JEGrlR3KTKLt
pPLt7I4KjTt1e8vDvFG/vjWSbr5vicWy/As9idf4SN1aLLOR17pv6uXo9KK9itDH3nJqtnx3xM6G
rTyGxcYG8d3tYn+aGCmN3vZeudMyLL0EQGy00oLz/YPRdindwK3ILSvb1aHDWQOteNqQZN0o1B71
PlNpwqGO1P0BgL3JXTDtcYMm2YKVmYybjGTtXiQ1Pq13jVeZsyXS5MrNpDvfzz6PHJ2LHYvSeHHS
DTmUsVC3U77PYODB8KXrdOuA+z3qfWJrDQdnlE45R5LsPJTYGslwbA7nHy/0htX0+H9eHc0r8nUP
mPHexD6sp86p5aOX6a7JnV7lnyPEedzEjI20Gq06lHu5SSQm2tq9QoOCMqcSu8psMwlloByV5Fh0
rQT7kWsyLYrPIoAAAggggAACCCCAAAI3gZmHM/7evuTb/DDD49nm6+/f+//tN9PsVVjxr9ol3uaG
sXjQxoKWhPgwv1+8r4r7K06poi1JiA+KSrEHsjpQF4+MRTYwQiceVsol89txH+nXPuYWcK+62Iav
F4TON+6Er8LLK95Fvqyvz0c6s21J8kg71dblE42rCpnvI5mpkEcaVM3mNF9pmWlan2P7dQ1/IN3c
ftddxsqUgjVSC1K3ZhVQbhN+E3j9XmX/seu1sd/LKvyejfBDJ85vkJf9mhUL4mqq/WqDeZdolXhN
hveIN7cf9/jrBjWvXTYDuXjsxtv63LRjI9VAK7ZLkRiS7D4n0h7VVq3kMZSi+ONh5TymkDVuOGva
c81Mxun/FeiBvKnTwUyVWXsi3X28VpeYnAyMkgOp26kM3FCpvXxDTs5DzEH3VSac+TrNu+0QW69m
nSmYV8XFrwqIE4BJs9aBDOfZs519/fBZ9+pm6mfV0QlFzr2ZUTzSMVB3A/Ow5Gg40sQGePOPMDMz
doH8yADtjBapkJvJpS5zZUclZ6aRWMQYmzX5z6J8AgEEEEAAAQQQQAABBBCoBNrDGYl1zO0+yzVf
X/3ZjGWxYPmTsJtgH0wX/6peouwRNUs4kVMCQ0nsmybe9kq58XnwG+b6rpmlIWd1oC4eMWAEzdjD
7XLV1/f3PaTkXah8AdXMFB7axk3Tb2hWtqGYAUOvzNfRL2vub1weaaf7trHxQtugXRkz8zqNXF8h
DQPbko4UcHtViC1WCtH9fn6fY35WiJB8XzcQmXqmtMBMvNNkqwG5VaqLjkUtBXen15Rev1cpTgcs
Y8iNYWtxe9HFN1QYw2t7ZlJdUlXC/Ifwmku8XYFuaJ2a4dJfO1+5Jyh9ItLm5TtsV1oHO0eC/0Ar
FrKSGpJCfY49ynsB0WTRTtEcxdKTomnTnktmMo6M5pzPW/aF/GLG9NYgZSg/GnrFlUdsZ6gKT40O
RGmkg9kLl55bZuch5uP0kvrQHG/ihDPUC9WDZ9ptd1hnGX//Pb6rkH94yVXxRbPWfBMbYFejKT8b
NFM/q44mFlm71UgS+bob6B6zo+FAExso+xMnP+XKyNHHQ+Og18AArdTuWMhdUY/ZUUlb6Mk3hMFZ
E6vMCCCAAAIIIIAAAggggEBOoDmcYZ+Ddx7qpJWocjdBOLNg792LfzWyqKxTafsJqpSRRHEvabf8
vhrnb8UqRypsDeVFp+rjYzarA3XxELSCZuzhtooaFTRbwOJLZnsVbaU2D4K0oaIVeaQGi8KWkVNG
bRNVQ6203HjsV4yddlrWsvwzC+oy9EWdhmiSg3qIr3uF/Y8ziY17ep8jv3uoSzub7khkFg2mXF1s
IrMMWrPHjr9NplrXLNu/kbQ/7r18r1J1pvdtnfsGzwO4LHozwsixUFm1Y5LU6ZQ+whh2wSVbEvrr
4svmMT/DNXO7pF6Jem+R93qL8HmZZpC931fZ8pPbQCA8mjF2fEgSX2akzIhi7TG1k+D0clX1huZv
xg0nTnsumMm4Mlr/mc5b8oX8SsbEoFUbfHY01Ao7NEoqw2TVY/sv+SqzNKs4Zc6yc8vgPMQadq0H
r+smnCN1mu8Ad4d6Gqk9DU6r4qtmrdkMj7DrwVSE0h7I2Yno2XU0scja4DKWRLbuBrrH9GiYb2JD
Zb9y8mMscDUztldekmpL4bBfUo/qwC/2C/qvC2YbwvCsyX8a5RMIIIAAAggggAACCCCAQCHQHM4w
jsHrbMoTz/0C9anqcTt7717MTbOkL3/jadnBaP9P5NhEk+luI6TdNG/S8H/QpCq4+BXozNf2rMfH
XFZjdSHkzQia0Ydbc2OxXWNTKlvYwFJBfkqVOkaiBaF9E03DqadH+aRfQslt1B1op1XMHmtcS+0d
yIwYb/F/dFYef1qkXh9Sy3RqL9jnNNvCgnF7o1y6Y5FZbTCVeVrODbS7xGYtZLo1oRfekh/7qn0x
2oj75i/Qqzyy0PYtXSgYXbDcNO33nffX+C+N6K6Zc8kW0sl2Nif1vmdSRxbrAKZbf/LpitAgkOvw
t5DW7q204rapaAz2z+3YP4tUzv6c9ih12MpQGhjEp02Kpk57crkyBgttJhOQ0QbGZN60WY5TZV0b
NpPtOojcaOgVNTlUqcPk0nKa2wXqYk5xBhpyfh5yyoPhxAnn2Mwn1B2LYVJdaTwOzqnipVVdMWvN
ZXiMXQ0mdVSVZ4NG59yNPNPqaGKR5z5guo06+JBiFHBgNMw2sUHe8yc/v2NJaizkBoaYoXr0gsX9
isDSs8zsxPSRj78ggAACCCCAAAIIIIAAAjmB+nCG9fWHyBpc/4y/P/FZLwFPfae6ebzSv0Vc7ykE
T01Iq4LRJBKv2ZceUB19+Zk29qQb2ICzz1EYf7WyHViGFuPK2rqvLtB3wuT6lp7Nq2+mB8NEDsKh
GuwWC/fKumVGIrSbgNZSd6tsO13vqK+Q/mQ099+BzIhBFf9HZ2PmbiMu4agrmLNYjG1dMSwT6Q5H
pnhW4CYRBh9PWqyG7Dmqorpfvldx25n+WqZELDipWAv1SiqTLtnuHnorxdZNTkpdDhThYEJgiMgs
JT/SDR7pSL0ywymR2oqlOdDqoAxJ4d6g7H3D7bHvsuWJhT2H8VtXV7MXTnvOncnEZPyZQ9schLag
zfmUfzemiBqJfXQifHCsL+2BoSrR/8bqInFDe+al9EXa3NLuiwJ9356bA3O85SYTEIbrNO/W9+Q2
14TSrVOka2atiQwPs+vBrDy7jQw9ZUGm1dHUIk9+wDQmGPEmPX00TDWxcV4pbGdOftrnlOh6UVy+
eyKNJpFYklJKERmtLqnH+NPx4JpYXx2RsueWPfg0AggggAACCCCAAAIIICAK1Icz1qeazFPjz22t
hyHzlrGnqD43xcNg5FvByeK0exOhjdHgd0+aRcfq1vajoPzXyONj+1QrZ9WuC+OvRg1Hcic3zK2G
I66xAkp7M7v/eo9Ieo87CUE4UoNb+esVji0fyj29JiCxDrfT6mbNSsxI4yr5rC/Jmy3Pbzup8Fs+
vN21CqpAVExi6XdpHeBYuociU/pJDfGG8X/cIsqto7J8j2pwVmmNkf4NepU9991ipB+EsVjQW7Of
QtPxxX7Dq1laVlMJVU9VxtMy3JXzvuWb7OuU3SQxQJX17WKnOZl4n0rfkram5JyOfeTCGJIGGn4l
HKvHvm00+bbnMJVIbM5w9bQnlquBmUxCRus/g3nTOnTl31P9v9/dDPSARXmPjZLdDprTqgJBf6w4
e8lyc8v7dd3wM9D/vMKE81CdDriVURCo4XZLNKn8hFlrLCYPsetTuOBsMNGrzKuj2UWe/YBZP7Me
mM8IZOOLAOEmdoz3xMmPthow+6nZe+QPDtDG85G0phGdOVxTj5FRyW384YYQLbtFyt8QQAABBBBA
AAEEEEAAgYBA87MmgSv4CAKnC0Ser0/PBAkggMDrCQwvmb1br3IraHKz5vVqixwhoAi8W3ukIhFA
AAEEXkhgeDb4QmUgKwgggAACCCCAAAIIIIAAAggg8LECHM742Kp/4YIb38N54VyTNQQQOF1guHMY
vvD0IskJ3PeuI1/sfFL+SBaBIwLv1h6PlJVrEUAAAQQmCzCITAbldggggAACCCCAAAIIIIAAAggg
cKkAhzMu5SaxgMD4zxYEbs5HEEDgtQWsb9RvnUP2lRLv16vw4ozXDlNyd0Tg/drjkdJyLQIIIIBA
WuCU2WA6F1yAAAIIIIAAAggggAACCCCAAAIInCHA4YwzVLnnuMD226XZ3dfxJLkSAQReR2D/YeGm
Dyh+1zj5Rgl6ldepXnKCAO2RGEAAAQQQcAROmA1ijgACCCCAAAIIIIAAAggggAACCLyIAIczXqQi
Pjwb+wrcn8d/HM348Iig+J8r0PUGS6ew/Z/g0Qx6lc+NIUr+egK0x9erE3KEAAIIvK7ArNng65aQ
nCGAAAIIIIAAAggggAACCCCAwMcKcDjjY6v+pQpefCf+tgMb3Hx9qSKQGQQQmCRgLcjHz23Rq0yq
Dm6DwAQB2uMERG6BAAIIfJDAnNngB4FRVAQQQAABBBBAAAEEEEAAAQQQeBcBDme8S0397nwW2zbx
vdffTULpEPhsgX5NPtk30Kt8dgBR+tcSoD2+Vn2QGwQQQOAtBA7PBt+ilGQSAQQQQAABBBBAAAEE
EEAAAQQ+S4DDGZ9V35QWAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBC4W4HDG
xeAkhwACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAKfJcDhjM+qb0qLAAIIIIAA
AggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAhcLcDjjYnCSQwABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQQQQAABBBBAAIHPEuBwxmfVN6VFAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAgYsFOJxxMTjJIYAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggMBn
CXA447Pqm9IigAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAwMUCHM64GJzkEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQOCzBDic8Vn1TWkRQAABBBBAAAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBA4GIBDmdcDE5yCCCAQFrgv79ff/58/0tfxwUIIIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIIIPASAp9xOOPf95/qv7E9zvUu1dXiP75E1ZKJAwL/vsdi
5ECSXPpUgfNrfKgXai76+vufp3Q/xVH+F4rk+qqzLtHyfk3qA6k8IcNbLt1KCHyyjbmfuLBDqIue
OpbGzwc9d6DsW/dz8/O/1TnQoL0Gf//7+d1XKBuzPvRy9ZUsWDK69mYajYf8FckSnPLxt4lStxtM
D7CneIo3PdK1PMJqH3nUe+XjT7tiiLrN53W6E1LKVdCQzy2XqQubeYF7bXxy0k4g5P9d9Xtu6us9
3FzstVXMhOyrtg8mbq4Gxfxm0g0QO5ab4TU39T0C4ZibRoarrwiFNUvha6XCbtmMjaJ7WrHPl7Us
5DNwk3Dp6l5Y73OGb6hW+uFw6jMbbnplnzUYzF3qfeg69ZQi9R6n8qPFhTGcz9zRK4R+RBgOAg3p
aEZ+zfW5B41AV9/LDF00EzjzMCgssuWIZmZceCp/YmYio8isXneqonOzlyQdB3j34jw3/5m+YryO
Eleme895jwxaLlMzHLdHSN1tGe1/2wj/3JhPRCMffSOBDzickX46U6pP7GXTXe8bxcaHZvVRpb9t
+PjQygwV+4oaH+iF1JUMPTYHLvkRkq8yZ2UDl2hVMXCray55UoblFXQpM+YnnTmzVrtnLZ89b6CU
W/fz8vOoyYnT+Su6r1A/OudDr1lfubIlo6tsq+7D8C0n6Qty2T/j0+8VpbFu8LHU8GpTxfGu5XFl
GYHavdIBqF8wRL3cLtRYzojmQ/fMVdCQTznExE5H1GEcS9SK/NgdtrxVt0pdG2p+1WzIvGL7ZOi+
Thyc0Ey6kC+wnOaw5ab+nBmOI9PIVPUtEbBmKXXtkYMqxRiaq+kRkTVMMqUL5Wr4hmqlHw6ntknE
m179KDoWzEXqto16+wzpGaN/4nTMcAwfGr7GL07s44Sifzwnv+XK5INGbuaxIA1dNAk49zDYN92f
MEoSTcr40jTbMH5SZpxCTe/zZyFG7vOapJGci5959+I8L/+5vmK4gpIXZnvPiY8MWk4zMxx/HM7c
beBQfZL7KR9/Xsw/pbgkeo3Arz+cIT0PjC3pib1stuu9plZJZViACh2me9MLr6jxfC/kTHmkLmxs
1dBISJuYDVwyME+cmPo7ZVhZQRcAjU+GVsEGYmj8++JXtDIpyLR0n5WfJY/zpvNPLsjsTv9F6ytZ
zGSllP2T/zRcn6aLfD6Z+xM+ngQ5IQepW4b6zxddaRjuWh51VI0K6r3mRewg9RufzshV0KBP+sUZ
VcUnElUephN3uLej8jbZa/1f+2smgNYCwPbRsVWCupuZ30yM8wjuTvGam/CLM0IV0TmFrmpODKVf
nKF0vuGd7SKTiYqOFU0dkmOXN8dVrIFr+IZqH1Td0Z5byOHUZjfR9Jo+60jqoR0DMYEUaTtepiYZ
2odPjuEpeRy7SahWtp4h0S7HsvMLrsrNq3Mzj+mPqmnv3MOgssiWI0rnUb4gl/NJiY7eZnqfP5qR
oeueUr9DOQ1d9O7FeVb+X7TFpbvceY8M/vwicGzfHYSz86VX/DpLqGEaH3pWzB/NN9e/tMBvP5yx
dR2H17DFXjbd9b50MJC56W+bh/TVBa5owuleqJihPWZH9zt8/y1+saSdNZWTJOm7r/I3fLuEbont
/yj1mgOXaDEwcKtrLnlihmNLnmU1mev17R+r9bF+5u2l7lyuNvYrWpmUuJbus/Kz5nHadP7ZBZnc
vb9qfeWKmayU+vnWfRyu33TkfzyX91M+nQQ5JQ+Zm07doc0kPOGzY13LUkPNcKHda2LEjlIv171F
+Ne1mqugUZ+tlxh58j0+DfDuYAV6tMj7ZMSOgn36/fX4xT+DJD1VtwpyQjOxXpzhnM7YchN8cUaZ
++Q0skOJhsPxqg/doShavAOp5r5dENV1LUdYKG+ZIWD4hmofVBfS6juUcKqzn2l61ZOnt4ZvpW6X
ofyrUL5opGaqKfvZULUOxXA2J1M/H+pbi2KNjFxTM/zyN0vOq3Mzj6X0QxdNkcs9DCrRlSSaknF1
GfkpmXGLNLnPd9Ob+oHXJB0u4rsX51n5z/UVw9WTvjDbe058ZNDyGppdpAtaXvAKc6hDBchd/KyY
z+WST7+bwG8/nDGvmxB72WzX+27h8XH5pUI/rcqvqPFkLyQ8Zt7u8FirUKZW2z9n1pv0i9QcD1zi
ThK7LM9M/c0yHFryvIuqn3TXwPYHAO2tm+Z3UUdWBZMtYFonpLXuK1q9UYh50/knF2RaRT19JXBm
SZKVEl8cq9q9t884s0QH75UEOZja4cvdHvRwCufdYKxreVRQ+CXM8yJ2mHq9ML65ep556s65Cjrs
479Wos99aBJiTgNCd1DYEkUOJVPMmP1+aOrK5fxmoh7E/fLPnchzMDUc3WrQp5FdxYbqqZzT+vuy
6i2tJ6FHxkYmsNWiudrnFJ8SSuCKpvqRHFd9a7fSh8OpTifV9PaKOZJ6WVFqFO0Nc+gJKFtP6c+f
E8PpbEy+INq3Rj83OXtveDt/PCsLlZt5LFcOXTSHMle6Zy0xSGXN5XyO1vBdthoenUIMpzzjwleq
9hnlefN7PKs6fstC37xHBi2Qps9Ch2f7bx7qZB+BUwV++eGM6KKAbyzOUZ84cfVzzCfyAlRo3uy9
r7iixpO9kPDx+z8tS4Li3ewk5JmrOUubdsnQHHFa6gNlfG6G4w832icj0aaxBFPfF0CDW2ORPJ3R
j2it+4pWb5Vn2uLNswsyudJ+R3GSpdg+/v19/2K33aaW0Pn+vm+nj2y9Tq6zwO2SIIE7nvuRd94d
GOla1pi6v6Sr+E+718SIHadervS3cc+NlfTdcxU07BMcysXsB681pgHBO5ipR/q2QM9SCrr2U1cu
T2gm6oszvv7+fQwI/m9qxF6cEZmyhbWi4ZCJdrXqnVztURsJsEeEFtd4/c2+uq6/5CQ4a3a7lgxX
qGMvXrczGE5iMjdot+n9XLmHyWjq4co9+ATkVs3BD5wQwwdzdPzycG9x6KVPx/P5PncIjH6R6ZxZ
4Ei7PUcsV7rIeHVOPvu75nJ+Va60dI73uk8swStV+xMZXiXpZ1XHb1nom/jI4DX3+Ow3G1zR2X72
vnwegU8S+JDDGd4TtV/l4hz1eRNXP8N8YkCACh1Ae+tLrqjx5LOa8PFbLrc+TJr72MUwOy95kibm
2V4MTBVz4FbXXDI2p53FFV/AUh+DYtUghkQ89eSy8LOm61qzuKLVG93ivEfYJxdkdtf/O4qTLMXW
OJbDGZF3iW+/cDVrn2d2TVb3S4KcmpfAzePdYOBmF38k1vtXmVov6UJJ7abmRewBajXbF4snk8tV
0LDPkSEmnqg2Dbgm9R95V7Mui/vx5LzGrvsTmon6Y4b79rc2fMhzsDOmkZ1KNBzigWdW/XYb4dFm
/1ticXqLitA1bqM4vhr1EE5xlZXiVnpxmiIXTmIq91u4Ta9+F+CCmEw9UVGyQTRSk51+/uPTYzif
hdlXxG0DsTI7c295v9y8Ou5fYDyxKsZKN6tzPRIQuZwfSWnCtWVYjPW6EzIxfItnrSwNZ/h3X/is
6vgtC33zHhm0OBueNIYDd2icCd+dDyLwIQLd4Qx3MqY3veK51/wOh5OE1M+2C3HuQ1ibl3uG1g3O
Axmopn6ulRxEbd6CP5hbFaG/c5uZMpUmhSoDxus318t2bfPnepc8NcVzv8mzZ65Mx1oGiQbajlTd
Wfp+UfeBh3ZfNdG0jYj9e3sNwsB3nNo615OIVFP9y/VKftwIb3qDiUEoPCA6LWBb/wlFVLjG3VmO
ni2zF1JHmB79/i9mN2FOisVa9KbRQjc8cIlWyIFbXXPJczOs/WiNkCt1GyE2NzZjIrAG7dVGleM2
S2Xbk8I62+30PHbr7u5fN1Rnwzs6wOlTSLl8bX9e/IRRPxaFu69obu0pVipvgeowf8N+6VD38deL
h9FB2RQeKMXjEi+3zY2LxvGoVGN5cfns199/9hziGhA/tMJRGpqNFLb91HQ7ruJOON3eLdaDdvHh
1HvkuSYVkOK0PL8urC/IagWaGLGD1Hd7ZyXZfoJ3nwjqthyaUrYpTqigYZ/U+NxkXJ1a6F1i0+gu
Sn1vLtqQ3RbFax5HMq49n/30N0c7drVrqfJrFk6JJPXOsciLjXVR1UTg2VWv3agYseKbePtFwZOQ
DrU7/tg91/7XWBUJdzsvnIrEsk2vfHHGz2GOgWDOVZSIEI3UaBUd+NyxGM7OAJXmEJm07MNc6J1v
1vx24TJqwZ9zygO3mMko0laL0QvOflw69KDhHXiqnkGcAXO0Og4827bttq2TzFJ/e63TwbuTRvt5
xxssL4ouZbZ3D4t8r3vgqWcNxLLcg1sV65haVoA/Wo9Gb7THW6XFXC3/WOWyw/RCxlptGC1d6iG0
DqZ2RuJVh1s8d4qTbHG1yZMW+vyFwQmPDFoiruiBKcvj0vQcKhqoZc7crlgeIg90NR6oNKxGe/TD
5tzgFwr0b85Yot6bwdV/V7pI+bXL9nzPemaq5g/a9nnVQezr/MV++HgG6udqb6WnjxdlIqksyA+r
Csms45CUA2WnRNAuj7gIrUHNsBRNVaDJMMLgqSahDLSqeBWb5gRfmmLVcaXnU4zYoSamLozmqylm
UrajaG9QtoiDQVgP9E07lhtMJqJCNa51+NGGqXzOWxds+5V7Xp2L9BnRVtLqDu4Mqu/cBi7xANVC
TUn97TKc+C6cXKv12OfFWVs7xj27ikzN8cuKkNpEl9GjPaTTuqvokhupPJykBjh9uihPHKZ3X6nc
lqnXF94oMnmryh0bazL1VbNGu+J+NAtPOX4ujZViyVlyWlg0juX/VdcM9lrQ07gGJBZaoUF2xPbM
adXA6kI9V5Grz32uUYJMvJtodvuku9ql9flSMlqQTYxYd4g2nrmT7Wy903C8PaOCRn0yQ/n8acDl
qSsznb2q28df7wJ/Yd9fDDqhmbTZamZhVotQIikwkGSnkcNzxUzYOJ/dxqcy80XLT5Qpk6ul6HIv
PNqW1UgbvqFW6cfDactruunVL874uU86mAcq6kjX53cABz8xGsMjM0DlwJA7aemfFNRCx58Yt5vW
3V1szllPwvUVY/VuStc/ovroZV7tQUMo4Mh8crg6PJDkw6CSj4e93omo00BxaIhNGpM5L5rKNdEl
tc2jfb49C7c6kHufI5ANbFXcV0clQ3eBMb2wnOjxbtxSSNwztf6hao095vIvbjmC+3HzNmLEfr6c
kYSqwymeO8PJtDi5jT1hoc9fGPyppiW3h9aCpIRc0oOzlj3iI89QqUHkkbNYVyysxR3sajQXrQPM
TjCOunP9bxPoD2eIg8ZWbCkS9YmF+IIAZ/1QSmHPU9vAklvy948fyED1DBN/2vB7lX7zdVxVvvI2
LKv9mnJ8c/nlz24CI+4UOxnW1pV+/l3vbpu5hNUvi9vXTp62ciifK/I8WB3f/4SIzTcxa1a9/dpw
rJrEOWx5aWWey2rRso4G4a3EZ0aUX+PeYNxzt92dFq7urGXL3OOTt//prib2q2ENofgUoN6279zc
WV28Pxy41TWXeHV+NldT8cZsw/pkFdtusO2JxFNPvlJ5G9O/H7+L7vZUuW6nZ7Jbt9tNye8vSHZH
euXJE5CizP6A6HdfydzaU6xU3kJrTvUxz3B91ajDg3J8yhEYh5q+Od4N1s+a+0TI2TksZkztJ68B
CYeWH6XeGK/Ynjit6jaJoo98Bx4rMnNga3Vi/Yq+85otoYFKo4RWoDLAl/9/NGKTbaWuCnuQUKot
+kRQP7I9q4KGfTJDeSuVuXbLoHjyd+QdAfHU96pUwm/7wP53B9SdZUY7g3IR8XjHrnYtTX6N0il/
sjqt0WlkZxRVnVj10jFnN17Eyh1qgtuE9/781iz/JGbkZrgN5ex2x/PCactvvul1ZyKzwSx3RYkG
uzxp32ems+ool3zzaaFEbgwPzgCdHrTksJ8U9AJHewF3IUN6ftRXF9tnqWoJOPQwWsZF7IKzH5eG
OhS11etPm/p8MvwIMDCPyj0Mark3l/r1IouP/tFJYy7nfWd5bnTJTfPgFOLAU8++Ay18hVatIfnA
1nd0ZemOMBy9Ymeid3lWy5KenARM+yFHHCOHS5dbFZGKvc17otVhF0+aRtXJhlucjtIPfUlAv/7d
xXthiijFRz+ZNNaC5GwNTxrDM5noOJ9uhpGmq6wXzehqFAC5A0wOMGFcPvhBAvrhDKlDMQaP7pGm
6OHkU33y848Y68u9vtof5f7vv9vpWPU/ZeJgd/lWBuqmH++F2glBUfSiFcv/Wj8o+qrLsLjfbMvm
olc8eMrPs83YJKxs9U+vZWck7gJ328tNnSrFVO4lPSz2mSoKUgebjK4vWlRdbXkrrzrkiLUmtM5k
t4j1kWqqBsSISS6r7dzsQBBWE8XTIiqu3U6f2hUcNRK8s2Bi71UE3PIK+8hqkTGpbC7fPqnetpMZ
uETrmAdudc0lz81w5sSD3HOrj/uxp4PMgOZkoJJsZqp7zKmjRq7b0apNu8tWzq/H70tVS7FKpOUH
OH1eIk9ABgZElSmfW3uKNZC3eg0kMtaoPaUcl4cH5VC9p0fM7Fsdq8It/0Psldc6UN8+ew1IPrSM
xpydoZlROqfTeMaLM9aZ+XhHtMIsPVqsv3feYqw9J82L2GxbqbvU9ORtuC2v0/hQjyE3kLEKyozI
JU5mdO7GqUyiYkLnp17PdJVJrDiSm4XbMh6ZbXurRCc0E20Lcu83tDartTT18+JKaLRbkdupu+Md
CrxQ1f9koA3C/bpUKcZCQmINlc4LqvLvwzfUKr0vazac1uzlm550JjKX+jDHbnqo48pUXfCz2Rge
ngGOvDhjcDHW6QXEZpqfc4YeZ5onvyKVqqMdVk1NG4YnJ/73dh7RprSmgenK4eqIzKPch/f7mzHW
/5RPy31COYS4S/H5ehnLeXoDJRldSpdztM+3JxAixXbJ9+OrOgdWiaumWd5JDdH6bPvQwvI2f3C2
n+Qwax4ExByUM5T181W4N6vRxecvaZvegtty2IaFvuW7aNHHiYmPDN5SdmoSHJyu3D8WnTDnA3Vg
RXFiV5MC3Zt+bKcyA8xnP0WgP5xh/JRvP9XR5rP1dLCaPjpLatJgr86OnVqSJw4DGZAntva8RFmg
EJ5NuifZA6rtl/fLWbnwjQBPu+vC96zJ8xppGBJnyFWd6ocXpb6t/7Q4INijhLV80hTiaHX0JvqE
K77CUAJGq8kZOQ0TYSw3ewPtLI70tRSxHYnLO+3aj/iSD6kJaM+mmSZcTjzEr9ZsWc4cRgvOcyNz
qeZJZT2OLlwaWIRqa3fgErdoeqEmpP52GS7nnu5s3mwe1VpM8a0E567RaXU9Bw9EZvGU2mXBHlBC
3Y46CwgsRwkrW5O6I31uIk9ABgZE9aWtBztPo5J+oik2WHtPaYnxVyzm0UH5+lLIAVE3uuV/SW1q
AbvXjRRBF4HkQ0t/tXB6huY8CMyYVtmvzey+2tavPw18BzUxY9kFunSKXtbdBl1i0Zz+aM9J0yJ2
mFpfmrQfCI/FW6zHmFpByoxO/oKlcmhc/nD1r8rJOXcSUs6IhfXkQMr9I2mqyOLXXJvoaKbqZgzE
n7789aETmona5Is/KEVQiq218b14Za/Srbn7CuX6gzdTnFb1Te926w2NZmkXYiwkhI41VbpU04u0
s9A6ilDWXDgdaHrimchM6mMVVVe+EOwGbqSSYi1E/1Q53fJi+MAMcHTSkiDwn1jrFlLcOj/ntCeK
eq8ndJMHVBPzupmPS/aDRl3VI9OVg9URm0e5z7Z16CkzWemflRWHetFE2FcWY/1dHmPdNZLBKYQz
gbD9+8UE4xnGXpYRFmSVej4YveEezwizopT+UvF6HyFhgeRg6YJtU4umomAs9Lmv6O8UJz4yuGvv
2UljeCITnIflA3VgiKymAU1sZ7saL+ZDM+zw0ZUwNx/8vQLC4QxtNLAmrsrkfrtEGPrlUU4c7Pd2
5D3fVxWlTBzMFUn525vinfyFDWGJIzLDk9TKglmqzWHw22XG/FtcsnG0hc7Xk5D+bj/zmODC0rPw
eXuQkEquFONYdYgRm2hi7qxa+cULsfh5E/mn8cqokh6i+6lyMggvjqhgtzISCUMvzmhXF9eZlP5k
UE1DjC2kx539CVTXPgYu8aYzuvqM1N8uw6F6WU390pX1XAZE/ESMOesRhw57WUh5WbBYlBk9pNaJ
5Ieeke7IX3BV9sREpswMZCS39qCfF/P6mMT4ax9FSMw8n1sKJR6a6F/+p7q08fiDMIUdGZvyICOh
pV6Tn404U9MZnYb7sk1tW/vAY4UwaS9mWOK2txT2xhxLDj47y9pfZ0VsEcdjK0VeMLaFzsdbvoFs
ETq1grI+9mywvZuyahWaFFv9eCTX5iFu9wZGFtWKMII+MafxV4ROaCbtqCDmV45yJfbtHqCdbMam
kS2NNzJtny+XS49UfX/D4qXn4c2Vx10GQ0JwzZQulMnhG2qVfjyc6qfL7hivGW1HU4+FcuyZxo0+
61SY3zVkPrHVshvDXjuT/j4waXHmYFrZxockb5iX/h58nOnbWXezEdX8tGHm45LzoFGVeWC6MlId
AyBaMvK/K582QyOyFD+zXqTMXBVdao9zbq8rVksxbqkTmv4bIGJG97gyK7OYJo5E71CPt/fbxS+b
rdWg5FvMnJpjQWSkdANt0+/jo9WxT7BCc1slYa+viD9fjwBmF/r8zy/xOrwWZMwvhieN0TmL16PV
02rtAc7sOOSLpA5iYlfjxXyVKbGzipU86sznfr+AcDhDeXWG0F7MAah6tg0fzhCfFvx0xIpSnjzs
BxIzA+LENvAkbS8LNpn3S9tXhTVLMe5nT56UgnVC7mOxOUcWu1p7hUk5jxBureUIJZwtkM9k699G
HGgZSw6asntTg6qAXpi4tVJzKSbb00kgq/OC0M37pIhy00mJy0dlc2n0MdxOp6RWqVWe8lhhzB3W
9NsGOHCJN50xvt47IfW3y3BqJThQuo1fmo9LQZS5Z7GF6Q9/3uhnDbmBbkft9rWWt4GYQ487MBTp
pjpuxTmWqyrLcgHdDsfsPM3HavkNul79dhWUGn/Fowje4Cd2xQPC1pQiWQr5Vm0wLP+7jcx6vjAw
4ZgDcii0/I5iJ3JstaFjwrTK+N3dfstGOjchF9Pq4+LHa52w91uFOJ+JL3bcLp8VseWtApthQiZT
va79cODFW/QhZW4FFeumrpC8OOReZry6ItRepQJnst2eRIxe62TOeBQxoiY9kBlBdUYzaQst51f6
VyUuM1O+6DRSH+3DdWZHbSgu65XQ7YaJa4+spQqu0cAOb/sP31Ct9MPhVC35Gd9ljvamP/cLB/OM
wSBDqpw0D69BhT/YZcpZlhtapxqZtOTaktR9KO28Do9Dc04FY8+MdwLRn1JZk/FEoBvxcMqDxj29
genKseqIg2jppJ55hQ9PHOOT9WJkZqjNTnkYP9rn2+Eg/XXeKrF3ckZYOTgWvYkezwkzueGZsd02
HaPfUfM5adlc66u8tiWWb/nHz1noU3v6iY8M7tp74MHQGxrlNPwBsxh3UoGaX4ub2dV4Ma9sGkZ/
cyw8HeSDnyMgHc5YY9o7iuD1xcJKnveqH/GWmSWEouaUxzX7Kc7MgLgAFujDAlD9urRx3x7E6BIt
PXPypPWc7UXxxy7hBKvceWl5bpIK0HvP2+6h2kDd9bl1IzbaxPSOyE1Cn4cmTIom7PUG5qNeLgiv
iqjcgs5QJBx5cUaxPPj99+/XPp9qWuZen0KDkHoGvyRdfQ1c4k1n9NY7I/W3y7CyEmlPgQM94HKD
tk31V/piwuAamHy6rUzsqeb1kNpTrT30CMdJ/QeaUHXIzlvMi/cQ+08Z9ljnKQ76A3lbQyUx1mhx
YqxCpOZIzy2F0hu2wSCGffuPfYMJtF1jlhINuUOhpdXVQISoa5nHOw1/wU+pyAOPFeE5sFfLbk9b
5d35tJbYpIgt9wZC3Wbv7nHINTUSb9kG4sV6sMBjBUxNJA6qin3qcLbtnG+9j69nfVR/FLEHiOSa
0AXNRCtJ76+VLF9T/jSydQqrGplJVH2dfN3YE5sr7fzFj7kyYaEoeWon4IZvqF14PJzKPl0a1KxV
gKOp58a+WU9VyU5h6OOxGA6Egz4DlKNbvKVVh0bxvFFX+QWQzHvMpNVFbw1zeaRTW/ch1fC8bnfz
mNzlSrUKpJJ4pTOevFLPwgNPXomHwWKN0t/h9YrszO61Ygfq5QUeY/vCndvritjbWN63Tatv6Z95
vb2kYja5JnXoATb6W5GRJxsxvJWYFx9qpZdnHitd9BlH7+TdIViqQ/nV3OIntZS9vuK1Fvp0v4mP
DFoig91ffN4Smu0PBWo7Iw90xRO7mmzsNUXMPT7EufnkbxaQDmesHX8xekrDZmSa3n/G7HblW6Z6
6r2ylMtmZSCeK3fQKgNsSDUyqxH6BzFjbrmaD3jPE0UnKmx4JR4L70pCalrHFxoCvO9HD1WH/ivr
W00vmXOamNHzZKvpcausSXGNm9VpQXhZRLmGhxum+LXv3Hhyy+QS4lvtVSHvhKjwZ7/c3SRu4BJv
gqgvlc5I/e0ynHmFcqRXkvir5q+sZISeQfcm6i94L4nq81O5qk7rIe0HlD4zY92R2siVuovlqkIU
3cZya7eWgbwNjDVaHowaMYJPn3nmphzZEdPvd+xBRWgtS1G2fPfTtkh/cBxkemjNtz3caYy+x96e
8tnPNdGAdGs5F3v2p7XE1DAqS+FHbDWX98cQsTfNFfdAj/SkCnLr21sxCg3lzU1SiYrTgNQdcqlv
HYYdMfI8eU1KHc2Cdw9N369oJm5BtrhVSjZaU9Y0stOJqjqZid5GC6jBNx2ISyReBAh5HaXOTiO9
rBlPGYfDqVpgEPtMYyJ5NPWhiqq1pteRXxmhTxTzLverplaXr88A7SPZ/tdy/GIMNt/Me8yE1cXI
88G+PtmHbCQijs+rx5bmcrMfqSRu6WY9Cw88P8YfBstep6lvPf+JzbJZDynPfYxVWujRXtcMQjG+
rKAz4kTs3oVn5bqcTfamP8B6E3A1zNachLpWfUmgOnZwrHTRZxyjq89Wx3Krw8/sXl9hF22vgjHA
aTO0iY8MXlyOPBj6Y3w1/8sNvYGTFvnn95ldjVZ6vQMUoikx6sS0+dSvFrAOZ+yxJI6akdlhN6ba
z1BiOoOPXcplszKQyFUEaguyyIe7tLOzmnpYrLpqcdpQNoE28dDk9dH57gFl6zlPDZFuz8jVTy6s
R5W6Bx2qjkBsdBMZ+wGm64PS1WSPbYrJPdloVqcF4VURFaimwn0oEma9OGP93ULBOP+I7V7Rv0x2
4JLxCWJXyIHUBy55boaL9unO4bbCuZ8UyrS3Lrmni9xz74H9fbWRnirT7XjV1pYnPfQMdUf6pFHu
JtO50r43MpTbWOpyZMgtbdb4a/4chxV8yZnn2aUIL44tGelXmq3Z+NDYFKv0YnI4N7TmRUg/SDtP
LtYT3ZYtv2ezZsV1EgPPNUJAurXsro7Fcyy+T75cfenm8rmIjS7kGFWVm75Z55JTTwRrjk6vIO+l
49khJraOkXkOkacBmTu0efKujUw8hMdDee1PewFdZAbkYfYFGerY5Rb3SNyY4jZNQ5svetqhftJ5
d1p4ruplJlL1xnR3sFKTnUz5wFwdBxkcVvw2nl5n15yPh5P41ZlQ05sQzCMV1eh6Eei1+dP+HsiY
OzcoOgzjlG9ZhIFJiyoQ7gW0AcF/U0N4dXFPQhgpTl/9e5kHjcD3hvr55NxHAH0Y01qz/O/Kp4V/
jjQTqQ0I4ZecNA5mZspjrN8s9bdYLA1CbsB2pyt2IEZfZt1Nqrj18+6JtbVRnxC9Mq0bZtKTmlr8
vpziR08onTEn6Auero7lFp1FYLiLPMjawWk8vqXGGn+GFp15Tnxk0DI1fRbaJhQa54cCdeD5fWJX
o4E6s053gnHalJEb/wIB8XBG283KY407AkmPXOZF9sQ1t1qqfZVtVgYCpV/CIzXCBV660K/KWCkY
+bSGePfM57YoEIcomot9UehBu+34yvCo/ibEjdRtXxXk9cpa+8K1aJBLsztpxrKv3YyYVE9PTlbn
BeFVEZVLJ/BpY8IXrdduSLkl2+97CFtmagpdxt3m1c/hBi4ZnM9Iv44zkPrAJc/NcObFGfKs0y3y
WkBxip4ZpfauJBDWXk+ljXixSYj/bNRm0W7HAm2g4ScmgorzwIA4PGD1uY2ZyJUthd3IWKPlQfj3
QI30zHnhmaWQI0QKhiWfK3a/5DE2ykwACbCHQ+sk26Odxj63ji6p1CtM8lbZyHON0Kw8f+/vdeXI
g8j6GW04mBWxP+lst0pSt8NY6PL58XZ6BQ37ZIZydXUtMLLL04Apqet1uk1etI9sH3DXWJs7iNOi
xMhuP35t89pUx142E20iI0HUwamUTK6pY9NI8fnlXhNOQAWi3a16obKOROP9drlOtbyiLHGgdLlI
G76hCmKNB6FwCr6XU/4Bi8OpZyu6r9fsHXIVduDTkYwFwjQ5AxyZtKiFHO5bA+UKzzmV3FVTg7qj
CqSeVBW/rTN/chIa1r3SjT15ZatDHG8SD4N6H324ic+slzHMGdFl9DxHe10zgsQOxOrLjLtZQeLP
FUcX2QcnAP6kYS1O6MUZ/Q9/XLbyE54LFvOeeHUsgdlieN1SG8+5vsJ4eswmbA/pkVH7pEcGLWPD
k8bw5CU0zo84D3TFM7saDSBUFGOCEYblgx8oIB7OqL8sr/TQgc6n7wzM3l7uO1IDxF6DSvZmZSCR
q8RHQzt0PZMxx7ISl3oWN7NtWoEw6JuVk0qoy3vctej4hPdTKRMFaQiRsxQoXTLIm8nBI4euuvaw
pc6EupDwBmZjWF3+5GR1WhAGzKdEVA49kCuBOHBVURaB8H6DYg7fZdpNQfiA076kUBi4ZHBCMyn1
t8tw/IX6SlMNx7MYMtZzelOTW/qRbTE3V2rCsW7HexA5/OIMt4HlZo1ycQcGxOEBK9t55vM2MtZo
qQj/HqiR9KAsdBczS6HEiBQMS7L1N5bKKO4vugYkkEo4tE6zPdZpSCcDQ83bbCNyYdPNyvN3e9q6
JPYAqfXKkyK2WMxzt22VCkgV94R46/1OqiBvW7vzSQzlR65VpgEzUg+9nF/80FYH7tGM9ryAFyKh
fqB5upNeO53p2Mtmok1k7PNg96uUpj40FXFXWJTJojdXNJ4/tzvulRs4OvS46kg01gsMwSTlVhEp
3VCATesc7M6r6myVcBpvet6yVyT1XE1LpT0cKqkKTHw4kDFv6CnWmPY+af6kxXsYc96yo08fU2Ge
mhmsaRbRu7X0+apiX+yNPFLnkSyjMW1T519jT17hR4B2qJS+XBd7eFeqyci/Nxg98jazXsYw+xyk
nxqsfuZonz/QgRh9mXU3achxm0CbVqA5p6NX5nUWH8WcG8WpHmq1zYIzSueUoyp8ujq2q489s2vp
2vkRtIYA9eYVGLWriyc+WWuZmj4L7RKKFHrEeaQrntjVePOayHgiTTASk0A++nkC8uGMJZCqJ/su
AN2eW2hR1jV79MYOFNqVpTTNWRlwC7/nLjefcm+sT9ikJ5hsD+X0rskq9UaJ6HdxTZR+yHEMxfUT
5ZqB6oh92SbUxNQQn15NosmSfCir1pCbDELXXGCxrxGbYDIZ9+PSBCIyWel7iua9GNLOXHgBwcqV
2PzkeZBZem/qVNfXwK2uuWSot5rHFZ20qlNsN0KXAko3iCZefj0utFyd76ma5TJ7EuL2kIdfnBHr
zqMzR8V5XvcVDYIyw/Y1+bxNHH9FfbeM6WmKNEDMLIUcHnIwLOneA7f8/+vRWHxpuvuC12J1fHq1
imUcm1ZlZmh1sqG5it5YcyPZfh8Lc+i5Jh+Q6rCQib2m7+0W7qdFrL/07fWobicQ7+H2V6aaU60q
R9dVUGTlp8xafCjviePXbuHWPHXG72CkbhZZaU/328Xarxg6yfZjhucZzSTz4oxC4odSWYDQaira
sEJg0XCIVZxV9WKFRJM3ajOWs3ZqXbeK1D28ji8e58KdVJDgPD0QTmbjdeYDyrVFBCvBvM+VIgcA
pMJOCJVIzeU/E8qY22iTU+KhSYv7MJY6Y3G/m1suIU3jGvN2fYfmpp5Ulb+G9eQHDaVSxO7dBclV
Rz1O+dNpOXml6xI+rHceUrDNrBfpXi7mlOgyOpyjfX7+qcfqy4zciPUWzH34kVeGcitJusy+SJ4N
GNesF5RLAn2rHclpLKOhJ5B8daxyS/E+YqFPbY0THxm0NKbPQruEQlOVkTF9oCue2dVoon22khOM
/GyQKz5HQD6csS8KGyHuPJzv0/u9bzcmR/vH66crZz7ldkT1msasDKRyFZrDtK/eUqbMo6rCsr2U
Len2BfFW5cVw7XXIwqCdmyO7z91tQewcFYFmvZGgXW2xz5GUPz0RjI0l19//PEBriUVeiRCqacRk
SziS1UjLygahstc0I6LybysZ6O7iL0PQou2uWjAIRbcnxVKLNV/UIl5gaymXeF2zNOWfl7rVCl8y
w8FvbSurZjfsYEcilt6J7rYr+PnaaeiJrXizkfh5syYi3U42yNJDj8/qPJXWOZTr6ECucl+oFb9K
aqeez9vIWKOlIv77QFf83FIoUSpHzpLVW6+/tQBhDibtI8+ZNN4SMyeHh8flmRHS2B7pNPy2rlRk
ZPLT9Jj5gDS/G+xM3Pt8xxp9G1DTInaYei2Js1wT6XWbxcHmtxdepYLSG1qp8UhuPe7bTLYeuJsG
XJC6PgUKTuykqt1uGpvX2CtEFzQTd7K3fuD7n9JUtDu4d36UPWYdDYfg/NOY/Yo1ErytWZvxrlVt
FUHS+LLj8A2HK90Lp1g4KI+ebnG81KvHH+/RRA6KaKTGK2nSJ0MxPHdKvIf83MXYfN/qRYZQa9bA
bVdy11HOVVXm1SdOhrsZjvi1R6lWlE5vcnU4DxqzX5xhn/VpyzazXp73GGt0Ql5lFr+nIU4h5j71
WLkxn0bDa+DuAlmyM9FprbIUfWvkZTHlXKtYEhCeCrzqzJcu84zlzJLM6cGRZ3YtCK1RYN76xlgI
SFdNfGTQMuUFyPH5SmwO5eWjv8vMrvinlNmuxgMtxqnkBOM4OXf4vQLK4YwlfL++v382B7WdGKvD
3Z9TzZfyt0tt/a+TZgaIopqUvlntsov8NitTYgZSudr6AvXlI+X8Iq9qZMboK+z5orT5thuJc3xp
krQP2eUltp6QMfk2S3X3ZTRKXdZz6GtyU6uj7EdCTUzpeIpS9E9WcjUNmGyJR7I6MQjNdb9JETXw
nYx8JOR/rqatu9v/3mpYmf9u1/R7Zvvt6jBRZydKE7cmNPol3pyme8CYmvpAGZ+b4fKlFNoqllah
j5yb3WTTW9YdvD5ECWOqOiEQ/IpH0WhPlep2tCrTuqP00FM8yMUHOH2+KHfDA7lSL7G6qZHO80De
hCjOjr9y6vmu+LmlkONB66Eeef36++92Mq+diMkXXQIycVwemI3YNTil0xh+C/6sx4q2FPLWSL91
rrUq47nVXrxS6mdixOZeQ98VxAggqdDT403k08ffgQpKlrALnfShjnICYe2iFUN6PyJ6C2/WSkpo
EmLMcyLTn3qeJJzOH2FrC3VFMwmEx/KRry95DUe9Q8TRnoimg/Fo1SthFb6tFZblvFzf9zdbRaCy
7By0fx2+oXZh4IZWOEVCRm16oWHXCeb7zYteVnuHWFFN8te2ZrT/XF06nw7GcH4GOH3SohZky1vo
XYv1beY+ziQXEs0Zrxxtr/igocy1RqYrc6tD+dKPZij/uxLH4of1xiQUbOKkUc55vs0ORFd2LaL6
vNnrzu1ADG1nsi0MynvbPHWrIt/jqdMEs2aXq7Ylgfh5lDWHIys/9iNiU3YW+uRxK3geceYjgxaV
gTnesZlL9OFvfBCZsKJYzBOFu6ViXmiz2QnGMXCu/tUC9uGM9Ydbna/KNSvIxay1WUeUQndrDuJB
kFRjKWpKGevGM1A9tEV7oSVD4sBVDtPyDKJel1dVDSIrn5JQmaf6i2RqbsslPT3DVTfo1KmXMQ1r
/3flCbwSrH9y2IAqrhK/rNp84S68Id9YZ55bR6opb7K3pkBWJwbhxREVh89GwsABkGJ/fduee2Sw
DN4my36+ulmIFA1WE6/rZEs/1Cv0U6ALUn/HDNeV3D6AVb2X8/rfx7jdRnbdjKu/OgNa2+EEnzi2
sF0PeUovXBIy2g+bUnGseZlWnoGhZ6A70nMm52sgV0aFibONOrYSw/FA3spOTDqTKcwqs/Xld3l1
+D+3FEo8aA/Lj39f24zS2TdN8AqQ8vRX1TuVXZMYWlZXlIwQr+sJzFWyzdN/ANwT7Seh8gH3kYAs
25XwZY2lWQXnM2YGtOY4MWKdESe4YxUs7IEeSYk3e5+gfiBunjtieR722VKLpVNBH58GnJh6kVOp
vQVf5XC/ixDHM1cuL2gmofCoA6+Jh9gMIjeN7FptVDVUnPvdxarXeosj0Vjd05R023e8dP5QUyNk
27g3zzKPJugI1mp7UyQpIkI+ZhWITwxt162uoFXP11nSMia9yUmweuXQczKWnQGKLWmbFZyyGDvE
c87jTPdAWfApM9IZq3/itEFaDmlWfJS3egVPErmtvlq4tbuzudWRekOfMmW1p2JNzInZL58ZpPOa
9bNk4yN9+bStF22ynW2zQ08NWqdzuNed+tRj5Uas4fopObiyNDl69f5c7E6MlhWq2e1RUhkKJpdO
bljmYgYLfe1MRJt1K+s38pwltRa0RYHW9Q3McGITl+hsX1nIqPY4lG+Tprti4wCGNHpmYl5ss8pY
Ejo5HFPmU58ioBzOqNqJMQVsJyrruvvyf7uWoX7+p8MwhmD3Nattdalj3XAGqqKEe6E1X6ZT11nm
VI3+xMinKLTfquxjqkoVH7CcDCv7C/Ig4UzBmwjb/md9MydD3eZlW1yxtpWklWHWHQLNtQKz/xmr
pqxJkQU/qxODsFo+VcyPR5RZ4+5in5yttm04M245FbXlGSfl7KrNNlltDcVIRbhk+7R0u9ytvIiY
nGGp5V6T4Wr7U+vpzMHQjZ7HXc11Sz1h8Vqrq1qy85OcDqh3lH634y48NBMRe+6tt9fcAOc/s9f+
A7k6MmClOs+BvAkLi2JIyetgVYzpE5hpc6RicX2oVRSlyE0L1eWosm61IUX/ydkZs5RZDcGI0uxs
xI7D1Fwl2zwDz4LDjxWpOXBzeq+oaec7VVIJLFAlkGdGbK6tKDt87hx7v25uvKWfL9MVNOoTnADs
saPuOY1MA6akHtjD6/cFpJ0CteH2gZzIuJu9K5pJLDzKoNcGN7E4QQ6PQpVoayZWnPtViYrWlqkD
HXr3kaCI2CMlShfL2vANtQtjN1TCKVEj8iulD6WuDA1m79XXUrB6nW4zMR7FKrrw8tqaPjl4lcXY
4FGCTib38OVNFL2JQFuH7/+gYbykTCucNV2ZWR3iPEp7apC7CWe9OL4F0Z3XcVtUeBn5uY+xSldz
vNed+dRj5MYKkvTK0szoDax89cOQfFjfqYwq18ZIMLN0qdVrFvrqYPAmWs1YeMYjQ+wnSaPzkMDn
wrN9f4Gwn0h5w/a9oQlrccKULNvVeLtBygZgbKcy4MpHPlRAO5xRTJDspwL9qUa+Tmpkjwa0/KW6
LDVACCu0RtMs246egWlHRjQne3VWGNv//leHqkFk9ZbSM0x1KynDeiioYdBf4vTh6sOV8fxs7Q3X
hLdPSik0Yamdd+zqo0s5HrHhJtZ2TcPVpAxwmsmerpvViUG4pnp2RNk1rg0Hqe7OWyfIJ6Kub6j5
0teLlEusBabEJd6KXeJWTkTMzrDSyV2T4Wr1WZre+Wt09cFK4R4HliYDqVdBvbSAe4pi72MuZ7rd
jj5pk1v36NBj1UqGRE5/KFeDA5Y+HMvlGMpbMZtrwy85/lo9aKIrvqgUuf4+smKnLi1KtXUFiHF8
TMqSFaWp2cgF0yr7t0i958Ox5xplPDcCSarkWyeai72iOxZ6YK25TIxYp0Xa2umy3m83FG/PqqBh
n9AiVv/sexcynrD6pyixiqakHtnibKeX3nSzyW0XyYmMu9m7oJlEw6MolvZIqxbHI3EdQj9XUQde
4J6J0xlRJa9vX/9ui6gTwdnZMH8z2yyLlpNoDsVwOtr0wsXRg1leEZNXx+UQ84K9u5d0sjjzKBCN
uWjVOB34KyzGjh7OSM05AxNFY5hLnlRTV//S04bU5CQ52TNDaGQ+qQJOWuxVnhrkaZ8yGUxOoYWT
GflJYzfHXDuJZz/GSp1NtGMxe92xpx6hlVm5EfWWfxxYWRqI3rGOXcGxiqN1kWWeB/fjJrVNZdwa
r47mwSNLLfcVwws+qbFGH8PdiZayod+Xfi9fcC1oS7r5fLS9Rycm/ef05x6z/5GWCKULUkOkMQ1I
dzWaiNqrZycY4+Rc+ZsFtMMZ+1JW7nm5O0wq4MkjzfqvVXpjy3DeT0skMiA28sADgBwzdf/iybZt
3H6snHBErKuCMr9ebvvlT2eX094Nim6RWoO5wq1UX/Hp0DNaujramNgSDNCW1x6qppxJtyylZNVo
p8aQ7bejZkCeHFFejWsdf6xher2QM6x0I7wfJcpDrZVQdYmfwu1WsUu2T4Ub6MzUqyInM2zlI3ar
gdS7OuqnosnHFv25WZ9W6183Sya+JLHE8HZxFdOB+h7tIdt1lTUDdoczqzvS25vcF47myuu+Yp2n
PcUazZuwsrXVd3z89Wsk1hVfUwo/t9IoLqzH6I+P/hP2BSDhmZ7WDHeG8Gwk8SAw3mnkVhe6Zn70
sWK7oRdIwrTcu6Tvk4wrFAcj+NIRe4R6uXZkSArH26EeQ/oNumQFjfp0s0bn/RfqzxPJ1znmU1KP
1eueVHk0Sf4xbj341843kXE3e1c0k3B3qE7DowGWmUa2zv5otVwRzczj43XVG083uds6z2Pbn3sR
ZyY7PRvDN9QuPBJOoQe9wrYPiiOp65XWN2mjmhLt/9Exlr3A+IQjEHL5uo7NANekE5OWcD3VxdpS
cHtOm2PG48yeQlfjyaFNDqZD04ZJkxN1KUEt4dB8MlYdoyDF3e2HdyUBd7qlYEshGK6XsWWHekiz
NlBGMcV2FW7NTgc/pwMxujmx1Ms/Hl9Z6rvzQitspPZcfdMSJ0WJcA0sm7Xn0CcvmwuFnVYdkdK1
6Yf7inbM07v8WOemj1feRCv64gzrCyPK7FqbjeQnEoHZSfmR8GxfaGDay72Mqi5flNE3H6PlZrsa
jcFps9kJRlKbj/9+Ae/NGQcn1L8f8DeVcOm2qPNLKtWdkGm5uL6ahrN6CeRvTeRWz2/dFu9hMzLd
/q01SrlSAnQ7KS4+jAACdBrBGLh+GhnMmP2xtX6ZV0zh5CYIIIAAAggggAACCCBwosDxAx8nZu5J
t+aZ/UnwJIsAAq8qoBzOOP2Y1at6fHK+WPa8svaHm9j11TSc1Ss9f11a73+04RY37KH8usC8qkB0
O1dJkw4Cv0SATiNakdfPI6M5sz631C/TihmY3AMBBBBAAAEEEEAAAQROFeD5tOfF5NSQ4+YIIPCG
AuLhjJEX1Lxh2clyLfCmX6V7y2o80MSurqYDWX3LqnmVTL/7izPe/3DJq0TCR+aDbucjq51CIzAu
QKeRsGvfRZu49FkffcMsP4uKdBFAAAEEEEAAAQQQQOACAes1ENvz6Vu/EXkuIs/scz25GwII/AYB
4XCG8xtjv6HUlEEQ4N1Sl4XFkSZ2cTUdyeplniT0cgIczXi5KnmnDNHtvFNtkVcEXkCATiNXCW/3
FopHhlnYzFUzn0YAAQQQQAABBBBAAIHTBLbDBu2DyvZ4yvuEd3ye2U8LRG6MAAJvLLAfzvj7dVv4
2v9jCeyNq3Uk6/wY2oha4pp91ra0sqEmdkk1zclqwoaP/iaBW/gMBfdvQqAsaQG6nTQZFyDw2QJ0
GsP1/16nMziaMVzRXIgAAggggAACCCCAAAInCXQPpPXGGkcz/vc/ntlPij1uiwACv0RgP5xx/1LS
9h+/6ftLKjhcjIvfyBDO1+/5YHFy9tbOxprYNdU0Jau/p+YoCQIInC9At3O+MSkg8KsE6DTGq/ON
zjs8Jr4c+RyvbK5EAAEEEEAAAQQQQACBMwSs4xk8wfyI88x+RthxTwQQ+D0C0uEMho/fU7/xklzy
RoZ4dn7hJ4sZyXgTu6aapmT1F1YhRUIAgdME6HZOo+XGCPxOATqNA/X6LqczOJpxoJK5FAEEEEAA
AQQQQAABBM4W6E9ojK/6n53Xq+/PM/vV4qSHAALvJbAfznivfJNbBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQAABBBBAAAEEEEDgLQQ4nPEW1UQmEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQOBdBTic8a41R74RQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
4C0EOJzxFtVEJhFAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDgXQU4nPGuNUe+
EUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQOAtBDic8RbVRCYRQAABBBBAAAEE
EEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA4F0FOJzxrjVHvhFAAAEEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQAABBBBAAAEEEEDgLQQ4nPEW1UQmEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQOBdBTic8a41R74RQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA4C0E
OJzxFtVEJhFAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDgXQXihzP+ff9p/vv6
+59V7P6C9ga3/y3f5Mi1RZ7++/tVpPn9z62lOt3ABf/7X5NVB2XJQShj9YckvP7fjOSF28Uy66rx
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBDQBSKHM+yTEtoJhsTRgu6MwJFr
17LKudbPWyiltM4vqNk0T3WEMxY7oVIf0BBTdm7EEQ16CAQQQAABBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQAABBE4U8A5nhA4IiLv7oSu3kwX1oYIj196xjBuImTVPg8hHLewDJNqBh0TGEidU
NsY+2SBl6CUhJ8Yht0YAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQODXCpiHM+qN
/Xbjv/xrfyhgO1oQfPVE+bEj195qqsjZfuhg/0fzRR3SBX/6owvlyYnir0XKUrmzGevCbr1B8GUX
zk+uVH/meMavbeQUDAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDguQL64Yzy+IF6
FkA/8bD9xdn0lz535NriaEaXa/low15Q9YLudMZe7q5425/0uwUz1kfGltXISYpQBf6v+FTkps+N
VlJHAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDgDQW0wxnhPXvlLRehl1/cvfoT
E0eu3Q8bWG+uKI8hmKkppyHs0yPrX+vDDmY68iVdPCVenBGuwOI0C6cz3rAFk2UEEEAAAQQQQAAB
BBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAgZcXUA5nGG+GaIskn1+Iv+Nh/eR+mOLItfaxiT6t/TCH
fDJBuOJ/Q9ekMyaETh7mzx/hJ1m6O8fv+/LxTAYRQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAA
AQQQQAABBBB4OQHxcMb+1oXIuxTE8wvxdzz0nzxwrXdp94YK91yC9E4L5z0XEkg6Y1KkeDfZrkmc
rblfI1bhy8UqGUIAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQOAtBaTDGfGfFVGL
7J552K7s3yhx4Fr39EJ3rML9PRH3A72BcEk+YwJtGGYzlX7a5S3DlEwjgAACCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggggAACCCDwvgLC4YwJZzP+5x5GWMSkd3SMX+te2R1vcM87uB/oq17IRT5j
UkS5d1kusn9B5X1jlZwjgAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCDwlgLC4Yzo
GQCjvMEjDfuvbxSveBi/1j+U0P18h5uYf8+GQXprhX+TyO+KuHldcuKn9paRSqYRQAABBBBAAAEE
EEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBB4U4H+cMaEsxnbizP+fP9TXPZXZvz5+a/8mH+2QLk2
8MaPvmzOr5bsx0fUklQFFD8/lLHeLVgx0TMcbxqwZBsBBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAA
AQQQQAABBBBAAIF3E9APZ8ROI4jlrU9P3E5fWP8Vb834udv4tf7hBeHcgnkUpMhLnUm5lveTGfWh
lKGMdUkEjnjcr/FTe7cYJb8IIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgi8tUB3
OCPyAxteiYtTCrmTGT83Hr428MYI6dyC9DMkj/JVOXGPqmgnM4rjJvo9Igcq/DeK3HNtn+Ewjr5E
jp94Fc/fEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQKATOONwRujlF8pZgPFr
/QMO8vGN4lRFkafmjIhzckG+x4I9mrG6rqIvzrDPqBhHX9zjJ7QeBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQAABBBBAAAEEEBgROONwRvAdD2J2x69dr0y/nyJyHsQ6uWCezPgp5HDGKqAoTPR3
WrpXmnA4Y6T9cA0CCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAKuwAmHM6LveJDy
Nn6t/6smxhsspBdK/Lwsw79n+dsnyukG/yb+qzW8XyspKP3kavcZP2PjRhkfQAABBBBAAAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBA4IMF1MMZf4JvUujfCxF9x4OkPn6te8jAu3X5Ao3Hr5h4
5xzKK3Stwxm7O3m53zCzx1v893p8cOug6AgggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggg
gAACCEwQ6A5nbMcAHgcUnP+E8wvZ0wFlCkeudU4ZhE83bPlxrijetmGfY5mSscjbNR45zxXUO4Di
BQB/RwABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAFHoD+ckTmdIR0ZyB0OqLM3
41rxpIR86sM8NmHmJXwy46d8ZirB4yiZMxQpxPihD5oSAggggAACCCCAAAIIIIAAAggggAACCCCA
AAIIIIAAAggggAACCAwJCIcz9l/rcF6eIZ4CCJ42EDN75NrtR0ikTG85rf9onExQrrhnu/g5k8hv
v1i/a2IlUxKlzlBEb1oWJVKQoQDjIgQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BD5dQDicUfwyxp8/2qa9ekAh9d6GRv/ItcWZiTbP+2sumr+oh0HUK24ZTpx9eJRPfe2FmUxpk3lx
RplDvf7qjP0J/YbNp7cVyo8AAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggMCQgHg4
o3w3xJ8/3c598Zse/e5/6h0PTZaPXPtzK+nASPFvwkET7wrhzMLI2z28ZNzXVqRhqirqS1H9+aeG
2yoeKeRQ+HERAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACv19APpzxU+5u//62
h9/+158qSL7joQI+cu3jRkamlXdDWMWUzkyUhz0kkeXf9Jd3dBcFXloxAhOqvzUzanbdYyO/v4VQ
QgQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBA4KqIczfu5rH0SQ9+2P/DLJkWtX
BiXPxiGD1BXBsxnWGzfq0xmh0w/pF2c8NCKZFY+G8OKMg62KyxFAAAEEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQAABBBBAAAEEECgErMMZj4/1W/yRkw6B90G09TDvSID5uytS9ddlNMoXfB+Fdod0xuoa
CJ3kUFmrUyHmrWackqGVIYAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggMBDwD+c
gRQCCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIDAsACHM4bpuBABBBBAAAEE
EEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEPAFOJzhG/EJBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQAABBBBAAAEEEEBgWIDDGcN0XIgAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IIAAAggggAACCPgCHM7wjfgEAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCAw
LMDhjGE6LkQAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBHwBDmf4RnwCAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBAYFuBwxjAdFyKAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCCCAAAIIIIAAAr4AhzN8Iz6BAAIIIIAAAggggAACCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAggMC3A4Y5iOCxFAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEE
EEAAAV+Awxm+EZ9AAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQSGBTicMUzH
hQgggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAgC/A4QzfiE8ggAACCCCAAAII
IIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAALDAhzOGKbjQgQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAQQQQAABBBBAwBfgcIZvxCcQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQ
QAABBBBAAIFhAQ5nDNNxIQIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggg4Atw
OMM34hMIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggMCwAIczhum4EAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQ8AU4nOEb8QkEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQAABBBBAAAEEEEAAAQQQQGBYgMMZw3RciAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAA
AggggAACCCCAAAII+AIczvCN+AQCCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IDAswOGMYTouRAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEfAEOZ/hGfAIB
BBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEBgW4HDGMB0XIoAAAggggAACCCCA
AAIIIIAAAggggAACCCCAAAIIIIAAAggggAACvgCHM3wjPoEAAggggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCAwLcDhjmI4LEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAA
AQQQQAABX4DDGb4Rn0AAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBIYFOJwx
TMeFCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIICAL8DhDN+ITyCAAAIIIIAA
AggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAsMCHM4YpuNCBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQAABBBBAAAEEEEDAF+Bwhm/EJxBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAgWEBDmcM03EhAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCDg
C3A4wzfiEwgggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAwLAAhzOG6bgQAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBDwBTic4RvxCQQQQAABBBBAAAEEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQAABBBBAYFiAwxnDdFyIAAIIIIAAAggggAACCCCAAAIIIIAAAggg
gAACCCCAAAIIIIAAAgj4AhzO8I34BAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAA
AgggMCzA4YxhOi5EAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQR8AQ5n+EZ8
AgEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQGBbgcMYwHRcigAACCCCAAAII
IIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAK+AIczfCM+gQACCCCAAAIIIIAAAggggAACCCCA
AAIIIIAAAggggAACCCCAAAIIDAtwOGOYjgsRQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQ
QAABBBBAAAFfgMMZvhGfQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEhgU4
nDFMx4UIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggIAvwOEM34hPIIAAAggg
gAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACwwIczhim40IEEEAAAQQQQAABBBBAAAEE
EEAAAQQQQAABBBBAAAEEEEAAAQQQQMAX4HCGb8QnEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBA
AAEEEEAAAQQQQACBYQEOZwzTcSECCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAII
IOALcDjDN+ITCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIDAsACHM4bpuBAB
BBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEPAFOJzhG/EJBBBAAAEEEEAAAQQQ
QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEBgWIDDGcN0XIgAAggggAACCCCAAAIIIIAAAggggAAC
CCCAAAIIIIAAAggggAACCPgCHM7wjfgEAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggg
gAACCCAwLMDhjGE6LkQAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBHwBDmf4
RnwCAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBAYFuBwxjAdFyKAAAIIIIAA
AggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAr4AhzN8Iz6BAAIIIIAAAggggAACCCCAAAII
IIAAAggggAACCCCAAAIIIIAAAggMC3A4Y5iOCxFAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAAB
BBBAAAEEEEAAAV+Awxm+EZ9AAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQSG
BTicMUzHhQgggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAgC+gHs749/3n8d/3
P+8u60e//v7nffRF/r7m2C/bWIb/fbd3PjvFsXxyVVagr1ntDlNrfGuN4UaZLdhrfP6NGs5zsxqP
w9eo2JFcTG1BagbEVFJJpz48AvHUa162dOc3gU/pdVPx9d/fr/sg9D6zvVTxhj48y+R4XzSU/Ve8
yOh2tIZ/fofwilDkCQEEEEAAAQQQQAABBBBAAAEEEEAAAQTeUkA7nLEuNwfW4LePnnXUYTrsrLV0
KWOPe7c7F2emOJ2HG4oCcs1qWBNrvN0j/J0bY2/UcJ6b1Vwcvm9jntiCDAQxlVTSqQ+/XXW8ZOmu
aAKf0evm4/FlD+vkizLtikkmx/uiaSV69o2Ubkdr+Fd0CM8mIX0EEEAAAQQQQAABBBBAAAEEEEAA
AQQQ+E0C2uEMXpwxVMvaKv2k1fuhPHHRDIFsDWY/r+ZxO/u0vDQj9jabGUW+8h5v1HCem9VpcXVl
7Q6ldU1Jj39Z/Zp8DhFOuOgFS3dFlj6j1x2Ij5c8rDNQjpmXzDI53hfNLNVT75WiuKJDeKoGiSOA
AAIIIIAAAggggAACCCCAAAIIIIDArxNQDmfw4oyhmtZW6Wet3g9liosmCGRrMPt5NYuJljihmM+6
xRs1nOdmdVpcPaumw+leU9LjX1a/Jp9htskffMHSXZGlz+h1R2KFjfBebZLJ8b5opEJf8poUxRUd
wksqkSkEEEAAAQQQQAABBBBAAAEEEEAAAQQQeF8B73CG/1MliXdsvC9TMOfP/VJ9MJN8bEBg0v5L
PuU14faHcvJ3euUrnsabR3luG38jqDxtdcU1JU19Q1ss0Wc00IOVOfPyKwKDSlVqjI3wHuZUkyui
fWbrPPFezx15TywYt0YAAQQQQAABBBBAAAEEEEAAAQQQQACBzxNQDmfENyf4jukeNM/9Uv3nBe91
JT51/8Uqxta8/FNS12lMT+lpvPmSPLeNvxFUnra64pqSpr6hLZboMxrowcqcefkVgUGlajXGWYFe
5kyTK6J9Zus88V7PHXlPLBi3RgABBBBAAAEEEEAAAQQQQAABBBBAAIEPFJAPZyQ2J3hxxh41fLXv
t7agM/dfTLPP2Jx5Gm8+Xp/bxt8IKk9bXXFNSXlxxsFquv7yKwLjM3rdgboDpkc71eSKaB+Ig2dc
8tyR9xklJk0EEEAAAQQQQAABBBBAAAEEEEAAAQQQ+L0C8uGMiS/O2M55/Ln/p/46w/q59Q0B26GP
P807A6I31OrMWktv7q3ntrn5ntlHKZf/lpx3KdbJOO9EmFXeHtaojb18XdGEGnS3J5rDPu3nyyJa
tW3/tEcUqo20//2vKmOdAbtmUzGWSvd/bSguIdURREPWbl/n1EgNKzX/ZMNpvEfLbta4VqfJrKba
+FEoKc/Wtp7xN70tR9uXM1jGKu1Yn+l7PjIpFtbtzcoCtscYyyjxX3ZzlNTJaqCa16FK7lpNn714
Rv/ZBoPos/yj4xXuimMRJobpdb1uM0eochOoN6G+IiN1EfPeRKu83ZZa+KyA24jahrMDRKtvIPjt
QdCf5Q+ZJOulqtm+jAdgq/KFKaJ91MQpRHMrreF/f1ezbaNFJatgqGdrw9buzaKqfkzyCQQQQAAB
BBBAAAEEEEAAAQQQQAABBBB4SwHxcIa+cN6VcVv2FNYilUVV+UzA8uH70nR9obJUWi7M2rv2daa9
L+B1C77pPbZ2w6tKUUaR00gBauGnwwaPy0gr4E1+yzSEfKzlWKupBBG2w9abSztlck2noKrMyrtx
W/Gim3VNocUYy6TbtoGtDsryq6VuzzPdcme3r+k1osDdy7EHj8NrbAUOl1059eI28kxWM218CpTU
9o1dvC1/fWtqm+rjzqn2ZYyD6n06/uE+M+a55NFqp1Ij6otWekmlU0emKaRGA/nJqhYBg7WwFj7X
j+1kanfe7hPLARTqilPdgjFSNWPeCb3u19eXdlo13TxHYt6aaIn3uzVR92TATpqcEjj9jDGiKe1U
zKo9CJoT+BGTkXqpiyM08CFYteOaN9+eOIWoS601/O/vRwsy2uoesYFJbD1HyswT1HoWe/8pPf9b
PmySaQQQQAABBBBAAAEEEEAAAQQQQAABBBDYBcTDGfIOncS2LUt2y5D6NkmzQfu47b5Bo30DLXlD
rZLl/QVrGd1/wYSStf7FGXoZ9G1S8duB5Q63Hc57ef/KXzRUiudw1y8VsbfX2h2GogrkVG4Yao2o
+7hBqCKveq23x0iaewdfdiJ+CfbnWj9d9SNbyk79tPFkt6/JNaJv5jfv/wg3nDrGh8sekVeaUzir
mTY+CUrMsrqRWkSf2muXAT6p4zXs+3equAEpvYblhyHVa+05Kou78bjndW7sW9/2neldJ5E6e+Xi
0Y3MSCfeP9V/bpGpJ/v9b/mbc8LSboBlXSgDQeAEp5bLE3rdv3+VwxnZ5jkY8xv7ptVMV3rF73/2
YaCqG8pNCQaqbyD4/Umm0vlb0auajPVF4pyhPLGRh7Um7moYeGHV9o5ujx2f1DU1qzV8pcdNDF3N
m/Fm92zSGDWp5+cRFgEEEEAAAQQQQAABBBBAAAEEEEAAAQTeXUA6nJHYnlI+Wi5mV6u4+9pks1Wy
fh9w+Trc9tf//vt5k0b9BbjYDbWKEQ+e7NkSt+liRyG03YItxeW7stV3I7eU68XuPKBX3mZb3CBt
NkrUjBV/sDZK+r+1G5r9q1G+1m9FCi9ZqAMnD9VEWl23cnU4+0AdvRVjW9EyYaAdtejOORVr//U1
dvuaWSPVDnkkeNxv+Os1Lhu2x40GalxrTTPb+DwoObfKrmG10Si/AUd5t8mcjrducEVupF219f0C
kcaS9zTHgsBefvfFbPmdMA1yvsvy+nb5GIkYq6mRblo/Vm5Jdr39WsehwzBqV6yGtdolqlM3LY18
ElavO6V5Dsf8OgzFJlprOfS3fXSYuSlBfXJL2V8XR7SRF2cok0yv06/f+uSYFGEXGgTDfVESVinU
9Pn2xCmESOE1SqG7HqyC1AxN7tnKcBamyXUg/dSQ+lj07o+X5B8BBBBAAAEEEEAAAQQQQAABBBBA
AAEENAHpcEa54Kh8C9V+l7By4uCRB3PpVXyrRrF4KW3giDdUa1zaE9HXuxPnVPYvUf+8/7v4r9rS
0V+QIe98Hi5vmXqX+P5HZf9A3C+TBNe1/tuP0tT/9bVjgShr3WXcHIQ6Uh2h7cMixJWTPtK7SuSt
OvlflQ3txX0jLKvb2pFvtpjlU1NSw3Rylw2ebpvNaqrVr6NsEeeXPdgAte4jsqMaTcLuW4xWFvvB
jeJlRGVp6j3Fqo6EbjDfk2tyqT52pJHK78HoQkPa+BW7lFBjV/f/ftJVetcrSKuBVuqF+hiVotFp
gNF+TBtmip7ypzmHDsMEBlmh3sRuQZ8TXtPrrip1fnPNMx/z0YHAfBNapmkYm+b7nw6MaErYuKEb
KsMtRva8JUwm9e1iJGbmWl5vPG2+fcakLtQtKyNyUW/RecjAoGP0bPuBiy39eT0/z7MIIIAAAggg
gAACCCCAAAIIIIAAAggg8PYCwuGMapEyeDgjtgH00JK26cyvjlnfFdRu6K1Ku2+PXm7gpb2n4321
T95Olq7y0vT+Xpbd+U6eeLDFPu0i7jvIO03yCrmxoWlshEjJehAWbrw61Dc7jMWY+BoWuST2v2rb
mZtUEeN2GMyrEeXs1SaVqUWz9pQ9Oans9s6fF0NVJc9r4zOhlDgUM7tU9ffyqwp2v+3ZeH8vM2Zs
V/a3GamyfK8lZj9TJu38xVpuIUve7b2/S3174sUZQ7Ugj5Xhfszck9xrOrRbrvBIzV4MvsgJkIt6
3ePNc6QPMQcC8xCLMTaLHVBiSjBSfeqmvDrHHHs/wZDJqX1RAlZ9PJk+3544hUh1y2p/OV4F8Zmh
2bO158i8nt37+9s/alIABBBAAAEEEEAAAQQQQAABBBBAAAEEECgFhMMZxTpr8GhGtU1sr5r/pC2t
QlrLnEM31GpZvplzhCEUMlo2zdV96ZvAp5RX+7a9m1hd9jI2pC82Nptf5v66kCUjDIRbuXk3I03c
pzMPq4S2D3+8xHzFwiDw5nZ7R+BWXQaVGAbWBmmuRuyGogSP13AK9mNlj9d4rvco3uxjJhGNn2IX
sdoBd+O9yfby+SJPS03//Mv+/60X9Xd300vtJ2X62FhjOcXTLXSJbO8ACj2Be/cMqX2z4yPdnH7M
MfKbtAQuvm7C+PWxw6x+NrO97trKh5vnoc5WGgicMvoE4lzBnxL4N04P/nbohjsO551t7p+FKsoM
gkoD7/t1ZdTXI8QiH+mjZk4hzLprq87NbL4K5EOvQjfm9f5V0m5GM11U6IGEDyGAAAIIIIAAAggg
gAACCCCAAAIIIIDAKwv0hzO8rzEWpZG+h1suPttnO/pdCXF3ZeiGGrm2AtokEvmCbZOEdmeb09rR
cU/GBHIp1VBwzXhfb5czUqcufp0zu0Vi5VbfO3adytX2WHVUW1fZVXPzm5/yyr9ccPFGgR2B/m52
GBg7Nbka2SMrETxewxHe/2EEvlF28Sq3gZStxcuqXbl2+koUC51koNk/Mt01meJERnc4Y1L7Msa6
eB8ba6SneKYau/vhFvWSsayu/FYpXwvyizOircnrr1xDvwF6SfzcIdHMr+p1jzbPQ52tdTZD619S
NVWYe79OMVR9dmbEWyZiYLP18mZm4+gg6A03HqzeF0eG9dS0auIUwpw+tbHpxeRAFYRnaF5siPPs
lOorPziSNwQQQAABBBBAAAEEEEAAAQQQQAABBBA4JtAfzvC/x2gtXXurocXipLDrKnylceyGGor+
/TUhnfAu6C017c72Em5/1VXlXYTs8xPWUnJTV8uNin+1Txwkt4f6VfgxqFh1VDXvfuexCTaTVI4p
OVfSjSI7TP1nzCJEdmqEbGv7IqENaOFtGP53Yo+VPSOvdB+z2vjj9rOg1AFg9VplqybatNflf0oH
5ubtJ0X72Fgj1QLGzK8XeKnG3qPZXcFYl5Ufy36uMNpKthbEPeBoa/J2T7eMht5mINbOWLegtpqL
et29isaa5/Q+xGVMNY2ig3OmBG66YjQPjGjp/AfO9Hgjwjl9UXSulYrx5cNDfdTMKYQ5ffInCd3D
SbYKpvVspf6Q6rEnO65GAAEEEEAAAQQQQAABBBBAAAEEEEAAgZcW6A5nbMuI/tEEaUk2tOP4WC/d
E7B2cIZuqJE7W0XRbSvh9tqd7RQFwVPKq+58CdkzMrD/HEL/jpNuo1LZYTU2XiNhsJdkCCpdHfKv
8FhNWkxiIF3x53/UM0DSTsDWwOzUp9XITx7ywZNoOJHdte52I/JK9SayWtxB3riaB6XHYpPfekuv
+l9SyYbalz/WBfrYkSqb5OmeJZDaWbh3nUo63KhvRRithXTVuG3WPeAiNKV6auImUZQ3PqlJn89L
9jzdwJJsnlM725uxy5ipqXulxaYEbrpS9Q0Ef6ppL0Hn5k00Ob0visGGB4bqg0N91MQpRGr6pNbp
pCrYZIyJeupUmX/QUX5xhz+u8gkEEEAAAQQQQAABBBBAAAEEEEAAAQQQeC8B/XCGv+i4rYB6X0f2
SKw1cHd93Lt5+ffQzdqNKx9i34jo3gdhpygsaoeyGC2zt5/SrzlXhRdKri7Drxc23wNuQeSt6kd5
jKIbh0hC1bOC2briWn+2QsTPD6QrcwRy0xObF82rkXrLNxg8Wt6Ef59fdvkAjNa6ElktbiG1mJFW
Fih9m/Gy+bftc/nf9z3o9m9ea4x2P/rnzD423WfOC7wUste7di04dXcPebhR7zfO10K6H/OKLIZe
rgF6SfzczerjmsQGuu/7HZK9bjHeiU3QbZ7zYn4B8Bi9v/eVFpsSBO6btJVrO5BOVwTvGunv0/p2
PfEYrN6BWMXyiizddeIUIjfrUTI7rQq2wvYzw0SXYk9tvX6evyOAAAIIIIAAAggggAACCCCAAAII
IIDArxToDmdIe4lyybflSeld+Jktc/X7b+Kex4FqsBJqb1ss8AbKot3ZTlFY4M2t+ToWbnm7ypbq
tExkvUD4UZJqq1JN2QgvK7f9KvwQlANi7bX437l+OIlJDKQrbygGSt1XoZ36tBrZc6xYScGTaDgn
lN3/ungR/ImsCleVLWaklbltWegLiia5XN69rej2D+1eXxXH0s8PHeiCq0vlPjbdZ84LvBSy++G2
ZQUCOC5rpu4F2Ggt5PsxZ6/XNSxzKn84oJrgEO82lIRftCPNc17ML8JeGf3i9LEbmRJ46ZZzwNir
oOTaHsm/lzfpnl6shQdBK8MRWL0rse7sFVm867wpRG76pBRkWhXshdXnn6F54ZBqfCzgkwgggAAC
CCCAAAIIIIAAAggggAACCCDwfgLt4YzEMqK4UT/yxTvzmpEbatXQ3yuUdGB/UruPnXtpUfuM8ioL
yMIitpP6VuXCHZe73f+irtZb4ZVc4h+Bsq8R1/qzyYifH0hXMXSzk63U62pEDJ5Mw5le9me9OGOk
lbmFFzq9tW6//5WN8/HB/V+qnT5rP2pwdMv2sdP7zHjgpZCN/qoALn++K3V7R9u619ami5FrSi3k
+7HYFbF3+Sv3clW9zdpSWq7UkSTca4pDUfObZzzm19LHml1gLrRzhqYErlNyRJOC/ydLbjJSexsw
mda3Bxq4PddSO5BQVxA4kbwmMHEKMeXFGdOqYBMUZobOYZ8mD0PRNzjecmPD2mIAAP/0SURBVBkC
CCCAAAIIIIAAAggggAACCCCAAAIIvINAezjD2XEqimR/QVFbxBfuby9zeqdF4hmWtmLtq8NLqloR
ciu4d9wTylvuDxYVKOzf2Inv2x5i7S7323aC/wifMkAiS/zljsEA1IHqCH1Bsqi+6vMD6WpxsFWa
vHmyV1H3jgRlA3RejYwET6rhTC97asculdW9mQl9yEwoc4zZDmf8/ft1+7X7KmYeGfv6+/f+f/vo
GGhfcmaSfWy+sUzzdJKuiyc1Nbt3PaNv10/JNZU6oxbyVWMW2RlP2mjSUh/oFtRWo4TSQBKBWDrS
PKfF/EZhBYgT6ypnZEow17aIqcwQrJQgbzKtXuz4icBqtWLfeaDbnzeFmPviDHGKJHc6Az1b4PDM
PuIOqL7D0yN5RAABBBBAAAEEEEAAAQQQQAABBBBAAIFhgeZwhreIKO0+1Wug4lc2l8v2hdHympEv
upk31DCsfVLpuEH83IdWBLto8opwHtArr7T1uu1L11u2RpG3fIm3+8nDUpyv7+/7TrB4GMAAMZKe
BTVQHeHzOUsliJ8fSFd9qYMVHkUVFQ3MTn1ijQwET67hzC57YAN1b1u5rK7XiUlMhLI7/iWhr6/+
bMbSASx/khrrrI5I7vWb9rLvpA40llmeqcaubQPfC7a3ROFAjNw3mkxdNauhW3bT4k/pxEa6af3Y
lp92q7TwO/TijAJbOKkid4lqq3EPgISTiMTSkeY5K+Y3Cj3+3JHf4zSnBHtbCdumg3/wxRn70aKu
0agms+ol1A3asEq1OJGZ7/ZPmEKIvWbbg3gjsh1PVacT0q67qa29dMlIgnnV4Uc6LkQAAQQQQAAB
BBBAAAEEEEAAAQQQQACBdxBoDmcYS8tNadS1SWXTSl3NdndIszfU2MWEyi0ieR9NegGEZtEsHw98
H+/nzrPLez8pUX1vXi10mbZ0fqa/VSFR3rX9mv7ysYnfsRyAGqiOxGGlexHNGFPeviFvDKjbBUVD
Km+obco6P90xs0aKAAgGj8br7pHWR38Gyz704ox2m24gqAZaWTYOl/ZmbatWzdX7krEWaaEXyqT6
2Cd65pAftuv2aAGh967z+nZx23ijE4/HHa+Fkaopz6kIP7LSDU36nMmonWyXqCdyUa87oXmOd7Za
m5XnHVXfGpgLVbaBKcHP59PVt99277diwR/qr+Q5TXGpZTKpXpy+KAYrxrnTivN91MQphDl90kbe
7uDMpCrY9eReQWwvai8rt64y9tsjatsV2aB9h+dO8ogAAggggAACCCCAAAIIIIAAAggggMDHC9SH
MxLbU7G9knU3v/y/zUZg4CumzWJ4e9Pgz2MrB0+cm0d+ZL1ZKt9WWQe+j3cPyLnlLX5npJVzvlso
1V5zzqNsQOoe+fahid+xHIAaqA6tZrV+Q4yxgXTNIxVOeDSVaqc+t0a8ttQGj9dw+vicWXZ3m6qs
ZS+r9s/MtAWZBeWMX8qO0P0q6wzBets5HZHboZV97Ehj8RNoA89sp6HNsIXv57M6khQTp5P+JKoY
ekEnvG2nohirmirQqhEl971/syvOdQtqq7F6hFwSgVmNug0ebJ5edWoxr0+XtDt+/f33+F2k2CtO
Cl5/ShAZx/V3EwhTQVE+VB1iWORNRutFbGtaXxSEFYoUoMj1UROnEGK3rDV8o0MYrYLcIG7P07t7
5VSLMS34fPPxz7IAIIAAAggggAACCCCAAAIIIIAAAggg8FYC9eEMcXFULo/50XbddFtD79eaYzuk
iRuq/Ooisnrz8GZEs+66rKY6R12sVfIJ5a1gpfuF1v333Y/bx2Pbc+omzsTvWK61nIAaqg65ZlMx
dihde7ugPz3TVardvqbXiLIDoQWPwjvUNPqdwyF5rW4nt/E5UM5Ys7eOPo72v1mHERLty8xLtI8d
rrKc5/Ed3OUOd1cxbXVLbRaplOoj0eUv+mArnLqrPyz5DFeNDKQfIlECye6K9QALnbV5pBke3xrA
ZK/bD15jzTMV86GJlqR4y1tgR1+sNet0WHVBtvpywR8qu9Z/5U1S9SLbeuBh2LZUMYpEHzVzCiGX
Wmv4VoeQqoLxnk1zSk7bpC5qu3Wi/3qrR08yiwACCCCAAAIIIIAAAggggAACCCCAwGcLVIczEuuB
znpmudOxbGSkftRBrJRmwTW5aOmsSnfLrMnbF5lbrxzeF19KP6e82zpxeTvvy3h10tunHcPtKuX2
xo7D+idB3d9OiEGNVodQs94+TlWKkXT9MtevPqh/tqbMnr3Lc0qN5IIn3XBupWtba/Irr3egiHFd
0emsRttL/SV35apEHG7ZtmrfaHJNfMfalz+QBvrYkcbSlbb5Gaf+nmIqqXhYPrw19KpoXu/6k+E5
pGWqe5+z/quSjdFaOFQ1dYkfOQvMIfRAFIfnWLegBmokAmJJeHvrzQhvvWLFmYeEO9tglqSKGugq
V2NvSlDVRcx2vSQR/OGya6EhzJ1m9e2DfVEKtihWgiLWR02cQuik2thnjonhpjGxZ/N/+Cemyosz
/OkEn0AAAQQQQAABBBBAAAEEEEAAAQQQQOCtBeo3Z7x1Uch8JbAsAifPmBxBjOxtHbk/1yKAAAII
/A6BxD7x7yjw55WCKcFJdQ7sSbDcFgEEEEAAAQQQQAABBBBAAAEEEEAAAQQQuEKAwxlXKD8hDefr
0yfkKP5N/BMS55YIIIAAAm8jwHjxNlU1mlGqeFTOuQ7Yk2C5LQIIIIAAAggggAACCCCAAAIIIIAA
AgggcIkAhzMuYb4+katfnJF/R/31JqSIAAIIIHCNgPX1/m28uPDVTteUmlQWAaYEJ4UCsCfBclsE
EEAAAQQQQAABBBBAAAEEEEAAAQQQQOAiAQ5nXAR9cTIXv/Z6+yFtttourmiSQwABBF5RYNtE/tMM
C9tw8efP979XzDl5OizAlOAwoXwDYE+C5bYIIIAAAggggAACCCCAAAIIIIAAAggggMBlAhzOuIz6
0oTWFfxT9772zbc/j/84mnFpJZMYAggg8LIC3QCxjBPb/zl1eHpZll+cMaYEJ1UusCfBclsEEEAA
AQQQQAABBBBAAAEEEEAAAQQQQOAJAhzOeAL6+Ule8+KM4gvQt+02dtrOr1hSQACB/7N3Z4ex4twa
QB2XA3I8jsbJnGC67RpAwNZY1LzOw71/26BhSVCDPgsCzyJQimeI8j3LKLa301uCdquuI8F2cTmY
AAECBAgQIECAAAECBAgQIECAAAECDy0gnPHQwzPauJtsnPFfsmBgoW10qJxHgACBFxbYJjS8XLzo
cHtLcKWBBXslWMUSIECAAAECBAgQIECAAAECBAgQIEDgDgLCGXdAVyUBAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECDwPgLCGe8z1npKgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQI3EFAOOMO6KokQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE3kdAOON9xlpPCRAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgTsICGfcAV2VBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAwPsICGe8z1jrKQECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IHAHAeGMO6CrkgABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEHgfAeGM9xlrPSVAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgTuICCccQd0VRIgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQLvIyCc8T5jracECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIDA
HQSEM+6ArkoCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDgfQSEM95nrPWUAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQuIOAcMYd0FVJgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIvI+AcMb7jLWeEiBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAncQ
EM64A7oqCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgfcREM54n7HWUwIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOAOAsIZd0BXJQECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIPA+AsIZ7zPWekqAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAjcQUA4
4w7oqiRAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgTeR0A4433GWk8JECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgACBOwgIZ9wBXZUECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIDA+wgIZ7zPWOspAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgcAcB4Yw7
oKuSAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQeB8B4Yz3GWs9JUCAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBO4gIJxxB3RVEiBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAu8jIJzxPmOtpwQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgMAdBIQz7oCu
SgIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOB9BIQz3mes9ZQAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBC4g4Bwxh3QVUmAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAi8j4BwxvuMtZ4SIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECdxAQzrgDuioJ
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACB9xEQznifsdZTAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBA4A4Cwhl3QFclAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAg8D4CwhnvM9Z6SoAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECNxBQDjjDuiqJECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBN5HQDjjfcZaTwkQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAIE7CAhn3AFdlQQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gMD7CAhnvM9Y6ykBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBwBwHhjDugq5IAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBB4HwHhjPcZaz0lQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIE7iAgnHEHdFUSIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
7yMgnPE+Y62nBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwB0EhDPugK5KAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4H0EhDPeZ6z1lAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIELiDgHDGHdBVSYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECLyP
gHDG+4y1nhIgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQJ3EBDOuAO6KgkQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAIH3ERDOeJ+x1lMCBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIEDgDgLCGXdAVyUBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECDwPgLC
Ge8z1npKgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI3EFAOOMO6KokQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIE3kdAOON9xlpPCRAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAgTsICGfcAV2VBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwPsItIQz
fr4+1v8+v/89ttG/789Dm6/a0HMlHx9fP/tydJR8Hp2r9nTf3g2WZh4Owq1PmyAvnbbngi4tZ6d+
KYYAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQKPKlAOZ8wZgU084+8HD7wme5Nl42mRe/dg
RJJDqCBPQ/TAg3Hx5DcPLyZMCpg0L522N0lA7dlzZREgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQOBOAoVwRmVF/BjXeNBIwE2WjTu2t+gc3QV9eQn99TfOMA87Z0/t8N3yPDdJQNV64/cECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBB4AoFsOCNdEl/HAxbPl3jIeMZNlo1vsnFG5dEsuy20
P+pcNQ93H5m98jw3SUDt3nsFEiBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4A4CuXBG9dEH
86L5pQ9HuEKvb7JsXCUa7tjU/M/Pw/YkBeG9FtqH23rlE6vI5mHvCOyW57lJAqq3d44nQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIDAIwpkwhktC7jVdfP79fcmy8bXS4DMiYvvr3I6o2Wc7jcM
l9fc0r93n4e9ynvlea43/3t75HgCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAg8ukAlnFHe
FeMmGYgBwdssG59q2X/jkDSRcCLOVbLXQvsA8k1OaRvJ956HnQPRkndpKvJR1Zsa7yACBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAjcVOCCnTP++6+8dD4/cKJt94evn3PPz8u+h/PmH0cw6aFT
gqGybDzasLmuQ5uOxWxjE6viSw8liYd6kbgo9qW60L6QzD8hZT2QaRdW/ovelYMprc6T5Xa0qx08
CFYiHJ0I152H57aea1lNqmRGNDa793ZRy/O0Tt8N+vLE6LJd9/2//zqu9N5+Op4AAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIPJJAJZ8yrpkMbQ2zWlQ8xizDKMC3QHitaLw0fTwsDGuGhf0cW
1uqHG7Y88dCew49WPNnyyxGTxYRYBRJK6YzCQnsMmeFM6whOPDc+KjOeH13OyahvSpsKGpqHIwhX
nofpVijBpDrMhK5md95MSnGX7KhFF+BiXsZnrif9YhuYjiu9s4sOJ0CAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIECAwMMJ5MIZF/xNe36JNwxaJH9On1+U3izN5w/9+snFGYYbtq4sk7QoralHW2xk
JsMqcVFYTc//qtLVVaYkSbPEJ/7xZ3u30eh0/mWYztjKXrC3whjCbxOuOA/n2Z6dVH3N7r2jZPM8
lWrXV2B1zmyzWGNXem8HHU+AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIEHFMiGM1aPHMjs
erHpUbrkvFhpn9d+480mPr++/p4SstomI7NqH9dy/sP8z2NBy3oubthU3L9/f1t8bP/NPUw7ntTb
tnvGNnGR3Tsjt9CerLMv68w0Zqrg628zkHSjkqmG8+jM5U1lXepc2jjjT3mdGmjcQ2MQ4crz8DxH
T57bSdXb7M57SsOoLbbJSJqzdJ9mxulqi8+KbgFdwp3dczgBAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQeU6AQzgjWxXNPGJn6VtgCYV5lX6zyLjYQ2K67R8GE+ZT8UzDWLb2wYQ25imQrgdVQ
l54ksZ0VQeIiE8LIlFuuLgAtDUGajFghhJmRfueW6yLY1aEyIrsiJFdCnLtpnocL6qAL3c1u0UuP
CcenHCCaTkl7edFlGya9shGk3i46ngABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAo8nUA5n
/LU3etxBZvOCfD7h2PHo9+WF4eiMcLH4LBsHN0Yalt3qIzOIhXX1WvVpiWExcTqj56cbnyQZkN2s
IdnTYrUPSTKY6ZYJtY7Wfl+6PjrmYSYHNJUeGd9kHiYXU3gJ5bZCWV4+i00q+m4p4cYZmd00tlxJ
m/u50ofFFGIpjfui9HXb0QQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIELizQD2ccWhgtDK+
WWGt/u17d9RiqjdZsi3uzZC0NEgfZHdbKDasYdeMldElC8ztiYsKRDCx4m0w5nX2bU8LdQRmQxOg
6wJomoflEjN7gRQjPzvNw3h2tgEUtjBpK2A5R6NsTs/l0cYVP9hoIJfS0UWHEiBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBA4AEFGsMZx5avl8aXq6zRwvnvAwyCf9Ff4IcrtsEWB+XNBcLdOS5r
WGs2Y8MzFNHI7b+x7XZ5v4PFYyfKQ9AQwIj2atgmMYacR66J8jycS+xAaNScJ8PAPEw2IilOqo5m
d+pFvax15LeK7aRs44qfX9R6pXf2zeEECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBB4YIGu
cMahH4uV8fhBB3EkY/5psA9GnGQoRBJyy9vbzEBtrTtpbdCw9mzGr01QVWdEI7tUvvlFNlPRlJGY
u1V4HEu0YcR5Lm83zhhzHr42cvPwWGAnQqmnSWnzYJbUFg0IN4/ITqreZnfqBa2udiSd1lO7y4mO
cAeVtlM6L5fO/jucAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECdxLoD2f8NXReQ54XU5vW
lY9BiO0ad7wmW9g4o7a8HT0HpZYZaW9YabAuCmgUlspXaYjM3gWFUfgliYovLJpHT3s5d/2SjTOS
CXDZtI/m4WJ6boY8g1DqabjhRZg/SDtzOiDcPCKTQOgfu169YKyrHfmtY6PTzRXuaJO0viUi0ttZ
xxMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOBxBFbhjPKqa9Ls7b4NLau8m36XTwpaU6sl
+H3tlHAwhk5KSlpHNBo34ChtL7DUCDfOWNQaVBkU37JxRlDSwND0TPoL5uFy+5I2hCltFA/TQGeL
m0eEtQyMXY/o37GlbE5hgm5PK18d4STuvtJ7O+d4AgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIPLDAaDhjs2A79LfvlTX47YJurZagwNop4eA0hwPKQ5ssuDelM8ptXTQqWuzO7KYxtTEKdOy2
ccaQc46v2X9b6wjCLeZhZfOIkWb3Tt3ObM6p+G3Ldt84oxKOeeD7p6YRIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECgSWD9WJPWHSP22Tmj/8/p2/5m/yONQrR2KfXqOqd4cLjDRW5oShtn/J6T
LIqH6+OVVk9NSR6r0bJxRvAUjqj6LrPa5GwtrHseRgj9G2dUzogHvWmeZEI8YbNDxHwt4VhXoYPU
SNtFuJg2/Vd6bYb4PQECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBJ5IYB3OaNz/IFgrrp1Z
+LP9YPX/jzBa0C3FF+Z9KhYF7t+w1fiWIxXV1e+5tFpLp3TG109x44zaYzOS3xc2QWjJbaQ11Vpf
SZ4sUWuFnY7ezsO2zUc+PtL8TvfGGfPcjKZuZR6Gs32k2fFtJjvf4r05KuGhqC8DXAOnPNFNVFMJ
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBQE1iHM6b1/8Xq9aqUaUH3I11oLv1x/7zG276n
RbigGxf01765UR+LZiW/KWwBsexuR6Dit+Z8m5JmZQIoC9mG+MLpkM/Pz998Qa6jQV05nUJPC62J
19r7J0BhdhZRT+eF87DQ7AGEZHjjxM9ik5biPBzYb+LYz/zMjvxyaYvcc1NKo5bUnFy3A7tgDJxS
u3P5PQECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBJ5IYBPOSKIGf6v/600Y5jXz7W+T34Wh
jXWYYOjP6eNK0vXrbat3bthmeFOUBVi8uJ2bHk27RSx7uhyepB1RBuYQ52jeM6J344zfXvU6J2ds
8ySlmbaoajXeuyL8DVUcLOieh5XZ3t/szDQKC8qUfigimVG5y3ZxGxi4bAdOeaJ7qKYSIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECgKrANZ6wXvs9L+pv/X9mfIThvlfQY/HP6VRBjqufz++c7
2k9itQAd9aerYZFprk1BHiI/JA0bZ/yeXNxHodaOdThjz40z+p2TzkRPYlnGM9rn4Y4Iv33KBgs6
52F1N5beZmcnUqmgcAOXSs2rcwYu24FTqncuBxAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QOCJBKJwxl/zayvF0WL633nZBfXC1ghxWYUF3aiSv0JKa8ADDWt5Esk01IUkQWM5TRtnLIcmkMsM
3F8btjyFHQ0GNs44WHQ4FzfOOMGOzcMhhM0uMccm7DQPK5tHLKpa5VDisSvdZHJquas2P2qrJwSl
A9x+2VZmdjW28kT3U00lQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIBAKJALZxwPjhZ5swu8
c/mr0zLhhPJydW0xO63j1KTaKZv+ZBo2vli8SSY05jIW2FXgqeu50pf+U3lbn703zjhPgbYJUNk4
I5mvI/OwGeE287B1UjU3u3ZDax2DuZz17I0n4gDXwCm13vk9AQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIEnkqgHM54qq5oLAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDg8QSE
Mx5vTLSIAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQeCEB4YwXGkxdIUCAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBB5PQDjj8cZEiwgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAIEXEhDOeKHB1BUCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDg8QSE
Mx5vTLSIAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQeCEB4YwXGkxdIUCAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBB5PQDjj8cZEiwgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAIEXEhDOeKHB1BUCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDg8QSE
Mx5vTLSIAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQeCEB4YwXGkxdIUCAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBB5PQDjj8cZEiwgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAIEXEhDOeKHB1BUCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDg8QSE
Mx5vTLSIAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQeCEB4YwXGkxdIUCAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBB5PQDjj8cZEiwgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAIEXEhDOeKHB1BUCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDg8QQ6
whn/vj8/Vv8+v/89XpfaWvTz9fWzPPLn69i79c/byrvtUee2rgdk+d9PPDy35fytbTsfbt4EFRIg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIDAiwo0hTMqUYBnywAcUybrVp+zJ8/QmyAnk4tp
PEN37nxtxfPhzo1SPQECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAi8jkA1nNG2R8NTbDdx
HLXcBhmvt3HGOa/xDFuB3O+KeqZxv5+SmgkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDg
AoFyOGOVzFjtwrDcv+FJtmjIbZDxlBtnlMyT0XmSoblgGo+f+kzjPt5LZxIgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIDAPQUK4YxF9iK7+8KzZQBeYaOEKTNT2ROj9bh7TsB71/0K8+Hehuon
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgbJANpzREbqY99d4/AdovMJGCdPQVHfEOI9M
9cC3vUxeYT687eDpOAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBJ5FIBfO6AtcPM8WDa+w
UcIUzqhmYUQPatfhK8yHWh/9ngABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgTuLBCHM+Zt
M6rr/8f2h3mBSjSgsi6+eKrKx0dt84fV4Zvj57TJR/rv1L9aiqFW+nkUz8fNbIt6M5rtaYtDPe37
YZSO3IAEwjWW9bivj0/ZVn1fiJYHt3UmdOGX58Odr0rVEyBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgMBLCYThjPbnZkwW0TJ+OX1RWPjPLJxnExrZ4+dAwDpdcQponFMBhaZmC/8Iohang4/F
xnVuz+rkbo9yZDY0yVgcSFatW/RnO/HX4Y+UMajlXHjUgDif0TUTevAr8+GlrnGdIUCAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAIE7C0ThjPbV/6Tx23jD4MYZ+TBElB7IRSDOu2Os0xeLbTOm
LEK2qZXGrBMFyd4N+QTEJobQ+VCY1o0zMtufVLq03HJkuxlFOl/XY54wxrX8dT0Ls0mtdM6ELnwb
Z9z5xqN6AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIvJNAEM4YymZMz9rYPtAjfpZHnIfI
PgdjXkxfZRvmX6T1JOVEP17nI+K8wyJGkBaTrOwvSzrv3fD19XlMgYRn5R7wUXtyy3FeNg5PBixp
+rIZsVgpYLP93RTW+Ppa9X4CPssEe5osuz88Ez478Cv5oXe6EegrAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECFxPIAhntO7MsGzUZueMkY0zintIhO3K1xKFGLbbexw7Ef08k+849XpqaRop
WKQ5tlGLXO19o1vfZ2O5N0WYWQkDM1H7Fo8KWTR0Oxyl7qc7VcSPTlk+UWV8JhxiIY34+4xI3/g5
mgABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgTeTmAbzmjcmWEttVnnLi98R6GK2j4GpXOW
S/uHxm0Pz1VQLDizn8XElPy+nOeoda9t7mWfCrJ6YMvxP8ONPYLoQuq1oEyeFbJoXyn6stoxZFH2
tu6opBpVccCW+5WcWh2WWKumbUQcRYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEygL5
cEbbQzbOpW+X2MvhjOC31W0MwrX07PNOgn53bJzRvj9F4hRupzG1Y2xHknU30i0owjzG/MOeEczu
bBFvnVFOR2z35ShwBkUNzYQB/Go97h4ECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQGAH
gW04YyxDsDmrvCtBUEl76iDeDuIUSigkEjo2zmhA2MZRos005iEa3JFkNcZNG2eUUxm1IpZnh1tn
hKmGhgBGsL9J8EiZoZkwgG/jjB1uIIogQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgbrA
TjtnbMMMxV0JgqBCLTKQ7BKxzh4Ep4bxhPaNM1qCFPm9QuJsREPeoz5cU5QhijnUT2+KPaz2vTid
k/w01imZFfq+TUiMzYQyb/hbG2fUZ4wjCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQGAH
gX3CGdtNE664ccZHFH5oCGgMbJyxfT7HTL4prtzllrxHfTzL+0OUzy8kM35Fs+3bPNckE4PoCmDM
Ld0mJJoSJMeszjwTBvBtnFGfb44gQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgT0EtuGM
4DkTlYrmYMQcZujdOKO/1qhR64jGIlzRvnFGS2PyG2fEiY59ogCFR4c0D9JvqCFoYjZbsX6uSYax
ZeOMoNqAZWhDi/JJIf5QPXtcc8ogQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgTcTCMIZ
xT0efn+5XmKfAgPJjhbFKEK0/cM+G0scBy+JaAQP48g9FCVofeHZIds+VNIXu0QBxjfOqJ1ZSH0s
ts7I9nK3jTOGZsIA/j5pmTe7XeguAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECIwIBOGM
OdywfX7Idgl+fghFGtooRRGijTb+a9msYtm/YtohyBp0bJxRb0yQdRjYu6F7wMY3zqhkQ6KAzbl1
p74eJkN1f42uTTnmHE0600ZiLAP4I9V0D5gTCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIDAf/9F4Yxk64ncNhO/j8b4+1WyR8XiyMKuBLVTcrtVbGIBha0afgd2s/Sea1LxiReZxsx9mHs9
sHdD9/yrbX+RL7C8HUW81chU2sny66dQSsuIFx6mkv6qtnVGMPAD+DbO6J5+TiBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgACBQYEwnDFnG04hjEXhyVL+76+P/1YhjurDLw4nLRfrGzZvWJwT
b8Bxaml+h4/cQ1naGzPvFZL2YGDvhu4hG984I7/lxWKs1+N4bOF564yvr89osA/HFLrf/cCT3plQ
23almL8JEiPd4+IEAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBQEsiEM5ZL9uscxbQU
f4pmbBa4o+DEtESeW+PP7KmRZCGW2YE0JLJoQZSeyG3HkIuRJGVEj9zYhEsG9m6YH+oRZyK2w1be
LKQ40TObY6S427zMqcRVGieKM+y4ccZvpaMzYfsYnkMPouBIbXsOdw0CBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQILCbQDackSxqT/tjhP8jXBFfL/pPZ/6u7Oc3WciedTx9kwqoHJ+esN7u
49zqfN6hUviq20MbZ3TuhHFZoKCGFRMfJtri1HCriT03ztjUuJ12qzYM4Ofmw24XloIIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgMBZoBTO+D0meoTJdq08ymdEWYDjmvrpN2GoI1tfZlOE
Qvsy6YnVZh+ldf184ZvWVHITcS3TSZnOrWdpZ5ZjM8kz8Yy/2tviDdFDbg617LtxxqLIzWzbYg3h
rzA83cQ9kQABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSuJlAJZxzr3a7qn1bIoweIzE1N
ww3zkvr5p/nl8FV11ejCJkRRebxFkjGoPI3kty/rwuNml8vJ/LYzbNGb5QjnzNJ26kzjU1m2m5cs
J0iAU3gUSx2/bSYM4S+ndXWOXe0CVDABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIvL5A
UzijwDAvn1vfftnZUk9RvGzXdYwAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECFwscGk4
Y9pawnMhLh6LRy2gsPnFozZZuwgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwOMIXB7O
+O3L784KshmPM6b7tmR6mooh3hdWaQQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECDwLgK7
hDPeBesN+zk9tsZTa95w9HWZAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBHYREM7YhfHF
Cpl2y/g4/RPNeLER1h0CBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQuKGAcMYNsZ+mqmm/
jGM2wwNNnmbkNJQAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEHlBAOOMBB+XuTUrCGfbM
uPtoaAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIPLmAcMaTD6DmEyBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAo8tIJzx2OOjdQQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgMCTCwhnPPkAaj4BAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECDw2ALCGY89
PlpHgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIPLmAcMaTD6DmEyBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAo8tIJzx2OOjdQQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgMCTCwhnPPkAaj4BAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECDw2ALCGY89
PlpHgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIPLmAcMaTD6DmEyBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAo8tIJzx2OOjdQQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgMCTC3SEM36+Ps7/vn6evNv3bP7k+Pn9r6Ed/74/T+xtxzcU6RACBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIEDgdgLN4Yw5I/AbFRATuGSEZsp6yEU04xJp5xIgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAgQcQaA1nJNtm/O3jUE8VPEDfHrYJzemMJBFD/GGHU8MIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgEBRoC2cMYUEPj+Pz9iwdcZl82rKupQyF6IZlyE7
mwABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIPIRAUzgjiRKc/6eNHC4bvok0G3MRzbhM
2NkECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOBBBFrCGek2D8IZOw3cvBnJ97+gSNGM
nZwVQ4AAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE7i1QD2csUwTn//Jck0tHbo5fbHch
Ec24VNf5BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDgYQSq4Yx024y/Vle2zjjHCubA
wVTAx++/wtNQkkDC35Ef2/hHqerC7/J5klWV1SeMnFs/d+n0k6mgrqe9rGnPkyJpViED09r4U6l1
3oeZkxpCgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgReSqASzthu71DZOuMUODiGCtaB
gEPoIgpoLAIcx6OCYwtVTwXkIx3L32RrjOIQaaeWJx7DGJVHlGQnTNjqhmhGV+N/a2/jfal5rTME
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOBhBMrhjCA9UN4kItk4I05mBHti5A9cHZwN
Z5TyDNGWGtmwQpwemTu1buppo4zcFhjVUQ5SHXPbMrtmdDY+k5A5p188nqY6SA4gQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQKXCRTDGWHo4PzDcFn/vMfE19fnNugQZxg2zwg5dCiJQWyf
kLJ6eMgiMrH63TabkR69ODgbi1h1aur4v39/24OMb5wRnFqLZgw3frVjSch72UxyNgECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIBAKFMIZ20eaHEoobZ2xiEnknzCSZiLyzyrZVpTsy5F0
5pyd+DzkQRZ5i6Dw4i4XYfCklP24fFal7Zlrqu2asYqgHJpRbPy2wPIWKJd3TAkECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIDAQSAfzggeaXI0ix4UctIsb8gQ5jAKGYHN8WEBp+Z8fX9v
whnb4/NRkGMPot/XdrO4cCbN6Yyvr/PDRqLsRdy4tPKo8T28F3bE6QQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgEAkkAtnZLbN+CuiEHDIBjoOVccPROnJPpw3yfg+Pk9kbszvxhCnZiVb
RGxTJIVcyaK4j3SjieJeGzvMqsXOHJvNP+YKhho/oX8s+rRDsxVBgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQINAlkwhmllEV+M4bpN+FTOXLnzemMw8YRmSd6HHqzySckiYxNOCOfzThv
UJH//0kjrv/4j2U6I941Y+57V+MTs9N5Jd6mGeMgAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAoEsgDmeM7IDxW228Nca5PfnfbvaOyEU0zgeeAwznR5r8/NWx+I/pP6OYRVe+obphRZd3
eHD5aTDHUwKjXDfW+Ytm3st7ogQCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEBgLRCF
M5pzAKtNHgrPO/mttrIDRVOCYFXDMo6x+K+oLastOkoRjTnfUO7UThOqnGo5VDLU+Ll5Tbw79UYx
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAQCIQhDPaYwCrLRrKe0w0pRzWGYL1Mz5O
VRwqXm+jkT7XZP27Q4+H9sAYOqlzirU8OWWXdtR4O9vtcAIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQKAusAlnNG+b8bfxxCI7UUlfdKQL8o/5SMIZaRbj0M/kB2E2oyUCsQFripTUmctH
NGycMdT4TK0tT1G5tEvOJ0CAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBE4C63DGtHC/
2hVjCRYGLXo3zigeP23fsdw7Y05dbLIZSTgj3V8jaXhHPGQ6a+Sc3snVFLzobsgIb2/LHU+AAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAhUBVbhjCkTUcxmTE8USY/q3jijvF9EnC2Ywhnf
35+brTuOp3x+fx/+/+//+H32SfqvFoIIGnTTjTOWG5Gshq678SO81eniAAIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQKBXYBHOmJ93sdyvYlNolBTo3ThjSnisno5yrCwXLThvivG5zWac
zvk8/mqbzZgLjX4X9717v4pe/9/j2zYrmURaG18czIYHqQz0xCkECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIDAViANZzRum/FXyHZxv3vjjCSV8BulWKRBpoZscxvJrzYphTmOsC7w3O/k
iHRfjWyhtY0zGnMV5YmXeYLL5qTRxmd2GDkkWCohHFcMAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgcLHAHM5o3jbjr87t1hndG2f8FZOmIg5hgdW/bXYgk1A4OKThjFzqoFLj6rTaxhmt
uYrSMHUEPPoaP8J78XxSAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQILASmMMZ38en
gTTuprDeOiN60ElSVTblsNjtYhnNSPe2mIoqRUjm34WnnsrI1rg96SYbZ1TkluPV0fjDib28rg8C
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEBgb4E5nHFKRpRyDUnl60hBOcdQSTlsIgSF
RpQ2s9g+bCXHtdqDIlPfLTbOCJ4QUxvjtsbPpfTw1ur2ewIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQKBTYA5ndJ7ocAIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgbqA
cEbdyBEECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgWEB4YxhOicSIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBOoCwhl1I0cQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBIYFhDOG6ZxIgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEKgLCGfU
jRxBgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEBgWEM4YpnMiAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQKAuIJxRN3IEAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQGBYQDhjmM6JBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIG6gHBG3cgR
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIFhAeGMYTonEiBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgTqAsIZdSNHECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAgSGBYQzhumcSIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBCoCwhn1I0cQYAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAYFhDOGKZzIgECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECgLiCcUTdyBAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IEBgWEA4Y5juWif++/78+Pj6uVbxyiVAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgRu
KtARzvj5+jj/Ex04DtLP134SCe8f8+f3v+JEWB0+Dc3pf9RO35Z9yIT0jfBtTskxXLH2qejq8DYc
GYxUeXSWHVuP7O9/V1t101uIyggQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECgJtAczlis
GPcv/dfa8XS/P3rsA5ENWuSLr2QzOtfv49KKGYDbnJKbFleu/Vx8fXiLR1ZSFjnf2tge4xr1tj3d
NaXBBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQeFWB1nDGasH43f92/8yxg0NlKT6uoba5
Qs/KfaEBuWJuc0pfMqOYWOhrcMN2GKe2FY6sDdGhvdHoNp14DGjsMP1e9camXwQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIEHgkgbZwxrRe/Pl5fPZFz+L/I3V3p7acPS5n2Dwr5lD013fyjJFg
BX6/+n+fzBI8ymT+YdTD25ySG6ob1L7DxhlpwmJtuEiKbEe3Vnvl9J2muGIIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAYD+BpnDGtBz89bPjlhH7deLmJe2lMIdevv+dOvFX9HE1P1Ffd/D0
qx2zIZuishGBqVlXPSU3pLeofYeNM4KBXfZoDm+sGZtqT7Ifds+4+eWvQgIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECPQKtIQz0pTAXrGE3nY+1PF7bVwRrMMfN874OXQ3u0y/1yAUIwRxJbc5
JTfat6m9tnXF3LrckS0Bi1xfGmuf4xnSGQ91c9AYAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIBAL1cMZyEXmvXMJTD8be4Yhk84R544xfocw6/XkMLl2VL2zN8Vt5ONK3OSU3OW5Te3XXi6l5
2QxG21USTqP22ssaT32BaTwBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgReTKAazlgvAVeC
Cetl6eT5C+cNIc6C6a9OT/LI4S4O/Tg/92N7dGVRPGr7Nukw9fi3olWbF7/6+/Xx35ySaNkyYW72
trnpxhk1jEsfalLboiHQus0puY7fqPb20EP2yLZpMBB/WdDUPF7sVqU7BAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQeF6BSjhj+/CEjvzDOlKRxBiCX2XyGZk8xCY3cRiCcnKktBh+TDpEzUqy
F/Gvk4a3b3twnDHr9h7Or4UuTif9HbZsUN9OGtWl/S3mbU7JXUw3qr19DAtHTrO2NpjrzrbXXnjs
zfPejrScAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECrylQDmcEa8zlTQGS/EOcqthGCuYN
KNZba8zhheSY9H+uV747giPn0Uw2zshEL5KNOqobZ7RvunCqf3XC33/WlvPzaZXtTh+FOVuNOmwH
+jan5Np8q9rbx7B0ZGH7leKNpL32JJxRmzKveevSKwIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECDyPQDGcEa4UFxfJp90Wvn5jBovdLabTvr5+d4eInwWyXGRehCXSXSGShe/lGRdsnHFu1SIg
Eq+U5yMgPdsenKZI0snP75+/jTMq+1/kIySn2ErbSn09BLDp5m1OyV06t6q9fQwrR65DNH3j0nJ0
e1Of536kpQQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEHhJgUI4Y/tIk4NAaeuMRXJgtb6c
LlavAghRqCIpKogrhI+NuGDjjEOwYbsgHqY9yhGQ7lmySltUVuVziZWCbtSihnX9dQjnNqfk+G5W
ez0Ecm5i/chgk5PW6E3LI2oaTLpnoxMIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4AoC
+XBGmH/4a0EhnJCsRq9Xl+dUwSZ+EMU9agvP0e8HNs4oR0DCtEclAjI0RptV/OzafCERkAnThA0q
bn9yOGMzKLc5Jcd3s9rrFZ2b2HZkENAoPLmmrcxTC0oxqaFp6CQCBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQuI5ALpxRWOlveKxH8GyOhlRBktqo70oQtKIczijughGvl4dL5TtvnDEP63oZ
PwponI7JbK9RVzvV1rCsn904I7+nww6n5Cb5bRr8W3tDRe2GU2+ihEbk2F77X9HNw32dW4dSCRAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQKBVIBPOyG6b8VtufgW5IYARxDaCrTgaNhDYtqK8
pUVYZHl/jrCj19g44zhYh5K/vv/+7/lfy8Mt0pFuDY7UfW2c8a9yCdUN1wWsIxr5x+hUnm2TzJf4
cTytV7/jCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOAmAnE4o5TNmP9if72EXPq7/8JS
9jbu0LKBwPaYYjIhLrK8wH7bjTMOGZUTadk/PzF6wxntu2CUHmZzas+Gq96a5oDDQFEDp1xr44x0
wBYJjdUV1DLvp7J6nmJzk1uJSggQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEAgJxCFM+Zl
33kPh/B/rdb2uwIYc4O2q+j1dfV5GX1a375g44x4o4K7bJzxc3Jpji0sBrYF7veEeghgU/1tTsnN
0pvV3sxe3nOlcr+ZAxrLmddcezqG0V407ncECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
8FACQThj/fSFfEJjubTcsnFGsFVDEKpoyBjsunFGvIFEmPZoaNrg+CYbZ/yVMFJRPcNwbFv10SzT
FJhkbnNKju5WtbcC5naPqbbz3MEtcDIsLXEL+2YMXmZOI0CAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAwD0ENuGM5m0z/jIbi1TDbhtnNCyRbzcuKK6Lx/scVJbSo3hE8+p7eSwDqkPJieempnrV
DW6nZlWSH1F04Dan5NhuVHuzYBiuaEi9nPsXVtSxccZUf7zpyz3uJOokQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAgazAOpzR9riGUm4h+rv/ztxGddeIoJWlczLbDJSruebGGcHa/GrjjO3O
GdXgQNvQHWZCsedxObc5JTdPb1R71fjUvkw2o32/k6iA1srnWlbpKLc5AgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIEHhQgVU4o/Ev8qPgQmFnh9Kyc7Tsnl37PiLOWYt534CW2j8+FvsM3G/j
jGR5/bxXxnrjjCDOUt5YIULJzrlS1zNT4Dan5Fp8q9orM289/zbPw2kMWITGTXX/tmA6bjmdH/QG
o1kECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg8N9/i3BGZoeJrVO0Bl3Y3KBz44xk/Tl4
asO8Nl18Csi50cnhmcewbFbYD6dec+OM3+LnZh2rX2yckQlaFJIzMUp+gmdDBOt2zUXc5pRck29V
ez35UD6i5RKKS2ja+GQufpU0ci8jQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgQcWSMMZ
jdtm/PVmG7do2boiiEFkIh1J2CDNZ+STFtGi+NSkr6/Pj79/e22csX1yS9PK+mIaLJfZf75/WzjH
NA6t/f23AkvOSbrSgLLNuCRFTZWkTQqG6ganFBhvUPthfBaaK7fS786DmyJuHztS+G1l043lmaIZ
D3xP1TQCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAhsBOZwRsvf/E+nb1eS99w446+axUL4
Oaww//9N3CB7/G/MIGxbocG/1YdZk/wCeesjKVL/dWmbThayLFuQTfgkNQw2ICkBR4dXxmSfU4qM
hRmxT+2nwamOSxCb6RrX4+Ct21yZ8IsRz/XXDY4AAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIEHlJgDmf87d0QbdcQN3u9dca+G2cc6swvksdr09Hq9mIzinDjjO0uGIfK4+jGqortnhOdq+aF
HED8sJWCSml3jPKTW1ZJj1zNhTHZ6ZTa/iMZrp1qTyZ6OZ/RMMi1pMW2yU2RkCjU8ZC3FY0iQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgVRgDmeclugbVp7/zl9vnbH3xhnnNq7XrEsL8cvk
wtyRcxmLcwtpkqR7W43wgSsjG2dMo7BZli/38e+81dp/ZsyaHlOTfyhK9kK52ilNjFerfdPfbcKi
8eI4l5SPC21ta3EODzJx4yZAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgMDzCszhjOftw9O3
/G9hvnPdv7XPh+RHPe3RWpzjCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgV4B4Yxe
sf2Pv2aA4i/3IZux/5gpkQABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQINAsIZzRTXe3A
622ccc3Yx9U4FEyAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBF5LQDjjtcYz7Y1oxuuO
rZ4RIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwBMJCGc80WD1NPUvmfH5/a/nFMcSIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECVxAQzrgCqiIJECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAmcB4QxzgQABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBw
RQHhjCviKpoAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgIJxhDhAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIEriggnHFFXEUTIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBIQzzAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwBUFhDOuiKto
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgIBwhjlAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIELiigHDGFXEVTYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAQzjAHCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQJXFBDOuCKuogkQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECwhnmAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIEDgigLt4Yyfr4/Vv8/vf4WWnY//+tkc9PMV/PB6nZxaftNar9cfJRMgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQJPJNASztjGMtKURibx8O/783DUKsFx/HE51rEz37klt611
504ojgABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEHhSgVo4oxzMOIU0oqxFuHFGYTeN
6/nZOON6tkomQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEagLFcMYymbGOYKS/3cQz
wo0zMrtp1Np42e9tnHGZn7MJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBiwTy4Ywp
1FB6DMkc0Gh5UomNMy4aKycTIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECTyiQC2ck
0Yyvn1K/OjamsHHGE04QTSZAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQuE8iEM+Yd
McrRjN/Kp3RG9UgbZ1w2Vs4mQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEnlAgDGfM
22ZUAxe/XQ42xFj9aI56/D4iZf73V3h1N41t9qM9DfI3Hue6Wx67shm/9NEuf+3OFbLuRXreinBR
ZLlRq9qHevCEU1KTCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIDAawlE4YyOJ5VkMJZ7
ZKwjDqd4xjFscDo0lzzYZiv6WteX5Ei6kwmU/LY9CKyk/Q16ez4jgog7nq1dQuO1rj+9IUCAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAIE3EAjCGX3ph8iofeOMeeuMeI+O4EkoU26hZVePwY0z
8smMQ7BknZBI+huf+XdCJqIShD0qtTf1+w2mri4SIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAIHnEAjCGRc9COTQ67CEzANMSs81iX7XFx0Z2ThjkaJIkxBJaGIZz5giJF9fh/DGfNIk8fX1
ufpVpiPZJ6LMtds94zmuLa0kQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIHgW044/Js
xhTOWOzxEGyCcWhB/rkmO7akPc+QhCMKzy9ZbJ6xSHOsakp3wViVF4IU9wXZAcS0J0CAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBG4skA9njD89I9wMI7tDxvkX6/pGNr1Y4/WXUduYI/p9
EsDIdmLzKJT5MSfJOaVtRP76Vvv9jSeP6ggQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AIG6wCacscP6f3FLiG3mI946Y4d2ZJ6vUkIpblxxOHHbrnnjjGzf0iednKsPOpjbXGRq8S4o9Unh
CAIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQGA/gSuEM/o2zpjjDk0PQenqeW0bjE1h
Dc8N2W7GUQh0lHbu2CYx0kegfBT/tT+lpcvLwQQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgMD+AlcIZ3RunDHtbxE83uNj/Nkqf1T1fTCWoC1PQdkc0xLACMIUpR04ysmM398KZ+x/JSiR
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAhcSWD/cEbvxhlThmJOHDRsYNHCMbxxRikS
suleobGlp5BcsnGGcEbL8DuGAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAg8hkA2nNG6
acUmZNC9ccZ/5wzDORMRljDA1btxxrTTRiGcMbRxRlBekNvYq9sDUk4hQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIEriawCWdMCYWmR2dsn+nRv3HGauuM0nYTXQzdG2c0PNVkW+ZuG2c0
1N7VfQcTIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECDyGwDWf0pDOCZEL/xhn/nbfO
OMRBdnqkyS9u98YZ9Z0zttmMUqSiM7dh54yHuCI0ggABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQI7CwQhDOmvMFHZfOMKP0wsnHGlKP4+tlx+4jujTOqcY4IprDPR0tuI33gSa3v+8VWdp5D
iiNAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQKAkE4IwkpfHyk6YG0mDmosDxkZOOM
eeuMr6/Pj79/TY9UqYxrLesQnT7FTYIGTL9boBS2u+jcOCNhD2qfvXMjYpoTIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECDykQhjPOzxk5BCW2UYkkprBObxQ3zsgmPVb1ZY7r3ApjaKOJ
pGtpQiLb4x03zvidHkniJVf7HrGVh5yIGkWAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BF5VIA5n/PZ2kUc4pTQ2/2+zi0OYiEgyB7l9MQp5j0k+eopKfljWlea6sA6fVPq9CkfsuXHGX18q
tds141WvQ/0iQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgRcWyIYzfvtczjeEQYE4rLCK
HARnZvaMSOEHN84ohDLOv1q1J9/tzbYV+26ccehttnZ7ZrzwZahrBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIPDKAqVwxrHf27RAfv+GfFgh87yQiba+LUb9iHScRjfOOJexPj/u894bZ5xr
X6VZ5DJe+RLUNwIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBB4dYF6OOMmAoUtKG5Sv0oI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAtcReIxwxnmnCFtEXGeUlUqAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAjcTeARwhnTI0Tyz0u5m4+KCRAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIXCdw/nHHeNePDthkXjaSTCRAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAgYcUuFc4Y9ot4+P0TzTjIeeHRhEgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIXCtwrnDHtl3HMZnigyYXj6HQCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IEDgQQUeIJxhz4wHnRuaRYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECOwgcK9wxg5N
VwQBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4PEFhDMef4y0kAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIEHhiAeGMJx48TSdAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQeX0A44/HHSAsJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBJxYQznji
wdN0AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4PEFhDMef4y0kAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIEHhiAeGMJx48TSdAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQeX0A44/HHSAsJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBJxYQznji
wdN0AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4PEFhDMef4y0kAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIEHhigfZwxs/Xx+rf5/e/5+35z9fXz4O0/t/355q2+t9Pbf8g
7ppBgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgRuItASztjGMtLswMNkHJq9jmGIx4k3
lH3jnMbzqTcPjwMJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgMBrCdTCGU3BgcfJOTQM
zrlHDxNvsHFGw6g5hAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIPK1AMZyxTGasIxjp
b58nnnGOQjx2i8+2j93Kp531Gk6AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBG4okA9n
pBs6ZDMCc0DjWWIED7dxRjTYk/3DbO9xwympKgIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAg8FoCuXBGEs0oBwSmA58jnWHjjNeav3pDgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQeXiATzph3xKju3fBcuzzYOOPhp6QGEiBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACB
1xIIwxnzthnVaMavRnkzivTpKB+//3IbbFS2tIhCFedT5kbOmZLfmpZtX/zqryHHf6eD1kUl4ZTv
78/DkdmdQbbplEvzKufamzcjaUWe527KMUGdfpgb9P5aXuta0RsCBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIDAkEIUz9npSSSYPsclNHBpe3tIijG6cTjlGGNbJgUX2IvfrKXGRFrVs9tfP
opot8jZJcalfV7ajC/nQ+kjqYLgNu0y97a9laDY6iQABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQIvKBAEM64NFtwVMov54f7UFy2cUaczEj3u2jdOGNd0u8uEoXMQtLPZLuJqa6WfUeCOdW+
cUYn8m9deamvn9PvNvt19NfygheKLhEgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgVGB
IJzRng3IVrqIAKQRhWSdf5kBuGDjjK+v43NHFs8xiQMSuQjIeXOMU1FT0/79SzaUiB4yEhV4abil
deOMfuRFYmbzMJjPz6PjMlIyUMvoXHQeAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBB4
SYFtOOPybEaynh/sHTHFJtKwwwUbZ4QbcWQek5KJgGQDCMchzz/X5HKrYFK1FXoB8jp/MXcx3Wvk
0LKBWl7yMtEpAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAwLpAPZww+kyNZ0I+2mvht
abSzxMDGGeXcQJj2qGyc8RtNCJuce65J6xYXfaPTtvFG7ajg98Wnrcya6bgP1NLXWUcTIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAIHXF9iEMyp7WNRFihmAw+lBFeVwRvjbcAeOqXnh9hO5
WmpNjrfOuFgqtqw15nBW/aBN6yobckQlDtRSnx+OIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECbyawezij4aEc2x0nyjmHsMjypg7hpha5Wqo7YIRbZ5TzJMPTqLZZxaHgAeTaKYXETGYH
lL+GVOmGHZxIgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgReRWDvcEbLav32mGLQIS6y
HDYY2jgj/yCXU3HJAS39HJoj9d0q2iIRqwZWt/nYBlBauthyzBCDkwgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAwMsI7B3OaNlPYhMUuGDjjHhXh6GNMwobRGyea1Lbh2J4fnRtnJFPk2ye
HVMdl+2jW6qn/HaymvkYhnAiAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBB4FYFsOOOj
tPKf9H61hN+wor/rxhlxK8PMQK5pHU0+19ZwyuAEadk4Y3qqSWGI1si1FgdPbqmd8ttDG2cMDrPT
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOCdBDbhjGnlv7CRxAy0Xp1vWK3f7g1R3H4h
3kqismNDlCvIndK0+cNiX4mmMwYnUcuWHCPIlaRF0KWRWgY77TQCBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIPDKAttwRk86YxslqG63EIQtSudMhy938ihXs/fGGdMWEYfASkt+YnTKNCQi
foseQG4D+0gTOQO1jHbbeQQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4IUFgnDGnIeo
bJ4RPYKj8liOqOzCRhTz4YvYwPQ4jUwDd984Y8pDfP20xSdGZ0xb8OMC5OhpNYly+qCUgVpGu+08
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECDwwgJBOGPemeHjI1rLP2hkFvSTc4PgxLTc
vyg3G85IDl81pG0fiEULcqdU94c4Dv6pkZ9fX5+/Kh/LrMhu86M1+THJNCNPu21sRjQZyVWvBmrZ
TUJBBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDgZQTCcEYavQiCCPnQxB9L8tvoIRmH
aEO6QUNS2fzjKbARhiEKe22kLUhrycUeKkVN45xGGLKZleCRLX0TpW3jjBHkTOJmMZSbKE73UPb1
1tEECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOAtBOJwxm/XV8v2h0zF5t8iZDF5VU7d
7PaQPf63+HBji4GNM1bhimmPiMaNM1Yecb/zu1M0TqXWjTP+iutFXmx2shzIwnYg/bU0dtVhBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDgbQSy4YxfgXWgYbmknwkoHOTyZwbP4fg9PsoA
HMs//WZxWiXDEOct4j0iWjfOWHQp7kNySO6A2qxq3jhjCDnvXEqo9A5lrY9+T4AAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIE3k2gFM44WmxX50uxjMRvfWL5tPToOdxw/mn4HJQ4A5HPWwRP
6WjeOCMJkGT7MZXfCLSeaz0bZ5zP7UP+OytNqRwbWq+3v5Z3u470lwABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIZAXq4Qx4B4H2PTaeDqwjofJ0fdNgAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBwdwHhjLYh6HviSFuZD3LUC3ftQYQ1gwABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgTeW0A4o2X86w/+aCnlbseUdv2YuhY/JOZubVYxAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBB4FQHhjPpInreW+HjW/MIUwFj3YOrZx8fXTx3CEQQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgEC/gHBGzmxONHwc/z1rNOO3g5u+nLo0/T/RjP5LxxkECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQKBNQDgj55TsKvGXYXjy+EIpnvHEqZO2Se4oAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBwTwHhjJx+Es54lfTCNqHxKj275yWkbgIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgUBYQzjBDCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQJXFBDOuCKuogkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECwhnmAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDgigLCGVfEVTQBAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAQDjDHCBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIXFFA
OOOKuIomQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECAhnmAMECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAgSsKCGdcEVfRBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAHhDHOAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIHBFAeGMK+IqmgAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAgnGEOECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgSuKCCccUVcRRMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
hDPMAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIDAFQWEM66Iq2gCBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAgHCGOUCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQuKKAcMYVcRVNgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEBDOMAcIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAlcUEM64Iq6iCRAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQLCGeYAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOCK
AsIZV8RVNAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEBAOMMcIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAhcUUA444q4iiZAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQICGeYAwQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBKwoIZ1wRV9EE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAeEMc4AAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgcEUB4Ywr4iqaAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
ICCcYQ4QIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBK4oIJxxRVxFEyBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgSEM8wBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgMAVBYQzroiraAIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQICAcIY5QIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBC4ooBwxhVxFU2AAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQEM4wBwgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
VxQQzrgirqIJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAsIZ5gABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBA4IoCwhlXxFU0AQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQEA4wxwgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECFxRQDjjiriK
JkCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgIZ5gDBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAIErCghnXBFX0QQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAAB4QxzgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBwRQHhjCviKpoAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgIJxhDhAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIEriggnHFFXEUTIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBIQzzAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwBUFhDOuiKtoAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgIBwhjlAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ELiigHDGFXEVTYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAQzjAHCBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQJXFBDOuCKuogkQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECwhnmAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDgigLCGVfE
VTQBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAQDjDHCBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIXFEgH8749/35cfz39VNpwM/X8cDP73/FI+ciTyUPnTKdm/sf52YE
1ZXOrbU+37XOilKrM13V+Epz4Fx/fSTODQjPuEBg27G5hvNw1YbmCnN1J+9wfNt/mGvFvafNTjp7
F/Pz1Xcd9R6/d3uVR4AAgUsFzq9/tdfJS+u56vnrl/3fW/mTvsxtX1aetCPReAfDdNVpofABgSea
b7s29V0m5xPdYd7tPfbAfH6Jl++Bu9Sep7zKNHuXO9ieY68sAgQIECBAgAABAgQIPLtAPpyRfEqs
rHhOS+P54yqL9/kTtwv11WzGFCbpPHd8ZaOzoiTxcu+vZdJxaVrXzpwwLrC4gMrFNM2Sy+fqnpd0
OL7tP8w15d7TZk+inco6krRfw73H79RMxRAgQGBfgYEFoX0bcHFpYSDzCV/m4peVJ+xIPKL9udmL
Z4YC+gWeaL7t2NT3mJxPdId5x/fYI/P5+V+++29RO57xOtPsPe5gOw69oggQIECAAAECBAgQIPAa
AtlwxiJOUV7zrGyc0barQlxF27nLvMbgxhn1DUJqS+X12Mh8xKmR9/5aJv02oGVhO3P8JcN0Vm3K
dzTMkovm6t4XdfseGV0zoevgvfv0iOX1gvQe/4h91iYCBAj899/IgtBDuUXvH55x44zcy8qLvNxk
humhppLG/Ao80XzbranvMTmf6A6z28g+1SU90Ounf/m+6wANgN+1vfnK3+MO9qD4mkWAAAECBAgQ
IECAAIF7CmTDGcu18tKKd3HjjNWS+6qc5cfRlnRA8s1j7fBKZuS66m2V3/trmaV/feuMxWhWDm8T
OI1CebbVIiQ7zdX9Z0T7HhldM6Hr4P179Xgl9oL0Hv94PdYiAgQIHASefX1iehuyfEv3dHfpXIOf
riPxdZUZJhfhowk80XzbranvMTmf6A6z28g+2tVVbM9Ir5/95fuuAzQCftcGZyt/jzvYY9prFQEC
BAgQIECAAAECBO4rkAtnTJ95P/+eFlB8XkB+FX6x9p9dy0+OquUt/qyKWZAEs/W4q/g3Vn7vr2Xa
Uw0L+dqM6BmmdERL82xu6maS7DJXrzINwkLbd9PItakr93K7jt2vpt7rqPf4+/VMzQQIECgJPP36
xKu8nr34y8qrDJO7yQsKvMfkfKI7zBM19b6Xw9O/fN+V73Wm2Xvcwe46WVROgAABAgQIECBAgACB
BxXIhTPmT4rfX+W1+GwOoSN0Ma+8t+/eUAty3PWzblvl9/5aZqr/6+sQwSnjn/r09XWcEHttnJFM
k3KR2b8s2WGu3vDqDAe9ayY0Jn9u2Kc7V9Wl99vW3uPv3D3VEyBAICfw7OsTr/J69uIvK68yTG4k
LyjwHpPzie4wT9TUO18Oz/7yfVe+15lm73EHu+tkUTkBAgQIECBAgAABAgQeVSATzkg/KZ6+PMiF
IXI5hJ7Axbwzd23N38YZO86kaehO4YyWh9d8fX8f91JpTFLU0jYd0yTz9cUOc3VH02pRNs6oEvUe
0Pv1Zu/xve1xPAECBG4j8PTrE0/fgdM4v/jLyqsM022uSrXcVOA9JucT3WGeqKk3naibyt5j5l7L
+HWmmXlwrTmiXAIECBAgQIAAAQIECDy8QCacsUhcFD8A11bMq2mLE1HrHw607Ukxpz1qG2yEI9Ta
mNzwtjVy83E82USimn6YUyqHqMTHQD+TXh4bXCjjdOzn988pm1Gpr00g6UQtxPFHHX9/cfFc3Y5i
acYXfrdqX/seGV1fzKwn5xxvqSVmUsPjrBmZNgeu5VRtKGd1Qnb6nI+b50Pav9XdZPGrU49KV07z
8d3dS+dQdTTzt5fWeitV9HxlOVDUeoxS1vp9qLePTTOh8EL7mMORNyzdCZvPar3c1q++xdvCo497
68yaJkunUvHd3ObeEl8JXfe3dYXhhdZzsc8Frq0as5aVl42+3q3bsCi8+Op5bu0y2rk+JS1/1b9F
1Y3vZaYXmPXx5ZeV2v2nddb22bZ+8miovThMxffP1Vf57aeDhvYs3oBkqoguiuab56JPzWf130wu
ugarttMBtQlY2kqstVPNSqHtoi9987x1cvbOK3eYv5EqvPeuffad345nsvytI9JyJ2t48d3xKsh+
aqrUcdHL976fSePpnVC3jk7f1boay/Cj6+mHjTtzxtOs/XbUMHPSz7v9H0aqkIN3sOr9vzrELReW
YwgQIECAAAECBAgQIEDgBgJxOGP6vHj8gNfyvcDys+JUQH2t7tzJ+pcn6YfkWuZjoAHBwkV768Pv
HNueE3I8Kv6CP/6GIrMY0LBEvmhmkmqofSFy+v2vR9tS0GoC5SbyRaN0KvTiuRo0rjAXJ/vt3FgH
UkKq9h/Wvv38a0A0E7KTdqdpk5mq+VBEV73zTPvtf/y91XRRtH6ttb7FrL7QW4JlW1u75czjtejD
dhjX8+R4RFe95auw7VZ6athAUWn/9pmBkW3PTCi+Uj7kcOQNS99Kt5zVdbmVJt6mIY887tleZzh7
lQozLHMbitekRmd1WMlf37ou9kMvsu0NXzlG7kvHgip37+wt73h26YXycMSyYX8U6SlB5eeZELWr
nKRZvWCsRrbyMlS4wV7RtuGzQ2vtmeNKb47dcivZ152uwWgMtvOtMhrTddP4RqhQaebCrL2VXdyj
++6QDZOzdZ6nb8PcYap373hQm96Td41I+U7W/uK781VQeMsa/Oryl+9dP5MWr9Ou0em7WoPvWJLX
1kOrtmmPaAK0veYWu9k+c+Z3I41vbM4tbnpTeskdbL9bccP7BYcQIECAAAECBAgQIECAwJUE4nDG
+fPi6bNfYak9/lXj2vyyT4UvsecDVy3LqkyfeFt2ZFiXctHJybpCOduRfNuS/zokHwGIFgzq+20k
XU0H6fS/Mw1Ovi9pXApqG6Y9shnrb5a752rpq5+NR/KFzuZ36/kbUoXN67tcpnq+DtudBP9KX9lf
Om1KX2lF6aD81A4XL4Optm3xuX+ZsrOXfP34Smsbw1rl7xej+1xnvZWrsOlOepr3A0XNp3zvOAPX
tl0zofjq+IjDMbdpfUG1bZyRO6vzcssv1Af76jzuuF/5plScXRXx1V/Zjs3qfP++fnou9t+O9FEN
3pd+53C+nulCzx1yuAKKr55fP9H8r76h+qs5267tddc6suWXlewN9qq21Y8L7bUXhylTj1vu9K4l
uJ/veg2uyw/mW3k0wnefna8jl72cLV76++6Q1cnZPs8PU7ncEXeY6lZ7t3qPfbzxtN6il0Nb+si3
CFYMTp7cW9ntJ5mOl+/s60hyCTR/Jo1fQBtE93mXXno/c/pd5aNW22tuvptdM2eRGGl4Y3OcX413
+UvvYLn7f6Hv1TcHDiBAgAABAgQIECBAgACBmwqE4Yzt13XZFYBwFT78uq/arZZVhtaSp+Mal1MX
jbvo5PRTeTkYMtF9fh6/tUkPz+RD0k/y0Z+b/ZbS3OPF0J3+I2zx6XeLP0Np3AW9UeDwp7Jj/y6d
q3Gtmbm4+CJl1bfNKeGl0f7DHMfqy5y5Fdm5sfye6LJpM38plpaTVL0ovn+6nufa11fzRZH91jIj
2PItZ+ZibLy6Si0qrJ3kbwLresu3yi6QgaJWX4zON5zGGZj7/jv8w93PjpnQO+DJvTqpO3uRJ/2O
/8Y4vtuFw7Ga5/OC9b/SrbBy1vDltnrxyVzODzvu174pFV6ZEpPl6JcNu2Z1PKznuXB699D2sp8O
YdLeuLHDF8JQ71YdKL1QnstfXTXTrezrmBmbOzgVdr6ZBK+aa8Dekc3ddcOOLFduwjd+mXt+l23T
e4im2js3aHnEV8AHueUulgW7rsG2d27RfEvfwa/nROUNSVelmQszNw1brvGuj0XFVfGmeV6eJO4w
LU9OjN9WHSdB/2vKfi++F1wFXW9ZM3nM8CXps/3le8fPpLnrtH90Bj6vZW6Aq/czTX9UU3nNzd+O
el/cB7o59qa0dAe75q147MsXZxEgQIAAAQIECBAgQIDALgJhOCP40iz+inn+riX84Ni2YHDuR3mZ
8HhUphkbitU6Uma7gPOPm74IaOdua+Tii5D8BhnxJ/KoxW3VTt1YphpO/1XYc+FQZeO6b1tT2o6q
uF86V+Piz/1cOi+/PwpjCAlgOJ/bf5jrd/Z7pcVXn1eaNvnx34Zkkr9ra56ul1wUrVdx/UvOoKRJ
vemelv8aeDtdc8GW5Q1v8YVv5SpsuZGeZ9dAUekYFbaWaQnvNPQx2MEhfSFoGvTHG47st+DFu13l
rGmOdl9u20kdXM4PO+7XvinlxyS66c1HB9fhwP1tPiX7d7HrZGeuwXNR2xmymTwX3Je6rtnM3Sr6
cXn+l2zT94Orvsf1d49s9tl/lY60vtYMzJz6NChtzb9aYux5UZnfqAevlvd6BXywW260yr29gfff
0sN5GL+dXXyiSuZhf6VjtsWm7nAPufQOtrk03WFat2a8wXvszHcPhW8Tuq6CgclTmc+Xvnzv8Jm0
+QX0iq9KhTchyat008es3Gtu7XbU/eI+8OLb96Y015Ob3Yrbv91yJAECBAgQIECAAAECBAjsLhCF
M8LPrvFKevjT8mffbBcavv6dSq58dl98mK4kMxr/Gr5dvrGR5W9/os/2+c/7x8bVfr/swmroTv+Z
/ULy+IuGMUoaUtnFo62wMvylczVTekh5Jvr+PuzqEK0+B3+AH+7ZXP9hfWUlXJcOLsjatKj9Pm1J
4dLeFFMrtzTF4y+BwxJr1awhM8fXrtra7xfV5L4GDvhq5Ua/L184XSADRc1foYZ34cIMzN0Poj72
3x7zN4rHG46KYaYrxbNqw1683Lb3ku3hDzvu174pVd+xxO9Gomb1z+pJPapkLq5lMaP4Tf/6pX3k
vtTfu+ybltILcGZ7sGRpZz2dC07x5Inf7Z4nQmlkc3/6mv78VrbxxB2pvfO95WIf+PCVMXn3NNKe
/V+2bnTLHboGcxd36Zbe9g6z8Iakp9Kxl7Pi27mOd4CZyXnJvIo/FLrDNH5YvsV77P1u0VHMY2Ty
RL3e8eW79JL41faZtOkN5HXfpRdvgPPrdFPoOveWt/+danL33+dtWyk6FDQ77snIW/qxW3H7t1uO
JECAAAECBAgQIECAAIH9BaJwRnsOI/6kXftMWfnOuPC5vPzV5lxuskpQjWY0btTaTt/YyOLXNlEK
oppl6HJffwdx+u/M5t6nHzfW0CbQWFgZ/tK5miv9VO42bPH7ky1VNDThcLX/sNKwbO5lU8O+06b9
u5+hetsuisUkrdazkoyPr0/Zrum6nT5/rQiKGKq33OUukIGiak3OzsDo77SPg1OCaU2AlO4TDzcc
NcO4M6WzqqM+sNy9bEWtzfcb9yvflNpf+ucj5yYt3l90398q6LUxSZsev1RmOlcveKdrNjNviy+U
4Y2ktLBT6E3Xbf0PqzKy63evQUduZhuO7VDtjancpEK33OhlbegazH4gKq0Gx3u+rZ4fWLiGhyrN
v8JHU7F8jXe87vffLKrvOqLuu8O0PjUzvqmP3XkGXn0zt+jzTawhPzfU1P5bfb2a4AV8h8+kxR2T
eq784fczmejXEMimN32lRO+RVn980d3NxfuEhuBs8Q7WMyDz25OmeMvAxeUUAgQIECBAgAABAgQI
ENhdIAhnRH86kH4rPX/WzPx5Sff33YdeNXxvWftzlkmnoazdJc8FNjayfFhp5aMaN2n4MmDLfa4x
fFjH+YeZlZSVZadAU3Mzw3XxXM1Ng01Pk0TGJpwRsYQXQfsPc+2qXlvrtqRfVJZnTts4rMrLnjRU
b9tFUduzpHhpx4ANN4zcVAtrC6+mwpe3BfttveU50NCVuckDRVUdcjNwqI9xCKnahswdKXzYT7Ch
+1BT45PC4eht/7E7pbOGLrfkVfd0a+jq+Ar5juN+1ZtS7Z3CvGYX31+3iymZvz8Oxrd2MVdfDubG
14padLPh4Px9qeeazXSg+EIZLwEXVmRaLpzyamzzyHYs/HVdawOvjPG8HRrZ7o0zMltn3PEV8CFu
uQ342wu2+n4/vd4q11MY2Uivp6HXkctswz0+Lr6HNFAX7mDFXemC+487THq3uc177OV7osxVEk6u
5qtg6BadnFSbhB0v38l7tan5o59JS9mMoS43X627guT02m5H13zbtvhCq/7WOn55veGtuPYW1+8J
ECBAgAABAgQIECBA4KoCQTgj+wF684vM9+F9Xzmculf4bn0CaDmmtop1Vc2/whsbWf6aYvstdu27
hOTLqZZV9u0XGKcq05OTL38W34RVKmgUGJommQXBTYta52puOqxX10/lLR7usoqsLNsQrENkngoT
HplrVzBKy0NXpe08bRZX1zzlthNirN62i2L/jTNavs5rOWazspJ8DRyd31Jm9kqN1yhbitw0s6eo
6nTNzcDCn3Ll+9iReCjd2ReX7+Jelna9xW6X4agahn0pnDV2uS0hSpdz8rqWHca7jnsAsNdNqTCv
mr4/bw7/bO9+1QnZPpG6Xm2r9f6a7HPNZjrQ/up5Hp2W5dHgZpJjGR3Z+sYZt7ONJu5Y7f0bZ0zv
dR7lFbD9SknZ9r3ljl2DXeGMXIPz7+4XC6tjryM72g68Awwn59g8L3bEHeaijTPGRmTHF9/5PUzw
GTe+Cobess4nVbvcd+Fc5zPpUbja1F1e8as3wHUXi9/WVO51+W18Rl/cez+MNL0pTb8zSufbLW/F
V/9STAUECBAgQIAAAQIECBAgUBLYhjMKH9NXn62nI1cfWqsfwYMGtSzo5+rbFFf+hu+6E6KxkWWj
YAyavlA4fo/bEs7ILgRt/+5n/kmba6PAwF9jbkbu8rmamwyr8Vku7i7+Kx7J8KftP6w1q3mn052n
zbld1a+dhurtvijCB2IUr++4ipYvSfvuapvFkPDaGaq33JC2a/RkNFBU9evTdZlX6GPLN8nLWfBY
w9E3ldbXXHR7H7rcZqLq5Zz5S/gU+f7jXu3FhUqrG0uhtN8hiuZo9/2teu0Ea66Z21+1qPS8loM3
nenu3W+NmXMGXigL971Sw8Ke7jeyQdU3sw0nwlDtQ2/V3HKnVMX5jt2CP43a0M0qO9W3L9vhoftW
2v9ObLd7SAt15x3MHeZ3Qrc8LOEW77H7b9F/k7HzKih1tmXyVCdh+8t32vrT7WSfz6Sna7Ta1OjF
uvtqrdbSA5KrvfLuun/mdHczeltc+uOZe9+Kr/uNmNIJECBAgAABAgQIECBAoCawDWeUVveWnyKn
j7n1PxesNSNbVLRkUPuGqH/trta8jt839SSzh8Lmi9lgh9SWb8fqzY2Ill/2FL7Iqvg3CjT9sU5l
xWiHuZqzSr8mWn+nd/rvw+hsv+87lFhc9Kk9XSE/ftUvr9b1Vr8Oq8+V/BHr1dDgb8S7pmu5scWv
sFrriatoUOq8qaynxY71Fhvb18yBomozcNOAIduBmVCcyI81HA0kQW9KZ42VuKqkcDlPN7Rs8O9h
xn3vm1JmXi2qCe4+0UtT96yuDWvt93Pbd7wvHAvdFtjdu9zLZM+r57mHpf4VGha9puw5skHVDWO2
j204cYdqH9k4Y/PG6L6vgA39vvotd/drcNvibDc3b1OHh6Oj0uJLcvs75KmYeJV0oKCRO5g7TGPu
P3O3aLgCmy+QkVt0+qI1vV4PXwVNt+hal2u/X18/6XvvvT6TnupoaEpTl5MmB1drrZbMh+n4PpIr
rO1DzV9Y4lpv27btLb617viyoPZpvQZcO9/vCRAgQIAAAQIECBAgQOAeAptwRvkLksUH7uo3gbUU
xbnD54Iqez5E6x2RWetxV/Fuqzz+mvHcoGAMmr+3autU1MpTFaflt+V/HUpt61rrYc3lHXsUAewx
V3NeyRdhG4rkB5mvk8Lxbf9hrlHlaRMM0c7TJmxX8sXT+mnIbX9pt572mQXg6HZTBVm1OD6+QWk6
pGVbmuRiOR6eaeZQvcUu9zVzoKgq+PomsX8fa8m2cJKm32vfeziqhsXLLJqADchtrwvpjXZx8Vbb
/GDjvttNKeNWu9CmtzTBYx2a72+1Ya0Oytz4jkOb9jjf9r9SQ/hmMdPBgRfKwnuTUsOCVu05skHV
tTFNr8BpoozYll+r82HGoP8NbQ5qc8ttevNRu8G0xk7TN8nBHWYxGhe8Am6b23VrmU4fuMbjRcxw
cjbM2N47mDvMRRtnDI1I7dqovJJuv3/Y6ypomjy1LndfOFf4THoSrjV1p1elSnagCyR3cLGQPV/c
T3QdeYjoTWnmq43uP11ZfEnS+iG1+UOBAwkQIECAAAECBAgQIEDgmgKbcEZlBT756Fv4FFz7DLzo
0bSOUclyNHyBsPiE2hoN2ZW3sZHlT/TRGHR8B1DvUNzKUxWHb4PT/93+BU76BXEDf2Pa46/U8NBd
5mrlC8CvnxNW8oXH/JPll31zUeFotf8wN4KNHU5CTrtNm2JB26XIkXrL54S3m95qMsdXi+m6oaV3
ocOsyQ7bSL2lc+Zv/5qWdAaKusIMDGwHZkLlppdexPcejuqoh31pugCbhj29n4YnbC/nWuKuewx/
29B9zvYO0GQyvxCNwfePxnzFL+Km/bO6fEYwTNmroLLusaqoCrXD2GVf1a+1cUawYhGpVPo+oafF
5c6Jfn4r23gqjNReu/jL76Ae4hWw2u3+i7y/yJ2vwaDJhTadK0/f3WcXrxveu1fe61Y/g4RN7b9D
5t5cVUen9w5W+mBXeFP0fneY3DY7IyOSmUUjt+hjUTtdBY2TZ7+X73Xrd/tMehYeGZ3+q7XtjKbn
smZ3c2p6T5h5n9z14r4ck+3DX1vfWmdeXqsj0nf/r96QHUCAAAECBAgQIECAAAECdxNYhzOq4YL5
643Sp8d5lbAW4p8+D1c/khe+AlvwtR53FfO2yvu+pl18rZT73rSt4lOf44NPrfr7UuF0QPj8jcqI
djSkeZZMk2TRoJ3majwPpnn+/f252QL12J7P7+/D/9/O3HB823+Ym5kVr+irpZpR82iVD9zcCwbq
veCiqN1kzqC5KuL5NQ1D8zxNB266hAoUA/UWlOZmVu+li1tKYdVyPbkHZuC+ffxr+MjXlvMd7e7D
UZnnmYu/fFb35TZ6OcczK7rz3GLcR3tR+W6+fjspeycX4iUbZ5T3luq8KTW9Wzu39hZj9zvPM4oD
L5Qtd8XCJubBKMXvs/pGNmzVjWwzt5GB2sf+iHe+T0/x1mgfrYH2tIz16j3ZQ9xyyy9cm2uh+5ae
3aDrOBWSt/T5e8HelWY/Yw1c4x0bZyTpuPhWH908215hS2+W3GGSm/oaauBKr7wV6rlFn4tqugoG
mlq81Rc3O2t7r754w/6122fSSXivLs9DFt1kSreX+KW17wZSmH4NX6L0vbivvkqZr/3ON6W1d0Ed
X/iMvcxd5XswhRIgQIAAAQIECBAgQIBAh8A6nNGwWHs65PPzsGid+25h+rSfPSL5cr7laba17w3P
nW49rgOp/dDGysvLi/Fn7GgBbNPntj95y7XynDn4OeYRlt8pNXat8bDVVxuFHXPzX9rsNlfD8V1O
8+X3rieo7BUQjm/7D3PzLaHYft83X3BhpCa6Tufy6n9xXzw2GIn+6TpwUZRP2TJmjy+1NrmT1Znm
Ok9gn19fhftkf73ZL+CSZjY+TWagqOJdfWAGxrYDM6F6i36c4eidtMeuVc7qvdw6L+cHHffOXkyI
l94Mi7GJ9EJs2V7hOL7F1crtBZ2rJHsdTFSbV45o6vTfl0au2cwNaOCFslB74V1C74pa78jGrbqN
bW4i9NeeewaGW+7h80/4+Sa+Me13DcblF6/B8wvg9+ndfXvMrvwpY8eXsx3vIcVb/ci7DneYwgf5
9FZwg/fYhVt67XXx8qugY/Lk357Umpm5t55OO331ssNn0rme/teFkat1qmX9SSr5aNuWVsnVXm7V
wMzp7mbvm9Lcx7Det/TVzwjVF2wHECBAgAABAgQIECBAgMB9BFbhjKal9fSbhcI6YPGw5S9bMgWF
z9ULuuzn/wbg/LenDSenS2jlv73Nrooea8l8HZBZmh/4nidHefz5eTl59f1Jo3/jYdsvfKOvuRez
JPt9TmnJvHGuBsNbck2/SgrmbnmhLW1u0/U2NW45PtvnrByWCgpfe0VLhYV81RIl7XMc/1jW3Ttd
By6KPr3sX2qn19xqraU4Acv3hMUcyd4mkwpyo7Mc0eirv4muGATZNLe/qFWfkpZlZ0dyP2u1HZgJ
9dvzwwxHpXeZnlTPGr3cMrsCLW8lDzvu174pZedVUnF6M1y9rUkv3aFZHY9q/5un336ERbX4Nd2X
hnoXv9EpvnrGb6wKtZdeI8L6x0d2/dKba1XvPX/INjtze2sfX/Nxy12vM/Zdg6O39MyHj+W7x9xH
rp0r7Xk5G5rn2bXT3nlert0dppCfTwe5cL/tHZEdX3znotqugt6mZqbHji/fi28Gog9ti7vt9iNx
9Q3kTl0+S2euy6SW7V4Tp7BbSwQ+N80q3Rx/cS/eVLOffJd32bDz+dj13rfiqbz6znD1z1OOIECA
AAECBAgQIECAAIH9BFbhjLal9dY8wHpN5/The/3/Wj6MN++t3Fjn3IZw7aGpSdtRKK0EJEdnv1A8
HJP/fmG99LOCbG10tpWp3foDfGPXGg9L6Sq9OnaysGNw+auG1rm6GczMVyOLIeppWXhptV1vp7ad
WvTb3zxZNAd2mjbzHyM2Xsd99Q5cFOtrvfalU/H4SmtrZW9vBo3Bjt56s8f/Dn3ZcNPE7qLm8rO3
2dCpr48DM6HhBfFRhqNzjE49azir73JLdn6vX84PPO61V4/1DbFXKTuzahWvX7YGZ3Wums/yX8BH
zS62ePPCcf1rNvNGp/hCGb/HKdgWXmFzb7R6Rzb3spKv+vq2pRtiX+3DG2es7jD5d6ed7cm/D8m9
AjbcPAOvhrP6bya7XoN98elF1YW3M32dalAq2C6aMXKHLC3G9s2rcu3uMI0bZ9zmPXbvLXqegjtd
BavrJ3ur3/HlO/lqINquJ4Vve77O+rrc83rJXpf5Dw5dyfLaa2721aZ35ozclLo+KV9wB8vc/3N9
n7re+kVRw6cqhxAgQIAAAQIECBAgQIDADgLLcEbr0nryCbf2Qa/8Ybh9ybPwJfuCoenDd7oclXTg
4j8taGtkRbn0dUDfmmhmfrQsGmQ3P6+MWJvAumHlRE08xa4wVxfNmtuU/zOkzBeW4Qi2/zB3WZ9K
ODQnnObZa3GXabP8cnC9ohvOi456hy6KlULtXrRWWx2fn4Xtt6lk7ErpnsxMa0INx/7YlZNHe3Oj
aZQvavE9YqSVr7jDdmgm1F8KH2Q4St/F5jvRdlbH5XaoqnDbXYzkY497ay/Our1K2VHJvNX4o9ve
7MdnddTev2t0YFk01/XKK+wmvbO5zod6l+lA9OPy/C/8ttSwPGDXyG5ejM+aQ2/ktpu6D9mW74cd
d+OheXaq3S03GIadrsHgpbZyQ0grbtvZr37djyZ3Std4Zk+PuHdta6fb+GH2Dnb9rXkOc+J17zDl
9+Rdd57STawTcCpqh6sgeO5GYRru+PKdvGHr/Ux64RvIXV+VMh86+t7PxNOspZtdM2f0xbf9Teng
HWx7n6r1fWpS+8fD+qcqRxAgQIAAAQIECBAgQIDADgLLcEbz5+PpA27jB73tB+LqQuqyc5UPyeF3
L5m/CF7++A4bZwyvNJx6ucJsHIItUeELns2vGv0bD4sn7vYLjdIkudpcPTeuVMH5d8VIQn6Bc71O
2DSEJ57p2AVXw+V04bTJX2KNeZ3TdZc5evSiSHrVpFg7fj0JG2Bzt+GpqpYy+uqNv2I+/7Slvsq3
1WFRmx+mU6qh0rY+js6E2qvhYwxH821r0Z2Os3ov881dN/+17zTEjzfuDb0IQcs3pdqUWi+yTUDb
OXzZrA7Aa9/FZ9u+nCC1y/Z612ymA+GPy/O/8NvCC3UFMMOUOSt4WakP0PVs6xN3G83KzIR6N/KV
ueVmbPquwbZbenWg5mJqF/1fq9sq7XhhSijCpo7cIat9/q207Spzh8ltjdFivJzmt3qP3XeL3nx6
brkK2iZPjWi3l+/hz6Qd1+keXa6BLO4ux3GYqm0ZlsNYBtOstZvNM2fkptT1SbkqtdutuOvFuOUN
hGMIECBAgAABAgQIECBAYCeB1WNNdipVMQQIEHhrgZav3Z4L6PT1XlMI5uF69nrDcTPipx73mymp
iACBVMAt13wgQIAAgapAa66iWpADCBAgQIAAAQIECBAgQIDAcwkIZzzXeGktAQLPIFD4m+1naH7Q
xs3GGc/Uj9cbjpvpP/W430xJRQQIpAJuueYDAQIECFQFvFhUiRxAgAABAgQIECBAgAABAi8qIJzx
ogOrWwQI3E2gf5PeuzW1ueIn3kDhFYejedwuPfCJx/3SrjufAIEhAbfcITYnESBA4PUESvsoTS8W
z7kr3+sNlh4RIECAAAECBAgQIECAwA0FhDNuiK0qAgTeQGB6vO8rfdf4vJvUv+Rw3Owyet5xvxmR
iggQSAXccs0HAgQIEDgJTAGMj9Wnoum14uPj6wcXAQIECBAgQIAAAQIECBB4NwHhjHcbcf0lQOAa
AvO3jx/Hf68Uzfjvv2d7KPSLD8c1ZnBY5rON+81gVESAQCrglms+ECBAgMBWYPPqcPqYNP0/0QzT
hgABAgQIECBAgAABAgTeUUA44x1HXZ8JENhbIPkLsL/vG1/sq8an20DhtYdj78mbLe/pxv1mMioi
QCAVcMs1HwgQIEAgEijFM14ryW78CRAgQIAAAQIECBAgQIBAs4BwRjOVAwkQIJAVSJamXvCbxqfb
QOG1h+Nm1+HTjfvNZFREgEAq4JZrPhAgQIBAVmCb0HjBT0vGnwABAgQIECBAgAABAgQINAsIZzRT
OZAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg0C8gnNFv5gwCBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAQLOAcEYzlQMJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAv0Cwhn9Zs4gQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECDQLCGc0UzmQ
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQINAvIJzRb+YMAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgECzgHBGM5UDCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQL9AsIZ/WbOIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAg0CwhnNFM5kAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECDQLyCc0W/mDAIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIBAs4BwRjOVAwkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAEC/QLCGf1mziBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQINAsIZzRTOZAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg0C8gnNFv5gwCBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAQLOAcEYzlQMJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
Av0Cwhn9Zs4gQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECDQLCGc0UzmQAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQINAvIJzRb+YMAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgECzgHBGM5UDCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQL9
AsIZ/WbOIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAg0CwhnNFM5kAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECDQLyCc0W/mDAIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIBAs4BwRjOVAwkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC/QLC
Gf1mziBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQINAsIZzRTOZAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAg0C8gnNFv5gwCBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAQLOAcEYzlQMJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAv0Cwhn9
Zs4gQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECDQLCGc0UzmQAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQINAvIJzRb+YMAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgECzgHBGM5UDCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQL9AsIZ/WbO
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAg0CwhnNFM5kAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECDQLyCc0W/mDAIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIBAs4BwRjOVAwkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC/QLCGf1mziBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQINAsIZzRTOZAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAg0C8gnNFv5gwCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAQLOAcEYzlQMJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAv0Cwhn9Zs4gQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECDQLCGc0UzmQAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQINAvIJzRb+YMAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gECzgHBGM5UDCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQL9AsIZ/WbOIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAg0CwhnNFM5kAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECDQLyCc0W/mDAIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIBA
s4BwRjOVAwkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC/QLCGf1mziBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQINAsIZzRTOZAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAg0C8gnNFv5gwCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAQLOA
cEYzlQMJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAv0Cwhn9Zs4gQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECDQLCGc0UzmQAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQINAvEIYzfr4+jv++fgolTkcVD2srq7/hT3PGGSCw/PkqAj9NFzU0K2CIn3pyvOfw
FW5ZudEcOOWpJ4bGFwT+fX8e3j58fv+7hVPXK+zuE3V7i9i9ilsgqmMj8J43fxOBAAECBAgQIECA
AAECBAgQIECAAAECBK4uMBzOOC/AVDIcb5/NyCxUHX98o9Wrq88iFQQChvipp8X7Dt/A2vrAKU89
OTS+JHDbdELXK+yuEzW+RexahYl2F4H3vfnfhVulBAgQIECAAAECBAgQIECAAAECBAgQeDOB0XDG
vG1GaYONKcHxvimEcKHqtqtXbzalH6O7hvgxxmGwFe88fAN9HzhlcGCc9vACN04ndL3C7jhRc0Xt
WMXDD/VrNtAIvua46hUBAgQIECBAgAABAgQIECBAgAABAgQeRSAMZ0yZiuxTNxbZjOxjTWQz/gsX
qm68evUoc+2d2mGIn3q033n4Bvo+cMpTTw+NLwncdm276xV2x4maK2rHKkyzuwgYwbuwq5QAAQIE
CBAgQIAAAQIECBAgQIAAAQLvI1AOZ2Q2vEgfaVJ4tvzbP9IkN41uu3r1PpP5gXpqiB9oMPqb8s7D
d+57x25H78zVP7le/IxHWNu+wa4WN6jixSfKo3bP3exRR0a7CBAgQIAAAQIECBAgQIAAAQIECBAg
8CICI+GM5bYZv+mMeBlPNiMzRx5h9epFpu+jdsMQP+rINLXrnYevvm3ShvCduZrm01sd9ABr2zfY
1eIGVbzVrHmczrqbPc5YaAkBAgQIECBAgAABAgQIECBAgAABAgReU2AgnDH/ZfXP9+ffxhlhOMMj
TbIT5gFWr15zMj9Orwzx44zFQEveefhsnDEwYZwyCTzC2vYNdrW4QRUm1V0E3vnmfxdwlRIgQIAA
AQIECBAgQIAAAQIECBAgQODtBMJwxn+FBbo0dFH4I+tyNmP1XJSODfS3A9Swj0fp6/bC7wrLTA2V
/rV0VcLmrEO05ePrJz/tqktd+TFofPjMupHrtkRA56LPLZ87dkFfLqkolZ2acPphrkmtk3Dd2f+m
yyMYvYEhTuZJ1bO1zadBbJsC+dHMbYqTzpG2OuZrYR6OhVVh4rTWMLWqUynpTfPwdbepfokfbwaZ
bYiCSVKch0P3yfSWdb4rhVfWsvTaPWp8OI71VMpvuIOfaPO24egMXBfNN8b++bM+o3Snnc1qkyrp
eP8wpXNjeg1vXdve50W56xV2M5GWXS6LnqXKt4jLqugfgvhK3L6OtNzJl69rmXtR7WKfrtbTDWR9
fNrFFfii9+X3hK1QN3jtfruPDDpMgAABAgQIECBAgAABAgQIECBAgAABApcK9IYzpqWJvy/P88GA
/CNNMmsbTYsHq86ul6vOS3CbwENhQWHRn2X5cUKludK/wpYLUJlTy8sQpyJyB8WNzCJHQZDyalpo
lzZqWVdpges6FUWqB63twsw0vl2TcDEC8RhOvR4a4mmeHJqd8+xqczL30qsizgLlR7MSFxiZZseJ
XGGcL8RsFZmJ1qsUL26uzJYXX1evy/fnrtb2zMNFtUO3rL8+R80LbkSF67qrgzmrgftG1/yv1Btc
ldnrouXG2D9/sgMYvioMmPefEjbp75qsLt6ftfd5Ue56hV0cHPe5GtCo3OGHq+gfgmDaDt/JO24R
nW9IUpCglrN39lV808suqJ575uBr96XvwJ1PgAABAgQIECBAgAABAgQIECBAgAABAm8nUA5nbNby
p2/GD1+q58IZ08/Xi0f5danqFhLtX9Jv/1g4uw6UfB+/WecKlwQrHVj9afaq3szJbX8CHR8VNbLS
xnVHK6tpUQ1z7GG9oHG1jTMyFeUXlb5+Tr/LDuwmtRDvY5JkPPK1nSsZGuIkRpLz7L1wSott6xzU
PAG+vzIm0SLw4DT7nSF1xtPF3tOLv1N6lVrvKcms7ux16fWks7Vd8zCpt++WNefJvlonQ+5V4PLh
OPZi4AZVmGLFzUkWwzVwXdRvjP3zp+8i6JxUY1dN6a4bvmxGl8EuL8pdr7DJwXmm2i5e5Tv8YBUD
o1Ym7bqT1y7U5bua5D4UtGE9/FWQRch3/fKzeTvRCdV1zxx77X67Tww6TIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIXC8ThjMx62zp0sf7vU2tqP14/yWP+Vry2MjJ3N/kmffkNfrJwtH2AQmkf7dXvKrGH
hkrnxeJFtyprjesRLR0e/G6xbpY2MgGLNwKIcxVh9ee/R/36+lysdf7797crQu5fedlupKJ0OWUz
2J+fh8at8kWpz6LH2Um46uyywOmsRVmdQ3xeHPvMeA63edX78NLYRBrm2ZGtd7XzRc80O/exhXEe
krSGTC8W6/FtI5ubqC3Lxpn2N28ANDymPYC/Hey7T262NJkh85PhXMXyttLfwf3vG23zv1bvJuxX
pWi5kBf3pezNebFunszq+CIYMN/tlPNt8nTXbXglz7weZF/Bktk8SYRzL3f9Tgef2hgPQXXvjL/5
smcVA0OQmbKreEHTnbz7FtH3hmQa5FPWa/uebLqhBfeaS28pN3jtvvgduAIIECBAgAABAgQIECBA
gAABAgQIECBA4O0EesIZ03f/m+fLL75Ejxes52WmaPUjXuDLjkYmPHI6frvqk/wFZVLmcj2purze
W+n6qSbZ1pUn3bmVh8dBLP5t0XIL18fTtuP3+8NKkCBaQSstn+X7sntFczNKf2CbeSRF8yRcdHa7
5BiuMJZjKBuiimfmelqOaWP+J5jDae2b/sXGF0yzw0p3G2N+wkRXYr9SdqrWV42D2RNeXNUqoscM
TddpOKYdgPmtjZZzJxOu2XQyc8EVL4H2DuasLrhvbOdZ+R6+aMJl18U6FJbcaTeBsfrNOSgteD0d
uAT6T5lZilsSNUQc9nhR7rn9jtzJc5Oyfotovcvt+L5oYMZecIsoPN8oToWsb/tplKQhGTsANTLi
na/db/eBQYcJECBAgAABAgQIECBAgAABAgQIECBA4HKBjnBGsJATZSri9Z7KElstI7DsaDnKUViA
jkIkX9/fhy0Wor/OT4/vrjTuUw1iM6TxGla4qjL1PPN3y9Hvy4sRYWvnVZWGv4+e+rN3RcV1xXKo
INfsqLflIELo0zvERc9aYcU2b5dIt4dXRjOY9iPT7ALGYH1/04sRpc40wEivM1WMtHYAME55TG2K
ggqZ/MX5nOgeWJqAPRda5zr46fDiBdgyc7Ivof3XRbLNRdDxgflTvMXND6D5e8JYNWUXvRoNzMNi
BmmePS0vDKXXlsYX5a7b78gV1HkB91cxMAR7ztiBW0TyEK5lS6L7SRLAWL8WFSZL6c7Uc0vpH46+
96GXv/1WAgECBAgQIECAAAECBAgQIECAAAECBAi8o0Aczgi+sw/XlraLdZklqOrfI9YWKVrHJvPn
mKcfJ9/snyr8/cn8v86VVFu7ak250uUSYW/Z04YXq1WJAKy8lpdZwCu3p/iXyeFfxVeXjuK/qe6u
qJyVCf/MtkofTcLiamS8tlWtJzd9ChsyZP8SvbTEGW5SsR6g2qzZ9KZ2QrhOPMBYXutedKMq3nN7
iQsb6nV8MQy1dgSwcNOs3LJyS6BB09t+tGzK5cNRvlVXghVNLya1AS/0O7ox1ooLrprKLW7Zi4FJ
dcEpmVt/vZNpoy9/US6+aqxvmW1XUMOWH5ktsX671l3FwBDkJ29Nv1pZPKM2zwbbjlvmvUXyqKkt
a6G1hfc2Xa+DbcMRbvvWNA2a7iMOIkCAAAECBAgQIECAAAECBAgQIECAAAECK4HWcEb8PffmTxwr
2YzDnvzFfy1/cnvqQvJ3kWGR0bYX03fuSSJjE84oLWJ0VNr1Z72FiXkuZ7FeUFgaLBBu/yS1vEwa
Lg9Gf9dav652rqi2cFnKrtTmYLr/emZCr6dgbduVkk/JM11D77lwVud1TYlVW9czrSb/e3p+mgVP
NAmPP7ahtRejSuGoxBN1qNfxqA+1dmAenmu/9JaV9mJ729nrQstdIQP3jY6Zk78uq3e5PEUpm9Fx
JTZMubn5A5PqglN6ti/IE28Ee1+Uu15h266gpvcfuTnZXcXAEOw6YwduEcntuvaGJLmB57dwKl0s
81lDUG3DUd0yrf7exhEECBAgQIAAAQIECBAgQIAAAQIECBAgQKBHoDGckfkzx/UKVu2w6rJ4vHi7
6VDTd/XRgvn5Z6cCjv+5+I/cThWbpeK4L0mlYcaj869XDz1ftS9ZHkkXFqqrieF5xfbERY50If/H
xocedldUXrJNSpzHo7ZGnYxnskJXXiENf9vrUzh+rM1L0rlf4cJjtbWrAy6bZvHaZx45ANgWcYFS
cKsMQcZ6Hd6Ix1o7MA/nO0f5tru9ZeVXqLc4hYzCLnf7gRtU1/zPvVj2XheJdvBX9wPzp3qLS1s+
MKkuOSW3r0AVbaF9bsHwi3LXK2zbFdS0Y0Kum71VDAxB4b1dFb/IVbxYVyptb0hyL+vHLhSstjN/
DKptOGyc0fN5wbEECBAgQIAAAQIECBAgQIAAAQIECBAgsINAJpyx/H48+yeIqzWnTDZj/RfwpXWA
hr9cLSQz5oeUfCz/InP1ff9ydWHxX5k1sf5Ku/6stziSp7pnmuFMwKZN5RXAsJ6uRcOpXztX1LoQ
FcQs6kvGSUKo3OxwybXXp3R8YdatuxFdOA3RhvUC6WYirptXlf8toXOaVVauq724UGnZ5Xg4hnod
X9RDrR2Yh6W7buY+2T0Zyk+wqV9p9bv9wH1jZq/OnPxtdx+KqfyB+dNyyqb8Ovl0cxuYh9UmbV6p
iq9rl74od73CjlxBmdbniuquYmAIdp2xA7eIQ/1tb0g6Axhzz7bTbAiqezjCu9kO77EVQYAAAQIE
CBAgQIAAAQIECBAgQIAAAQIEFgKVcMYh4pDNZqx2PMj+pWJ1UadjUBYLbsHfuWYaka4nrBfeTv99
WCmMF+VGKi3+nWrTH+ieVdZtijkbkLer4MWT4kXzhnqC4dy5olojgmGsnRLOwfJJ4fJPbz2l43vL
CvuwXqKOdqPPrpJv5kBDkzqn2TbLEXWj0IuGJrXfYHa8uOJKh1rbPw8vuGVdtnHGtOLbdY/LDdDA
fWNdVHH+V+ptvy7KewM1DPrqqqkkllbtbih/3dMrnNJb5GUvyvFEy7Wh3LbsW5dofuxWRa9X8S6W
YhZvoOtHux0DPe1vpbZvkjL9KE3hQteDV9UhqP575r63rvbXHEcSIECAAAECBAgQIECAAAECBAgQ
IECAwHsJNIQzzl9yV76/z38V37fOVPbPB0WO5+UamyxdpFmMwznJDyrZjMxaXVRp15/1VubcYt0l
8wehDchbu+Ifl8bUxVOy/di7ospqTVBdg8+29ZXORq3o9SkdP9TmzBgkS9TJOly1tetVy4YmdU6z
yhNvlt2JetHQpPZ7egzSUEXtxnRuQkNRe8zDWnOab1lpY7ZL2HtdaLkBGrhvdM3/oXrT15n5FeHC
C3k9YNUrM236wKS6wildTU4Q/wxP5244k9+t33+E1eXaUG5bF8ZuVXTVWruFVfE3F+/ILeLUipY3
JDtunDEEVQHZ47W7NiZ+T4AAAQIECBAgQIAAAQIECBAgQIAAAQIEIoF6OKP4R6XzL3++P6t/grnD
31JXVuWnNcd1jOL8Vf3Xz2YZKFkYyvz16Uil4TlDfwG6SI/kn9VeLTtYjCmdM6+DR1stREmdwhW2
d0VtfxWbPJ+kKwIwdaStmoueWl+sojqma/Km0pKxK17c4Z451SZ1TrNoL/nOXlSb1HHrz5RVraK2
0jk3oVpU0Nr+eXjJLSsTQYu6GFUz0sHcCPXeNzpnTnZi9F8X5b+6r5pscPvWlqvlb3u6+ynTa2/z
y/xFL8rPv3HGrhs19M/YkVvEaRql76GyFZciFYXWRhN/YK5WcMPLa6SajtcWhxIgQIAAAQIECBAg
QIAAAQIECBAgQIAAgYNAJpwxZwA+T5mLeMVu+pL96+t4XHhY7S8fKwsLyVCVS5rzBJvswLQO9H0M
kQSBg8/v76+wDyOVdv1Zb3UunoCmZEm0C3llbWy2afpj74RyMaTVP9AN+1I4a6yi0pBkZsHAJOxb
ID30vNenfHx3m8tX0mbxKZoUyQBOUyqZAvtOs7/Ktktio73IrAq3316ywzfQ69xF3T2m1UkVrCle
csvK3MSjyVC8y+XyWzsMR3KhLV9zOmdOdYzi17Nmiqn8gfnTlEs5z/gLJlXHMJV0K7eS4svC18CL
cnyp5m6nA3fy2tzIxT8ru2ul96iBUau1qv3iHblFrOdz8Q1Jy4t+4WEqF0INjHjva3f1bZsDCBAg
QIAAAQIECBAgQIAAAQIECBAgQIBAJFALZxx3wwgfSf5X3LTodDqs9tyP4Pfzqk7LX9wW1ofStmzr
Of328xg2WVZ1/N3pV8F63ECl4dLa8F9mnow+ixGYaMnwPOSJTctDLRbDusPGGdllj+GKsqudSTRj
NZIln3gSlscr7FTvEFeO721z8WLaTuOEP3vFxEmmOIcVT7Nuxs5ezHeh3W4vm1tR/8WVfb3pHdPa
pi/FeRiI5O6TuZjUoSPzSSlNPLL9HcxZdd83emdOpuKB6+KCC7l8c96+HkW+A+b9p+R1y6+9tUvh
ohflppen7ltQ/t1irqiRKvqHINeuC2Zsxy1iqr3lDUkBZPtKtC75GV67faIgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQJDAm3hjEzmInnuQTHCsTguLat/TSdZQYxWCfNZklJV6YJ+9IfE/ZUO/Ul5fgAX
DWxJyuSQl72LFtumlodBkME/Lr1CReGKZjrI20RRZvU5OzMu+OPb1qe+VD1H25zZHWYZtFpNq6TR
6W/WQYXMOuDCPvqz55E/Kc8mQ5YXQa9SLQwQDF9vrwt3497WDszDaV+BFeDqAln08/i7cwAsGa3s
ZJh+sZoivR3MWo3fN5rmf3UabJKJVYrsK2X//AkVqw1YpaYaX/YaX5TjgS3fdUtvSxpbF73idb3C
jlxBmXbX5nzHXe63hr2ulJE7ef+7mpmk4Q1JwTxn+Fd+JtLRCzUw4vlWTb/JXt1D772dRIAAAQIE
CBAgQIAAAQIECBAgQIAAAQJvKpALZyy+fc9/J738kr609cV6WfAcoyjvzBGMSqWgXEok8+3+oYLS
UvSxBb2Vhn8aul5B6fiqP7vyvQSqNHNTYfb435EMV0nKfx6cv4auUNFac5pQhf1F+ibhwN9C9w5x
g2dfm+tTNblI59rzmN//tqPaN80GGLt60XKBtmzKs7oRHJIKae/7el18Pekb0yHA7lvWaQr8djnf
uhVjeJfbcTgK0yB3g+qfOYWXmOSpDauXq812FpdfyJUNDNYN2OaH+ibV4DDlKvn8/jk+Lyx+EEzm
arjgRbnrFXbsCgobnbvDj1YxMGr7zNiBdzVJxfU3JAWQ/H2j8FiwPqiB4ci/dk9VN76MvOlnCd0m
QIAAAQIECBAgQIAAAQIECBAgQIAAgUaBlnBG4Svp9AvtWtagbwG40P7M1/R/9ee/k4/+CPtcx/y7
fB/6Ko3bsSqj46v+0irWQiqLnFk0i7p1bNfpNwuRyt+jlubcNSrKlFlamOmYhKW/701WtqIn5Ewr
qZUhbvPsaPNhBPJzYDkJFrVHJ7WGstKF4805Q4wdvdhexKt17Np9KZ215Su09+IqXA8dYzoKmItn
ZO6Tp64fZmx4s9vO5Z0utOJLVd99o2v+5+oduC4uvJCziYbcNInvKx2TavyqiSr5a05DPGUDfsGL
cs8r7PAVFM2Q+BZxQRUDo7Zu18CMPRbR964mqbb6hqRwRZSsdrqlDA1H7uY/FdbzStL4/tthBAgQ
IECAAAECBAgQIECAAAECBAgQIPCGAvVwRvkb6fkL7bZvrldfgLedFIzLspxpraqwJFD62r/wp5xp
3c2V5puRFNHV964/3lwv95QjAunRc5vOP12cO7IANwNepaJ0TI5trSzMbJekMgNRXnDdZYg7PHsv
nM2S37aXmyHeWhbviG3TbJTxUHVDLxZN7FUq31nCidHW66aXkrbWXgTYe8uaurzoZjkL0Jxo67rj
Dd034pM6d3QYuC46LuT++ZMZw8wUa5tUl101wY2iPEtzl8Pwi3LX7feiK6jtFnFpFQOjljRsYMbO
ZzffIuI5k3lzURjZwrut+ixqgxodjvDtWdd7r6Ybv4MIECBAgAABAgQIECBAgAABAgQIECBA4K0F
cuGMt0Z5vM7XFy0er833alHHQum9mvgY9Z6gBpfMH6MPWkFgbwHXxd6iyruuwO1nrDck1x1RpRMg
QIAAAQIECBAgQIAAAQIECBAgQIDACwsIZzzD4Dbu7PEMXbl6G1k1Eod7ozSe6zACryrgunjVkX3V
ft1+xnqRfdW5pF8ECBAgQIAAAQIECBAgQIAAAQIECBAgcH0B4YzrG19aQ/05HZfW8Fznl/5qd7Ky
H0RtUG//59a1Fvk9gfsLuC7uPwZa0CNw6xnrDUnP6DiWAAECBAgQIECAAAECBAgQIECAAAECBAgs
BYQzHn1GTE88Fzc4DdW0NvSxIkmeGP/18+jjeu/22Zj+3iOg/kcUcF084qhoU17gxjPWGxKTkQAB
AgQIECBAgAABAgQIECBAgAABAgQIXCIgnHGJ3vXOnQMIH8d/ohkz9gbnZDT9P9GM+sw8r7Gxqls5
4n0EXBfvM9av0dObzFhvSF5jsugFAQIECBAgQIAAAQIECBAgQIAAAQIECNxfQDjj/mMQtSDZBOIv
cmAFfYlUimeIsbRM6Rv/uXVLkxxD4O4Crou7D4EGdAncZsZ6Q9I1KA4mQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQJZAeGMx5wcyVqIsEFmiLYJDVTNs/kmf27d3BoHEngMAdfFY4yDVrQK3GbGekPSOh6O
I0CAAAECBAgQIECAAAECBAgQIECAAAECZQHhDDOEAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIHBFAeGMK+IqmgABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAgnGEOECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSuKCCccUVcRRMgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIEhDPMAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIDA
FQWEM66Iq2gCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAgHCGOUCAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQuKKAcMYVcRVNgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIEBDOMAcIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAlcUEM64Iq6i
CRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQLCGeYAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQOCKAsIZV8RVNAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IEBAOMMcIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAhcUUA444q4iiZAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQICGeYAwQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgACBKwoIZ1wRV9EECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAeEMc4AA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgcEUB4Ywr4iqaAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQICCcYQ4QIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BK4oIJxxRVxFEyBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSEM8wBAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgMAVBYQzroiraAIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQICAcIY5QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBC4ooBwxhVx
FU2AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQEM4wBwgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECVxQQzrgirqIJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAsIZ5gABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4IoCwhlXxFU0AQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQEA4wxwgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECFxRQDjjiriKJkCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgIZ5gD
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIErCghnXBFX0QQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAAB4QxzgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBwRQHhjCviKpoAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgIJxhDhAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEriggnHFFXEUTIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBIQzzAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwBUFhDOu
iKtoAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgIBwhjlAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIELiigHDGFXEVTYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAQzjAHCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQJXFBDOuCKuogkQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECwhnmAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIEDgigLCGVfEVTQBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAQDjD
HCBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIXFFAOOOKuIomQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECAhnmAMECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAgSsKCGdcEVfRBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAHhDHOAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIHBFAeGMK+IqmgABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECAgnGEOECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSuKCCc
cUVcRRMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEhDPMAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIDAFQWicMbP18fx3+f3v4aq/31/dh3fUOQVD5lbe2p06f99/VzS
krNjUMrP1/qHhYOLbZgG69SPsSaHtY826RIz5xIgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAgZcTCHfOmAMM9aX+54pm/PffOsxQyma0hVNyc+Issyrl+ON10ZmDK/Nt25mxJoe1jzXp5a4Q
HSJAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAhcKxI81aU5nJPtQ1HMcFzZ1l9PvvXFG
bjeKkV0qor6MjYKNM3aZXAohQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQKRQBzOSDaY
KK32P100o7Ffu0yVrt0oRnapmPTHdstIOtnV1F1wFEKAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBN5HIBfOmB//kV36f8Joxn/7BRrGpsieG2ecy7o4mzGN9SKIM7KXxxiKswgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAwEsLZMMZtRzDM0YzbrlxRjhrchtkXLRxxtiTTJL22Tjj
pS9xnSNAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBewvkwxlzOuNju/z/nNGMWuDk6oOx
58YZI4GOuINhq2yccfXZoAICBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQeBOBQjgjv89E
Es0oPFMjOerj71/u0ErKIEoJnE85p0bOx3wEOZJ0IKcmje82MVd16FSuW6tObc46nntqxkZgSbdq
7Np10Y4LMBfjsy2nGgbZAfdNLjrdJECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIH3EiiG
M/6bMgXpwn1DNCOTRZjjCClyeY+GMBNwOuXQqmVd5dTF+dhCpCQ//JlQRJqymE9edqqYp5i7cGh8
bJf0K4N77NM45jLXEpSTogdKF9m+10WntwQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECDw
XgLlcEb0HJA5G5CJOOSTGeFOExfs9fD1s049lLMZl+ztUOnVal+Q0Y0z8rVM2LmMyKHrF2AuwhmR
1Hq7kuWF4jEo73Xj0FsCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQaBeohDM26YxaNGMR
HUijEknsYBnqGN/r4fPr63PxYJF///42j8j/G9/cIWn9Mv+R9DfqbeVRIcemTs36PHRn+WyWqea4
3lVAZhxzmS6JpErBj+ozT9pnpCMJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgMCLCdTC
GcljNn7TAXMUId41IxNVOJkVH5JS2YYjjSZkAyCVsSk9l+SYikj+hRWGG3NEgYgwJJFJTiyatWWI
z4p/OrBxRvwglFL7g5EaT7282OWkOwQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAYCtQ
D2dMq/cfX19f5/RC/PiQKWaQSVpEv79gr4eP1bNEauNbezJJLptRjh6UngDStHFGOdES5y0yKYwB
zLCkcqXrwb/kaTG1MfN7AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECDw9AIN4Yxkv4xT
fCGOZiQpjp+MS7DoX84TFHeg+Mi0o1L5ZpOM4AeZdMmm5DTvkbSmZ+OMmS2stGfjjHgXjKnR7a3K
DMrpx6t2eqTJ098FdIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIErirQEs5YpTNykYiG
Z1ts91gor+yHRY5u1NDQvhJ17aEoSWShYzeK/2bcMJvRtXHGBZhde3wsZkA5XXPVyatwAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECDwDAJN4Yw0nZHdraIlM7E9pri0Hxc5mAZoaV9mxJqe
h3LpxhnxZh37bZzRgZkVPv0i6eoFqs9weWgjAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBC4XKAtnDE9L6PwuI+WzMRma4e99npogJgSFn0PQ5mfOrJ9+smvRhROGNo4I5aNsw8Ztb0w8+Vs
nmty4W4kDcPmEAIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg8OwCbeGMlv0RGsIZD7Fx
RiFfEgzm4mEmQawjSieEEjmeMlscfsics9cuJPlyzhpniIYxf/YLRPsJECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgMClAm3hjIYNEhryG9MhUz6iuNfD9vC/zpa3h8hzjG2cEbdhriUq9WYb
Z3wswyJ7YZbKWWydMToUl85Z5xMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgacSaApn
NAQv/pse/5F9akgQdCjtvDDvWbEocXCzhlrKIjNoldqmbEa6HcfNNs5Y7QGyF2ZDOYeaGxI7T3Up
aCwBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIELiOQFM4o2nXicpBc9ZizhQUtl5IHiey
yCCM7tbQlC/ZEJfPStqY5Ed23DgjDrxkGrUXZln4NMpfP2Og15nESiVAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAo8s0BLOaNx1ItxG4tT36Xcf6bM4sjmA5PDFCQ37c2SwB7d5KJyWtvFK
G2fEPhm1vTDLm4Wcavn8+vr8+Pu32r3jkae6thEgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAgfsItIQzmjbO+G1+EleI0gqHxfzFI0qiJ5dMGYNw+f+2G2f891+8OUba03WvihtnLLs/F5+J
OMQ5iUx6Yhxz0arKjhgJyGY4pxncmOa5z4xXKwECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQuLFAQzijY6l9sePFIbaw+LfJIGSP/81whBmE8rYOebvBjTN+C6x06di/JHIS1rSKNEz7TTTt
UrHanaJth4zUvYa5GJaaVH5Tk1m/tIfKjee36ggQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAwN0FesIZi10vMi1fxxDmkEC8O0SUfTjWc/rN4rRbb5xx6GQmnvHXsCBcUd7t4sRxkqzsUtG1
cUaupXnMsfhLMsC5R5pEe3jcfaJrAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQuJdA
QzijtpfCtunrhEY51ZEePa/3n3+6OPf2G2ecOrcMaExt2oZF8vGR4KEv5axJ/NtKPqUDs/gAllzw
IsmqZEfVxhn3uprVS4AAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIPKdAQznjIdmvUnQQa
Ny85HNay1cqduqFaAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBwMwHhjJtRv0RFrduo
/B0nm/ESQ64TBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIHCpgHDGpYLvdP700JRK7sK+
Ge80K/SVAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBCoCwhmmSKvAedeMj8/vf6VzRDNa
RR1HgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAm8hIJzxFsN8QSen3TI+Tv/K0Yy/wyvh
jQsa41QCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIPB8AsIZzzdmt23xtF/GMZtReaDJ
bdumNgIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg8AQCwhlPMEh3bWISzrAjxl1HQuUE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg8KQCwhlPOnCaTYAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECDyHgHDGc4yTVhIgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQJPKiCc8aQDp9kECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIDAcwgIZzzHOGklAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg8KQCwhlPOnCaTYAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECDyHgHDGc4yTVhIgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQJPKiCc8aQDp9kECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIDAcwgIZzzHOGklAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg8KQCwhlPOnCaTYAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECDyHgHDGc4yTVhIgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQJPKpANZ/x8fZT/ff08aZe3zT73dZcuzXCf3/+aiPrPaCrWQQQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgMAjCOTCGf++PyvhjMOvd4kz3Nvh3NfWMEWxvSlck073CffmUj8BAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECDQI5ALZ1Q3zjhHN3ZJNPS0eP9jr7RxxsdHi00K3XL8
/t1XIgECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIHBNgUw4Y9rOIbf5Q5IpaNof4pp9
uLDsq22c0bKxyCIE8+ySFw6E0wkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwEsKZMIZ
58xAaS+HKVfw5JmCq22cUd86Y/nwGBtnvOQlplMECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAg8O4CcTijunHGga0lwfH4vlfZOOPz6+vz78Ev5dzKCfDr6+vwkJgnD7k8/lBrIQECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQuIdAHM5o2xRj11jDPTp/qPMqG2ecwxmlzTDOfF/f34cg
h3DG3eaAigkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwBUFwnDGtHFG+UEb0c4ZU+Lg
59joKeaxiR7MvzkkE3LPAFkXmBZZDTQsHxsSxh82CZPlOX27WSQ7jhy7VwA8Hfv5/XPKZuSOvQXU
SrXS9NXIJq7nvUBOo7+euKvxyOpUZ1Hb1i5XvG4UTYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIEmgXCcEbbxhnhU01OPzwsuy9TBXPMYR2ZOIUz4t0j0gL/i88MAxTZSlaJgMXGGZscROeO
FglJJaZwxvltTX7rjptAZVCPgxJnU6J2HVi3SZppIsa0mfxKbRY1poearwIHEiBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgACBKwpE4Yympe9psX0RdphX57ObVmRX6cMNNJLl/nxUYbMDQynV
sNzOItk4I9+w8gYi8+ik+zlMG2P8Zha2/4JubSq5BVSwY0YalYmyE3nbr59cpytdWUdAqrOoLT10
xctG0QQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAoF0gCmdUl76TtfZ4H4rPr6/PxaMx
/v07ZRSSU5dL8smaf/qL8x4KpwKXezlkGppGAZKywhqmvS4+Dw1uKj+ru9hLpLR1RrQzxCqhcAuo
RTSjaTgWu6HMZ5z7c0LMFrXciWPuYucsakoPtV8DjiRAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAlcVCMIZ5W0n0o0VyjtWBI/ESLeW2HYreL7Hoi3bDSyiJ4LMp2wbsElzDJSfH45l7wpb
Z6S5jWTrjqTkW0DNTyGJn15SHI6NbTawU4z6LOIs5/4vBiV+sMpVrwmFEyBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgACBXQVGwxnxkz6yeyEc2xwuxU/diQIJmf00TudEwYbyth/rwEF/+QX9
VfdyW2ckjzRJTBYhhFtADQxH0TYOxcTZkxmxOIQfrc+T2fWSUBgBAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIENhXIAhnpE8FSffJ2P7vzdp59YEo2cZnHkUy/Thcpg8yDOVYw6b6tvLbdm9Y
h0syW2ekjzSZN69ojiHsBVWeRmEtFdto8KOtTRY19+dr9p3/SiNAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAlcX2IYzyk/UODQo9wiLhlN/z649NiXJKUyHhtmFoLrObEZb+Y3BiXXdyx0y
TiO5/mEpvXBNqHliNddSsw1yFkM5n7ZZdPUrQwUECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQGAngW04o7YGf6w4fIhFdZ+EpsX6ZJ+Kclu2v609RGON1lb+2MYZU4YlzXasd9PINfjaUEeI
rlqqttswSi33kezFkiBVZ9FOU18xBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEDgNgKb
cEbrvgVBOqOyfF+IAvwuzUf1lgssbJzRFqeYEibx1hitEqeBykZFtrmD+SdhOuQGUL9t7q2lmplY
Pq/lT6Up/HFMaMwk1RDIbS4MtRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgb0ENuGM
aUG9FnDYHlhcvl/sohCUHcUUynmA7Sp+Z5xiCg/EXW3bQuQ8EFHdpxLOxRe2lkhacAuo1bNlmoaj
Fs4InuJSOyWcxEMn7XU5KIcAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECOwvsA5nTNGA
eDeJpAGbcEZxy4NauVEmpHEnjjla0LnpQvfGHCX+KMqxfIjJ+pEmv6UFJ90Ean4qTWaYg+GohCYC
zN6szIG3cwz3vyKUSIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEdhZYhzOaN87YhgiK
q/eVpf2p2jQs0LtxRm1hf11eufzLN86YwheH+MhqG42/cSzstvGR2bhkH6jKniHz80i2j2TJNCzM
lIxsgjFyzs7XhOIIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgMCuAqtwRm3fhqnuKSQw
Lda3bZwRZg6SJ3kEz/eobO6Qllha2d8kIbo35ijBx1GOUxV/TQyyGcWNM64KVd7TojwcUTojPiMM
n6SIhY1Dqhu37HoVKIwAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECFxRYBXOaNw4Y45m
zIvow/tQzIV9fFy0cUayFcVmbT/YcaJ7Y47CMOTSDsc6Pr9/vj+P/+NfUkh4UmHDjh2hBmqZt9NY
50aSaMaqh+E+HyeB+azWeM0VLwNFEyBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACB6wks
wxkNG2ekK/EfyS4KlX0oMnsrpHmDv/TCZRtnJOmMNAiRNnmu4AYbZ5z3y/j8+jpkM1aphjAhcRuo
/lp+p2AyWHNPViO46mJST5S72QRWKoMy78VhZ43r3ROUTIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQI7CyzDGeusxN/ieeFf15YHTWUnJQ5ubFGspr3BtZTAchiyjwnJbyqRBEkymY1W+jGo
3uH46/AymJO075w+WW2c8XdOpZ5M38OHuqSl5Q7Y+fJQHAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQuFxgEc7ILr4HOYHV1gUtYYbMOv1fSduAQTbvcOxzIZGQ68VyPX+8/AA9/5SQ6AEw
pwJaTlrA7wzVNRznTkcn/cKWAiL5SMfiIS9/NdRmUcPOLpdfEkogQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQL7CizCGU27KaQPDJnbUt6+YXvcMXUw5SW2q/LldfraKv5qw4Zgn4ULy1+M
QiHokf9VYzrkulDLES8MR9rf9KTjKZW+/B6xmlqZp5LUZtFUio0z9r0NKI0AAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIErimwfKzJNWtS9usK1EIVr9tzPSNAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAlUB4YwqkQNqAvkHtNTO9HsCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIPD6AsIZrz/GO/Sw9BCY6ZkmmWeV7FC9IggQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAwBMLCGc88eDdrulTAONjlcA4b5rx8fHx9XO79qiJAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAg8j4BwxvOM1T1bOqczfmMY0T/RjHsOj7oJECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBA4JEFhDMeeXQeqW2leIYHmjzSSGkLAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CDyYgHDGgw3IYzdnm9CQy3jsEdM6AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIELi/gHDG
/cdACwgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIEXFhDOeOHB1TUCBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIEDg/gLCGfcfAy0gQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIEXlhAOOOFB1fXCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgfsLCGfc
fwy0gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEHhhAeGMFx5cXSNAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgTuLyCccf8x0AICBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIEDghQWEM154cHWNAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQuL+AcMb9
x0ALCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgRcWEM544cHVNQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQOD+Avlwxs/Xx+nf5/e/pob2n9FUbOdBcyvO7f9o7UBnTfsd
/u/789DYh2roBPn1c1lPzwVdWs5lrXA2AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBC4
k0A2nHHOC/xlBppW1btP2L3HaQumYMb8P5p6sXujmgp8xPjCpHlpYuQhkydNw+IgAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECCwh0A2nJFuQNGyPN97/B6NT8uoJDOOGY0HzWc8ZHxhAr0U
7RGTJ3vPPuURIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIG8QC6cscw61NfnFw8TqR++
95CkzV1HSe7ctIauPmR84dyolmROqY8PmTxpGBSHECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgACBnQRy4YxFouGjtkC/jHLUjt6p6Ukx1UdwzA28feOq3X3I+IKNM6rj5gACBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQINAkkAlnTHmBr6/P+vNATkmOr6+v+zw8pCVIUM1vNHFd5SAb
Z1yFVaEECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOAxBDLhjOmZFqdwRmm/iXPs4ev7
+xDk+Lj5U03atp54yAzE7yxoa/2Np0tL3qWpSY+q3tR4BxEgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAgR0E4nBGsjR/XFwvhDNOx35+/5yyGemxU3Dj59jW+WkpSYJj+VCUYmVxj9uSBJUQ
xPJBLtk+r3uU9qkaTEnrmJii+EKT20r0EIwZ0AtNp3TO97/w961jtkFfnhjleC4S3uGSUAQBAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIENhXIA5nJEvz50eWnNIVm9pPv/+NGkQhg/m3qxDD
eXuNTSTiFDHo235jKqW0w0cWbh00ODch3AUk7dG06UV6RmbnkLCSv2xCmBmpuv12pqvZnbOmFHfJ
Dlk0ZotJEZ+5DmiMCnd20eEECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOBGAmE4I12a
nzbGCHdQSHY5iEIG86/XSYLDinwpXtC3BcRi2b/vsSr5rEG0E0XQ4WUyI9y8It/Rr5/yxhmh29/c
6Gt272zKbpxRqXYdjkkmRf7M5Uljwr0ddDwBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ELiZQBjOWCzNl7bOiDZ4SKMR599/fX0uQgv//v1FPcJnnKSJjZ6UxXrtv3EPjeS0ZW1JKqLQo+U2
GVNh2aKSzSXONp8HmuWDY8puC7rVdhWZZnfOp6mUODdxaHCgsg2mTDPp1Mv4rAXXqu9Nwp3dczgB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIELilQBTOWD7TorB1RprbKG2csVnMP/QwfJ7H
4jd9zzaJNpOoxDtKT++YwyNJIYuNLLb5j9IuGMFWIGmgJG3popagC93N7p1RYcqknPsInyxzEdc6
sXLoRCTc2zvHEyBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBmwpE4YzVMy1yW2ckz5/4
bXK0aj6HD6J9LAoRg3xuo4YTPTwju4lG9uEdh1qi9pUDClGzw9DCuRdzcYs2lt0m6rhf5eRGzW/Z
87SGzG4aU4HR7/u5KhunjE+Mpo47iAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIXEEg
CGesV/czW2ekjzTJbIOReczHuRuVDMJ4b6OERs8jUn5rzmxqUYxaRKmJVoLoyR7LJ3o0ceT24mg6
eTooanSlI7/nBsGJNq6472H0pByn6eulowkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
wG0EgnDGegF8uUPGqVnrHxYf6JELRqxCFNkdLsYo1hGNXPGLZ28cnsCy/JecV949ItizohYmCHeC
aNv7oqPZnX5RL2sd+a1i2+w2rsW4tJ3S+bybzv47nAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQI7CqwDWds19iXe2Qcql/vphGGDKLAxqrxQcJg34jGIqGxKTraYmMTzkiiJeWIwva31ZhF
SFR1621254wJWl3tyDwnkv0+2rjCjTPiWdAQEensq8MJECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgMDVBbbhjGzEIFkv38Q1olXzMLARdOiGAY3Fkn8h4vB7XBRHKPeosHFG9pkqQe4lejxI
qtbf7N5JFAxmNS/yW8dGp5sr83Scqf0tEZHezjqeAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAhcW2ATzogWwE9L8+eIwfY5J+GqecuC/ty9dUQjm2dIRFrjH/9NgYa51EV9QWVR3KTco6A1
NYL+jTNGmt05hwozoPQ4keyOK5mTwl0wuoU7++ZwAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBwe4FNOKOwB8Zp34n1I01+G33JxhmLPifZg4Z0RnM4YxscmH6SeYpKEOdo3dIiCIBk+hK2
v9ipkWaHk6pQSzCYDVtWbFu2+8YZ0zRrmBq3v5LUSIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIEMgLrcEa8DJ9unbHaRuOv3N6NM4r7I0SxiNzw1XamOJ+3LbNy5nRCmt3o39ahfEbc0yac
TDwhbHaIl6+ldzCPxQepkbbOh0+aibvXnMVxtRMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAgUcSWIczwodNnJfe/9bMg2xG98YZcSVnltbExe/xDTs6/JW6zSyUT4z376hkA6Jmlzo6V7LI
J7RtnBGmF3q2Hckix3tzVBIzUV8GuAZOeaRrSVsIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgEAosApn5EILx8X5z++f78/j//iXlNa918K8lh+EDMrJjVUniiWdjp2CBWmzC7XMxy872r9x
xhwe+Vh3NFdJ5dEdA82OJ34ubZF7bkppT46kL0k3B7gGTnFZEyBAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgACBhxdYhTNyi/+ncMbX1yGbsUoaRCeVN0FI9nhYFhav8xcU05I2DUvCEetmZ3aZ
SDMTq1OGtnVIqkkCLataUs5KLf3NztiFBZW23kjaHD3q5Q9rMZQDXAOnPPz1pYEECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQOC/ZTgj+7SPNAOx3DYj82yR2rNJ1imIw+J+8i98bkc4Xst4
xrqc6b/XrZ6fdZI9ZRk3GNzWIdfRaReS5TYkNbfuZmfneGkENlh/pVSGbHXOANfAKa5gAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECDw+ALLcEb+qRnhs0GO3evfOONwWiFUEWYDSpZjUY/M
WX+1b3MC2dzKEiEIlUQd/TssyiJUNo9YeK9CJXGzR9SywZj8kG1GbIBr4JTHv760kAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQILHfOKCyP538V/qa6AcSJfrPc353LSMYwClvUduBYnjMd
vQ1JtD2nJdf8tJpTJWGBrW7Nza5N8ZVZA/96yGLhAa6BU2q983sCBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIPAAAsudMx6gQZpAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
EHglAeGMVxpNfSFAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQeTkA44+GGRIMIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBVxIQznil0dQXAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBA4OEEhDMebkg0iAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
EHglAeGMVxpNfSFAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQeTkA44+GGRIMIECBA
gAABAgQIECBAgAABAv+3d2/ZbeNKFEA9rgzI48loMpkeTNoPieKjAJIyj0LZu3/uuolUBDaKzAeO
QAIECBAgQIAAAQIECBAgQIDAdxIQzvhOq2kuBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAwOkEhDNOtyQGRIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECHwnAeGM77Sa5kKA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAicTkA443RLYkAECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIDAdxIQzvhOq2kuBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAwOkEhDNOtyQGRIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECHwnAeGM77Sa5kKA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAicTkA443RLYkAECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIDAdxIQzvhOq2kuBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAwOkEhDNOtyQGRIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECHwngWY447/fv152
/Pfr938nYfnz+jns1z+jAZV/eJIBf30Y17WKLsJhF3nkWvx5nbTB16mLnprdJbkLHjr6u4v9wPvr
bitfJECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwEWgGc647sFuDGhEcwF7VqvMEBwWLNgz
ksd99iFxh6Mu8qi1+LxOoC+33RmBCz+uoTpX+on31yngDYIAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAgacWaIUzdh6cMT2n4l+K/MAf9j8k7jA0xFdPhjgq5NFvssxVnvauOO6O/IH313F4KhEg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg8GMFWuGM6x7ssx0A8BN/2J8JIsxuiaMa4iFJkr+J
qyxOzKhiKov4xlezLCd7Mv3E++tkS2A4BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAg8o0Aj
nHHYOQmnMHlIeuGfzTQRRMhN5jFrcfhVJsmM9cDFJKOx/vEc90MqH679kFG7CAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBB4n0AhnHHVOwuMm0rnSc6UXdpM91db4Y9bi4KuMohY7TpIZfet7
xzMO1t59A/gCAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIETi9QhzMcnHH6hRsG+Fxb449J
khx6lVvIYnfG4gtffZ4G/Huo9hPN21AJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECCwWaAO
Z9x/cMY8KjB+v8Nsb3vy6oeVEwkmb5V4efuv/nwZVGilF3oJlMnYmpcbK+8c4c1ixwszxh8dAKqt
8evwr1e5fXGeL9g57NYybfVarMX0i1X6YT6Xv0MY4L0PXqZfWUzn4zOzD70t2/hzQ4XLH84G0b4X
RkUGl2E+1z/ZdittHs8T3V8rz6B5y/STL/c22EHPn83PUx8kQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIBAKVCGMxY7zNvxxlGB+YbqaJe8+KuVvMXnLvv0v8V+bvkb/sYP+5uTbGzwF1v8nyzV
XK7jbI3wc+u+/ma5S11+9P2TZfTkMoWPq0yncyt+z7CXKYd5/fECFROZrEUN3YiPbBJrzGkSKWm2
3jIFMlqiWSqlWWSY0yJ9szXXMkSPyvE8z/3VeWY0W680at6PvW4c3xqjnryuyvbnz/ZHn08SIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECgLVCGM4od5o2Go6hAvaf6vv3a3Jtd7ua3N2Y/d1yn
u7ndgzPmW7n1JHuRhfK8jvtG+DbT9pUWW9Ttj77+6R+c0Tye4L5hL0a2Umb++dX2aK/pJrHVgzN6
kpe/m4y5DPZ0sgV/fv9aNuY4KjO7j/aOZxXwPPdX64mx7w47usF2+mx87PkYAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIEVgSqcMYRB2e8vs7eJzG83uH19XP/ujjDYb6XP9qZneY2Rhu8478o
XyHReK9EPcnbFcd1G5d7o71zhL+uCJPESCMTM97NXp7H8OtTs4oUXK9ye+fGf+/nddw/7Fm+ZbLN
Xq3DIj0zrMVl1Bumf0027BArMzqzaY+ufL3CZUzFRMq8xmRyi8TBpF8vV2ge9DK9HTrjGfrt9fT3
V+O5Mw5bjDzqO+z+BjvGx78eBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQOEqgCmesnarQ
frfIZDd1FrVobMuON80bCYzyPR/VmQblOQflHzb4mtv6t3MuytDG1hH2fN6G1DsFozi2oyXa3NO+
zHr4+73Dbr0epPP+kklq5EvTXyQ9mp3TWPHb1XspifEci1BFu0hP/bNQ89UoW8czPW3lie+v6hVB
i2hSOxI1XvnNDbb7+XPUI1YdAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI/P1bhDP67x2Y
JjPqIxumJwF8Mt+qtt/aUR03MN/SvixaFTAooxXtvEWx/J3YQlGncSbHhhFODg4ZxlENddhSXqi1
TW/b0NWXhghI+ZdlCqUbGqnL3OqM/r6/3d5dvu1iw5Vb4YUqklJmLopsRuNwk88lbLtfypf9PX/d
zuxuqb+zTDc8w/3V1ZtHk+qjbW43bfX3zYNs7nj++NeBAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBwp8OZxRv/5juQHe2ZjdlZ94n3vjR/BfPjijt7++z3xlhHXipAh79LezbxDbt/2786iH
3Q2NlKdvvF+kk2bpT7+ey/acSX1wRj9JU0leJlAcpbEWGFrkLYpwxv7xjAMGT3l/rUx52pkrnV82
WOdslMat8nHN3c+ffc8CnyZAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgEB5csauPdSJ4YYA
RnVOQO/FI2vHeBTnMpQv32ic77BogXE8oXyRxuIbd4yw3twvju1YW4r+YRPN3MT7HHYMu1iftYGN
LzAMo38WQn8u5fqVB53UW+1rA25GSbZnM8rDQj7bZRnOuGM8o+jQcmU7Z76Ub8u5tvGONvjqwTRr
Uy4fJZ37djnlwPPHvxEECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBA4RGB5ckZvm7d/yS0b
xMVma+uH67OcxPx9Kp//f7RNfcDBGe/zK7arWzvE946wrrfcvF5divumvHPYxfqsDmzsOCxRf3e+
O5etYm/XLeusHo5w/cCtn+4PVCwDSJ9jWgsSje+u5XhGnVmkbjq2J7m/Vpegmn75zpfLBxdNePDz
55DnqyIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBD4EFiGM3b9wH2ieMcG8fv3u7vyRSTj
bY+72ob96g/7x1PZEtDoRBz2jHC4bOfgjOYRGJcx7DkrZP+wOwdn9M7mWKxHf3d+xykYV7IdX+kd
zvJRbim5CGesJlKaE2znPHat7HPfX6tLML4Bt3x4wX2kj38eCBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgcKjAIpyxugXdvPyWH64Xm9HVlvYkHLHxnID7TpHoYc4jGuOBHDbCYQCFw9oW9f4p
3zHsan3WBvY2qWU39L9U7qz3v1KGIeqvrA24OKiiGc5onaPS7P/LX1QvSGmFM4rxFKK35u3M7yT3
177nytp6VQ124PPn0EesYgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEChOzjjDwRnDNmtj
H3w4/WG0t33kwRmTxhjlGRbvvJi8qmL8rc0jvH1puR+9tp+9f8r3wBb75GsDe5vU8krNcyU+CMqS
/a+UR640vrKy2d8JMCzXvNGUTdvqeJM7xjNMd8ebgd5le+v3wPtrZS2nz+K7GszBGf5BI0CAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIEDgvALzkzM2bIs2JrPlh+sb95VXtq6H5MPqWQQbfoB/mU33
k8uoxWEjHDD3H1BRBUAaL4m5XuWOYe84nmLUGEVS4V8enFG/Omdh/zJuqJ0nZ9wyPNPTMOpQwrYj
QarxvLxsPEnmMrv9fTVgTS6//5SW6XNiX9Bm9cZth38O8Dnv49rICBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBB4VoF5OOP+gzM6u69bchvFIRjVNuvojIXJNvX+UySmK9af93yvuB9hKc/aGM6G
WDmtYLz13hvU7SKTgt098HuG3d2Ur1fodgzGaHt/3+b8x+Ic+JXezOvlul59pNtbjiEpM0k0NF9F
cv94pvU/m/gJ7q9+amgxgTp4NNyyVfMf9/x51me5cRMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgcGKBWTjj/oMzeocT7H3hQOfzt03w6StFjvph/8aDCY4a4WK7uUxaLAfVYlg5ImL/sBsNUR5e
cpnMaGyjpMm2syIm0z/yK83d/lE0Y5Z7uHylmMM8XzOuMc6rdCIG+8fT0XiC+6t60c21+atmOrTB
9vqc+HltaAQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIPKnALJyx8ov1ziSP/OF64yyDcSLh
5e2/4rCNOttQv2iiccbBtPDtF/+TPz9ohDfPeut9dJniBId3g9loVw6bOG7Yo7WoXi6zd2DV9L9w
cEZxokcZGZl1VNkn9fSGP50EM96nffmbOqKyXPDJAnbG8+z31+TIm5vpmG/Cf2CD7TxY5Emf5IZN
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgMCpBabhjMVO82X/v/k/t13WA3/Y/wY2z2GUA1g7
06Dxg/lWAGXtmtOt+7VPz+MJdxwD8d44rcv8+v3n96+Pa+w4bOILsNPJ90Z2WarZ6RJ3TP+Or8z7
dzyIZm//en0tJG9Wm4q8vr7WN8kS7vI82Dme57+/Vu7qBdTKHba9wRyccep/ggyOAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAwM8QmIYzNiUOxpvQw47qkT/s/5BvDOV9S7bYpy63rldOo5i/nGLy
2/75Tvvyw/tGuPK+mM7We7WJ/85+x2ET+2G7p1e0ozwLrTumf8dXFl0z2/CveurtI038bleNOuTj
Mo3ivcfIjvF8i/urfYfVEZY7Gqy4Tx2c8TP+KTNLAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AucWmIQzTnNwxsVsunndCYKUW9eN/ezWwRnXdWq9p6Jcx6+NcCi58gaPycb/haH8Sv+widsMNg97
veCcq7/NXuydv41qz/JdJtEWG82tutp46p9j7WzeN85caLyLY6LaPDFj1kcbx3PswRn/8v6ax1jW
oLY1WMTn3I9uoyNAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA4JkEpidnPNPIjTUvsBYayY/g
AVfoBVCGZECdKtk6us6LNZYl1gMxWy/rcwQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBw
DgHhjHOswylHsfJmkVOOefegVoITt4Mb1o54aF15Z8BjV5Bj92x9gQABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgT+gYBwxj9Af5JLfpdsRu/8jw3JidF7Nfafn3F7a8ntu18cz5N0j2ESIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAwCAgnKEZGgJDrmB/JOFcprd0xWwmt+TES/dYjFE8
4+Wl/9HRzCffGtf/8njOxWs0BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQILAmIJyxJvTT
/n6UWHhLIuwII5wXahqu+JzV5L/1N5YUJVpfWn50/skDxnNebCMjQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAgYWAcIammAjMgwPPfmzG5+R6cYjNM1zNVBSxj8Y5G4eMR+cSIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAwJMICGc8yUI9apiTgzM25xYeNbovXWeZiLhjfjsSGmvncRwy
ni+J+DIBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIPERAOOMhzC7y7QQWr38Zzs24I/Lx
7XRMiAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgRGAsIZ2oEAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgEBQQzgjiKk2AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQEM7QAwQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBoIBwRhBXaQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQICAcIYeIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgEBYQzgrhKEyBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSEM/QAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQCAoIJwRxFWaAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQICCcoQcIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAkEB
4YwgrtIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAeEMPUCAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQCAoIZwRxlSZAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQICGfoAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIBAUEA4I4irNAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEBAOEMPECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgSCAsIZQVylCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQLCGXqA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIBAUEM4I4ipNgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIEBDO0AMECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
gaCAcEYQV2kCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAgHCGHiBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIBAWEM4K4ShMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIEhDP0AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEAgKCCcEcRVmgAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAgnKEHCBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQJBAeGMIK7SBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAHh
DD1AgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEAgKCGcEcZUmQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECAhn6AECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAQFBAOCOIqzQBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAQDhDDxAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEggLCGUFcpQkQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECwhl6gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAQFBDOCOIq
TYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAQztADBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAIGggHBGEFdpAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gIBwhh4gQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECAQFhDOCuEoTIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBIQz9AABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAICggnBHEVZoAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgIJyhBwgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECQQHhjCCu0gQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAAB4Qw9QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAICghn
BHGVJkCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgIZ+gBAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgEBQQDgjiKs0AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQEA4Qw8QIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBIICwhlBXKUJECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAsIZeoAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgEBQQzgjiKk2AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQEM7QAwQI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBoIBwRhBXaQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQICAcIYeIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgE
BYQzgrhKEyBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSEM/QAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQCAoIJwRxFWaAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQICCcoQcIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAkEB4YwgrtIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQICkFTo6AAAhoElEQVQAAeEMPUCAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQCAoIZwRxlSZAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQICGfoAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIBAUEA4I4irNAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEBAOEMPECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgSCAsIZQVylCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQLC
GXqAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIBAUEM4I4ipNgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIEBDO0AMECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAgaCAcEYQV2kCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAgHCGHiBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIBAWEM4K4ShMgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIEhDP0AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEAgKCCcEcRV
mgABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAgnKEHCBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQJBAeGMIK7SBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAHhDD1AgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEAgKCGcEcZUmQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECAhn6AECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAQFBAOCOIqzQBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAQDhDDxAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEggLCGUFcpQkQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECwhl6gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAQFBDO
COIqTYAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAQztADBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAIGggHBGEFdpAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgIBwhh4gQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECAQFhDOCuEoTIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBIQz9AABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAICggnBHEVZoAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgIJyhBwgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECQQHhjCCu0gQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAAB4Qw9QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAI
CghnBHGVJkCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgIZ+gBAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgEBQQDgjiKs0AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQEA4Qw8QIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBIICwhlBXKUJECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAsIZeoAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgEBQQzgjiKk2AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQEM7Q
AwQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgACBoIBwRhBXaQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQICAcIYeIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgEBYQzgrhKEyBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSEM/QAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQCAoIJwRxFWaAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQICCcoQcIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAkEB4YwgrtIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAeEMPUCAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQCAoIZwRxlSZAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
CGfoAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIBAUEA4I4irNAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIEBAOEMPECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAgSCAsIZQVylCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQLCGXqAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIBAUEM4I4ipNgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIEBDO0AMECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgaCAcEYQ
V2kCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAgHCGHiBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIBAWEM4K4ShMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIEhDP0AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEAgKCCcEcRVmgABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECAgnKEHCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQJBAeGMIK7SBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAHhDD1AgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEAgKCGcEcZUmQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECAhn6AECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAQFBA
OCOIqzQBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAQDhDDxAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIEggLCGUFcpQkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECwhl6gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAQFBDOCOIqTYAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAQztADBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAIGggHBGEFdpAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgIBwhh4g
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECAQFhDOCuEoTIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBIQz9AABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
ICggnBHEVZoAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgIJyhBwgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECQQHhjCCu0gQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAAB4Qw9QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAICghnBHGVJkCA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgIZ+gBAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgEBQQDgjiKs0AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQEA4
Qw8QIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBIICwhlBXKUJECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAsIZeoAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgEBQQzgjiKk2AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQEM7QAwQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgACBoIBwRhBXaQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQICAcIYeIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgEBYQzgrhK
EyBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSEM/QAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQCAoIJwRxFWaAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
ICCcoQcIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAkEB4YwgrtIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAeEMPUCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQCAoIZwRxlSZAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQICGfoAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIBAUEA4I4irNAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIEBAOEMPECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSCAsIZ
QVylCRAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQLCGXqAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIBAUEM4I4ipNgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIEBDO0AMECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgaCAcEYQV2kCBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAgHCGHiBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIBAWEM4K4ShMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEhDP0AAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEAgKCCcEcRVmgABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECAgnKEHCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQJB
AeGMIK7SBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAHhDD1AgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIEAgKCGcEcZUmQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECAhn6AECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAQFBAOCOIqzQBAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAQDhDDxAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIEggLCGUFcpQkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECwhl6
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAQFBDOCOIqTYAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAQztADBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AIGggHBGEFdpAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgIBwhh4gQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECAQFhDOCuEoTIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBIQz9AABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAICggnBHEVZoA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgIJyhBwgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECQQHhjCCu0gQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
4Qw9QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAICghnBHGVJkCAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgIZ+gBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgEBQQDgjiKs0AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQEA4Qw8QIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBIICwhlBXKUJECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECBAgAABAsIZeoAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgEBQQzgji
Kk2AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQEM7QAwQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIECBAgACBoIBwRhBXaQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QICAcIYeIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgEBYQzgrhKEyBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgSEM/QAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIECBAgQCAoIJwRxFWaAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQICCcoQcIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAkEB4YwgrtIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAeEMPUCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQCAoI
ZwRxlSZAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQICGfoAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECBAgQIBAUEA4I4irNAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIEBAOEMPECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSCAsIZQVylCRAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQLCGXqAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIBAUEM4I4ipNgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEBDO0AME
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgaCAcEYQV2kCBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAgHCGHiBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI
BAWEM4K4ShMgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEhDP0AAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIEAgKCCcEcRVmgABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAgQIECAgnKEHCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQJBAeGMIK7SBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAHhDD1AgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgAABAgQIEAgKCGcEcZUmQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECAhn
6AECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAQFBAOCOIqzQBAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgQIECBAQDhDDxAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAA
AQIEggLCGUFcpQkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECwhl6gAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAgQIECAQFBDOCOIqTYAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAQIECBAQztADBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIGggHBGEFdp
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgIBwhh4gQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIAAAQIECAQFhDOCuEoTIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BIQz9AABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAICggnBHEVZoAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQIECBAgIJyhBwgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECQQHhjCCu0gQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB4Qw9QIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAICghnBHGVJkCAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgIZ+gBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgEBQQDgj
iKs0AQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQEA4Qw8QIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBIICwhlBXKUJECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBA
gAABAsIZeoAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgEBQQzgjiKk2AAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQEM7QAwQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIECBAgACBoIBwRhBXaQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQICAcIYeIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgEBYQzgrhKEyBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAgAABAgSEM/QAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQCAo
IJwRxFWaAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQICCcoQcIECBAgAABAgQIECBA
gAABAgQIECBAgAABAgQIECBAgAABAkEB4YwgrtIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE
CBAgQIAAAeEMPUCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQCAoIZwRxlSZAgAAB
AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQICGfoAQIECBAgQIAAAQIECBAgQIAAAQIECBAg
QIAAAQIECBAgQIBAUEA4I4irNAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIEBAOEMP
ECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSCAsIZQVylCRAgQIAAAQIECBAgQIAA
AQIECBAgQIAAAQIECBAgQIAAAQLCGXqAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IBAUEM4I4ipNgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEBDO0AMECBAgQIAAAQIE
CBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgaCAcEYQV2kCBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAgHCGHiBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIBAWEM4K4ShMg
QIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEhDP0AAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIEAgKCCcEcRVmgABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECAg
nKEHCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQJBAeGMIK7SBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAHhDD1AgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB
AgQIEAgKCGcEcZUmQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECAhn6AECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAQFBAOCOIqzQBAgQIECBAgAABAgQIECBAgAABAgQI
ECBAgAABAgQIECBAQDhDDxAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIEggLCGUFc
pQkQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC/wOmt5lMVJ3jlgAAAABJRU5ErkJg
gg==
--001a113fb8a4b9e53905294b8e8a--

From jwollrath@web.de  Thu Jan 14 08:37:04 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 10C497F47
	for ; Thu, 14 Jan 2016 08:37:04 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 000D58F8040
	for ; Thu, 14 Jan 2016 06:37:00 -0800 (PST)
X-ASG-Debug-ID: 1452782218-04cbb05f7f195660001-NocioJ
Received: from mout.web.de (mout.web.de [212.227.17.11]) by cuda.sgi.com with ESMTP id ODv2hxYcOMUZBV0u (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 06:36:59 -0800 (PST)
X-Barracuda-Envelope-From: jwollrath@web.de
X-Barracuda-Apparent-Source-IP: 212.227.17.11
Received: from saldaea ([134.76.63.1]) by smtp.web.de (mrweb102) with ESMTPSA
 (Nemesis) id 0LtnzN-1a9S8h0wzt-011C0Q; Thu, 14 Jan 2016 15:36:54 +0100
Date: Thu, 14 Jan 2016 15:36:52 +0100
From: Julian Wollrath 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, Jiri Kosina 
Subject: Re: Suspend does not work with v4.4-rc8
Message-ID: <20160114153652.49598601@saldaea>
X-ASG-Orig-Subj: Re: Suspend does not work with v4.4-rc8
In-Reply-To: <20160104194014.GD19802@dastard>
References: <20160104202611.5b100540@saldaea>
	<20160104194014.GD19802@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Provags-ID: V03:K0:noQ24WjQX+A4gknBXmB6dUmItxe5EmC8KL2ldEWSWp4xaKISJpq
 or8jpVhbeRCFFMYLA2h3oE1yMjvgNSRK8LQaL9sbsUz1Mreq83OmlyX0BRjW/RBHjlPgVbI
 0KuNCEyHsWv6DKoBOgkU1RnFWMYNtIBomIezu7zAjXw7Xy1RuEyrwqjadPLdKexpvNCNdx8
 PqTfQA4hewOn3vlzFmK0A==
X-UI-Out-Filterresults: notjunk:1;V01:K0:VNVNKEOQ6dU=:aGQEerE6ozzYMCVEM+39sY
 MFeqQhLfQ/xhhADUDCft9O5kyt3eTwPv60uiRiGHpKD9Yy7oaGGyXXriIcNShnGCzQq19Au9b
 QmBuOywjIApp5Cn01p3f+CnB0xjrM0XzaxXRp0VoBozBRibwGIC0WoP+gxpSMr9Tswf9OPV/2
 hJpPK75zqYQe2ZJSdgBB6NDO4gO9Ur2n2fM4JOrJaA7SYn8HcpYUh3r+mNcnttDLkcPwstRy6
 8OGWHgo7ct5tqPPnAQcAy2s0KUuhw801ozkQIxttud/tBvgv4hUWTLXrnTfLZ4tgtkEyILgjh
 hJY3nb6NbBwhbS8UOrb4ti19fIsS51hwo9Lus3uQDRu8jTYjMjmSlBbGZkmuHLB3TMrPOLkF9
 JNwmGT/LY+d+latYSLaTypKR0/b7T0+Uc+L19L9duO3FTjG4QWXLwTKqNOX+pNxJ+fFntyz+A
 sBX9BI/KGaD15phBJCQG7VGjC4n+NoYhuI9Uy2CV4vZ0nS53+/fIzOiJYma7xL/kt57+uRBGs
 Dz42MLCvWkLlB3XJYxHbTU92B0cIQjyyr2k0EQsW5a4ZiIfba2sgvGYfD0JyE2dZaKrSBnTpt
 eFUmwoiGnKHbgLEb2miN9Xfg8ez683nWIs9htJxnlIYyK/bMs7jNE6Xn+Ed0OjjxQiOWCweaL
 Zyfi9vrK629spikYDhrpdPaTBlVmjgyHVZYtUP3zgp5yPmG1GVGIj2ac8r8T6ULhxBWDFdv0M
 sJctNVhpn5Suz4WtAIqNKuBc+lGLh7+MbC602IIV3HV1aNbCFqn9Ueos2bMbDzSZkkkHqy0H8
 jkxUdZf
X-Barracuda-Connect: mout.web.de[212.227.17.11]
X-Barracuda-Start-Time: 1452782218
X-Barracuda-Encrypted: DHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26121
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Hi,

> The freezer detection is broken. The thread is sleeping in schedule
> until a wakeup occurs some time in the future, which means it cannot
> "enter then freezer" because it's not a running thread. This is a
> problem introduced by commit 24ba16b ("xfs: clear PF_NOFREEZE for
> xfsaild kthread").
reverting this commit gets rid of the problem and lets me suspend my
machine again.


Cheers,
Julian

From sandeen@redhat.com  Thu Jan 14 10:35:59 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 42D3A7F47
	for ; Thu, 14 Jan 2016 10:35:59 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 1832130405F
	for ; Thu, 14 Jan 2016 08:35:55 -0800 (PST)
X-ASG-Debug-ID: 1452789354-04bdf06f6d1028e0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id EVS0SVDIddynxNOc (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 08:35:54 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id 25CE032D3AF
	for ; Thu, 14 Jan 2016 16:35:54 +0000 (UTC)
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0EGZrjk028172
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for ; Thu, 14 Jan 2016 11:35:53 -0500
To: xfs@oss.sgi.com
From: Eric Sandeen 
Subject: [PATCH] libxfs: don't pass negative errnos to strerror()
Message-ID: <5697CE69.3050905@redhat.com>
X-ASG-Orig-Subj: [PATCH] libxfs: don't pass negative errnos to strerror()
Date: Thu, 14 Jan 2016 10:35:53 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452789354
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

The error negation work in 12b5319 tripped up a little bit
when we're reporting errors via strerror().  By negating
the error before passing it to strerror, we get i.e.

   mkfs.xfs: pwrite64 failed: Unknown error -22

Keep the error positive, but return -error, just as we
do in the else clauses in these functions.

Signed-off-by: Eric Sandeen 
---

diff --git a/libxfs/rdwr.c b/libxfs/rdwr.c
index 7a04985..7b23394 100644
--- a/libxfs/rdwr.c
+++ b/libxfs/rdwr.c
@@ -913,12 +913,12 @@ __read_buf(int fd, void *buf, int len, off64_t offset, int flags)
 
 	sts = pread64(fd, buf, len, offset);
 	if (sts < 0) {
-		int error = -errno;
+		int error = errno;
 		fprintf(stderr, _("%s: read failed: %s\n"),
 			progname, strerror(error));
 		if (flags & LIBXFS_EXIT_ON_FAILURE)
 			exit(1);
-		return error;
+		return -error;
 	} else if (sts != len) {
 		fprintf(stderr, _("%s: error - read only %d of %d bytes\n"),
 			progname, sts, len);
@@ -1081,12 +1081,12 @@ __write_buf(int fd, void *buf, int len, off64_t offset, int flags)
 
 	sts = pwrite64(fd, buf, len, offset);
 	if (sts < 0) {
-		int error = -errno;
+		int error = errno;
 		fprintf(stderr, _("%s: pwrite64 failed: %s\n"),
 			progname, strerror(error));
 		if (flags & LIBXFS_B_EXIT)
 			exit(1);
-		return error;
+		return -error;
 	} else if (sts != len) {
 		fprintf(stderr, _("%s: error - pwrite64 only %d of %d bytes\n"),
 			progname, sts, len);

From cmaiolino@redhat.com  Thu Jan 14 10:47:49 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 149E67F47
	for ; Thu, 14 Jan 2016 10:47:49 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id D90A8304053
	for ; Thu, 14 Jan 2016 08:47:45 -0800 (PST)
X-ASG-Debug-ID: 1452790063-04bdf06f6d102d60001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id AYecKTFtZXERxCR4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 08:47:44 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	by mx1.redhat.com (Postfix) with ESMTPS id E5F2FA0792
	for ; Thu, 14 Jan 2016 16:47:43 +0000 (UTC)
Received: from redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0EGleWR006990
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 14 Jan 2016 11:47:43 -0500
Date: Thu, 14 Jan 2016 17:47:40 +0100
From: Carlos Maiolino 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] libxfs: don't pass negative errnos to strerror()
Message-ID: <20160114164740.GA20151@redhat.com>
X-ASG-Orig-Subj: Re: [PATCH] libxfs: don't pass negative errnos to strerror()
Mail-Followup-To: Eric Sandeen , xfs@oss.sgi.com
References: <5697CE69.3050905@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5697CE69.3050905@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452790064
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

This looks good to me, and simple enough.

consider it

Reviewed-by: Carlos Maiolino 

On Thu, Jan 14, 2016 at 10:35:53AM -0600, Eric Sandeen wrote:
> The error negation work in 12b5319 tripped up a little bit
> when we're reporting errors via strerror().  By negating
> the error before passing it to strerror, we get i.e.
> 
>    mkfs.xfs: pwrite64 failed: Unknown error -22
> 
> Keep the error positive, but return -error, just as we
> do in the else clauses in these functions.
> 
> Signed-off-by: Eric Sandeen 
> ---
> 
> diff --git a/libxfs/rdwr.c b/libxfs/rdwr.c
> index 7a04985..7b23394 100644
> --- a/libxfs/rdwr.c
> +++ b/libxfs/rdwr.c
> @@ -913,12 +913,12 @@ __read_buf(int fd, void *buf, int len, off64_t offset, int flags)
>  
>  	sts = pread64(fd, buf, len, offset);
>  	if (sts < 0) {
> -		int error = -errno;
> +		int error = errno;
>  		fprintf(stderr, _("%s: read failed: %s\n"),
>  			progname, strerror(error));
>  		if (flags & LIBXFS_EXIT_ON_FAILURE)
>  			exit(1);
> -		return error;
> +		return -error;
>  	} else if (sts != len) {
>  		fprintf(stderr, _("%s: error - read only %d of %d bytes\n"),
>  			progname, sts, len);
> @@ -1081,12 +1081,12 @@ __write_buf(int fd, void *buf, int len, off64_t offset, int flags)
>  
>  	sts = pwrite64(fd, buf, len, offset);
>  	if (sts < 0) {
> -		int error = -errno;
> +		int error = errno;
>  		fprintf(stderr, _("%s: pwrite64 failed: %s\n"),
>  			progname, strerror(error));
>  		if (flags & LIBXFS_B_EXIT)
>  			exit(1);
> -		return error;
> +		return -error;
>  	} else if (sts != len) {
>  		fprintf(stderr, _("%s: error - pwrite64 only %d of %d bytes\n"),
>  			progname, sts, len);
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

From bfoster@redhat.com  Thu Jan 14 11:32:23 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 0258E7F47
	for ; Thu, 14 Jan 2016 11:32:23 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D820D8F8040
	for ; Thu, 14 Jan 2016 09:32:19 -0800 (PST)
X-ASG-Debug-ID: 1452792738-04bdf06f6e1040f0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id lYrmHsHFGLEMK4qo (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 09:32:18 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26])
	by mx1.redhat.com (Postfix) with ESMTPS id EA982A04BA;
	Thu, 14 Jan 2016 17:32:17 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49])
	by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0EHWHEa006509;
	Thu, 14 Jan 2016 12:32:17 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 6E9F11245A1; Thu, 14 Jan 2016 12:32:16 -0500 (EST)
Date: Thu, 14 Jan 2016 12:32:16 -0500
From: Brian Foster 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/3] xfs: handle errors from ->free_blocks in
 xfs_btree_kill_iroot
Message-ID: <20160114173214.GA11448@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 1/3] xfs: handle errors from ->free_blocks in
 xfs_btree_kill_iroot
References: <1452707454-15229-1-git-send-email-hch@lst.de>
 <1452707454-15229-2-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452707454-15229-2-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452792738
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 13, 2016 at 06:50:52PM +0100, Christoph Hellwig wrote:
> Signed-off-by: Christoph Hellwig 
> ---

Reviewed-by: Brian Foster 

>  fs/xfs/libxfs/xfs_btree.c | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
> index a0eb18c..3143577 100644
> --- a/fs/xfs/libxfs/xfs_btree.c
> +++ b/fs/xfs/libxfs/xfs_btree.c
> @@ -3209,6 +3209,7 @@ xfs_btree_kill_iroot(
>  	int			level;
>  	int			index;
>  	int			numrecs;
> +	int			error;
>  #ifdef DEBUG
>  	union xfs_btree_ptr	ptr;
>  	int			i;
> @@ -3272,8 +3273,6 @@ xfs_btree_kill_iroot(
>  	cpp = xfs_btree_ptr_addr(cur, 1, cblock);
>  #ifdef DEBUG
>  	for (i = 0; i < numrecs; i++) {
> -		int		error;
> -
>  		error = xfs_btree_check_ptr(cur, cpp, i, level - 1);
>  		if (error) {
>  			XFS_BTREE_TRACE_CURSOR(cur, XBT_ERROR);
> @@ -3283,7 +3282,11 @@ xfs_btree_kill_iroot(
>  #endif
>  	xfs_btree_copy_ptrs(cur, pp, cpp, numrecs);
>  
> -	cur->bc_ops->free_block(cur, cbp);
> +	error = cur->bc_ops->free_block(cur, cbp);
> +	if (error) {
> +		XFS_BTREE_TRACE_CURSOR(cur, XBT_ERROR);
> +		return error;
> +	}
>  	XFS_BTREE_STATS_INC(cur, free);
>  
>  	cur->bc_bufs[level - 1] = NULL;
> -- 
> 1.9.1
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Thu Jan 14 11:32:28 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id E2C7A7F52
	for ; Thu, 14 Jan 2016 11:32:27 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 7ECAEAC004
	for ; Thu, 14 Jan 2016 09:32:24 -0800 (PST)
X-ASG-Debug-ID: 1452792743-04bdf06f6e104100001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id VXeNFjb8shScLe3p (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 09:32:23 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	by mx1.redhat.com (Postfix) with ESMTPS id 5C25FC0BF2CD;
	Thu, 14 Jan 2016 17:32:23 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49])
	by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0EHWNXa006097;
	Thu, 14 Jan 2016 12:32:23 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id D7DA81245A1; Thu, 14 Jan 2016 12:32:21 -0500 (EST)
Date: Thu, 14 Jan 2016 12:32:21 -0500
From: Brian Foster 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/3] xfs: factor btree block freeing into a helper
Message-ID: <20160114173221.GB11448@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 2/3] xfs: factor btree block freeing into a helper
References: <1452707454-15229-1-git-send-email-hch@lst.de>
 <1452707454-15229-3-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452707454-15229-3-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452792743
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 13, 2016 at 06:50:53PM +0100, Christoph Hellwig wrote:
> Signed-off-by: Christoph Hellwig 
> ---

Reviewed-by: Brian Foster 

>  fs/xfs/libxfs/xfs_btree.c | 23 ++++++++++++++++-------
>  1 file changed, 16 insertions(+), 7 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
> index 3143577..77afb4a 100644
> --- a/fs/xfs/libxfs/xfs_btree.c
> +++ b/fs/xfs/libxfs/xfs_btree.c
> @@ -294,6 +294,19 @@ xfs_btree_sblock_verify_crc(
>  	return true;
>  }
>  
> +static int
> +xfs_btree_free_block(
> +	struct xfs_btree_cur	*cur,
> +	struct xfs_buf		*bp)
> +{
> +	int			error;
> +
> +	error = cur->bc_ops->free_block(cur, bp);
> +	if (!error)
> +		XFS_BTREE_STATS_INC(cur, free);
> +	return error;
> +}
> +
>  /*
>   * Delete the btree cursor.
>   */
> @@ -3282,12 +3295,11 @@ xfs_btree_kill_iroot(
>  #endif
>  	xfs_btree_copy_ptrs(cur, pp, cpp, numrecs);
>  
> -	error = cur->bc_ops->free_block(cur, cbp);
> +	error = xfs_btree_free_block(cur, cbp);
>  	if (error) {
>  		XFS_BTREE_TRACE_CURSOR(cur, XBT_ERROR);
>  		return error;
>  	}
> -	XFS_BTREE_STATS_INC(cur, free);
>  
>  	cur->bc_bufs[level - 1] = NULL;
>  	be16_add_cpu(&block->bb_level, -1);
> @@ -3320,14 +3332,12 @@ xfs_btree_kill_root(
>  	 */
>  	cur->bc_ops->set_root(cur, newroot, -1);
>  
> -	error = cur->bc_ops->free_block(cur, bp);
> +	error = xfs_btree_free_block(cur, bp);
>  	if (error) {
>  		XFS_BTREE_TRACE_CURSOR(cur, XBT_ERROR);
>  		return error;
>  	}
>  
> -	XFS_BTREE_STATS_INC(cur, free);
> -
>  	cur->bc_bufs[level] = NULL;
>  	cur->bc_ra[level] = 0;
>  	cur->bc_nlevels--;
> @@ -3833,10 +3843,9 @@ xfs_btree_delrec(
>  	}
>  
>  	/* Free the deleted block. */
> -	error = cur->bc_ops->free_block(cur, rbp);
> +	error = xfs_btree_free_block(cur, rbp);
>  	if (error)
>  		goto error0;
> -	XFS_BTREE_STATS_INC(cur, free);
>  
>  	/*
>  	 * If we joined with the left neighbor, set the buffer in the
> -- 
> 1.9.1
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Thu Jan 14 11:32:32 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 54D307F55
	for ; Thu, 14 Jan 2016 11:32:32 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 28AFF8F8064
	for ; Thu, 14 Jan 2016 09:32:32 -0800 (PST)
X-ASG-Debug-ID: 1452792750-04bdf06f6b104110001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id edoT6QT1LEDCDCwy (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 09:32:31 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id CCF388E008;
	Thu, 14 Jan 2016 17:32:30 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0EHWUXF020442;
	Thu, 14 Jan 2016 12:32:30 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 4DDDB1245A1; Thu, 14 Jan 2016 12:32:29 -0500 (EST)
Date: Thu, 14 Jan 2016 12:32:29 -0500
From: Brian Foster 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 3/3] xfs: move buffer invalidation to xfs_btree_free_block
Message-ID: <20160114173228.GC11448@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: move buffer invalidation to xfs_btree_free_block
References: <1452707454-15229-1-git-send-email-hch@lst.de>
 <1452707454-15229-4-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452707454-15229-4-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452792751
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 13, 2016 at 06:50:54PM +0100, Christoph Hellwig wrote:
> ... instead of leaving it in the methods.
> 
> Signed-off-by: Christoph Hellwig 
> ---

Reviewed-by: Brian Foster 

>  fs/xfs/libxfs/xfs_alloc_btree.c  |  2 --
>  fs/xfs/libxfs/xfs_bmap_btree.c   |  1 -
>  fs/xfs/libxfs/xfs_btree.c        |  4 +++-
>  fs/xfs/libxfs/xfs_ialloc_btree.c | 12 ++----------
>  4 files changed, 5 insertions(+), 14 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_alloc_btree.c b/fs/xfs/libxfs/xfs_alloc_btree.c
> index 444626d..d9b42425 100644
> --- a/fs/xfs/libxfs/xfs_alloc_btree.c
> +++ b/fs/xfs/libxfs/xfs_alloc_btree.c
> @@ -118,8 +118,6 @@ xfs_allocbt_free_block(
>  	xfs_extent_busy_insert(cur->bc_tp, be32_to_cpu(agf->agf_seqno), bno, 1,
>  			      XFS_EXTENT_BUSY_SKIP_DISCARD);
>  	xfs_trans_agbtree_delta(cur->bc_tp, -1);
> -
> -	xfs_trans_binval(cur->bc_tp, bp);
>  	return 0;
>  }
>  
> diff --git a/fs/xfs/libxfs/xfs_bmap_btree.c b/fs/xfs/libxfs/xfs_bmap_btree.c
> index 1637c37..e37508a 100644
> --- a/fs/xfs/libxfs/xfs_bmap_btree.c
> +++ b/fs/xfs/libxfs/xfs_bmap_btree.c
> @@ -531,7 +531,6 @@ xfs_bmbt_free_block(
>  
>  	xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
>  	xfs_trans_mod_dquot_byino(tp, ip, XFS_TRANS_DQ_BCOUNT, -1L);
> -	xfs_trans_binval(tp, bp);
>  	return 0;
>  }
>  
> diff --git a/fs/xfs/libxfs/xfs_btree.c b/fs/xfs/libxfs/xfs_btree.c
> index 77afb4a..1f88e1c 100644
> --- a/fs/xfs/libxfs/xfs_btree.c
> +++ b/fs/xfs/libxfs/xfs_btree.c
> @@ -302,8 +302,10 @@ xfs_btree_free_block(
>  	int			error;
>  
>  	error = cur->bc_ops->free_block(cur, bp);
> -	if (!error)
> +	if (!error) {
> +		xfs_trans_binval(cur->bc_tp, bp);
>  		XFS_BTREE_STATS_INC(cur, free);
> +	}
>  	return error;
>  }
>  
> diff --git a/fs/xfs/libxfs/xfs_ialloc_btree.c b/fs/xfs/libxfs/xfs_ialloc_btree.c
> index c679f3c..89c21d7 100644
> --- a/fs/xfs/libxfs/xfs_ialloc_btree.c
> +++ b/fs/xfs/libxfs/xfs_ialloc_btree.c
> @@ -125,16 +125,8 @@ xfs_inobt_free_block(
>  	struct xfs_btree_cur	*cur,
>  	struct xfs_buf		*bp)
>  {
> -	xfs_fsblock_t		fsbno;
> -	int			error;
> -
> -	fsbno = XFS_DADDR_TO_FSB(cur->bc_mp, XFS_BUF_ADDR(bp));
> -	error = xfs_free_extent(cur->bc_tp, fsbno, 1);
> -	if (error)
> -		return error;
> -
> -	xfs_trans_binval(cur->bc_tp, bp);
> -	return error;
> +	return xfs_free_extent(cur->bc_tp,
> +			XFS_DADDR_TO_FSB(cur->bc_mp, XFS_BUF_ADDR(bp)), 1);
>  }
>  
>  STATIC int
> -- 
> 1.9.1
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Thu Jan 14 11:32:49 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 08AB17F55
	for ; Thu, 14 Jan 2016 11:32:49 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 8D61CAC004
	for ; Thu, 14 Jan 2016 09:32:48 -0800 (PST)
X-ASG-Debug-ID: 1452792767-04cb6c75dcf8b30001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 2hRzuskEfIdwj9Bx (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 09:32:47 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	by mx1.redhat.com (Postfix) with ESMTPS id D7D811204B7
	for ; Thu, 14 Jan 2016 17:32:46 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-49.bos.redhat.com [10.18.41.49])
	by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0EHWk08006288;
	Thu, 14 Jan 2016 12:32:46 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 5A9E61245A1; Thu, 14 Jan 2016 12:32:45 -0500 (EST)
Date: Thu, 14 Jan 2016 12:32:45 -0500
From: Brian Foster 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: remove unused function definitions
Message-ID: <20160114173244.GD11448@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH] xfs: remove unused function definitions
References: <5696CDB1.3060404@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5696CDB1.3060404@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452792767
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 13, 2016 at 04:20:33PM -0600, Eric Sandeen wrote:
> Old leftovers.
> 
> Signed-off-by: Eric Sandeen 
> ---

Reviewed-by: Brian Foster 

> 
> diff --git a/fs/xfs/libxfs/xfs_sb.h b/fs/xfs/libxfs/xfs_sb.h
> index b25bb9a..961e647 100644
> --- a/fs/xfs/libxfs/xfs_sb.h
> +++ b/fs/xfs/libxfs/xfs_sb.h
> @@ -27,7 +27,6 @@ extern struct xfs_perag *xfs_perag_get_tag(struct xfs_mount *, xfs_agnumber_t,
>  extern void	xfs_perag_put(struct xfs_perag *pag);
>  extern int	xfs_initialize_perag_data(struct xfs_mount *, xfs_agnumber_t);
>  
> -extern void	xfs_sb_calc_crc(struct xfs_buf *bp);
>  extern void	xfs_log_sb(struct xfs_trans *tp);
>  extern int	xfs_sync_sb(struct xfs_mount *mp, bool wait);
>  extern void	xfs_sb_mount_common(struct xfs_mount *mp, struct xfs_sb *sbp);
> diff --git a/fs/xfs/xfs_fsops.h b/fs/xfs/xfs_fsops.h
> index 1b6a98b..f32713f 100644
> --- a/fs/xfs/xfs_fsops.h
> +++ b/fs/xfs/xfs_fsops.h
> @@ -25,6 +25,5 @@ extern int xfs_fs_counts(xfs_mount_t *mp, xfs_fsop_counts_t *cnt);
>  extern int xfs_reserve_blocks(xfs_mount_t *mp, __uint64_t *inval,
>  				xfs_fsop_resblks_t *outval);
>  extern int xfs_fs_goingdown(xfs_mount_t *mp, __uint32_t inflags);
> -extern int xfs_fs_log_dummy(struct xfs_mount *mp);
>  
>  #endif	/* __XFS_FSOPS_H__ */
> diff --git a/fs/xfs/xfs_mount.h b/fs/xfs/xfs_mount.h
> index b570984..a4e03ab 100644
> --- a/fs/xfs/xfs_mount.h
> +++ b/fs/xfs/xfs_mount.h
> @@ -327,7 +327,6 @@ extern int	xfs_mod_fdblocks(struct xfs_mount *mp, int64_t delta,
>  				 bool reserved);
>  extern int	xfs_mod_frextents(struct xfs_mount *mp, int64_t delta);
>  
> -extern int	xfs_mount_log_sb(xfs_mount_t *);
>  extern struct xfs_buf *xfs_getsb(xfs_mount_t *, int);
>  extern int	xfs_readsb(xfs_mount_t *, int);
>  extern void	xfs_freesb(xfs_mount_t *);
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From felix.janda@posteo.de  Thu Jan 14 13:09:22 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 9127B7F47
	for ; Thu, 14 Jan 2016 13:09:22 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 83944304053
	for ; Thu, 14 Jan 2016 11:09:19 -0800 (PST)
X-ASG-Debug-ID: 1452798553-04cb6c75defad60001-NocioJ
Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id 3FFc2fmFPEGqeTA5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 11:09:15 -0800 (PST)
X-Barracuda-Envelope-From: felix.janda@posteo.de
X-Barracuda-Apparent-Source-IP: 185.67.36.65
Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13])
	by mout01.posteo.de (Postfix) with ESMTPS id DDA3920B88
	for ; Thu, 14 Jan 2016 20:09:12 +0100 (CET)
Received: from mail.posteo.de (localhost [127.0.0.1])
	by dovecot03.posteo.de (Postfix) with ESMTPSA id 3phFYh21c8z5vMK;
	Thu, 14 Jan 2016 20:09:12 +0100 (CET)
Date: Thu, 14 Jan 2016 20:06:34 +0100
From: Felix Janda 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
Message-ID: <20160114190634.GA627@nyan>
X-ASG-Orig-Subj: Re: [PATCH 6/6] Add configure check for members of dirent for use in
 io/readdir.c
References: 
 <20160112200022.GG568@nyan>
 <20160113075855.GG21939@infradead.org>
 <20160113170938.GA10942@nyan>
 <20160114101729.GC27644@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160114101729.GC27644@infradead.org>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Barracuda-Connect: mout01.posteo.de[185.67.36.65]
X-Barracuda-Start-Time: 1452798555
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26128
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Christoph Hellwig wrote:
> On Wed, Jan 13, 2016 at 06:09:38PM +0100, Felix Janda wrote:
> > Christoph Hellwig wrote:
> > > I think we're having a deeper issue here.  For an LFS build we
> > > absolutely should have these values as the kernel dirent64 has them.
> > > 
> > > Either we manage to not pull in the LFS version of readdir/getdents
> > > or musl is doing something inredibly stupid.  Can you investigate if
> > > and why we're not using the LFS getdents64 here?
> > 
> > With musl you always have LFS. However still musl does not define the
> > _DIRENT_HAVE_*. You can see the full dirent.h at:
> > 
> > http://git.musl-libc.org/cgit/musl/tree/include/dirent.h
> > 
> > I will ask on the musl list whether these constants can be added. So
> > you can disregard this patch.
> 
> Oh, ok.  That's a little weird, but I guess working around that
> with a new autoconf test is fine.

Actually, it should also be useful for other systems. For example, on
FreeBSD dirent has d_type (but does not declare _DIRENT_HAVE_D_TYPE),
but currently dump_dirent() does not print it.

Felix

From felix.janda@posteo.de  Thu Jan 14 13:10:17 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id A0AC67F47
	for ; Thu, 14 Jan 2016 13:10:17 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 932628F8065
	for ; Thu, 14 Jan 2016 11:10:14 -0800 (PST)
X-ASG-Debug-ID: 1452798612-04cb6c75dcfadb0001-NocioJ
Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by cuda.sgi.com with ESMTP id jsTS6kKmGdxuwtDY (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 14 Jan 2016 11:10:13 -0800 (PST)
X-Barracuda-Envelope-From: felix.janda@posteo.de
X-Barracuda-Apparent-Source-IP: 185.67.36.65
Received: from dovecot03.posteo.de (dovecot03.posteo.de [172.16.0.13])
	by mout01.posteo.de (Postfix) with ESMTPS id 68DD320B9B
	for ; Thu, 14 Jan 2016 20:10:12 +0100 (CET)
Received: from mail.posteo.de (localhost [127.0.0.1])
	by dovecot03.posteo.de (Postfix) with ESMTPSA id 3phFZr22Fcz5vN9;
	Thu, 14 Jan 2016 20:10:12 +0100 (CET)
Date: Thu, 14 Jan 2016 20:07:35 +0100
From: Felix Janda 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
Message-ID: <20160114190735.GB627@nyan>
X-ASG-Orig-Subj: Re: [PATCH 2/6] libxfs/linux.c: Replace use of ustat by stat
References: 
 <20160112195945.GC568@nyan>
 <20160113075501.GC21939@infradead.org>
 <20160113174241.GC10942@nyan>
 <20160114102009.GD27644@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160114102009.GD27644@infradead.org>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Barracuda-Connect: mout01.posteo.de[185.67.36.65]
X-Barracuda-Start-Time: 1452798613
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26128
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Christoph Hellwig wrote:
> On Wed, Jan 13, 2016 at 06:42:41PM +0100, Felix Janda wrote:
> > Is this really performance relevant?
> 
> I'm not entirely sure.  There are systems with giant amounts of mounted
> file systems, in which case iterating each of them and requiring
> multiple syscalls each might not be a good idea.

Of course, it is not very elegant...

> > Hmm, so we couldn't we use for linux just something like
> > 
> > int
> > platform_check_ismounted(char *name, char *block, struct stat64 *s, int verbose)
> > {
> > 	int fd = open(block, O_EXCL);
> > 	int e = errno;
> > 	
> > 	if (fd != -1) close(fd);
> > 	if (e == EBUSY) {
> > 		if (verbose)
> > 			fprintf(stderr, _("%s: %s contains a mounted filesystem\n"), progname, name);
> > 		return 1;
> > 	}
> > 
> > 	return 0;
> 
> Please give me a couple of days to come up with an idea to use O_EXCL
> on Linux and work around it on other platforms in a more integrated way.

No problem. Thank you!

Felix

From s.priebe@profihost.ag  Fri Jan 15 03:08:07 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id A06A67F47
	for ; Fri, 15 Jan 2016 03:08:06 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id C3E50AC004
	for ; Fri, 15 Jan 2016 01:08:02 -0800 (PST)
X-ASG-Debug-ID: 1452848872-04cb6c75dc10c930001-NocioJ
Received: from mail-ph.de-nserver.de (mail-ph.de-nserver.de [85.158.179.214]) by cuda.sgi.com with ESMTP id lWxOPM5evZ8Zi4Ki (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 15 Jan 2016 01:07:53 -0800 (PST)
X-Barracuda-Envelope-From: s.priebe@profihost.ag
X-Barracuda-Apparent-Source-IP: 85.158.179.214
Received: (qmail 29036 invoked from network); 15 Jan 2016 10:07:52 +0100
X-Fcrdns: No
Received: from phoffice.de-nserver.de (HELO [10.242.2.14]) (185.39.223.5)
  (smtp-auth username s.priebe@profihost.ag, mechanism plain)
  by mail-ph.de-nserver.de (qpsmtpd/0.92) with (ECDHE-RSA-AES256-SHA encrypted) ESMTPSA; Fri, 15 Jan 2016 10:07:52 +0100
To: "" ,
 "xfs@oss.sgi.com" ,
 "xfs-masters@oss.sgi.com" 
From: Stefan Priebe 
Subject: xfs deadlocks since upgrading to 4.1.15
Message-ID: <5698B6EE.4000502@profihost.ag>
X-ASG-Orig-Subj: xfs deadlocks since upgrading to 4.1.15
Date: Fri, 15 Jan 2016 10:07:58 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-User-Auth: Auth by s.priebe@profihost.ag through 185.39.223.5
X-Barracuda-Connect: mail-ph.de-nserver.de[85.158.179.214]
X-Barracuda-Start-Time: 1452848873
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26149
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Hello,

since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle 
call traces show xfs.

Attention all those are down top instead of top down (sorry for the 
wrong order).

NR 1:

2016-01-14 00:17:23     [] rtnl_lock+0x15/0x20
2016-01-14 00:17:23     [] rtnetlink_rcv+0x16/0x40
2016-01-14 00:17:23     [] netlink_unicast+0x13a/0x1d0
2016-01-14 00:17:23     [] netlink_sendmsg+0x20a/0x600
2016-01-14 00:17:23     [] SYSC_sendto+0x111/0x170
2016-01-14 00:17:23     [] ? 
context_tracking_user_exit+0x13/0x20
2016-01-14 00:17:23     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-14 00:17:23     [] SyS_sendto+0xe/0x10
2016-01-14 00:17:23     [] system_call_fastpath+0x12/0x71
2016-01-14 00:17:22     [] xlog_cil_push+0x22f/0x3f0
2016-01-14 00:17:22     [] xlog_cil_push_work+0x15/0x20
2016-01-14 00:17:22     [] process_one_work+0x14f/0x410
2016-01-14 00:17:22     [] 
process_scheduled_works+0x2c/0x40
2016-01-14 00:17:22     [] worker_thread+0x28a/0x510
2016-01-14 00:17:22     [] ? __schedule+0x4f1/0x990
2016-01-14 00:17:22     [] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:22     [] kthread+0xc9/0xe0
2016-01-14 00:17:22     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:22     [] ret_from_fork+0x42/0x70
2016-01-14 00:17:22     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:22     INFO: task qpsmtpd p:25 :::12850 blocked for 
more than 120 seconds.
2016-01-14 00:17:22     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:22     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:22     qpsmtpd p:25 :: D ffff88071f807c48 0 12850 6124 
0x00000000
2016-01-14 00:17:22     ffff88071f807c48 ffff8805aabf0000 
ffff8806c5780000 ffff88083ffd7000
2016-01-14 00:17:22     ffff88071f808000 ffffffff93ce6444 
ffff8806c5780000 00000000ffffffff
2016-01-14 00:17:22     ffffffff93ce6448 ffff88071f807c68 
ffffffff9363c167 ffff88042b9f0000
2016-01-14 00:17:22     Call Trace:
2016-01-14 00:17:22     [] schedule+0x37/0x90
2016-01-14 00:17:22     [] 
schedule_preempt_disabled+0xe/0x10
2016-01-14 00:17:22     [] 
__mutex_lock_slowpath+0x95/0x110
2016-01-14 00:17:22     [] mutex_lock+0x1b/0x2f
2016-01-14 00:17:21     [] __kmalloc+0x1c7/0x240
2016-01-14 00:17:21     [] alloc_fdmem+0x24/0x40
2016-01-14 00:17:21     [] alloc_fdtable+0x77/0x100
2016-01-14 00:17:21     [] dup_fd+0x262/0x330
2016-01-14 00:17:21     [] 
copy_process.part.37+0x76a/0x1a80
2016-01-14 00:17:21     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-14 00:17:21     [] do_fork+0xbc/0x350
2016-01-14 00:17:21     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-14 00:17:21     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:21     [] SyS_clone+0x16/0x20
2016-01-14 00:17:21     [] system_call_fastpath+0x12/0x71
2016-01-14 00:17:21     INFO: task kworker/7:0:23773 blocked for more 
than 120 seconds.
2016-01-14 00:17:21     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:21     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:21     kworker/7:0 D ffff8805243bbb38 0 23773 2 0x00000000
2016-01-14 00:17:21     Workqueue: xfs-cil/sdb1 xlog_cil_push_work
2016-01-14 00:17:21     ffff8805243bbb38 ffff88082d941d50 
ffff880790560000 0000000000000000
2016-01-14 00:17:21     ffff8805243bc000 ffff88010af7c0b8 
ffff88042c067d28 ffff880790560000
2016-01-14 00:17:21     ffff88042c067d00 ffff8805243bbb58 
ffffffff9363c167 ffff88042c067c00
2016-01-14 00:17:21     Call Trace:
2016-01-14 00:17:21     [] schedule+0x37/0x90
2016-01-14 00:17:21     [] 
xlog_state_get_iclog_space+0xdb/0x2e0
2016-01-14 00:17:21     [] ? xlog_bdstrat+0x34/0x70
2016-01-14 00:17:21     [] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:21     [] xlog_write+0x18e/0x700
2016-01-14 00:17:20     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:20     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:20     [] ? aac_fib_send+0x2c4/0x4f0 
[aacraid]
2016-01-14 00:17:20     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:20     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-14 00:17:20     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:20     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:20     [] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:20     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:20     [] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:20     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:20     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:20     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:20     [] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:20     [] shrink_slab+0x216/0x3d0
2016-01-14 00:17:20     [] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:20     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:20     [] try_to_free_pages+0xda/0x160
2016-01-14 00:17:20     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:20     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:20     [] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:20     [] new_slab+0x32b/0x450
2016-01-14 00:17:20     [] ? zone_statistics+0x98/0xc0
2016-01-14 00:17:20     [] __slab_alloc+0x2de/0x433
2016-01-14 00:17:20     [] ? alloc_fdmem+0x24/0x40
2016-01-14 00:17:19     [] ida_pre_get+0x69/0x100
2016-01-14 00:17:19     [] ida_simple_get+0x4e/0xf0
2016-01-14 00:17:19     [] ? 
finish_task_switch+0x11e/0x1a0
2016-01-14 00:17:19     [] create_worker+0x4b/0x1f0
2016-01-14 00:17:19     [] ? mod_timer+0x14a/0x1c0
2016-01-14 00:17:19     [] worker_thread+0x3ef/0x510
2016-01-14 00:17:19     [] ? __schedule+0x4f1/0x990
2016-01-14 00:17:19     [] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:19     [] kthread+0xc9/0xe0
2016-01-14 00:17:19     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:19     [] ret_from_fork+0x42/0x70
2016-01-14 00:17:19     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:19     INFO: task httpd:22880 blocked for more than 120 
seconds.
2016-01-14 00:17:19     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:19     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:19     httpd D ffff88005557b108 0 22880 22644 0x00000000
2016-01-14 00:17:19     ffff88005557b108 ffff88042dbdbaa0 
ffff8807e65a57f0 ffff88005557b1b8
2016-01-14 00:17:19     ffff88005557c000 ffff88005557b270 
7fffffffffffffff ffff8807e65a57f0
2016-01-14 00:17:19     ffffffffffffffff ffff88005557b128 
ffffffff9363c167 0000000000000292
2016-01-14 00:17:19     Call Trace:
2016-01-14 00:17:19     [] schedule+0x37/0x90
2016-01-14 00:17:19     [] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:19     [] ? cfq_set_request+0xa5/0x4c0
2016-01-14 00:17:19     [] wait_for_completion+0x9f/0x170
2016-01-14 00:17:19     [] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:19     [] flush_work+0xff/0x1d0
2016-01-14 00:17:18     [] ? 
__cfq_set_active_queue+0xf8/0x230
2016-01-14 00:17:18     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:18     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:18     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:18     [] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:18     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:18     [] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:18     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:18     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:18     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:18     [] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:18     [] shrink_slab+0x216/0x3d0
2016-01-14 00:17:18     [] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:18     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:18     [] try_to_free_pages+0xda/0x160
2016-01-14 00:17:18     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:18     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:18     [] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:18     [] new_slab+0x32b/0x450
2016-01-14 00:17:18     [] ? try_to_wake_up+0x1f3/0x330
2016-01-14 00:17:18     [] __slab_alloc+0x2de/0x433
2016-01-14 00:17:18     [] ? ida_pre_get+0x69/0x100
2016-01-14 00:17:18     [] ? __wake_up+0x53/0x70
2016-01-14 00:17:18     [] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:18     [] ? pick_next_entity+0x90/0x170
2016-01-14 00:17:17     [] worker_thread+0x3ef/0x510
2016-01-14 00:17:17     [] ? __schedule+0x4f1/0x990
2016-01-14 00:17:17     [] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:17     [] kthread+0xc9/0xe0
2016-01-14 00:17:17     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:17     [] ret_from_fork+0x42/0x70
2016-01-14 00:17:17     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:17     INFO: task kworker/7:0H:7845 blocked for more 
than 120 seconds.
2016-01-14 00:17:17     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:17     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:17     kworker/7:0H D ffff8806ec7ff108 0 7845 2 0x00000000
2016-01-14 00:17:17     ffff8806ec7ff108 ffff88044840d7f0 
ffff88082762baa0 ffff88044840d7f0
2016-01-14 00:17:17     ffff8806ec800000 ffff8806ec7ff270 
7fffffffffffffff ffff88082762baa0
2016-01-14 00:17:17     ffffffffffffffff ffff8806ec7ff128 
ffffffff9363c167 ffff8806ec7ff148
2016-01-14 00:17:17     Call Trace:
2016-01-14 00:17:17     [] schedule+0x37/0x90
2016-01-14 00:17:17     [] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:17     [] ? 
ttwu_do_activate.constprop.91+0x66/0x70
2016-01-14 00:17:17     [] ? try_to_wake_up+0x1f3/0x330
2016-01-14 00:17:17     [] ? cfq_set_request+0x2ff/0x4c0
2016-01-14 00:17:17     [] wait_for_completion+0x9f/0x170
2016-01-14 00:17:17     [] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:17     [] flush_work+0xff/0x1d0
2016-01-14 00:17:17     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:17     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:16     [] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:16     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:16     [] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:16     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:16     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:16     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:16     [] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:16     [] shrink_slab+0x216/0x3d0
2016-01-14 00:17:16     [] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:16     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:16     [] try_to_free_pages+0xda/0x160
2016-01-14 00:17:16     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:16     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:16     [] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:16     [] new_slab+0x32b/0x450
2016-01-14 00:17:16     [] __slab_alloc+0x2de/0x433
2016-01-14 00:17:16     [] ? ida_pre_get+0x69/0x100
2016-01-14 00:17:16     [] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:16     [] ? native_sched_clock+0x35/0x80
2016-01-14 00:17:16     [] ida_pre_get+0x69/0x100
2016-01-14 00:17:16     [] ida_simple_get+0x4e/0xf0
2016-01-14 00:17:16     [] ? 
finish_task_switch+0x11e/0x1a0
2016-01-14 00:17:16     [] create_worker+0x4b/0x1f0
2016-01-14 00:17:16     [] ? mod_timer+0x14a/0x1c0
2016-01-14 00:17:15     [] do_fsync+0x38/0x70
2016-01-14 00:17:15     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:15     [] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:15     [] SyS_fdatasync+0x13/0x20
2016-01-14 00:17:15     [] system_call_fastpath+0x12/0x71
2016-01-14 00:17:15     INFO: task kworker/7:2:22195 blocked for more 
than 120 seconds.
2016-01-14 00:17:15     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:15     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:15     kworker/7:2 D ffff88009ac93108 0 22195 2 0x00000000
2016-01-14 00:17:15     ffff88009ac93108 ffff88082d941d50 
ffff88044840d7f0 0000000950d64c61
2016-01-14 00:17:15     ffff88009ac94000 ffff88009ac93270 
7fffffffffffffff ffff88044840d7f0
2016-01-14 00:17:15     ffffffffffffffff ffff88009ac93128 
ffffffff9363c167 74612e656e6f6568
2016-01-14 00:17:15     Call Trace:
2016-01-14 00:17:15     [] schedule+0x37/0x90
2016-01-14 00:17:15     [] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:15     [] wait_for_completion+0x9f/0x170
2016-01-14 00:17:15     [] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:15     [] flush_work+0xff/0x1d0
2016-01-14 00:17:15     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:15     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:15     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:15     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-14 00:17:15     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:15     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:14     [] ? tcp_write_xmit+0x2b1/0xda0
2016-01-14 00:17:14     [] 
sk_prot_alloc.isra.38+0x41/0x170
2016-01-14 00:17:14     [] sk_alloc+0x33/0xb0
2016-01-14 00:17:14     [] unix_create1+0x51/0x1a0
2016-01-14 00:17:14     [] unix_create+0x73/0x90
2016-01-14 00:17:14     [] __sock_create+0xf0/0x1f0
2016-01-14 00:17:14     [] ? rcu_eqs_enter+0x68/0x90
2016-01-14 00:17:14     [] sock_create+0x30/0x40
2016-01-14 00:17:14     [] SyS_socketpair+0x6c/0x1d0
2016-01-14 00:17:14     [] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:14     [] system_call_fastpath+0x12/0x71
2016-01-14 00:17:14     INFO: task imap:26593 blocked for more than 120 
seconds.
2016-01-14 00:17:14     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:14     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:14     imap D ffff880675a5fdc8 0 26593 4498 0x00000000
2016-01-14 00:17:14     ffff880675a5fdc8 ffff88082d943aa0 
ffff8807abab0000 ffff8806b0702f00
2016-01-14 00:17:14     ffff880675a60000 ffff88042b1f9928 
ffff88042b1f9800 ffff88042bba4c18
2016-01-14 00:17:14     ffff8807abab0000 ffff880675a5fde8 
ffffffff9363c167 0000fddf00010800
2016-01-14 00:17:14     Call Trace:
2016-01-14 00:17:14     [] schedule+0x37/0x90
2016-01-14 00:17:14     [] _xfs_log_force_lsn+0x153/0x2c0
2016-01-14 00:17:14     [] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:14     [] xfs_dir_fsync+0x86/0xf0
2016-01-14 00:17:14     [] vfs_fsync_range+0x46/0xc0
2016-01-14 00:17:14     [] vfs_fsync+0x1c/0x20
2016-01-14 00:17:13     [] 
__mutex_lock_slowpath+0x95/0x110
2016-01-14 00:17:13     [] mutex_lock+0x1b/0x2f
2016-01-14 00:17:13     [] 
xfs_reclaim_inodes_ag+0x307/0x3d0
2016-01-14 00:17:13     [] ? spin_unlock+0x10/0x10
2016-01-14 00:17:13     [] ? __call_rcu+0x3f/0x360
2016-01-14 00:17:13     [] ? inode_lru_isolate+0x89/0x1c0
2016-01-14 00:17:13     [] ? 
__list_lru_walk_one.isra.6+0x158/0x170
2016-01-14 00:17:13     [] ? 
insert_inode_locked+0x180/0x180
2016-01-14 00:17:13     [] ? 
insert_inode_locked+0x180/0x180
2016-01-14 00:17:13     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:13     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:13     [] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:13     [] shrink_slab+0x216/0x3d0
2016-01-14 00:17:13     [] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:13     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:13     [] try_to_free_pages+0xda/0x160
2016-01-14 00:17:13     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:13     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:13     [] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:13     [] new_slab+0x32b/0x450
2016-01-14 00:17:13     [] __slab_alloc+0x2de/0x433
2016-01-14 00:17:13     [] ? 
sk_prot_alloc.isra.38+0x41/0x170
2016-01-14 00:17:13     [] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:12     Call Trace:
2016-01-14 00:17:12     [] schedule+0x37/0x90
2016-01-14 00:17:12     [] _xfs_log_force_lsn+0x26b/0x2c0
2016-01-14 00:17:12     [] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:12     [] xfs_file_fsync+0x18f/0x1e0
2016-01-14 00:17:12     [] vfs_fsync_range+0x46/0xc0
2016-01-14 00:17:12     [] vfs_fsync+0x1c/0x20
2016-01-14 00:17:12     [] do_fsync+0x38/0x70
2016-01-14 00:17:12     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:12     [] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:12     [] SyS_fsync+0x10/0x20
2016-01-14 00:17:12     [] system_call_fastpath+0x12/0x71
2016-01-14 00:17:12     INFO: task pop3-login:7921 blocked for more than 
120 seconds.
2016-01-14 00:17:12     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:12     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:12     pop3-login D ffff88082c5234f8 0 7921 4498 0x00000000
2016-01-14 00:17:12     ffff88082c5234f8 ffff88042dbdbaa0 
ffff880827443aa0 ffff88082c523568
2016-01-14 00:17:12     ffff88082c524000 ffff88082bb63b6c 
ffff880827443aa0 00000000ffffffff
2016-01-14 00:17:12     ffff88082bb63b70 ffff88082c523518 
ffffffff9363c167 0000000000000000
2016-01-14 00:17:12     Call Trace:
2016-01-14 00:17:12     [] schedule+0x37/0x90
2016-01-14 00:17:12     [] 
schedule_preempt_disabled+0xe/0x10
2016-01-14 00:17:11     [] flush_work+0xff/0x1d0
2016-01-14 00:17:11     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:11     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:11     [] ? 
try_to_del_timer_sync+0x4f/0x70
2016-01-14 00:17:11     [] _xfs_log_force+0x6b/0x280
2016-01-14 00:17:11     [] ? 
xfs_buf_delwri_submit_nowait+0x20/0x30
2016-01-14 00:17:11     [] xfs_log_force+0x2a/0x90
2016-01-14 00:17:11     [] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:11     [] xfsaild+0x143/0x5f0
2016-01-14 00:17:11     [] ? __schedule+0x4f1/0x990
2016-01-14 00:17:11     [] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:11     [] kthread+0xc9/0xe0
2016-01-14 00:17:11     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:11     [] ret_from_fork+0x42/0x70
2016-01-14 00:17:11     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:11     INFO: task mysqld:7538 blocked for more than 120 
seconds.
2016-01-14 00:17:11     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:11     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:11     mysqld D ffff8803e708bd98 0 7538 3771 0x00000000
2016-01-14 00:17:11     ffff8803e708bd98 ffff88082d941d50 
ffff8803f1ebd7f0 ffff88042bba4c00
2016-01-14 00:17:11     ffff8803e708c000 ffff88042b1f9928 
ffff88042b1f9800 0000000000000000
2016-01-14 00:17:11     ffff8803f1ebd7f0 ffff8803e708bdb8 
ffffffff9363c167 ffff8803f1ebd7f0
2016-01-14 00:17:10     [] ? __schedule+0x4f1/0x990
2016-01-14 00:17:10     [] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:10     [] kthread+0xc9/0xe0
2016-01-14 00:17:10     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:10     [] ret_from_fork+0x42/0x70
2016-01-14 00:17:10     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:10     INFO: task xfsaild/sdb1:2068 blocked for more 
than 120 seconds.
2016-01-14 00:17:10     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:10     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:10     xfsaild/sdb1 D ffff88042529fad8 0 2068 2 0x00000000
2016-01-14 00:17:10     ffff88042529fad8 ffff88082d948000 
ffff8800b94ad7f0 ffff88082a8401d0
2016-01-14 00:17:10     ffff8804252a0000 ffff88042529fc40 
7fffffffffffffff ffff8800b94ad7f0
2016-01-14 00:17:10     ffffffffffffffff ffff88042529faf8 
ffffffff9363c167 0000000000000000
2016-01-14 00:17:10     Call Trace:
2016-01-14 00:17:10     [] schedule+0x37/0x90
2016-01-14 00:17:10     [] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:10     [] ? part_round_stats+0x5c/0x70
2016-01-14 00:17:10     [] ? kernfs_path_locked+0x49/0xa0
2016-01-14 00:17:10     [] ? elv_rb_add+0x62/0x70
2016-01-14 00:17:10     [] wait_for_completion+0x9f/0x170
2016-01-14 00:17:10     [] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:09     INFO: task xfsaild/sda3:598 blocked for more 
than 120 seconds.
2016-01-14 00:17:09     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:09     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:09     xfsaild/sda3 D ffff88042a8d7d18 0 598 2 0x00000000
2016-01-14 00:17:09     ffff88042a8d7d18 ffff880102c93aa0 
ffff88042c248000 0000000000000282
2016-01-14 00:17:09     ffff88042a8d8000 ffff88042c248000 
0000000000000000 ffff88042b1f9800
2016-01-14 00:17:09     ffff88042bba4100 ffff88042a8d7d38 
ffffffff9363c167 ffff88042b1f9928
2016-01-14 00:17:09     Call Trace:
2016-01-14 00:17:09     [] schedule+0x37/0x90
2016-01-14 00:17:09     [] _xfs_log_force+0x189/0x280
2016-01-14 00:17:09     [] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:09     [] xfs_log_force+0x2a/0x90
2016-01-14 00:17:09     [] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:09     [] xfsaild+0x143/0x5f0


Nr 2:

2016-01-15 02:01:04     [] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:04     [] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:04     [] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:04     [] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:04     [] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:04     [] xfs_read_agi+0x95/0xf0
2016-01-15 02:01:04     [] xfs_iunlink_remove+0x4c/0x420
2016-01-15 02:01:04     [] ? 
xlog_grant_head_check+0x4e/0xf0
2016-01-15 02:01:04     [] xfs_ifree+0x27/0xd0
2016-01-15 02:01:04     [] ? xfs_trans_ijoin+0x36/0x60
2016-01-15 02:01:04     [] xfs_inactive_ifree+0xc1/0x230
2016-01-15 02:01:04     [] xfs_inactive+0x96/0x150
2016-01-15 02:01:04     [] xfs_fs_evict_inode+0x6f/0xb0
2016-01-15 02:01:04     [] evict+0xc0/0x1a0
2016-01-15 02:01:04     [] iput+0x18b/0x1f0
2016-01-15 02:01:04     [] __dentry_kill+0x198/0x200
2016-01-15 02:01:04     [] dput+0x1c0/0x220
2016-01-15 02:01:04     [] __fput+0x164/0x1e0
2016-01-15 02:01:04     [] ____fput+0xe/0x10
2016-01-15 02:01:04     [] task_work_run+0xbf/0x100
2016-01-15 02:01:04     [] do_notify_resume+0x88/0xa0
2016-01-15 02:01:04     [] int_signal+0x12/0x17
2016-01-15 02:01:03     [] xfs_inactive+0x96/0x150
2016-01-15 02:01:03     [] xfs_fs_evict_inode+0x6f/0xb0
2016-01-15 02:01:03     [] evict+0xc0/0x1a0
2016-01-15 02:01:03     [] iput+0x18b/0x1f0
2016-01-15 02:01:03     [] __dentry_kill+0x198/0x200
2016-01-15 02:01:03     [] dput+0x1c0/0x220
2016-01-15 02:01:03     [] __fput+0x164/0x1e0
2016-01-15 02:01:03     [] ____fput+0xe/0x10
2016-01-15 02:01:03     [] task_work_run+0xbf/0x100
2016-01-15 02:01:03     [] do_notify_resume+0x88/0xa0
2016-01-15 02:01:03     [] int_signal+0x12/0x17
2016-01-15 02:01:03     INFO: task php5.4:9661 blocked for more than 120 
seconds.
2016-01-15 02:01:03     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:03     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:03     php5.4 D ffff8800189278e8 0 9661 15202 0x00000004
2016-01-15 02:01:03     ffff8800189278e8 ffff880624309d50 
ffff88000b4a57f0 ffff880000000002
2016-01-15 02:01:03     ffff880018928000 7fffffffffffffff 
ffff88000b4a57f0 0000000000000002
2016-01-15 02:01:03     0000000000000000 ffff880018927908 
ffffffff8e63c967 0000000000000000
2016-01-15 02:01:03     Call Trace:
2016-01-15 02:01:03     [] schedule+0x37/0x90
2016-01-15 02:01:03     [] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:03     [] ? blk_peek_request+0x87/0x290
2016-01-15 02:01:03     [] __down_common+0xaf/0x118
2016-01-15 02:01:03     [] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:03     [] __down+0x1d/0x1f
2016-01-15 02:01:03     [] down+0x41/0x50
2016-01-15 02:01:02     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:02     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:02     php5.4 D ffff8800017338e8 0 9657 15202 0x00000004
2016-01-15 02:01:02     ffff8800017338e8 ffff8806242f0000 
ffff8800010a1d50 0000000000000002
2016-01-15 02:01:02     ffff880001734000 7fffffffffffffff 
ffff8800010a1d50 0000000000000002
2016-01-15 02:01:02     0000000000000000 ffff880001733908 
ffffffff8e63c967 0000000000000000
2016-01-15 02:01:02     Call Trace:
2016-01-15 02:01:02     [] schedule+0x37/0x90
2016-01-15 02:01:02     [] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:02     [] ? __enqueue_entity+0x78/0x80
2016-01-15 02:01:02     [] __down_common+0xaf/0x118
2016-01-15 02:01:02     [] ? zone_statistics+0x98/0xc0
2016-01-15 02:01:02     [] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:02     [] __down+0x1d/0x1f
2016-01-15 02:01:02     [] down+0x41/0x50
2016-01-15 02:01:02     [] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:02     [] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:02     [] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:02     [] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:02     [] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:02     [] xfs_read_agi+0x95/0xf0
2016-01-15 02:01:02     [] xfs_iunlink_remove+0x4c/0x420
2016-01-15 02:01:02     [] ? 
xlog_grant_head_check+0x4e/0xf0
2016-01-15 02:01:02     [] xfs_ifree+0x27/0xd0
2016-01-15 02:01:02     [] ? xfs_trans_ijoin+0x36/0x60
2016-01-15 02:01:02     [] xfs_inactive_ifree+0xc1/0x230
2016-01-15 02:01:01     [] xfs_ialloc_read_agi+0x43/0xe0
2016-01-15 02:01:01     [] xfs_dialloc+0xf1/0x2d0
2016-01-15 02:01:01     [] xfs_ialloc+0x5b/0x550
2016-01-15 02:01:01     [] ? kmem_zone_alloc+0x77/0x100
2016-01-15 02:01:01     [] ? 
xfs_trans_mod_dquot+0x47/0x190
2016-01-15 02:01:01     [] xfs_dir_ialloc+0x6a/0x290
2016-01-15 02:01:01     [] xfs_create+0x2bb/0x6e0
2016-01-15 02:01:01     [] ? native_sched_clock+0x35/0x80
2016-01-15 02:01:01     [] ? get_cached_acl+0x21/0x70
2016-01-15 02:01:01     [] xfs_generic_create+0xd3/0x2b0
2016-01-15 02:01:01     [] ? __d_lookup+0x11d/0x160
2016-01-15 02:01:01     [] xfs_vn_mknod+0x14/0x20
2016-01-15 02:01:01     [] xfs_vn_create+0x13/0x20
2016-01-15 02:01:01     [] vfs_create+0x77/0xc0
2016-01-15 02:01:01     [] do_last.isra.57+0x73d/0xd30
2016-01-15 02:01:01     [] ? path_init+0xc4/0x430
2016-01-15 02:01:01     [] path_openat+0x92/0x5a0
2016-01-15 02:01:01     [] ? user_path_at_empty+0x68/0xa0
2016-01-15 02:01:01     [] do_filp_open+0x43/0xa0
2016-01-15 02:01:01     [] ? __alloc_fd+0x134/0x1b0
2016-01-15 02:01:01     [] do_sys_open+0x13c/0x230
2016-01-15 02:01:01     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:01:01     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:01:01     [] SyS_open+0x22/0x30
2016-01-15 02:01:01     [] system_call_fastpath+0x12/0x71
2016-01-15 02:01:01     INFO: task php5.4:9657 blocked for more than 120 
seconds.
2016-01-15 02:01:00     INFO: task php5.2:6976 blocked for more than 120 
seconds.
2016-01-15 02:01:00     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:00     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:00     php5.2 D ffff88015bd87568 0 6976 15202 0x00000004
2016-01-15 02:01:00     ffff88015bd87568 ffff8806242c0000 
ffff8801938e0000 ffff8800374ded00
2016-01-15 02:01:00     ffff88015bd88000 7fffffffffffffff 
ffff8801938e0000 0000000000000002
2016-01-15 02:01:00     0000000000000000 ffff88015bd87588 
ffffffff8e63c967 00000000000001ed
2016-01-15 02:01:00     Call Trace:
2016-01-15 02:01:00     [] schedule+0x37/0x90
2016-01-15 02:01:00     [] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:00     [] ? dequeue_entity+0x13c/0x580
2016-01-15 02:01:00     [] __down_common+0xaf/0x118
2016-01-15 02:01:00     [] ? 
arch_vtime_task_switch+0x40/0x90
2016-01-15 02:01:00     [] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:00     [] __down+0x1d/0x1f
2016-01-15 02:01:00     [] down+0x41/0x50
2016-01-15 02:01:00     [] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:00     [] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:00     [] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:00     [] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:00     [] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:00     [] xfs_read_agi+0x95/0xf0
2016-01-15 02:00:59     INFO: task kworker/4:1H:6823 blocked for more 
than 120 seconds.
2016-01-15 02:00:59     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:59     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:59     kworker/4:1H D ffff880302813c88 0 6823 2 0x00000000
2016-01-15 02:00:59     Workqueue: xfs-log/sda3 xfs_log_worker
2016-01-15 02:00:59     ffff880302813c88 ffff8806243ed7f0 
ffff8800038b0000 ffff880621729400
2016-01-15 02:00:59     ffff880302814000 ffff8800bb179128 
ffff8800bb179000 0000000000000000
2016-01-15 02:00:59     ffff8800038b0000 ffff880302813ca8 
ffffffff8e63c967 ffff8800038b0000
2016-01-15 02:00:59     Call Trace:
2016-01-15 02:00:59     [] schedule+0x37/0x90
2016-01-15 02:00:59     [] _xfs_log_force_lsn+0x26b/0x2c0
2016-01-15 02:00:59     [] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:59     [] xfs_trans_commit+0x1e3/0x220
2016-01-15 02:00:59     [] xfs_sync_sb+0x60/0xa0
2016-01-15 02:00:59     [] ? 
xfs_log_need_covered+0x74/0x100
2016-01-15 02:00:59     [] xfs_log_worker+0x4d/0x50
2016-01-15 02:00:59     [] process_one_work+0x14f/0x410
2016-01-15 02:00:59     [] worker_thread+0x119/0x510
2016-01-15 02:00:59     [] ? __schedule+0x4f1/0x990
2016-01-15 02:00:59     [] ? 
process_scheduled_works+0x40/0x40
2016-01-15 02:00:59     [] kthread+0xc9/0xe0
2016-01-15 02:00:59     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:59     [] ret_from_fork+0x42/0x70
2016-01-15 02:00:59     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:58     [] ? inode_lru_isolate+0x89/0x1c0
2016-01-15 02:00:58     [] ? 
__list_lru_walk_one.isra.6+0x158/0x170
2016-01-15 02:00:58     [] ? 
insert_inode_locked+0x180/0x180
2016-01-15 02:00:58     [] ? 
insert_inode_locked+0x180/0x180
2016-01-15 02:00:58     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:58     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:58     [] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:58     [] shrink_slab+0x216/0x3d0
2016-01-15 02:00:58     [] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:58     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-15 02:00:58     [] try_to_free_pages+0xda/0x160
2016-01-15 02:00:58     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-15 02:00:58     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-15 02:00:58     [] 
alloc_kmem_pages_node+0x65/0x130
2016-01-15 02:00:58     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-15 02:00:58     [] 
copy_process.part.37+0x109/0x1a80
2016-01-15 02:00:58     [] ? kmem_cache_alloc+0x193/0x1f0
2016-01-15 02:00:58     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-15 02:00:58     [] ? account_user_time+0x99/0xb0
2016-01-15 02:00:58     [] do_fork+0xbc/0x350
2016-01-15 02:00:58     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:58     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:58     [] SyS_clone+0x16/0x20
2016-01-15 02:00:58     [] system_call_fastpath+0x12/0x71
2016-01-15 02:00:57     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:57     [] ? syscall_trace_leave+0x95/0xf0
2016-01-15 02:00:57     [] SyS_fsync+0x10/0x20
2016-01-15 02:00:57     [] system_call_fastpath+0x12/0x71
2016-01-15 02:00:57     INFO: task auth:9631 blocked for more than 120 
seconds.
2016-01-15 02:00:57     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:57     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:57     auth D ffff8809eea33588 0 9631 9616 0x00000004
2016-01-15 02:00:57     ffff8809eea33588 ffff8806242c1d50 
ffff880b97f19d50 ffff8809eea33568
2016-01-15 02:00:57     ffff8809eea34000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:57     ffff880b97f19d50 ffff8809eea335a8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:57     Call Trace:
2016-01-15 02:00:57     [] schedule+0x37/0x90
2016-01-15 02:00:57     [] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:57     [] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:57     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:57     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:57     [] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:57     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:57     [] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:57     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:57     [] ? 
__call_rcu_nocb_enqueue+0xe6/0xf0
2016-01-15 02:00:56     [] xfs_file_fsync+0x18f/0x1e0
2016-01-15 02:00:56     [] vfs_fsync_range+0x46/0xc0
2016-01-15 02:00:56     [] vfs_fsync+0x1c/0x20
2016-01-15 02:00:56     [] do_fsync+0x38/0x70
2016-01-15 02:00:56     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:56     [] ? syscall_trace_leave+0x95/0xf0
2016-01-15 02:00:56     [] SyS_fsync+0x10/0x20
2016-01-15 02:00:56     [] system_call_fastpath+0x12/0x71
2016-01-15 02:00:56     INFO: task mysqld:20600 blocked for more than 
120 seconds.
2016-01-15 02:00:56     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:56     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:56     mysqld D ffff8809218ffd98 0 20600 5893 0x00000000
2016-01-15 02:00:56     ffff8809218ffd98 ffff8806242d8000 
ffff880123ef57f0 0000000000000000
2016-01-15 02:00:56     ffff880921900000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:56     ffff880123ef57f0 ffff8809218ffdb8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:56     Call Trace:
2016-01-15 02:00:56     [] schedule+0x37/0x90
2016-01-15 02:00:56     [] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:56     [] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:56     [] xfs_file_fsync+0x18f/0x1e0
2016-01-15 02:00:56     [] vfs_fsync_range+0x46/0xc0
2016-01-15 02:00:56     [] vfs_fsync+0x1c/0x20
2016-01-15 02:00:56     [] do_fsync+0x38/0x70
2016-01-15 02:00:55     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-15 02:00:55     [] try_to_free_pages+0xda/0x160
2016-01-15 02:00:55     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-15 02:00:55     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-15 02:00:55     [] 
alloc_kmem_pages_node+0x65/0x130
2016-01-15 02:00:55     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-15 02:00:55     [] 
copy_process.part.37+0x109/0x1a80
2016-01-15 02:00:55     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-15 02:00:55     [] ? account_user_time+0x99/0xb0
2016-01-15 02:00:55     [] do_fork+0xbc/0x350
2016-01-15 02:00:55     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:55     [] SyS_clone+0x16/0x20
2016-01-15 02:00:55     [] system_call_fastpath+0x12/0x71
2016-01-15 02:00:55     INFO: task mysqld:7069 blocked for more than 120 
seconds.
2016-01-15 02:00:55     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:55     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:55     mysqld D ffff880f0abbbd98 0 7069 5893 0x00000000
2016-01-15 02:00:55     ffff880f0abbbd98 ffff8806242c0000 
ffff880f0ab53aa0 0000000000000000
2016-01-15 02:00:55     ffff880f0abbc000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:55     ffff880f0ab53aa0 ffff880f0abbbdb8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:55     Call Trace:
2016-01-15 02:00:55     [] schedule+0x37/0x90
2016-01-15 02:00:55     [] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:55     [] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:54     [] system_call_fastpath+0x12/0x71
2016-01-15 02:00:54     INFO: task mysqld:6683 blocked for more than 120 
seconds.
2016-01-15 02:00:54     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:54     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:54     mysqld D ffff880f0b2f3588 0 6683 5893 0x00000000
2016-01-15 02:00:54     ffff880f0b2f3588 ffff8806242dbaa0 
ffff880f09913aa0 000000000000254e
2016-01-15 02:00:54     ffff880f0b2f4000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:54     ffff880f09913aa0 ffff880f0b2f35a8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:54     Call Trace:
2016-01-15 02:00:54     [] schedule+0x37/0x90
2016-01-15 02:00:54     [] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:54     [] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:54     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:54     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:54     [] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:54     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:54     [] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:54     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:54     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:54     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:54     [] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:54     [] shrink_slab+0x216/0x3d0
2016-01-15 02:00:54     [] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:53     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:53     mysqld D ffff88064549bbc8 0 21561 5534 0x00000000
2016-01-15 02:00:53     ffff88064549bbc8 ffff88062426d7f0 
ffff88000b4cbaa0 0000000000000040
2016-01-15 02:00:53     ffff88064549c000 ffff880f1631b674 
ffff88000b4cbaa0 00000000ffffffff
2016-01-15 02:00:53     ffff880f1631b678 ffff88064549bbe8 
ffffffff8e63c967 00000000000282c2
2016-01-15 02:00:53     Call Trace:
2016-01-15 02:00:53     [] schedule+0x37/0x90
2016-01-15 02:00:53     [] 
schedule_preempt_disabled+0xe/0x10
2016-01-15 02:00:53     [] 
__mutex_lock_slowpath+0x95/0x110
2016-01-15 02:00:53     [] mutex_lock+0x1b/0x2f
2016-01-15 02:00:53     [] do_last.isra.57+0x25b/0xd30
2016-01-15 02:00:53     [] ? path_init+0xc4/0x430
2016-01-15 02:00:53     [] path_openat+0x92/0x5a0
2016-01-15 02:00:53     [] ? handle_mm_fault+0xb47/0x1860
2016-01-15 02:00:53     [] do_filp_open+0x43/0xa0
2016-01-15 02:00:53     [] ? __alloc_fd+0x134/0x1b0
2016-01-15 02:00:53     [] do_sys_open+0x13c/0x230
2016-01-15 02:00:53     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:53     [] SyS_open+0x22/0x30
2016-01-15 02:00:52     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:52     [] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:52     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:52     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:52     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:52     [] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:52     [] shrink_slab+0x216/0x3d0
2016-01-15 02:00:52     [] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:52     [] kswapd+0x527/0x980
2016-01-15 02:00:52     [] ? 
mem_cgroup_shrink_node_zone+0x150/0x150
2016-01-15 02:00:52     [] kthread+0xc9/0xe0
2016-01-15 02:00:52     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:52     [] ret_from_fork+0x42/0x70
2016-01-15 02:00:52     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:52     INFO: task mysqld:21561 blocked for more than 
120 seconds.
2016-01-15 02:00:52     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:51     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:51     kswapd1 D ffff880622e87848 0 195 2 0x00000000
2016-01-15 02:00:51     ffff880622e87848 ffff8806242d8000 
ffff880622e33aa0 0000000000000000
2016-01-15 02:00:51     ffff880622e88000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:51     ffff880622e33aa0 ffff880622e87868 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:51     Call Trace:
2016-01-15 02:00:51     [] schedule+0x37/0x90
2016-01-15 02:00:51     [] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:51     [] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:51     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:51     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:51     [] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:50     INFO: task kswapd1:195 blocked for more than 120 
seconds.
2016-01-15 02:00:50     Not tainted 4.1.15+68-ph #1

Nr 3:
2016-01-12 03:18:50     [] 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:50     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:50     [] ? account_user_time+0x99/0xb0
2016-01-12 03:18:50     [] do_fork+0xbc/0x350
2016-01-12 03:18:50     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:50     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:50     [] SyS_clone+0x16/0x20
2016-01-12 03:18:50     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:49     [] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:49     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:49     [] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:49     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:49     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:49     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:49     [] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:49     [] shrink_slab+0x216/0x3d0
2016-01-12 03:18:49     [] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:49     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:49     [] try_to_free_pages+0xda/0x160
2016-01-12 03:18:49     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:49     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:49     [] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:49     [] new_slab+0x32b/0x450
2016-01-12 03:18:49     [] __slab_alloc+0x2de/0x433
2016-01-12 03:18:49     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:49     [] ? do_page_fault+0x37/0x90
2016-01-12 03:18:49     [] 
kmem_cache_alloc_node+0x8a/0x240
2016-01-12 03:18:48     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:48     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:48     sshd D ffff880859e6f198 0 6171 1 0x00000000
2016-01-12 03:18:48     ffff880859e6f198 ffff88085c2e1d50 
ffff880855d53aa0 ffff880859e6f248
2016-01-12 03:18:48     ffff880859e70000 ffff880859e6f300 
7fffffffffffffff ffff880855d53aa0
2016-01-12 03:18:48     ffffffffffffffff ffff880859e6f1b8 
ffffffffb163c967 ffff880859e6f298
2016-01-12 03:18:48     Call Trace:
2016-01-12 03:18:48     [] schedule+0x37/0x90
2016-01-12 03:18:48     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:48     [] ? cfq_set_request+0xa5/0x4c0
2016-01-12 03:18:48     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:48     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:48     [] flush_work+0xff/0x1d0
2016-01-12 03:18:48     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:48     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:48     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:48     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:48     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:48     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:47     [] shrink_slab+0x216/0x3d0
2016-01-12 03:18:47     [] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:47     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:47     [] try_to_free_pages+0xda/0x160
2016-01-12 03:18:47     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:47     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:47     [] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:47     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:47     [] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:47     [] ? make_kgid+0x12/0x20
2016-01-12 03:18:47     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:47     [] ? account_user_time+0x99/0xb0
2016-01-12 03:18:47     [] do_fork+0xbc/0x350
2016-01-12 03:18:47     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:47     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:47     [] sys_vfork+0x1c/0x20
2016-01-12 03:18:47     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:47     INFO: task sshd:6171 blocked for more than 120 
seconds.
2016-01-12 03:18:46     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:46     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:46     [] flush_work+0xff/0x1d0
2016-01-12 03:18:46     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:46     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:46     [] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:46     [] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:46     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:46     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:46     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:46     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:46     [] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:46     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:46     [] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:46     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:46     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:46     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:46     [] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:45     [] ? make_kgid+0x12/0x20
2016-01-12 03:18:45     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:45     [] ? account_user_time+0x99/0xb0
2016-01-12 03:18:45     [] do_fork+0xbc/0x350
2016-01-12 03:18:45     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:45     [] SyS_clone+0x16/0x20
2016-01-12 03:18:45     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:45     INFO: task qmail-rspawn:6145 blocked for more 
than 120 seconds.
2016-01-12 03:18:45     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:45     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:45     qmail-rspawn D ffff88085af93348 0 6145 6125 
0x00000002
2016-01-12 03:18:45     ffff88085af93348 ffff88085c300000 
ffff8808554c9d50 0000000000000000
2016-01-12 03:18:45     ffff88085af94000 ffff88085af934b0 
7fffffffffffffff ffff8808554c9d50
2016-01-12 03:18:45     ffffffffffffffff ffff88085af93368 
ffffffffb163c967 ffff88085af933e8
2016-01-12 03:18:45     Call Trace:
2016-01-12 03:18:45     [] schedule+0x37/0x90
2016-01-12 03:18:45     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:45     [] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:44     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:44     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:44     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:44     [] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:44     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:44     [] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:44     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:44     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:44     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:44     [] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:44     [] shrink_slab+0x216/0x3d0
2016-01-12 03:18:44     [] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:44     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:44     [] try_to_free_pages+0xda/0x160
2016-01-12 03:18:44     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:44     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:44     [] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:44     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:44     [] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:43     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:43     INFO: task zabbix_agentd:6130 blocked for more 
than 120 seconds.
2016-01-12 03:18:43     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:43     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:43     zabbix_agentd D ffff881053bc7348 0 6130 6118 
0x00000004
2016-01-12 03:18:43     ffff881053bc7348 ffff88085c300000 
ffff88105a4d9d50 ffff881053bc73d8
2016-01-12 03:18:43     ffff881053bc8000 ffff881053bc74b0 
7fffffffffffffff ffff88105a4d9d50
2016-01-12 03:18:43     ffffffffffffffff ffff881053bc7368 
ffffffffb163c967 ffff881053bc7368
2016-01-12 03:18:43     Call Trace:
2016-01-12 03:18:43     [] schedule+0x37/0x90
2016-01-12 03:18:43     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:43     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:43     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:43     [] flush_work+0xff/0x1d0
2016-01-12 03:18:43     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:43     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:43     [] ? __free_pages+0x27/0x30
2016-01-12 03:18:43     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:42     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:42     [] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:42     [] shrink_slab+0x216/0x3d0
2016-01-12 03:18:42     [] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:42     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:42     [] try_to_free_pages+0xda/0x160
2016-01-12 03:18:42     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:42     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:42     [] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:42     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:42     [] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:42     [] ? make_kgid+0x12/0x20
2016-01-12 03:18:42     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:42     [] ? account_user_time+0x99/0xb0
2016-01-12 03:18:42     [] do_fork+0xbc/0x350
2016-01-12 03:18:42     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:42     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:42     [] SyS_clone+0x16/0x20
2016-01-12 03:18:41     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:41     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:41     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:41     [] flush_work+0xff/0x1d0
2016-01-12 03:18:41     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:41     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:41     [] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:41     [] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:41     [] ? part_round_stats+0x5c/0x70
2016-01-12 03:18:41     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:41     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:41     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:41     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:41     [] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:41     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:41     [] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:41     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:41     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:40     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:40     [] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:40     [] ? make_kgid+0x12/0x20
2016-01-12 03:18:40     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:40     [] ? account_user_time+0x99/0xb0
2016-01-12 03:18:40     [] do_fork+0xbc/0x350
2016-01-12 03:18:40     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:40     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:40     [] SyS_clone+0x16/0x20
2016-01-12 03:18:40     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:40     INFO: task zabbix_agentd:6129 blocked for more 
than 120 seconds.
2016-01-12 03:18:40     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:40     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:40     zabbix_agentd D ffff881053bd7348 0 6129 6118 
0x00000006
2016-01-12 03:18:40     ffff881053bd7348 ffff88085c2e1d50 
ffff88105340d7f0 ffffea0017e65180
2016-01-12 03:18:40     ffff881053bd8000 ffff881053bd74b0 
7fffffffffffffff ffff88105340d7f0
2016-01-12 03:18:40     ffffffffffffffff ffff881053bd7368 
ffffffffb163c967 ffff88085fa59650
2016-01-12 03:18:40     Call Trace:
2016-01-12 03:18:40     [] schedule+0x37/0x90
2016-01-12 03:18:39     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:39     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:39     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:39     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:39     [] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:39     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:39     [] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:39     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:39     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:39     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:39     [] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:39     [] shrink_slab+0x216/0x3d0
2016-01-12 03:18:39     [] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:39     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:39     [] try_to_free_pages+0xda/0x160
2016-01-12 03:18:39     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:39     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:39     [] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:38     INFO: task zabbix_agentd:6128 blocked for more 
than 120 seconds.
2016-01-12 03:18:38     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:38     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:38     zabbix_agentd D ffff88105341f348 0 6128 6118 
0x00000004
2016-01-12 03:18:38     ffff88105341f348 ffff88085c2e57f0 
ffff880858739d50 ffffea00009e2100
2016-01-12 03:18:38     ffff881053420000 ffff88105341f4b0 
7fffffffffffffff ffff880858739d50
2016-01-12 03:18:38     ffffffffffffffff ffff88105341f368 
ffffffffb163c967 ffff88085fad9650
2016-01-12 03:18:38     Call Trace:
2016-01-12 03:18:38     [] schedule+0x37/0x90
2016-01-12 03:18:38     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:38     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:38     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:38     [] flush_work+0xff/0x1d0
2016-01-12 03:18:38     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:38     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:38     [] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:38     [] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:38     [] ? __free_pages+0x27/0x30
2016-01-12 03:18:38     [] ? __free_slab+0xcc/0x1c0
2016-01-12 03:18:37     [] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:37     [] new_slab+0x32b/0x450
2016-01-12 03:18:37     [] __slab_alloc+0x2de/0x433
2016-01-12 03:18:37     [] ? sock_alloc_inode+0x1d/0xc0
2016-01-12 03:18:37     [] ? 
ttwu_do_activate.constprop.91+0x66/0x70
2016-01-12 03:18:37     [] ? set_fd_set+0x21/0x30
2016-01-12 03:18:37     [] ? core_sys_select+0x242/0x2f0
2016-01-12 03:18:37     [] kmem_cache_alloc+0x193/0x1f0
2016-01-12 03:18:37     [] sock_alloc_inode+0x1d/0xc0
2016-01-12 03:18:37     [] alloc_inode+0x26/0xa0
2016-01-12 03:18:37     [] new_inode_pseudo+0x1a/0x70
2016-01-12 03:18:37     [] sock_alloc+0x1e/0x80
2016-01-12 03:18:37     [] SYSC_accept4+0x7c/0x210
2016-01-12 03:18:37     [] ? vtime_account_user+0x5d/0x70
2016-01-12 03:18:37     [] ? 
context_tracking_user_exit+0x13/0x20
2016-01-12 03:18:37     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:37     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:37     [] SyS_accept+0x10/0x20
2016-01-12 03:18:37     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:36     [] ? __free_pages+0x27/0x30
2016-01-12 03:18:36     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:36     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:36     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:36     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:36     [] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:36     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:36     [] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:36     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:36     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:36     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:36     [] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:36     [] shrink_slab+0x216/0x3d0
2016-01-12 03:18:36     [] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:36     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:36     [] try_to_free_pages+0xda/0x160
2016-01-12 03:18:36     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:36     [] ? __pollwait+0xf0/0xf0
2016-01-12 03:18:36     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:35     [] SyS_clone+0x16/0x20
2016-01-12 03:18:35     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:35     INFO: task nrpe:6076 blocked for more than 120 
seconds.
2016-01-12 03:18:35     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:35     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:35     nrpe D ffff880853e6b1a8 0 6076 1 0x00000002
2016-01-12 03:18:35     ffff880853e6b1a8 ffffffffb1c134a0 
ffff8808554e57f0 ffff88105f5d6240
2016-01-12 03:18:35     ffff880853e6c000 ffff880853e6b310 
7fffffffffffffff ffff8808554e57f0
2016-01-12 03:18:35     ffffffffffffffff ffff880853e6b1c8 
ffffffffb163c967 0000000000000003
2016-01-12 03:18:35     Call Trace:
2016-01-12 03:18:35     [] schedule+0x37/0x90
2016-01-12 03:18:35     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:35     [] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:35     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:35     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:35     [] flush_work+0xff/0x1d0
2016-01-12 03:18:35     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:35     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:34     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:34     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:34     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:34     [] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:34     [] shrink_slab+0x216/0x3d0
2016-01-12 03:18:34     [] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:34     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:34     [] try_to_free_pages+0xda/0x160
2016-01-12 03:18:34     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:34     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:34     [] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:34     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:34     [] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:34     [] ? cp_new_stat+0x140/0x160
2016-01-12 03:18:34     [] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:34     [] ? account_user_time+0x99/0xb0
2016-01-12 03:18:34     [] do_fork+0xbc/0x350
2016-01-12 03:18:34     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:33     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:33     [] ? cfq_set_request+0xa5/0x4c0
2016-01-12 03:18:33     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:33     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:33     [] flush_work+0xff/0x1d0
2016-01-12 03:18:33     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:33     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:33     [] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:33     [] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:33     [] ? 
blk_attempt_plug_merge+0xd7/0xf0
2016-01-12 03:18:33     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:33     [] ? 
scsi_prep_state_check.isra.22+0x80/0x80
2016-01-12 03:18:33     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:33     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:33     [] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:33     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:33     [] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:32     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:32     [] xfs_file_fsync+0x18f/0x1e0
2016-01-12 03:18:32     [] vfs_fsync_range+0x46/0xc0
2016-01-12 03:18:32     [] vfs_fsync+0x1c/0x20
2016-01-12 03:18:32     [] do_fsync+0x38/0x70
2016-01-12 03:18:32     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:32     [] ? syscall_trace_leave+0x95/0xf0
2016-01-12 03:18:32     [] SyS_fsync+0x10/0x20
2016-01-12 03:18:32     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:32     INFO: task cron:6006 blocked for more than 120 
seconds.
2016-01-12 03:18:32     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:32     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:32     cron D ffff881053b57348 0 6006 1 0x00000000
2016-01-12 03:18:32     ffff881053b57348 ffff880858738000 
ffff88105a8057f0 ffff8810595db080
2016-01-12 03:18:32     ffff881053b58000 ffff881053b574b0 
7fffffffffffffff ffff88105a8057f0
2016-01-12 03:18:32     ffffffffffffffff ffff881053b57368 
ffffffffb163c967 ffff880000055000
2016-01-12 03:18:32     Call Trace:
2016-01-12 03:18:32     [] schedule+0x37/0x90
2016-01-12 03:18:31     [] system_call_fastpath+0x12/0x71
2016-01-12 03:18:31     INFO: task mysqld:6762 blocked for more than 120 
seconds.
2016-01-12 03:18:31     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:31     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:31     mysqld D ffff880843887b58 0 6762 5590 0x00000000
2016-01-12 03:18:31     ffff880843887b58 ffff88085c301d50 
ffff88084385baa0 ffff880572ac09d0
2016-01-12 03:18:31     ffff880843888000 ffff880843887cc0 
7fffffffffffffff ffff88084385baa0
2016-01-12 03:18:31     ffffffffffffffff ffff880843887b78 
ffffffffb163c967 0000000000000100
2016-01-12 03:18:31     Call Trace:
2016-01-12 03:18:31     [] schedule+0x37/0x90
2016-01-12 03:18:31     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:31     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:31     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:31     [] flush_work+0xff/0x1d0
2016-01-12 03:18:31     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:31     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:31     [] ? 
xfs_submit_ioend_bio.isra.14+0x40/0x40
2016-01-12 03:18:31     [] ? 
xfs_file_dio_aio_write+0x2a4/0x2f8
2016-01-12 03:18:30     [] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:30     [] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:30     [] ? __switch_to+0x1eb/0x590
2016-01-12 03:18:30     [] try_to_free_pages+0xda/0x160
2016-01-12 03:18:30     [] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:30     [] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:30     [] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:30     [] new_slab+0x32b/0x450
2016-01-12 03:18:30     [] __slab_alloc+0x2de/0x433
2016-01-12 03:18:30     [] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:30     [] 
kmem_cache_alloc_node+0x8a/0x240
2016-01-12 03:18:30     [] 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:30     [] ? make_kgid+0x12/0x20
2016-01-12 03:18:30     [] ? inode_init_always+0xe3/0x180
2016-01-12 03:18:30     [] do_fork+0xbc/0x350
2016-01-12 03:18:30     [] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:30     [] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:30     [] SyS_clone+0x16/0x20
2016-01-12 03:18:29     [] flush_work+0xff/0x1d0
2016-01-12 03:18:29     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:29     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:29     [] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:29     [] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:29     [] ? 
blk_attempt_plug_merge+0xd7/0xf0
2016-01-12 03:18:29     [] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:29     [] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:29     [] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:29     [] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:29     [] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:29     [] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:29     [] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:29     [] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:29     [] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:29     [] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:29     [] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:29     [] shrink_slab+0x216/0x3d0
2016-01-12 03:18:28     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-12 03:18:28     INFO: task proftpd:5592 blocked for more than 
120 seconds.
2016-01-12 03:18:28     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:28     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:28     proftpd D ffff88105487b198 0 5592 1 0x00000006
2016-01-12 03:18:28     ffff88105487b198 ffffffffb1c134a0 
ffff8808587b9d50 ffff88105487b1d8
2016-01-12 03:18:28     ffff88105487c000 ffff88105487b300 
7fffffffffffffff ffff8808587b9d50
2016-01-12 03:18:28     ffffffffffffffff ffff88105487b1b8 
ffffffffb163c967 ffff88105b9881d8
2016-01-12 03:18:28     Call Trace:
2016-01-12 03:18:28     [] schedule+0x37/0x90
2016-01-12 03:18:28     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:28     [] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:28     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:28     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:27     [] ? xfs_imap_to_bp+0xc9/0xe0
2016-01-12 03:18:27     [] xfs_log_force+0x2a/0x90
2016-01-12 03:18:27     [] xfs_iflush+0x202/0x210
2016-01-12 03:18:27     [] ? up+0x32/0x50
2016-01-12 03:18:27     [] xfs_inode_item_push+0xda/0x140
2016-01-12 03:18:27     [] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-12 03:18:27     [] xfsaild+0x2ce/0x5f0
2016-01-12 03:18:27     [] ? __schedule+0x4f1/0x990
2016-01-12 03:18:27     [] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-12 03:18:27     [] kthread+0xc9/0xe0
2016-01-12 03:18:27     [] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-12 03:18:27     [] ret_from_fork+0x42/0x70
2016-01-12 03:18:26     [] ? try_to_wake_up+0x197/0x340
2016-01-12 03:18:26     [] wait_for_completion+0x9f/0x170
2016-01-12 03:18:26     [] ? wake_up_process+0x15/0x20
2016-01-12 03:18:26     [] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:26     [] flush_work+0xff/0x1d0
2016-01-12 03:18:26     [] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:26     [] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:26     [] ? xfs_buf_read_map+0x37/0x110
2016-01-12 03:18:26     [] _xfs_log_force+0x6b/0x280
2016-01-12 03:18:25     ffff8810566e7a28 ffff88085c31d7f0 
ffff880859430000 000000000000137e
2016-01-12 03:18:25     ffff8810566e8000 ffff8810566e7b90 
7fffffffffffffff ffff880859430000
2016-01-12 03:18:25     ffffffffffffffff ffff8810566e7a48 
ffffffffb163c967 ffff88105b988060
2016-01-12 03:18:25     Call Trace:
2016-01-12 03:18:25     [] schedule+0x37/0x90
2016-01-12 03:18:25     [] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:25     [] ? sched_clock+0x9/0x10
2016-01-12 03:18:25     [] ? sched_clock_cpu+0x85/0xc0
2016-01-12 03:18:24     INFO: task xfsaild/sda3:730 blocked for more 
than 120 seconds.
2016-01-12 03:18:24     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:24     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:24     xfsaild/sda3 D ffff8810566e7a28 0 730 2 0x00000000



I can give you many more if they may help.


Greets,
Stefan

From jack@suse.cz  Fri Jan 15 03:35:01 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 458E87F47
	for ; Fri, 15 Jan 2016 03:35:01 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 17AB88F8040
	for ; Fri, 15 Jan 2016 01:34:57 -0800 (PST)
X-ASG-Debug-ID: 1452850494-04bdf06f6c1195b0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id FQ8ZEyOtTLaMLGpN (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 15 Jan 2016 01:34:55 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 13DD5AAB4;
	Fri, 15 Jan 2016 09:34:53 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 27CAD823D8; Fri, 15 Jan 2016 10:35:07 +0100 (CET)
Date: Fri, 15 Jan 2016 10:35:07 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: Jan Kara , Christoph Hellwig ,
	fsdevel , xfs@oss.sgi.com,
	Eric Sandeen 
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
Message-ID: <20160115093507.GA15950@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org>
 <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net>
 <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5696D27A.9070700@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1452850494
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26150
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed 13-01-16 16:40:58, Eric Sandeen wrote:
> On 1/11/16 10:28 AM, Jan Kara wrote:
> > On Mon 11-01-16 10:07:22, Eric Sandeen wrote:
> >> On 1/11/16 7:26 AM, Jan Kara wrote:
> >>> On Fri 08-01-16 23:26:00, Christoph Hellwig wrote:
> >>>> On Fri, Jan 08, 2016 at 10:56:12AM -0600, Eric Sandeen wrote:
> >>>>> This adds a new quotactl, Q_XGETQUOTA2.
> >>>>>
> >>>>> Q_XGETQUOTA2 is exactly like Q_XGETQUOTA, except that it will
> >>>>> return quota information for the id equal to or greater than
> >>>>> the id requested.  In other words, if the specified id has
> >>>>> no quota, the command will return quota information for the
> >>>>> next higher id which does have a quota set.  If no higher id
> >>>>> has an active quota, -ESRCH is returned.
> >>>>
> >>>> Please add a flags argument to Q_XGETQUOTA2, and then make the
> >>>> new behavior the first flag.  Keep Q_XGETQUOTA behavior for the
> >>>> flag-less case.  That way we get future etensibility for free.
> >>>
> >>> So this is what I wanted to suggest at first as well. What I somewhat
> >>> dislike is that 'addr' must now point to something like:
> >>>
> >>> struct getquota_args {
> >>> 	__u64 flags;
> >>> 	struct fs_disk_quota ret;
> >>> };
> >>
> >> Ok...
> >>
> >>> which is not as nice as passing pointer to fs_disk_quota directly. But
> >>> probably still OK. So I agree with the flags idea.
> >>
> >> I understand that flags are for future-proofing, for possibly-not-yet-imagined
> >> scenarios, but on the other hand, I wonder how many different flavors of
> >> "get me a disk quota" we really expect to have?
> >>
> >>> Another issue is that OCFS2 and ext4 with quota in hidden inodes would need
> >>> call with these capabilities as well (they have the same problem as xfs).
> >>> For reporting VFS quota we use bytes for space (since some filesystems need
> >>> it) and we don't need RT fields (easy so zero-fill) and warning fields
> >>> (there zeros may be confusing). So we would need something like struct
> >>> qc_dqblk (currently only internal) extended with ID and maybe flags field
> >>> XFS is using to return quota type which can contain all the information and
> >>> tell which info is actually valid as return structure.
> >>
> >> Hohum, yeah - I forgot that ext4 has (can have?) a hidden quota inode.
> >> And I didn't know about OCFS2.
> >>
> >> And TBH I did the xfs-specific one because it was trivial to extend, with the
> >> id already in the returned quota struct.
> >>
> >> So ok, it sounds like this needs a more significant overhaul at the vfs
> >> quota level...
> > 
> > Actually, what I want from you is just an interface which is usable for VFS
> > quotas as well since I'd like to avoid adding GETQUOTA2 quotactl shortly
> > after XGETQUOTA2 :).
> 
> Actually, that's exactly what I thought would *need* to happen ... we already
> have this weird 15-year-old split-brain quota interface, so if xfs and ext4
> both need the same functionality, then we'd probably add both GETQUOTA2 and
> XGETQUOTA2.  If we were doing this all from scratch, sure, but adding a new
> handles-both-quota-types interface when every other operation is already split
> between the two almost seems to make matters worse.

Well, currently GETQUOTA and XGETQUOTA (and all the other quotactls) are
actually translated so they work regardless of the underlying filesystem.
So the only difference between XFS and VFS quotactls is in the formatting
of input/output structures. So from kernel POV it seems somewhat pointless
to add two calls doing the same thing and differing just in the formatting
of output - especially when we want the call to be extensible.

I agree that having a unified call means having a new structure for passing
dquot info between kernel and userspace. So just for adding that one small
feature you want it seems like an overkill. But when thinking about new
extensible getquota quotactl it IMHO makes sense to unify the VFS/XFS split
brain. Thoughts?

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Fri Jan 15 07:10:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 13DB67F47
	for ; Fri, 15 Jan 2016 07:10:31 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 91E60AC003
	for ; Fri, 15 Jan 2016 05:10:27 -0800 (PST)
X-ASG-Debug-ID: 1452863422-04bdf06f6b11fef0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 3pjfRuNZ73Rwp7Jw (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 15 Jan 2016 05:10:23 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id AFA0CAD98;
	Fri, 15 Jan 2016 13:10:19 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id E1A77823D8; Fri, 15 Jan 2016 14:10:32 +0100 (CET)
Date: Fri, 15 Jan 2016 14:10:32 +0100
From: Jan Kara 
To: Ross Zwisler 
Cc: Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 6/9] dax: add support for fsync/msync
Message-ID: <20160115131032.GK15950@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH v8 6/9] dax: add support for fsync/msync
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-7-git-send-email-ross.zwisler@linux.intel.com>
 <20160112105716.GT6262@quack.suse.cz>
 <20160113073019.GB30496@linux.intel.com>
 <20160113093525.GD14630@quack.suse.cz>
 <20160113185802.GB5904@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113185802.GB5904@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1452863422
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Barracuda-BRTS-Status: 1
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26153
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed 13-01-16 11:58:02, Ross Zwisler wrote:
> On Wed, Jan 13, 2016 at 10:35:25AM +0100, Jan Kara wrote:
> > On Wed 13-01-16 00:30:19, Ross Zwisler wrote:
> > > > And secondly: You must write-protect all mappings of the flushed range so
> > > > that you get fault when the sector gets written-to again. We spoke about
> > > > this in the past already but somehow it got lost and I forgot about it as
> > > > well. You need something like rmap_walk_file()...
> > > 
> > > The code that write protected mappings and then cleaned the radix tree entries
> > > did get written, and was part of v2:
> > > 
> > > https://lkml.org/lkml/2015/11/13/759
> > > 
> > > I removed all the code that cleaned PTE entries and radix tree entries for v3.
> > > The reason behind this was that there was a race that I couldn't figure out
> > > how to solve between the cleaning of the PTEs and the cleaning of the radix
> > > tree entries.
> > > 
> > > The race goes like this:
> > > 
> > > Thread 1 (write)			Thread 2 (fsync)
> > > ================			================
> > > wp_pfn_shared()
> > > pfn_mkwrite()
> > > dax_radix_entry()
> > > radix_tree_tag_set(DIRTY)
> > > 					dax_writeback_mapping_range()
> > > 					dax_writeback_one()
> > > 					radix_tag_clear(DIRTY)
> > > 					pgoff_mkclean()
> > > ... return up to wp_pfn_shared()
> > > wp_page_reuse()
> > > pte_mkdirty()
> > > 
> > > After this sequence we end up with a dirty PTE that is writeable, but with a
> > > clean radix tree entry.  This means that users can write to the page, but that
> > > a follow-up fsync or msync won't flush this dirty data to media.
> > > 
> > > The overall issue is that in the write path that goes through wp_pfn_shared(),
> > > the DAX code has control over when the radix tree entry is dirtied but not
> > > when the PTE is made dirty and writeable.  This happens up in wp_page_reuse().
> > > This means that we can't easily add locking, etc. to protect ourselves.
> > > 
> > > I spoke a bit about this with Dave Chinner and with Dave Hansen, but no really
> > > easy solutions presented themselves in the absence of a page lock.  I do have
> > > one idea, but I think it's pretty invasive and will need to wait for another
> > > kernel cycle.
> > > 
> > > The current code that leaves the radix tree entry will give us correct
> > > behavior - it'll just be less efficient because we will have an ever-growing
> > > dirty set to flush.
> > 
> > Ahaa! Somehow I imagined tag_pages_for_writeback() clears DIRTY radix tree
> > tags but it does not (I should have known, I have written that functions
> > few years ago ;). Makes sense. Thanks for clarification.
> > 
> > > > > @@ -791,15 +976,12 @@ EXPORT_SYMBOL_GPL(dax_pmd_fault);
> > > > >   * dax_pfn_mkwrite - handle first write to DAX page
> > > > >   * @vma: The virtual memory area where the fault occurred
> > > > >   * @vmf: The description of the fault
> > > > > - *
> > > > >   */
> > > > >  int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
> > > > >  {
> > > > > -	struct super_block *sb = file_inode(vma->vm_file)->i_sb;
> > > > > +	struct file *file = vma->vm_file;
> > > > >  
> > > > > -	sb_start_pagefault(sb);
> > > > > -	file_update_time(vma->vm_file);
> > > > > -	sb_end_pagefault(sb);
> > > > > +	dax_radix_entry(file->f_mapping, vmf->pgoff, NO_SECTOR, false, true);
> > > > 
> > > > Why is NO_SECTOR argument correct here?
> > > 
> > > Right - so NO_SECTOR means "I expect there to already be an entry in the radix
> > > tree - just make that entry dirty".  This works because pfn_mkwrite() always
> > > follows a normal __dax_fault() or __dax_pmd_fault() call.  These fault calls
> > > will insert the radix tree entry, regardless of whether the fault was for a
> > > read or a write.  If the fault was for a write, the radix tree entry will also
> > > be made dirty.
> > >
> > > For reads the radix tree entry will be inserted but left clean.  When the
> > > first write happens we will get a pfn_mkwrite() call, which will call
> > > dax_radix_entry() with the NO_SECTOR argument.  This will look up the radix
> > > tree entry & set the dirty tag.
> > 
> > So the explanation of this should be somewhere so that everyone knows that
> > we must have radix tree entries even for clean mapped blocks. Because upto
> > know that was not clear to me.  Also __dax_pmd_fault() seems to insert
> > entries only for write fault so the assumption doesn't seem to hold there?
> 
> Ah, right, sorry, the read fault() -> pfn_mkwrite() sequence only happens for
> 4k pages.  You are right about our handling of 2MiB pages - for a read
> followed by a write we will just call into the normal __dax_pmd_fault() code
> again, which will do the get_block() call and insert a dirty radix tree entry.
> Because we have to go all the way through the fault handler again at write
> time there isn't a benefit to inserting a clean radix tree entry on read, so
> we just skip it.

Ouch, I wasn't aware of this asymetry between PMD and PTE faults. OK, so
please just document this all somewhere because I'm pretty sure casual
reader won't be able to figure this all out just from the code.

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Fri Jan 15 07:22:43 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 0D4FE7F47
	for ; Fri, 15 Jan 2016 07:22:43 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id DAC2A8F8040
	for ; Fri, 15 Jan 2016 05:22:39 -0800 (PST)
X-ASG-Debug-ID: 1452864157-04cb6c75db113330001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id QTLPbQCvss2guBwV (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 15 Jan 2016 05:22:38 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id D7836AD98;
	Fri, 15 Jan 2016 13:22:35 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id CAEFC823D8; Fri, 15 Jan 2016 14:22:49 +0100 (CET)
Date: Fri, 15 Jan 2016 14:22:49 +0100
From: Jan Kara 
To: Ross Zwisler 
Cc: Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree
Message-ID: <20160115132249.GL15950@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-5-git-send-email-ross.zwisler@linux.intel.com>
 <20160113094411.GA17057@quack.suse.cz>
 <20160113184832.GA5904@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113184832.GA5904@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1452864157
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26154
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed 13-01-16 11:48:32, Ross Zwisler wrote:
> On Wed, Jan 13, 2016 at 10:44:11AM +0100, Jan Kara wrote:
> > On Thu 07-01-16 22:27:54, Ross Zwisler wrote:
> > > Add support for tracking dirty DAX entries in the struct address_space
> > > radix tree.  This tree is already used for dirty page writeback, and it
> > > already supports the use of exceptional (non struct page*) entries.
> > > 
> > > In order to properly track dirty DAX pages we will insert new exceptional
> > > entries into the radix tree that represent dirty DAX PTE or PMD pages.
> > > These exceptional entries will also contain the writeback sectors for the
> > > PTE or PMD faults that we can use at fsync/msync time.
> > > 
> > > There are currently two types of exceptional entries (shmem and shadow)
> > > that can be placed into the radix tree, and this adds a third.  We rely on
> > > the fact that only one type of exceptional entry can be found in a given
> > > radix tree based on its usage.  This happens for free with DAX vs shmem but
> > > we explicitly prevent shadow entries from being added to radix trees for
> > > DAX mappings.
> > > 
> > > The only shadow entries that would be generated for DAX radix trees would
> > > be to track zero page mappings that were created for holes.  These pages
> > > would receive minimal benefit from having shadow entries, and the choice
> > > to have only one type of exceptional entry in a given radix tree makes the
> > > logic simpler both in clear_exceptional_entry() and in the rest of DAX.
> > > 
> > > Signed-off-by: Ross Zwisler 
> > > Reviewed-by: Jan Kara 
> > 
> > I have realized there's one issue with this code. See below:
> > 
> > > @@ -34,31 +35,39 @@ static void clear_exceptional_entry(struct address_space *mapping,
> > >  		return;
> > >  
> > >  	spin_lock_irq(&mapping->tree_lock);
> > > -	/*
> > > -	 * Regular page slots are stabilized by the page lock even
> > > -	 * without the tree itself locked.  These unlocked entries
> > > -	 * need verification under the tree lock.
> > > -	 */
> > > -	if (!__radix_tree_lookup(&mapping->page_tree, index, &node, &slot))
> > > -		goto unlock;
> > > -	if (*slot != entry)
> > > -		goto unlock;
> > > -	radix_tree_replace_slot(slot, NULL);
> > > -	mapping->nrshadows--;
> > > -	if (!node)
> > > -		goto unlock;
> > > -	workingset_node_shadows_dec(node);
> > > -	/*
> > > -	 * Don't track node without shadow entries.
> > > -	 *
> > > -	 * Avoid acquiring the list_lru lock if already untracked.
> > > -	 * The list_empty() test is safe as node->private_list is
> > > -	 * protected by mapping->tree_lock.
> > > -	 */
> > > -	if (!workingset_node_shadows(node) &&
> > > -	    !list_empty(&node->private_list))
> > > -		list_lru_del(&workingset_shadow_nodes, &node->private_list);
> > > -	__radix_tree_delete_node(&mapping->page_tree, node);
> > > +
> > > +	if (dax_mapping(mapping)) {
> > > +		if (radix_tree_delete_item(&mapping->page_tree, index, entry))
> > > +			mapping->nrexceptional--;
> > 
> > So when you punch hole in a file, you can delete a PMD entry from a radix
> > tree which covers part of the file which still stays. So in this case you
> > have to split the PMD entry into PTE entries (probably that needs to happen
> > up in truncate_inode_pages_range()) or something similar...
> 
> I think (and will verify) that the DAX code just unmaps the entire PMD range
> when we receive a hole punch request inside of the PMD.  If this is true then
> I think the radix tree code should behave the same way and just remove the PMD
> entry in the radix tree.

But you cannot just remove it if it is dirty... You have to keep somewhere
information that part of the PMD range is still dirty (or write that range
out before removing the radix tree entry).

> This will cause new accesses that used to land in the PMD range to get new
> page faults.  These faults will call get_blocks(), where presumably the
> filesystem will tell us that we don't have a contiguous 2MiB range anymore, so
> we will fall back to PTE faults.  These PTEs will fill in both the radix tree
> and the page tables.
> 
> So, I think the work here is to verify the behavior of DAX wrt hole punches
> for PMD ranges, and make the radix tree code match that behavior.  Sound good?

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From sandeen@sandeen.net  Fri Jan 15 11:31:04 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 1BBBB7F47
	for ; Fri, 15 Jan 2016 11:31:04 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id EACAB8F8035
	for ; Fri, 15 Jan 2016 09:31:03 -0800 (PST)
X-ASG-Debug-ID: 1452879061-04cb6c75db119840001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id j595OGlTQ9NTmFob for ; Fri, 15 Jan 2016 09:31:01 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id DC729608F751;
	Fri, 15 Jan 2016 11:31:00 -0600 (CST)
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
To: Jan Kara 
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org> <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net> <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net> <20160115093507.GA15950@quack.suse.cz>
Cc: Christoph Hellwig ,
 fsdevel , Eric Sandeen ,
 xfs@oss.sgi.com
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56992CD4.6030408@sandeen.net>
Date: Fri, 15 Jan 2016 11:31:00 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160115093507.GA15950@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1452879061
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26159
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Hi Jan -

On 1/15/16 3:35 AM, Jan Kara wrote:
> On Wed 13-01-16 16:40:58, Eric Sandeen wrote:
>> On 1/11/16 10:28 AM, Jan Kara wrote:

...

>>> Actually, what I want from you is just an interface which is usable for VFS
>>> quotas as well since I'd like to avoid adding GETQUOTA2 quotactl shortly
>>> after XGETQUOTA2 :).
>>
>> Actually, that's exactly what I thought would *need* to happen ... we already
>> have this weird 15-year-old split-brain quota interface, so if xfs and ext4
>> both need the same functionality, then we'd probably add both GETQUOTA2 and
>> XGETQUOTA2.  If we were doing this all from scratch, sure, but adding a new
>> handles-both-quota-types interface when every other operation is already split
>> between the two almost seems to make matters worse.
> 
> Well, currently GETQUOTA and XGETQUOTA (and all the other quotactls) are
> actually translated so they work regardless of the underlying filesystem.
> So the only difference between XFS and VFS quotactls is in the formatting
> of input/output structures. So from kernel POV it seems somewhat pointless
> to add two calls doing the same thing and differing just in the formatting
> of output - especially when we want the call to be extensible.
> 
> I agree that having a unified call means having a new structure for passing
> dquot info between kernel and userspace. So just for adding that one small
> feature you want it seems like an overkill. But when thinking about new
> extensible getquota quotactl it IMHO makes sense to unify the VFS/XFS split
> brain. Thoughts?

My first lazy/hacky thought is "how terrible would it be to overload the
quotactl syscall return value with quota ID for Q_GETQUOTA2 calls?"

For a purpose-built interface of "find the next ID" that wouldn't require any
structure or interface changes...

We could name it Q_GETNEXTQUOTA / Q_XGETNEXTQUOTA to make it explicit about
the purpose, and document that return behavior.  Done & done.  ;)

A new grand unified extensible quota call sounds like a great idea, I just
hate to gate this work on designing a brand-new interface.

-Eric

From ross.zwisler@linux.intel.com  Fri Jan 15 13:03:49 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 820967F47
	for ; Fri, 15 Jan 2016 13:03:49 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 630AE8F8040
	for ; Fri, 15 Jan 2016 11:03:46 -0800 (PST)
X-ASG-Debug-ID: 1452884625-04cbb05f7d1c04c0001-NocioJ
Received: from mga01.intel.com ([192.55.52.88]) by cuda.sgi.com with ESMTP id 1gGAY7wDhNhNjqyA for ; Fri, 15 Jan 2016 11:03:45 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.88
X-ASG-Whitelist: EmailCat (corporate)
Received: from orsmga001.jf.intel.com ([10.7.209.18])
  by fmsmga101.fm.intel.com with ESMTP; 15 Jan 2016 11:03:44 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,301,1449561600"; 
   d="scan'208";a="861349945"
Received: from rzwisler-desk.amr.corp.intel.com (HELO linux.intel.com) ([10.252.206.118])
  by orsmga001.jf.intel.com with ESMTP; 15 Jan 2016 11:03:43 -0800
Date: Fri, 15 Jan 2016 12:03:42 -0700
From: Ross Zwisler 
To: Jan Kara 
Cc: Ross Zwisler ,
	linux-kernel@vger.kernel.org, "H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree
Message-ID: <20160115190342.GA20168@linux.intel.com>
X-ASG-Orig-Subj: Re: [PATCH v8 4/9] dax: support dirty DAX entries in radix tree
Mail-Followup-To: Ross Zwisler ,
	Jan Kara , linux-kernel@vger.kernel.org,
	"H. Peter Anvin" ,
	"J. Bruce Fields" ,
	Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Dave Hansen ,
	Ingo Molnar , Jan Kara ,
	Jeff Layton ,
	Matthew Wilcox ,
	Matthew Wilcox ,
	Thomas Gleixner , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
	linux-nvdimm@lists.01.org, x86@kernel.org, xfs@oss.sgi.com
References: <1452230879-18117-1-git-send-email-ross.zwisler@linux.intel.com>
 <1452230879-18117-5-git-send-email-ross.zwisler@linux.intel.com>
 <20160113094411.GA17057@quack.suse.cz>
 <20160113184832.GA5904@linux.intel.com>
 <20160115132249.GL15950@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160115132249.GL15950@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.88]
X-Barracuda-Start-Time: 1452884625
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Fri, Jan 15, 2016 at 02:22:49PM +0100, Jan Kara wrote:
> On Wed 13-01-16 11:48:32, Ross Zwisler wrote:
> > On Wed, Jan 13, 2016 at 10:44:11AM +0100, Jan Kara wrote:
> > > On Thu 07-01-16 22:27:54, Ross Zwisler wrote:
> > > > Add support for tracking dirty DAX entries in the struct address_space
> > > > radix tree.  This tree is already used for dirty page writeback, and it
> > > > already supports the use of exceptional (non struct page*) entries.
> > > > 
> > > > In order to properly track dirty DAX pages we will insert new exceptional
> > > > entries into the radix tree that represent dirty DAX PTE or PMD pages.
> > > > These exceptional entries will also contain the writeback sectors for the
> > > > PTE or PMD faults that we can use at fsync/msync time.
> > > > 
> > > > There are currently two types of exceptional entries (shmem and shadow)
> > > > that can be placed into the radix tree, and this adds a third.  We rely on
> > > > the fact that only one type of exceptional entry can be found in a given
> > > > radix tree based on its usage.  This happens for free with DAX vs shmem but
> > > > we explicitly prevent shadow entries from being added to radix trees for
> > > > DAX mappings.
> > > > 
> > > > The only shadow entries that would be generated for DAX radix trees would
> > > > be to track zero page mappings that were created for holes.  These pages
> > > > would receive minimal benefit from having shadow entries, and the choice
> > > > to have only one type of exceptional entry in a given radix tree makes the
> > > > logic simpler both in clear_exceptional_entry() and in the rest of DAX.
> > > > 
> > > > Signed-off-by: Ross Zwisler 
> > > > Reviewed-by: Jan Kara 
> > > 
> > > I have realized there's one issue with this code. See below:
> > > 
> > > > @@ -34,31 +35,39 @@ static void clear_exceptional_entry(struct address_space *mapping,
> > > >  		return;
> > > >  
> > > >  	spin_lock_irq(&mapping->tree_lock);
> > > > -	/*
> > > > -	 * Regular page slots are stabilized by the page lock even
> > > > -	 * without the tree itself locked.  These unlocked entries
> > > > -	 * need verification under the tree lock.
> > > > -	 */
> > > > -	if (!__radix_tree_lookup(&mapping->page_tree, index, &node, &slot))
> > > > -		goto unlock;
> > > > -	if (*slot != entry)
> > > > -		goto unlock;
> > > > -	radix_tree_replace_slot(slot, NULL);
> > > > -	mapping->nrshadows--;
> > > > -	if (!node)
> > > > -		goto unlock;
> > > > -	workingset_node_shadows_dec(node);
> > > > -	/*
> > > > -	 * Don't track node without shadow entries.
> > > > -	 *
> > > > -	 * Avoid acquiring the list_lru lock if already untracked.
> > > > -	 * The list_empty() test is safe as node->private_list is
> > > > -	 * protected by mapping->tree_lock.
> > > > -	 */
> > > > -	if (!workingset_node_shadows(node) &&
> > > > -	    !list_empty(&node->private_list))
> > > > -		list_lru_del(&workingset_shadow_nodes, &node->private_list);
> > > > -	__radix_tree_delete_node(&mapping->page_tree, node);
> > > > +
> > > > +	if (dax_mapping(mapping)) {
> > > > +		if (radix_tree_delete_item(&mapping->page_tree, index, entry))
> > > > +			mapping->nrexceptional--;
> > > 
> > > So when you punch hole in a file, you can delete a PMD entry from a radix
> > > tree which covers part of the file which still stays. So in this case you
> > > have to split the PMD entry into PTE entries (probably that needs to happen
> > > up in truncate_inode_pages_range()) or something similar...
> > 
> > I think (and will verify) that the DAX code just unmaps the entire PMD range
> > when we receive a hole punch request inside of the PMD.  If this is true then
> > I think the radix tree code should behave the same way and just remove the PMD
> > entry in the radix tree.
> 
> But you cannot just remove it if it is dirty... You have to keep somewhere
> information that part of the PMD range is still dirty (or write that range
> out before removing the radix tree entry).

Yep, agreed.

From sandeen@redhat.com  Fri Jan 15 13:38:45 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id ED28A7F47
	for ; Fri, 15 Jan 2016 13:38:44 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 88CE3AC003
	for ; Fri, 15 Jan 2016 11:38:44 -0800 (PST)
X-ASG-Debug-ID: 1452886720-04bdf06f6b129f90001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id kmkrJwbM2925vqar (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 15 Jan 2016 11:38:40 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	by mx1.redhat.com (Postfix) with ESMTPS id F0C5618B319;
	Fri, 15 Jan 2016 19:38:39 +0000 (UTC)
Received: from Liberator.local (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0FJccSs005598
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Fri, 15 Jan 2016 14:38:39 -0500
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
To: Eric Sandeen , Jan Kara 
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org> <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net> <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net> <20160115093507.GA15950@quack.suse.cz>
 <56992CD4.6030408@sandeen.net>
Cc: Christoph Hellwig ,
        fsdevel , xfs@oss.sgi.com
From: Eric Sandeen 
Message-ID: <56994ABE.8000607@redhat.com>
Date: Fri, 15 Jan 2016 13:38:38 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <56992CD4.6030408@sandeen.net>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1452886720
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On 1/15/16 11:31 AM, Eric Sandeen wrote:

> My first lazy/hacky thought is "how terrible would it be to overload the
> quotactl syscall return value with quota ID for Q_GETQUOTA2 calls?"

Never mind, we can use all 32 bits of the unsigned ID, I think, so that
doesn't work.  (thanks Zach) :)

-Eric

From darrick.wong@oracle.com  Fri Jan 15 14:57:08 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 004B47F47
	for ; Fri, 15 Jan 2016 14:57:08 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id BE5EA304059
	for ; Fri, 15 Jan 2016 12:57:04 -0800 (PST)
X-ASG-Debug-ID: 1452891417-04bdf06f6d12b7f0001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id k7fENTpRcrQkGFWj (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 15 Jan 2016 12:56:57 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0FKuVQD019225
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 15 Jan 2016 20:56:31 GMT
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75])
	by userv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0FKuVa7028730
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 15 Jan 2016 20:56:31 GMT
Received: from abhmp0007.oracle.com (abhmp0007.oracle.com [141.146.116.13])
	by userv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0FKuVFf009043;
	Fri, 15 Jan 2016 20:56:31 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 15 Jan 2016 12:56:30 -0800
Date: Fri, 15 Jan 2016 12:56:29 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: Eric Sandeen , xfs@oss.sgi.com
Subject: Re: [PATCH] xfs_db: check on-disk structure sizes
Message-ID: <20160115205629.GA5757@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH] xfs_db: check on-disk structure sizes
References: <20160111234644.GB7831@birch.djwong.org>
 <20160112140122.GC12156@bfoster.bfoster>
 <20160113012945.GC2455@birch.djwong.org>
 <5695C01F.7030807@sandeen.net>
 <20160113054730.GQ10456@dastard>
 <10021348-43A2-465E-BC00-E0A0882FFD22@sandeen.net>
 <20160113074640.GA21939@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113074640.GA21939@infradead.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1452891417
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26166
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Tue, Jan 12, 2016 at 11:46:40PM -0800, Christoph Hellwig wrote:
> On Wed, Jan 13, 2016 at 12:02:36AM -0600, Eric Sandeen wrote:
> > > As I've pointed out previously to Darrick: xfstests:/tests/xfs/122
> > > 
> > > Make that build again, update it.
> > > 
> > Oh, I went looking for that and missed it somehow, thought it had been removed.   Ok then!
> 
> The real issue is that build environment change alignments.  32 vs 64
> bit builds are obvious, but on some architectures different ABIs have
> different alignments (we had some fun with ARM in that regard),
> nevermind the equivalents to IRIX n32 popping up everywhere these days
> that make things complicated.
> 
> I'd really love to have Darrick's check in xfs_format.h as an opt-in
> if a build time assert is provided - that way every user can check it
> doesn't screw up the structures.

I've fixed xfs/122, so I suppose we no longer need this to end up in
xfsprogs.  The kernel-side patch can stick around in fs/xfs/ without
touching libxfs.

(New patches out soonish.)

--D

> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From darrick.wong@oracle.com  Fri Jan 15 14:58:07 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 2B3117F47
	for ; Fri, 15 Jan 2016 14:58:07 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 11103304059
	for ; Fri, 15 Jan 2016 12:58:06 -0800 (PST)
X-ASG-Debug-ID: 1452891484-04cbb05f7c1c28c0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id 5Ebf2sMMeVMCFqLd (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 15 Jan 2016 12:58:05 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0FKw3L3015660
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 15 Jan 2016 20:58:04 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0FKw3Gx024399
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 15 Jan 2016 20:58:03 GMT
Received: from abhmp0010.oracle.com (abhmp0010.oracle.com [141.146.116.16])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0FKw28W000419;
	Fri, 15 Jan 2016 20:58:03 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 15 Jan 2016 12:58:02 -0800
Date: Fri, 15 Jan 2016 12:58:01 -0800
From: "Darrick J. Wong" 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
Message-ID: <20160115205801.GB5757@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
References: <20160111232657.GA7831@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160111232657.GA7831@birch.djwong.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1452891485
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26166
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Mon, Jan 11, 2016 at 03:26:57PM -0800, Darrick J. Wong wrote:
> Check the sizes of XFS on-disk structures when compiling the kernel.
> Use this to catch inadvertent changes in structure size due to padding
> and alignment issues, etc.
> 
> Signed-off-by: Darrick J. Wong 
> ---
>  fs/xfs/libxfs/xfs_format.h |    4 ++++
>  fs/xfs/xfs_super.c         |   45 ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 49 insertions(+)
> 
> diff --git a/fs/xfs/libxfs/xfs_format.h b/fs/xfs/libxfs/xfs_format.h
> index e2536bb..a589b47 100644
> --- a/fs/xfs/libxfs/xfs_format.h
> +++ b/fs/xfs/libxfs/xfs_format.h
> @@ -1509,4 +1509,8 @@ struct xfs_acl {
>  #define SGI_ACL_FILE_SIZE	(sizeof(SGI_ACL_FILE)-1)
>  #define SGI_ACL_DEFAULT_SIZE	(sizeof(SGI_ACL_DEFAULT)-1)
>  
> +#define XFS_CHECK_STRUCT_SIZE(structname, size) \
> +	BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(struct " \
> +		#structname ") is wrong, expected " #size)
> +

Since xfs/122 will handle this on the userspace side, this can move to
xfs_super.c.

>  #endif /* __XFS_FORMAT_H__ */
> diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c
> index 36bd882..31b69d1 100644
> --- a/fs/xfs/xfs_super.c
> +++ b/fs/xfs/xfs_super.c
> @@ -1812,11 +1812,56 @@ xfs_destroy_workqueues(void)
>  	destroy_workqueue(xfs_alloc_wq);
>  }
>  
> +static void __init
> +xfs_check_ondisk_structs(void)
> +{
> +	/* on-disk structures */
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dsb,			264);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agf,			224);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agi,			336);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agfl,			36);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp,		8);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dinode,		176);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_disk_dquot,		104);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dqblk,			136);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dsymlink_hdr,		56);
> +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_rec_t,			8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_key_t,			8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_ptr_t,			4);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_rec,		16);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_key,		4);
> +	XFS_CHECK_STRUCT_SIZE(xfs_inobt_ptr_t,			4);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmdr_block,		4);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_rec,		16);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_key,		8);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_btree_block,		72);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_acl_entry,		12);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_acl,			4);

Forgot the dir/attr structures here.

> +	/* log structures */
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_trans_header,		16);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_32,	52);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_64,	56);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_ictimestamp,		8);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_icdinode,		176);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_extent_32,		12);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_extent_64,		16);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_32,	28);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
> +}

Perhaps this huge function ought to hide away in its own file?

--D

> +
>  STATIC int __init
>  init_xfs_fs(void)
>  {
>  	int			error;
>  
> +	xfs_check_ondisk_structs();
> +
>  	printk(KERN_INFO XFS_VERSION_STRING " with "
>  			 XFS_BUILD_OPTIONS " enabled\n");
>  
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From darrick.wong@oracle.com  Fri Jan 15 15:06:12 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id C00E17F47
	for ; Fri, 15 Jan 2016 15:06:12 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id B6154304059
	for ; Fri, 15 Jan 2016 13:06:09 -0800 (PST)
X-ASG-Debug-ID: 1452891967-04cb6c75dc11dab0001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id HpJoISvJJViP6hOi (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 15 Jan 2016 13:06:07 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0FL66QJ030904
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 15 Jan 2016 21:06:06 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by userv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0FL65Kl025545
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 15 Jan 2016 21:06:06 GMT
Received: from abhmp0007.oracle.com (abhmp0007.oracle.com [141.146.116.13])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0FL657Z015421;
	Fri, 15 Jan 2016 21:06:05 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 15 Jan 2016 13:06:05 -0800
Date: Fri, 15 Jan 2016 13:05:56 -0800
From: "Darrick J. Wong" 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
Message-ID: <20160115210556.GC5757@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
References: <20160111232657.GA7831@birch.djwong.org>
 <56969696.2040206@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56969696.2040206@sandeen.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1452891967
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26166
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Wed, Jan 13, 2016 at 12:25:26PM -0600, Eric Sandeen wrote:
> On 1/11/16 5:26 PM, Darrick J. Wong wrote:
> 
> > diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c
> > index 36bd882..31b69d1 100644
> > --- a/fs/xfs/xfs_super.c
> > +++ b/fs/xfs/xfs_super.c
> > @@ -1812,11 +1812,56 @@ xfs_destroy_workqueues(void)
> >  	destroy_workqueue(xfs_alloc_wq);
> >  }
> >  
> > +static void __init
> > +xfs_check_ondisk_structs(void)
> > +{
> > +	/* on-disk structures */
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dsb,			264);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_agf,			224);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_agi,			336);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_agfl,			36);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp,		8);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dinode,		176);
> 
> I'm not sure why, but all this hard-coded stuff in the codebase bugs me.
> 
> What if we did it as part of the build, but outside the code, something
> like (pardon my bad Make-fu):
> 
> diff --git a/fs/xfs/Makefile b/fs/xfs/Makefile
> index f646391..da9ec6d 100644
> --- a/fs/xfs/Makefile
> +++ b/fs/xfs/Makefile
> @@ -122,3 +122,8 @@ xfs-$(CONFIG_XFS_POSIX_ACL) += xfs_acl.o
>  xfs-$(CONFIG_SYSCTL)           += xfs_sysctl.o
>  xfs-$(CONFIG_COMPAT)           += xfs_ioctl32.o
>  xfs-$(CONFIG_NFSD_PNFS)                += xfs_pnfs.o
> +
> +check_structures: $(obj)/xfs.o
> +       $(src)/scripts/check_structures.sh $(obj)/xfs.o
> +
> +$(obj)/built-in.o: check_structures
> 
> where the script executes pahole or whatever we want, to check the
> structure sizes and possibly even alignment explicitly?  Presumably
> we could make running the check a Kconfig option as well...

We certainly /could/ do that, but now the build depends on pahole or whatever
being installed, whereas this macroey doesn't add more tool dependencies.
I think between this and xfs/122 we're probably fine?

--D

> 
> similar tricks could surely be done for userspace as well.
> 
> -Eric
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From david@fromorbit.com  Fri Jan 15 16:50:50 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 9B9577F47
	for ; Fri, 15 Jan 2016 16:50:50 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 771FF304053
	for ; Fri, 15 Jan 2016 14:50:50 -0800 (PST)
X-ASG-Debug-ID: 1452898246-04bdf06f6d12f580001-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id 6v7LBkyl2vMbsEH4 for ; Fri, 15 Jan 2016 14:50:47 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2AkEQBvd5lWPGwtLHleKAECgw+BP4Jkg3qBeJ9uAQEBAQEBBotchUOEBIYJBAICgS9NAQEBAQEBBwEBAQFAAT9BDgGDZQEBBDocGgkQCAMOCgklDwUlAwcaE4gawR8BLBiFdYVHhDuFAgWXGY1VjwpEjhmFAio0hCZAgUoBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 16 Jan 2016 09:20:23 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aKDCI-0006yc-MT; Sat, 16 Jan 2016 09:50:22 +1100
Date: Sat, 16 Jan 2016 09:50:22 +1100
From: Dave Chinner 
To: Jan Kara 
Cc: Eric Sandeen ,
	Christoph Hellwig ,
	fsdevel ,
	Eric Sandeen , xfs@oss.sgi.com
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
Message-ID: <20160115225022.GO6033@dastard>
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org>
 <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net>
 <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net>
 <20160115093507.GA15950@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160115093507.GA15950@quack.suse.cz>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452898247
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26171
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri, Jan 15, 2016 at 10:35:07AM +0100, Jan Kara wrote:
> On Wed 13-01-16 16:40:58, Eric Sandeen wrote:
> > Actually, that's exactly what I thought would *need* to happen ... we already
> > have this weird 15-year-old split-brain quota interface, so if xfs and ext4
> > both need the same functionality, then we'd probably add both GETQUOTA2 and
> > XGETQUOTA2.  If we were doing this all from scratch, sure, but adding a new
> > handles-both-quota-types interface when every other operation is already split
> > between the two almost seems to make matters worse.
> 
> Well, currently GETQUOTA and XGETQUOTA (and all the other quotactls) are
> actually translated so they work regardless of the underlying filesystem.
> So the only difference between XFS and VFS quotactls is in the formatting
> of input/output structures. So from kernel POV it seems somewhat pointless
> to add two calls doing the same thing and differing just in the formatting
> of output - especially when we want the call to be extensible.
> 
> I agree that having a unified call means having a new structure for passing
> dquot info between kernel and userspace. So just for adding that one small
> feature you want it seems like an overkill. But when thinking about new
> extensible getquota quotactl it IMHO makes sense to unify the VFS/XFS split
> brain. Thoughts?

A new unified userspace quota API would be nice, but it's a
completely separate piece of work. We'd also need to do userspace
work to support it, so I think this is a valid medium term goal,
but not somethign that should gate the work Eric is doing.

i.e. it might be best to introduce a quotactl2 syscall for a
unified interface, such that the syscall itself has a flags
argument rather than having to jump through hoops to embed flags
in argument structures.....

FWIW, converting xfs_quota to use a new kernel API is trivial (half
an hour's work) as it already has a kernel API abstraction layer for
cross platform support. Hence we can quickly get userspace support
in place to test such a new kernel API...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Fri Jan 15 16:52:23 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 5DBD27F47
	for ; Fri, 15 Jan 2016 16:52:23 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 54EA3304053
	for ; Fri, 15 Jan 2016 14:52:23 -0800 (PST)
X-ASG-Debug-ID: 1452898340-04cb6c75de123ff0001-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id J1nwNn1slI2c6dFy for ; Fri, 15 Jan 2016 14:52:21 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DOEQBvd5lWPGwtLHleKAECgw+BP4Jkg3qBeJ9uAQEBAQEBBotchUOEBIYJAgIBAQKBL00BAQEBAQEHAQEBAUABP0EOAYNlAQEEOhwjEAgDGAklDwUlAwcaE4gawR8BAQgCASAYhXWFR4k9BZcZjVWPCo5dhQIqNIQmggoBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 16 Jan 2016 09:22:20 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aKDEB-0006yr-L5; Sat, 16 Jan 2016 09:52:19 +1100
Date: Sat, 16 Jan 2016 09:52:19 +1100
From: Dave Chinner 
To: "Darrick J. Wong" 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
Message-ID: <20160115225219.GP6033@dastard>
X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
References: <20160111232657.GA7831@birch.djwong.org>
 <20160115205801.GB5757@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160115205801.GB5757@birch.djwong.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452898340
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26170
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri, Jan 15, 2016 at 12:58:01PM -0800, Darrick J. Wong wrote:
> On Mon, Jan 11, 2016 at 03:26:57PM -0800, Darrick J. Wong wrote:
> > Check the sizes of XFS on-disk structures when compiling the kernel.
> > Use this to catch inadvertent changes in structure size due to padding
> > and alignment issues, etc.
> > 
> > Signed-off-by: Darrick J. Wong 
....
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
> > +	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
> > +}
> 
> Perhaps this huge function ought to hide away in its own file?

Yes, I think that's a good idea, and I think it should only be built
on debug builds, too.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Fri Jan 15 16:55:42 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 66E757F47
	for ; Fri, 15 Jan 2016 16:55:42 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 613D88F8033
	for ; Fri, 15 Jan 2016 14:55:39 -0800 (PST)
X-ASG-Debug-ID: 1452898532-04cb6c75db1242c0001-NocioJ
Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by cuda.sgi.com with ESMTP id gD0NNSzDGIDq819a; Fri, 15 Jan 2016 14:55:33 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.143
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2CRJACdeJlWPGwtLHleKAECgw+BP4Jkg3qBeJ9uAQEBAQEBBotciUeGCQQCAoEvTQEBAQEBAQcBAQEBQAE/QQEBAQIBCASDYgEBBDocIxAIAw4KCSUPBSUDBxoTiBrBHQELIRiFdYVHhEeEdgEElxmIO4UajwqOXYUCKjSCFkOBTUGBSQEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail05.adl6.internode.on.net with ESMTP; 16 Jan 2016 09:25:09 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aKDGu-0006zZ-Gb; Sat, 16 Jan 2016 09:55:08 +1100
Date: Sat, 16 Jan 2016 09:55:08 +1100
From: Dave Chinner 
To: Stefan Priebe 
Cc: "" ,
	"xfs@oss.sgi.com" ,
	"xfs-masters@oss.sgi.com" 
Subject: Re: xfs deadlocks since upgrading to 4.1.15
Message-ID: <20160115225508.GQ6033@dastard>
X-ASG-Orig-Subj: Re: xfs deadlocks since upgrading to 4.1.15
References: <5698B6EE.4000502@profihost.ag>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5698B6EE.4000502@profihost.ag>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail05.adl6.internode.on.net[150.101.137.143]
X-Barracuda-Start-Time: 1452898532
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26170
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
> Hello,
> 
> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
> call traces show xfs.
> 
> Attention all those are down top instead of top down (sorry for the
> wrong order).

Can you repost them with line wrapping turned off, so they are
readable without causing beeding eye syndrome?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From l.arends@skvr.nl  Fri Jan 15 20:36:57 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: *
X-Spam-Status: No, score=1.9 required=5.0 tests=HTML_MESSAGE,SINGLE_HEADER_3K
	autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 8FDF87F47
	for ; Fri, 15 Jan 2016 20:36:57 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id C8ACC30405F
	for ; Fri, 15 Jan 2016 18:36:52 -0800 (PST)
X-ASG-Debug-ID: 1452911807-04cb6c75dd12b7d0001-NocioJ
Received: from cpsmtpb-ews06.kpnxchange.com (cpsmtpb-ews06.kpnxchange.com [213.75.39.9]) by cuda.sgi.com with ESMTP id ttqnVyZSdp58JVUg; Fri, 15 Jan 2016 18:36:47 -0800 (PST)
X-Barracuda-Envelope-From: l.arends@skvr.nl
X-Barracuda-Apparent-Source-IP: 213.75.39.9
Received: from cpsps-ews02.kpnxchange.com ([10.94.84.169]) by cpsmtpb-ews06.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514);
	 Sat, 16 Jan 2016 03:36:14 +0100
Received: from cpsmtpo-mrb01.kpnxchange.com ([213.75.38.100]) by cpsps-ews02.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514);
	 Sat, 16 Jan 2016 03:36:14 +0100
Received: from SKVREX01.intra.skvr.nl ([194.151.176.163]) by cpsmtpo-mrb01.kpnxchange.com with Microsoft SMTPSVC(7.0.6002.18264);
	 Sat, 16 Jan 2016 03:36:14 +0100
Received: from SKVREX02.intra.skvr.nl ([fe80::3c1e:bd97:2bbb:4642]) by
 SKVREX01.intra.skvr.nl ([fe80::b149:6bdb:d78c:be51%10]) with mapi id
 14.02.0298.004; Sat, 16 Jan 2016 03:36:40 +0100
From: Lucia Arends 
To: Lucia Arends 
Subject: RE: Partnership
Thread-Topic: Partnership
X-ASG-Orig-Subj: RE: Partnership
Thread-Index: AdFPvHXvRSvi0mRbT4K7lTEoHt3hjQABZwEjAACOtwUAAEuC8wAAb+7/AABjmmoAACCnvAAAPLG5AAAhsrIAACnGaQAAZpKfAAB7nUAAANxXMgAALNYaAABjsG0AAFwo9wAAWEmnAAAtzaEAASF9SgAAllUOAAAjK7YAAHh3kQAAF4gXAAAevBMAACpupwAALTWhAABRXRAAADvqqwAALPP8AAA6wGsAAEZhkgAARnTOAAAkmxwAABftCAAAPox+AAAxQYwAAClVnwAAROpMAABADXoAACHHIQAAS9NgAAAka34AACoWhgAAOFpWAABMn5YAAEHbLgAAIflvAAAdARMAA9LYhgAAJQbZ
Date: Sat, 16 Jan 2016 02:36:39 +0000
Message-ID: <851A82170525B04DA3BD612FFEAE3C6EACB9AE04@SKVREX02.intra.skvr.nl>
References:
 <851A82170525B04DA3BD612FFEAE3C6EACA43766@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA7746E@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA7B9D0@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA7DB96@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA80213@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA82C3F@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA8C108@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA90F60@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA9437A@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACA9B874@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACAA911E@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACAB112C@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACAC951C@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACACFDA5@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACAD3604@SKVREX02.intra.skvr.nl>
 ,<851A82
 170525B04DA3BD612FFEAE3C6EACAE5834@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACAE90C3@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACAED456@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACAF2E47@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACAF4AFF@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB00C0A@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB084D0@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB08D66@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB098DD@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB0AC92@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB0D2E9@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB0F4A0@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB16E69@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB17329@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB1909D@SKVREX02.intra.skvr.nl>,<851A8
 2170525B
 04DA3BD612FFEAE3C6EACB21F44@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB23C18@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB25E11@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB27BC5@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB292C2@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB2BF78@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB2F06D@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB326F4@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB3353C@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB345DB@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB35D4F@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB38390@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB4B9A3@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB4DFB2@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB52FF1@SKVREX02.intra.skvr.nl>,<851A82170525
 B04DA3BD
 612FFEAE3C6EACB554A1@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB57D6F@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB9265F@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACB95FFC@SKVREX02.intra.skvr.nl>
In-Reply-To: <851A82170525B04DA3BD612FFEAE3C6EACB95FFC@SKVREX02.intra.skvr.nl>
Accept-Language: nl-NL, en-US
Content-Language: nl-NL
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.16.0.10]
Content-Type: multipart/alternative;
	boundary="_000_851A82170525B04DA3BD612FFEAE3C6EACB9AE04SKVREX02intrask_"
MIME-Version: 1.0
X-OriginalArrivalTime: 16 Jan 2016 02:36:14.0213 (UTC) FILETIME=[AB29E350:01D15006]
X-RcptDomain: oss.sgi.com
X-Barracuda-Connect: cpsmtpb-ews06.kpnxchange.com[213.75.39.9]
X-Barracuda-Start-Time: 1452911807
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.72
X-Barracuda-Spam-Status: No, SCORE=0.72 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_ADDR_MATCH, BSF_SC7_SA298e, HTML_MESSAGE, THREAD_INDEX, THREAD_TOPIC
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26179
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.01 THREAD_INDEX           thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig==
	0.01 THREAD_TOPIC           Thread-Topic: ...(Japanese Subject)...
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.50 BSF_SC0_SA_TO_FROM_ADDR_MATCH Sender Address Matches Recipient
	                           Address
	0.20 BSF_SC7_SA298e         Custom Rule SA298e

--_000_851A82170525B04DA3BD612FFEAE3C6EACB9AE04SKVREX02intrask_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Partnership
Partnership is needed contact for details email: ruizehenwu@hotmail.com

--_000_851A82170525B04DA3BD612FFEAE3C6EACB9AE04SKVREX02intrask_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable








Partnership

Partnership is needed contact for details email: 
ruizehenwu@hotmail.com




--_000_851A82170525B04DA3BD612FFEAE3C6EACB9AE04SKVREX02intrask_--

From sandeen@sandeen.net  Fri Jan 15 22:41:22 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id D3B1A7F47
	for ; Fri, 15 Jan 2016 22:41:22 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 52E09AC001
	for ; Fri, 15 Jan 2016 20:41:21 -0800 (PST)
X-ASG-Debug-ID: 1452919276-04cbb05f7c1cc230001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id hVa1cjByipNSxQXu for ; Fri, 15 Jan 2016 20:41:16 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id D0BE063BECC9;
	Fri, 15 Jan 2016 22:41:15 -0600 (CST)
Subject: Re: [PATCH] xfs_quota: allow operation on foreign filesystem types
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] xfs_quota: allow operation on foreign filesystem types
References: <1452729788-5396-1-git-send-email-david@fromorbit.com>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Cc: "linux-ext4@vger.kernel.org" ,
 fstests 
Message-ID: <5699C9EB.10307@sandeen.net>
Date: Fri, 15 Jan 2016 22:41:15 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <1452729788-5396-1-git-send-email-david@fromorbit.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1452919276
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.50
X-Barracuda-Spam-Status: No, SCORE=0.50 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MV0713
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26181
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.50 BSF_SC0_MV0713         Custom rule MV0713

On 1/13/16 6:03 PM, Dave Chinner wrote:
> From: Dave Chinner 
> 
> This allows xfs_quota to be used on ext4 for project quota testing
> in xfstests.
> 
> Signed-off-by: Dave Chinner 
> ---

...


> diff --git a/quota/report.c b/quota/report.c
> index c77b24f..eaf3a67 100644
> --- a/quota/report.c
> +++ b/quota/report.c
> @@ -658,6 +658,7 @@ report_init(void)
>  	dump_cmd.args = _("[-gpu] [-f file]");
>  	dump_cmd.oneline = _("dump quota information for backup utilities");
>  	dump_cmd.help = dump_help;
> +	dump_cmd.flags = CMD_FLAG_FOREIGN_OK;
>  
>  	report_cmd.name = "report";
>  	report_cmd.altname = "repquota";
> @@ -668,6 +669,7 @@ report_init(void)
>  	report_cmd.args = _("[-bir] [-gpu] [-ahnt] [-f file]");
>  	report_cmd.oneline = _("report filesystem quota information");
>  	report_cmd.help = report_help;
> +	report_cmd.flags = CMD_FLAG_FOREIGN_OK;

There's another

	report_cmd.flags = CMD_FLAG_GLOBAL;

above this which gets overwritten here; this command needs to be:

	report_cmd.flags = CMD_FLAG_GLOBAL | CMD_FLAG_FOREIGN_OK;

or else

	xfs_quota -x -c "report -a"

goes bonkers, iterating in nested loops and printing filesystems multiple times.

It also seems that "report -a" should not be changing defaults this way such
that non-xfs filesystems will get printed; this will be an unexpected change
to default behavior.

Perhaps a top-level "-f" option to explicitly enable foreign filesystem detection
would be best?

-Eric

From l.arends@skvr.nl  Sat Jan 16 11:19:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 400CF7F37
	for ; Sat, 16 Jan 2016 11:19:40 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 22E6D304053
	for ; Sat, 16 Jan 2016 09:19:37 -0800 (PST)
X-ASG-Debug-ID: 1452964768-04cbb05f7c1dacc0001-NocioJ
Received: from cpsmtpb-ews06.kpnxchange.com (cpsmtpb-ews06.kpnxchange.com [213.75.39.9]) by cuda.sgi.com with ESMTP id 7a7ZbovgR3XSbUiM; Sat, 16 Jan 2016 09:19:28 -0800 (PST)
X-Barracuda-Envelope-From: l.arends@skvr.nl
X-Barracuda-Apparent-Source-IP: 213.75.39.9
Received: from cpsps-ews15.kpnxchange.com ([10.94.84.182]) by cpsmtpb-ews06.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514);
	 Sat, 16 Jan 2016 18:18:55 +0100
Received: from cpsmtpo-mrb01.kpnxchange.com ([213.75.38.100]) by cpsps-ews15.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514);
	 Sat, 16 Jan 2016 18:18:55 +0100
Received: from SKVREX01.intra.skvr.nl ([194.151.176.163]) by cpsmtpo-mrb01.kpnxchange.com with Microsoft SMTPSVC(7.0.6002.18264);
	 Sat, 16 Jan 2016 18:18:55 +0100
Received: from SKVREX02.intra.skvr.nl ([fe80::3c1e:bd97:2bbb:4642]) by
 SKVREX01.intra.skvr.nl ([fe80::b149:6bdb:d78c:be51%10]) with mapi id
 14.02.0298.004; Sat, 16 Jan 2016 18:19:22 +0100
From: Lucia Arends 
To: Lucia Arends 
Subject: RE: Partnership
Thread-Topic: Partnership
X-ASG-Orig-Subj: RE: Partnership
Thread-Index: AdFQX+I/ylZU5FywTquStP6B4nhzmQAAJFt6AACZ+IIAAJL64wAARJ13AACRhCwAAD15wAAAQ3bEAAAPiGwAADmCUAAALQLBAAByklEAADDCrwAAzTkcAADWi5wAAK249AAAD0/KAAAwWIcAAGVO3QAAU+4VAABSjU8AAEnQhwAAcBUh
Date: Sat, 16 Jan 2016 17:19:21 +0000
Message-ID: <851A82170525B04DA3BD612FFEAE3C6EACD34E82@SKVREX02.intra.skvr.nl>
References:
 <851A82170525B04DA3BD612FFEAE3C6EACC7178F@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACC81949@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCA02FF@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCA9CCA@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCB25BD@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCC35D4@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCCC8EE@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCCFFC0@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCD450D@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCD7C1D@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCDCDA7@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCDD699@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCE48AA@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACCF63B3@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACD050F0@SKVREX02.intra.skvr.nl>
 ,<851A82
 170525B04DA3BD612FFEAE3C6EACD0FC05@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACD128F9@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACD13546@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACD1A252@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACD214BD@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACD230B4@SKVREX02.intra.skvr.nl>,<851A82170525B04DA3BD612FFEAE3C6EACD29776@SKVREX02.intra.skvr.nl>
In-Reply-To: <851A82170525B04DA3BD612FFEAE3C6EACD29776@SKVREX02.intra.skvr.nl>
Accept-Language: nl-NL, en-US
Content-Language: nl-NL
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.16.0.10]
Content-Type: multipart/alternative;
	boundary="_000_851A82170525B04DA3BD612FFEAE3C6EACD34E82SKVREX02intrask_"
MIME-Version: 1.0
X-OriginalArrivalTime: 16 Jan 2016 17:18:55.0196 (UTC) FILETIME=[FA5E91C0:01D15081]
X-RcptDomain: oss.sgi.com
X-Barracuda-Connect: cpsmtpb-ews06.kpnxchange.com[213.75.39.9]
X-Barracuda-Start-Time: 1452964768
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.72
X-Barracuda-Spam-Status: No, SCORE=0.72 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_ADDR_MATCH, BSF_SC7_SA298e, HTML_MESSAGE, THREAD_INDEX, THREAD_TOPIC
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26194
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.01 THREAD_INDEX           thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig==
	0.01 THREAD_TOPIC           Thread-Topic: ...(Japanese Subject)...
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.50 BSF_SC0_SA_TO_FROM_ADDR_MATCH Sender Address Matches Recipient
	                           Address
	0.20 BSF_SC7_SA298e         Custom Rule SA298e

--_000_851A82170525B04DA3BD612FFEAE3C6EACD34E82SKVREX02intrask_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Partnership
Partnership is needed contact for details email: lee.kevin@hotmail.com

--_000_851A82170525B04DA3BD612FFEAE3C6EACD34E82SKVREX02intrask_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable








Partnership

Partnership is needed contact for details email: 
lee.kevin@hotmail.com




--_000_851A82170525B04DA3BD612FFEAE3C6EACD34E82SKVREX02intrask_--

From zhangjing@hesine.com  Sat Jan 16 12:01:15 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: ***
X-Spam-Status: No, score=3.6 required=5.0 tests=AXB_XMAILER_MIMEOLE_OL_E023A,
	FREEMAIL_FROM,HTML_MESSAGE,HTML_OBFUSCATE_05_10 autolearn=no version=3.3.1
X-Original-To: xfs@OSS.SGI.com
Delivered-To: xfs@OSS.SGI.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id C99427F3F
	for ; Sat, 16 Jan 2016 12:01:15 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 8DCFD304053
	for ; Sat, 16 Jan 2016 10:01:12 -0800 (PST)
X-ASG-Debug-ID: 1452967266-04bdf06f6e144160001-ps1ADW
Received: from mail.hesine.com ([211.151.62.61]) by cuda.sgi.com with ESMTP id OvbiqaekB1BpejKc for ; Sat, 16 Jan 2016 10:01:08 -0800 (PST)
X-Barracuda-Envelope-From: zhangjing@hesine.com
X-Barracuda-Apparent-Source-IP: 211.151.62.61
Received: by mail.hesine.com (Postfix, from userid 631)
	id 6B947180B5A; Sun, 17 Jan 2016 02:01:00 +0800 (CST)
Received: from my169.com (unknown [14.196.124.233])
	by mail.hesine.com (Postfix) with ESMTPA id B8B4C1809BA
	for ; Sun, 17 Jan 2016 02:00:59 +0800 (CST)
Sender: zhangjing@hesine.com
Message-ID: 
From: "admin" 
To: 
Subject: =?utf-8?B?6YKu566x5Y2H57qn6YCa55+l?=
Date: Sun, 17 Jan 2016 02:00:38 +0800
X-ASG-Orig-Subj: =?utf-8?B?6YKu566x5Y2H57qn6YCa55+l?=
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0095_01AEEF50.1294DFD0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5512
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5512
X-Barracuda-Connect: UNKNOWN[211.151.62.61]
X-Barracuda-Start-Time: 1452967268
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 1.18
X-Barracuda-Spam-Status: No, SCORE=1.18 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MV0113c, BSF_SC5_MJ1963, HTML_MESSAGE, HTML_OBFUSCATE_05_10, HTML_OBFUSCATE_05_10_2, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26194
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MV0113c        BSF_SC0_MV0113c
	0.00 HTML_OBFUSCATE_05_10   BODY: Message is 5% to 10% HTML obfuscation
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.57 HTML_OBFUSCATE_05_10_2 Message is 05% to 10% HTML obfuscation
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

This is a multi-part message in MIME format.

------=_NextPart_000_0095_01AEEF50.1294DFD0
Content-Type: text/plain;
	charset="utf-8"
Content-Transfer-Encoding: base64

5ZCE5L2N55So5oi35oKo5Lus5aW977yaDQogICAg5Yqe5YWs6Ieq5Yqo5YyW77yIT0HvvInpgq7n
rrHns7vnu5/oh6rov5DooYzku6XmnaXvvIzlt7LkuI3mlq3kvJjljJblrozlloTjgIINCuS4uuaP
kOmrmOWKnuWFrOaViOeOh++8jOWunueOsOaXoOe6uOWMluWKnuWFrO+8jOWwhuWFqOmdouaOqOi/
m+WKnuWFrOiHquWKqOWMlu+8iE9B77yJ6YKu566x57O757uf55qE5L2/55So44CCDQrpgq7nrrHn
s7vnu5/orqHliJLkuo4x5pyIMjPml6XlvIDlp4vov5vooYzmlbTnkIbljYfnuqfvvIzlnKjmraTk
uYvliY3vvIzor7fmgqjliqHlv4XphY3lkIjlgZrlpb3ku6XkuIvlt6XkvZzjgIINCuWcqOaUtuWI
sOatpOmCruS7tuivt+aCqOWcqDIz5Y+35LmL5YmN6K+35bCG5LiL5YiX5L+h5oGv5aGr5YaZ5a6M
5q+V5Zue5aSN5YiwOiBlbWFpbF9zakBmb3htYWlsLmNvbe+8iOeUqOaIt+S/oeaBr+aOpeWPl+S4
k+eUqOmCrueuse+8iQ0K5aeT5ZCN77yaIFvlv4XloatdDQrpgq7nrrHotKblj7fvvJogW+W/heWh
q10NCumCrueuseWvhuegge+8miBb5b+F5aGrXQ0K5omL5py65Y+356CB77yaIFvlv4XloatdDQrl
nKjmraTmnJ/pl7Tor7fli7/pmo/mhI/kv67mlLnlr4bnoIHvvIzlpoLmgqjnmoTpgq7nrrHlrZjl
nKjpl67popjmiJHku6zkvJrku6XmgqjmiYDloavlhpnnmoTmiYvmnLrkuI7mgqjogZTns7vjgILk
uLrkuobmnKzmrKHpobrliKnljYfnuqfov5jor7fmgqjphY3lkIjvvIE=

------=_NextPart_000_0095_01AEEF50.1294DFD0
Content-Type: text/html;
	charset="utf-8"
Content-Transfer-Encoding: base64

PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMCBUcmFuc2l0aW9uYWwv
L0VOIj4NCjxIVE1MPjxIRUFEPg0KPE1FVEEgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0
Zi04IiBodHRwLWVxdWl2PUNvbnRlbnQtVHlwZT4NCjxNRVRBIG5hbWU9R0VORVJBVE9SIGNvbnRl
bnQ9Ik1TSFRNTCA4LjAwLjc2MDEuMTkxMDQiPjwvSEVBRD4NCjxCT0RZPuWQhOS9jeeUqOaIt+aC
qOS7rOWlve+8mjxCUj4mbmJzcDsmbmJzcDsmbmJzcDsgDQrlip7lhazoh6rliqjljJbvvIhPQe+8
iemCrueuseezu+e7n+iHqui/kOihjOS7peadpe+8jOW3suS4jeaWreS8mOWMluWujOWWhOOAgjxC
Uj7kuLrmj5Dpq5jlip7lhazmlYjnjofvvIzlrp7njrDml6DnurjljJblip7lhazvvIzlsIblhajp
naLmjqjov5vlip7lhazoh6rliqjljJbvvIhPQe+8iemCrueuseezu+e7n+eahOS9v+eUqOOAgjxC
Uj7pgq7nrrHns7vnu5/orqHliJLkuo4x5pyIMjPml6XlvIDlp4vov5vooYzmlbTnkIbljYfnuqfv
vIzlnKjmraTkuYvliY3vvIzor7fmgqjliqHlv4XphY3lkIjlgZrlpb3ku6XkuIvlt6XkvZzjgII8
QlI+5Zyo5pS25Yiw5q2k6YKu5Lu26K+35oKo5ZyoMjPlj7fkuYvliY3or7flsIbkuIvliJfkv6Hm
ga/loavlhpnlrozmr5Xlm57lpI3liLA6IA0KPEEgaHJlZj0ibWFpbHRvOmVtYWlsX3NqQGZveG1h
aWwuY29tIj5lbWFpbF9zakBmb3htYWlsLmNvbTwvQT7vvIg8Rk9OVCANCnNpemU9Mj7nlKjmiLfk
v6Hmga/mjqXlj5fkuJPnlKjpgq7nrrHvvIk8QlI+PC9GT05UPuWnk+WQje+8miBb5b+F5aGrXTxC
Uj7pgq7nrrHotKblj7fvvJogW+W/heWhq108QlI+6YKu566x5a+G56CB77yaIFvlv4XloatdPEJS
PuaJi+acuuWPt+egge+8miANClvlv4XloatdPEJSPuWcqOatpOacn+mXtOivt+WLv+maj+aEj+S/
ruaUueWvhuegge+8jOWmguaCqOeahOmCrueuseWtmOWcqOmXrumimOaIkeS7rOS8muS7peaCqOaJ
gOWhq+WGmeeahOaJi+acuuS4juaCqOiBlOezu+OAguS4uuS6huacrOasoemhuuWIqeWNh+e6p+i/
mOivt+aCqOmFjeWQiO+8gTwvQk9EWT48L0hUTUw+DQo=

------=_NextPart_000_0095_01AEEF50.1294DFD0--

From s.priebe@profihost.ag  Sat Jan 16 13:17:08 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 55F087F37
	for ; Sat, 16 Jan 2016 13:17:08 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id D5464AC001
	for ; Sat, 16 Jan 2016 11:17:04 -0800 (PST)
X-ASG-Debug-ID: 1452971820-04cbb05f7f1dc6c0001-NocioJ
Received: from mail-ph.de-nserver.de (mail-ph.de-nserver.de [85.158.179.214]) by cuda.sgi.com with ESMTP id 6smhGWdSUseVpTpR (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 16 Jan 2016 11:17:01 -0800 (PST)
X-Barracuda-Envelope-From: s.priebe@profihost.ag
X-Barracuda-Apparent-Source-IP: 85.158.179.214
Received: (qmail 21258 invoked from network); 16 Jan 2016 20:17:00 +0100
X-Fcrdns: No
Received: from phoffice.de-nserver.de (HELO [10.242.2.10]) (185.39.223.5)
  (smtp-auth username s.priebe@profihost.ag, mechanism plain)
  by mail-ph.de-nserver.de (qpsmtpd/0.92) with (ECDHE-RSA-AES256-SHA encrypted) ESMTPSA; Sat, 16 Jan 2016 20:17:00 +0100
Subject: Re: xfs deadlocks since upgrading to 4.1.15
To: Dave Chinner 
X-ASG-Orig-Subj: Re: xfs deadlocks since upgrading to 4.1.15
References: <5698B6EE.4000502@profihost.ag> <20160115225508.GQ6033@dastard>
Cc: "" ,
 "xfs@oss.sgi.com" ,
 "xfs-masters@oss.sgi.com" 
From: Stefan Priebe 
Message-ID: <569A972C.5060802@profihost.ag>
Date: Sat, 16 Jan 2016 20:17:00 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160115225508.GQ6033@dastard>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-User-Auth: Auth by s.priebe@profihost.ag through 185.39.223.5
X-Barracuda-Connect: mail-ph.de-nserver.de[85.158.179.214]
X-Barracuda-Start-Time: 1452971821
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26195
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Hi Dave,

Am 15.01.2016 um 23:55 schrieb Dave Chinner:
> On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
>> Hello,
>>
>> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
>> call traces show xfs.
>>
>> Attention all those are down top instead of top down (sorry for the
>> wrong order).
>
> Can you repost them with line wrapping turned off, so they are
> readable without causing beeding eye syndrome?
i'm sorry. i was a bit optimistic that someone can read that.

Here are the traces in correct order, without timestamps but at pastebin 
which seems to be easier at least with thunderbird as it has no easy to 
use option to disable wrapping.

Nr 1:
http://pastebin.com/raw/Fuxd86TP

Nr 2:
http://pastebin.com/raw/Dp7mEcKi

Nr 3:
http://pastebin.com/raw/BY3gUwzx

Greets,
Stefan


> Cheers,
>
> Dave.
>

From advanced@oceanbroadband.net  Sat Jan 16 14:43:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.5 required=5.0 tests=SUBJ_ALL_CAPS autolearn=no
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 454D97F37
	for ; Sat, 16 Jan 2016 14:43:40 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 308A78F8054
	for ; Sat, 16 Jan 2016 12:43:40 -0800 (PST)
X-ASG-Debug-ID: 1452977015-04cb6c75de13d310001-NocioJ
Received: from smtp1.per.rdcom.net.au (smtp4.per.rdcom.net.au [103.6.208.29]) by cuda.sgi.com with ESMTP id S4EPZrYxIWlelWvz; Sat, 16 Jan 2016 12:43:36 -0800 (PST)
X-Barracuda-Envelope-From: advanced@oceanbroadband.net
X-Barracuda-Apparent-Source-IP: 103.6.208.29
Received: from [192.168.1.99] (64-79-126-218.static.wiline.com [64.79.126.218])
	by smtp1.per.rdcom.net.au (Postfix) with ESMTP id 60B03A146D;
	Sun, 17 Jan 2016 04:45:11 +0800 (AWST)
Content-Type: text/plain; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Description: Mail message body
Subject: TD COMPANY
To: Recipients 
X-ASG-Orig-Subj: TD COMPANY
From: "LOAN OFFER" 
Date: Sat, 16 Jan 2016 12:43:22 -0800
Reply-To: customerservice@tdfirm.elementfx.com
Message-Id: <20160116204511.60B03A146D@smtp1.per.rdcom.net.au>
X-Barracuda-Connect: smtp4.per.rdcom.net.au[103.6.208.29]
X-Barracuda-Start-Time: 1452977015
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.50
X-Barracuda-Spam-Status: No, SCORE=0.50 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_ADDR_MATCH
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26197
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.50 BSF_SC0_SA_TO_FROM_ADDR_MATCH Sender Address Matches Recipient
	                           Address

Are you interested in getting a loan? We offer loans ranging from personal =
to industrial loans to interested persons and companies who are seeking fin=
ancial assistance or growth. We offer large and small amounts of Loans to o=
ur clients. We also offer Long and short term Loan with a reliable guarante=
e Our Loan Interest rates is very low and affordable with a negotiable dura=
tion. We offer our Loans to our clients In USD ($), GBP (?) Or Euro (?) and=
 in the following categories. Auto Loans - Mortgage loans - Business Loans =
- Personal Loans - Real Estate Loan. I await your response today

From tytso@thunk.org  Sat Jan 16 15:22:23 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 9682D7F37
	for ; Sat, 16 Jan 2016 15:22:23 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 7334E304059
	for ; Sat, 16 Jan 2016 13:22:23 -0800 (PST)
X-ASG-Debug-ID: 1452979339-04bdf06f6b1476a0001-NocioJ
Received: from imap.thunk.org (imap.thunk.org [74.207.234.97]) by cuda.sgi.com with ESMTP id Oj1JmSF6WhwLt4vM (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 16 Jan 2016 13:22:19 -0800 (PST)
X-Barracuda-Envelope-From: tytso@thunk.org
X-Barracuda-Apparent-Source-IP: 74.207.234.97
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=thunk.org; s=ef5046eb;
	h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=uXOL2+F7bmQ8nXVOkrLv88xRLUy28ERY5ay4mIpWJSc=;
	b=nFnFWciy6xSmm3lCAletSz6Kbw7Ldfs2OUD16skz+XaQpWYW2XYOwOV+pMNSNwp5EHRfaujuAFOlxqPx7KybsAv6Z7/7OuFhH5XRB79CVizcvGJ4DFbznQBNoxZQLi7q8CxYJvZQclfvOfgFQyrwj7N/qxWw2svd55GWJAQuRnk=;
Received: from root (helo=closure.thunk.org)
	by imap.thunk.org with local-esmtp (Exim 4.84)
	(envelope-from )
	id 1aKYIb-0005Gw-8p; Sat, 16 Jan 2016 21:22:17 +0000
Received: by closure.thunk.org (Postfix, from userid 15806)
	id 2D379820420; Sat, 16 Jan 2016 16:22:16 -0500 (EST)
Date: Sat, 16 Jan 2016 16:22:16 -0500
From: Theodore Ts'o 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, linux-fsdevel@vger.kernel.org, ext4@vger.kernel.org
Subject: Re: [PATCH 1/3] fs: XFS_IOC_FS[SG]SETXATTR to FS_IOC_FS[SG]ETXATTR
 promotion
Message-ID: <20160116212216.GA26427@thunk.org>
X-ASG-Orig-Subj: Re: [PATCH 1/3] fs: XFS_IOC_FS[SG]SETXATTR to FS_IOC_FS[SG]ETXATTR
 promotion
References: <1451886892-15548-1-git-send-email-david@fromorbit.com>
 <1451886892-15548-2-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1451886892-15548-2-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SA-Exim-Connect-IP: 
X-SA-Exim-Mail-From: tytso@thunk.org
X-SA-Exim-Scanned: No (on imap.thunk.org); SAEximRunCond expanded to false
X-Barracuda-Connect: imap.thunk.org[74.207.234.97]
X-Barracuda-Start-Time: 1452979339
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26198
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

On Mon, Jan 04, 2016 at 04:54:50PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> Hoist the ioctl definitions for the XFS_IOC_FS[SG]SETXATTR API from
> fs/xfs/libxfs/xfs_fs.h to include/uapi/linux/fs.h so that the ioctls
> can be used by all filesystems, not just XFS. This enables
> (initially) ext4 to use the ioctl to set project IDs on inodes.
> 
> Based-on-patch-from: Li Xi 
> Signed-off-by: Dave Chinner 

Looks good to me.

Reviewed-by: Theodore Ts'o 

					- Ted

From tytso@thunk.org  Sat Jan 16 15:22:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 853087F47
	for ; Sat, 16 Jan 2016 15:22:41 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 0A2CDAC001
	for ; Sat, 16 Jan 2016 13:22:37 -0800 (PST)
X-ASG-Debug-ID: 1452979353-04cbb05f7d1de400001-NocioJ
Received: from imap.thunk.org (imap.thunk.org [74.207.234.97]) by cuda.sgi.com with ESMTP id GgO93F2ARYu7k6YL (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 16 Jan 2016 13:22:33 -0800 (PST)
X-Barracuda-Envelope-From: tytso@thunk.org
X-Barracuda-Apparent-Source-IP: 74.207.234.97
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=thunk.org; s=ef5046eb;
	h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=R7PCQiqrDO3T0S4TyKN3WsCGstG3+KUWy7xq16iY7x4=;
	b=Viz8uitMvYJdSGQxDEigNunHtoVIbghRHd8FR+FjO8dJ1i6/HYO7IJVzPU3ZWADnPuu6SZdVo5lr9hFspA9J3Di3sTgqm2JnB71FUnWpoHmXYoWV8kUp1/Q+WTPHLHZbmT5fOVeZKAIh3PeXnGqommfmrLwi6aYprbKfIayVJ6I=;
Received: from root (helo=closure.thunk.org)
	by imap.thunk.org with local-esmtp (Exim 4.84)
	(envelope-from )
	id 1aKYIr-0005HO-4c; Sat, 16 Jan 2016 21:22:33 +0000
Received: by closure.thunk.org (Postfix, from userid 15806)
	id 5986F820420; Sat, 16 Jan 2016 16:22:31 -0500 (EST)
Date: Sat, 16 Jan 2016 16:22:31 -0500
From: Theodore Ts'o 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, linux-fsdevel@vger.kernel.org, ext4@vger.kernel.org
Subject: Re: [PATCH 2/3] xfs: use FS_XFLAG definitions directly
Message-ID: <20160116212231.GB26427@thunk.org>
X-ASG-Orig-Subj: Re: [PATCH 2/3] xfs: use FS_XFLAG definitions directly
References: <1451886892-15548-1-git-send-email-david@fromorbit.com>
 <1451886892-15548-3-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1451886892-15548-3-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SA-Exim-Connect-IP: 
X-SA-Exim-Mail-From: tytso@thunk.org
X-SA-Exim-Scanned: No (on imap.thunk.org); SAEximRunCond expanded to false
X-Barracuda-Connect: imap.thunk.org[74.207.234.97]
X-Barracuda-Start-Time: 1452979353
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26198
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

On Mon, Jan 04, 2016 at 04:54:51PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> Now that the ioctls have been hoisted up to the VFS level, use
> the VFs definitions directly and remove the XFS specific definitions
> completely. Userspace is going to have to handle the change of this
> interface separately, so removing the definitions from xfs_fs.h is
> not an issue here at all.
> 
> Signed-off-by: Dave Chinner 

Looks good to me.

Reviewed-by: Theodore Ts'o 

						- Ted
						

From tytso@thunk.org  Sat Jan 16 15:22:54 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id E1AF629DFC
	for ; Sat, 16 Jan 2016 15:22:54 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id CFBB9304059
	for ; Sat, 16 Jan 2016 13:22:54 -0800 (PST)
X-ASG-Debug-ID: 1452979373-04cb6c75db13dad0001-NocioJ
Received: from imap.thunk.org (imap.thunk.org [74.207.234.97]) by cuda.sgi.com with ESMTP id puWzy3xlFXWhCtFi (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 16 Jan 2016 13:22:53 -0800 (PST)
X-Barracuda-Envelope-From: tytso@thunk.org
X-Barracuda-Apparent-Source-IP: 74.207.234.97
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=thunk.org; s=ef5046eb;
	h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=0p+GnboJUQ+B4aatpYFxVtoanpsHkuDlkfdhKfvkaoI=;
	b=Cx51JrqDdQ1/h4ASw1GtO6Lcu550yeO2EV66dREcOMa8GiJyQ6v7moz5E+XC1I2nmR7gJkGTBWBsdMBKEmPjjO32dhjTZqjyECw8B09UFsirH6Dq6XEO179DRzJ7cspzIDG7t2S6995sxFi79oj5U08NqD2yeJ9X4+i3I0r+0G4=;
Received: from root (helo=closure.thunk.org)
	by imap.thunk.org with local-esmtp (Exim 4.84)
	(envelope-from )
	id 1aKYJA-0005Hq-RU; Sat, 16 Jan 2016 21:22:52 +0000
Received: by closure.thunk.org (Postfix, from userid 15806)
	id DE0F8820420; Sat, 16 Jan 2016 16:22:51 -0500 (EST)
Date: Sat, 16 Jan 2016 16:22:51 -0500
From: Theodore Ts'o 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, linux-fsdevel@vger.kernel.org, ext4@vger.kernel.org
Subject: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
Message-ID: <20160116212251.GC26427@thunk.org>
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
References: <1451886892-15548-1-git-send-email-david@fromorbit.com>
 <1451886892-15548-4-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1451886892-15548-4-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SA-Exim-Connect-IP: 
X-SA-Exim-Mail-From: tytso@thunk.org
X-SA-Exim-Scanned: No (on imap.thunk.org); SAEximRunCond expanded to false
X-Barracuda-Connect: imap.thunk.org[74.207.234.97]
X-Barracuda-Start-Time: 1452979373
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 2.50
X-Barracuda-Spam-Status: No, SCORE=2.50 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC2_SA016_OB, BSF_SC2_SA016_OB_ENV, DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26198
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	1.00 BSF_SC2_SA016_OB       Custom Rule SA016_OB
	1.50 BSF_SC2_SA016_OB_ENV   Custom Rule SA016_OB_ENV

On Mon, Jan 04, 2016 at 04:54:52PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> Rather than just being able to turn DAX on and off via a mount
> option, some applications may only want to enable DAX for certain
> performance critical files in a filesystem.
> 
> This patch introduces a new inode flag to enable DAX in the v3 inode
> di_flags2 field. It adds support for setting and clearing flags in
> the di_flags2 field via the XFS_IOC_FSSETXATTR ioctl, and sets the
> S_DAX inode flag appropriately when it is seen.
> 
> When this flag is set on a directory, it acts as an "inherit flag".
> That is, inodes created in the directory will automatically inherit
> the on-disk inode DAX flag, enabling administrators to set up
> directory heirarchies that automatically use DAX. Setting this flag
> on an empty root directory will make the entire filesystem use DAX
> by default.
> 
> Signed-off-by: Dave Chinner 

Looks good to me.

Reviewed-by: Theodore Ts'o 

							- Ted

From david@fromorbit.com  Sun Jan 17 13:49:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id D0A5B7F37
	for ; Sun, 17 Jan 2016 13:49:05 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 6DB05AC001
	for ; Sun, 17 Jan 2016 11:49:02 -0800 (PST)
X-ASG-Debug-ID: 1453060135-04cb6c75dc152130001-NocioJ
Received: from ipmail06.adl6.internode.on.net (ipmail06.adl6.internode.on.net [150.101.137.145]) by cuda.sgi.com with ESMTP id 8hZcxuf1BUi7uDkS; Sun, 17 Jan 2016 11:48:55 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.145
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2BmEgDw75tWPGwtLHleKAECgw9SbYhWn3YGi1yJRh6FawQCAoEaTQEBAQEBAQcBAQEBQT9BDgGDZQEBBDocIxAIAw4KCSUPBSUDBxoTiBq+RQELIRiFdYVHhEeEdgWXGoVIgnSFGoFnSoclhTSOXYUDKjSFDEqBSQEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl6.internode.on.net with ESMTP; 18 Jan 2016 06:18:54 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aKtJk-000214-OH; Mon, 18 Jan 2016 06:48:52 +1100
Date: Mon, 18 Jan 2016 06:48:52 +1100
From: Dave Chinner 
To: Stefan Priebe 
Cc: "" ,
	"xfs@oss.sgi.com" ,
	"xfs-masters@oss.sgi.com" 
Subject: Re: xfs deadlocks since upgrading to 4.1.15
Message-ID: <20160117194852.GR6033@dastard>
X-ASG-Orig-Subj: Re: xfs deadlocks since upgrading to 4.1.15
References: <5698B6EE.4000502@profihost.ag>
 <20160115225508.GQ6033@dastard>
 <569A972C.5060802@profihost.ag>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <569A972C.5060802@profihost.ag>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl6.internode.on.net[150.101.137.145]
X-Barracuda-Start-Time: 1453060135
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26220
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Sat, Jan 16, 2016 at 08:17:00PM +0100, Stefan Priebe wrote:
> Hi Dave,
> 
> Am 15.01.2016 um 23:55 schrieb Dave Chinner:
> >On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
> >>Hello,
> >>
> >>since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
> >>call traces show xfs.
> >>
> >>Attention all those are down top instead of top down (sorry for the
> >>wrong order).
> >
> >Can you repost them with line wrapping turned off, so they are
> >readable without causing beeding eye syndrome?
> i'm sorry. i was a bit optimistic that someone can read that.
> 
> Here are the traces in correct order, without timestamps but at
> pastebin which seems to be easier at least with thunderbird as it
> has no easy to use option to disable wrapping.
> 
> Nr 1:
> http://pastebin.com/raw/Fuxd86TP
> 
> Nr 2:
> http://pastebin.com/raw/Dp7mEcKi
> 
> Nr 3:
> http://pastebin.com/raw/BY3gUwzx

Commit 7a29ac4 ("xfs: give all workqueues rescuer threads") should
fix it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From masterdestroyy@gmail.com  Sun Jan 17 18:33:49 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: **
X-Spam-Status: No, score=2.2 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,
	MIME_HTML_ONLY,T_DKIM_INVALID autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 5335B7F37
	for ; Sun, 17 Jan 2016 18:33:49 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 25A068F8033
	for ; Sun, 17 Jan 2016 16:33:49 -0800 (PST)
X-ASG-Debug-ID: 1453077225-04bdf06f6c165c10001-NocioJ
Received: from mail-qk0-f193.google.com (mail-qk0-f193.google.com [209.85.220.193]) by cuda.sgi.com with ESMTP id UmkluKvHjB3V1aLO (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 17 Jan 2016 16:33:46 -0800 (PST)
X-Barracuda-Envelope-From: masterdestroyy@gmail.com
X-Barracuda-Apparent-Source-IP: 209.85.220.193
Received: by mail-qk0-f193.google.com with SMTP id s5so2730751qkd.2
        for ; Sun, 17 Jan 2016 16:33:46 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=from:subject:to:content-type:content-transfer-encoding:date
         :message-id:in-reply-to;
        bh=lHVno10qY4F1H0//4BkEOzB1XkO86V9rvu45zyNdRTQ=;
        b=iVn4HhhIUqubm45idHrIiMaRhCYxw3TjnztlXqUaZrVhTRRA9j+he3HfViUHBsXAoE
         5dCL4EmfbaIT0jEmPZV/eqCqwquoSTpzR/KCZLkulI5hOk61dvT3qQJXibtPsedmC9bg
         gYlePwTSVGZ51myKXi8thx6HKHJJOagslNeRkkkHNC46/51WjM1nJ6FqgjmCZZQ3f7UO
         b9kMQZ5q3JrALe4MySwUEQEe3Rj9qGkN5iD+uaPxO6QbsgaXyCdpY4LzYWSc5JBc6RX+
         +yk0GtCLkDnMeSZpBrB8au2VVOJZTDaAkxl1soSDr7sGiPYDuEwxfWpZRjl0f99OR5+f
         A4XA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:from:subject:to:content-type
         :content-transfer-encoding:date:message-id:in-reply-to;
        bh=lHVno10qY4F1H0//4BkEOzB1XkO86V9rvu45zyNdRTQ=;
        b=nOopwhO18xHZlOp6w+zEqrBxKiUKXHnVptB5PaTlaVlQOwKKJiXM/PGfyMSdzsDX83
         AZhhlQv7xMkIExuPnE1Ioz84Qia13l+GHgvo8hfQS7rbkWjgHlKw0OL63aR20is+hYSc
         7zMZKVjOxXRV9Fi53qh8KCVe0/uEeOdW4T914ftkp4gDPjPnpP1v118Ayw4M91KNes9P
         +7PYwrMG1huvMT8Ao1unCgVAScgMsA9msplWR2W4hP9Y1RRNwEMDS8TT3/5+6HQMtAKC
         U73sW5S8fhYn8iTelmBsXdPvLdL51nfwDTqZIEhqYc0MArKgIB0GZgEHPoLx8shy2Uu1
         5LHg==
X-Gm-Message-State: ALoCoQk3xUnCsaaXi2RwMiCNbR/OBKlgwV2wJNPbvTLlRsxGYYdzOhD01NdIKCRlZ9wlw7dz+CbgNbsZxCn3TGXAOd9uekOxXw==
X-Received: by 10.55.73.199 with SMTP id w190mr27456083qka.77.1453077225368;
        Sun, 17 Jan 2016 16:33:45 -0800 (PST)
Received: from DESKTOP-TTME6S7 (201.22.163.143.dynamic.adsl.gvt.net.br. [201.22.163.143])
        by smtp.gmail.com with ESMTPSA id z132sm9183674qka.14.2016.01.17.16.33.44
        for 
        (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
        Sun, 17 Jan 2016 16:33:44 -0800 (PST)
From: "Deploy Ai" 
Subject: Divulgue seus produtos na internet - 467535
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Divulgue seus produtos na internet - 467535
Content-Type: Text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Date: Sun, 17 Jan 2016 22:33:35 -0200
message-id: <390704057@localhost>
In-Reply-To: <390704057@localhost>
X-Barracuda-Connect: mail-qk0-f193.google.com[209.85.220.193]
X-Barracuda-Start-Time: 1453077226
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE, MIME_HTML_ONLY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26225
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	0.00 MIME_HTML_ONLY         BODY: Message only has text/html MIME parts
	0.00 HTML_MESSAGE           BODY: HTML included in message







Olá, que tal divulgar sua empresa para todo o 
Brasil?


Temos as ferramentas perfeitas para que você possa 
realizar marketing da sua empresa em Território Nacional no conforto da sua casa 
ou no escritório.


São elas:


    
  
  • 
  
    Atomic 
  Email (Envie 
  e-mails em massa usando suas contas de e-mails gratuitas ou pagas, suporta 
  texto ou HTML. Também disponível para Android): http://deployai.ddns.me/AtomicEmail


 



 



 



  
Como diz o ditado, Marketing é a alma do 
  negócio!


 


Att 
Deploy Ai



 




132663

From s.priebe@profihost.ag  Mon Jan 18 01:20:10 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id C650E7F37
	for ; Mon, 18 Jan 2016 01:20:10 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id A400B304032
	for ; Sun, 17 Jan 2016 23:20:10 -0800 (PST)
X-ASG-Debug-ID: 1453101606-04cbb05f7e1ff7b0001-NocioJ
Received: from mail-ph.de-nserver.de (mail-ph.de-nserver.de [85.158.179.214]) by cuda.sgi.com with ESMTP id L6vEZiCoSuNFtGtk (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sun, 17 Jan 2016 23:20:07 -0800 (PST)
X-Barracuda-Envelope-From: s.priebe@profihost.ag
X-Barracuda-Apparent-Source-IP: 85.158.179.214
Received: (qmail 28469 invoked from network); 18 Jan 2016 08:20:06 +0100
X-Fcrdns: No
Received: from phoffice.de-nserver.de (HELO [10.11.11.93]) (185.39.223.5)
  (smtp-auth username hostmaster@profihost.com, mechanism plain)
  by mail-ph.de-nserver.de (qpsmtpd/0.92) with (ECDHE-RSA-AES256-SHA encrypted) ESMTPSA; Mon, 18 Jan 2016 08:20:06 +0100
Subject: Re: xfs deadlocks since upgrading to 4.1.15
To: Dave Chinner 
X-ASG-Orig-Subj: Re: xfs deadlocks since upgrading to 4.1.15
References: <5698B6EE.4000502@profihost.ag> <20160115225508.GQ6033@dastard>
 <569A972C.5060802@profihost.ag> <20160117194852.GR6033@dastard>
Cc: "" ,
 "xfs@oss.sgi.com" ,
 "xfs-masters@oss.sgi.com" 
From: Stefan Priebe - Profihost AG 
Message-ID: <569C9225.6020408@profihost.ag>
Date: Mon, 18 Jan 2016 08:20:05 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160117194852.GR6033@dastard>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-User-Auth: Auth by hostmaster@profihost.com through 185.39.223.5
X-Barracuda-Connect: mail-ph.de-nserver.de[85.158.179.214]
X-Barracuda-Start-Time: 1453101607
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26232
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Am 17.01.2016 um 20:48 schrieb Dave Chinner:
> On Sat, Jan 16, 2016 at 08:17:00PM +0100, Stefan Priebe wrote:
>> Hi Dave,
>>
>> Am 15.01.2016 um 23:55 schrieb Dave Chinner:
>>> On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
>>>> Hello,
>>>>
>>>> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
>>>> call traces show xfs.
>>>>
>>>> Attention all those are down top instead of top down (sorry for the
>>>> wrong order).
>>>
>>> Can you repost them with line wrapping turned off, so they are
>>> readable without causing beeding eye syndrome?
>> i'm sorry. i was a bit optimistic that someone can read that.
>>
>> Here are the traces in correct order, without timestamps but at
>> pastebin which seems to be easier at least with thunderbird as it
>> has no easy to use option to disable wrapping.
>>
>> Nr 1:
>> http://pastebin.com/raw/Fuxd86TP
>>
>> Nr 2:
>> http://pastebin.com/raw/Dp7mEcKi
>>
>> Nr 3:
>> http://pastebin.com/raw/BY3gUwzx
> 
> Commit 7a29ac4 ("xfs: give all workqueues rescuer threads") should
> fix it.

Thx!

Greets,
Stefan

> Cheers,
> 
> Dave.
> 

From kumba@gentoo.org  Mon Jan 18 04:29:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id E26F77F37
	for ; Mon, 18 Jan 2016 04:29:05 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id CC4738F8033
	for ; Mon, 18 Jan 2016 02:29:02 -0800 (PST)
X-ASG-Debug-ID: 1453112939-04cbb05f7d203ba0001-NocioJ
Received: from resqmta-ch2-01v.sys.comcast.net (resqmta-ch2-01v.sys.comcast.net [69.252.207.33]) by cuda.sgi.com with ESMTP id GCyMhSohqDBQk1LE (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Mon, 18 Jan 2016 02:28:59 -0800 (PST)
X-Barracuda-Envelope-From: kumba@gentoo.org
X-Barracuda-Apparent-Source-IP: 69.252.207.33
Received: from resomta-ch2-10v.sys.comcast.net ([69.252.207.106])
	by resqmta-ch2-01v.sys.comcast.net with comcast
	id 7NUb1s0042JGN3p01NUzD9; Mon, 18 Jan 2016 10:28:59 +0000
Received: from [192.168.1.13] ([76.106.83.43])
	by resomta-ch2-10v.sys.comcast.net with comcast
	id 7NUy1s0060w5D3801NUyDt; Mon, 18 Jan 2016 10:28:59 +0000
To: xfs@oss.sgi.com
From: Joshua Kinard 
Subject: [PATCH 0/3] xfsprogs: misc uClibc patches
X-Enigmail-Draft-Status: N1110
X-ASG-Orig-Subj: [PATCH 0/3] xfsprogs: misc uClibc patches
Message-ID: <569CBE4F.6000306@gentoo.org>
Date: Mon, 18 Jan 2016 05:28:31 -0500
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101
 Thunderbird/42.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net;
	s=q20140121; t=1453112939;
	bh=9h/5ArSTgNkzntOZQ2UMjZoRnLvwYZ4/TJzWLj/g/CU=;
	h=Received:Received:To:From:Subject:Message-ID:Date:MIME-Version:
	 Content-Type;
	b=tTPq9w8pcgcjXQRF2FFs+a5vxZfkmOA60Ufll38By8DHiPCa0ulGeVaWfgXwMKDoa
	 bG7a5YxszppwD39vuLZnTtyFkrdbE9Ts1HZk9k5eW1aR/z4NmlNmpyMju12nqHzldX
	 TGO8tLFRDddh3M2sJnNXXkvOQ0Ev93skzXaHEGN9hJL6ZptUl5W+UFyZ2dvvZahHn7
	 HMJ6fx0Zq/oARv6Hqe0qRDPyWbOV5maVwu4ZHzmdJC0DOOs4YJ1dCRbB0ctIBgsqx6
	 mLTshmjA85qlRzY8+QigByHnXy5l1EVXPlvViuw9wPwHQO/gRfzYf20uhJZEVHO/wa
	 ki0/aBm2fmlhA==
X-Barracuda-Connect: resqmta-ch2-01v.sys.comcast.net[69.252.207.33]
X-Barracuda-Start-Time: 1453112939
X-Barracuda-Encrypted: DHE-RSA-AES128-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26235
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature


Hi,

I'm sending these three patches upstream that allow xfsprogs to compile and
work on uclibc-based userlands.  Two of them were sent upstream a few years ago
(2012), and one was rejected, but I don't know what the status was on the other.

Patches were contributed to Gentoo in our Bug #477758:
https://bugs.gentoo.org/show_bug.cgi?id=477758

xfsprogs-4.3.0-remove-aio-check.patch
Removes the configure check for aio.h, which is not provided by uClibc.  It
appears xfsprogs has no logic (that I can find) to compile against or utilize
AIO capabilities.

xfsprogs-4.3.0-use-sigprocmask.patch
uClibc doesn't provide the sigrelse or sighold calls, so this patch replaces
them with POSIX signal API calls (sigprocmask(2), etc).  This was previously
submitted in the below thread, but I don't see a status update on whether it
was accepted or rejected:
http://oss.sgi.com/archives/xfs/2012-03/msg00015.html

xfsprogs-4.3.0-use-statfs.patch
uClibc doesn't provide the ustat() call and (apparently) statfs() should be
used instead.  This patch was sent upstream before, but rejected because
xfsprogs apparently relies on ustat() reporting if a block device is mounted
anywhere.  I don't know if that's still the case now, but the attached patch
was re-written by another Gentoo user after the first one got rejected, so
resubmitting seems prudent.  The newer patch looks for mounted block devices by
checking mounts and/or mtab.

Thanks!

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

From kumba@gentoo.org  Mon Jan 18 04:29:27 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 615A87F37
	for ; Mon, 18 Jan 2016 04:29:27 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 07963AC004
	for ; Mon, 18 Jan 2016 02:29:26 -0800 (PST)
X-ASG-Debug-ID: 1453112959-04bdf06f6b170ce0001-NocioJ
Received: from resqmta-ch2-05v.sys.comcast.net (resqmta-ch2-05v.sys.comcast.net [69.252.207.37]) by cuda.sgi.com with ESMTP id 1HT7HOIXIC1E1Rrk (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Mon, 18 Jan 2016 02:29:20 -0800 (PST)
X-Barracuda-Envelope-From: kumba@gentoo.org
X-Barracuda-Apparent-Source-IP: 69.252.207.37
Received: from resomta-ch2-07v.sys.comcast.net ([69.252.207.103])
	by resqmta-ch2-05v.sys.comcast.net with comcast
	id 7NVK1s0022EPM3101NVKVL; Mon, 18 Jan 2016 10:29:19 +0000
Received: from [192.168.1.13] ([76.106.83.43])
	by resomta-ch2-07v.sys.comcast.net with comcast
	id 7NVK1s0010w5D3801NVKYL; Mon, 18 Jan 2016 10:29:19 +0000
From: Joshua Kinard 
Subject: [PATCH 1/3] xfsprogs: misc uClibc patches: remove aio.h check from
 configure
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 1/3] xfsprogs: misc uClibc patches: remove aio.h check from
 configure
X-Enigmail-Draft-Status: N1110
Message-ID: <569CBE64.6090603@gentoo.org>
Date: Mon, 18 Jan 2016 05:28:52 -0500
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101
 Thunderbird/42.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net;
	s=q20140121; t=1453112959;
	bh=/If7xFD84yw9Y4xMJpKh3P7/Aw0qseGPtSjgUNrbrt0=;
	h=Received:Received:From:Subject:To:Message-ID:Date:MIME-Version:
	 Content-Type;
	b=BmsCMxkIoAEoBMGkeho5yGKfUCLPLud4RHpIHIDHft5Lr/iBcrfX1JgzKZ3eOGf/t
	 Vk/7jNi0Dlbx1ZM/g4AcppfI9+KC27cVckhrHlZlMqF7V0wxiltMAXAkCCHIHhud13
	 vYqZ6r5l53qUuzwsiSy2C1V4zasFrfo2dyU2zDVMd2rEq0g43Og+n2qNckTVmvuQyu
	 F6bdQEoXob5Yw7WJGzkdLAz1zW6+dd3ZWhpI7u6/OKL+vvOofW2NkJhHD2wm6pOZoS
	 Y2Z6QyTuLrGXk/PBoyBk0BpygQeZP1qRKESmB5a+O0sS0utr2LesJVP2ZbuZ/La5+C
	 a3pAY3l6rgF8g==
X-Barracuda-Connect: resqmta-ch2-05v.sys.comcast.net[69.252.207.37]
X-Barracuda-Start-Time: 1453112960
X-Barracuda-Encrypted: DHE-RSA-AES128-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26235
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

From: Joshua Kinard 

Remove the configure check for aio.h, which is not provided by uClibc, and
which does not appear to be used by xfsprogs.

Refer to Gentoo Bug #477758:
https://bugs.gentoo.org/show_bug.cgi?id=477758

Signed-off-by: Joshua Kinard 
Reported-by: El Goretto 
Suggested-by: René Rhéaume 
---
 aclocal.m4           |    1 -
 configure.ac         |    6 +++---
 m4/Makefile          |    1 -
 m4/package_aiodev.m4 |   36 ------------------------------------
 4 files changed, 3 insertions(+), 41 deletions(-)

diff -Naurp xfsprogs-4.3.0.orig/aclocal.m4 xfsprogs-4.3.0/aclocal.m4
--- xfsprogs-4.3.0.orig/aclocal.m4	2015-11-23 04:30:01.000000000 +0000
+++ xfsprogs-4.3.0/aclocal.m4	2016-01-01 10:47:09.492229000 +0000
@@ -19,7 +19,6 @@ m4_include([m4/ltversion.m4])
 m4_include([m4/lt~obsolete.m4])
 m4_include([m4/manual_format.m4])
 m4_include([m4/multilib.m4])
-m4_include([m4/package_aiodev.m4])
 m4_include([m4/package_blkid.m4])
 m4_include([m4/package_globals.m4])
 m4_include([m4/package_libcdev.m4])
diff -Naurp xfsprogs-4.3.0.orig/configure.ac xfsprogs-4.3.0/configure.ac
--- xfsprogs-4.3.0.orig/configure.ac	2015-11-23 04:24:24.000000000 +0000
+++ xfsprogs-4.3.0/configure.ac	2016-01-01 10:47:09.502229000 +0000
@@ -56,6 +56,9 @@ AC_ARG_ENABLE(lib64,
 	enable_lib64=yes)
 AC_SUBST(enable_lib64)
 
+librt="-lrt"
+AC_SUBST(librt)
+
 #
 # If the user specified a libdir ending in lib64 do not append another
 # 64 to the library names.
@@ -101,9 +104,6 @@ AC_PACKAGE_GLOBALS(xfsprogs)
 AC_PACKAGE_UTILITIES(xfsprogs)
 AC_MULTILIB($enable_lib64)
 
-AC_PACKAGE_NEED_AIO_H
-AC_PACKAGE_NEED_LIO_LISTIO
-
 AC_PACKAGE_NEED_UUID_H
 AC_PACKAGE_NEED_UUIDCOMPARE
 
diff -Naurp xfsprogs-4.3.0.orig/m4/Makefile xfsprogs-4.3.0/m4/Makefile
--- xfsprogs-4.3.0.orig/m4/Makefile	2010-01-15 00:27:10.000000000 +0000
+++ xfsprogs-4.3.0/m4/Makefile	2016-01-01 10:47:09.502229000 +0000
@@ -14,7 +14,6 @@ CONFIGURE = \
 
 LSRCFILES = \
 	manual_format.m4 \
-	package_aiodev.m4 \
 	package_blkid.m4 \
 	package_globals.m4 \
 	package_libcdev.m4 \
diff -Naurp xfsprogs-4.3.0.orig/m4/package_aiodev.m4 xfsprogs-4.3.0/m4/package_aiodev.m4
--- xfsprogs-4.3.0.orig/m4/package_aiodev.m4	2010-06-24 22:23:27.000000000 +0000
+++ xfsprogs-4.3.0/m4/package_aiodev.m4	2016-01-01 10:47:09.502229000 +0000
@@ -1,37 +1 @@
-#
-# Check if we have a libaio.h installed
-#
-AC_DEFUN([AC_PACKAGE_WANT_AIO],
-  [ AC_CHECK_HEADERS(libaio.h, [ have_aio=true ], [ have_aio=false ])
-    AC_SUBST(have_aio)
-  ])
-
-#
-# Check if we have an aio.h installed
-#
-AC_DEFUN([AC_PACKAGE_NEED_AIO_H],
-  [ AC_CHECK_HEADERS(aio.h)
-    if test $ac_cv_header_aio_h = no; then
-	echo
-	echo 'FATAL ERROR: could not find a valid  header.'
-	exit 1
-    fi
-  ])
-
-#
-# Check if we have the lio_listio routine in either libc/librt
-#
-AC_DEFUN([AC_PACKAGE_NEED_LIO_LISTIO],
-  [ AC_CHECK_FUNCS(lio_listio)
-    if test $ac_cv_func_lio_listio = yes; then
-	librt=""
-    else
-	AC_CHECK_LIB(rt, lio_listio,, [
-	    echo
-	    echo 'FATAL ERROR: could not find a library with lio_listio.'
-	    exit 1],[-lpthread])
-	librt="-lrt"
-    fi
-    AC_SUBST(librt)
-  ])
 


From kumba@gentoo.org  Mon Jan 18 04:29:54 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 709A97F53
	for ; Mon, 18 Jan 2016 04:29:54 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 285088F8033
	for ; Mon, 18 Jan 2016 02:29:54 -0800 (PST)
X-ASG-Debug-ID: 1453112991-04bdf06f6c170d10001-NocioJ
Received: from resqmta-ch2-08v.sys.comcast.net (resqmta-ch2-08v.sys.comcast.net [69.252.207.40]) by cuda.sgi.com with ESMTP id 8VuEId5xbSGuPk5C (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Mon, 18 Jan 2016 02:29:52 -0800 (PST)
X-Barracuda-Envelope-From: kumba@gentoo.org
X-Barracuda-Apparent-Source-IP: 69.252.207.40
Received: from resomta-ch2-16v.sys.comcast.net ([69.252.207.112])
	by resqmta-ch2-08v.sys.comcast.net with comcast
	id 7NVr1s0022S2Q5R01NVrsR; Mon, 18 Jan 2016 10:29:51 +0000
Received: from [192.168.1.13] ([76.106.83.43])
	by resomta-ch2-16v.sys.comcast.net with comcast
	id 7NVr1s0020w5D3801NVrR3; Mon, 18 Jan 2016 10:29:51 +0000
From: Joshua Kinard 
Subject: [PATCH 2/3] xfsprogs: misc uClibc patches: use POSIX signal API
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 2/3] xfsprogs: misc uClibc patches: use POSIX signal API
X-Enigmail-Draft-Status: N1110
Message-ID: <569CBE85.4030800@gentoo.org>
Date: Mon, 18 Jan 2016 05:29:25 -0500
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101
 Thunderbird/42.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net;
	s=q20140121; t=1453112991;
	bh=SQrXF7V/gijXP41ISzmi0nfGmaGFNtPkMDDRl2LIYpg=;
	h=Received:Received:From:Subject:To:Message-ID:Date:MIME-Version:
	 Content-Type;
	b=QBaYLqaarCjDoccGj5sAEVsHFK9zbzrn4djRF2EuFoaYFJDifa5+5pxxmPR3u7Jm/
	 T9fHnlAWQ6QcxZ/TTaSsbCvLr7QerPHPvndDXeQp6NBq93KDE1l0Md04ronpKOgE9y
	 E5ycdd+KQljAT6rBxiegVPsYN5T2O2RzALJs0x4FDo1VFE8WrJeW7bWMapYRLEpuJb
	 DLYSUjWIT+PhV6Tvxpvh46ph9hd4VukVbY33G/iOatHCJPcWSTep1Crzs5UW5qC/rI
	 b8LC1TCHRoiC8+zxHphFgTKVnnPBjR9i1EqD4tDDd4EM9iz+/ThS5amfdfHc7MK5Ns
	 xXhQkYc/B8wDQ==
X-Barracuda-Connect: resqmta-ch2-08v.sys.comcast.net[69.252.207.40]
X-Barracuda-Start-Time: 1453112992
X-Barracuda-Encrypted: DHE-RSA-AES128-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26235
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

From: Joshua Kinard 

uClibc doesn't provide the sigrelse or sighold calls, so this patch replaces
them with POSIX signal API calls (sigprocmask(2), etc).

Refer to Gentoo Bug #477758:
https://bugs.gentoo.org/show_bug.cgi?id=477758

Signed-off-by: Joshua Kinard 
Suggested-by: René Rhéaume 
---
 copy/xfs_copy.c |   16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff -Naurp xfsprogs-4.3.0.orig/copy/xfs_copy.c xfsprogs-4.3.0/copy/xfs_copy.c
--- xfsprogs-4.3.0.orig/copy/xfs_copy.c	2015-10-15 21:31:26.000000000 +0000
+++ xfsprogs-4.3.0/copy/xfs_copy.c	2016-01-01 10:53:04.722229000 +0000
@@ -75,6 +75,16 @@ static int	format_logs(struct xfs_mount
 #define LAST	0x10		/* final message we print */
 
 void
+signal_maskfunc(int addset, int newset)
+{
+	sigset_t set;
+
+	sigemptyset(&set);
+	sigaddset(&set, addset);
+	sigprocmask(newset, &set, NULL);
+}
+
+void
 do_message(int flags, int code, const char *fmt, ...)
 {
 	va_list	ap;
@@ -477,9 +487,9 @@ write_wbuf(void)
 		if (target[i].state != INACTIVE)
 			pthread_mutex_unlock(&targ[i].wait);	/* wake up */
 
-	sigrelse(SIGCHLD);
+	signal_maskfunc(SIGCHLD, SIG_UNBLOCK);
 	pthread_mutex_lock(&mainwait);
-	sighold(SIGCHLD);
+	signal_maskfunc(SIGCHLD, SIG_BLOCK);
 }
 
 void
@@ -893,7 +903,7 @@ main(int argc, char **argv)
 	/* set up sigchild signal handler */
 
 	signal(SIGCHLD, handler);
-	sighold(SIGCHLD);
+	signal_maskfunc(SIGCHLD, SIG_BLOCK);
 
 	/* make children */
 


From kumba@gentoo.org  Mon Jan 18 04:30:16 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 238297F3F
	for ; Mon, 18 Jan 2016 04:30:16 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E3AAF8F8033
	for ; Mon, 18 Jan 2016 02:30:15 -0800 (PST)
X-ASG-Debug-ID: 1453113013-04bdf06f6e170d40001-NocioJ
Received: from resqmta-ch2-04v.sys.comcast.net (resqmta-ch2-04v.sys.comcast.net [69.252.207.36]) by cuda.sgi.com with ESMTP id u0cuBkCvF7eAgykx (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Mon, 18 Jan 2016 02:30:13 -0800 (PST)
X-Barracuda-Envelope-From: kumba@gentoo.org
X-Barracuda-Apparent-Source-IP: 69.252.207.36
Received: from resomta-ch2-16v.sys.comcast.net ([69.252.207.112])
	by resqmta-ch2-04v.sys.comcast.net with comcast
	id 7NWB1s00E2S2Q5R01NWBEV; Mon, 18 Jan 2016 10:30:11 +0000
Received: from [192.168.1.13] ([76.106.83.43])
	by resomta-ch2-16v.sys.comcast.net with comcast
	id 7NWB1s0070w5D3801NWBRm; Mon, 18 Jan 2016 10:30:11 +0000
From: Joshua Kinard 
Subject: [PATCH 3/3] xfsprogs: misc uClibc patches: use statfs() instead of
 ustat()
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 3/3] xfsprogs: misc uClibc patches: use statfs() instead of
 ustat()
X-Enigmail-Draft-Status: N1110
Message-ID: <569CBE99.4000309@gentoo.org>
Date: Mon, 18 Jan 2016 05:29:45 -0500
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101
 Thunderbird/42.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net;
	s=q20140121; t=1453113011;
	bh=f5hL9KVe3JDMFLk/fBB8oBFR40M4691qoPMZ9qBneuw=;
	h=Received:Received:From:Subject:To:Message-ID:Date:MIME-Version:
	 Content-Type;
	b=UxzM8oRK+6U9beoYpfy1TDmC8xw4vnxT4k3KaTbhXZbbuIBFW9AGyRnTVrIkfoScU
	 jokUmhOc7ly6Enps6gILeB8B7coxTvih+VQttcYzhPq43t9se5eWss7MemhvZXJf+s
	 ByF68hjAp1WdIMB8vesPov6eCSDwEX2YoODa+Z4/Kez3YfoxIEwXIdj2iUWZdxY2qs
	 cAQ+VsFOsS0bFmXR2DAECNzPczg9FS3U3dOYDTh9jbMuDfwVv9kk7obOyrMbwNeFss
	 Et850q60lkAj3xRy7MlQQTjYeyZYq9Tsnlp0EPklTKFx0YiKH6tqvTauygIN5ketyn
	 /woBb58Lr4sig==
X-Barracuda-Connect: resqmta-ch2-04v.sys.comcast.net[69.252.207.36]
X-Barracuda-Start-Time: 1453113013
X-Barracuda-Encrypted: DHE-RSA-AES128-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Barracuda-BRTS-Status: 1
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26235
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

From: Joshua Kinard 

uClibc doesn't provide the ustat() call and statfs() should be used instead.
Patch was previously sent upstream, but rejected because ustat() checks for the
block device being mounted anywhere.  This version of the patch takes care of
that check, as well as replacing ustat() with statfs().

Refer to Gentoo Bug #477758:
https://bugs.gentoo.org/show_bug.cgi?id=477758

Signed-off-by: Joshua Kinard 
Suggested-by: René Rhéaume 
---
 libxfs/linux.c |   35 +++++++++++++++++++++++++++++++----
 1 file changed, 31 insertions(+), 4 deletions(-)

diff -Naurp xfsprogs-4.3.0.orig/libxfs/linux.c xfsprogs-4.3.0/libxfs/linux.c
--- xfsprogs-4.3.0.orig/libxfs/linux.c	2015-08-03 00:39:42.000000000 +0000
+++ xfsprogs-4.3.0/libxfs/linux.c	2016-01-01 10:49:06.272229000 +0000
@@ -20,7 +20,7 @@
 #include 
 #include 
 #undef ustat
-#include 
+#include 
 #include 
 #include 
 #include 
@@ -51,9 +51,11 @@ static int max_block_alignment;
 int
 platform_check_ismounted(char *name, char *block, struct stat64 *s, int verbose)
 {
-	/* Pad ust; pre-2.6.28 linux copies out too much in 32bit compat mode */
-	struct ustat	ust[2];
+	struct statfs	ust;
 	struct stat64	st;
+	char		mounts[MAXPATHLEN];
+	FILE*		mtab;
+	struct mntent*	mnt;
 
 	if (!s) {
 		if (stat64(block, &st) < 0)
@@ -63,7 +65,32 @@ platform_check_ismounted(char *name, cha
 		s = &st;
 	}
 
-	if (ustat(s->st_rdev, ust) >= 0) {
+	if (strcmp(name, block) == 0) {
+		/* Device node was passed as parameter. Find its mount point */
+		strcpy(mounts, (!access(PROC_MOUNTED, R_OK)) ? PROC_MOUNTED : MOUNTED);
+		mtab = setmntent(mounts, "r");
+		if (mtab == NULL) {
+			if (verbose) {
+				fprintf(stderr, _("%s: %s contains a possibly mounted filesystem\n"), progname, name);
+			}
+			return 1;
+		}
+		else {
+			mnt = getmntent(mtab);
+			while (mnt != NULL) {
+				if (strcmp(block, mnt->mnt_fsname) == 0) {
+					if (verbose) {
+						 fprintf(stderr, _("%s: %s contains a mounted filesystem\n"), progname, name);
+					}
+					endmntent(mtab);
+					return 1;
+				}
+				mnt = getmntent(mtab);
+			}
+			endmntent(mtab);
+		}
+	}
+	else if (statfs(name, &ust) >= 0) {
 		if (verbose)
 			fprintf(stderr,
 				_("%s: %s contains a mounted filesystem\n"),


From jack@suse.cz  Mon Jan 18 04:32:55 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id DC2F67F47
	for ; Mon, 18 Jan 2016 04:32:54 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 6AEEAAC001
	for ; Mon, 18 Jan 2016 02:32:54 -0800 (PST)
X-ASG-Debug-ID: 1453113170-04cb6c75de1616e0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id Mvml221moesyIGrW (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 18 Jan 2016 02:32:51 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id E1D22ABB5;
	Mon, 18 Jan 2016 10:32:48 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 97E67823D8; Mon, 18 Jan 2016 11:33:02 +0100 (CET)
Date: Mon, 18 Jan 2016 11:33:02 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: Jan Kara , Christoph Hellwig ,
	fsdevel ,
	Eric Sandeen , xfs@oss.sgi.com
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
Message-ID: <20160118103302.GD6850@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org>
 <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net>
 <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net>
 <20160115093507.GA15950@quack.suse.cz>
 <56992CD4.6030408@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56992CD4.6030408@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453113171
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26235
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri 15-01-16 11:31:00, Eric Sandeen wrote:
> Hi Jan -
> 
> On 1/15/16 3:35 AM, Jan Kara wrote:
> > On Wed 13-01-16 16:40:58, Eric Sandeen wrote:
> >> On 1/11/16 10:28 AM, Jan Kara wrote:
> 
> ...
> 
> >>> Actually, what I want from you is just an interface which is usable for VFS
> >>> quotas as well since I'd like to avoid adding GETQUOTA2 quotactl shortly
> >>> after XGETQUOTA2 :).
> >>
> >> Actually, that's exactly what I thought would *need* to happen ... we already
> >> have this weird 15-year-old split-brain quota interface, so if xfs and ext4
> >> both need the same functionality, then we'd probably add both GETQUOTA2 and
> >> XGETQUOTA2.  If we were doing this all from scratch, sure, but adding a new
> >> handles-both-quota-types interface when every other operation is already split
> >> between the two almost seems to make matters worse.
> > 
> > Well, currently GETQUOTA and XGETQUOTA (and all the other quotactls) are
> > actually translated so they work regardless of the underlying filesystem.
> > So the only difference between XFS and VFS quotactls is in the formatting
> > of input/output structures. So from kernel POV it seems somewhat pointless
> > to add two calls doing the same thing and differing just in the formatting
> > of output - especially when we want the call to be extensible.
> > 
> > I agree that having a unified call means having a new structure for passing
> > dquot info between kernel and userspace. So just for adding that one small
> > feature you want it seems like an overkill. But when thinking about new
> > extensible getquota quotactl it IMHO makes sense to unify the VFS/XFS split
> > brain. Thoughts?
> 
> My first lazy/hacky thought is "how terrible would it be to overload the
> quotactl syscall return value with quota ID for Q_GETQUOTA2 calls?"

As you said, that won't quite work...

> For a purpose-built interface of "find the next ID" that wouldn't require any
> structure or interface changes...
> 
> We could name it Q_GETNEXTQUOTA / Q_XGETNEXTQUOTA to make it explicit about
> the purpose, and document that return behavior.  Done & done.  ;)
>
> A new grand unified extensible quota call sounds like a great idea, I just
> hate to gate this work on designing a brand-new interface.

OK, ok. I like Dave's proposal for quotactl2(). So let's leave the unification
for later and implent Q_GETNEXTQUOTA and Q_XGETNEXTQUOTA with the
functionality of your original Q_XGETQUOTA2. Having separate call to get
next ID would save us one new quotactl but OTOH we would need two syscalls
(and quota structure lookups) to report one structure and there are
potentially *lots* of them.

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From BATV+6a48a0219c47f280597b+4530+infradead.org+hch@bombadil.srs.infradead.org  Mon Jan 18 05:00:51 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 3E84B7F53
	for ; Mon, 18 Jan 2016 05:00:51 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 26488304032
	for ; Mon, 18 Jan 2016 03:00:51 -0800 (PST)
X-ASG-Debug-ID: 1453114848-04cb6c75dd162170001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id p89dehkkKYGTlEMI (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 18 Jan 2016 03:00:49 -0800 (PST)
X-Barracuda-Envelope-From: BATV+6a48a0219c47f280597b+4530+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aL7YE-0007vG-Hm; Mon, 18 Jan 2016 11:00:46 +0000
Date: Mon, 18 Jan 2016 03:00:46 -0800
From: Christoph Hellwig 
To: Jan Kara 
Cc: Eric Sandeen ,
	Christoph Hellwig ,
	fsdevel ,
	Eric Sandeen , xfs@oss.sgi.com
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
Message-ID: <20160118110046.GA25594@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org>
 <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net>
 <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net>
 <20160115093507.GA15950@quack.suse.cz>
 <56992CD4.6030408@sandeen.net>
 <20160118103302.GD6850@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160118103302.GD6850@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1453114849
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26235
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Mon, Jan 18, 2016 at 11:33:02AM +0100, Jan Kara wrote:
> OK, ok. I like Dave's proposal for quotactl2(). So let's leave the unification
> for later and implent Q_GETNEXTQUOTA and Q_XGETNEXTQUOTA with the
> functionality of your original Q_XGETQUOTA2. Having separate call to get
> next ID would save us one new quotactl but OTOH we would need two syscalls
> (and quota structure lookups) to report one structure and there are
> potentially *lots* of them.

Oh well.  Guess I should give in and just accept the original patch..

From BATV+6a48a0219c47f280597b+4530+infradead.org+hch@bombadil.srs.infradead.org  Mon Jan 18 05:11:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id B623A7F54
	for ; Mon, 18 Jan 2016 05:11:29 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 56B38AC005
	for ; Mon, 18 Jan 2016 03:11:26 -0800 (PST)
X-ASG-Debug-ID: 1453115483-04cb6c75db1625d0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id R98nM9dft9b4oaFV (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 18 Jan 2016 03:11:24 -0800 (PST)
X-Barracuda-Envelope-From: BATV+6a48a0219c47f280597b+4530+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aL7iV-0005fo-OP; Mon, 18 Jan 2016 11:11:23 +0000
Date: Mon, 18 Jan 2016 03:11:23 -0800
From: Christoph Hellwig 
To: Joshua Kinard 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/3] xfsprogs: misc uClibc patches: remove aio.h check
 from configure
Message-ID: <20160118111123.GA17495@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 1/3] xfsprogs: misc uClibc patches: remove aio.h check
 from configure
References: <569CBE64.6090603@gentoo.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <569CBE64.6090603@gentoo.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1453115484
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 2.60
X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC7_SA_HREF_FROM_MISMATCH_TEXT_URIx1_HL, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26235
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	2.50 BSF_SC7_SA_HREF_FROM_MISMATCH_TEXT_URIx1_HL Custom Rule
	                           HREF_FROM_MISMATCH_TEXT_URIx1_HL

On Mon, Jan 18, 2016 at 05:28:52AM -0500, Joshua Kinard wrote:
> From: Joshua Kinard 
> 
> Remove the configure check for aio.h, which is not provided by uClibc, and
> which does not appear to be used by xfsprogs.
> 
> Refer to Gentoo Bug #477758:
> https://bugs.gentoo.org/show_bug.cgi?id=477758

This looks fine:

Reviewed-by: Christoph Hellwig 

From BATV+6a48a0219c47f280597b+4530+infradead.org+hch@bombadil.srs.infradead.org  Mon Jan 18 05:17:15 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 84EE77F54
	for ; Mon, 18 Jan 2016 05:17:15 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 1F0A6AC00D
	for ; Mon, 18 Jan 2016 03:17:15 -0800 (PST)
X-ASG-Debug-ID: 1453115833-04cbb05f7f2050d0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 88hFkG09UDxVGxpV (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 18 Jan 2016 03:17:13 -0800 (PST)
X-Barracuda-Envelope-From: BATV+6a48a0219c47f280597b+4530+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aL7o9-00089o-0j; Mon, 18 Jan 2016 11:17:13 +0000
Date: Mon, 18 Jan 2016 03:17:12 -0800
From: Christoph Hellwig 
To: Joshua Kinard 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 3/3] xfsprogs: misc uClibc patches: use statfs() instead
 of ustat()
Message-ID: <20160118111712.GC17495@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfsprogs: misc uClibc patches: use statfs() instead
 of ustat()
References: <569CBE99.4000309@gentoo.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <569CBE99.4000309@gentoo.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1453115833
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26235
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

A similar but slightly cleaner patch was posted recently, but we'll
try to solve this in an even better although more invasive way.

From sandeen@sandeen.net  Mon Jan 18 09:18:27 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 48CCB7F5D
	for ; Mon, 18 Jan 2016 09:18:27 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 2A928304032
	for ; Mon, 18 Jan 2016 07:18:23 -0800 (PST)
X-ASG-Debug-ID: 1453130297-04cbb05f7d20d2c0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id Ew08AhUPNzbthcTI for ; Mon, 18 Jan 2016 07:18:17 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from Liberator.local (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 2255C63CBCE7;
	Mon, 18 Jan 2016 09:18:17 -0600 (CST)
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
To: Jan Kara 
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org> <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net> <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net> <20160115093507.GA15950@quack.suse.cz>
 <56992CD4.6030408@sandeen.net> <20160118103302.GD6850@quack.suse.cz>
Cc: Christoph Hellwig ,
 fsdevel , Eric Sandeen ,
 xfs@oss.sgi.com
From: Eric Sandeen 
Message-ID: <569D0238.5060407@sandeen.net>
Date: Mon, 18 Jan 2016 09:18:16 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160118103302.GD6850@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453130297
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26239
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On 1/18/16 4:33 AM, Jan Kara wrote:
> On Fri 15-01-16 11:31:00, Eric Sandeen wrote:

...

>> For a purpose-built interface of "find the next ID" that wouldn't require any
>> structure or interface changes...
>>
>> We could name it Q_GETNEXTQUOTA / Q_XGETNEXTQUOTA to make it explicit about
>> the purpose, and document that return behavior.  Done & done.  ;)
>>
>> A new grand unified extensible quota call sounds like a great idea, I just
>> hate to gate this work on designing a brand-new interface.
> 
> OK, ok. I like Dave's proposal for quotactl2(). So let's leave the unification
> for later and implent Q_GETNEXTQUOTA and Q_XGETNEXTQUOTA with the
> functionality of your original Q_XGETQUOTA2. Having separate call to get
> next ID would save us one new quotactl but OTOH we would need two syscalls
> (and quota structure lookups) to report one structure and there are
> potentially *lots* of them.

Ok, I can re-do it with the new Q_[X]GETNEXTQUOTA names, I've already done
the non-xfs one as well, just starting testing on that.

With or without the flags argument?

Thanks,
-Eric

From jack@suse.cz  Mon Jan 18 09:40:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 453947F5F
	for ; Mon, 18 Jan 2016 09:40:29 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id C0257AC001
	for ; Mon, 18 Jan 2016 07:40:28 -0800 (PST)
X-ASG-Debug-ID: 1453131625-04cb6c75de169cd0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 1BC2pdYAjPsO5VeG (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 18 Jan 2016 07:40:26 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 3ABAFAB9D;
	Mon, 18 Jan 2016 15:40:24 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 58F84823D8; Mon, 18 Jan 2016 16:40:38 +0100 (CET)
Date: Mon, 18 Jan 2016 16:40:38 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: Jan Kara , Christoph Hellwig ,
	fsdevel ,
	Eric Sandeen , xfs@oss.sgi.com
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
Message-ID: <20160118154038.GE6850@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org>
 <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net>
 <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net>
 <20160115093507.GA15950@quack.suse.cz>
 <56992CD4.6030408@sandeen.net>
 <20160118103302.GD6850@quack.suse.cz>
 <569D0238.5060407@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <569D0238.5060407@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453131625
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26240
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Mon 18-01-16 09:18:16, Eric Sandeen wrote:
> On 1/18/16 4:33 AM, Jan Kara wrote:
> > On Fri 15-01-16 11:31:00, Eric Sandeen wrote:
> 
> ...
> 
> >> For a purpose-built interface of "find the next ID" that wouldn't require any
> >> structure or interface changes...
> >>
> >> We could name it Q_GETNEXTQUOTA / Q_XGETNEXTQUOTA to make it explicit about
> >> the purpose, and document that return behavior.  Done & done.  ;)
> >>
> >> A new grand unified extensible quota call sounds like a great idea, I just
> >> hate to gate this work on designing a brand-new interface.
> > 
> > OK, ok. I like Dave's proposal for quotactl2(). So let's leave the unification
> > for later and implent Q_GETNEXTQUOTA and Q_XGETNEXTQUOTA with the
> > functionality of your original Q_XGETQUOTA2. Having separate call to get
> > next ID would save us one new quotactl but OTOH we would need two syscalls
> > (and quota structure lookups) to report one structure and there are
> > potentially *lots* of them.
> 
> Ok, I can re-do it with the new Q_[X]GETNEXTQUOTA names, I've already done
> the non-xfs one as well, just starting testing on that.
> 
> With or without the flags argument?

Without. For further extensibility I'd really go for the unified API in the
end.

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From BATV+6a48a0219c47f280597b+4530+infradead.org+hch@bombadil.srs.infradead.org  Mon Jan 18 12:16:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 526D17F37
	for ; Mon, 18 Jan 2016 12:16:05 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 42A3B8F8035
	for ; Mon, 18 Jan 2016 10:16:02 -0800 (PST)
X-ASG-Debug-ID: 1453140959-04cb6c75de16d490001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id vSpDaNmIqft5p16Q (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 18 Jan 2016 10:15:59 -0800 (PST)
X-Barracuda-Envelope-From: BATV+6a48a0219c47f280597b+4530+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aLELD-0003VG-JZ; Mon, 18 Jan 2016 18:15:47 +0000
Date: Mon, 18 Jan 2016 10:15:47 -0800
From: Christoph Hellwig 
To: mchristi@redhat.com
Cc: linux-f2fs-devel@lists.sourceforge.net, linux-ext4@vger.kernel.org,
	konrad.wilk@oracle.com, drbd-dev@lists.linbit.com,
	philipp.reisner@linbit.com, lars.ellenberg@linbit.com,
	linux-raid@vger.kernel.org, dm-devel@redhat.com,
	linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org,
	linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-scsi@vger.kernel.org, linux-mtd@lists.infradead.org,
	target-devel@vger.kernel.org, linux-btrfs@vger.kernel.org,
	osd-dev@open-osd.org, xfs@oss.sgi.com, ocfs2-devel@oss.oracle.com
Subject: Re: [PATCH 00/35 v3] eparate operations from flags in the
 bio/request structs
Message-ID: <20160118181547.GA30789@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH 00/35 v3] eparate operations from flags in the
 bio/request structs
References: <1452543693-4440-1-git-send-email-mchristi@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452543693-4440-1-git-send-email-mchristi@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1453140959
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26242
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

The whole sries looks fine to me:

Reviewed-by: Christoph Hellwig 

There are a few pieces of future work on top of this I'd rather see
sooner than later though:

 - use REQ_OP_FLUSH for all empty flushes, including those at the bio
   level.
 - merge the req->cmd_type values into req->op.  Right now that's
   basically only a new REQ_OP_BLOCK_PC (or better REQ_OP_SCSI while
   we're at it) and a REQ_OP_DRV_PRIV as the last value.
 - drop as many of the crazy defines based on REQ_* in fs.h (e.g.
   (READ_SYNC, WRITE_SYNC, WRITE_ODIRECT, WRITE_FLUSH, WRITE_FUA,
   WRITE_FLUSH_FUA, RW_MASK, RWA_MASK)

From sandeen@sandeen.net  Mon Jan 18 18:31:57 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 66F3E7F37
	for ; Mon, 18 Jan 2016 18:31:57 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 575CC8F8040
	for ; Mon, 18 Jan 2016 16:31:53 -0800 (PST)
X-ASG-Debug-ID: 1453163511-04cbb00afe048a0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id faCSfGJmv3iTtcz1 for ; Mon, 18 Jan 2016 16:31:51 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 7BE8C63737DD
	for ; Mon, 18 Jan 2016 18:31:51 -0600 (CST)
Subject: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH 0/4] quota: add new quotactl Q_XGETQUOTA2
References: <568FEA2C.6080708@redhat.com>
 <20160109072600.GA21636@infradead.org> <20160111132617.GD6262@quack.suse.cz>
 <5693D33A.5090307@sandeen.net> <20160111162807.GK6262@quack.suse.cz>
 <5696D27A.9070700@sandeen.net> <20160115093507.GA15950@quack.suse.cz>
 <56992CD4.6030408@sandeen.net> <20160118103302.GD6850@quack.suse.cz>
 <569D0238.5060407@sandeen.net> <20160118154038.GE6850@quack.suse.cz>
From: Eric Sandeen 
Message-ID: <569D83F6.6090300@sandeen.net>
Date: Mon, 18 Jan 2016 18:31:50 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160118154038.GE6850@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453163511
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26250
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------



On 1/18/16 9:40 AM, Jan Kara wrote:
> On Mon 18-01-16 09:18:16, Eric Sandeen wrote:
>> On 1/18/16 4:33 AM, Jan Kara wrote:
>>> On Fri 15-01-16 11:31:00, Eric Sandeen wrote:
>>
>> ...
>>
>>>> For a purpose-built interface of "find the next ID" that wouldn't require any
>>>> structure or interface changes...
>>>>
>>>> We could name it Q_GETNEXTQUOTA / Q_XGETNEXTQUOTA to make it explicit about
>>>> the purpose, and document that return behavior.  Done & done.  ;)
>>>>
>>>> A new grand unified extensible quota call sounds like a great idea, I just
>>>> hate to gate this work on designing a brand-new interface.
>>>
>>> OK, ok. I like Dave's proposal for quotactl2(). So let's leave the unification
>>> for later and implent Q_GETNEXTQUOTA and Q_XGETNEXTQUOTA with the
>>> functionality of your original Q_XGETQUOTA2. Having separate call to get
>>> next ID would save us one new quotactl but OTOH we would need two syscalls
>>> (and quota structure lookups) to report one structure and there are
>>> potentially *lots* of them.
>>
>> Ok, I can re-do it with the new Q_[X]GETNEXTQUOTA names, I've already done
>> the non-xfs one as well, just starting testing on that.
>>
>> With or without the flags argument?
> 
> Without. For further extensibility I'd really go for the unified API in the
> end.

Thanks.  I'll get new patches out in a bit, putting some polish on them and
the userspace, and seeing about a regression test as well.

Realized I need to think semi-carefully about what we do for IDs found on
disk but not in the user database; today I think extN will show everything;
XFS only shows IDs in the DB, unless an ID range is specified - probably
need to keep all behavior the same, without some explicit commandline switches
to do differently.

Thanks,
-Eric

From dave@fromorbit.com  Mon Jan 18 22:32:09 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id E8CF57F37
	for ; Mon, 18 Jan 2016 22:32:09 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D843F8F8052
	for ; Mon, 18 Jan 2016 20:32:09 -0800 (PST)
X-ASG-Debug-ID: 1453177921-04cbb00b0009bb0001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id E1fm8NgKu7Df7KJE for ; Mon, 18 Jan 2016 20:32:02 -0800 (PST)
X-Barracuda-Envelope-From: dave@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2AdCwChu51WPGwtLHleKAECgw+BP4ZegXigBgEBAQEBAQaRH4oSgTtNAQEBAQEBBwEBAQFBP4RiLxYKGxgYUgMHLRuHf58+oBCGDYpKE4QnBYdjA4cQiCSIPIcBjSEChW+IboIuAQsBNwMbgXEqNIZuAQEB
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 19 Jan 2016 15:02:00 +1030
Received: from disappointment.disaster.area ([192.168.1.110] helo=disappointment)
	by dastard with esmtp (Exim 4.80)
	(envelope-from )
	id 1aLNxX-00054q-9v
	for xfs@oss.sgi.com; Tue, 19 Jan 2016 15:31:59 +1100
Received: from dave by disappointment with local (Exim 4.86)
	(envelope-from )
	id 1aLNxX-0004ea-9H
	for xfs@oss.sgi.com; Tue, 19 Jan 2016 15:31:59 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH] [RFC] xfs: allocate log vector buffers outside CIL context lock
Date: Tue, 19 Jan 2016 15:31:59 +1100
X-ASG-Orig-Subj: [PATCH] [RFC] xfs: allocate log vector buffers outside CIL context lock
Message-Id: <1453177919-17849-1-git-send-email-david@fromorbit.com>
X-Mailer: git-send-email 2.5.0
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1453177921
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26256
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

One of the problems we currently have with delayed logging is that
under serious memory pressure we can deadlock memory reclaim. THis
occurs when memory reclaim (such as run by kswapd) is reclaiming XFS
inodes and issues a log force to unpin inodes that are dirty in the
CIL.

The CIL is pushed, but this will only occur once it gets the CIL
context lock to ensure that all committing transactions are complete
and no new transactions start being committed to the CIL while the
push switches to a new context.

The deadlock occurs when the CIL context lock is held by a
committing process that is doing memory allocation for log vector
buffers, and that allocation is then blocked on memory reclaim
making progress. Memory reclaim, however, is blocked waiting for
a log force to make progress, and so we effectively deadlock at this
point.

To solve this problem, we have to move the CIL log vector buffer
allocation outside of the context lock so that memory reclaim can
always make progress when it needs to force the log. The problem
with doing this is that a CIL push can take place while we are
determining if we need to allocate a new log vector buffer for
an item and hence the current log vector may go away without
warning. That means we canot rely on the existing log vector being
present when we finally grab the context lock and so we must have a
replacement buffer ready to go at all times.

To ensure this, introduce a "shadow log vector" buffer that is
always guaranteed to be present when we gain the CIL context lock
and format the item. This shadow buffer may or may not be used
during the formatting, but if the log item does not have an existing
log vector buffer or that buffer is too small for the new
modifications, we swap it for the new shadow buffer and format
the modifications into that new log vector buffer.

The result of this is that for any object we modify more than once
in a given CIL checkpoint, we double the memory required
to track dirty regions in the log. For single modifications then
we consume the shadow log vectorwe allocate on commit, and that gets
consumed by the checkpoint. However, if we make multiple
modifications, then the second transaction commit will allocate a
shadow log vector and hence we will end up with double the memory
usage as only one of the log vectors is consumed by the CIL
checkpoint. The remaining shadow vector will be freed when th elog
item is freed.

This can probably be optimised - access to the shadow log vector is
serialised by the object lock (as opposited to the active log
vector, which is controlled by the CIL context lock) and so we can
probably free shadow log vector from some objects when the log item
is marked clean on removal from the AIL.

The patch survives smoke testing and some load testing. I haven't
done any real performance testing, but I have done some load and low
memory testing and it hasn't exploded (perf did - it failed several
order 2 memory allocations, which XFS continued along just fine).

That said, I don't have a reliable deadlock reproducer in the first
place, so I'm interested i hearing what people think about this
approach to solve the problem and ways to test and improve it.

Signed-off-by: Dave Chinner 
---
 fs/xfs/xfs_buf_item.c     |   1 +
 fs/xfs/xfs_dquot.c        |   1 +
 fs/xfs/xfs_dquot_item.c   |   2 +
 fs/xfs/xfs_extfree_item.c |   2 +
 fs/xfs/xfs_inode_item.c   |   1 +
 fs/xfs/xfs_log_cil.c      | 258 +++++++++++++++++++++++++++++++++++-----------
 fs/xfs/xfs_trans.h        |   1 +
 7 files changed, 203 insertions(+), 63 deletions(-)

diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c
index 9220283..3f39d96 100644
--- a/fs/xfs/xfs_buf_item.c
+++ b/fs/xfs/xfs_buf_item.c
@@ -833,6 +833,7 @@ xfs_buf_item_free(
 	xfs_buf_log_item_t	*bip)
 {
 	xfs_buf_item_free_format(bip);
+	kmem_free(bip->bli_item.li_lv_shadow);
 	kmem_zone_free(xfs_buf_item_zone, bip);
 }
 
diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 9c44d38..4569cc4 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -74,6 +74,7 @@ xfs_qm_dqdestroy(
 {
 	ASSERT(list_empty(&dqp->q_lru));
 
+	kmem_free(dqp->q_logitem.qli_item.li_lv_shadow);
 	mutex_destroy(&dqp->q_qlock);
 
 	XFS_STATS_DEC(dqp->q_mount, xs_qm_dquot);
diff --git a/fs/xfs/xfs_dquot_item.c b/fs/xfs/xfs_dquot_item.c
index 814cff9..2c7a162 100644
--- a/fs/xfs/xfs_dquot_item.c
+++ b/fs/xfs/xfs_dquot_item.c
@@ -370,6 +370,8 @@ xfs_qm_qoffend_logitem_committed(
 	spin_lock(&ailp->xa_lock);
 	xfs_trans_ail_delete(ailp, &qfs->qql_item, SHUTDOWN_LOG_IO_ERROR);
 
+	kmem_free(qfs->qql_item.li_lv_shadow);
+	kmem_free(lip->li_lv_shadow);
 	kmem_free(qfs);
 	kmem_free(qfe);
 	return (xfs_lsn_t)-1;
diff --git a/fs/xfs/xfs_extfree_item.c b/fs/xfs/xfs_extfree_item.c
index 4aa0153..ab77946 100644
--- a/fs/xfs/xfs_extfree_item.c
+++ b/fs/xfs/xfs_extfree_item.c
@@ -40,6 +40,7 @@ void
 xfs_efi_item_free(
 	struct xfs_efi_log_item	*efip)
 {
+	kmem_free(efip->efi_item.li_lv_shadow);
 	if (efip->efi_format.efi_nextents > XFS_EFI_MAX_FAST_EXTENTS)
 		kmem_free(efip);
 	else
@@ -300,6 +301,7 @@ static inline struct xfs_efd_log_item *EFD_ITEM(struct xfs_log_item *lip)
 STATIC void
 xfs_efd_item_free(struct xfs_efd_log_item *efdp)
 {
+	kmem_free(efdp->efd_item.li_lv_shadow);
 	if (efdp->efd_format.efd_nextents > XFS_EFD_MAX_FAST_EXTENTS)
 		kmem_free(efdp);
 	else
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index bd9808f..45a882f 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -649,6 +649,7 @@ void
 xfs_inode_item_destroy(
 	xfs_inode_t	*ip)
 {
+	kmem_free(ip->i_itemp->ili_item.li_lv_shadow);
 	kmem_zone_free(xfs_ili_zone, ip->i_itemp);
 }
 
diff --git a/fs/xfs/xfs_log_cil.c b/fs/xfs/xfs_log_cil.c
index 4e76493..bde71e5 100644
--- a/fs/xfs/xfs_log_cil.c
+++ b/fs/xfs/xfs_log_cil.c
@@ -79,6 +79,144 @@ xlog_cil_init_post_recovery(
 	log->l_cilp->xc_ctx->sequence = 1;
 }
 
+static inline int
+xlog_cil_iovec_space(
+	uint	niovecs)
+{
+	return round_up((sizeof(struct xfs_log_vec) +
+					niovecs * sizeof(struct xfs_log_iovec)),
+			sizeof(uint64_t));
+}
+
+/*
+ * Allocate or pin log vector buffers for CIL insertion.
+ *
+ * The CIL currently uses disposable buffers for copying a snapshot of the
+ * modified items into the log during a push. The biggest problem with this is
+ * the requirement to allocate the disposable buffer during the commit if:
+ *	a) does not exist; or
+ *	b) it is too small
+ *
+ * If we do this allocation within xlog_cil_insert_format_items(), it is done
+ * under the xc_ctx_lock, which means that a CIL push cannot occur during
+ * the memory allocation. This means that we have a potential deadlock situation
+ * under low memory conditions when we have lots of dirty metadata pinned in
+ * the CIL and we need a CIL commit to occur to free memory.
+ *
+ * To avoid this, we need to move the memory allocation outside the
+ * xc_ctx_lock(), but because the log vector buffers are disposable, that opens
+ * up a TOCTOU race condition w.r.t. the CIL commiting and removing the log
+ * vector buffers between the check and the formatting of the item into the
+ * log vector buffer within the xc_ctx_lock.
+ *
+ * Because the log vector buffer needs to be unchanged during the CIL push
+ * process, we cannot share the buffer between the transaction commit (which
+ * modifies the buffer) and the CIL push context that is writing the changes
+ * into the log. This means skipping preallocation of buffer space is
+ * unreliable, but we most definitely do not want to be allocating and freeing
+ * buffers unnecessarily during commits when overwrites can be done safely.
+ *
+ * The simplest solution to this problem is to allocate a shadow buffer when a
+ * log item is committed for the second time, and then to only use this buffer
+ * if necessary. The buffer can remain attached to the log item until such time
+ * it is needed, and this is the buffer that is reallocated to match the size of
+ * the incoming modification. Then during the formatting of the item we can swap
+ * the active buffer with the new one if we can't reuse the existing buffer. We
+ * don't free the old buffer as it may be reused on the next modification if
+ * it's size is right, otherwise we'll free and reallocate it at that point.
+ *
+ * This function builds a vector for the changes in each log item in the
+ * transaction. It then works out the length of the buffer needed for each log
+ * item, allocates them and attaches the vector to the log item in preparation
+ * for the formatting step which occurs under the xc_ctx_lock.
+ *
+ * While this means the memory footprint goes up, it avoids the repeated
+ * alloc/free pattern that repeated modifications of an item would otherwise
+ * cause, and hence minimises the CPU overhead of such behaviour.
+ */
+static void
+xlog_cil_alloc_shadow_bufs(
+	struct xlog		*log,
+	struct xfs_trans	*tp)
+{
+	struct xfs_log_item_desc *lidp;
+
+	list_for_each_entry(lidp, &tp->t_items, lid_trans) {
+		struct xfs_log_item *lip = lidp->lid_item;
+		struct xfs_log_vec *lv;
+		int	niovecs = 0;
+		int	nbytes = 0;
+		int	buf_size;
+		bool	ordered = false;
+
+		/* Skip items which aren't dirty in this transaction. */
+		if (!(lidp->lid_flags & XFS_LID_DIRTY))
+			continue;
+
+		/* get number of vecs and size of data to be stored */
+		lip->li_ops->iop_size(lip, &niovecs, &nbytes);
+
+		/*
+		 * Ordered items need to be tracked but we do not wish to write
+		 * them. We need a logvec to track the object, but we do not
+		 * need an iovec or buffer to be allocated for copying data.
+		 */
+		if (niovecs == XFS_LOG_VEC_ORDERED) {
+			ordered = true;
+			niovecs = 0;
+			nbytes = 0;
+		}
+
+		/*
+		 * We 64-bit align the length of each iovec so that the start
+		 * of the next one is naturally aligned.  We'll need to
+		 * account for that slack space here. Then round nbytes up
+		 * to 64-bit alignment so that the initial buffer alignment is
+		 * easy to calculate and verify.
+		 */
+		nbytes += niovecs * sizeof(uint64_t);
+		nbytes = round_up(nbytes, sizeof(uint64_t));
+
+		/*
+		 * The data buffer needs to start 64-bit aligned, so round up
+		 * that space to ensure we can align it appropriately and not
+		 * overrun the buffer.
+		 */
+		buf_size = nbytes + xlog_cil_iovec_space(niovecs);
+
+		/*
+		 * if we have no shadow buffer, or it is too small, we need to
+		 * reallocate it.
+		 */
+		if (!lip->li_lv_shadow ||
+		    buf_size > lip->li_lv_shadow->lv_size) {
+
+			kmem_free(lip->li_lv_shadow);
+
+			lv = kmem_zalloc(buf_size, KM_SLEEP|KM_NOFS);
+			lv->lv_item = lip;
+			lv->lv_size = buf_size;
+			if (ordered)
+				lv->lv_buf_len = XFS_LOG_VEC_ORDERED;
+			else
+				lv->lv_iovecp = (struct xfs_log_iovec *)&lv[1];
+			lip->li_lv_shadow = lv;
+		} else {
+			/* same or smaller, optimise common overwrite case */
+			lv = lip->li_lv_shadow;
+			lv->lv_buf_len = 0;
+			lv->lv_bytes = 0;
+		}
+
+		/* Ensure the lv is set up according to ->iop_size */
+		lv->lv_niovecs = niovecs;
+
+		/* The allocated data region lies beyond the iovec region */
+		lv->lv_buf = (char *)lv + xlog_cil_iovec_space(niovecs);
+	}
+
+}
+
 /*
  * Prepare the log item for insertion into the CIL. Calculate the difference in
  * log space and vectors it will consume, and if it is a new item pin it as
@@ -101,16 +239,19 @@ xfs_cil_prepare_item(
 	/*
 	 * If there is no old LV, this is the first time we've seen the item in
 	 * this CIL context and so we need to pin it. If we are replacing the
-	 * old_lv, then remove the space it accounts for and free it.
+	 * old_lv, then remove the space it accounts for and make it the shadow
+	 * buffer for later freeing. In both cases we are now switching to the
+	 * shadow buffer, so update the the pointer to it appropriately.
 	 */
-	if (!old_lv)
+	if (!old_lv) {
 		lv->lv_item->li_ops->iop_pin(lv->lv_item);
-	else if (old_lv != lv) {
+		lv->lv_item->li_lv_shadow = NULL;
+	} else if (old_lv != lv) {
 		ASSERT(lv->lv_buf_len != XFS_LOG_VEC_ORDERED);
 
 		*diff_len -= old_lv->lv_bytes;
 		*diff_iovecs -= old_lv->lv_niovecs;
-		kmem_free(old_lv);
+		lv->lv_item->li_lv_shadow = old_lv;
 	}
 
 	/* attach new log vector to log item */
@@ -134,11 +275,13 @@ xfs_cil_prepare_item(
  * write it out asynchronously without needing to relock the object that was
  * modified at the time it gets written into the iclog.
  *
- * This function builds a vector for the changes in each log item in the
- * transaction. It then works out the length of the buffer needed for each log
- * item, allocates them and formats the vector for the item into the buffer.
- * The buffer is then attached to the log item are then inserted into the
- * Committed Item List for tracking until the next checkpoint is written out.
+ * This function takes the prepared log vectors attached to each log item, and
+ * formats the changes into the log vector buffer. The buffer it uses is
+ * dependent on the current state of the vector in the CIL - the shadow lv is
+ * guaranteed to be large enough for the current modification, but we will only
+ * use that if we can't reuse the existing lv. If we can't reuse the existing
+ * lv, then simple swap it out for the shadow lv. We don't free it - that is
+ * done lazily either by th enext modification or the freeing of the log item.
  *
  * We don't set up region headers during this process; we simply copy the
  * regions into the flat buffer. We can do this because we still have to do a
@@ -171,59 +314,44 @@ xlog_cil_insert_format_items(
 	list_for_each_entry(lidp, &tp->t_items, lid_trans) {
 		struct xfs_log_item *lip = lidp->lid_item;
 		struct xfs_log_vec *lv;
-		struct xfs_log_vec *old_lv;
-		int	niovecs = 0;
-		int	nbytes = 0;
-		int	buf_size;
+		struct xfs_log_vec *old_lv = NULL;
+		struct xfs_log_vec *shadow;
 		bool	ordered = false;
 
 		/* Skip items which aren't dirty in this transaction. */
 		if (!(lidp->lid_flags & XFS_LID_DIRTY))
 			continue;
 
-		/* get number of vecs and size of data to be stored */
-		lip->li_ops->iop_size(lip, &niovecs, &nbytes);
-
-		/* Skip items that do not have any vectors for writing */
-		if (!niovecs)
-			continue;
-
 		/*
-		 * Ordered items need to be tracked but we do not wish to write
-		 * them. We need a logvec to track the object, but we do not
-		 * need an iovec or buffer to be allocated for copying data.
+		 * The formatting size information is already attached to
+		 * the shadow lv on the log item.
 		 */
-		if (niovecs == XFS_LOG_VEC_ORDERED) {
+		shadow = lip->li_lv_shadow;
+		if (shadow->lv_buf_len == XFS_LOG_VEC_ORDERED)
 			ordered = true;
-			niovecs = 0;
-			nbytes = 0;
-		}
 
-		/*
-		 * We 64-bit align the length of each iovec so that the start
-		 * of the next one is naturally aligned.  We'll need to
-		 * account for that slack space here. Then round nbytes up
-		 * to 64-bit alignment so that the initial buffer alignment is
-		 * easy to calculate and verify.
-		 */
-		nbytes += niovecs * sizeof(uint64_t);
-		nbytes = round_up(nbytes, sizeof(uint64_t));
-
-		/* grab the old item if it exists for reservation accounting */
-		old_lv = lip->li_lv;
+		/* Skip items that do not have any vectors for writing */
+		if (!shadow->lv_niovecs && !ordered)
+			continue;
 
-		/*
-		 * The data buffer needs to start 64-bit aligned, so round up
-		 * that space to ensure we can align it appropriately and not
-		 * overrun the buffer.
-		 */
-		buf_size = nbytes +
-			   round_up((sizeof(struct xfs_log_vec) +
-				     niovecs * sizeof(struct xfs_log_iovec)),
-				    sizeof(uint64_t));
+/*
+		if ((lv = lip->li_lv))
+		xfs_warn(NULL,
+	"lv %p: next %p, size %d buflen %d niov %d bytes %d iovecp %p, buf %p",
+			lv, lv->lv_next, lv->lv_size, lv->lv_buf_len,
+			lv->lv_niovecs, lv->lv_bytes, lv->lv_iovecp,
+			lv->lv_buf);
+		if ((lv = lip->li_lv_shadow))
+		xfs_warn(NULL,
+	"shadow lv %p: next %p, size %d buflen %d niov %d bytes %d iovecp %p, buf %p",
+			lv, lv->lv_next, lv->lv_size, lv->lv_buf_len,
+			lv->lv_niovecs, lv->lv_bytes, lv->lv_iovecp,
+			lv->lv_buf);
+*/
 
 		/* compare to existing item size */
-		if (lip->li_lv && buf_size <= lip->li_lv->lv_size) {
+		old_lv = lip->li_lv;
+		if (lip->li_lv && shadow->lv_size <= lip->li_lv->lv_size) {
 			/* same or smaller, optimise common overwrite case */
 			lv = lip->li_lv;
 			lv->lv_next = NULL;
@@ -237,32 +365,29 @@ xlog_cil_insert_format_items(
 			 */
 			*diff_iovecs -= lv->lv_niovecs;
 			*diff_len -= lv->lv_bytes;
+
+			/* Ensure the lv is set up according to ->iop_size */
+			lv->lv_niovecs = shadow->lv_niovecs;
+
+			/* reset the lv buffer information for new formatting */
+			lv->lv_buf_len = 0;
+			lv->lv_bytes = 0;
+			lv->lv_buf = (char *)lv +
+					xlog_cil_iovec_space(lv->lv_niovecs);
 		} else {
-			/* allocate new data chunk */
-			lv = kmem_zalloc(buf_size, KM_SLEEP|KM_NOFS);
+			/* switch to shadow buffer! */
+			lv = shadow;
 			lv->lv_item = lip;
-			lv->lv_size = buf_size;
 			if (ordered) {
 				/* track as an ordered logvec */
 				ASSERT(lip->li_lv == NULL);
-				lv->lv_buf_len = XFS_LOG_VEC_ORDERED;
 				goto insert;
 			}
-			lv->lv_iovecp = (struct xfs_log_iovec *)&lv[1];
 		}
 
-		/* Ensure the lv is set up according to ->iop_size */
-		lv->lv_niovecs = niovecs;
-
-		/* The allocated data region lies beyond the iovec region */
-		lv->lv_buf_len = 0;
-		lv->lv_bytes = 0;
-		lv->lv_buf = (char *)lv + buf_size - nbytes;
 		ASSERT(IS_ALIGNED((unsigned long)lv->lv_buf, sizeof(uint64_t)));
-
 		lip->li_ops->iop_format(lip, lv);
 insert:
-		ASSERT(lv->lv_buf_len <= nbytes);
 		xfs_cil_prepare_item(log, lv, old_lv, diff_len, diff_iovecs);
 	}
 }
@@ -784,6 +909,13 @@ xfs_log_commit_cil(
 	struct xlog		*log = mp->m_log;
 	struct xfs_cil		*cil = log->l_cilp;
 
+	/*
+	 * Do all necessary memory allocation before we lock the CIL.
+	 * This ensures the allocation does not deadlock with a CIL
+	 * push in memory reclaim (e.g. from kswapd).
+	 */
+	xlog_cil_alloc_shadow_bufs(log, tp);
+
 	/* lock out background commit */
 	down_read(&cil->xc_ctx_lock);
 
diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h
index 4643070..74e6819 100644
--- a/fs/xfs/xfs_trans.h
+++ b/fs/xfs/xfs_trans.h
@@ -52,6 +52,7 @@ typedef struct xfs_log_item {
 	/* delayed logging */
 	struct list_head		li_cil;		/* CIL pointers */
 	struct xfs_log_vec		*li_lv;		/* active log vector */
+	struct xfs_log_vec		*li_lv_shadow;	/* standby vector */
 	xfs_lsn_t			li_seq;		/* CIL commit seq */
 } xfs_log_item_t;
 
-- 
2.5.0

From njgnyiul@ceqp.com  Tue Jan 19 03:45:21 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: ****
X-Spam-Status: No, score=4.9 required=5.0 tests=DC_IMAGE_SPAM_TEXT,DEAR_FRIEND,
	HTML_IMAGE_RATIO_02,HTML_MESSAGE autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 26B0D7F37
	for ; Tue, 19 Jan 2016 03:45:21 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 1358F304032
	for ; Tue, 19 Jan 2016 01:45:17 -0800 (PST)
X-ASG-Debug-ID: 1453196695-04cbb00aff104c0001-NocioJ
Received: from ceqp.com ([115.220.69.252]) by cuda.sgi.com with ESMTP id ZFEAjJt4G1fV5sUR for ; Tue, 19 Jan 2016 01:44:56 -0800 (PST)
X-Barracuda-Envelope-From: njgnyiul@ceqp.com
X-Barracuda-Apparent-Source-IP: 115.220.69.252
Received: from sunny-PC ([127.0.0.1]) by localhost via TCP with ESMTPA; Tue, 19 Jan 2016 17:48:13 +0800
Disposition-Notification-To: mrrcym@126.com
MIME-Version: 1.0
From: CoCo 
Sender: CoCo 
To: xfs@oss.sgi.com
Reply-To: CoCo 
Date: 19 Jan 2016 17:48:13 +0800
Subject: =?utf-8?B?VGluIGJveGVzLGNhbi50aW5wbGF0ZSAgdG8geGZzQG9zcy5zZ2kuY29t?=
Content-Type: multipart/mixed; boundary=--boundary_997_2870cf01-1463-4ef9-9d5d-023701733bf7
X-ASG-Orig-Subj: =?utf-8?B?VGluIGJveGVzLGNhbi50aW5wbGF0ZSAgdG8geGZzQG9zcy5zZ2kuY29t?=
X-Barracuda-Connect: UNKNOWN[115.220.69.252]
X-Barracuda-Start-Time: 1453196696
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 2.29
X-Barracuda-Spam-Status: No, SCORE=2.29 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, DC_IMAGE_SPAM_TEXT, DEAR_FRIEND, HTML_IMAGE_RATIO_02, HTML_MESSAGE, MISSING_MID, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26261
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.14 MISSING_MID            Missing Message-Id: header
	1.00 DEAR_FRIEND            BODY: Dear Friend? That's not very dear!
	0.55 HTML_IMAGE_RATIO_02    BODY: HTML has a low ratio of text to image area
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.00 DC_IMAGE_SPAM_TEXT     Possible Image-only spam with little text
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963
Message-Id: <20160119094517.AF400106C0A0@cuda.sgi.com>


----boundary_997_2870cf01-1463-4ef9-9d5d-023701733bf7
Content-Type: multipart/alternative; boundary=--boundary_996_99e233a1-ff66-4b8f-8f0c-968ac97f691b


----boundary_996_99e233a1-ff66-4b8f-8f0c-968ac97f691b
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64

DQoNCkRlYXIgRnJpZW5kLA0KDQombmJzcDsNCg0KJm5ic3A7DQoNCiZuYnNwOw0KDQombmJz
cDsNCg0KJm5ic3A7DQoNCiZuYnNwOw0KDQombmJzcDsNCg0KJm5ic3A7DQoNCiZuYnNwOw0K
DQoNCg0KDQoNCiZuYnNwOw0KT3VyIGZhY3RvcnkgcHJvZHVjZSBhbGwga2luZHMgb2YgdGlu
IGJveGVzICxjYW4gKHRlYSB0aW4sY29mZmVlIHRpbixjb29raWUgdGluLGdpZnQgdGluICxl
Y3QpJm5ic3A7Lg0KJm5ic3A7DQoNCg0KJm5ic3A7DQpJZiBuZWVkLGNhdGFsb2d1ZSBjYW4g
YmUgc2VudCBmb3ImbmJzcDsgeW91ciByZWZlcmVuY2UgLg0KJm5ic3A7DQoNCkJlc3QgcmVn
YXJkcw0KDQombmJzcDsNCiZuYnNwOw0KDQombmJzcDsNCg0KDQoNCg0KSWYgZGlzdHVyYiAs
cGxzIHVuc3Vic2NyaWJlDQombmJzcDsNCiZuYnNwOw==
----boundary_996_99e233a1-ff66-4b8f-8f0c-968ac97f691b
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64

PERJViBzdHlsZT0iRk9OVC1GQU1JTFk6IGFyaWFsIj4NCjxESVYgc3R5bGU9IkZPTlQtRkFN
SUxZOiBhcmlhbCI+DQo8RElWIHN0eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsgQkFDS0dS
T1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsgRk9OVC1WQVJJQU5UOiBub3JtYWw7IEZP
TlQtU1RZTEU6IG5vcm1hbDsgVEVYVC1JTkRFTlQ6IDBweDsgRElTUExBWTogaW5saW5lOyBG
T05ULUZBTUlMWTogQ2FsaWJyaTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsgTEVUVEVSLVNQQUNJ
Tkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IEZPTlQtU0laRTogc21hbGw7IEZPTlQt
V0VJR0hUOiBub3JtYWw7IFdPUkQtU1BBQ0lORzogMHB4OyBURVhULURFQ09SQVRJT046IG5v
bmU7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PEZPTlQgZmFjZT0iVGltZXMg
TmV3IFJvbWFuIj5EZWFyIEZyaWVuZCw8L0ZPTlQ+PC9ESVY+PC9ESVY+DQo8RElWIHN0eWxl
PSJGT05ULUZBTUlMWTogYXJpYWwiPg0KPERJViBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5v
bmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IEZPTlQtVkFSSUFOVDog
bm9ybWFsOyBGT05ULVNUWUxFOiBub3JtYWw7IFRFWFQtSU5ERU5UOiAwcHg7IERJU1BMQVk6
IGlubGluZTsgRk9OVC1GQU1JTFk6IENhbGlicmk7IFdISVRFLVNQQUNFOiBub3JtYWw7IExF
VFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBGT05ULVNJWkU6IHNt
YWxsOyBGT05ULVdFSUdIVDogbm9ybWFsOyBXT1JELVNQQUNJTkc6IDBweDsgVEVYVC1ERUNP
UkFUSU9OOiBub25lOyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPjxGT05UIGZh
Y2U9IlRpbWVzIE5ldyBSb21hbiI+PC9GT05UPiZuYnNwOzwvRElWPjwvRElWPg0KPERJViBz
dHlsZT0iRk9OVC1GQU1JTFk6IGFyaWFsIj4NCjxESVYgc3R5bGU9IlRFWFQtVFJBTlNGT1JN
OiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUpOyBGT05ULVZBUklB
TlQ6IG5vcm1hbDsgRk9OVC1TVFlMRTogbm9ybWFsOyBURVhULUlOREVOVDogMHB4OyBESVNQ
TEFZOiBpbmxpbmU7IEZPTlQtRkFNSUxZOiBDYWxpYnJpOyBXSElURS1TUEFDRTogbm9ybWFs
OyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCwwKTsgRk9OVC1TSVpF
OiBzbWFsbDsgRk9OVC1XRUlHSFQ6IG5vcm1hbDsgV09SRC1TUEFDSU5HOiAwcHg7IFRFWFQt
REVDT1JBVElPTjogbm9uZTsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4Ij48Rk9O
VCBmYWNlPSJUaW1lcyBOZXcgUm9tYW4iPjwvRk9OVD4mbmJzcDs8L0RJVj48L0RJVj4NCjxE
SVYgc3R5bGU9IkZPTlQtRkFNSUxZOiBhcmlhbCI+DQo8RElWIHN0eWxlPSJURVhULVRSQU5T
Rk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwyNTUsMjU1KTsgRk9OVC1W
QVJJQU5UOiBub3JtYWw7IEZPTlQtU1RZTEU6IG5vcm1hbDsgVEVYVC1JTkRFTlQ6IDBweDsg
RElTUExBWTogaW5saW5lOyBGT05ULUZBTUlMWTogQ2FsaWJyaTsgV0hJVEUtU1BBQ0U6IG5v
cm1hbDsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigwLDAsMCk7IEZPTlQt
U0laRTogc21hbGw7IEZPTlQtV0VJR0hUOiBub3JtYWw7IFdPUkQtU1BBQ0lORzogMHB4OyBU
RVhULURFQ09SQVRJT046IG5vbmU7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweCI+
PEZPTlQgZmFjZT0iVGltZXMgTmV3IFJvbWFuIj48L0ZPTlQ+Jm5ic3A7PC9ESVY+PC9ESVY+
DQo8RElWIHN0eWxlPSJGT05ULUZBTUlMWTogYXJpYWwiPg0KPERJViBzdHlsZT0iVEVYVC1U
UkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUsMjU1LDI1NSk7IEZP
TlQtVkFSSUFOVDogbm9ybWFsOyBGT05ULVNUWUxFOiBub3JtYWw7IFRFWFQtSU5ERU5UOiAw
cHg7IERJU1BMQVk6IGlubGluZTsgRk9OVC1GQU1JTFk6IENhbGlicmk7IFdISVRFLVNQQUNF
OiBub3JtYWw7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiByZ2IoMCwwLDApOyBG
T05ULVNJWkU6IHNtYWxsOyBGT05ULVdFSUdIVDogbm9ybWFsOyBXT1JELVNQQUNJTkc6IDBw
eDsgVEVYVC1ERUNPUkFUSU9OOiBub25lOyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAw
cHgiPjxGT05UIGZhY2U9IlRpbWVzIE5ldyBSb21hbiI+PC9GT05UPiZuYnNwOzwvRElWPjwv
RElWPg0KPERJViBzdHlsZT0iRk9OVC1GQU1JTFk6IGFyaWFsIj4NCjxESVYgc3R5bGU9IlRF
WFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2IoMjU1LDI1NSwyNTUp
OyBGT05ULVZBUklBTlQ6IG5vcm1hbDsgRk9OVC1TVFlMRTogbm9ybWFsOyBURVhULUlOREVO
VDogMHB4OyBESVNQTEFZOiBpbmxpbmU7IEZPTlQtRkFNSUxZOiBDYWxpYnJpOyBXSElURS1T
UEFDRTogbm9ybWFsOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xPUjogcmdiKDAsMCww
KTsgRk9OVC1TSVpFOiBzbWFsbDsgRk9OVC1XRUlHSFQ6IG5vcm1hbDsgV09SRC1TUEFDSU5H
OiAwcHg7IFRFWFQtREVDT1JBVElPTjogbm9uZTsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0
aDogMHB4Ij48Rk9OVCBmYWNlPSJUaW1lcyBOZXcgUm9tYW4iPjwvRk9OVD4mbmJzcDs8L0RJ
Vj48L0RJVj4NCjxESVYgc3R5bGU9IkZPTlQtRkFNSUxZOiBhcmlhbCI+DQo8RElWIHN0eWxl
PSJURVhULVRSQU5TRk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjogcmdiKDI1NSwyNTUs
MjU1KTsgRk9OVC1WQVJJQU5UOiBub3JtYWw7IEZPTlQtU1RZTEU6IG5vcm1hbDsgVEVYVC1J
TkRFTlQ6IDBweDsgRElTUExBWTogaW5saW5lOyBGT05ULUZBTUlMWTogQ2FsaWJyaTsgV0hJ
VEUtU1BBQ0U6IG5vcm1hbDsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsgQ09MT1I6IHJnYigw
LDAsMCk7IEZPTlQtU0laRTogc21hbGw7IEZPTlQtV0VJR0hUOiBub3JtYWw7IFdPUkQtU1BB
Q0lORzogMHB4OyBURVhULURFQ09SQVRJT046IG5vbmU7IC13ZWJraXQtdGV4dC1zdHJva2Ut
d2lkdGg6IDBweCI+PEZPTlQgZmFjZT0iVGltZXMgTmV3IFJvbWFuIj48L0ZPTlQ+Jm5ic3A7
PC9ESVY+PC9ESVY+DQo8RElWIHN0eWxlPSJGT05ULUZBTUlMWTogYXJpYWwiPg0KPERJViBz
dHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09MT1I6IHJnYigyNTUs
MjU1LDI1NSk7IEZPTlQtVkFSSUFOVDogbm9ybWFsOyBGT05ULVNUWUxFOiBub3JtYWw7IFRF
WFQtSU5ERU5UOiAwcHg7IERJU1BMQVk6IGlubGluZTsgRk9OVC1GQU1JTFk6IENhbGlicmk7
IFdISVRFLVNQQUNFOiBub3JtYWw7IExFVFRFUi1TUEFDSU5HOiBub3JtYWw7IENPTE9SOiBy
Z2IoMCwwLDApOyBGT05ULVNJWkU6IHNtYWxsOyBGT05ULVdFSUdIVDogbm9ybWFsOyBXT1JE
LVNQQUNJTkc6IDBweDsgVEVYVC1ERUNPUkFUSU9OOiBub25lOyAtd2Via2l0LXRleHQtc3Ry
b2tlLXdpZHRoOiAwcHgiPjxGT05UIGZhY2U9IlRpbWVzIE5ldyBSb21hbiI+PC9GT05UPiZu
YnNwOzwvRElWPjwvRElWPg0KPERJViBzdHlsZT0iRk9OVC1GQU1JTFk6IGFyaWFsIj4NCjxE
SVYgc3R5bGU9IlRFWFQtVFJBTlNGT1JNOiBub25lOyBCQUNLR1JPVU5ELUNPTE9SOiByZ2Io
MjU1LDI1NSwyNTUpOyBGT05ULVZBUklBTlQ6IG5vcm1hbDsgRk9OVC1TVFlMRTogbm9ybWFs
OyBURVhULUlOREVOVDogMHB4OyBESVNQTEFZOiBpbmxpbmU7IEZPTlQtRkFNSUxZOiBDYWxp
YnJpOyBXSElURS1TUEFDRTogbm9ybWFsOyBMRVRURVItU1BBQ0lORzogbm9ybWFsOyBDT0xP
UjogcmdiKDAsMCwwKTsgRk9OVC1TSVpFOiBzbWFsbDsgRk9OVC1XRUlHSFQ6IG5vcm1hbDsg
V09SRC1TUEFDSU5HOiAwcHg7IFRFWFQtREVDT1JBVElPTjogbm9uZTsgLXdlYmtpdC10ZXh0
LXN0cm9rZS13aWR0aDogMHB4Ij48Rk9OVCBmYWNlPSJUaW1lcyBOZXcgUm9tYW4iPjwvRk9O
VD4mbmJzcDs8L0RJVj48L0RJVj4NCjxESVYgc3R5bGU9IkZPTlQtRkFNSUxZOiBhcmlhbCI+
DQo8RElWIHN0eWxlPSJURVhULVRSQU5TRk9STTogbm9uZTsgQkFDS0dST1VORC1DT0xPUjog
cmdiKDI1NSwyNTUsMjU1KTsgRk9OVC1WQVJJQU5UOiBub3JtYWw7IEZPTlQtU1RZTEU6IG5v
cm1hbDsgVEVYVC1JTkRFTlQ6IDBweDsgRElTUExBWTogaW5saW5lOyBGT05ULUZBTUlMWTog
Q2FsaWJyaTsgV0hJVEUtU1BBQ0U6IG5vcm1hbDsgTEVUVEVSLVNQQUNJTkc6IG5vcm1hbDsg
Q09MT1I6IHJnYigwLDAsMCk7IEZPTlQtU0laRTogc21hbGw7IEZPTlQtV0VJR0hUOiBub3Jt
YWw7IFdPUkQtU1BBQ0lORzogMHB4OyBURVhULURFQ09SQVRJT046IG5vbmU7IC13ZWJraXQt
dGV4dC1zdHJva2Utd2lkdGg6IDBweCI+PEZPTlQgZmFjZT0iVGltZXMgTmV3IFJvbWFuIj48
L0ZPTlQ+Jm5ic3A7PC9ESVY+PC9ESVY+DQo8RElWIHN0eWxlPSJGT05ULUZBTUlMWTogYXJp
YWwiPg0KPERJViBzdHlsZT0iVEVYVC1UUkFOU0ZPUk06IG5vbmU7IEJBQ0tHUk9VTkQtQ09M
T1I6IHJnYigyNTUsMjU1LDI1NSk7IEZPTlQtVkFSSUFOVDogbm9ybWFsOyBGT05ULVNUWUxF
OiBub3JtYWw7IFRFWFQtSU5ERU5UOiAwcHg7IERJU1BMQVk6IGlubGluZTsgRk9OVC1GQU1J
TFk6IENhbGlicmk7IFdISVRFLVNQQUNFOiBub3JtYWw7IExFVFRFUi1TUEFDSU5HOiBub3Jt
YWw7IENPTE9SOiByZ2IoMCwwLDApOyBGT05ULVNJWkU6IHNtYWxsOyBGT05ULVdFSUdIVDog
bm9ybWFsOyBXT1JELVNQQUNJTkc6IDBweDsgVEVYVC1ERUNPUkFUSU9OOiBub25lOyAtd2Vi
a2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHgiPjwvRElWPg0KPERJVj4NCjxESVYgc3R5bGU9
IkZPTlQtU0laRTogMTYuNXB0Ij4NCjxESVYgc3R5bGU9IlBPU0lUSU9OOiBzdGF0aWMgIWlt
cG9ydGFudDsgRk9OVC1TSVpFOiAyMi41cHQiPg0KPERJViBzdHlsZT0iRk9OVC1GQU1JTFk6
IGFyaWFsOyBGT05ULVNJWkU6IDEycHQiPjxGT05UIGZhY2U9IlRpbWVzIE5ldyBSb21hbiI+
PC9GT05UPiZuYnNwOzwvRElWPg0KPERJViBzdHlsZT0iRk9OVC1GQU1JTFk6IGFyaWFsOyBG
T05ULVNJWkU6IDEycHQiPjxGT05UIGZhY2U9IlRpbWVzIE5ldyBSb21hbiI+T3VyIGZhY3Rv
cnkgcHJvZHVjZSBhbGwga2luZHMgb2YgPEZPTlQgc3R5bGU9IkJBQ0tHUk9VTkQtQ09MT1I6
ICNmZmZmMDAiPnRpbiBib3hlcyAsY2FuICh0ZWEgdGluLGNvZmZlZSB0aW4sY29va2llIHRp
bixnaWZ0IHRpbiA8L0ZPTlQ+PEZPTlQgc3R5bGU9IkJBQ0tHUk9VTkQtQ09MT1I6ICNmZmZm
ZmYiPixlY3QpJm5ic3A7LjwvRk9OVD48L0ZPTlQ+PC9ESVY+DQo8RElWIHN0eWxlPSJGT05U
LUZBTUlMWTogYXJpYWw7IEZPTlQtU0laRTogMTJwdCI+Jm5ic3A7PC9ESVY+DQo8RElWIHN0
eWxlPSJGT05ULUZBTUlMWTogYXJpYWw7IEZPTlQtU0laRTogMTJwdCI+PEZPTlQgZmFjZT0i
VGltZXMgTmV3IFJvbWFuIj48SU1HIHN0eWxlPSJXSURUSDogNjQ3cHg7IEhFSUdIVDogMjQ2
cHgiIHNyYz0iY2lkOjBAbG9jYWxob3N0IiB3aWR0aD03NjIgaGVpZ2h0PTI1Nj48L0ZPTlQ+
PC9ESVY+DQo8RElWIHN0eWxlPSJGT05ULUZBTUlMWTogYXJpYWw7IEZPTlQtU0laRTogMTJw
dCI+DQo8RElWIHN0eWxlPSJGT05ULUZBTUlMWTogYXJpYWw7IEZPTlQtU0laRTogMTJwdCI+
PEZPTlQgZmFjZT0iVGltZXMgTmV3IFJvbWFuIj48L0ZPTlQ+Jm5ic3A7PC9ESVY+DQo8RElW
IHN0eWxlPSJGT05ULUZBTUlMWTogYXJpYWw7IEZPTlQtU0laRTogMTJwdCI+PEZPTlQgZmFj
ZT0iVGltZXMgTmV3IFJvbWFuIj5JZiBuZWVkLGNhdGFsb2d1ZSBjYW4gYmUgc2VudCBmb3Im
bmJzcDsgeW91ciByZWZlcmVuY2UgLjwvRk9OVD48L0RJVj4NCjxESVYgc3R5bGU9IkZPTlQt
RkFNSUxZOiBhcmlhbDsgRk9OVC1TSVpFOiAxMnB0Ij48Rk9OVCBmYWNlPSJUaW1lcyBOZXcg
Um9tYW4iPjwvRk9OVD4mbmJzcDs8L0RJVj48L0RJVj4NCjxESVYgc3R5bGU9IkZPTlQtRkFN
SUxZOiBhcmlhbDsgRk9OVC1TSVpFOiAxMnB0Ij48Rk9OVCBmYWNlPSJUaW1lcyBOZXcgUm9t
YW4iPjxJTUcgc3JjPSJjaWQ6MUBsb2NhbGhvc3QiPjwvRk9OVD48L0RJVj4NCjxESVYgc3R5
bGU9IkZPTlQtRkFNSUxZOiBhcmlhbDsgRk9OVC1TSVpFOiAxMnB0Ij48Rk9OVCBmYWNlPSJU
aW1lcyBOZXcgUm9tYW4iPkJlc3QgcmVnYXJkczwvRk9OVD48L0RJVj4NCjxESVYgc3R5bGU9
IkZPTlQtRkFNSUxZOiBhcmlhbDsgRk9OVC1TSVpFOiAxMnB0Ij48IS0tU3RhcnRGcmFnbWVu
dCAtLT4NCjxESVY+Jm5ic3A7PC9ESVY+PC9ESVY+DQo8RElWIHN0eWxlPSJGT05ULUZBTUlM
WTogYXJpYWw7IEZPTlQtU0laRTogMTJwdCI+PEZPTlQgZmFjZT0iVGltZXMgTmV3IFJvbWFu
Ij48L0ZPTlQ+Jm5ic3A7PC9ESVY+DQo8RElWIHN0eWxlPSJGT05ULUZBTUlMWTogYXJpYWw7
IEZPTlQtU0laRTogMTJwdCI+PEZPTlQgZmFjZT0iVGltZXMgTmV3IFJvbWFuIj48L0ZPTlQ+
PC9ESVY+PC9ESVY+PC9ESVY+PC9ESVY+PC9ESVY+PC9ESVY+DQo8RElWIHN0eWxlPSJGT05U
LUZBTUlMWTogYXJpYWwiPjxGT05UIHNpemU9MyBmYWNlPUFyaWFsPjwvRk9OVD4mbmJzcDs8
L0RJVj4NCjxESVYgc3R5bGU9IkZPTlQtRkFNSUxZOiBhcmlhbCI+DQo8RElWPg0KPERJViBz
dHlsZT0iRk9OVC1TSVpFOiAxNi41cHQiPg0KPERJViBzdHlsZT0iUE9TSVRJT046IHN0YXRp
YyAhaW1wb3J0YW50OyBGT05ULVNJWkU6IDIyLjVwdCI+DQo8RElWIHN0eWxlPSJGT05ULUZB
TUlMWTogYXJpYWw7IEZPTlQtU0laRTogMTJwdCI+PFNQQU4gc3R5bGU9IkZPTlQtRkFNSUxZ
OiAnVGltZXMgTmV3IFJvbWFuJzsgRk9OVC1TSVpFOiAxNnB0OyBtc28tYmlkaS1mb250LXNp
emU6IDEyLjBwdDsgbXNvLWZhcmVhc3QtZm9udC1mYW1pbHk6IOWui+S9kzsgbXNvLWZvbnQt
a2VybmluZzogMS4wcHQ7IG1zby1hbnNpLWxhbmd1YWdlOiBFTi1VUzsgbXNvLWZhcmVhc3Qt
bGFuZ3VhZ2U6IFpILUNOOyBtc28tYmlkaS1sYW5ndWFnZTogQVItU0EiIGxhbmc9RU4tVVM+
PEZPTlQgc2l6ZT0zPjxGT05UIGNvbG9yPSM0MDgwODA+SWYgZGlzdHVyYiAscGxzIDwvRk9O
VD48QSBocmVmPSJtYWlsdG86ZW1tYS5tYWduZXRpY3NAeWFob28uY29tIj48Rk9OVCBjb2xv
cj0jNDA4MDgwPnVuc3Vic2NyaWJlPC9GT05UPjwvQT48L0ZPTlQ+PC9TUEFOPjwvRElWPjwv
RElWPjwvRElWPg0KPERJViBzdHlsZT0iRk9OVC1GQU1JTFk6IGFyaWFsOyBGT05ULVNJWkU6
IDEycHQiPiZuYnNwOzwvRElWPjwvRElWPg0KPERJViBzdHlsZT0iRk9OVC1GQU1JTFk6IGFy
aWFsIj4mbmJzcDs8L0RJVj48L0RJVj4=
----boundary_996_99e233a1-ff66-4b8f-8f0c-968ac97f691b--

----boundary_997_2870cf01-1463-4ef9-9d5d-023701733bf7
Content-Type: multipart/mixed; boundary=--boundary_998_71d08529-2a97-4e03-a5fa-4d4ed47c58a4


----boundary_998_71d08529-2a97-4e03-a5fa-4d4ed47c58a4
Content-Type: image/png; name=5.png
Content-Transfer-Encoding: base64
Content-Disposition: inline
Content-ID: <0@localhost>

iVBORw0KGgoAAAANSUhEUgAAAxgAAAFJCAIAAACXUhxIAAAABmJLR0QA/wD/AP+gvaeTAAAA
CXBIWXMAAA7EAAAOxAGVKw4bAAAgAElEQVR4nOy9Sbgc13XneYZ7bwyZ+UYABDhTBGAbomxV
l+1yg/X1onegNvpqoa134JLYaKeldt4AS2LnrVbamPh6Uf6qukto2e3212XTkExAljgAxPSm
nCLi3nvO6UVkPjxwAKmBpE3nj/jAh8jIuBGRGS/+cc65/wOwYsWKFStWrFixYsWKFStWrFix
YsWXCZrZV70PK1as+MpAXP0SWLFixYrfHPqqd2DFihUrVqxYseLfKishtWLFihUrVqxY8Ruy
ElIrVqzoufY69rx+7VOX/K64deXVz9zsZ69zuH+P8fq1z7f932bfVqxYsWLBSkitWLHiI7z9
zi0AALj1zttw/vz5z/GOW1de/SqEx4U3zczMbl4+D3D+8s3+X29e+DLG/ooOecWKFf/aWAmp
FStWHOH8xYvnr9+4CQAAN29cP//KK5/5lltXXj1z6fqvN8zpN37yhUqeL3j7v8khr1ix4uvJ
SkitWLHiKOfOvbIISd1652145dy5oy/2Wa8Fr165BQC3rvz5pesAcPW1ZTrsSMKtX+Vw4auv
vrrIvT2ePvuEzX6EW1de/ZRXPpmj2++HvnLlo3nKfvnrr7/6keWftKXf5JBXrFjx74KVkFqx
YsVjnD23CEndvHH9/LmzR1659vqZS9cvvmVmZm9dhOuX/vzKLTj9xl9ePg8AF98ye/MCXHsd
X7u6yLK9dfH6pTNHhcV1+N7NjweKPnGzv2uu/wi+328frv7w6C5dhR8skoNXX/u4CPqtD3nF
ihVfd1ZCasWKFY9x5mwfkrr1ztvwytkzR1658KaZvXnmyquI+NpVAFjmAB9x7cdXAc5/7zun
AQAufP/yebj+o786lBXLFx7nc2y2T9X95I1PePfnZDH0mXPnH9/++cvfvwAAp9/4wcVPGvi3
PuQVK1Z83VkJqRUrVjzOmT4kdfPG9YvffSx21Cewzvzoezf70M7HufXO2wBw/dKZPs115tL1
T5RFj/PZm/3SWJbZH/IFHfKKFSu+PrgnvHZss15k/Q362TAEgIgAQICICGBmogjgnBGT99Vg
UA8qkhybOagWhAaQRW2JiIKaGqiaqqlKFmlTympmCIAAgMi91TKiMRMREiMjeOecc8H7oihC
CPu7ewf7+4gYnEdE7xkRAI0ZnWd2zITOMTvy3hWFDyE45xgLYmJmIiJERGRmBmRmZvbeA1hW
JS4zD+ZW/+p+86t7BwK8vTF8at1Ru2PNji+JN7bfn7mf3+2My+MV/advbL1QNXXeZVT1VVIk
6TyIAaiqqvajiYiq9ucQDBEdEmUTMUuauhSbpuvaLqeIakwuth0jGEA2RBcMXRLICpPJvJmn
sqxzTA/u3Z9NxwbSHwshgSkiECIR9h+lmhE5Is4iooIIiGyAaoZIAERkYkKECEAGhMCEjh0j
MREzOSRCNWuRDImBmNgBkgKIGSLHlO892OmSsPPknCN0xCH4uggheO9oUA22Nje3tje4oLZt
JGUCRlc04FK9/vS5/zB8+hth4+ljp15mKtuumc8n7F1ZV5PpdHdvr22armnapmWk4INH86rj
hw8f3L+7v7+vhtXa+nB9s6prS22w+MEv//nurX+8/Q8/lTjbOraVUjcdj5npf/vPf/Ynf/qn
pjqfNV3Xpq7TnFJqcxYAAzDRrCpEoKYIRsxkLFlTymaLb7Jq/0kKLC4EACPnPCKYmfe+twvv
v8YL33A0IkQERBIjUwTk/f2DGzd+NpvNB8Nqc2tzPJmYiIgAmHPOEb93Z/wFXv2fxunvfO/8
j965AlfPn/s+wDuHy6/9+CrAxbf6zNxNAIDz58585K1nXwG4fv7yzV8jePQ5Nvul8crZ0wBH
tNQXdMgrVqz4GvE7ikjZ8j81NCMiZkZEUT3UDYiIgGBgAIiPvblXa4AI0N94AJe3JzA4/J+K
iEjOuZcm+OgmBqoqoiIqKqoqSSXJYskCFdGcJUvMEnOOWaJIyjmlFEUzgCKaakZMziWkDqF1
Lq4NsPapcF1VZIdt4eLWut+oIKTZhpOnhjSwds1rzeI0OsssCXOr0mpOcjj28sa7vA2rmamJ
aDZTQIwxzqbzrk0xpZwkJ5lM5rNpY2YAaNBLWTCzrCJZnfNlWRIRAAKRESJyv4KaWn8aAQyW
2hQAzMyAiTw7ZgfESOScI+fIETL3p7r/7IiYiBdbM1VVUc1qhghA/R8z7AUC2GIJ9vsKjOCW
S0gNJItk6bp2PB4/fPhwf2+/bZucc79TaqYiaFaEUJWlgXUpi4iZHewf3L97L6VUhOCYnQ+D
wWBQ1945QBxPZw92dyez+WA0eO75555//oWTJ086ZkKQHAm0Loq6Hq2vHWMI7bRNnW2MRtub
254cG3mkQC6QZ3LOVcFXjguiQBCYSsKSsWIaOKyZS3LBh+BCWZR1UQ3KaljWw6IchKJyoST2
xOx4weLpwxZfbFoCi34si++9ZBmPJ20bzYAIQwiDulYVMwUz0K+wbcvps69cv3Tpai8rPsLV
H18DgFtXfvgoz3X67KOZfRe+exEOU1u/hiHTJ2z2MX7dYvPPzfVLf/Fo5I9E4D59334Xh7xi
xYqvB7+VkFrcos0ADNTMFMEYifubSv8ADvBIFiEsNBPgUWj5d79J6LUW0mEsTM0ICYkOH/EP
N3EY8lnIFAURE9GUJSVRVRXTbJItZxGRLF3OKeWYc8opiiTVDKa2+CNqUa1Rm5vOPDRrVV6v
ZXOg61X2MMW8h2nfdQc8ebAms+cH/FSRNlwcUgyYHGaCTBZJE4Bof0tEMAQgRELo9SIhICIT
kqUc22b+4OHD+w8ezJtmPm339sY7u5MHD/cPxnPRXnxCf9RZJCfJImooZrPp/GA8STkRkQEo
oC6aJ6Lh4pPt5RQioRmIWi+JUk4xppQ6STGnmFMbu5ST5mxmAAqLEKSKSlIRkaw5a1ZDBVIg
W2gpUiPrDw8WAT4zNiNTVkUVWwhIRVUTyV3s2raNMXVdbJsmdR2ZmmrsOjBF5JRylszMVVl6
5pxS7LqUoqSsOaNZTnl6cHCwuz9t5uVo+MwLz73w8kvPvPDcU0+d8N7NppP5dNrN56xKakRe
BKazeYwZDDY3Nrc2tiwmjZ3lpF2HKozkwHkOwRWeSs9VoDrQINDAY+WoJgpIDOQA2ZAMkJz3
oXCh9KFkF8gFx454oZZ6rfzoMlnSS6v+3CJiznk6nfaCUlXMzHnf62Ww/mr5yrTUmXPn4ePB
lwtvvnWxn6qGZy69cvny+WUq7MJ3F8tfvXLrwpt9wfUizbWM5jyBT93sl8L5y+d+3O/q+cs3
P7arX9Ahr1ix4mvEk/qVfnZqD81AxRTJGRI6Vw8Go+GACXLXSM5o/e2EVAX6RGAWVQMA1YUA
SjlF0WyLe+0ip9fLKlBEQERic0T9E793rk/tzSbTg/19AHDEZuYcE/fKComQCIjJeyYi5ziE
4L13gYOnPujCzARIhM45h+ScW8g/lwEjc1AsuNhoNbx/dy8b1qVzMg06GbkUFCazPIZRW2zO
EwwKd/rkaN21XmcAWZE6JVBFQgTs76lEhISSRVUMoD88EZFsUeTdD27vj2cnTpxsmri3u0/E
zXw+rAbbW+uBRQGSmaHParM2tZ2kbO08Tiazbt4ZCICJZERgdgSAaEjIAERoBoDAgKCmhshk
YCnnmEUJiRmAFMBUGKRw7HxwiGAGCGjWS99e+BFhcERMSI76U4xsYCpAzCmnhw8etskUArFj
MkIInoNn79A7KgtfV2U9qF3pkZDM2JBdIaFqQn3y97/90h/+2ejpM+JGasGxae7aGNuunTeN
qGqW1EXvHKl2s5Yc+SqUpWeV1DXNvB2PZw8f7qFKSTrfuXtw//3bt248ePcXuZtLaom08PRn
f/of/uxP/xgkT8fj8Xh/b2evizmKqRkTMzsiR4REzjETMTvnnANW0SyqRMTcK9SFHgJANc0p
MahjVtWU0jJOCkcvLgMF0BAKERVDBJ7P25/9/ObduzvMUFZ8/MQJZt55+ADUENEH55n/5f2D
L+Cqfwz899u0eDnbbpWTW7FixW/Bk2qkHv16XaRw4DBJB32qCawPMgECIYKCxJy7CIwiamb9
HdmW+sgOE3WLt/eJD0YzyLZc2Kf2Fj/2I9pSyfVvF5Gu61JK/aM/IxGRGZgaUC8O0QxVLJoQ
qWpf+IQiqgLMxOwQ0dTo8RopZiZn7Cw4IEcibZJsXey6nGZCaRKsSUFZdNZI58mkiNGa/a6y
dhoy65xZjZ0AeoSiLy9CAlycp17WgIHgIqlliG1MqZOc7OBg1rYxJfHeF75GcmpLKYocVbOY
mBmgGXQxpyxADAAASujMTPtPB1HNDIwBiclURQ17gdkX6ThGs0W+sL/pIxAQ9kVSYIYAYArA
CIggprDILxIkQBJmBiTt07jEfSQGEFPK5BQQcs7B+z7HKCIIpN73mVkUBkOPaGY5JQVS4GY6
AbMyFJF8EhbpYoxmVhYlAHZd18SIppAFAeuq9N5DoBi78c7D8d5OM29il7zzkGXejA8e3tu9
+8H0YMegM4vICghbx4anTm0NBi53KZVAk65r9nb2x9MmpZxFFBb5NOxPBWEfWnXAaGTE7J0P
IYTg+6QqO/bOO+cQofQMAETUF0gdpnEPfwYzIFSR/koCxJRzjLGPUapqjHE0GjGRSHLOExzG
IlesWLFixb9eniSkAMDAEPAJLy8EAmC/muWsKRu6/oaNixoYWCQ1DBY3bjx8fx99IgA1WGRD
Du9Ai1Hw0cLDFVQVzIioD2sxc3+7QgVEVLQ+qEWMApCT5qTMmRmIFZGQFruGCEyL0BQhETOg
kkOH3rkCedwmurc3nTQdEXhMBcaAidFSpuS1w2bcRDCd7D0cBvEQkdQIESGYBkIffAjBO09E
zjtmDmGRBkKERSE3+sFow7iKSZmLug4qCqihCCEEsAiIAKSSEYnJFSUzQ4xqRqYmlvpTpapg
4BgAIKVoIn0Qz2xR20zEBqBg3hjZiaoiEDpEVFPU5Jmd48MPR0Gp/xDVABCJASmlJDEREyL1
xT1lVTp0atnAVNU0SczOkQGokSqrKQKnlLxjyWoxkycmIgVAA/WgOptMxgcHYX9fPHIYEiKY
5ZSImRAlpdxFzZpSm7uY25hyjpa62M0n+5jicDBYHwzm82a6v6vzg9zM2oN9aRvnQFgIgBm2
ttZHa8OsSUGcp8GoPvXcqeHm+nTexZhijDGm2KWcRbKomUhMqUsJ+0K2vsbczJjYzAyMcBna
JCyC946dc9577z0RuSP0ZWfISESIxMAqMJtOu67thVT/lS6Loghh3DRgxmVA5F/7gl6xYsWK
FV8unyGkPhPrA0hgYIZoQGaqfVEQLCNZR0vLEdEAEQyX8uhwOTwqQv+EB/F+ihT1VdDMRFSW
ZV/KfbQkxQ5DZQaCAHnxdqaMxI77KM9SwAExE5HRIt5GSKighsbAjC1Q00Q6mHaTNgHhaFBw
WUmGlDpDMJBpnE6bjgAxdbnkQYEASU0cQScZVA2BiZxzTFwUBTsOIRRF4X1wjsoyOOeRfF2P
mhYkt4BIjDHNY+yKwhkCGppBlixiRghIAJQkqhkxkyOPzkD7Sn0EIGZCiLnLXUKAPrlJfFia
v/xkjAzAEBx5QjRTAvGM7HhRzUO4qLACIALnfAjek1c1A3PsfPDM3MtBREwpnzo1a7oOXel8
2Nvd2d/bdcSSEyIQYUzR9XmxFMOg8D70n7pa1pxT2zaz2Ww6kxBKLLzrq4Sgny/gHQcfssRO
NDZdM522XSeg3vP2cC0wkWnTNLO9nW42tXY+H++3k4mlqFlzNsdYeJez/fLd2x/evRucY0TJ
Qgg+DNa4TiktKujyYkqDqOWcc0qi2sfjRCTGGGNUUVzW/5mZScrZcmzVzA7TuIh9Btk516eV
idF7VxYFEvui8r6czeZtm0T6Ly3EGEVlc3O9mU1zEjMBWwmpL5QLb5q9+VXvxIoVK/6t81sJ
KUPrJ+uBGZL1IQhTBTUCMEQBUDACsmWSDtCW9eewnNZ0WI37aMreJw3WFz+L8sIcoSjLlFLb
tqamooDAjEejVsv8IwAgMDKAgGVZzCJ0jpHYFEXBFhPWABGUARjMsqiiYVLOIlFUkQvgFiBm
TW0k5xw4Qi6YNIuknIgzsiP1kMkUzKmhqSUBFSWCGBtYTotjZud4MAjsCkUXo07nXdslXxZl
cMzsHbFDM13MWDTzPoSqbpM2TZS8mAxIziGRaiIkJMM+RkJooCqKBs4Fdv1ncziNrK/bX0wi
8xwcs5kyCbtFuTSQIVOfmCNCIgpFCKEYlGXok1veO+e8d0UR+g+UEQHRiLgsDey9d9+TKIX3
P7tx484Ht7uudc7FLooIW0HBZRSHSIgKGVhFUhYhoi7l/f2DMoBnBVhoOce+8AJROjUErMqy
KsqiKBAUcqcpTqfT3Z2Hs9lMuha6dnZwMB/va4opG2QnBinxnTt7H97ZEYm9swAjM1NwrnQO
EbzvDTKW1hiEVSigMAA0UwM1tZRSjLHX7odCqp8oaovrYKHRETGlpGop5UXVFJqagAEiAQXv
wt7eGICKAqu6KMtQ1pVzrq5L9zyqSFmWIfy2zzkrVqxYseKL5jf8TW2LGUVo8Cjzp2Bkqofa
BfoYCSHCYX4QAQFMl6Eq6+8ri0lKsBRRCkfzib1fAlK/vUP7AO9dP9VcQftcXsqCh+4KC73G
vf+CqgGoKRJTr9Ukm6H0uoL6KVJmhKgmpgJgaEiEJugdDCpWIu96B6GMCKgmsUNwFZE5LJwv
GEnVIxCiJrM+9NabOiiqoRmoqVru85uIsLNr6JwoioAhEnHOiaqKEOu6LgcDHzyDcyGUgEU9
Gm5s7B1M33/vtvdclB40mmYFU8wIBMu5c0qkiNBbZXmPzilKH9wCAEJkZFQ1NU/siRHQEBmp
n3G5OOVEYIhkYEaGli1bnGkSLdGEQBlNGbPmPrcF2MfLEDW7IqxvjGLS5557brA++pvrP73z
/m0iJwZo6IlVJeeUkZjIkE00x5hiw4xEnGLWlEsABEgp5ZhAVVPuYhO7edvOUhtRdf9hyrEj
S4Gs65pmsp/a1pF5D+QgWs4palZ2hZnO52kOCRcTEsURMjtTJVUC67/HSItIExGG4MuiCN6x
Y+cxeFr6GoD3rk8c95ILEUVF1Pppi/03tnea6kVrP0PCeceEoloWpRoaML5E5D0T13VVVEX/
bBEYA5OZicgTkuorVqxYseJfCU8SUgQIttBBZgaIff5sUZ2MYICLDNHiWVxUIWsW8MxOsx3a
JiERAPQCSGRhYbCsQSciIyY1NellWP83PjI6MFIgJABi6+eJCYJpr6VSVoNltXOfoXuUIzwS
OZCsiEzESIuKq96wUvMiMqZkaKAGaH2czDQXSEVJamAqBJ2JeTUltywAy8wuFCEEB2AIigyG
3lDAEpoioCqIAiCKQpdU0VQtSVYzIkbWbMJMoFI6H+d5uqPOB1cWu+NZEfzaINSDOpQl19LE
Zt5O98c7RSiPbQ9j3RJyGzsFzjmJqpEtDDLJyDGiA/IKnIDROQZUEdeXhwNojMiEntXMs3No
ZMDsADmrKKKqOnOoBoCUSZKoqAdW8zF25kwHBOQUzTN6T5BzYRg0oAVCP2725iYv/P7Zew93
93cOctM5xpIc5ORSBpAEaD6YcG46PzRnWWQmCK6uCEFMJebUtSaiXZfm7XRvf3pwIDF383ay
v9/OJmVwx49tDqpyrfBPbx2vCl9VzjE17cvjgz+J04lPHQFktZTFgIjIMSGaZ6qKgGBdjPvj
g53dvf39g/3xZHwwnk0nTCQAmk1QKYsXVGAH0Bc8ee+9Y0JmprquR6M1IuReh/aTM9EQzLtF
1XnOEQE8uz69SsSGqAoKoGCAICJEhowSM5mBIJiBHD6TrFixYsWKf7385rmD5e94/MiC3nuS
yAAJlpmpI/OrHxnq9J46fXnTR9J5fVTgqN2UgamZSEZDRgdsKSVVRSRDsL5SvY8BweP3n8di
W2Y5G/OiJEsRCFR1oagWaUc2QgQkBARGRAJcWCsJaO92abkPYhiias6SLLl+C9R7O2g2Tb2G
BGAzThKbLndRFMHAsgICOCL2RJ6KugiMg1AUSLmNWbVN0qU842Y6ViYEwsFgUNWDnIWzbG2N
nn3mmaoovXNx3lrSrJJSiilnlZQlZk3JYlbJkMUyAXlWUUAJ3jvGEHxRFMDY+7zHHImAENW0
CEUoiiwZEAM7WEhqEwMfKstakB+EYjQYlnVlbIaauzmnyIroYK4NSpZpU85TsTsdWfEH9bpu
n7h//0EGcQ4VjIQ9IBk4MWA1kjrZttFmypwbxyHwIDapnaecUJK209juj+f3H8zHY4eUm7bZ
vb+x5r/9h2dOv/ziqCpKT6XH0jvmPrSpgESoZGpmYqCGQAu/exMxzYX34/H+z268/Y//8D+3
t9bO/t7La2trOcl0MjHJaBrQvEMGIg/oEAChr5ZCdMzMLqeYYupznYYKaKDQ+5eaqWVRlZwj
gwMzJEQ1MNOcBFTEFlMyqY+zMiiqivambKpg6H34jS/P34hrr/ft5B7xRXgD3Lry6ieaLfWj
Hy7/tNW+FPrBF/84chIeW3701cNTd7jDH1+yYsWKrye/4yIMW5iMCy/Eii1kzULbLPTTIlRl
ixhV/17Ex9TUYbnJkc0vLQoR+gn2MUYzLXxA5hgjESLTkZLqR4YLy/cbIGovtlTVlJgISVT7
4iwAU0TJy1ykJTCjxZEBAQKgahbrQMExIDEgGuZFhxayvsrIzIwcsUN07AtV2tufSdtFWfqO
AziCQFwX1drm6PiJ7SoUJXNJ5IDn8/bu3k6TUz0o6+Bi284nU57lgaPRcN1v+OHa+na9Xg5r
YG7LJBkQsZMUY8qmSczIuVADuSwgYjpvK2YmatrWTJkZHbnCC1iynEHTbB4FVczMCjeoyjVH
vQeFKZovCl8EIYYw1GReLMXYhfL4yWe+9UfffObUVmzne3fv7979sImzjjpJ+fj2SRlP487e
zi/f4en0TLnx7DP1zJIGAmMSD5o9kifHzmd0kpz/lzsSi7IciQ8NoKo6Mg+QY1dItIOdycPb
W6Aqqbl//2TgP/vmt//T//KNU6dOgiQGBRHLCRZ1SwpAhtgxArk+xojskAjMQClFvXv3w5/8
5H/8j//zr8cHu6dPnzl5YrvyuHXihIOn2maS2lZzR5oJIap0or2PJpipqcRMTtDUcmxTl1qn
vcgCwN7wQ5UJpE/wARAimhByv2+L7DQgmGUxJDJTW6SfAYDqenRse3tjY/N3eXF+Th7d96+9
jq9dOvP62S9VB1z94ZXvX/hqfZ2W9lL2xmnotdMZvHFUDT1JGr39zi24cLrvwHf+/Pnr169/
yoorVqz4uvC7rmY1672C+hngiKhmeCQi1dcyH/Z6saWHOS0DVABA1DtzHgosAAA1IVqk3Mws
ZwHAEIKIztuYU+7noMkiF3i4O/b43qmaLrNyioikSI8SgoBASo9KuAgNDZx3wQVCJEeeiBgJ
EzP6EJgdIBGAd54JHUNw7B1b70Ku1nXZ0KUMzbh1YiyLpi0MUBJsl4Otja2nTp04efIpNKWU
CyJvLJVulcN56ooyeMdx3sAwF8ijsq7LyvtCHGJrDKmzNhF27EVkOmum81mXJQMq+QzzqCiA
yM6DFUTOu6ouh8Mhe5dSnqUug7miDlVx/Lk68oD94NTWsRJw7+69NG/WR8Ot7c1jJ46tH9/G
4KbT6e7dB9KmbjZv5s3pl1/+X189v36sQgdDhuKZl/n23ZPH16uCPEEwCynL7sGdn998+2/+
9s57vxqMBvXmyA9KQO8wmKQHd+7u3r9fFWFtbXPt2FM8WJtm4MJhGdBL087nzbRrmulkPzaz
jTiv/JxRJaXnt/LaWvmSm+Vf/MPd9/4pBDeoa3ZOzdh7DsG5QN6D88yemJBZoZ+laDml2Wz2
L7+49X/99//2N3/z0/H04MWXX3zxxRcGgzJ3ncQOESXFrp3ldg6QCDCrRhEz62uoFvZXxEiE
IkSIkkXF8FD3G4CCIoiAKBIA9l9pVdGs0nt4AZCoxpgEYPFQgRjKwfMvvPCNl76xsbnh3Zcc
kfoIF757Ea5eXSqDL4vrl/7i2htfYQjn2uuvXYXzl/9yKeZOv/GXl3905tJrr3/3MwXl+YsX
4eqNmwCnAW7euH7+lYuw0lErVnz9+U2E1OPq5NDwaWm6eRgNWtpBHRVShz/0UglxUWLVs6xn
wkOLqUeb6mvFFQ0BjPrNjEajUBTT23ezCBLlLIh9F15YTAKEZTBsuS1VXc4TpIXrAREBLudq
EaKqCAA4zwSIYN5zFUJZBHaIiJ7ZMTIzEwOSmGpW7z2BgWXKBjk7wpqDgCZ27EvxVG3Tths4
dMH74FxZFMOyHpWDqqqGa8Oi9JaSc+oUPCAgPlOtR9UkSVLsCInt+MZmHaqUcso6j6mqyuFw
i8siOX/Qdk0bZ1LuRy8Bt08+Pdo+xvVwb9a988tf3bl/fzfPDXVtOBqd2BxubqGaTGazven4
YGwAw9Hw2RfXnnvhmWow3KoGO+/dvnfzF83e/m7w6ZlTa3JGNB008zvvvvurt9+eHoybeVvX
Q3lwd7b3YD+nxnDrqRMPHux08+6b33jp5HAwGNYpR7BcIk6j3SkGO+UocTUcPnX86ZPFxkgY
UtfOCvdA5o3BiVNbZ//om089/ZwAD7eOlYMRstcYc+xi1/ziH/6/v/u/f5KSPv/yC7XD2WQs
sWubSfver/YsqSo6JM/gPVelqyooS2UHzjlfDocbdT0KVWXkeuuMyXT+85/d+Ou//q9////+
HZieOnny1PGTmxubhQ9oAGJimqPGLseYe9WkklWyqTE7cAxq3GefTaEvVCd1hNJ/4XsdrqJI
oAYqaoACakqAuZ+euGzjE3Nu5i06xz5I1uH6+pk/+NYffPObJ44fF7G+e8xXx7UfXwU4/73v
nIZllKaPsFx8y3wYyDUAACAASURBVN688MnJr0Uw5/Irly59JK11uP7Fy5efMObFy5ffvvTj
a29e+Ihm+cThFtm/y5ffvtS/dvGtt+C1RXbySNjoSMryUZru1pVXz1yCj6Uujx70gtPf+d75
S9ev/vjamxc+o5fyuXOvwI/euQUXTt9652145Xvn3n7y+itWrPg68NnO5kc10MeNCT6+cNEd
b9mruF9ERHYEAABAZu6FlAAssn6EoI8E2bKO6rHhlu83xN552tuyDgsX3t142AZQcsalPuot
F5D6eYJAS+MFIlRRUF1Ex8xAhAkga84ZATBzl8U0U19CRYzIjrwLHpZ9Ag3QIztygbkui0FZ
jnwgQyTnfMHkAahwAUQdUO0LBtMkngI5Jkfeobouz5rKOUtZFUabm2vbW957VNl7uJu7fPKp
kyGUt+/d/eDePe/L0WC7KEfmXVKcz5vZeCbTWbt/oOxndDAdRyjrSL6ZprbRNolhhjR9r7n9
od3RLkkTLatlAdVp6H757t5d+OmQaegL16XjMQ2JvKq/c2f6T/8Yi5A1cdc8k3OvP3U2mf3t
T//+7/4mVjWO1m5xaNssYjf/j/+GXcLKH+RGArDn9mCCbRdSLhUKgOCcG7hoHZkEAIht5fhf
/vmdv//b/2djfaOoBlvbx9bXt1wxJOdLH0Z1uXPng7w/4dTZ7ti8C23HpgMInQtGKGoRspiK
JYdubb2qt7cjwMF83s6nB3cODpSyqAL5svKh6Lo0v3930/MfvvwiGri10ant7bWiKlwwtWAs
kvI8pjb3Ey/V0ARAenWUTZUQCdFEbNkMGntbtL40ql+kppBM1VSWxh6q0NvFg6lmURFtYiqK
erS2lsWyyy+9+I1vfetbm8eOASCglGX5uS/k3x1XX8NHdVJHIjMAcB2+d9N+0gumR/VL117H
1y79+ZXvHCqS6z+CvzR789rr+NoyUXft9b6X3U/eOH3ryqtPkiNn3/jBRfzhle9f+M6RhU8c
7uqNs2Z268qrZy5dfe2Hl2+anb515dXDGNLRPjDXXsfXzrwKT6j7uvXO2wAfbdZ8+uwrANff
fucWnPnYKXq8iuzsufPXb9wEOH3zxvXz534AKyG1YsW/Az4jIoXLnjCfqKI+eX2DnHPOuddJ
amKqxL3N46N+rn0buMX2bVFv3puff6IhZ08/KwoWbV8h5yx9Fz81ZCREWnit9y1OTAHd0n66
r+ntLdgPNVo/B5ARqZ/8DqiKZiQKmlQEyqp0RYHOoS9c8M774ArHPhR1PaydLxSAnS/YeUJS
9aaFdwFg3YCzRBEWHlbDzfXNUVVqm6Y7u/Pdvcq77Y2tja0TRV1x4KoqZvt7H9y8Nd89QJGC
XRFKgL2oCil3e/vzrovjWTR4OJkcpDhHvD1t5ioH81k0AUcueF+WPKiA/RytSzEzlVuDp7/5
eyecm2kLDFUoCnQBfeW8dcmyeeSyKJgdNs3GeKdObZ41LuUiSxofuBgLM5JsMaGS+ToWlnOC
bDHnViV5l3KCdi6QHBWI1Jh1BMlk6nBuOp5M8u5ObrsuJkGcmxGR30NMnTNTz16lM/M+zPmA
ioeDepD4lw+ct/UqBVLJZXCFo9GgLGuax/uiHDyaQQguwAAQgaAEBQZzAJ7INM0m5P16ETZD
gC7HNs3bKKbOaVHQsfWNp7aGZ148dbC///Dh/d3prCz9yPLAl548E8/nedp1HCMRxJwM1UDV
oLfRNxPvnFGfYsaFWefC0l3A1IjAAKyfVNB7lFlvrmGgCEBMIJJT6mL2odrY2AihnIyn6+sb
L7744qCucuxUjZmX/X++XD4SRDpaHvQoUNNbWd668iouwkTXFxmtI6udOXd+ufxomKeP7zwh
43Xh+5d/eOYvrn3n3JFFTxru4ncvwFLsPB5KAvhIhOnC9y+fv3rpR3916403Tp9+4yf2xm97
ij7KmbN9SOrMO2/DK9/9jPjVihUrvh58vl57n1z6/alkyUvTwsfevuwFtxBnffBKVHtv5z44
BAZEKPLRbfav9r1y+27JvSYjohB8ViMERmCAhZoCUFTG3oEKaZljJDpsLnfYjBlzTGLqnXM+
OFdAMQqhCEVBzC6EoiqdL3wRQlX6UHgfTC2mjKGo1td8VQ/q4dpwOCyLktkbdPPZzu0Pmvt3
uekUoB4Nnzv17EvPPw8p7X94N4tI7FyKgrSfchKJqSMC7dpm/8BlY4VsMI5dRstdrJCzQac6
jVGKsnWYXGEuZGJVc95DbNkEkSRmdM40T8aTzvuUdUrOG1JV02AQ6jKE4DlURbU2HJah0Gwh
hKquAYgABsYVUG5bm8+hbWC879uO5k0+GOfxRLuOpJvt3j6496HM586s8Dh0FSZh7epQYAeU
1RdBCsoMyXFmL1W4l/PO/n7jqUEb59hYiqLgqCgLQCTTQMTMqKaYs7RojLkTjZmgie2exqab
k4NQsKE5Tz4wO2JyxIGIHYP3PKhDVYXBcBBUSlFfFkyspkZWj0IxqJouZUtq85TFDEOBG1sD
F44Pdl2ctfDww+hKVw3L0XrtnVtfe6DxYD5FkJRzIjPHhoRItPDLWvi/q5qqgiqiHtqhIfZV
6bCMjCIAEGD/uLDQWapmtjYcOnYH+wfs/HPPPn/82DHNItqXpUPM7ee53L4wTr/xg4uXXuuL
pD7y0iJZdv7yTfvJzY/N9fsonxjmecK43/ne+Us//qsjGcBfb7iPD3390hm89GjhERX2sbEP
g09H6sJ+jf0/04ekzt64fvG7F+CdH37uHV2xYsW/WZ4kpEQVl+12e0Mn5z66/sJ/8IjHQG/h
vDA1ACJEQzTVpc34o/l0aqBmOedFTxlE7NtrEINmA1AAQrPexwoMDNXEFi3y+nyceA6lC4LZ
EJBUwdCUEBDQ9cEnUBQzFSBCA4qySAt6H7wn58i5wWBQD4fVYDAYDoH9tE3Evh6OgD35EIrS
cfAheO/MbD6f7433qChfPHP2j/7w288/98LW1lZg180n+/fu7d69+8Gv3p0eTHU8rsHY+Xns
fvnue7c/uJ3mDXYRY/QqSXXa7eT9cR9PU8iMUHgnzhxgzpKtFUMk7cwS2DincTcdT3Av5QaQ
6xGXNTqvQEGhJGbAlEViG5uubZuZylSsRc6OOxOHPCir0aAelYO1ejCtq2E99L4QAF/Wo9F6
AzBd38a19Xow9D4Ug621Y88UQK7tfNNWXWdtC9Odyc9tZ+d+Al0vAjgkVA/mRKFNJZAz02Yq
LJ6oQEJgT24LtKnrqekB6Y64fYl73o2Zxm0bZ1POqSTyajVQTTRoaM0HB0xQMRUFwNDXpXI7
T2mak1mX2sjqNmoakPlOc7LYaRsttqS5dL6uy82NjbX1NV8EQcGKTj373NrmdptwPJtnaHzw
7Ml5Gm1Um0+fonw8Tpvx3v7Dezt3P7zX/SI6dCGUzXwOsTt54sTm8acmqvea2bRrDJCZNSUD
M0dKKAwI4KwvL1+Ye+DSZU0XlXi9f4YxkIGJLboRE5BzLDk3TfPs88efPnWSAHOKxAwOReRJ
bS6/Wq79+OqjuMxNAIDz554QfvlEdfIETr/xg4v42iPl82sO9/Ghfw0Xh77Cvg9aLRbd+qsf
XV8Evm595njf+d75H71zBa6eP/d9gHc+306uWLHi3zRPElLaT2dbhH+g95bEZU3QAlRYNiTu
VyYgJERFVCRmAzYENUVAI1AFUc2gBiZgYgBMAEqEiGyGkqV3kTIzRTVGQzFVB4zgzFQ0E/a9
eDWnBIoeiBWUMLMhIVt/E1MCA1UUYAbnvHNceb9eFFVZ1HUd6oqCM0fkQxitV+sb5frasZMn
kez2nfdj0jBYr9a2NrdPbR87sTYYDYpCY3v/zgc3/unthwfzsqiDNgfv/+JX9z+864qdnQeT
/b1uNsmzmTVN3TV+rUSwlGUmbdOkFLOKkRlINMmoAmZGfVJRASmmbjafI1lZVVVdkcOUk2R1
yFm18SIUOoO9SRrPGt9oyFHNGFxVkpZU+cI7wHlX6bQkMU2dRgVjds5z7YqabZDbQTuuWqCH
6pBEsc2WgI1CJB4XeicnrNfXT72w9vQL26eeHw7WWJTbOcwOpg/uzt5/V27/stvdKwxUYwRz
lfeFT0Zm2kmiJJ1JDFawq4B8TiaABsFgSOCQa1dt+/KOCaTWt5ITOixK5wvmmmgdecNgU3FA
rMG5kitAM01UtFrGSF2nE8q7XmYbRXh+u6oHFYd1X61hBdM43dlX0VCW1foweZjkdt5OD+4/
kEF4afvpsFEPBqnLESkhztF1GqCDeeRIp+rN51/Y6p6Ne839W7c//MW784cTnyBEa3dup+14
7MSx0Wgjbh37cLy7u7NXqDmijC45E1QHFgSzmSwSebSwybdFC6X+asF+uiigmYKCKZpBzhbT
PITw9KmTg1HdO5t5Jib812DGeSQtdvPjr1798bU3L1y4deWHV+HxXNvHOapOrv3Fpc+eydbn
4B5b79cY7tOGfsyb6lOKzeHCm29dvHrE9uHWlT+/dB0uvvU5JxKePvvK9dcuXYeLb32lFg4r
Vqz48nhijVTfexUAHu8uvCjnRuxzdUc5rAU/zLt9fKt9QqSvJmdettToZ9r11d+AbISiiEDQ
Cyxic4RMpkQATAacEecqgVVrn5JLsQ0RHBoBMmBBrnSh3ijLslxbW1tbWx+tjUIZXMDe3trA
soqAKVBUy7N20kq7Nx8Niu3CH8R52+wOqvV1ApqNd/d37rZNMxvf/eC9/d0HJyjXzWR288Z7
v7jp2CNhSslATVPqWo0tqZhkZM69LyMiEoXCE0HwVIYyeObCa1X4EIoQAHR/bz893HFMVVmu
jUZlEdqmncxms3kzm3fztkuC2SD5RGVih0QG2dDQ5poOkpl5w4p4y4VRWQ+rcjgaDAY1MKac
ubPtwdqoKprJfjM+cEyi2Blk9gldK1ASNpJiN2vHDejd2e40/vMvJWboOt81awxDwkHumAt3
/OkgMnJUoEHunAMCc4smPVgZg/iSXEHsHaKnDDbRPLW0m7sH8/lO1+xFaZOCWOmcZ29ZMqmW
ocPcItlgaGpVjuW0jakd52YH9b4qVuuD0bFqfXtjvfQniuLU5jODDZzFu+9+OM/NS8++dPLs
70Mo54x7lj6Y7t+Z7kMc1dUWPP0yP3uOQ5XG48neburGBQaXp7FtNLYpz1w5MTGIMnD108++
cHLj1L/cuLV7+0HwhbXy3vt39f3b609tPn3u914cPlVrOZ3OowgYcM4Aifo23Uc8y/DQE23R
pwgAgJDUBBbTE5gQAWw2nYnaCy+9+MyzzxRlkSU7ZOsfXD53Jv13zGPF5nDxrU8K5fRiY7Hm
xcuX4dKlJ8ebLrx58/LbZy6dwUtw8fLl8/CZWuqxSqpPG+5zHc+FN+0twNeWub3PYY954U27
ee7VM4fn4dBSaslHT9FjW+xLwz53yGzFihX/5kH79Cff9bXqsNnKopkLYO8OxYtSJ1MTePw3
PuJinlxZlkVRmIpJ1n6yEoIqiKmYad91d2Gfo4CG6MxIsmhSE1AxNGVauI4LekMmMiTzjrwj
7zA4Hg2HDLi7t98c7B0jGoUwqOv14drm+vpaPRwOBr33T79vCTSCMKNbmlypakqSRVOSlK0q
B+vDEYhMmtacD3VVra25EHb3d2azsfcs0jnGyhOjEjGxM8CoElXQI3hSEwN1RAMMBQX23Ntk
+eCQBNEGVfAeQcUKbocegrcsqOqQSZEU2ACySBY0NII2SxYFo5RzM485au6kmXWSTETn47nc
3y9nsWJXh3KjHm0MR8OyZmJCZO/JMyCGUCJR287atlFQ9t5CsKIsNjapHnUG2dez+sROo21O
UHjwHhlDWTgCBAkAZCrtnHPnJNt0UqsUYLmbEgipWJe6ybg5mPIsjjI6MJOcuy7l3KA8aJt3
x7vvzce7qK0jJQxEtQ81+4EvhlUZHFSlO7G+9sKJ4+vBT+7en9/5MORIaMnhvOBdhTZRkcut
sDFcX9f1gEN/YmOdXfHz99+/9fAhbG3C9lZbVjPnGqaJpIi4zuXJXL78/DdOvvDSJHbv3rnz
4OF9y23FuYBuSDlAKroDnty79+Gde3fvV2X9H7/9x9/6gz9K0+5n//PGwYP99XI4P5jG3Z2g
2cqaNjbp+DHdGD2YjGPbBMpmXcY2OwFDUDAzR4QIqtqb8QMCkhGgqWbLAKhibczzNnZRc9Zq
MPyPf/LHZ86e4eCyWgil877veEhE//l//y9f2LX/6IJ9wi+BFStWrFjxZJ6Y2lM1gF6FLKJM
BmZGRItn7o89MB+xieobZdgjP6clfZRrYa2pamYESkiLPixIQtD30lBFMENDY4pkypkRnIFT
5WQh4UZRPIN+a7jmhpv+1DNbQSqHjh0hEJCjWOS+Y4eqqBmYmAMkJCZixH4Gn2Q1A0IHxi5m
N+7aTk8AFaXPuy3sNOxwIzZqUhTsPQUurAhSe/ReCI2JvPdlYYzGaI4SSGeanBcD79g50pzY
hElyN0OHYllTDORH0ShnydnESBFEUUCi5JSkSznmDJbNYkxMrIqQklPUpDRrLUPwBVnWYVjf
3hxUdelKEphnmadGBbIqAJNjJOpIEluX4u7B/sF00opSVR5/9rmXT2ytb4+MyQ032nK98NXA
91U9YKgAwEQihkBMHhC1KKJYyIk0O7QCMkIm09x0cWe3+fAh7O+5OPcKFrvUzFPspl07nU1h
ux6oKGPLyNhV2Fbw/7P3JsF6XNeZ4DnnDjn90xtJACTBAaAsmp5tuQR2uMJVinARrnapIyrc
3StH1IJYkhvtvPROG3JJ7LTVSh1uCe6K6hq6LdrVFRWySxRpkqA4Ynp44z/kcO895/Qi//fw
ABIQJdGSbb0vQADMl3/mzXv/RJ485zvfZ8bGT1w+8j5DDc2sgIU2MNtvu8Uc1jJTrY0yr0ZT
PV9sbe8sah+yYLmKrdTWFjZdu+W9R5WJxf16OouLzjk7GD48WjlFJjRdDe310omZ7ES/dXBw
bXaj4cW48KuZ8zb3zo4dDbscfTBGJxvrAISFX2iXVX71zIZ3riRfeo8qVUg2G80wv74bPty+
WYOMMufRWsO1MxEYAY6+1bgs7wEQ9M7PhCgIDp2IijAiWutYOHF8+NTDp8+cVlBmNtb2emZw
d6vHCU5wghOc4B8sfoT8wVKB6VjX3l0imaqAd0p+Rzv0FjF9dc/QoaqBHipjAvZPHEIEY60V
o4ZARVEQlDAZEBGGBAaUKKkYECeMzCjqEVeqcn04fGR1/fRkZb2sCjTadg6SwQZBjDHO+b4m
ackaMswSui7GaIFyySwSgLIwq5AxxlKMiYwVxRB1IW2yRIopHBiFjNAEHvQOs23n2GAKMRXS
FuoZMgfOgIF0EGJMi6aZd10jqUHddTjjYEAyg5iCk5RbdMrDzA/LvMo8OYdohKVtuy5GVQDV
tgmiS8UrZhVREOhCxymKSM8ei6KBtUsKphPEmJvZiDIPsZnt7u7s7R50TVTpO8usEiEiQ5ek
U8SkqGR8MZgMB8P1VX1o3T20hmTB0Mhy0hmJhKaZzxeosFg0129sKfjVzVOT8UY+HEEn3DbC
MXBMKVgDPrPGGku+GK6Po+vysu0OdvZ2Z4v9lNrJeLi6+fjD47FUg1QWtbcNYQezEA7CdBr3
57Ko9+Z1u5iyz6vMHuQWOmuKqpAMIPOAGBMqjcYTN1pvW7h+++DG1s0DE82k2BxXVbIDU1Ro
VhrcmIHEMBiMHnk8W9lcu9XO3rp+vRsPzz360BnP9UPl3vh0QraWCoO5SiacKTshOuUSPxaT
KINRqym8//57W+9dW7FlUhOmM7eYQwCHA4coHLvUJQRNlGXWqGkAwPahjwAAyrJbdSkLi7Q0
4VY4skhCIpEEAOPx5OzZs0VRHt5upldC+1zv8ROc4AQnOMHfIx4USNGhlCUiikjvm9dX7lCX
FQGFu4QK+sfEkdiBiBDSUWtfD10+TojM0gKNOCIve54IlACsARVVYRJSYae6wbCWu5XN1ZX1
9Y3NjZW11TzLB2XhLFrE1AVDpAIxJgZAZ9FYVW2SdG03PZgfNLFrBVgxdgioKIzKyoyghIjG
eYxJujZEF6NvkNOAzCTLhtaNs3xUld5YFWWRJqbZop7uTWexXUhgABAt0WfkQFCd0zyLqmFe
Yz2XlIIkJ8kgWMLS2SLL8ix67zize6VZSGzbNqYEZICojV0SMc4Zl4lK7CJFIUSRBMyAKqgB
oBMNaATQZJkYkrbFrqsXzbSet6jJG2HoHU18ljnvB8PVvPDVYFgMRoPBZHV1Y319czQYGoX5
tdvdonaEFCDFJKDzenGwWIDz8y7WrGunHlsbbthsCNt79uO322vvT/f2c9LRoBhMhqkoOiIh
r+AsOlIE66vhRhHpQHYO9mbb+9Nic6165ExeFdnqmi2r3KBxjlhS0zIzKwfuIqQk7Xy6t5ju
ZcJ5iDqb3fzgo7C9m+euGg+NGqEWPEMhQNq69qbFlaosci3RrCWtomRihxM7LOcIcW0l/dpk
lI1X19akwK0GZDWLEVGUDYsRJk5WFAxIjsSmUmcYZ9v7H7374e6HN2yrSZtu1nbTOl8kjma3
k0VRbTM3IYA1HdkDgnxocQUNWAUmJCSkXpf/SF8fjt5DEAB7HdjeSSnLiiefeGJjYwNBCY0c
4xf+vDXNT3CCE5zgBJ8Vn9kiBkFFhZmWb8zLwtg9bPOjxJWqMvcPhk9UKHRJxyVARVBVRjhs
cEJA8KCq4lkgiFOZVPlj6yvnR+NT4/Foc5OqwUJlP4XdxcxqKifDajLaP9jf3dnVRuq6ndeL
NsQmhq6LbQgcWRkI0VBPVzLGGZ97V1ggmzjWdR26YFrrrM+yrBjZvMwHmRsPBhPvTRu6JFOj
XWz2prO92WJed3VLdYQDCQ2xWnJqC6YMnCE/2twoV1YWs3kx3xk2nPssd2VGUjpXWGtUVbRp
NSUKgu819VSDJBYVlsgqvdIjJTYxoKGMfJkbi2hAHIg3ZL1Da9Q6KqpkrZIdeT80RhV6zhmi
Z0URJGPJOCKjSKk0nFtRWjSxbrr92wd71/cghrRo0nxmmUtHgyoviiLLc+wCzReiMPH5GAx+
8FY4uI1FqQe3R/vXB7O9ejqtF7OYuWYySmUB5RjLIWbDrBgV5cpgvJZ5l4YrdPaJwM07H73/
X9/4wY2/+Zu1p59+5Fd+dXLmUT95yFeTPMvYSEJlR5FULBgLI+6GIVUcz+5cpx98bxdvCWUU
U8IuMzYnWVkdPfvQil8paVJiZooyHxSZswAcRaP1drS2Uk6GYtCjTsgCWMU0g+0kSz68EekF
7olQRUiMT1661Ozu713bOrh+m2fdw5AX1nHdRTENWOtIyNxO8+v7uzfrtosyzqvhZC0vfYgN
R284kz6dCr1Epxpceicd1wUx1oIhFcDAPsse2nzo8cfPFnlxaFO0vHcOPY0/kwTuCU5wghOc
4OeLHyHIecc6WJfVOVVVUVimnT6FxtGTqJYGrSJ9ieOT6IXORZSVkwqTooJFNYgEijGZwBt5
+cXTp556+PTqaJgNPRQ2GX+Q0vXF4uaibghJNAOyUT6+fu36h+/l3YJAyHuXF5i76GWxiOrB
O2+sDyGGbu6MGE+cOSryyWQ8LAcgFbfJgfFoM3IDa3IRm2I2Zx9n2AZkATCctGLKzEpb6Bx1
D1Nqp22s2y6GFEA9GeMKu7F+5tFffnY63e+A7N6ugjLqLHV7PJ/uzpquYwF03mS5LMycoNUE
okWej8fj0XCYe4+gIYTQtdaYyvkC0LJYiRnowBpPBgQ9FaUbdUpd5IzQaWRhBVQg1Q6NVWtF
U4hd23R12y1yp6ORy0qwRTVaEcVYNxA6n+fFytDEDroGNfJif38nNE2IXachLuo6zKdpPusG
5ZmHNydFniGU1k7KYg7CjvxomEbjrhxLtUrVhLJRm5fdMMudobKQsOhqTpWnKreQqqKqbJ6D
Ve/aykSLyiSEEZSsJURt41DLQtEd7Nk3b7V//Za59vHQJDuiyWOnNp58jJPObm7Pd3ey3I42
x9WpDapyKGzy0FoJBlqUkLgBNGgBEBhcr1EmZAG89qU2qwqIJKJtbGXaha3tg5u32905zru8
05yNiUljMIqayCZsm7DTte9Lcy3VdRdXMM9UqtHAGTOXYCEvKWtVGFVUOCVQMM70TtwEAIAK
vUvxUqZTAcqiePjU6cFwSISiCghEtOSYI/Z30E93a5/gBCc4wQl+FnhQIGWOlfaW9sO6LFYo
LV2Bl7ynQ1HO45bFh3xz0KUcFRxJP0P/VxUVBhZEJDSEgKLE7Nr2IV889fAjzzz66OnJxKNS
7ptBtpvCwWx/kVSNW5usRWPUui5q2KlHUk7OPOVxnyAY67z3xtBRzGetRcXZbNHUZpzjynhQ
+JxUcmtL6yyDWqtd6hZz7tg3qajZApCSN967TI2rkyZWo9ok2Z8twqIeWjvOMvF55JRCADFI
jlz+aLXy1MqpW2Df9f5m6rrQIgFzCjF2qh0aNUTGceS67tRbZ51V5XnU7iDuNySSurbtuhA6
EbGqhXCGmhF61QKodI4UiZzPChZMilpIykJkZiSwWVTokgSFwCkmFlUlYpOTL31W2KLMqoHx
GSMKKKhYVRDWrtNFI23kyIZMZpxRR3ZIA2/K0Rz43WldzNoSPUU1fQnLW4NUZCMYbdq1M7C6
FrIiELM20sy6nb3F1laazXJrn/nV37pw6lS+vulX1rQopoVrfSSJHqFAS100i5AnLrpU1F17
e2/3vR/ufPC3trmZrRAU3p0ZZU+v82ZOCsYVi7bb3ru5zdvY3DTDKh+Utsqp8OS9Wuuc82RU
gdCIc9EiAhGLSWyi2KTQSpw3zbyup4uDvYN2f5EOFtikLJJtE9UMTIk1ikakg6beW8xiig3E
BoOBtOrdw34wsVWK4fb2XFarUysj9IQBCIk5ESIZMtYaY3tlzt4lRkTAKCuoqiGcjMcbG2t9
y4NZNj4gxP2kIwAAIABJREFUIqQUiCwciiac4AQnOMEJ/oHjgaU9EVhGS4eqNoQsAgqKQD33
AwERGXobu6V0Tp/IIgFlYREgULPsxENVVOi7/1DUABChUysheW+0a08PBk+un/6l1Y2NLCsy
A6brBg4rSguGvTYeTFnZ5JnPMiWrSiggkZVBQYPzZKwL4BaROFnh3FlEVFBWZVaBLIswnJsM
VNqEXUtJHBhDhpnzFMHYHNE6VfIR8plk80T7re607U7bHMRmGhrVdLbwX1wdnBmOB4JOJIkc
qG6x7hgbqEsao8iN0HwwP+hCICQiUlEE523pyYCASVIoNW2MqXaKkyxfZaw6JhZJ0fvcr64E
Ye2aAWmVZda5umlCYmsMkAnM24tFYjXWoyoGiujrBJ0xgXxAUu+wsGSJVRInIBNjkmnjd6Yp
pcyZYjTIJmMtCrHeVmM7sdaotwQCqUuSxJCB3gbaEKEBogSwHzjULYfo0eUmc9Y3Zuj9Ktqi
JugyLgyuNgbEOVOMV09NTj81Hg/L4QjKorPUKguHcZMmNVMMvgtFF2i239662d6+MZ3tNxxI
4lC5XE3ZYyM7KuzKgCYlFNmMu65tguvcmUm+MfBkEhklB0m5jUQOEVSEQKxXJIOkCOwjxK7j
JsS6a6dN2KsX29O0vwgHC20SCXoWGyKxUgIImqK0ERrmg7bbaZuDFFplgzDM/EO+PI2Y+9K6
KoJpjGmQJuvrbnU4C9OURGMUVe+dtVYV0BCosmrfpSqoKGqJAuBkPH788cfKMldJSmTJExok
IwIKrHc46T/jpNSVT7ivHEmCH5ey/Cz4cff/7OM6OubR1k/d8rW3nzv/0mv3Oge+Bi98R7/6
Lbx4+c4Plp/6nMZ6P9xvQn6yiTo+I0ef/YTc6Oc1/z/VUH82+MnG9vkuygl+cfFA+QNVZT7q
1LuHtHGXL8ydrSpHpT1Y6g4cKiAo9v9BT8ZFQ0SCisrKket1Mz53+rHfPfvU+fFat70jsWnb
yAX5ySAZIzvzIvE4cwRQY+piUiVkxYahjtQmUikyMb3SNIIH9QiWiAjBGOnLK6oGwVIDilbJ
WWcMqahGQHLOuCS8Y2maZ0FNAFsHmaW4CLFOXcRAGW6MhkPvnnWD866qnOtdaBKhUbFkgdVb
ystsxayeWtmI27t11wooIiTpddhVJaKCsZSTWdFoE2wU1bnNhx4ZTQqRbjojhtF4NBqPZnW9
v71LSYuqEoPzsmk4UlVko2oWwvbsIKpkVSXVep2NI3mjiILDlbWVU6ddWRrnvHfWEhAsjCCm
tSas3N6z731I16/RdNctOlosFFSQuygJC7A5IASRgJgsBUOdIc6sHVTFeESDcp57deuFz8qs
qExmgsT92kG0+3tQzxOpSnIQjEUBoYLKKpZZi6EN02gCF0mU1cxmMJ2GetE18y51kmrhBi3n
QzuclKO10Xh1bAtlk5LFaClZYdORxSorRsPCPbxmGRAomRycR0MMwCIiyiwSE9dtCjGEQDXT
XhtmdbOou0UbZw0vggQ2CVxSIwSimIQ5hZTmkmYpLGI3C90ixo5TUkVDzrqM8jIryzInQ6LU
KnSqyRjKvMuKpScSC4EaRMJlSU+YVUVVew1OQlIFAXDebWxurq6tKUh/PyiCIBAAKgLhoav3
Hf2znynu2KlcuYQXXzr/HHxmexW4+spzX3/6u5/r0+fuZ9rVV547fxFfv1uP/FAL9Orbr8OF
Cxdee+01ODIKvPxnr3zt+RfPAfSC6i9859Xn4cq3jh3/yqWli9/nMOq/h8u/z2kuXl7GT1cu
4cWLzz3z2ZfoBCc4weeOB5b2jBERROypG0v7vEOZg0NZhE//LC4tXUVEjNJdWlJIgAqyjKkI
Eer66ZXJhaef/bWHzz6SjWwbrsuOrYamNLVhl1zpsumQG6O2jvmMbQggaK1FMErAhYXCEoDT
ZEUIwShYVEJEVTKWrFMyimgUHCMQCiIDMEIkiKJdCMIKqiGlnURbEeahE0omyzH3lPkVKHPl
EeFala1k2UM1Fa1GxUVGnZFkTN1xizYQNJyutwcpL3n9DMzYpGRAo8QUA3MCACKw1vrMl2TW
UlrL8mGWWYRtVgeKxVqStGWt8VnKqjicGDJFURpDopJiAoBA2IUwXj09Hg2KajCrxttlmfJ8
XFa1sVyUfjjOirIoqjLLPCIKk3R5t6j2pgMgt5hDe6A8hfmUUm2JQZkNNspJA4qqIAhIgMiS
RBIIAyJhsI7KImbeDAqajKAqnPVVUlh0LmppvLeWUMCwEgRIEZIa6EBi02obPGMBxora2ECo
AcU6NIWlgbEj7ya5GWdcoDjp3EGdmV6MFYgAyIiqALAoa1JkFWViIBFm5hRCt2hS00mXuOli
3aWmDXULdRo0JC0zswG0DMwsUUBEk8TIIYSWeaGpCV2bYgQR0CDJggyyrMzzzDmDxLaI6IOm
rus6RXA5u2IRY9CwTsYZB4mVEyAY0781SG+vJ8KqSqSEBglZKcS4vrH+0KlT1lkRlD7je+zm
UNHDm0r10+mFPzMsfVreeAfg3LkXv6svPnj3w5AHAAA+w/6fCVe+fpc/yycOe+GFF+Dy0i/m
nTdeu/DsC3CPHvo3v331xRfhlT+7DHDh5a/dE+RcuXTxMvwYTnwPwF2X/0l8XhMCAO+8cWj9
B/D8q6qvfv6n+EeHn+zaf5Fn7ASfJx70T/VR/umwUw/uYp8f+/s9nzokVt35iKqKat+Svzyn
Ql/s46Y7V43+7bO/+bvrp8+gz1LafOTMU7/zG9UTj2abG2urm6PWmpu1d6ZYqSajar3MT7n8
MVc8iv6MsQ95t5qZwoP3MsjcuCwng8FkPBoMR8VgmA/HvhqZosQsY2c7YxfGzokOLO1avEny
fmx/WE8/Du3NFHYRYlEWvlgXf8rkZ7LqTFZt2mxN4WGgxyh7RN2ZDtemoWvCO/Xstds3/v3W
R1d2rv/feze/F+dvdvNrEj9sF39748O/O9i65sytqjqYjOOpTf/446Pz5x761Wcf/e1ff+rL
v/NLv/flZ37vy0/93nNrv/+V8Vf+VfkvvmL++e+br3wl+9f/uvjqV90f/Rt4/l+Fr3yF/+D5
7OLzg3/xP9Fv/nJ7/on6kTN7o9H7Tfd317eufnD96tsfvP93H9x656PwwXvj2x+Mtt5zH7xB
771ev/5X17/7Fx/+529//J/+/Pp//D9v/qe/2P/P/x6+/efZt76V/vz/qP/yP4b33uimt9im
NMh5NErVJPoJ5eNh5SY5rxS6XuHGADdLPTM0j47d2ZE/O7SnS3zKyD+bdb+zNf2lj24+8sP3
Vt+9Onj/rfLGu9XeB8XsQzN9Dw9+qLMPZX4T5zfz+fZwvjPav71ysL0RZpumXS+7ySiO12R4
mqon/OipcvzUID9l3DpClSIu6m6vnu9xN+duISFKFK6jHLSyU+utudmq8Vadrs8XH+1tX926
/oMPb/63t27+5fdv/Jf/sfVfXt/5f9/Y/cs3Z3/9Tv1f3+Xvfeje2h68NyuvL+SgjbMmTJvu
YDE/mE73pnu7u1u3b1/buvnx1o2Pt25e292+Vc+nXSCmFSzO0Ph8tvnM4Oz5waOnylNDv2bM
iNUuYpi13V7bbtXz2009VemcwaK0eYFIIIpAZIwxFoBEQBWJLJEFoL5JFYBYIYoczBY7e/tt
F3vu4LFbSBXk+OuKinzy/vo54eorzyHipSsAAFcuIeJzr7xyaVmBvHQFDi3pAC5fRLx05Ufv
f+yoiPjcpUt39r/6ynOIz71yFQ7tir/6gBzPM888C6+/fRX6hNSzzzxz7GfPf+3lC/DaS1+/
cuXrL70GF17+xt3h0mFm5/5R1NH4jo96eTlX75qWT738/tLu2fPuA1/69t1nvHIJ7/ng3RNy
hMvfugL34O5T3HXEuwe//L9PPVd/dc89h59yqPvOzfHBfeqcffpCf+pMPmAeHjC2z/J9+3wX
5QQnuIMfIX+gqszcSy0zMxyLn0xPkT1e7NNlkklEsO/oBhBmaw0RJeGjaqCKWiIETCF4Mr91
/unHy6HsHnCunfd125z91V9+JLfXbl6rb+/aWbM335rf3KOcCuft2rqMOLZBQkRmDzoCR0Cg
ahGXb/dIIpBERJUVIy9NAQWQHXWcDuaz3fl01nWCUBbFqBjk1mfGDnyx4myJqsa2hLtNs3Vw
kJiHeT7x2cj4UhRS2le5afVDTbdSaIxxhsaijtmotPuz62+85VZHOYGPOySaq8+sqZy3UWmW
PGKVZ+NBlY/GNFzPMGUJMUToAkXGKK4NlhUUmUXbRax3ZtODumlDF7u2BeHK29xRVpHH/W73
1t7NRUBtiWYqnXMLxHngkASjOMUMbaG0EpsVlKHzA2NL6/xhpJtZl6N3ZDKQTBYZARnqbRDB
OEaQPuIlVQRgdB1Z1cyqGASTgCNFMZbIoFgSg4LWmtw7b601BgGVDKpFcQCWBDVoalEDAiEY
JYPGGVKVFJMGAaO6SEAckAOrhIQhacfSMTeRA6ckSYUVUKhIhgJTFyQmx5AJaErICqyaOKXU
qswhdRw5JRFQEWZNzEu3R0DyVFk/9mVuvEVr1BhyTDZYG6oiVcVUeade8HyX68UidC0KZ4W1
mZIV4/JqaPI8igIYNFZU+ChXi2itFREgY5xDwpQ4iAaW2x9/fO3m1qOPnn3q/PnxZGKdpWUi
S1lEuZddRURS+PkGUstq2KeHMa99E76h+uqVS3ixL5+9+I2Xv3mnCHf17R+1/zm4cun8S6/1
6aArl/Dip5zk6tuv/8hRPv3MhT5p9s4br1145k/h+CeW9b2LFwHghT+9O156+zD0+daVV5//
tEvseUh3KEcXEe5Pmjl3/PKvvvLcxdeOiFovff3Ki3d/7NiFX33lufPHtuPFy8v82JVLePE+
ZdVj1oOfIZnWuzYvL/JOYPrAc70Gf/yOfvdBx71y6c5aX7mEF1/6k1f+8LuHa3pv1fGd+1zv
/Q/+U42t3+3e7xsszaf/nhblBL/Q+BGB1FGchIj3OFeo3rexSJfazocZKRE0Zvkj6BnqAACo
6o1bnQwffeTRzOd7B4sB0Sgfzm5P3/yrv3nsi09/8ewX+UzcvX2rWlsx1z66uXWrqwPnTlSz
onSl0RggRiNArMoSNSSRyCIKSZRZ1VCXJHBMKoAAhNbawlpbFqWhEKKKepuNykHl84yMQxpQ
Kh3bHLUoH4Zyc1LszRddFzi1B23dihoy0bjBID+blasOg7OIRmeBFy1xN0AohGm2qAodNwdG
YOB9QWbofU5UOWNVZXsbVbDKdGWkpG0X43wRZwsMqSTnxVKCvoLlFTzIyBrM8tbablSpp4a7
RarnGkLXqEo+rmxeWpEM1HjvyGRJui7FNmlkShCYP4R4TXmdcF1hFLhgjXUtHBEx83ZQlmWe
jQs7yrLceaMISTCJJjYKRsWypJSCym2H6I0lMhJ9hwWZDIkAFKHj1Ko6kaq2ZCG61DiMFsEb
EISozqBDtSpJBBGYJaaIqiigosyJUwJVERXmmBJHxqSGlZJCFBJSARQhBeypb2Q4xtiFFAOz
gkCMURRYJMYUOYkykvT0KUAiY5yx3uXWWmMskQEgJhOAkrHRe82KSCZYJz5Pxs9j3Jl2s3k0
TfAxJFTwmcsrk1eCXsFa68m4pmubLiRJIqxdQAXF/oo4xgCA3vsYw8HBrE1diGk+nwPQ+x9f
f/+jj3/7S1968qmnjLXMQkTMvGyMVejVPX/K2/snwWsvHfr7woO4Qxf++A/PwaFF77K29iB8
cv8r37p8tHlZRex3vVNw+Sxv/+effha++fZVOP/26/DsV+99Sh8e+E51cInLL730wnf0T791
J667B/3wlsXAZUTWP5E/Iy5/68qrr35a6ej4hR83Zz6+fTnwb3776osvfrKu+vyrhzFC7wP9
QFb0sUjqWBx1v3MBwJ0f3B99SfHqK8/hctWOrem9VccrX//0670fftqxHd/tzvet/2Z8fovy
GcZwgl8Y/AgdKThkO+ExppQecTg+LZbCuzeJiKgoaC+AoKAqans1ThYC1MhKdP7Xnm3Otru3
dm0+fmT9VGLZ+uDmtfc/3jx7euXM5urZxzafefbc7Z3r164d3NpqpwfhoO4ggkrvA6cO1QqL
AIACsWiIqQ2hDiEpM6YoKQqbNg4T+KpaGwyzyYojIiBDxjuf57klw4lF6qTScdSuidZYI8PS
VZmVlDQmEgGRsTVDwlaljaQKRCTkkhcrmDtHBIIYAVpDoOyCOoW8g8JClbvSO7B5XS+aWwvd
X6BDQmQVRdI8O2yUc6oUQqoDTxUx82zcglMLqmQCqhkUeem0nRkDK5PJpCoBTSvMZMC6kGBR
h9lsXs+b2EWJ2sWCIq+Q28RswuyaxpTGZmRL40pnMkPGOHACNiIKg6ZAoqiIkTUxAuTkM0eF
p6XqPCePlKm1gClyF1NrpCUwqsILMTZ4iuqFnEVvyGoKsUs2JoxJu2gCA7OklJhVVERZmFng
MPI2AEZUk5AqMChL4qWIgIimlBJI6BXIhIEFFBVQVBhJAJRQnbPgc0VnLBorSGKMkmVABlTj
wHk0FqrCr08gzwPq9mK+N59PF3W9vRvrVppoYk+qi9Ehudxaj+RQjUcvxhI6RHtrZ+edH749
m+331WrnfZ7lieNivuhCpwBVWarqzs5u0uS8I2sG1WA2n7/+5ps3t7f/2Ze//Bu//uvVYBBC
FBFzGDz93HSkPh/G0I9Gn2169ukHnerc088CPPipC3C+T0k9/cZrL3z1eXj7zz7tEBee+UQe
pA+tzr984fJLh+mUH3d49x12n566fBHv7ip88JH77XfFsQAPCFGff1X11auvPHf+pfuEgkc7
fvUFuPz621cBvnV5GRr+uOf6BJZtgxdefke/+85RC+GnZhB/3Jn8qcf26fjZLMoJfjHxoEAq
pdS/HIsKKFCvzHRY2kNAuNePeOkRcydr1Uc23FPLUVGXwp6I0isnKLTzxbW3P5iefuKXv3xh
QfT9778562aPPvzIuUcf7ep69/aNd773fXR2UqysbDz8xV97rDBUp2bn9s2D7a2Dg739nZ1m
Nk1NBxy9MqiisZl1hXFclDSxYEBRk4QQI3BwnCyCI7KYcKnQ6BLqTFIUCSmJIKFBBWBGVIem
sI6MKnZExhFYQ0C9fGJGaBkNK/HApFUhVqtiYtQYA5loKw6Buw5SImYSpdiht9ZSoQE50Rys
c+AoAbIBRmyZZ82iTSKCkXUWm91YCxmT5eo8OO+5yHKPotSlIVpPNIhpFAMC4mxRNx2iMUFs
4BFrIRBZOHDJODF+7PIVl5eEdjz0mSEP5FRRQRKwpqgxpE64V0oVTYpgS2tdDgYXIpLYJMpc
lg0rtKbt2nnTUloKWTAqE7akTbF0mjNBfFLbRpQYUwipqzkwR9slF0VFmLlXGkMAw4Ix9bGI
MCsiq7KwqrIKMzMnBUVRUVVhVRUiA+QRDRoiS4BkDJBRQ0oIhAJIZIFcEOlE2HrN8+S9+twO
BpiXEWChqSaMi3Cwt7d9+1Yzn2piSAypy8lkZIGks6KZI58RGFBEVsPgvPPGWWtnzWJntr+9
dZvAOGPHk7ErTAgpgklKokq+yDI/azpIXTmosjwfDAZt0x5Mpz98773b29s3bt783S/97vr6
2lESF+56Y/mniT7COey4uw/uqkvd70B/+McXvvn2K3D5wjNfA7i3pHg/LNMmy+TDJys9n2l4
9x/TMod05RJe7Md/J5C735GXId+PF8cux//gR/vzX3v5wvlvfvvKH79+lHP5Sc51DH3maRmL
vAMAcOGZ8/cJfH/cmfxpx3b/A/+sFuUEv3B4UO2gzyQpqoCwMguLyp1fsBTbVCAFVCQFVMCl
0ACQ9K/+irIU7URURAUDCCLKTKBkSBFme3s//P4P/vt/+A+yt/fbv/Mbp5958ofb1/7bG9//
4MatYrx59ou/+YUv/OpwuDK9dfvG62+8++abt2/cKrLi3FNf+NKX//kf/C//6x/97//uj/7t
n/zL//l/+63f/4Mv/s5zDz/5xWr1dFasZLailmgqdi8ODnSjsRtSjrJR7gbeFpkrvCuMzQCN
IrIwC4Mx1juXZ3mRe2soBuwamC9gMXMxFKilgZzAWkHPzsbMpdLGwoYqT+NKq4EURcoLKUud
+HYjazbzbr2M6wNerVLlu9w3BmfMe8bWgzFVGwM/KbAgMZwkJG5RUmlpYGBoYLPMvrCx9qUz
D/3m+vjZUfGFzDyucWOxP969Nbx9K7t+bXB7p9rageu3uhu3481t2N6zewdud79aLFZDtx7D
aZRzVfErmyu/eWrtV06tPrlRjUdkK4ABhkw6o1FJE0KLXIuERJIsgnfGZ95lmS8y8lYMJFQm
ZYIIPA/t/mK238ynbX3QNbPYtilxYm0jHzRxb9bt7qWdXbi1Szdum4+u47sf6zsfwg+v6Ue3
9Oau3N5r9/fns93pdGc+36sXB/Viupju19P9djZrZ7Mwn6e65tlM5gdQz7CrTeycJI/gEKw1
ZZEPR8PBcLg6Wl0Zro4GK1U1qapRNZi4vLIuR3SiFILM23SrC9fb+mYMB8aE4UBXJrGqdmJ3
9eb1//7WD/7ye//fX/31X33v//nLH3z3r26+/ibe3BvO07jhCcvIkDOcTB2zDh1bUodI4AAc
gBMwaLwv8rpt57NFVVaj4Wg4qIajwWQyWpkMx+PB6srwoY31hzfX11cnq5PJxtrq6mS4MqpW
BtUoz0dFsTkZn3v8bFUUb37/9f/rL77z1ltvMScyqKDMnFJK/I/Kce/c08/+WPs//9UXAF77
5revwiEda4nj3Ornv/byBbh88W4+8D2M33NPP/vaSy9d/snSR+de/NMXAODyn93LIu6H99LX
l/z3P7t8SLO6Uy2Cq9/+5tGwj11+T06+dGW5773psPtd+PHtd/Gg7yab3zn60QgeSMeHPtx6
7aWLL712VBe777l+DPR8935qlhPy/FdfOJrLIyr5fRf6PjP5eYztk/i8F+UEJ7iDB8ofOAP9
OzIiAAgI0TH/Yer/JO330KNeo6XDWN+qxCDA4gSICFKiXkOzl+UkjaiCvJembZjO33rjg2sf
PfzEE+Mnn/jSrz0T2e5vL7Z29vm9vQQyrPK1lXJsxtJ1tGhkutdYb/NcvFt4l8rCrK6O/dmH
s0yyHIAwcurabj5P9SLMpnE26xaz1DaxbruuCV0XY4gxgLAng4kzVQNAiGI4SgcqyL2HjRgA
IgJAJJMAOUlkgYgBA0JEABClfmpEVUEVUISIRUVFBEVQBAQs9RVSa4wxBEgtg/TKW4ZK75xz
RAQKzNxzoh2gE0uZAUNMwACsutQZQgTV3iIXFwCIExyoW7bdA6IhQiICoqQkEfvkjyiogCj1
cpHcc8dAhRHUqGjPNFdA5l5kC1QNoAdQ7ocrIAoAvs9WivSBM6hmIqIsmhQEVJNoBAVFFlUR
RZQ+SdnvwwJERFZUhAGJgAiJFBAAvbIja0yvp0qKhMYmhaiqxiSQlKRR6k2IBCCKRE5qMKpE
ll5ZqhNuUowKgojBY7vQLWy7LsYoKoDiOVlQRAtEaAmEeqq6IiAZMoRoAYWULYslZONbcQyO
yEQJE2f3Z7OtWztZkZ174glDgKh55rIsG+QoknvnERCACM0wO83aqiRQQKChrxCGaGwYMat2
Xbu9tTUZjyaTcZZliIho8OeiI/WTo08fXcTLF15+5xufZf9Xv/PC5Ys9y+flly/AS59WxDv3
4nf16Uu4LMjApwpn9g/kTynffbZh9zSqTySlnn9V33nmufOHZz467yGDHS/3wz6SWzh2+e+8
/Pr5l/pPXnj5nRfP3U33ev7Vd15+/fwnL/z5V/U7gBcPy0j3oT4tP300I59FA2uZuDrGL/ps
57ofjvHdAV54+WV46aU+m3Ps0gAOOyLP3ed67zuTP9XY7jvkv89FOcEvNvAB5YO1SXGcbL7c
erQ7HgZSdzbA0s1Y73xEQYCoyDNrCfq+cABUBZA+9iLmX87cxceeOFeNBgzGuy7PaWV95eGz
K5tnRyurkLv9ehb25jxvF/OZdp0BADI0GeeTcT4apjwPhlCx6sSpRiIos3xzPdvcMCsD8AiO
ABOERuoW6pDaNtaLbjbtpvO0WMS9g7Cz223v8ryxDIZUkVVEk3CM0vN/ARQUkRhAhEUZ+4hJ
ZXnV0pcvdRkZgUYICbmPTGAZSACRQUJUYBEVBmE8Uiil/ieQOCGgtUYBkNX0fXMIgtBbGsPR
mvQ5wD6vuCzCsoj21j5wpD0hYgAIEA4FK7B3fRMFEUACWGqC95aKhAigwqIgvacJAmpP2TaI
R4dlERYF7aMoUQVeilAp6tHXARAFEHo1V0OqSqIGARTRGCTTJzUBCRAVQQFVIRqIIJxYVBRI
FaJIUlRj0Dl0zljLHGfT/UXdKAIgRZWkmIQTiAokkSQcNQmSIAFYJWIF4V5fQBFFmBUAyAgS
AoqooukJV9YhGuxb/Qyqdc66Uqhsk1WX22pkyqxYG+zVe99/639M1oZfePJ0blWUAdQQ9GsO
QASIYESBUyIrnKIkRiRjLKERhTbEwLyytvb4k09mZWGtGQ4HZVnleUlEf/Bv/t1Pe4v/KByv
xf/8cKIl/UBcfeW5P4Fv/FMoL50s9An+CeJBGSkC00uQAyy1yI+Lm6sqgirK0VszHpbw+nph
/8BnVLAqxEqIhqAvDAKALm2PDel+O79d7z05rKxolpLtUtjqdq7faMzf7lcVDYvi1IbZPO1O
D1fyDYvUNF1Td9y03fRgur0LYIqstJlfOARnk7MKMd+jUtk3A/DGVpmrcvQFjSvYtB7IA1Qg
AAzM0LQwW8CNrfqd93Z++EHa3ck5gIgwc0ySeNnRjmisASQA1cSakiSWFIWTCiMLqqKKqoAo
qqg4AAtL81kFBRZGJEICBEVVEugFGAEQUXSpGZEgAQABqaqgIEGv2SjQazj01m0KSy/cZdDS
/y6SsnBDAAAgAElEQVQCqsApqSohKhxaS6v2R+mZN6iKuGS82V6lQiRJ6FUisI+RhQEAmAEA
kUAEQNWQ0FLzAkEMgqgaQ4h0FN/R0jXOABESLuOrPqoio6CBJQEgoSiwAgMKAKuyoIICIhhb
o0Q0riyc86LQhVi3bZeScX44WcmrgYJ0892mpVkrKbGAJhEFSNxzi7QXOkdCIEXs+XgIoIpI
BmGZ1wMk7FtMZcn8EyIQVEDqXwoAANAoWEACZVRGQZLGkP34ww/evfnxnLkUK6IxtIkToiKA
MQQAKSZUUIEY+8p4FGUQ6F2SmIVVo2hRVpvr6yuTcWJm1fl8wSzW+qIofqqb+x847ni2vPr8
XSKTJ7gHV1/5k2/+8Td+dMv/CU5wgp8LHhhIHTbnHSaXFFTx0LdCFRgOH+GHwEPW1ZHUIIEK
MzCRsYg9r+qeXj/cDeF61zXOCohhMcyO1YDHdo5xkRa6des9Q4MsGzVlaQZlNh5XK2NYW0Xv
jfOObBRtYqzblmMrddKDNLvGewwxSq+DlXnnXGatc1lOhRdvxKtaIkfOGgQybQfzJiW1MVbt
HI8FUn0eBwmNdcaS9LpKBGpFMIlGkYAoCAwgqIIoCiJGAEDkUI4UgOiQpA+ABIomoWMBEDWW
CIyIMAhZIyIhJSQwgJTYLH0NgTX1saz206iqKqDaNy8iLjWxEycRxj4i6tNFh7lCOuzExF7K
vV9lBEIhJOn3WXrpIAAAWUDoC5cCytbIYV6LEFFBVBAR4dCTEVGIZJmt69l1mjjxMjMHIrog
iN4Z54CMICgSWgvL3jqIwilJaLoUkonJCCSWpm2bLkQR7cJ23TJo4kjSgoqqKiGLMqiqMoiC
AoAcJsZUVZkBUVAEiQwBgAIkEUtIZLj3aenNtWmZSetrtApgTL+13ywIAhSBwnS+c+3mjeli
RoPKZmVMabp/O3QtGFTmPl2bYupDzJT6flIWScrS+0SzCCvEJA+dPp0XmbEGiEgYAOr5goMM
h4Of+Mb+R4B72qg+qVBwgiXOvfjd7/68x3CCE5zgvnig155on4k57mGhsoyDELAPjPrnVl/R
UwBZ5kLu7GGSIrJiQiJU7cMJ7XMsoCLY2mILXDdY6Zqu5MRNQyqW1CEpMxsgC4Yb0zLIPCxg
ekM7VTYGyWY+r8phWVRSeFwp86ocjCvjKwZkUUsOgVChZWlDClG6WMvBLIQ2xMApQEyuDa4J
dtGaJpqQGOst06gKKgDLMkBAUASgts/IoSL1IaEKSu+owqDcx1eoAgjKIEu7DwQ0Kn0ocTSX
CCgAkQQUVCMfRjmoqgTgEQCAhSMIqwFVEAZVBBVJva8tAPQ+PCCKKEc5IWOQyKgqEvVFyV4h
FRCoj3iWH8cjy0RFABU69PxBANNrfkEfD/UJHBJEkUPrk76EqSpH0SJzBxqhTzuyaC/8pCKs
S08gFRUtSzso86JAa0OMTQgcRDBGlsjcphhDcCFSl4T7I4OIoCElA4QpNgygoKzLwqiIqigq
AqiBZdMhqYJCUgYyqiLAagwaNMawCkiviyYsIkq9HyIiLBds+QXvy6NIhtBaRUggYhEtTpvF
jf29ab0wzuZlvrq2StQ4Q+AsEoClPsB1hhDJoEUFARBkVdaeJCYqCgIoAAjy7rvvAuHGxqbP
cgCYx8V0ejCfzz6fu/wfKk4cOn7xcLLmJ/gniAcKchL2KYplJgkA0RwLqw7JUHDIB0HsYw40
xiD1kj/QP83RIBAo0TKNoj0VBgCQsBZ6fxpm2eB0MQ7TAyQDibuomtgAAAOhOGULCZJ3RF4x
iUgCZtG0l+TGXCSJtgaE6LY15Cw5S85nZQXOu6I0RUVZPsiKLB8YQ3ZSYrYC3gqCipAIRsGQ
IESWlIRZJKWUEjMzqyZVUe5pN6AKdaCQqE8spKjMwkn7/IsIqJAqtQxJlirhy0AEj+SBVIEk
mTgnEELCYxHMsr6GBKICFJ0SEqGiELAiKKkFFSRAQhAl1T6EOlqM5W8iS9ZR30ePeKhEoarA
wLpMIi2jhyTLkEWW2l/CvMxaifSpryVhXfvaGSwjqV7DABGFJSEkQ2nJFOvzkqpK2vOtetvq
usVuuzZWoWc+gRIJoCgoogEFESuJVBSEEBVVFFRYQUQBUamvxKHVnn2liH28pagKokI9+6sv
ah7aBmNPHTv8BvdfX12GTrJU3e+1ZPuTSupp9GhAQASVjasTT2ez7dl0nqI4M1kbP/rEY6vj
IuNQjEacIhIsq6acVBSUCC0qCihDFOhp7n19FVhVAKPIRx+8v71z+8yjjz35+FMrKyuT0TjG
1Hbd53ajn+AEJzjBCf5+8EBBTsT+wQfHHtJy9LxH1b4fCqmvV4kqEYJCF5K1vaAUsIBy8miY
wCwJzwhEuOzqA0BM6rK1h+zKegqhbutB6TQEaKJwkBQ1QR8LIAEpIyKBGgEWEVAkRBIRpigZ
EybRruvzFAAAYEQpADIQAAqikCqRWhLn2Bi1Br37/9l7lyC7ritLbH/OuZ/3y38C4F8UQJEU
S6VSteRuMLqr2263mijbVRG228MKhx3ApB3khDMNK8IDTcChMNPEEdZIDreIQYer7S4RKnW7
yl0qFksEIEokAeKb/3zv3XvP2Xt7cO59mfgwSbZIitX1VmRkJN5793fyPdyVe6+9ljpC59k7
YueodFw6Zu89+4zZ5XnGeU5M7B07h47BkTkEYkDUZHfUTrRJut+jGgWDKEkQlvwiCFHNWvck
VZKGql0LTYhBVFPvLT0FXf/OR4G6CVUVqgpiBFUQURFTsVT0SfJytfYoZgagoiJp+CwRKlVI
UrbUCbPEk1KTTmdNLGCzJIs3VUv2FgqtWL7tD+JsoqB9YySTi5SMYkAKJmmaEKjV6LcGGWqt
DolIFGNjGNSAAR2zkYmkXzAlehFTwzIVQAEMNZXejNpuZaeYp1Y/1lXj0jiDgmmiX+lRwmT8
hcyzcQiFpBubtZnTOzsVGhGQUlILtH88WCNxZ9rcnTZ7IUbn/WCwtLpw/NjqoO+dThwKOiY0
ZiBCNbGY2qeIyAhsphHQQJI3bfK4BYNxXQFRE8LOzi7yB2i0ubm5tra+uro+GPwn3dqbY445
5vhPAkcRqSZEREA8nFisredme+OEaIBk7YyYGpohoc8KZg5NKPvlaGnEuSvLstfvF3lBxD7P
y16/LMpe2XfOETERLagys6FqxhOty5zRAJsIiqZB0faJmdgjO3SQ6iqK6Y4JZJBcIQERkYFS
McdUVCMhE3pCUEDDiFCBAjXoaiRjABQjMRQAIRSkGkBNMfmFtrdxSNW0ZPNoiOrJHAGSIRmC
IRISIRITERETEhlyOh/HnNYneSjAQT3EYirmqBkkAwW0NC5nBgimhiJUhWp/P1Y1RNFQg0RK
oXGgmiYIUxv1UAxiW1NKvBbQ1FQjYuI0ZgYKaq2GLY0MJOrQ1qZStQkMFawVaEMr1hZkSDvt
JuxEVVATrzI0M2M1slbgjsAG2LY1FVBNQSMKsAFSa4KBZkBKoCZgaIZRNKJJqtu1VTqUVKRD
NABBUE2d0KTFMlMTUEMTBFUzmrmdExIaYTph6oqoHX2atfLaNwigJkEbo09KeY0qoa6mcafR
jdqqrEej0XBxafXYyvJib1BAhsHqPbKICEjoCJEABA0J2cBS0guaUdKjoQEYpfMOqmVRTOoK
1FaXlx59/PHRaElE725sNE0YDoef1ed+jjnmmGOOTwlHEakTX3qambz3jhmRuO2NUGqSMDMy
uyzzWVYUhXOOiZIU2jtXVZWqnvzyyS+d/JLLvfM+y7xzHpEJyXmX5UXmMiIU1d39vV/++Z/f
vvKLp5cWnecmWJNzAMkaziK5xpFagxpN0BhQk0yJUE3ULLIJWmS1qGlqDgCMEIFJU5MFgiEr
gJIopjt+0m9HNPKgGTlANIOoChSBkwgHjADNVFVUQSxZFiWq0k3EARAaQHuzTLWnJEsyNEMz
rVMBQg809gapmYTBvLFjaqnqTGzUUqLUd0PTGJOy20IAESLAtnE1e6m2Db3EwKwbE0j0KNVu
AFPIoQJYyz3B0l5aHqbJFSxxUjVtS4gwO22ARBzbS0jlHwU0bVtomnp+SVsFSDYbF0RL9SNU
E0JhBANBBQNETWU1S3pvAwUTxUSTpA3GNkvZL+nkU4XMFMwUBAwVxMyMrLUoSx06QzJEo2TS
0NahCFs/CcDkwyUIakagBECGrAREBD4ojZswlbiLYQ9iJFceXz2xdqI/WnaZi2E6LHh1Md/d
ukkkDtFi8nFAA0VAZDZVhHakURWIXPvrSoyWIWOum1hX1XDUf/zxx5YWF7KyKIqSkJoQdre2
Pu3P+9G42GV9zKbTH3zkE+HDZt0/xgx8O9N3GEf7S3/8ufov8gT+r7Fic8wxx28IRxGps//L
v8y8985TR6BS3B4iOiRmBiZiZue8c0SUukjpNh5DYOaFxcU8yyS2Bs3W9ZWSmDe1TqJKJBhr
dDt76ssFN9iKTUQE7zEzdoYxoESPhhg5DRAiKZqgGgkm5bUpmkKrpAaA2X2+K0CYkrb9HGJG
AIQ0vYXtXU3T5H+qcVBn12SJfSTXdrYZgSBJbSFCBNSkTE969JZ2oBEporW9JwBKTAzagwIg
ApoiRULE5HRK93Ap6IyYFFFVVdScAacUHjqk8hY1TfShM6ewdmLPuqMRqWGXRdjNDaIhQBp2
a8VEKIioSJpacZRe1llSdS6l3W+y/an91aadG8RuDYiSxxQkW4jZd0NEQUBMtELATNvOIyi2
VSjrAlIMLC0QmgCAtYU7NVMTNWjbywBkpmrcCvDafrMpkCECKCqiIdmsI4lGiX8iAREhsZk2
UQzIjCZRtqXZA6082eJSb33l2NLy8mBJJjHsT3R3L4y3tRTxAwcR2UDSjlJDnAhbHSDMxi5Q
GUhUUuEUiFIBT1WKLFtfW11bXsry3GeZI/SZH/R6qZD5m0AXlXH18ptw+vTpSx+RMvvZ4VNk
Dldfe/G7z7wx5yFzzDHHp42jiNRXnv2KIzczYCQiJmpttQ2IyAhFdWbop6rADADe+yzLiCiE
UFUVtUWFA5YwM6NS1RgjIz399JdvXH732ub2iSefGKg19dQr5MCUQUCpk8gYkAgdIwIBtWaW
aVYLiRWiiRyoubr9Hxyr1XeZqQKiWgSb3ahartD+2GlrUo2j816C2WsUTFpWkKRg1jG2A6RC
jx7st6saHaJ6XQEGQaCtu3RVpm4vyRw9CYY0SdQ0zRK266lq0nGadt+zzTsm1TZl20tFREQ9
oEGd+r3bpFWTt9rrVIFqe2CH+dPhH6zzW9duMQ+DiGBWOKODLTsK1wIP/fIMTak9N0NC7Opc
oIqYpiUj4cFYZKq0YWu9bl3vDpmMDAgIKDXuwBAE0dKUHxEgAwhgUJ2aVQQTjRODkHlcWBo+
euLE6spyXvQB97f3tn51Y+fOlidaXV5YXlrtk4/7E+faIVTA2SoidPr/lgQi6mzxqZUVKkBV
N02IC4uL62vrvaL0ReGyHJGZ2DmXZRl8/jh99ixcaLPbrrx16fQLZ+E3xqM+CY4cB+sKOh/9
yjnmmGOOT4ij/uSNIYYQmqYJIYiIiIQQ0iMzzB4MISTW4r1HxBhjXddJIZz20BogAqTOYHt4
IkICtSe+9PTXfu8fbvV7VycVLx7vF2t9GBSaE6J6kULVoSGlSTEFUERAVEQjUmIlVuQD3XCH
wyEbic119qKJHEiqZ8xadgBgap0c/KORCESqw9136NQRBFETNdE2PSX90H2piqqIxPRdRNIj
6StpvkXTLpKk3DqptZlppxQHOERo2qMfrhi1FkndibWlOlVLBu3QVqRaK4FO7d653re7SlW7
7pLtXswO+tA30v0vPhiWA4DWnP3g5/SbobYVaa3APSnKFVE7m3c1SFq21pa0cyaltHs1EzMh
jaQKJu1D0ESpVRujinDqaM/jhofbFK9bc8fBeGHATz3+6Le++dvf/va3/st/fvKrX+8Vi3vX
7978/9668Rdv7b5zTTf3skaHrnhkeXWxP/REECMIECWrkA6tbRfO+uDMjhjRITsix0hkZk0I
vV5vdW11NFpwecbOETl23ueZyzwyH/Hx/Mzw/PMvwJuXr0IqSL3w/PP3PJvixhJmoWOzWDU4
Ko9stum5H93/6Iv3J919KNqdHAqhazd+IJwu4cXXrsLV1/7olUsAcOEM4rmL97yyPfPXzt1/
SQf7ePHcuYdeUbub12bHOnfxYrebw6/+1Fdsjjnm+GLhKCLFzOnecPDqwwKp7uf0lHMuz/M8
z1P7L90yJUYVSVslzNJLEhDROacx3t7efOrFb73w3/83H6yMrhrElRP5yqNQjgK4qIammUge
JRfNomQavAYHgUEQI5AodlWae5EOiQ/i4N5tBpLskABSR+tgBAweIAEdFWjH0tqvjgF0N/R2
ohHSrRv08Bda9wO0Lkzpu4qoiMQoMaqKqqqI2UNY3QNnk9qE2Llp3vsD0aEVSBojNTNNY3fp
H7Pa1aF93reaswG4+17Z0QZ4YGEPbfsQJtWtfRJq2aGkIWgjr52BU3CGZMYGbMBmZJrE7Gzg
pftScGJs4Ay9ojN0SiyEBoIW0WrQGmyKOGGaeLfraYPtFsk1bN5zzfUFHj+61P/qyUf/s28+
/0/+8W//w997+rkXetnozi9uXP+zv7n7k5/v/vy98fXbtjfpAy/1eieW106sHyvLIkokx+gY
CDBNEsw+LA+845jQOXaek7k9IEQVJFpbX19eWXVZxuyQmJg5jSyQY+eP+Hh+dnjm+dMpSfbK
W5dOP//MoWcunsOk1DGzK+dPXzjzCQJcL5479cql0+evmNmr8IOPWeO6cOZgBRPtOPny98+f
TsG4iR+1eW6H0D1uV86fhkuvfPdiuw2cfd0e2im89AN41cxePwjcPXy234ELR5zthbeeecPs
yvnTABfO/PHzV9LKzHKWP/cVm2OOOT53HEWkrCsbEM1UUkiH/4GYrJaIKFGodJtPm2MK6HXO
OceH/rZu7yvMzGxmRLS4sDiN8afvXoFnn/jd//F/yL7x3DvePsi9HD+enXiMe4sgmYucCWbR
MpEsSq4xM8kwMiqiKESDgIfk8LNjtdOG2o3CpXaZHZyntRNlnSbpUJpcx1Lw0BW3Mio4VO6i
NNhoZmrJksDUOrmVQWropFm2rkuIKarXNKl9DmgWWpqHIwKiJAC37oTTi0U1qkW1aCAAnS+A
IaTEP4UUK5zEXYiMyIiULABS+ymCGkAqfAHMiJzMWFpal3S1OLvsB9lB19bsVjhJ7e95aVq3
GeuC1u0TGTr61a0k6mz9gdWcgBNgMa/ICh7IIbGxA2RABGS13Kgg74xYyZkjIzQm9EieKDfM
pgr7anuKY+fvot00uWbxBsnt3JrjC4Nnnzz+O8+f/ObvnPzG15/6ylcGo4Wtu9tvv3V58+bm
idFKde2uvHe3vzFZDJQBjsreoFf2yt7y6lLRLwSttlhbtMxFUtE2Sujw2y818qwt5elMes+O
kDCqDBdGK6srvf4gy4osL4jYOe99zuwB4KAd+/ni1DOpJHX18pvwwjOHkoAv/vACwOnzr86y
ew+Ix0cibZtCc0/+/r84PXvi5MtvmH2YiDwxkITuNYkWXXrlVAqY+TDV04UfXjz58hv2cOp0
L9rTStnHb12572xfevX86Q/fNsXanHzmhYP9PHDVn/KKzTHHHF8sHBkRQ4TtPBrMQnDTDyk4
pZEYYizLsigKIooxHi5RtPgYCfaOeG1t/dr+nX9/5a+/fPzRF07/vf5zv7X19i+3r13PFrxf
KPLtTdzcCvt7MTQZUeYZUVWCtMohJOwObKB2wAYekOtimnI3O1xpm1URDNLc10ed8Mz+stvS
WpW7tdpuQ8A2q6QbnGtFMtpWj7p+Gxoka6Q2z6UTnFHqyJkhtcqjlMNy/7khdM2xdh8HDTZM
va6DfUJSEQGAHRJs2UFf7z60i2it+MlM9ZDoKz17sIiI6Qw+cvU6uTskd9bUWzUASbOWgGrE
HQdNAnNEVEQ1VDJDVIAIREhApIhBJSIokQI0qkFVUNUsIFbsAxG4TLM8eOeHw16vV44GeZmx
Z0TTGGR7srV9e7o1DnVUwEDuGydf+N2v/fbln/7F7TsbuScEWxiNjDkYDhYGjcSN7a3eQmEO
xRQtGiUjjo55H5pzPKCJiAaqKuxdiM20Cnmer66ulv0eMQEhsXPonPPIjIhIjv1vpiIFp1JJ
6pm3Lp39w5fg8h+3D1+9/CYAvPDMf0zm26+z7QNow2UuzSjKQ5/usmfOvm7fO/Xgq47Cp3W2
X5gVm2OOOT5DHEWkmDmpZeFesZGIxLqpm1oR+oPBjEUBAHUOCPds8lHMBA1A5dji0mRK1996
O8K7J9effOyrX3Ff/+r07u3p7Vvh/fdt467sbFV7u0FCLQ1UU6zRYqCorg0ANECBNinFOveA
ew/UsYrOGKubcbuHS300E7hvQUwN7+WLbX6ztftOhMk6ltGpqm3WOSXsyB0mdwcA6IRbeFiE
/aEnYzaL2usOM/tmiXzYgQNnZ/hps1F8u/8SAMBUrU2gTr3PTod+SIM1Y2ntIOLHIVL3riG3
3k2tnB8QwFSRCLFtiyIpohAKgiAYkoIGM0FWpGhaCwSySFqJiENzDr0j75CZ8nzUH2ieNewc
uXI0UtVQV9X2voz343QKoeE6ZoH6DRKQElXexrduvfPzN+tmT6kml5U+E411XSu7qpoqQ0m9
cuQcJ98qc20Zs1u3jgV2awLdHxcKBAAYYgCEpaXllZWVPC+BmIAJyWcemQ0QsC3lfuy1/FRx
8vf/xekfXH4NLpx+/lWAy7OHn3kB4FI30fcJd/lrbPsAUs/r9OlLl145de6Zh5ScOjn5xXN4
5sKFH1783qu/mbP9wqzYHHPM8Rnio+erZzfO9D00ze7u7s7eDjMvLi72eiUitnNt95GJrjb1
oODmwUN4pKxqhvvheGXNL669demnf/L//Ns/vfL2u8MCv/Hba2fOrPz+S2vf/mcLv/ePsq/9
lj35JTzxOK+dcKNV9EODHMSjuVQDSlU0x+yce+iRH9ap6pRFidl8FBe4V2P1oa/Gw2Krex9E
S5PyBmjJggoPKa1SXpyhGshBpethX2k8DQ+U9IeuZmbmMKM/HWecaZW6ZtOHXifigRMV4qED
HXQ5u80/uvJ4CLONOi9TTsashITIxMBO2AlzZG4Ia8YJ2p7prskOhG2LmxpuQ7jJ8ba3zZL3
Forp6hCePMEnH/fPPuWfe8o9+xSffMI9+uh40N/J8mNf/eq3XnrpH/yzb3/zxX+0srQq2xO9
s9ffrvuVmRmalcR9pTJYL+jVv/zL/+tf/Z+TjdsDp6Oclod9z2SiZBDrQAYZO1akqBzVR6Mg
KDqz9eoGJQ+z+eQ4Qd47NTHExaWltfW1oizYuSzLfZYxO0osCoCIklftJ1nTTxEnn3nh0iuv
XLi/HvLSH54FuPTKd1tB9x9fADj7nZdP3tMUu/qjh6t50rY/+NFVALj43UMOUZ9QbA5w8dyZ
C3D6/PffeOP1szATI93zAsQk3j71/GmA08+faltvHx8ferafEJ/Jis0xxxxfLBxZkSKSKEmq
nKjSeDy+deuW9/74sWOD/oAcS+c4MGuizRp73Q2+JWvdoNzBi0y73geTxLi/tce7+yOlMtre
7ZvTD67X7/9iZ23wy0G2duz48RNPjJ58Yu2rz2USpxsb0zt3641N2NzWjW3b3tW9iU3GMt2B
GC0EjJFiJFNWQJMUNgJoCKzKpmjJHxLREKR9FozITFHRASarBOvmy9IlpQaOpbAUIkivIQBA
Q9QuxDeVhFiJTQ0AFQ1BjdSMjDoPcQMAxdYaPVWeEjFJ/2zH5FCFUi/OFBUQ0nA/tHp2AwAl
tpQBPZuL7BqHM/4UkSSZbaIYmKbEuWQGn86GkFoX8cSIEIw4mUsAgCEZCIpy8rsEMCBFM1BR
xDQpR2YmAMDtL7nrA6KBIQEYmimQOUWnIGCCoESCYM5HwEbFmBWxiTEAN4YRVBCFMSDUpkJg
3lnmkBmJochdnrF3wITMRhQkmlnTBJ1Om7qOVaTA4nyxvvp7v/ePv/1f/dcE8Mu3fk6bG3bt
g8nWrmtif3Vh8anH4rja+NV1m0aH6DPvJOp+PXIuy4sMjKRZHPbyvNivY2SH3vt+gTmDtMo6
h17at0pLb9N7HNK7P3m+gwIYAzehyfNybX29NxgiO2RH7J3L2DvkjMDAgPk3yKIAuvv86efv
a4m99D278vyLp9qe2YHL08mXv3P2lTMXzuAFOHv+/OmH+iW89L0r59889copfCW95mMxg649
B7PjwbkzMzZy8nuvn71w5sKZc394T1WqPVLa9PT5Ky+fBDj5h2fhwoUzeOH0+Svf/xgHful7
r5+9cOYTnu1D9/M5r9gcc8zxuQOPKKj86x//2MBijERUVdWNGzd29/aWFhcfe+yx4XCo2hpc
0r3K61lTg2BWkGrn4NRS8AqZoZp2AhIc19XeeK/a3/V1vRJ1cW+qv3gfbt+mpmp8rEqoB8Xu
YJVWji0dW19YX188tr6wvFQUeS7m6hD39puNTdnZka2t6dZmvbVj+3u6s297e1TX1NQmDaqg
KapxSkBDQ0oFLAOyZEUNCKLRgM28qXYUQ1QFUxsO2gRgQwaj5BOQGmeGafxvVr0zEiUwJOpq
R8mhEWYlCwU07hxGEZNDekuwujxgQzVsRV+gmgysWvdNbRmeYYpeToF7hzDrvgEokLYDewJm
ncLdOnspUzAlTrErXXoMEPGBG7uCgiiGpIyyLnTaFNs6l2Jy6LIkp0vECVJ0DEi6qFZgjykT
JhI2iAGhQgiIkSkSCUEwq9mLz5x36B1nGWWZctKRe/ROATRGq6KFUIemauqmaVQiqxZIUIUe
ORKxYLEBHo2y42t/9PK//Kdnzrzzy3f+j//tf/+rP/3T+sZNPx6XZscefeSxZ09tbmxcfmDg
zL8AACAASURBVOtvSNERMWJoGhUlJCBU0ybWvdGwHK3sTKVGXn7s+MqJpWCTpt5GC2RqUXRG
oawzt8I0MZBCuxFMAUzNEHl1df34I484XyA5IMfk2bdFqUR/U0UKEf/5f/c/fyqf8yOAeNR/
AnPMXcXnmGOOo3FURUpUUsbujRs37ty5471/6sknjx07xswhhNT0aWfZAGA2IndvNxAO7Byt
K0kliyVj58Cgquu93b39ahrNajJ05vv58S894dnL+zezqqap7N8Z3+T9veLabuY2+iUtLeLS
Yra63FteGq4uD5cWF768Puw9WSINQqQgVNXNxlbY3La9cb21U2/vhL1xM57YeJsmW9o0MQSK
EaJSFBIlQVbzhs5cw27CXlUQgQxAkcS1rbTW2MjAFCxSyqxLoXRggOC72pUhTDMNBIQp/Bc0
Cbzx8NIwgE+EMwX2JXl5W9XAVNlSJ5qUTdBNfakqtmUxAENDiJBs3bWVfxsYHkjAAKyQSBoA
UExNrUvOSa1BNTMlbFRTAkwKigFMQXmtg5MRoEImDi15hEMyUBDGYCKJTqIzIFUEIAWIaIoQ
wRqQCBZQjcmIppmfOKdJRM4OHAsTMJPP0DvnyDn2DtURI4qCiCBgrOpqMpUoiXlaI7C9H/an
k3oKhEVRlHnumQsiAuyRA1EgU48hKI+rX/30z3947YO3fvbmX/7ZvwtbW4tMuSHU9e1r127c
uRVjRDXnvUpAYkBT0mCChsAkiIGMUcVT0R8sriwX/dKaUAfSYMQMBBojtDS45a+iYgrgCJEJ
EcmpqcS4srK8urbuXIbESA6QgdJUZbKqoIdMbMzxOaMNqEnk6cpbl7rpvDnmmGOOB3DUH6P/
6k/+5ObNG9evX2+a5tFHH/vSl57q9/uqFkLjvffOB4mAbSAIHpQuUthdqx1Od7227mEGQEns
TOyQqK7r/f1xNa6aGINTIcHYLIT4NJePjAO9+0Hz7nXc2WcNwGE3TPZinHg3zbIdxmme1XnW
FJ4HZbkwyhdG5fJSfzRaWFgYDvrD/qBX9npZnqMjQ6ijTiud7MJ0sxmPx7t79fZ+2J/oeGrj
qUwqmVQWAkS1EKVpTGISKWEUCBFESYHMyIDRGqlDrMEU1BCMwMwUzShFuAEaQcUq7Yqke6u2
wqqOSSEgmktRuqkJata2DmfNuTaVBZKxgGKKWRFL/k+UVhpNIWUfa+JRnUtVp1tCIFFUBUMx
aYVXKX0ZUDrnchKDdnwPFVAZ1UyJDFHRVC0iBfSCqISAIEQRoVGJaBEsAgCiIFfEAKiIwiCE
SqCMkRA8oXPArOSNC+99Ig0iggigFmJUVYkSQqOxkqaKIVhUaWKcNhqjQyrY5y7PnM+AC4Fq
f38ynbJ3/WG/yHNGyphJtXS5SWzqpm6CMu+Emgf94cKiNGG6vUt1nZGRBNNgjAGSZwGJChIB
oYApQUQwQvIugpTDYX+0plzkw6Xh8kJvoSh7NJlsb925FauawCTW1lY4u6FRTOWo1jnDANSg
6BWPPPrYyvIqECIyIBsyMTufeeed8865RKSSOci3/9v/6dP9wD+IeUXqobg3629ejppjjjk+
FEdVpH7ykz+7e/fu6urq17729RMnThCRiACA9zkAhCCAaHh/Q+m+/5cNIbl2pqE0g2S9iIbQ
NGE6qWOIjOSBFACdq003Q51j4MV8KXsEFvLqzp3mzsZgZ29kxciTMgtxJJpOw9bG9tZkPNWg
iI1zm95J4d2w5xcX3OIA+r1iaaG3MBqMFvvDweJooTcYudUvF3k2LMpVciYKUVBMm6h1E+o6
TCre2+a9rTitq8mkHk/DtIrTCuqoTdA6aBQV0VBRqGOMEqPFoCJkiiqgolFMRFVBrW1xQZsM
h11EcKtFMlGtqJPHpzVpdU4zKTeYEiXTK1XE1BzVxKxQWytLYQXqvESTJTlimwUDCIBUcxaM
k0OoIqihERuipOgSA0Go2SSpeQwleYdTOzGnBGrWIE0yFxGBQBENUKlLsEMQNDUIBjWRYZvV
a+kLUcAETExElKO6yVTinokQgANyCM4gNmGytz8djyFKL4ZcxAOigSf27KVGj8gAEvYZqVcU
/bKYAvXYZ3lRusJRMrxkaQJHbeqAdQ2hEbACUHf3prtjbSKbOhNVCRCBgI09IhlHC2gWwWqN
XObgnDEKo2ZYDkfL68d9PhpPDXKmwhf9cnG5P1rsMdPuxvb+7m4UJDTAVsOGSAAght1vTVXV
5fny6rHFpRVwPtkdqKIhJJk9cfIbuceC61P8qM/xiTDPkZljjjk+Jo4iUu+88843v/nNZ599
djAYAEBiUbNZLWz/06fZFBgksUzXmGAiAEBC7VL2kFpdNSDEIPv74/3JWEUcMjp0hqGKPhg2
RMQTUZGQD/NieLy/vpBd35K7O7GuPEBJWJIjcE/kGNCFpg5NM5mG7e3pnoR9i3sO9hnGDpvC
B89YFHmvl5Ul9gY4Ws17g/6g3xsNeqNh0e8X/V7eL/NeL8vzbDgoHh3k7vHM+xESGzASC0AI
0kQTETUJgaoAVSNNkNhoE0II0jTSNLGpJcYQQqwrqiKJiYjEIDFKFIlRRS2KgKmoSVSZqEYV
FVMRsRSAo2oICKhmqEqiEkVTTqAamgE6aP0vzcwiqjjDtuahQQQQgQmZkBnAgmhFrkFOuibD
lORLmqLnzNQgkE6dBVQVi6YClhynBEwMollSgylhV3BpFfymQIgIqIklSQSpJYqogimIaozJ
8MsxM5NGCU0tMTpyPe97edHLCqemTdNMq2Jvv6lqMusJZAbE5Bw7dKxOOOn0CXOX+SwvsiiS
A/jMZ957RIgS6qZSMdGUdCMxGpgjjBISx8QoFkVMyTM5pw4ICeuoENVMCRUBHGLG1MuFgIus
vzgaLoz6o8VpbZEF0QSVc0eO2ZUra8fKYkh8e3vrtkkQFVHpYrlbqKpIjFEKcNFI0TvOiNgA
FDTJoZI/LSKpavfzvFA0xxxzzPG3AEcRqT/4gz9YW1vL8zy2+o/WjfPAM6AdvDuoQh22H0z/
VLMgYmaIRIapdRVD3JtMxpNxjJGIwZGaYlQfNA/gouvXtFBkMtmT3b3+qNcrh/6RQdVfqvb3
tJlW1TRIdEnPg+Ccz0o3VDvRSIhxKnFqMpY4MRnvN9NYTWU7iEjUKcKUs7GjXUZzZHkGudfC
SeawyLJemfd70u9Lr98re/1er8zLIs+zLMuyoszLoijyvMiLAhzRkDNfOsfe+cx5QAQDIkRm
QgIzTkomM1ATkdZPPbmHp5/EJJoJRI0xxCiS6JRIDCKqEkIEVTSVKDGKRGlCUFEzEFWLIlFU
dWLNxCIAiGnajYiEROFERDVKrLWOFlQhbaKqIcS2cKbplIBCwCCqIJrUWGgGom32oKihxCwG
TOl/UVEMokrTsBIZaFSRaBpBA0QBNQZABS9aAmXEvazIvVfRJk4Qo2fHyBZ3NMQmCBmUxAMD
AmRiY7JkiuAcIhoRM5Nzw4WFpeVlYp5MxvtbmzaRJoYm1HWo2+mHNAyAmEzkmTFF9nRRQUSM
psaZEzJBFQUHkOhPdKhM4Jzv9/qrS9DLsMgwd0AuqISoaqoQRRo1jWAOKe/1s2yY5aOy37t9
8/r2nY26aQAshLT26XcXW9N41Ft3Nx9//PbJLz997Ngx5xx5Zm5DZowMCUBa51vrcqDnmGOO
Oeb4IuMoIvX444+bwWGnzY5EAQASkYGiGbaOj+3f4NRRqNkoOCGnUkJyxIwi46qaTqeqlueF
c16kjiGqSkqmU7IaYjFaztC2Nu9Ob20ikGZ97vd6/WxS7Ycq845iaGRSybSCqtEmUpQ+C4sU
KgXAgqiJgqlGiW2tJFYSJ9YElboJ1VSq3b3KZAraADSgEawCnWI2pdwIkQiIAUGJ2GeceSQm
Js6ypsdSZHmWExN7n2U5O0fsXDIEco7ZWc7kyHuftY9wyslBJseOmcxILbPkDtEpw9Ugpuhi
U43SgExTp1AkRmmaJoYYVEUkxBibKCJWBauCqcYYQ0z5xxJibEJoc/pMTaamtYrFEFUiiJhI
mqMDUVUhsVwixTboBQxQEczY0CVqYmlyr0IQEEuiMVBgNTJmbM2wjM1YQY0RPTEBMhEBoIqr
Jq4iA/AOmNmCACghUZZFjKZARGREiMhsZWaZc96z80AoALUoe+/7xTTWk/1qb29PpxMLIcao
lmZAMVkwICIezBtiGp00h855jBqmDYA5QE+sKgRAjA7Q2MCjesZ+sbi2Ojq+NrY4sYiOmSgN
3M1ygpxz6V1tQOi4GGYnyszlfhJs69r1za2tyXiceGqaiyRiJlILG7t7H9y48+577z333HOn
Tp0cLgydz7zPoBslaJMDuuiez+ZTP8ccc8wxx6eGo4iUKmjbziOAh4o2klejHVSkZtNG3Sg/
IjFR0kGraQwSQggxqhoTA+B0OoVmShANMDJGAjHbt3rYsyeWVvN6e/L+tbg/le1t53w5KM2B
Fi44dr1eVgjWESrVqWhdT6tdCzXG6NQY1ZM5sEgNSMMOoCxysH5KAQYwwCASzRRQVBVADEIM
jcRGpA4hBAsWa5WpSjALpg1qNBNRhMigAcAQFUkAxBCdM3Ki7Z2zIRGydN/VdE+n1r6cmAlJ
wCRpiQCpsz84qEMQmpmYxvQIAqYRtlRpUQQzNQUDFuCYbKdSURBTvUu1U74nzwS1ZDvBEsmU
1BxgsoViMACIEA2NDR2xJ2YgNmQDMkxfgGbUIBsaISA5QgAyQkBsSTY5Ype0dAZIrdc6AQKD
qbZ6OTIAQ+eAGBLjKXrsvfMZkTNEQhQEYyQiVa2qumnqEGKttnvzdqreGZqyUmdqlSbjWhN5
xJR+iKqopmQAQMjOsxFZ1aAaG5JxCCoSFIzYsSdyVBMoMbrMmKumrkAdOI9p0DQQZsnz1LFj
doSgRqJI3rPnY48+WQ6WF1fW3r585dq1axAkI4dETOScZyKEgAREuD+t/ubty1s72489+ugj
j504tr6e5xkgIBIZtf4YB45snxsunsMzFw501Z/x2P+9cm44ff5Km6Z3z3Fnrzp4/qGn/eFP
f0a4eA7PvNkd9OI5PDMzvPrUV+yT/iJ+Mwsyxxx/Z3Gk/QGoUZLroKUYk5kjdWdRZKit9eCh
2b2U/ZbYlVoiAaCqoWmapqmquqlrE0EiU5VQkwZABaBkOCDIhrYVqsfWllafe3qvgOrajXwb
xrc2tjc+KIq8yHLX66HzAM6YpXTiTcSpOoiN1k2sA9YNNQFCJHJgrUcTmHklZhAEIPStPxMk
WZISBjOODcdGVQRRESNYQAwiTXoELERhERBR1YggAFEVvEPnolkQiaoKJhoJEYgETMzErKnr
FMGnodEQFSyi2Ox2SQTYRhAnQjQzBQ1NIyECdA5PnfEjAGDiVDSz54Qk804sir3T5JOQWAtA
1JBx5gBcNG/ACJzMURHAGRAS4GwYH9XIEK3TvYMJZtblz7Qzfgez+oQIDESGhtSSOEYFi2ZI
qRELBoBZxqlK5x0QJ+U7IIlaiLEOIYRYSx2lsRglCoimYYWkCWNCBjADAUNETFYNyAQIgJpS
WCCCIZoJiHoyBDBQJQAjZhCIUcgMNMbYKFnGmUMHCiriwIBk2lRNFPK5RS8KdZSm1qqph9gj
LpjzZGegYOnPCwVk70eLC1/pfWV5Zemdd1Zv3rw1mVSiRuSAnSN0FMwEiRBY1D64vXX77s57
124+9dQTjz/+2PLK8qBw3gytDZ8m/g1l7X0O6O71dog7nXoRHrj3X/zuK5cOONTV11787jNv
fCHm565efrNNFL6X5lx97cVTZ/DNT4HFfIEudo455jgKRxEpQzucniZgfDhYbdZ6wFSzammW
tYbZlMJ4U2sJESXGpq7quo4hgEZH6BwhIuWuJR7BQMwbOkIEB1VEg+HSIlfrW9P9HmWLWbm3
tTnd3pXx3u7tzf5gkPV7mOfC1BAImYkAARKbA8oIJKMQLQpGMVVM+u2QEuxMTAnRKTAoAQOg
IDSo5pkoa80kzcwIENWpSmvOpE4t3ULBko4HnTNGQwQCAQsqIkIKzjE6J5iEzmSqGTkUrfcn
zf5ERJQAEMVMVAGBmMmxdHk72mm667qJTVDT1iNeFQ8RWkBQbqe8EgUSsRAjExFxlIjoMHNc
ZJXE3b2dUX+QAel44oISALAZkqaJPTBEShYIAm0MXyosGqCYNRqtMw1ruVVaIMQDcTSysRmY
IiKhIqCpJf176lv5TL0PBlLHKFWMFkWDSNLSJQ+IaMFA0AwVUI2TtwN2duEIiOg1yc/RUrQM
IQKZRAPTKBqVCCmj1KRFYscZmAlFIEz1ViAD31pzWRMFDFAIM0ARjECoRtJQVU2tqafTuD+R
or+aF6P+aLnsuaoaN01NhCACkMYvrOzlTz7xWL+XLy6M3r127e7GpqIhQUBTZlBUIyKH5BWw
jrL57t1fvH9nbeWdZ5975rlTTy8PStfWx+gTxu78LUKX8jLjGvfMyH3IwFxHWD6nc/wIXHnr
ErzwnZMt1Tv7ekd4PqVpv3sudj5BOMccX2gcNV+N0LVIEAmT1w52tjjdo+nbof/xk1u2tVEq
LfkIIYQQVFVEzMw5l2dZ+5XnWVG6rESXITlkZnaAEKPUTUNEg+Gw7A+MORsOV088cvyxR5dW
ljLisLdf396sr9/S67fzu9vDnclw2gwr6TWS1wGqKlTTaNow1jlPSzcuedx3k6GfDn01zJpB
FvpeS0+ZZ+c8cw5UKDI4zTJzGbBHyogco88gy8GV6vuW9TXrgS/J5cgluB66PrgBuBG6RcqX
uVihYpHyHrgcXAncJzdw+YIrFvPeQlYu5oPl3nCxNxjlvcV8sFQMlovBSjFYLgfLxWA57y8X
/ZXecKU3Wu2PVnvDld5gvT84NhodHy6sD0Zr/eHaYLTWH60NRmv94fpgtFYOjxWD9f5wfTha
X1xcX14+trr86Nra2mhhqSiP90frRbleDB5bXF/vjYaULZf91d5oIesN83KYlQNX9F3Wc3lJ
WclZTlnmvCeXsfNMzMzsmD2xc85lWZ75PPO5d96xd85779l7do6YiRjYgffmvTlvxIIoBgIo
Yk0dJuPpeDzd3Nm6effWrTu37ty9s7W9PR6PJ9NxXU1iaEAFQcnMi2XRvKJDJMfmvWUe84yL
zPvMkUNAdSgeojfxFjIIGcQcY0GSk+QUCzSPBipRVIyIssx755P5BgCAgYqaIhOZSGhCjFFA
lVAJgB06HxW29/Zu3Nm4vbl/a2Nnf9oYe5+XeVlkeZbnWZZ5JuqMywGJxAzYLa2uP/vVF37r
a18/8dgTQeDu1s6tzb337k7f2WjeuVtfvTV9+8b+5et7lz/Y/dWd6dvvb/70r37xb//dm//3
v/+rH//88ju721sWJxyjj5/tp/8T4OprL3YzJge5eCnU7r4HH9jqwacu/vACtOWcDz/WuYtw
9bUXz1wAgEuvnELE1OG7cCbF6D2IH3Vnc+jpQ6c9e7h97NzFg3/MTvGhV/Swa7/4wwvJo/Pi
Dy98qFvnxXMHm8wuavb4uXMvPnjC3RH/6PDFPrjta6895GJ/owsyxxx/h3EUkaJEkVoWhZwa
PMliu/UqB0ZqJT5dkQpSGEgbOGZmIJJmxISYfZY5751zxK2tERKzy8mXPi+zoiyKXl6WZdlD
xkldBxF23pCCd/smMfP9leXeaFT2S5KYxaYfQrk/dbc39frN5tpNuXPX7ewXdewZFIAMgITC
FJgqggqgQasJKoYqo2nOkx5PBtl44PcGbmfgdgZumrOSU3JGDsgDeSRGdISeyTE6j57JOccZ
O8eEBhYiNkJBOahvLAuQRyyRCyBv6BQyhcyQo2rdaFWTaO58z+cluZJcL32h75HLgQvg3CgD
9Ia5UQ+5R67PfsCuz65Hrk+uz65PrseuR27APETuA+WIDoFUF0eD3/raC7/z9373ifX1AdGi
zwbIhWKp7BrVcYVRcuaMnUfyrbs2MhEhEyF1fVojAmRjVkYjUCZwHpwz5tmXEiuREEWAgDAF
28K4TbKNsgVhQ5qNWG+EalPqbW12LexamMYQQgCTFFGMFlGF1LyZB8gAMrASsIeuZJf7LC8y
3yv8oHS90hWlK4osL/Ks9LlzhXeld73MD3I/KKwgKMgyFAeRNVKSzkuy3jAAUVFROOT5Dqqm
KmJRJFoUUGAEdgEsIEbESRWmDY4b251K1hvlvQF7T+wNISt8v9frl2WRZ8hoYOyYyZkhkst7
/adOPvM7v/v3n37mecrK63e2fnFz/KvbzXt3w3tbdm2fro/5xj7d3IW702yzLn51u/mLt2/8
5D/84t9c+ss/+3//+oMbm6GRz/Kz/wmQbuxnX7cr5093Gbyz5pyZvX720iunPvY99erlNwHu
D0R+GE6+/MbrZwEgHeXK+dMAcPZ1e6hU6NIP4NX2RRfOpFO5eA5Tzy09fuFM4gonX/7++dMA
F/74tavddR0SOj1wRQ+79tTYe/5Udy3/Mbh0Ab5z3wkfuvDvf5yLff1suoyH7/9zXZA55vi7
jCMrUmhAgGSEKWvXQLUV/KhYmznbgpmp08Ym/pWMiKJKHULVhKiGxD7Lnc+QnQJG0SAaknKY
mH2Rlz1fFFmWDUejvCjrJlRRKCv9YGErhF9t3f2ba+/emuz311cGK4voCUFZNVfpmZUxljHw
eCLbO83WZtjesb0xTiquQt5YodQH3zfXEy6EnSAIRLMp6IY2Gxg3nG4UcLfE7ZL2M9oveL90
uz23U/JmQXdLutunuwO+O3S3h7Sdw5Sh8RgcBcYGtQFpTKoYKgkhWY8niZiqBAlNCKEBVQmx
qqomBAOgzDnv0ogfe8e+zUZJ1Z1WOYWzyDZARiLgpM9mQIL0eOqYJW160KaSelpXnLvlY2uL
i4veOTQDs6au0xzj/nRahRAB0jk3JkEliNQSa4mVxEpjpTLVOFWdglQmE4sT0KlpZTK1ODWZ
mkw0jjXsSbMnzU6sd2O9E6rtWG2GaivWOxp2TfZB9kHGpmPQCdqUsCIE5pydQ2IzMiMAT5gx
5kwZkyd0yesKhNDIIToCT5AxeAYmYGLn8iIVzpAdsUMgAwJyKSJbEY3AmMD7NDGJqlrXVVVV
ZqopAxANwAiB2XnnXe59lrFn9qyUZicNgAA9ct5EaAIMFpbysm/ExAwAabSOGL13ZVGy8218
NYBacrPC5dXVb/2Dv/9P/ov//Ktf+0Zv+QT0VqFYgXIVesdgcAKKY1CuQ/948Ms39+zK+/tX
363++u2dH//0nX/9b37240tvf9af/0+ECz+8ePLlNyzd2++pKr306vnTcOkHP7r/ln7y5TfM
PkIsdLhG8mvUN9pTOfnyd84CXHrrSneG51+dhQTPWEdiDpdeOXXqcFfuyCu659rh6o9+cOnD
K2of84TTmR064U98sSla+kM2/VwXZI45/k7jSI2UKGJqtZEpmEjrI9UJydN/fil3LyVapAdF
lYkMoAltw8R7F0K4fPnynTt3mNl7j4jOOedcFBFE9rlHZkJEdI7K0ofJ3g7Dcl4srR/vL6xC
/sHEVDTc2N3KHPaXFxbjibC1I5NaY/REKgZgTECKqmqqqqJVVARLeyY2MwQETxmjcxhNmVmT
saSYBHVoQjTNTFSN0Hvv84K9A0RjIuI6hMl4nI+n1gQTNU2j9yYxgBgl5pNMxYkAUsZLu2bE
pK1raZL2JKNzbCOAQcCMkYXarqh2AbjmEIBNVLVN0VNRSMq0pEgzUAZhVAIg2g/1z/7qZ4XL
KJowhibUIhF0KmFfGwsIwXGKvlFNVURJnqkpYo/QEI1ITNsYGQCVqAAppSbJmFJKclKLGQAi
KJqaZZKk7tiKlgwNO+OMZN19MNMJZmbtAKKqRZWYusVUspEpgjoDh8TJohQMNDaiCgwMDEiU
l4WgBdGiLJFwsjsOtYKYA87Y5T5D72uEQAgATKAOTZJLlgKY8w7JoweX5wFiZZWYmkpKrZZa
ScmTr6xOfCvPM+/TuzuNMhIqkSP2HiPXdWUi3QQriIka9sry2WeeWVg+sXDlxs8uv39nY0cx
d9koRDPNwWogNQLQZko4rbIN4VHpdzb5V5Ot//Wz/fh/XJx8+fvnf3DqlQtn8AIAwNnX7dXL
b0Lqub1y8LK3rgB8DHJx8pkXAC69efkqvHRyJgC6Z+7t18abl69ehSPqXu0FXZrxiq629OAV
vfzAtX/vpZlAqr2WT+WE4aXPcMbuM16Qz+7E55jjbwGOJFImrW2jiEiMMYa6CTEigAHUk+l4
fzweT5qmMbCi1+v3+0QURYioLMvBYJBlGRI1UXLKt3d3L/3kJ++9996JEyeWlpeLovDee+eQ
CJjKsk+ADNTv90LQqp6iNopwe2t/fUF81hsOl/ezjSDaqNze3Vnr9d2wV3hHk2b/zmY9CUBk
JplzRGiNgFgGDKAmyWUIELAhDFkW1WoJFjCGkBMXSt6SKZBF09pD5SBIVACfkZfgciDnCNmR
D4Q9F4MLmozIEdVU0IwRk31lDKEJGiPhoc4o/v/svUuQHdd1Lbg/55zMvPfWD38Q/BOgJIqW
bPlZtgHaftGvPwbUL0I9UU86wjNySEw889AzT8Ah0JP21N0RrY5ok6/79XvPfjZhfSxTlihS
JECCBAjiX7/7ycxzzt67B+dWoUCAFG2LlGzXCkRF1alE3nPyU7lz77XXIttqfpz31mFJiGwP
gBbaNqGoAiISFZ6ZbaszimqRNjUQka2ePgVVMBNQIARmYMfeqVdVALUsMZn0KinlWY7rsTUA
C+xFMQlpYcmjgtl8eghKgGAgilR8kA20aKqXsEnNwGA+A0Kz7cjRQrYqWtE+KNmhudXzvG0B
AbE1bcnIOaSStpsLV7F3WCIUsERZimkQFvIVExCogTBXFQKKakYAsC7LeDadxTho+qZuJAkC
MTk2DehCcOqQDLxzxE4QtU8ZAcEAzTkWQFVh9o5JgdgKAxCYHAvlNqVJz8QBCUKojKyHRwAA
IABJREFUPNfBDZuKHczdEJEQyYAN0RXbHIk5ZRVBgLmnT4KK+ciBlecWl48cOvT3P3n70ge3
hcAEEhj4AEymGYCgqmCw3IewOqy7QUP8i9CRevAjfYvv/MoLeOrcuW+/cvabzwKc/0e22J/8
5vNw7tyf/fnFFz+r/vxnnz56FHaGax/BKy8cO33++PHj58+fPvbC03b25FZI9KAV3bd2+PY5
eP7lkzvW8u1Xzp78J0UTn6LQ+U/CZ3tA/mlr38Uu/rnjkwKpv/3+9yeTyXg87vu+eJe0bYtE
TVU55ulksrGx2XUREauqGi0sjBZGLgRE9N6PFkZLy8sLCwvOBySHxMzu67/1Wyee+509e/Y0
TeOcK8RzU80qRGSijhgQr1z9YHVt9dDBAwdWVrgedcqAVUNhZL5HryKdxL6qo6oDbAZN2LM3
130/Hus0k6gZGRCyGghIUbWeazNAcNWBPYsrC53kNvU5Za8As5S6vm17VQEkzYJJakQA1L7N
GxNBJEBQYOeYyJspikIu/fSoRmIISEBAYITZoQC1XVfa/hGgZKd0Lv+0pViKyICl9VFxHnCI
WVYp/DJVUTAtCvKF7KNzFU3JWVUB7opOgRoCOHbM7NiJ853rs0of+5gkqmWwDmzctwZW1RUg
crHrmxv5oCEBAhjNWzHVioJBWbpqacRjKwQ4BAU0BENSNCIk54iIPXCwIpelCFnETIvdXkl0
ASI59p4MbS5whSpmGURAEZG9Y09u4AaNH/qqIU/ZLIl22cRAIWWZztpp3270cSZZEch5Ggx6
pr7rnECFzHNVBhYpFTY0ZANVE0MDMyithlQEZYmYi84ZgDnHjjjF1M9S3NRgblg30s1C5QaO
PFpgtJyAMqgYIAKBoRgYUGHNOXKSY4oRzSokAzFT7+yRpbC32ffI3tEP33z/jXdvfNhPM6s5
x+zRiMxZNUzVAoRGmsW4sARV/Vnf//eiBAWn/+SVF8+eLPWrOY+6pIuef9nOHnvmOAA8c+ze
YOjjhI4uvnTi2Gm4/1F88uzLz587tf3I3maVfyI+Oflzvkz74kt/vE3+3rma+fgfzbk/pWnw
1RcvvICnzp164Zt29uTHrQjuW3tRPji2tZY/PHP83On5Prba7Ur4cbf0drQczJ8x4U+92E+D
z/GA/FOmuYtd/AvAJwVSXdellAAghBCcZ+aVlZXgXFPXoaoAwMSYnffBh+C9I8fEzK70cjkX
fPAeybELzKyqTzzxpHOlUx9zzn3fq1pKUSQbKKpKkus3bl569xI6eviRh41cFBSu0bkRV1oN
WoNMjhkYnZh0ksDQhaquhj5UEUG7LoswoRrknGme15Fi9dsTmNfUUHY1VKOcY5cyKZCaZEmS
k2Sfch2zZNEsGsWSULacssbMWTySqaGqBzEwAioNWwSEAKAmit7YCMPASTGyK7x8K446oFt6
i1AsihEBYC7CaPOIT2DuLaOqGWAu0q2ARsXheK45BYCoAmoIhOCNWMkZai8RJJN1mqOKABla
UqPggvNiysSOyDl1Yh6BENUMgUtFaitTVnoM5qJghnMV09JJoFBSYfM0FQE59I6dDrivQAyQ
Uc2iShYxACUwnCe9FFBLGOPYe0eVA8+ucmFQV03N3gGBUmLWSpnalNanKUbLybLGmDdns/Fs
Ok35TsqtGRB5Jk9MZmgWENGMVZ1gYATJCoTBEZEZFMMWMEVkJBSQci7M5mRzcOCcQ4PU9e1m
bjet4qWaHamw94zmUB2baQaTeTKxnGBkw0KNI2ZjcIyWYo8AyJhyhpyD9qjw8Mpw+Gtf3L//
4Hd+eOG9a2tdNgAlYgYwVWkFkYLWDKNkg8/y3n8ATp61lwFPzYs2d8s2J89eOPP6sdNl/PiZ
Cy8eBThatt0q/PwDKzwnz5p98wXc/qSyi09Ob5UH+yl8oNbk8TPPfBvxVNlPmcnJs3bhmRPH
tj5ja4avvHBqO4Q4evbl589tRQ4nH7yij64d/vzE+ePf+tPtzz/64qv29M6lbB+Koy/+0fOn
T507hefg+TNnjsP5nZHR3QkfP3Ph/kO3c7F/+ikO6H34/A7IrurnLv6145OMUc/8r2cRkYm8
98F753xVV8654H0VKudcCYmcc44dbKk24pweBGoKaoDIiIWyrsXBDUCyxBRzzCoCYFmlZFna
6ex73//+6ura1379awcPHiSDUWieePTxvZV3b78N166lydRiG0DJcu5jO2sJnSV15CoibSdx
Mk6TGfYR+h5TCkimWYtlHeiYKR06OHry0T1ffGrwyKFNTRuTiXUpjWf95jS2bYoRcsIUQYSz
shiZQRYTRTEnhkm160kF1CRnSdlENAuaoQDp3QIemyKaAGZTM0VFMizmg2ImhICQVIuFSHF/
JsCUU85SVEzVLJtlQGJCAFNDpJzz3EsOwObexYJkhYPuAMnmBbXiPVxUrDRnQwTv1qeTzclk
38rehhna3lIigCJwyUBoUAQ4E0JvFsmUHDpnBmyQVYRVEBHRCI3ImJAoq6JnXwUkJ6BFKUlU
ygSKXigGR5X3wRMzDJgb55iC8z54Ds4cZdwKzpJITmA9mXIyGEfZ6HQz6iS1bby1un5zMukJ
o6NxFh9qVJU+MmBwbtgMPEIwgj4GI1/0zr0D59B7NYx9H7seRJjJERYBKRElDsjU21SHtufR
h3l5761NuHFjOt7sB3XYu2/56vWbrl469uUvP/H0Y09+4cBogVEzgUcoegqixQQ6ZzBBMwRT
LWaH0sV+Mpm0XS9IYEoqnqpp8m9fG3/3zQ/fvTHO6rHk+PxA6728uN/v2SPNQgLW/+1//sz/
BOy6I3/e2FUe38Uu/kXhkzJSTx97moi89845ZkeEzjmbZyqoOJUURamyfXEs2XKNA0YyMlKh
+SMfEAHZpSx9bHMUEZGYmImZexVDeuOtty5fufLlZ545cuiwY459nMymtzZWw56VYe0pBPN9
jXXIyQELMBoSu24WVUy9Q7dQ1VUYdro5S5ubpjMRQUNEcoiGRqbWS+yyeEd7lwejhrJ4I24T
9wJ9ltS3sZt1M+06mbVp2nazWR/7lFLqeozixLwtgCbTDGZUNB0BMYslxSyoAKqac0y9mWSA
rKJJOGvI6DKgKKoUqjPcNdOdC3T54FIXswoAqCqbFTMSAtIsnp3kuV3MXC8UDEGwpHsA59kR
AFV1SkCgYCJCXDnnMqLGmIGWXNU4l5JlEQNDMgNgUwesgBnABYeDAIFbIggVIBAxpJgwCYiC
iWLCeQMnBu9C8FVVDIa9d01dsfPIBEwueFd5IAKHRIxMLfeRew9IamYmhBk0Qk45lyQXQDEK
ypoNNKspGpiCRut76ZPNHHVohhzIVQgc2AMRgUcEU0IxyEScRZOR9w4VKKsBggIgGqECCAAB
Itl2TCqgzhN5IPKgruu6GGUwiDFO21k7DPvZL5KvuaqQMyiQOQCnkAGylbJlIYPN/5EZAlIg
5j5NNya3N6bWzUIaB82iVW2LI8iYxKgyM0fIVRUHAx1WqWLxavhJTbW72MUudrGLXwZ8UiC1
f//++wcL6XmnvrmIwJbKOW758W239cGc761K5SmfUhfTrFU1BiJiNkyionL5vfd//OMfHzp0
8Etf+lJV1zlGVU1d37WtyJLzjThHITCzSDYFQGPHpqCSSQmTmgkbu3qIrsp1nWez1E5T30nf
m2ZERPScTfu+G0+avu8qbtE6ZK7AB8eLXq0mWFgAdKpeALPmJFlyTDnGmNuuH8/yrI3tRNIs
xxT7XmKvWVAUspAaIaACiHgZmCQxA8QA5BSCMkZRyVDKfVlCFiiqXIVHQ0hMqetSTFaca82o
PKENdJoyZOZC2JpTvYvCuEEqDiqFmI4Ac6kkQwQDFUAoKkoppSwCAEVq05jVDFANJAMoGhB1
qlBxvX+5d9DnmB2JGaFpxZkRCdh7F4JnRse+quvhINR1Kela41PDNhdlzyqmpqlkziRniaAm
YGAMBpQVDYwRmKjkxUAIDAlAcxF4orKpKpnVCEshpBQxJxIlpEqsJjesqjoEBVC0LACiOQMQ
oEKyzDmjQ8BiPKwfuZ4RwQjmnZuE7BwSlebIPsUkmX2VJYPZYDio6ipUVbnCiQh0fvZKrROM
tvY5j4yRyNHcuLquB4Fv3ro8Gd9adyrZwp3Y9WNwwEKVAHNwvlrQagiuQnLoKvTVP/7O3sUu
drGLXXwu+OSuvY8m/EtgVPqaCv2Htv31dkRO94wwFeayYxS12Hepa3NKjFT5AIaSMqDdvHnz
v/7VXzHzs8/+ynA4NLOUcwnRYoyT6bTJuaoHqGZ9X40WKfeSupgl5YxmgYgAzFBVEwAy43DA
dSAZ4nQapzOJETSTgopZn7tZJ6ICoMwZDMDK81tMAYzMIGkg4sBQEXCtiKhWAw7MWIwgmeWc
ksSoMaGKppT7PvVd7Pu+7yzGMOuk7SymlFMf06xPHAVYUA3QCYIJ+cxQXPEIiYgcJ7PYQhRX
2vm8ISULQJili53GBIpoRiWLZAYGiqIohcoExUS6dPshIFJWRZNCJyckNRNRA2Bmc2yJS0Me
GipaMgWgTCCkRrquifevHHr8UaqCAZAjDGhoiARE20JMRgQASSWZZYbeSYpJLRMiO0RT0JIz
UwRFACVQIpN5TEPlerH5dQWaQU0Km0yNDS1r7jpsM7RpKCboEGSE5JkCEYPVakEhI/YmVDJs
gKSoCghQspJqgEjblzQaFAcWVVMRMwYmQDIkRTKDPqau74HQOW+ovqpGCwtVXbkqIKKq4pZF
NAIQoG11X+68WYL3qppzRsSV5aVRGOyvqvchX37n0up4cwqLswmbDNAzmBNqgBsjD4CmaGLI
Hw37dvEvAifPmp39RU9iF7vYxc8LnxRI7cT2E2jbH0YLzJh5e+T+2MsMAKk474nGvms1RWfg
kAlQRFT05urNv/zLvxxvbj733HOHDh5MKeWYYowmykQ55/FkMlCohsPJbDyezZYdDYMPdYii
fbeJiM4xZBOZ05yRERyZACiS42Y4xJRyTByzqmXnVWwrgAEHBAgApQe/UI6APJgIIAigkGUA
RUUwUmCwDNlKuFZVbN4ZOFQSqUAR0dBA1VKUFE0Vk8rmTO5s2u0JrE1t0kEvqY+xFytBEKgR
uorRWFQdhzabY3bOdSmN+2lCqgkTZJTEhqZKQFia61WNjFxhqBmoocLcr2QuT26gZmhqW151
YCrCzHONcoTSyIZmYgqIjjmadTnb0C8c3HvgqccyUy8JCLrUFu0s1eIFZFmzZDEAYlZTSFIl
ZElAgISgoppVMqmiAs1jPAMrIUxmIAa2rAriANgQJEuKIAI9QDSKmvoUZx1OI7YSBBbZB2Lz
HBg9ouYsWTX2hERkjGSInpkEwYioWB0LAiIB6LyQt+1pZGZS3KgVgRCARCAnabsUY3QeiShL
5jqEukYi7z17V94iUJWLO+GWfKptiWOVOwUAyt2hqqriHB068tDy0nDPvgNvvP3eW1fWp3Ga
zQtmAMxgQo4oADpVsBgh//JYxOxiF7vYxS4ejJ+RkdrmP20/G3AHKYqIbMthd3t85wYAYHPJ
JO1jnE3H/WwmIp49M3Vta0brG5t/853vXrly5ev/5jeOHTuWRdAs51xUAxBQRLLaVGxpOLDh
aLK+PptsLjIt1kEJEoiJGBkTCWRB894nzYRkzGqKxOyImH2onZpDmjQOHacuNS5ITgjAgCaQ
NRmBYWZWUkBBIlNRYiIyBSn2bCIixOYcApKAiWYVQgUTBQUAMVUVhIwsZlJ7CoNKxwGYCQkU
yIh9LVyNY69IYpJMjajvoquCoMZp55vB8sJCPzCsqtl4bIbIDBBLk6BapiJyCQpIZnN9h2LQ
a1uGy2ql4qeIZqqGyMwImEXMjBANTESQDQAI0KHLYkzAakQIOatozKk3zEWtwLEqSDbFPFfn
QmDGnMXECJHUfMqogo6UVItyJ4igAhnBVg4LTGK3dmu1n8727NmzvLLsCQ2Mspooi0mrcZYs
K/aqswxRMCsDGkJDHBgTGeSIalg+xBRDcOzELG/FH0QElkWguDmb6VxLFqCkkdAsiRRKk2Qh
z8HVmq1NnWYEhdRHROhj78KAHGUVFzwx5aimhqKICjRvysR7KU0isq37X340B8Zch+WjS4v1
gSP2k3evv/b2+kZnOoXRPqiCeVZG7wMwyT0OlrvYxS52sYtfUvyMjJSZbb9Yz3MJALAVLSGi
butT78hFFXGj+SOk2NYmmU6nk/FEcw9mphqzmHFO9r3XXrvwzjtf/epXf/VXf61pmtl0umWF
hkSoKacYkag33VBplhZ1feADzabjNBsPnEPv0ImamKl6rAejOlQbG2spxuBciTUMQY2UFJCQ
GJkBSbOSYIWeDBwYgAlQETTXKKjggKWPFXFGMxOzRChoYpadeRFFAw/ACEBmUpTUs3Eh+4jL
4g1MMZiRSFZNOabYSd9xVjaMWT64tWqOFKCX2KvOulZVp9OZxDhoms3NSbOyGBmyKnJgcgqE
gASKSAAGCEp3H7dYLKIBCk9NwQhAEYnKWSqBFtrWCYK5LzUCQqGsISKakRoKYFZkBFVQA0eG
JoUeN09EajkqaEBkpgJmjp2pCSA6nzRn0EySNQOacwRqosIEXiQI5Gyyvjlb3xiZ48ECcUgx
xjZJ22sbZdxCEsfeklgrJMDIAKoq2UwMhQzMGAm8Y2QO3oJXE00xZTOTgI6JUNBURZIBsnOG
aKpoW4ISpgZaFOiBmIhALPc5g0gkRnIugGrKmQJm1SLwIWYABLBFDTRVU0BUUNj5DlEMvLcO
NSAaWkZTM+fdoUcOn1jZt/zQw995/d03r25sYBSXkDIgAhF5Rk/kPm3CeBe72MUudvGLwid7
7d2DOYVlK2zaHtwe+RhOFYpo18eu60QSgPngs0nUjJW7cOW9H7/10yOPPfr1r399NBqmlIi5
lAjLEwixZKTEqjBWtcHALy+5hQVaHErlsQ4YvICpGToSh1T5wcKgqQLmbH3PSTkbqimaFPVI
VcuqSVOfQLXyHhEBDR2Cc4KoxjnT5nr7zoXLV967ljsjZTTCUqwiZAaC5C1WkDj10LcYO6e5
ZqwJSQTFwCyitqy90w4loaLDetj44IN3DmHt1s03X3/98jvvtZtt7mXt9satqzfXbqy2G7Ol
MDq4tG9fvWTjeOfKjY21zdyLJ1+5Co1MEYmNipcwGbORA3KGztBB+Tr3WvaIvvgRlwwVUkkW
mmRR1RJx4dZpVTBF0LtnE8qZVTDcUukW1aJaWWQSCMBMLEmFPKgqif3a6p1pOzNEUUPAwD4Q
V+B8gibbSHAQYZBd1WN/e1Inf+zAUytuefXS+tU3P3z/x1cv/eiDS69/ePkn18eX19O1cX9j
nFZbmwkL52wpqQAaOwgBQmVVrU2tVR0djyWtdtP12LWm6hidK3IbxExFj0LnF+d2AbpcZKiG
YpCVgR06jRKnfZr1qYsOedQMGTHFqGAxJ2R23osKEjARFmbbzhys2c57Yef3iIiARRg+RgGV
lWH19We/8M3/7rl/95u/cmTRu37T0sxMsmbJUXPUHH9+d/qnxysv3L3pX9hypC2OeC/88hnU
7pztvVP+h8z5lRfu3fKevW45AD5w8OeFiy+duG+XH53VDl/Cu9veHfslPDu72MW/CnzSK+/2
M2L7eUBE5RtVpS3NbhMBmBf/5nznLUs1Vc0CXddNJpOUonOOgMTUBY8cPvjw+qvf/S7X1a98
5Sveua7r2LkSexmAiDAROyciKSUYjvo+Yl0vHti3+t4lyMJlQuwMQcGQGcjG3aT2jAYsiqIA
ZkhAAA6BCRQoAonllGPf5ZS5CUpqpkiQCDq02Ka3fvTTWx/eQuO6HlR7H9qzNBJpEY2Z0ABU
AoNKJrHgeNbOrn94Hc0OHj5Uj0YmlsxQNCR1aDkLZDUxJRIHulhntNmm3NjQm15uS0ea2WgN
JAWGxmeAST+tiANwYBcG9dz3DpiQmJikCLVrsYU2NAUEYytHrSSlEAEQ5nYupIYleMA5RwpU
ZYuNjlQY4ACp1CURMqESIqERipmYGUJRXQcELSk+Ki2BRKAIJpIna+ObN67fubW2b+Xw8PGl
gAFVbZbbtQ0UaYjjZKyzblQ35sNm28skuxhWN8ZdFAU28oxNQwtUAWEO4/U8WUvQcXCmhlnM
0JwDJmDOZL1pJ9nUVDTlnMnEEVSVFSEJh1hcbHIR3UQFAxFANDUEAytpRCuS/abiyNgwJWml
nRp2rQfVKnhQk5RrJgPzzoU6EKkJSDHN2apmAyIBlSithE609dZRtiAkIshiWy2KiCaW20f3
LAx+/dkDh458/8333rw+nbACg2qGaL+IQKpojBfpxVdewFOnTjzzz0DuaFsC8+JLJ46dQvgH
iYMWze57h95+HeCjQk8PHPy54JUX8BQ8/zzAn5xA+KOdCun3zRPOXLAXL7yAp07/wUvfePXF
oxdf+oOiN37spRPH/nmcrF3s4l8cfgZHaq79eG+qafsN3Ar1BoAIgYgAzbRkOVS1KHfHftZO
Jzm2bEZIigSmAG59Y/rq+e+sr6//zu/97kOHDpokBNSspsaMZuoCG2pSZc/m0Dz32dZSvzIc
4HAIMfqolMlzyOxELZOwqmSZ6iYruCqkWScpO2IiJEFEMzBgVk0Qe5hNbDZ2QzKyjBo1KwEi
2nRst9YGuKQHHv/e6torb1751Zn8u5XFR1Yn09XLsADLB/ZEdS7UZh1bSmm8efvqeNwT8IEj
gQI5FtQYrNOUkqgABnSNZ0YnvdikH/T60GixfoQqtzGdzmQyySm54BYXllPOH964sWGGROz9
44sL+33TaZxSikHI5cqsymiIUqxuzYxNKQMA2T1nrsgIOBDRbFBoVcyIZqKWCY2xEKwVUJEA
5qkagnkIRQKYAcGzMiZJxghKqAKUASKaAqhj303l/XdvXL+6mnoYt0tXxsO3K1wa8ZE+Lt1Y
W730/o1uIqM6tW1oZ4tKh+qRg7Amtkpe9x1Y+sKT1Z69wMg502yG4426j3hnefOaX1+97vvx
0LM3Q+JIbjNLG1NW7WLfQWbHHjA4j8TeOWSvgJoyGagKAxI642JHqGRFoD0DlJCfwDADJDKG
ZGZkQxAvwONka3GaDfb7JY0acj2QpkKpm+SrCJgJPAOQkaEo6DxlZ8aEZlhkUotLDRVNKQAz
lWxbNxDMWy5RUfL+QfXbT42eWD7y99fbv7gJd1LfCwgEMP/zvt9/Ji68cX7bsGRnc9mWvdov
OeZa4n/80h+efPHop5nzA72S5+Y499rfPXDw54OTZ+3CSyeOnTt/V2v8/lld/PM/Ow/Hz3yj
iMrPT0wZ/NYxgKPf+Nbx06c/S//CXexiFx+DTyrt6RZ21Cu2ihRbwZRtWccRlTIRF15UKRil
mKazSY4zz8COSqs4omu79P2//cGFC29/7Ve/+qUvPG0qoAamlgtRO5saMwEgMdZNXTcVO1pc
Wlrb2Jj2GX0N6B1VmNAZMTIwGqhXDIa5jyKZnXN1BcRd33dtZ1E4A8YMIpAUYoSuk9nYUjTN
AMqEjpCyDJx/8rGnDh54bOAP6Gz09tur3/3eWz/4i7+78ddvXv+vP/7p//fdH/7F9374n/7m
5t+9Xd+aDa5v8rs3npLmt/Y9+TgsL23QaN2a1bQ8tZWeFjqqZlk3ZjJrPWLjPPVZV8e0Ol3Y
SPsiPzRaHCLA5qRu+8VkzbRvJvHx0Z4vH3nsi0ce29cMuRMXA1qjfmj1KGFQc4jeyAk5IVZi
NS8QBHwGn8ELeMEgGIS8kld0hqzI0SihE/aKPhllcEZBKQh6AW/omMxhdpg9ZJJE0jvNTvvK
UmUpSAwgZIBGaECABIBAfWdv/fSDN9+4arZnYfHoOO97Y6P6jxdv/aefXn79vZtrt9o8desb
+u6H43dvTC7fbC/fmn24li+P5cN6CZ/92uJv/Vs89pXZnkcnS4fvLKzc8P6O2J1Je8e7ePBg
WlrY6NrYTl0WzJZENvp4o2tXQftR4/Yu42jQIfSgGQDReVc5dghoSFIoS8RbbD4lNTQrTjK4
ZRetiMpsrEgZQEwpq5sl7TSrV1f7EJqlhX2eaiYbDNG5ZCAIjswRkBVq3LalDhITMbNj3q55
gxkUuYStm0bBjNCYwKHzVlFc8v3Rff53nz3473/76FcfaY6M8tJAwuAXxDY/9+37ikQ7ymR3
a0kfqTLtKH19pPL0866DfQJOfvP5ub/dR0p790x754Sef9kunDm+cx8X3jgPAOdO3VMwe9Dg
/BNe2lFdu3sQPv02F186cezPvnXBXoZT2xO7b1YX3jh/v7fxjsHizvfGhZ/brHaxi118SnxS
IMXMzFw449tZKNxig5QRAqItdxgohQ41FUGzFNN0No0xOnZMbFa8bRGZLr5z4Sc/+cmzX/7y
r/3aVxnQtNjgIjN757zzwXtC8s41dTNoGu88K3g1Z7B5+07F3rFH59E59M45RwrFORi29KUB
wTlX1RU7l0Wm3WzazVKMMm0pJp120vaxjSpqapaUEnCrVavOal7aL4uLS1XzG8sHvxYWR0ku
ra9eFtnQ5u2L63/9t+9890c//c9/9Vd/8f/8xaXvvU0fpP2bI7s4vfyf33zr/37t/f/wk8v/
1+vv/u8/vPTym7f++j17Y335Q1m8k3Ft3E02Z3m2mWbjyThuTmmaBogHlxYOLy8+um/PI0tL
i1n3AB5mf0jxcQpfGe15slmuzLNVbA3xCP1CdqPo6s67voKulr7OxsTSOBmwDlgHpA1pQ1aT
VagBNJh5MCJDBnBEDIUxb6TkjFkdC5MyQUDwaAzCaISKTokTcLSQsVausjFkAmFAJwFzDXkQ
Z05ytbL3yGBpjzkHS9WsUSTTXq7f3riwtv6B5A9n3c074831eHsst8HfOXSk/dq/4d97Lj37
9GxpKDH7yVTvrK3fvj3uOsWg4jdMNhvLHhXQqfPifCbOoFlmFqcD1APDas84o7hLAAAgAElE
QVTI1T6TZgBwhJUTAkFLYAKmVBYBBlaMeMRMyzVaLG4QAectEeyYQ1BEQzOznCIC1KEOIXDw
ftAkVCCqqgoB2EqS1gRKe+T8XUPNVFVEys/lXaKMlA6AktwtI8Wxe96NoSYGQG7BwVeX8N9/
4cB/+9TK0TDZm2985n8APoqTZ19+fh4ufFz0c/TFV8tL1cvPA8DxM386970ttiflF+dPH/sc
g6f78frb99GNiuXufN7nT/9Bmd7Js/dXAV/59jmAspSXn4dzJbh54GDBuTeeftVK2HPu1B8/
c8HKD+dO7QhIPnGboy++aq++eBROnrVSmXvArEphEb594p4I7O3XP3bh/+RZ7WIXu/iU+Bkc
qe3vt8t8sCOWAgBiUtWdI2omIkl1Op12bQdqigBmakCOVO3Da9f+7rXX9uzZ89xzJ+pQpZQq
71UyEd2VpDIzVSJqmiZUlZnlrmsn01Hw2sXRaFgfCpMPr8bpxAMhMCMH4l5kPmctJUdj75rR
wLGLsUulCJMyCPWUbXPYj2cas7EHM2n7bmPCUSS7G7fG16PF2UZW3yzgtck6cnrtxri5udFp
/eZP35u2t/YO3Mb+w+mIDngwaW/SaLnZd2i4tGca0+TObW2dXVu12a1Aum/vaN/hZbcYVCOb
1k1FrTVZQAA9DYOTEDy54LzEBKIIlFPyZq4K4N3Ycms9IRFnrMkpWFZkA04E2Zk6RSeMAHP7
XVQoCglFLx0yW2eQEMmDVZwCdZSVZRqochhN+2wZwHpiAVYgIZfRiJtEDOZbY0IfGRUi2QRQ
CBiRQSkncxz27z+gOHaeET11M0j9IMqKGs5kLUaRPIl5FBYeOng4adZR1Xz5q+Hw47OR6y3D
tEtXb69dvjLup/XhlT37VtaxnRCKw3Zz3WbtnlA7DskIyBXFKCSwinIFXdtJN0spVsEhARM2
TdPnJDFDyipZkbwjMy6p06K9ZXPJp8IwQwJyZJ6d91VGNueIvBmAQh1qX9WxnU2zYO2bxSEh
atbgCVCRi/8hzPU9zO5ph9wh8V++lp8Rkbala0vFXM0AJGdmbkBdXluo/L4jg8PV/ovX1n4u
N/k/DCfPmp195QU8de70MTz98WbEF1/6422r260441vfOAoAcPIPzxw/Ny8z/ZLUBEuZ8uJL
J3BOPDr/xgWABxfBdpQ0i3Xwn/35xRdffNDgNwBgqxJaMkJbh+Cj+DTb/GycPwcvm53cJq99
4xO2/fxmtYtd/GvHz+6vvr8Xb7tlr/zwkW4+AFDVvuu6tlVVJiYomkUGiKtra+e/8x0F+53n
TiwujFJMjgjNfKGZI8IWmV1Vi7eGYzZRUNEUY84j9rHtl4ZDd/DA7CZY14P31vfkiBnEtJiQ
EAAhKQgQ18OmGTYxJugiKAooxiyTKNNOuwSeCTG2/fTmmu/19s3xO+9dqw4/vLSw3CwNYDRI
t9uD2Q50oK66dvu29vqbT/86zCaoXlce18OP1AcPhMceseWlKTsWGwng6uboyofh+o3ZB5cn
V9619WuHDi5SBU6iZlMDBWSARqBCAOdVxGJPBOwYABUNKBsCgQbOXlvyjGAdRERlBDTjTAjM
AoaSqh5KqYpKf54aaFHOTND3GtUEGM25DJJcUtMOu8AILiWXs2UBBc0ECEiKAgQZu8xOaBpd
D047ELPIOTs0AgYgM7ZkfZskq3O8MV4T5FFF+9Y3w1r3WKhrdZtZp10LfecFIfZSu3B4P+7b
2yn6cYaN26tXLuP6qnWzAdo+HdBk4+0P3ott/8jexTye8Sw5N9xUMc/kw6yNohgMoUu2Mc7R
8rTVrgdwzgXO4s0AMAOaoYgRbVXTtr4UMc6dBpGIxATEjN4hOnAOjDQbZHXsXPDtzNb72XA0
CqNhEpltjmlUe3KGOJeXQAQz/EjYtAN3b5vSiLrDJxgRuYixIYIqIVQgGPuK66UDwycXBz+P
e/wfg5Nnzc5efOnEsdNzvtFHN7j40h+cPg/Hz/xhCbJKauT86WN4+u42Hx+qfOZ49umjADuT
UnPW0fEzF+zVCw/iRf2yo0Q7JfIpkdwbF+DFp58FuIeR/tHa3y52sYvPHj87kNpZy7uffo4A
jhl2vIubmeQcY0RExwyqTE5MgKDv+r/74WvXrl8/ceLEoYP7u7b15NSMmUqRpcRPiGiqTFRX
VTHZMBVEqILjpGSggDMVX9ctkzBWTZU3N3PK5gi36jUlIkMgmGtdIQfHxIZkpos1RQNsk7Sx
WhxlU8dhWA3Gt27cef/qARrs89Xy0kJrqV278z9Ue540mk4+eH11dbp268CoObj/EFYNHTi4
8pu/pQ8dGY9GtxiVqDLybV6ygKPQHZSF0XD5wL7QDOJ7F+5c3VhoGL1R1pxlknvKUJsDUDET
NGMQADPJpplUURhyEJakAqZZ0dWCThktKWggRG/ok0QvnTcpNiy0XWrSQgUSRvOs2SQZ5lpU
FIZZLFqdoDH22VPSpJrqFEkFmPskipxjlAGk3DWkfWq9g6zZskd0qYd+JoFqi3Lzw5vXb69F
oFmMoRmsDAdHne0JYa/xtEvT8TSk7omlpZXRcq6oq41GfuZTRYlurm6+9ppP02NHH3E42rx+
U955fxwjTTdl4K0JVWu5t76mtg64b8kNh+2tDZv1owwuYZomD8x1w752ys6QYna5KEsFM44s
8zqaqcG8iFZ0nwwRuYQvwMzAaIgZ0Spn6HInCDAIdV3VgDSL3Xps2YEfNoi4sbpmqVlYWAZi
LMpcBgRzFdTtJO49Lxtb+ukGNo/ptrYsV/tcRNUsizpynkzUAsZl/4s1LT76jW8dP33+gamb
V/7k9Pntqh5sJzU+i5a2fyBKauyZYw8Y3UquXQB4wCa/7Dj2zPGP+c3rb1+Ek0dL7e+f26p2
sYt/Cfikv9TbGSbYeskulKnt38IOcXMqbedEkvNsNosxzoUSssaURQGAX3/jzTffeuuLX/rS
k089lbMQAKiQSnE9AwNTlSx935tZXddN0+CWKKiqEBMTJZVJ36137YbmMdkYxJqKmzBpZ5Kz
mmnJPSAogoIpgBZNTkR1ZMPgR03TNA0HijlPZtpFSdI0gz179yPy2rXbH75x4fZ7l29fv3rt
1pXQ8EITNm/fppz2Lg2+dvTx33n6SUdyx+nm3tHt/cvXFgd3PBP50VRG1zaqd2/i21f12tr1
vvtJmr02G88OH66PfWnNqg9vTjZuzvIksaGvPDeeQwjNaLC00iwt+4WFamnZLy34pYVqZale
WaqWF6vF5aWlhxcXH+rEt0izAHeg3wiy4W3D46Z3Ex8SeC8UlIP5ICEkX2s1hMFQB8PcDNKg
6QejOFzMCwtpOEyDkQzrWFXJVzmE7L2wF8cSVGuRRnKtuSZpSGpoQ+hqGmPVhXrq/MxpplmX
bt+5denShfcuvXnlyk8vv/+Trru5b6/74pcefvTxvQMXl7E/FLiOsUI6sLTy5IHDTyzvrSRO
ZmuZMgdkl3N3e+PSm3Dr2lNLo70BcbxWjTcXbq/tvT0+1OZR2+dZG+oRhcHESA8far78TPjC
F6rHHl88+NDe5QN76sURuJC0MW7AeTMP2PjgENCMAJxDH9gH55iJaVvtbKchns1vAGRgQBSa
/8tqBDwcjJaXVojd5mzWmvBCXY0G3ntUm4wnG+vrfd8DIXHR6OAidrq98+0c7c4bqkylVPfm
FMPCPoT5xqoggmIERMBg/LnLHxTC8RYhufSpffO+yt4rL5w6tzOMgi2O95/9+UWAe1neny/Z
/N5640dQOPRlkzkf/YF45YVthlgJwP7oxaMPHvw8cfQb3zo+X8LdYPHkH545PmeYl/693fLc
Lnbx+eNnmxbvzEjBjifE/ElgoGrlcaCqOaXY9anrVYQdg0F5aLD3l65c/u73v7eyZ8+zX/0K
O5aYvAsOyXJSVUBUyyoCqgxYOT+oaockoiAKhNmEHYCiiE777tZk3VWc46yJsxp5cTjUWZ9F
tnQYwBDMwGjOPDcDJMxgmQwMsgipWTZrE2Sg2gGybxrfDEh5vDq1qrm9OX7v6uV9jz707uWr
4wuXnqiGhx5afna05Kaz67GbhZC6LnUddf2IgG7dnrz1zpWL79z48CoNBoe/8PTosSODvXtn
fX+tiwcO7q+/8KUPvvudpVm/b+AWaqLA4rAXbvueVIBRyYi4WK4wETMysofgqAZNOICVJxaX
Hoe8sQE5xSwpSx9z7NS14qKYqqmpmMSkAqRYVp4pJUqWk4Ex1B3JNGzM0mTsG2o60NxXvUCO
oL2RGTGxImaPsJRtQH4vWNNVixWbOBKmWcqtC7PRkniY5ZgePbZMdTNcHpiHqPbhZXzzRnr3
8h1ZT/Xicr008H2bN2fTOG4XuFleXFlYHvRp/b3L/dUrCyA63rxxYa1rpyGDIcwgZ1QA3ISU
QwV1WFpaGj32ZN6/f9qn7CsVaCdT1wg22FAFXU4xMiBXFTkCR6oizhwFAgM1EGFTu6vudN9F
DkVrDI1JEEVFJBOCJ9dUtSG0OVETqoVRaOqipc4AbTuLakY0aCpHZCLl3thWkd1ZvJvfPoS0
JfVVyGywrTUFRoAAqFjcp5FI8SOOM58PTp69cOb1Y6dP4bz0dfzMhfsYUluRyN063vEzF159
8ay9DHhqa+xjqVWfCc5tT/hjPvnk2ZefP3dqvtnzZ87A6dMlj/Ogvc0PQlnK1hF44ODniqMv
vvryG1iWsJ372zn4/Mv2i84H7mIX/yrxUY2onfj+T17f5szuVDCf/88CUQNwzIjY933btm3b
lpQSMyMSGqjZnbXVl//f/3Bz9fZ///u///AjD+eYApMnDsSWtY9RAVUFzNi54EPT1IPhEBFz
zkRFiydTythmTJZSXl2/kzRVZNVscljgYfQ8nm5ubjCRc44QS2CB2341BpolovQozkjF/Gik
ywP/8MGlpx9rGRFwAP7ia2/81f/5XyarceXpR2Ey6d+7sf/g/mubq6t3bh9dWTl2ZP9ygG79
9lsyfWeh8V/+lSf/m98Pywf8OF/5L3+z+oMffuXxxxYOrLy5dn3xySf2PfT4nbXNi+9eOLR3
zz4DfO9y9/ev75tM9rOMamAnPejVcXdzfcN5H+o6SzYT58gRBUJPzEZm1lK8M+pXfv3I137/
a4MFtbSpkDrLAgDClMA61h4lWU4pdim2MbVzWpQkiV3ftx3kTiSbd4lhY7qZNR8+eGBhNPAe
jBU9mofEAIieHChENWp8S5YDVYuNsaaYiTFbDxJZE1sMZDF2RpBEo2ImL+Av/FT+4/9xbfPt
8VK1svfQI4O6XorJjdeujW/fWMDFZ790+JEn6PZGunS16qJNpnk6U8xIaM5vxn7TJA7CJsO4
8pRgtN4/eviRxaOPba6Mpn2Ca3fcu1ea6QbUKXG3z7hJmLs+pxSaerC06EejNqeYU+WCxgyq
ljXFmHLWLXObckX5ELbryIiIA4fDJrKfJLu1EcetjJb3PX7saI/puz/6QVxaeuY3fuPEV768
vyIvPYMkhITkQlheGC3WlQcwQgFQEdgyRxIRuBtIoZniXUfjrYQu0rbDj6oagjIV+x0HhAb/
4//0v3w2N/5d4H1CcbvYxS52sYtPj0/KSBV/PbhP8gB21CyUEQwUwVT6PnVtr2JgWNqhRI2Y
2rb9/g9+cOnSpd/9veeeeuwRyZkIGMBEkxAYZgMFRUIzcEyhCc2gJkKRzGgACgCAlAGV2RRy
ykSVjzpAq7WW1LcDHwa1dhOLMSCzUU7CRAK2RWBHBAqGpOQUwSzFqMnl2TTfuuOJCZgHw2pY
2f5mffWOvvn6Y+qeFlra2Dzi+erKwgzzqpir9unSYprdcJUd2LvvYF3Jxvrs4gezN3/0xf1L
J776xZRTNZtO3n4///in45u3b97agN/+Dd2/X2+tPrk+fVxabvLMc0CvCXsXx0HANYCVGy0O
BwTadrP1RhRT8DjoCbsBcq1mkxnebgeiNGMQIcvMiFwJgUFGIuSANKAB2NCKQZ6aqYqopmw5
5tzHlJLmlJdUQbUUppCDslMjyFgDODCUnL1EsSnlGWmKiAAMjtRQLeH/z96bxth1XeeCa629
95nuUHNxFkWKlEVNnu2Y9CA7L4mlpDP6GS9/2njAgwQ0Gm2iAQONRoBGA/6XBlrqfukHqdEI
8p7Tr9vJ6zhOLMVzJFm0LVmWrYEUJ3GearpVdzjT3mut/nGqKEoiKXqQLTv1gShUse7dd+11
zq297hq+jyqPdcWSixME5pq5sNaixBFMJCthaiBiY21ly4QLRa3DYeKDD8ArAU/O9edHYVBu
HpvpZhP9+qIfluIh6WQj9hWoSZM0TlCkHOXDwSCuYTB3HqTynUxbSYUy9a6dM50xKare+XPl
4qXY1MgKXLORECEZdGwjRhOUARiUrRpFHwSEURVFSIEUUJWwIcQkJjBGrTKFyHO8KDKIjYkZ
sfT9HDGxNs6SKLFkUAiZgcUaJggQVkZ9Culk1gZjWKWR08FVcUZuxJERDIJBBIWgKCANGycg
gBI0HehkDAByCKCNYB8qqsp6fLOOdaxjHW93XC+Qet1HVV0TMIYrQisFQIQgUpdlWVYcBBGd
ixrmHjRUh/DiwYM/ePYH77j11ve+5z0gwTRiuiKgRlSRkIwDCABAANY5FzsyRlUAZK15StUH
BIhjhw5q9oKIAnEIrg4YvEqE1iatVul9VVSdOHMu9hxUlQkaJQ8EJEVgsKII4AMjsx8OB2dh
sjNuyBHTdHd8avvmg6dO1xeXZ1wH0m4Zqn6RD8GfqEYHLpwVE3daY4kJLqPpibPpzAUrZvnQ
kZttNJGX/RdedkRZb3k0d2k4dyJTM16bxdNnJ2amA2qoS0sCILmiUBS5GOuy8P2SmFKa2jLV
3TAeS+mXLuUX5oYDjlKn1mKiBDXUDKQcoQdwAoGwJhAQAUEUNgwKCgjcyBaLgioyoCqpWGFU
Bq6lDOIDBBZWQFRDbCN2jqwqivGKQRQBOEDNWgKUiDUBAUSANgQ0CAIKKELE7ICcqFUSQESx
JAnW2Irbgw4vhWr5/NngRao68XXky+mxdlKqRW/bY/HmTcPlfo+lPTXVdVSyN1JPRGMaGUH0
CsVyOfK5ePA9D2WeZQlEpnS4MjWdWTc5uaHDcHFpvsjzKM8jkMQZdsaDGEEnhBrUqBAQkBFq
+u+agpppVF2a6i8IIikCGkQEZfBMhZohVuPIIr7o9w05FyWtKILgEaQVR0HCCBhBATEwl2U5
YsBWBnFsnFENLOFyCIQK2rxe85EEUPHVlnO4omlrtQldARvrFPSXxMe5jnWsYx3ruHH8ZPLy
l1tAAICIZJW4GXxdl2UZgtem/4NIQButvrNnzz755JPtdnvfvn1pmlVVrqoAKCrATZu5AVw9
T6xzcRxZ83qrUIFADKoFiIxrpVGUGPXUci6JwJSaGIgUAlmMsnowqqRM49QAaaOIAk1VUgSA
UUzT7CUKQaAONY9GJSRREgWI0mQm6W6Z2XxpqXq5P1iueCxtUZwtVKPFMiyHesT1iXyxVeeb
Wq2pzsQr4ftF4InxsR27d/YPHjn94+dHg36VOewmWTumfj1t40vnz/V2bB7fMLEw3jpfV7GF
ESWF7cSYkqdQ90vno6TWjg8pS6E26QxlqahzaPvIom2kiIOKBwNOISatGwE+hIZknBUIgFRJ
VVEZSVZDoqYnB5SFWUgVFZxoFaBSDYhg0QokzG0kUuwDeQQSUFBBBSWDYhEAgAQQiVQMIisI
AhAY0aYnzYICghOhXl2fLvJeKUWQuvBOoCPaJkyydGJs3LW6tjvW3rrZt1s5h2j7FqPKHOpi
UJRCEY1PTWRZttxfoQEYC0YlUR5HP+YJagxJvHxxqSx5tCPkHEpHA1+7Im8blTqiwGRBICiy
teCtMkKkVoLiagWNkEBUEAmAFBSamTsFQwaRgkqQoCrCHoTF+7zIrSVrTTUaPvv9p5NqtGfn
tm1bt3ZbaQFQsCqLpzCCSnN0SGmaIIAqkFFhgUbyWRmEFQDo1bgJV0UPXzPc1/CowdU4R9ax
jnWsYx1vT/wEgdQbyaKaH+va+7puBveQjJcAImSNMWZufuHAdw8URfHJ3/nt2dnZ0WhoLYk0
GriggKqg0mRRFIniKEqShOiKA2cNxpARUWayLoqcxpFRsFVlKlFmFmNcFKB0LrYZVIM896M0
TQ2RrmZoeDWYIhBVo4Ci6MUEVe+rYd8lUtdQ2FFnsXxfMntqRo5Xpw+u9GR5qZWkDiBimPIw
jmziaKzVGo+zzen4WJRpgtM335xEcXFxruwtD/MiQB233cULF/KFkduwM7G2P7c4sXVjuXHj
2YVeK466t92abduBYunIkdBbRNuPsMa8VwNXA28rqrytBDxwO3WtNKvFl5KjVxBSIoTo1UAK
AMCrIIBFtQiIyAheEGg1liIFQFMhAGFqEBkq1Fw0FwhAgcgj1UhEpklCWlAQata3QgFVEJwC
ghICKDAqAwCCBTCgBtEAKIIVNlUNBUPt2YmmhB3ErsGpLDMucuOTna3b3ezGqp0shxInxnNf
nzh5UoqCMNRVHkI168ttmzaRSouiFlgNo1q8WMyUsoCW4wzwRG/5UnJulESlsCJGUWQsYNOe
D8DCiqyEYjQIW7WqoCqqYBCliagQtSExF0AFEkCloBpUhACMAisx+yJH5TRuWWcTwCNHjiyd
OHr06fSWnTveceddU9u2ZZOT6FwIVe5LAGITAVJkDRmrqKoeBFSFtCFCB8JVOcTXNxqujXE0
gdRlUaYrxwDXsY51rGMdb0+8SSB1JREOABhj4IpjQAFUNIQgIrQ6sI1WUFTB0Mpo8N1nvn/w
5Zc/9uGP7Nmzh5kj5wJ7bGIoJCRUBhFpVMqctUmcRFGkKioKKpfNUISgRo1FUCGHxtQC4DkB
I0qKbhTIWENRJF5cFJsWFv1BCMHGEQo2jSuNhI0iACopkCiGYBkUMBKEoq4qXmGfrlRjIapc
MtdtzXGxvLJMHmfS9o7xqXFrYmVjNLUZRC1Jprg1uWHzxont28rl5XmKsDU2s6stebGy1C/E
ydhkAGtqXT57KSIbq/Htyd233db6wHvCjpuGRQ3dzmRYGZ55oTVcTIt5lxaR7RYVQClUsiz3
EU0IWmvObY+MDiNWv1qqBNNcA0QFAcSYNLEuQgxBchQVBCSj4oiswRQt1DX3hyEf2ao0xqWt
NmDETJ7ckAhQncHMe4psTKoMHEIFUhtj2EtkkzJ4bXTt0CKoACEQEIE6UCGMWKgaVNCvJjTO
rOnGZtbFmQqKlIQmTcc2bOKZ2QJ83a9HoxGzbNyxM7WmHK74clgVAyuhGo1aNt7Zmk7T+lIe
hpIvIqcRgHNGteBQiy6t9FeG6IqyBZTYKAIxrBTYOURBI0CqKrzWIyZrEkQNK2ejlwMKCEQi
qACiGkBqFCYkJIcUKUpROFGLMNFq8Wg0mF+IkXxevvTcj48cPDa1Zcut77z7lj27xyY7aAic
AwZfBozBoRVUREtGUVlBUBRRAenyB4TX6QG88X23HkWtYx3rWMevBN6E/uDKv+YNveGVPObC
7L2HZqZPFRwJK6goKAu/cOjQd5/+/q5bb73zrruMMXVdEgEzY0OJQNjQExoyDEJEcRwbZ1VB
RECaXp/LQI+GgYzBQiBOEo+2qEK/KKLA40lnfnnFuDSLYvAhACdp6suqqr1XMdaSQRVExOYL
iAICqAizBVAWCOoZIKZhf7hUDkOQC1CMSJR4zNFNrexmG28SmULTzmIRCQNfBA1jk6277/Bb
Ny0QBWeTD32AlntjVR3Pr+ihY5s3zxQKw76fMc5Yc/bIyfLS/JapmU0TE5KlPQlD67sz2fTd
e1K/MjHvJkPtghGTjRx2OuiyJAllzEYC2jgpAKRmFEK0hNKwY2PTdANoyAIko4E899yzl+bm
3/3e7du2z5KxoEYlRmidPnH+6adfPHL4wvJSvjDPytDp2Imp5D0fuOl9H9oeRblCjiZmH68s
5UePnl1eKpyVHTu3TEx0f/jCwePHT3Y7U+997x2TkwlLSWQFREQQLCIBNO61IaiOqlmIxrKx
lnMtwNjXVIegAcAimZEPVVl5i1Y1tS6ZnJ7utBNDdTFW5X3kMgpe8sJUoc3WRONlPDw7qk7U
9SCqJ8lQnY/K0Le2IIHImrKOaokFMmsyF8fYSC6KEaCgrhGAEVgLo2G1b4kIyagCEikpoQFQ
Rg2EnhSMMdbYWlvGQulNCFalG0W98+d9kTtRRgUg9nLp1PkLFy4deuGFW27bedtdezbdtEPJ
BB8KX4U4StIYiZDAECkwAiOArH0wed3nkyu/XskvdZnkdh3rWMc61vG2xfWoai7/Tb8sWnwl
s3kIwXvPPiCAMcZaS0iqQpaMMydPnfred7+btVr3fPye8bGxsiyNMWTM5aUQ0ZCJIhsnibMu
jpMsy4wxIgIIVzyqISxERiyY1dpaRBCSdiYWhSjrjs1u265RCklGrXbtrERRKeLaLUYYDUd1
XSnr2j5FUQAAVsXOQEWsNQKA1jLooCp+NFx6HIaHXbgYCqnqm1rd28anNlhTrywuLs8v1KNF
KAMyIpcQyoiKVtKLo8U0vpi6Hw17/3DoxW8eP3pWg291x3fcsmvfB7ffsXvn1k1RORotzS31
Lh45+uLF44eXDx/USxeA8wtLC4dOX3z+5NzJhXJQU78/KAe9GHyG0DLOeOCi9qOyGhUYwIAB
aZqXVZEVAxIz++C1HIWvf/XJf/j7bx146qX/8rdfO3nqIkAaOGVpf/lLT/3Pf/Y3/+HB7335
b09946vzLzy39MJzS888Nf+tRy/+X3/x/a/8f4eGSy2D4yLOe/rRjw7/45cPnDqx1OuJL9Mf
/uDYN77+fBZvPPjixae/f9DXjEiXGSV1TbcXV9XmkERjX5uykLIMVRlCYFEkZ0yEgBJY6xCD
jlkz0+6MZdny4tLLh19+7vkf/ejF50+cPj0sciJEEPS14YAAJcNcEYQ4P2sAACAASURBVE7k
9eE8PzIczIeahW1etEZ5u+aWQtu4VhQb6xjUqzIAKBowToxTIiRVFVFQAkBZHQEFNITWGhuR
MYAgBoNFdkSRMZZisimZKEhK1LKuWu4vnT9vhMlSKYGtDaCIpCVfPHHme//85Jf/37/91j8+
tnj2jFNBz+pDPizyvPI1CxMqKaDoqxy2Da76ppM1wGtFLX8haOg4r8RlJs0rOTbfcjQvhlcl
8rz8u2v8/rWPu77BP/umrvTY5XWu4O58S/CWXYvXXP833c7PaMZjD/wc9vCLucqvX+fYQ/t+
Ye+FdfxK4M0DqTf+2KSmQgjMfGWMxRyYxXs/P7/wjW9+Y6nX+9gn7pnduIGZm4BMmMkYY6gJ
YliEWZg5iqJWK4vj2BqDTUswvPYIQUAUYxS0RvTej+LEGJSpybHp6fGxic707FTabbVnp7CV
5sLBkEmSOEmTNK1qn49yZbEIqIKgSKogRACotfdKFBDZUbDGO3tR9STYFdcqPMionrLpdJpO
jo91pidy0Iv9Qb8Wl9iUfLS0aM/Npb0882IqpiLcNL7hA+9878ZdO3pWX3zp8JNPffdcvrT1
9u3bdkzPtKlFhV85d/GFp8899hV59Gvd73wveuWV2XZrz0c+svm3Plnefsf89FTejkyiSSS1
jEqHtTXEEIExgtA0eTeaeiCAAhgAa1FZWRkcPXzi+PFzxQh23rz9Ex//5OzM1sCOqPP1r33v
P/7Vt48cWpEA79gT/8GfbP93/82df/Cpd9x082QUxXPnwzcee/7lg5dQ287GPvgjx8/5AC6i
VtaNk+7xY3MzU9P33vfJLVs3nzp5tixKWMudWEvGYNN81sxWMguQmgRzGS2Olhaq4QCkjqy3
xIAkGAW1ZU15HoOCr04dO/r97x34zlPf+eGLz//45YMvHTs83+8Fi0xQYFFoP0AdRdbZuKo4
r1XIRS6ajN0WgxuqasbajnPIfpQPR77wlkqrlSU2TsGBEgoioDIor2U3V4lkCY0hY23kwFBQ
ZYPijDdYsK85JJFrxYkVTYCcyGB+rlhaVmEPXKFWpDVCxV5CSIw1NS+dufDM44//l7/+wg+e
fBLq2oJBwVCHsgy1D0FQoaE+b0zQRumYXlsov/wuu6w1+bO+uX867H3w6Cp96aP3w4H9u3+S
iODYQ/t+DufLqv7MUdXXC8489gDu3g9rBh59EPbv/iUeaMce2nffI3D/o6u+euS+XxiB+1uA
Yw/tw/seaTajevTBvY/c99YGg59/5FdW0KYhmf/8r/DVXsfPGW9Cf3A5SLr8P41qWAiBhS9z
C8raP2PNykr/iSefPH7ixN59H96zZw+qioizhkNgZWOIma0xDbMTKiJAHEXOWVEGRQFAEXO5
crVa9FBrwEXO53lMWJfDiSSb6GaalyIuVKPYqWhl0lZnamppWIior0OcJi6ykA99VdfeoyEi
NAoEjd4aIYL3XhWEUAnV2Va3287GKAeo645m462p6e6EMUaUbRQnSVsAqDaLNoeoknoUjQa2
KiizNrUzM1NpHHcdbbvtpg133eIPnn3ppUPHzxyHMcxANs62l0+borc8VWcb5i5sXBrSmfPD
mVY9Ozu2+7bpD34ohLI6f2Lpuafsio+IvFQFQVkW7WDbmbNoQICQiJoe6YYbQhTEWtvpdG7e
Ofvpf73zuweef/aHz9b++2jee8fde1ZW8md/cOjCBQGEjVviz/y7T7z/Q5tFy34vevzrp774
10+xQK8Xzpyee7+/2Vng4O++69Y9t6Z1zt/61jO9pcU8r6LIcJDgwViLhsioiLIEBRS9HPJq
EwAAqW3b4K0PUKhhlkopRhqVRbvI7WhYFKUmEsd49sy5gy+91B8N2IACKZkatVfk2bCfiGDs
Q5cSSjt9hZoFLAbIrI2EW8IbW2mpYWHUNzZCg6WvgZGMkkU1GBQsADMwACKpimrDJIAKiMag
ISSDRIpAxoA1rFAFn9dQsBH2iXVt61BVVNnX/ZWeQ/CgNTNaE0QNKikioK+9hMo4APTzZ85+
7dL8saPHPvjRj23fvQsQJQSPgM6BIWMNCiuzCF9O8/LV3nSXK32/7Nm9ez/34N5H9l9Va+9q
OPbQvt37D9z/6Ftmz+uVaXZ99q8e/OLu/fc98IdX4TLf9dmn9LNvsuCNPOY6OHrwVQWdex9W
ffinX+onws9o9lXx2J/vPwD3P7rmx7fae42gzYM/s6DNW+GKG3nZ3/303v37v/iVY59dp5Jf
B8D1M1KwJvBqjbGreSQQ5uA9h6AsIhK4rnxZhVCzBMJS5aXDR5559tntN930vne/q2UsF4VB
Vvao7MiQgFHSoChAiGg0SWwSo8Gg6gUFCAEMorMQRRpZNSRo0JFGUqljypBigNqX7enxeKJL
qSurQWJ9YrmoKpu1xyemFECNspFg2bVim7rAdcgrKiWuAYICISKYEKKqSoStSpBgRCZdtq3d
TiRnGHZTeueWzVuztuSF8aEj1jDM5aPDg/nD/cFcgYsLw3Kub5eHuLzSAc4iimNXeV7s5SVG
yc5tH/y93/rI3g93KUnU7Lzllj133jEzNVks9arFxQz8pkg3L17YeOIV/f6PTj7xgxJi++73
du/7veK2u1bak0naNWUpZeGdQXSZRjgKqMrIjB7EUxATSCQJjHEUpak5d/6oseU737lrZaX/
3DMv1qOQJuZD+3b91//21t//1KY//vQ73vWuzbGzxVBS156dmUxSYFVDEJnuwnk+98qoGtpy
UE2PZ1u3TscOnMHpyfaFcwvfPfD00sKlm2/eGreRgQGNJUIMol6Q1ADa2lDlgkQVJ+whSKVu
hdrnQnSqkDOjYi4fLPQv9YfnmRdh8WJ58OXy+OG0WmmZykFJ6rM4S6POcFDPLy4v9IeXgvbT
TjS1wcSZ1IzeWwxkOEqjKEvROGPiqvZDX/dEhsYJZSYYV6sDBhs8VQFqUAb1AAFQlEgQBLXJ
oBI0g33MUNfEEmw5sl5jdeCgmJRqrPIRiyfu1T2m0oJPgVpi0kCRFyMsyJ58Ceydq8j5gOIV
aj76/Itf+o9f+N6jX63nFxIWJ+J96SV4UFBsassNKSiwGGqUAAmIFJuPFQAAIsLMb98eqavU
14499Jn9BwDgkfvWqh5XFIqumdq4cqHL2nz3rSnQvPZpjcLca9Tkdv3up/euaeg1Fah9+1aX
em1B59Vi4QMPvE4H8Epr9z300AOvsebNtQKbF78KvviZNxTJrl2XfKMfLv/flaauufrNzb7W
lq+BRj7wjaqK19vOT2jGlVtuVBzvvHXXq8s8dPlxDzz26p3zZq67MRuuxNXXuQH7H/jKFas0
It2r0pLrWMcNiBZfntFTvZyOaup6wqGuKx+CwCqR8/FTJ//5iSeMc7/5iU9MT03VZWEJnSGL
YMkYJAQkIlBgZhE2BlxkyBEQGqBIIGJwjEZAVGsNwaBGNhhVCS3mzV7d8fOjHx82cz0XxFob
pUkQrqoKAPOqFsUoS6MkVURBEAAwGGdRK8sQsRoV5TCvi4oDC7NhocDogyUMtQ917RRm0rRF
EsIoi3EsMsbXXPuVYTmoYeTa5216Jh0/25467M3z55eOHD259MrZ+pXTeuZc1u8vnXzlK3/3
t3/x7/+3P//f/9f/6S//4q8f/+ooL7d3Zjbb7rTJdm/buW37zdRuLYRizhfBwHTsthvcUOV6
5tSpF16Y6w1W0om5ePpHl4pXlsPpuX7OUCFUIhJAfGDmAMwoAEyiKKRKgAZRAIKL+PCR5586
8EyrRXfcdWuaujTF3/2vPvLf/fef+h/+xz/940/9Rqtdeb+cxqYshi+/fLDfr1lgajravmPj
iZOHn3nmKeEKsfrq1771rW9/7d3v3fixj9+976N7br9r89nzB9/9vu0f+vDtQJVwACFCMqhE
gEQCqsqgAViNp0wTq+lKqedHftTu6oYNw8gtjQb9xXlcWWmP8tb8cntheRvQbeNjt3THt7XG
ds9u3r1h63TadYKhqPr95XPziyfOX7pwaa4YjGKFTDRlbVnbSpOAeH6lf2ppsfKVsBdQGyVJ
nCXobFBSUZVGu1oBmtk9AF0bXkDVhvyqqVKLaGAQFRI2IgggsYWxxMUGFTT3VeELBs/sQcQA
GAGjgApKqhbZQCCoQWthQJLADojz4smvf+NvvvDXp48csQAo6kMoy1KEjbPWOUQUEWkmCddk
jM1aB2Hz9ZedjoLVLMVVjtfHHti9/8BqEejR++HA/s88dAx2ffavHtwLAPc/qvrwvfDYA3jf
I6tVwkfvv3qF8LEH8PJCTS3pgcdg12efevR+aCqMr6nsHTvyIqwdvpex69Y7AeDFI2trH4BP
H1V9fYLqsQd27z/QGPNn8MiBa+/4wBfhc82mbqRyc+/Dj97fRI5XibRWLblyqav67Vp+gDWP
PvL5h441Uer9j76+1Hlts294ywCw5tvr4SrbuWEz3rjlowcPwGsLe48cvPUp1aMP7gV45L7P
3360cQU8ct9akHR1192IDVfiuutc342fgy9e4cbdt++9rvD1Ov5l4U1Ke803ugbvfV3XdV2L
NL2zIqLWRXUILkoWFxa+88Tjg/7KPR/72E1bt4a6bBR4URRwdQCw+UqGlBURIxdFLlZjAcEF
sIwUlEErDRJRQAENibEpYgyezl70h88Wh0/UZdH+6G+k3fETiz1I0okkFRsNSh+3k+Fw2LW2
MzPTO1MYEFDDwTugKEmYURRqX2lg75UsgTWCWOWFa3XFe1FSAgsmAbTMsSGpK5/XhigHWCyl
2rjl9vf/QefW3RjpxSMvP/3YN5974YWJqbFb77o1FP0yHxx59gcvPfeDefTDdnxmNHrp6MH+
hfP/Zu8929IW1xhH3UF3etDtXegPD42WqZu9I3Fta1oJTMTh1LGXllNKNm6G9pbB+PYXL73Y
1iQt6gnuRZiaDguhYtPZ3bCYC0BD7dBcBfvBD75n86Ytr5w4tWv39tmNE4GHxlkBrX0N1sdO
tGKutC7k8cd/cODJQ6O+dtvwsXv23LK7Y9yMyniWmHs23fW+37ilyPOJiXFE3xX7+5vfGYIk
Sep9UYUakVQYlEGVEBVQWVVAlXytviLxMWs2sXl69qbt3Q1TVAwGx6Ma6nSUJxdXOn11dWm8
bxPGYFo22thpB5fUCpVUoOqsiGhwUJRBihAFbkUJehaWCZs5dL2VpUE+QNDJNMlcZF1k4yg1
pIGREJgUVJWa6q2w59CwXq3e0qraVNZklQweAaHZjKpYtNYlSZpWwkNfD6tcI1WRAA1tWtN5
RXKZr7zhPFUQZTJIQN7XABInydmTJ7/4n//zR37z4x/86IdjE+V5kTuTxImxRlVZhYhUtali
X66Pwtps7C8tkLpCjPga6rxNGevYQ/twf3O0vLH695rsUVMifEMhpHnMg5+7FwBg12f/7P79
9z3y+Yc+d+/PUqp5bcLqKsasliuv//RXT8ld1y8e3fuw6sOPPYD3PbJ/N+5/jV7yVZa6ht+u
6Yddq7XL/bt3A1xRd7sBs298yzeIq2znRh5z9S3DG2PiJlxvEj1XuYg3cMvdkJ3XXef6btz1
u5/eu//AmhsbS68tfL2Of1m4Xkbq8mxREzExc13XVVXVdR1CEJYQBACNsc7Fozz/zpPfefml
Q+965zvf/773hbpWZkIFYXptQAYAxhjnXBzHcZKQMQoIYBDIACoBk3gjwQoa7VqaEejML1XP
Pdd//InymWfHL8yND0f9M6dt8E5gZWF5cb5XCNbOmiiyzg3L0nVa6cQEGwfGIllVEiRKoqzb
7o51kywlY5i5LspylI8GQ/VBvOcQRqNRvjKQQgxQEkVZ5ELeH/YWWDjZuGHyjjs3fmDfxN3v
h9nN2m7vuHnrtk7kjx2kwy+lRw6FHz3XPn/hrnbn9u7YNPMUK5TF2cXzJ4qFlUgksppX3dpO
m04R4NnFC9+8cPJIv7hY1KcvnV2eO10vXTz38tEzJy+9fGH5AmULafci0MXBYK63tDxYGZWF
F1ZUJVVohs8UQUgFQcgoGq78YOtNk3v33blxS1tpYOPCy0qQIZngYl+HknB84UL2d//Pwb/5
60PnT2orMR/9yB2f+FfvS7NRkuWtNltXksk7Hd2ytYM0BFwBWEYaJFkReE6wbx1QU5SSoBJU
GUQAiciB2lDrMOecXWd2y553f2DPu95j4vTc+XP5ytKGLL25290oMNEfZfnIceF8HdWBhqWf
X5alYVbrBLkuUsahJX7GmBkbdcm2ne1kWauVxs5pCPlgWJdlEsfTk5PjadoyJjXUdjY11oJS
0zympIqgFtGqapAAa/ffajQPKKKNqPBqx/daaig1USdOI+t6o+FSMRjWRag9B2ZEQWAEBhUE
VmERbZT0iIDQWFt7H9g7Z5WlLstWHA+Xel//x0f/6e//Ybi41I1TDmFY5EVVMYISsqoiCsBq
i6GIsjRTsdef7Htr8Wqzub6x2xsA1mp2u7/46aPNx/c3oslwHNi/u6mK7N5/4I0f36+aYboO
Xp98uuFFftIX+klx78O6lku5fhbr6n67rnlreb7LgdaN4afz7VuAN79V3s7rvNV3zjp+bfAm
gdTlNBIzNyGU956ZmVlVrbFJ0gpBAocXX3jh6e8/vWPnLR/et89ZqyLWWmDGKzjQm6WaZa21
SZLEcQwAwNpoyngDpdMyAUkUsB5D2VBU0aFX6m8/jd95buzk2bFiCJBXrlyRUc1VN0uxDr4K
Yt3CKB/mZWd8vAy8nI9aG2Zt1hY1xjgyTgADKhtEY6MkybIsTVLnHAJy7UUEAVnEez8cVCFQ
KcREExMTN2/ZvH12enqyvfnmmZmdG6tYB9Xw4rHDh775tXju7L6NYztwGI7/iM4c21gWezds
+a3tt32wu/EubO+B5AObb9o4NX740olnzh++IMORBIs2MUmStiprjiz3v3n87D8dOv74oZd/
+PKLSxfPL50+2b9wcTgY5OIxc+B0Ynp8Ymoia2VgsOY6KGszrweMGBCEUAhBxAN6pIp1BFSI
DkRHCqVzgKgizMHGdsOLzy3/hwe/9bdf+PH8+TA2Ft3zm7f//qfeNz5dMwxYggqAICEpS11V
hKAihEAovi6MEdIAKqACwAiswLCqFUMq6ChiD54cTE+mmzcm491yOLx47Fh+5lwrLzqhGsPQ
Mpw60QRGsQ6sH2lZaa0aiH2M0LVu3EUtwLbSZDCz4rpAiDqUcjmUfS7mV5aGK8ttNBvT9rRN
unGcOZMYjBANAilgowYk0GhUA6CICr/aaYSr2Z8r+7hVARhBDRiiGKgVJYZoxNVQfe491x4A
ACEoM6iosHDgwByEpal9I0AzaUFEHAICkEIoq/GsY7w888RTX/27fxjML7Xi1BoDAFVVoSEw
CIQiwg1h6GpNUl9r6tsPTT/No1cEWW+YvmoO5teEZG+ot101MLoe7v3D+1/fl9L02ly3t+en
eKGfBk231nVrPdfw23XNa0pLe/fCgf27f4LxxJ/Ot9ds9/qpcQO3ytt5nV/InbOOXwdcL5Di
VVZoqet6NBoNh8OiKJq6g7WWiAAxhIBE586df+LxJ5Ik/c1PfKLb6bL3ceQMgjNW+dW/pLBW
2kPEJimFiIbQKqgPtdSlER9BMMFIPck8ubhcPfPj0be/mx08tmm+P1HWNRTDcYxvmTET8cLi
XBbHBLA87JcqNon7/b4iTG6Y7eWjmnBi2002awcGUVDCQMgWxSAYImdd7OIkybI0SWJDZJ0L
ISiAZ0WX2fYYtLKAGDvXyeI0MTbFbDxqdSm2Ph0tT+WDDX40EfozphiDIsPgOCRlmGZ7Rzb1
2zv2/Nvf/O0/2ffhu2c2mN7K+ePHDp44Mm+qaPemyTtuueWO29+x9ZbNE5vmxR0uQi9pVy4F
L+0iNxdOJ70LWT4/vHCcl+dMXULgfDga9Qd1UTYzcqiCyACKKIgCyIAsUiMF0VKxYi3JKChy
QBAX2TGupv/ub3784P/y5ae+c7o/1I03mT/8N7f+wZ/eNLX5guA5CZ6kjRqrGhWDGClbUAdq
hFEYDUXsm7OeVRlBEIVIEBVBjbEcRBhDrcn4RLZjS7pl2rUSqStd7nd8mCLqGCKSnKoFHJ2D
4riUZ6AcdS3NtOKZlm1bMMyhRBVH1jAlNUaV1nk5qMrTvflzw14OtY3N7ER3U6c7CbYdxKo2
aRsFARZlUVFqZHNkle1s9cZDRES9gvoSEZp8D7OoKqOW7FEhIWtZh8NhRVo7qpGD944MkVkV
PW7Uuxu2TVVgAW1Um5tfCyERolF0gFJW7ShqkX3x6Wf/6Utf7s0tTLS7qYs4cFWWhgwZA4SN
onajVNy84658y7w90Ry6xx76/COwFj9cmdZ4TdBzDSKf5jH7/3y1mfrzjwDc/2fXm4K69+GG
kOHVbufPvGbU7JrPu8KYpu/rJ8C1m82b7MaqMTcU0sFV/XZtP6yNKT71VNOMdeOh1A1t+crr
cu/nruhIem1v+8+KN275pwtQruK6n5M918S13bierFrHlXiT2oG1FgDyPM/z3Hvf5JMaRTBE
EGFQWOn3v/XNby8u9u65557t27erMAJICBoYVRBV1zjKmblJaBljkiSx1qoqAlkAA8LqvVYE
3Alhc62dE+cXv/lU/3vPtpYW2lUeqZQoeTdKd27KpsZH84ujsxdjgfbk2AqXC/2lLI5Rsaiq
bOuW7qaNC/2+tlrjmzYZG62KgRgEa8SiGFSDYI1xNkoTG0eKaKwBAEFMszSK4mCNj6LKOg9W
MWGKOUqpk2LigLShnqqBPWkgqBAqMiOGlSrkRVmPhpIP/Gg5rkbbydzp2ls9+uXeoQuvHDh3
+ILzO+6+Y0t3plVC2spovFPFCbh00qbviNOb8/7s/OmxhdNbwd82PrlranaqMzbZHRtLW6mz
RgSYYXUCgBWCQgAQAIYmRYRBIRhDIqBqVRJLExfOFf/n//FPD//7bxw5NLIO3r93+v7/9qP3
/dE7pjbkQD0iNpRpSJURml4gEVgtYSmuhm7NSQ+mGS8jMASEYFABFQEtWvYK4ly762Ym3OSE
xNGwqgb5MK+rEYclX10K1VmpT3B1pBi+vLI8j5Ju2zC5a3O2qZPMJNlkarsRRI4xyj2MWJaK
0WKRj4Jn1cnxsZ2bt26fmp6O0xTEaW0gqEgdQl4Uo7ysQ8A1Og1seslV6QbikCauDyI+BGtt
alzZH64sr+S+Gon3qgYRWYRZg0hgroP4oE1p0IdQBwhKhMYQGSCDSGpADagjtKJYhZaxbecO
PvvDbz/21bnTZxM0nTjlOtRVtRp4GdNkyUTXIqkraDnfdriix3r3/jsffHDv2pnYpDXuQ9z3
0LF7H256zFcre1d2D12x0Gpv9XUec+1n4Cql1Js9B16NwBDxS7c3pbKfA+59uGmHXqteXrWf
7PVWXNVvV/fDYw/cdzmmWnvuDYdSP/GWd332KX30/ldde+A1iZufEtfa8k/aqn1N1/2c7LnO
E44+uPdqbnxju/w6/iXjesNBzx89gohlWfb7/aqqCIkMQXOyEhljiExVh39+4onHvva1vR/a
+zuf/GRdlgiiIRCwJTWoLBBkrUKhoKBRFGVZ1ogTNx/u1XtACQ4BpYM4MarDi0fDwaN2bj72
JYfCq69UZbLd2rkJrVk+s1AvVt0tN+/8+D0LY+lzJ4/W/f7OiZkUIhtFN92+x1h37vkXo1o2
dMbKM2frQZ+1qqUGBAoBQkBms5asKBlsqw1xa34wLFgvVPb5QX7IX9oi/MHS3FqSM3Yxcfbu
u7IPf2g0u7Hy5uLLLz31f/+lOXbwzgS3tVwcxyZpq6Zcgee6tuUK5iNrJianZ7JOqMOwLEeO
FpFP9RZHVZhIu4PzvTCsACTn2od6Y6uz1ca7p6bGx1rLftQb9Nsu7lawPL+AiW6bafNYPvnx
LVO/tb2frCDkpCEgCjmrDgEYWJFWJ9NQyBhh4BBZMzY/V3zhPz36lb8/urJELtG7797yqX/9
8dmN2XLvfJEv1b5A5M2bZjdumoyjAZoS0YiwCAMAN2UnFUSV5ntU9QSihBUDs2ZViDR4JxLX
k6d+bI69ktHUnXZyS23jYyfPHHr26frciXbR63Ieo7gsYxsPfJ2HautNM3fdtdtFWgyHGkxi
WqQu74/OnjqzuDDfamVlCBf7Pa8cI+2YnNns0mhQWl+Bk4o4Z08mJhMBoRpK0jROUpamrxyb
7BmCllXe7w8ADZITXa0su8iRNYqgwB61F3zP2dy62KZZKeViDykMMczlK6rYpjR2rpaqqjwi
qaqCIBhQCBJAyFBTPSZEIQGr4AQtoBEgROccWWsj189HlYP37dv7O7/3SZdlHmVUlS6Ooigy
ZEhVg7Awqza88QCgqn/0J595y/8EvC0mBH9hWGO7upEA7NcE197ysYf2fQb+6meOl35Kk+DB
o7+El/654Ffd/nX8nPEmU3t1XRdFEUK43Pp6WTEGAFnkxZdeOvC9799x++0f/ciHOXhQQRUg
MIimkeeAVcZOBSVjrLVxmkRx1JwUjUyLkhKhEWmRaS+Pln/4kjtyckNZ2yD9elQ5HlrhTnds
++aQmMHZS8lyMQvtspeXg0G6fXbTjm3Lp85y7SGJy7pauHhh0y27J7dsXjryynBhMXWuJvSe
kQwIExlBVgQGbLRNkFlFImcBIDCnLprKok7uqlF1ul9EIU2d7ZMZMxlSOgzWg1npjJ2IW4u9
wcWqf3PiplvtVlYGDwu95fHN3dveu2smmUqWB1TXvsrH2t0sixeGQxR06djI8spwJBjU1LPO
jkoeBZmN4hl0s0pTohvb7bN1NXdpeVC5xHUoDvlw1K/mXa81VW0xKQqsTvKDoGpTFsKGGhUQ
m8YghcjZdq8Xvvylx7/+T0eH/chQqmF04Vz/P/3l18vCcxWqssiLujsOn/rTPVu2jCkVCgUB
KYg0SiooQAIigAgogIpiVRu2A2AFkYBKcWydulC7QBS1uiYbOkVotQAAIABJREFU9xQVCiPQ
JeWR8gihtm48jrJWW8iQiWEovmZAsNakWWJMVJUiDNjKVuowNyq6qcPUhALrwmc2NpXHCmNm
Y6lyUJvgBR1ZUfLBS1AwxkQRrl3SRrEGRFZTTnqFZCSAIjQx5xohAiiAiC4sLMa5T1icxVJK
EYmiBARFZZWhoGFNEEKyAGjVQCNotNYijqCGwSBYBUOkgV1MImzUppErq+KlH/5oemLiAx/7
MBi0ZKqqRkCMabUnigwKXy5/v017pH7l0AQSqzN1V7Jo/vrihrZ87KHPfPHTf/XULyMUaBgt
b5Tq9e2Howd/Lnyi6/h1wfUCKRFfVUXtKzIIiCGwNc46pwCIBAgnT5z+1hPfMcb9q49/bKKT
DAcrUdOi0hznTEEQBK1iAPYoajFKHMVGSQCgKWcEZQ8hM7aVw8aR7z992B48cvN43PeLCzCo
UhWMorjV2XaTWl05czrKR5NT3bLGPnF14cKOXbdsTaeiad9fWakst8UWvX4xN9fasoWXVuTk
eVH0qMFQymJqARAGCiYOKiieRGICrvJI2q007rNEld8uwLZ9xoZX2nRyUMe5HyezcymPlwYB
NE+MysBazjvpuRhsHCOkOLKMuhiBTrdat94ynZjlpfnBQk9Lz8YjUGzFDKttGic2XRB/rKxD
nIxPjBfD/mBhYXfLTSQZRrTAutIrBhqdVxn53o7ZrV2TnLx4oS+XpostEUshXFslBlJjwDE4
VLaorBqUFRktKVoJimpPHD994J8PLZwDY30QqMtw7GgfpB85QABjwDpIGJDQs6AqIkgTjSGI
CKzKwzVXlAwKigYAj5FQpFAh5inmqEngqSGO9Tsu95MYpV65qAoul6BYclpFlhKXJmkUOwQM
ApXYUVVD2R+OReOWYjBWYi0qwSRy7RaSdXVIuJba55HJNo7nWTowURwnGptBNVhcnKt6OdSe
wBFinERqCQw1WteqCqCCigrKZNUK4FqQqUqqhEwgqojEQIrWl/Vy0euXhVUdI5sFwryaShJB
W2FAY8SzIyBlEGARH4IAojGOrEMDqqxqyEQIRtmyRAgORUjU10jINaRJHNiXveWDTz+76+ab
p7dvYYO1McMQLEBi44Qio6rkWRibsOxfUKLorcQqi8Aj9+EjAHBdIoFfF9zQlnd99qmnftGG
vfraf3b//vu+9NjD9/4qXorHvvQI3P/oejZqHWu4XiC1vLzivRcRIFRZbX0VVRdFRGZ+fv4b
3/rmpUtzf/xHf7h5y2ZfD1EZFKwx2mjqIgIQoIIKIlpnoyRqtVJnDErDmLM6A+YITFFOVG7u
h4f88ZN3br9pNH9iZdiDGAQMJkl30+Y4SRcunoFh0Y6i3JcjcGZsqj8ajHor3S2bemijNEUQ
9FBV5dKlS5uSpDMxXsz1hisrihAZS8FbJFZWRFVUIFKD0gQQWkuQyEEGwQIxzFKyYnEZ+3Pl
cr406BYFzG3o9pYLrAZtw/kgVZ1stce6nfFOJ2FKvfF1FflyxRfBmcktW+x4Gnczk3su/PLy
wGZZRoldqakIXZtundqQG2q1x1NIaBTSdseNdS+N8pdPvXLq0hwTkjMK3g2WJsAMiyLEUPnK
WkL0jEM0NYmSGkCDSioIKtaokrAGBCC0wkuz0/kf/dGuj+zlUemLqgqsHBgEEcC6uJVFU9Nx
0ipv3pkhLoB6WtX8AVqdegMEUFDlRt+PyQ6BQSRmdAjeUO3Ih1qQJigib8pcC58PWYxIkS8v
QDVqGZyI0wx10OtxYsfGOlliJ2a2p9aigiGHKoJEBIQMhLOzM8VyP+VqLKINW8Z1plONpct5
fymvRmXfMFKEElnvWWuInYuiKIljZx0qKItoQ8YJjfEiKkC62su9OjXahPCgaowNYMu6HIwG
eTVUFCAqq9IIpS521hUIIQKwhjmIFxI1ZIxxqwwIqqhiwRAiIRhEi2RQDYhBJQWDAMCKRlWM
QuaceN+7dOmFH/xw7+SEthMgCCDsxQBFRGTRGFJR0Ib0cz2Q+vngl6Qk8svE237L9z78q3t7
/yrbvo63AtcLpKqqQkRjLRlUxcghkQksALiysvL0008feungBz70oXe/6+6qyjkE6xwIC5IK
qIKujjKpEhNRZE1ibAyGmg5mVUFgUABsM82wO/fsC/XJc+/avWNl6cJCfyk2kQ3MiLSpQzPd
wfxSOd/LIodglgcr3rXG4sRF7ujRIx/ctTPGKHCJnq0xbDUfDQdnzo6Nj0cbJkblQAccBQEF
toY5KCAAGkUCMmBVgjqbC1RpNGTtZ7YfPDJtbG/No4X5sl6K+xeX5qOzr0ydu6njNrNE9Uo/
qWWb68yOpWlizpw80Q9mqt12xlw8f+HwsSO7dm81caeTGFcIeUiH5cpSv3dmvi6qmM1yUV2o
yqW6NnNDByqCkJdc5OcvXRqWleu0OtYm/z97b9Js2XEcaPoQEeecO777ppyQExKJgQQniDRJ
gFQs02QFoGSmVpupzXqjHbAkNvwHsjZr4wZo6w3TeqMyU7dIqtQaikCrqWJJTQEUCVIkQQAE
ch5f5hvvu9MZYnDvxc0kEkBmCpI4QKr37dLfOXE9Im6e49fdw92YjLTvbKzL5X0rWuRJnKR2
ZjkyW2xYKk2BGBAYklVRRRH0oAEhkvGq5fHj2ckTH4FofAoh+hBDDCF48Y16D76JNmMiDxys
a/hmkxLWuV2gqAp48zw+gqJKJlIwZk5bMTnvQ/Bl5ZvY0GxqtnbGG2sYo5ZxLICamnK4k4N2
83zBMjd15UPWbbXarUrC0mC55Yw1NiWsfBDCKBqjGMCDy4vO+9mN9Y4hLvIdxc3d8XY5pvEM
p9XiQufQoQNFB7nwQGgoIyJMqj4qGrxlLoEq87zuZRRMiHODHlQEorI1jBSTpLoOaHysfVNZ
FUakkGyCzHCeZTXEQACOZ3WTfDCAlhkABTQmbSSlJBEFDObGECIjGiIWMoKsyvNUQiJBnneQ
zIxJ1vmYzr399iOf/Nhy+0gVAjMqoEiIDKiExIQIiCKCtBfa22OPPfb4sPOP5EgREQKokmFj
2CQRVKjr+o3X33j57145efKBz/7qr4Amicmy1eQTACqIqt7saIyKSVksU25tbixGASJiTpoA
gBAN0EqA8vW3JucufOr+kxb10pWLJiQSCiBm0M1Wl6apma1vZh6Yqa6TmlyMbVLIW9nOcLuZ
TBdW9m9euiqT8eDgMjJbgPH6JlZVZ2mBu3maTa2AkAnqgW7+zidFFnJIASEhN4DB2fXh7sW6
2YW0T1ur7aXV5dVrGuvJaDqbXt5Ye/v73/8ISX5gyU/KJaBW1l60Lhg5H8sU5MjSvoML+w/Q
obKpbox3u/u6Sa0xxMlkecuBef3l7+xe3Dhx4FgZ0pXJdN3Xwas2dT/jSSozSNKEw/3FfYPF
HKmF4CCx1VmPCCWarBm1vvHXlzbbdexgOze5yZ1pcStzGRWGrMkQ0RjICmLrgT1DYkJDmRBa
rcjU1kR10riAGEUgNDoZizXdmELSBGTnPehSkiSaosakPsTQRO9j8NGHOJ6Eso6zWT2dyGxW
NdWsmZX1LJZjBV3ct++jh453vd9lZ6rJSHaHA6YeaFtTqMtMJSdqOeercP3yteXBQrGyMpvV
dajRWkXMnbOJndJiURiXa+03h1tnmt31TMxCt2OcgvdgGkETiMhpmpcGIwNokMzNauCEpIpI
RDGlKJLmFhIAgIACI7AqhJSapmnqmk1MAjE4Ao3ikHqtwhIhQEiRbNYgjMvSCJNxZAwoxhib
FH2KIoKsgBYYCMEAWiRm5IRGwKiiAiMJQBJlSQZcQrKWyt3d82+dXti3ajIGIEWUFH3Sm6VF
rJlnR8nez9499thjjw899zakKCUhIvpx4okCIV28cunll19h5t/6zd9aWRyUsymipCQIaGw2
P+il77idFBmcM7m1jowCKqECEhGJYNI2gl64vv2d148dO9hdaV1+9dW8rtsJPfPEQr6ygJ1i
vHbdNCE3tmpipSlkZhZTrMtVg87Q+rW1Ix/9xNaNrZ1zZ46nE+2lwULWNknqnV1X2Fa7qI0h
kCASGY0QzkNXqgSIqkRGRIKIEFUSLk+2m267lUSub6S8aPd6S/tXdVbRdj0+f2FTUvfgUqvd
PpblVsHVdQWxhcEwLfU6qPLQ8fs7RxbrEArEaAhMHhqdlLW2iynq2bVr3fbCwuoBm2qdRbUy
WBx84r6D92Wul1IeoSCGEGNoMgbQ5KW2lFAJzGB3Wrz4ldf/33Nv1Z0CgBHRZTm3sjwPhfVF
1jacEUC74zpdk9nkHDhnjDGWXbuwKiFphZhmVZzNwnQayjJNJ01VhbpufIg+UogKQEym8SFG
TYIyL12O3NTeS+M1RkGJOUOL2cZY5gyoQsKF0dz5xYWxocAeeLzTbUobfJ8gTymKWGZT1zot
WwZV1AIxcPASI/imjim1XNZGi0Hj7oibIE0iLwSEZBJylcS5fBZlezIrfOCkrGDIZC6z1hg2
cDNei4go8zZ6SeZfY5jX+UACREaWJsYYQuM1JXSIKFZSStoqilZeOLbBN7UEk+eauXE1VVEk
FJUUExMxoTUGGVOMDCAphDraVmbZGgQDYIiMiEFAAAJQAEPIiJY5MUfVKOn6pcv1eFKsLiFi
IlJVFQ0pgRcgNMxwy722xx577LHHh5l7GVJELDL/bT9PnkFms7uz/eq3Xr127drTTz995PDh
0JRMcDMzBUBEAVgRkJCIQUAR2Zgsy5xxhKyAkTRJYhEOoQ1oJ+Xwh6cXlY/ff3hr68p0Z/1g
VONT1TblQsvuW/JJcDgzorPgxyHUhI1qDVqPRssSbMbT8S7l2cL+A2s/+MHFN9/uLQ0Wjt6/
Olge7mxXm9v9VluMVaMhNTBPZQERJQRBAJFEpCAJo3eo/SJzM1ODVeuS1Z3JpPK42utxqxjs
+hWJxfW16fWLsdfdV3QWLDOnicaTnXY9rnA47K4uXblwvhfHK0cPpFpQWYmUADJnCvOxz3zq
0hun17dvLO5fMRi5nllJh1qLJ9ruaJb1fCSfUgiRUp1LA144WuAsMCYXoa12qZPNUtio6qK3
tHrf0WMRdeSrq9cvzHY2ei0bfMpdLjFU9XaKZZJm3hg3huTIoSZjpQ4lIlhnV1YPZa5z9crI
uLzfX1FIIZTGcVlWo9GkaHWOHTsxnc6CT9OyQuD+6sJSuzhy8ohv0u7WZHlptV0U6xuXXnvt
7xkChCQyquqL443QcQWk2I6eUdDXi4bbRKbT0V6nhtRU5TxFfLK1Pdke2TwzeeFaRZZlFCD5
Olaey6atXKeYI7coh9jUs0YVM+Oq6HfrSgBYYpsQDCnPA4+QROZtXgABRUEVVVHEAOBNu36e
MyUxpRSipsSERhV9kwHkeWEzxwSAqZYoBrNOMW6qUNUta1DAztPpAJiQiZRYmRjAAVmgjDm3
nAGxilVgQCtyM4qNiMyAyIgWkQDJcjUZT3d3ewdW59XigefJ5RJSohARiJl/Rs+AW7z0LD51
6n3SZ17UL8LNDsTvPez9j5YSeOkuN/6cuHmWDQDurPPtK/DjK+4o/Klq9/jzL/7el5/6MNVo
+JfUjPh53bvHHj877mVIGZ6HGBCREQmRQgjf/c53X/vha5/85Ccfe+zTKcbkGyKYN69NMi+C
w0iEQKggKsRojGWyiEaVgkoEURCnkocwENp8+8Js7fpDjzyMENbPnTZNnSUTQUtCXVygXre+
vuXKAIzDyg+bJhkXARNpPZ1Us9lgeXU0GdXeH3r44YuvfMOGpry6cWnSDB5+uG1M2p3GWQBR
7hY2umo8dKKa5gWtVECQFDRaVCfRhGbVumWbTxuZmkhFO0pCaQZk+r1+fxIORjhodDaeTna3
2vv2H+73jYEK9NCBw77bjKdVsLvDcug1njx6HMZNEwJiY4wVQrTwwCPHPvNLn9h687z1k57U
A2lIJPeTON6qWzmISIguy9RwlaTSJJSySG0wKICCCGgsmowSBdvmX/sPv4q52W3K4fCxtfPX
P/nxT8dGDLnd4fDrX/9aSNW/++yvsnGz0k/GJUaUEL7/2re03P3MZx47cuRolrW2NobF4OLR
48diCszirBw6dODb3/r2t7/9rY99+peffvq3v/+D1/oLg43NzSj68Y9/Ymc4+tFbpw/uP/T4
Lx2XECbjneHoxtwaZRSHitWo2QT1wCEutoqDFjutvE/okqYYIlG0riHYTs2u901MTdQI5BFd
0VpdWl7t9F0AKhPOAokISMfYliSdjVIyyjaQJggYCREsJMvGq0iKnFkwdLMugygTEqIqoiik
iCJEN4tcxJhEBBBU56VEwQD02BYtzluFBymDb0Sp7VzhZr5u6jonw4CM4AxlYAwT07wKqhAg
A2aIBikZtIYdktFkVCiBQySkeZFQRATkeSkQQ0hsJEk5mxGCqoQkAJAU5wlY3idjiZFTiPf4
7/kTZ97Q9U41cn7SnUN+Ppx94ffnL+WTLzxx8qknPvIe8+7sC088deqmqfTSs/jUU0985MzL
T3/1DsKfjlX40heeewVuGp1nX/jyP+XWsy888YUHX/5QGRs/S5U+hNPf478z7l3+AJksACAR
AsYYr1658r3vfX9lZfXf//tfN8Y2zcyAICAgEhARiALQvN61qsj8dFhRtAw7ACsCETURGUIT
Q48tXN+Ynj7XaufZcm949Vq8sbkkAiSNNU1hTatlheOktqIT8TupLgmVjQAykUgc7w73Hb9/
Z1SPpuNDx48ePHYkXL6afDm5sXm9aO3rD3o2SxRiZrTbbtscmobGYwFQ0KQimhASoRhQEz3W
9TKbRTBXvGz52ShLASKE2jahF9JKZhYpLFI8vNgel5ilynp0KjY10EgmbtrqvTmdtlpFh7Px
petFWuYiQ8YoEQpWlqLjfvPXn7hUtMqN7fuO7P/Uo8fHGTNrbnGUmpJQo0AErWMz8sZHJynG
ZoZWIlVlGhmd1hPFiGDXr1/f2djO2+2JLx95+BNL/SMPP/xoDDIdT5QFM8xN6xee+GUBI2BS
ZAl+/ca177z9D/v2H3viN3/LcjYcjk2lj/3yrxSF/cuv/nkK9fFDxwa9+65dnqSpiU2+uVEe
O/Zo0WrFlCtgUSzZmbaLBUt2srM73Nr42te+OppsiiYNCQQccAfzLhNW0zidutTut4tFwnZQ
qr2ABOFI6Ighz1zL1kGDci04DUmAq1G9tds0UQfKeUBUUcRE1CHbs3nd+LwwLetqUMhMYiBv
U4Vl9AzJqM0JEYAVVZIqEKIoaEqkADcrhIuKSooiiuZWK24EktRn61HqqgaDUZNYbi92x1W1
Mxpatg4YkuTETtGBmCQoiCoJkiF2zDkQKwSi+Q+Im+cBQRmJEW+dF0RFYOcUYVbVgIAEO8Ot
y5cvjTXWkARIJUmEJChJmWk2K3eHw9/9H/7nn82D4J/Fh/5k2O2c/eqXX4HHf+/krSJGX/7q
2c/dbhLdPpknf+cZOHXqy189+7k7C3/a/rV/ysLe8tr8VBX6p/EulX7KX5IP4fT3+O+Pe7WI
UZW5QyqmVHu/Mxx+4+WXy7L8jd/4jZXlxRg9qNyMpOjNcudEjApEhERkbdZqtVpty06RI0jS
OL+UEMA3mY+TK1dhPFnat9yon6xv9JQy0QChMRozpswqAIqyQi0S2BhXMLD4qCIWcTzcIaZi
sTue7JrB4KFPfKzdamVIGZIvSz8Z7168IttjUwfvG25n7YUBGofGkTGCKCgJkveNpEQxcgh9
5i6xVSy93ygnNQoSthC7AH1DK71isTAHOsX9iwurmbGxMr5sxTBI0o/SavyxpZVFm+9cvvYP
f/vNb33tGxfeODPeHk5Ho9lojD7kiAeXlx49eWKQuX25e+TEsWOfeXT/Z3+BPnVi475e89DB
wa98avGXP7n4qY+09h8obGdAvZ7ras7a5ug0sXcZFg6ceuvDW99+7YevvHbph1de+s9f/X/+
4i9e+cbf/Oi1721dv1KXY1AffLm5ceNHb75x4dzFza3NWTlKMvuN3/rsp3/hk8Ptzb/5b399
7szpt9964zuvvry+fumxTz30yEPH6tn6tYtvTHevANajzUvTnavn33r1G1//86+99KW//a9/
8qU/+t//y//9f0xHp7/7zb/48n/6wstf/6Ny+CbL9cIOF9rVai8t5dpRvwCy6uyB3K06s4BQ
xOh8k6sUiJkAh+CStKxtF0W/210cLC4tLi8uLPZ7C+12m4hDCDEERjLMBgAl7V8YfPzEgwcX
Bo4oqjQSooIp8vZg4DptNSaIRNF5L5V5gtQ8oy+lqD7ahBwV5422NSrMz/bNq3QCRtUgKQRA
TZAmTZl1isH+lXE529reyozt2axHZiXLFjPXc7aTmY61PWOWrF1x+cDaLnMBaCUaS8ZaVuAk
VtUSEiqLGgFWBFABLNqd5aXVwuYsABHOnbvwpa985Y+/9KWvfOlP/uTLf/LHf/yV//P/+tKX
vvKnf/7nX33xv/zVX/75V//yz7/6M3kIfEC+/Pt4k3d3ZHv3v959xY+5rTHdS88i4hMvvPDs
+6996dkfj/BOn7fbxr0pvIvk3q3hzrz5yq3maA88+Oi9O5TcsZPaO8Kb837hx2o8+9I7mt9q
vvcBrrlt7BeeeOoUALzy3El84oWzty/sfLmeeOLmfe+Z+7zf4LztyZ1GvXnls8/edcB3b9zt
i/hj+bPv+R7ecZtu+9R3qfT+ubx/6+fyuZJ3XJ936/POB79/+vfWbY89fircyyPFBppQs8uC
6sw3L3/nO997843P/soTx+8/Asmb1IjGBIJCiCAqTKyAAiAiqsBsKMsyaxl0Jl5BMyAjaggp
hAFQPt69cWXNktqMJaZqMkMyiYUlOVGWEDPdzbxwWAZmamVZYRLaOpZIwVInt1bBl5P+Iye3
N67762sHHnmw2bi+9vqbcr3EULewE5pZXCvdbEHrxmeWBx2W/c1ot2lKNKBe5z1nEawISJPI
po41FqqWocLliLTcaX/y6GLeuZpfvryispikrYAkgRVVGRgVFMSjN2y6ZfnRTvvoge7aaHdt
Mrrwox9dvXzp0PFjR44dkXLcJC0TyzT2XRt36423Ln7zzOxSF9ygh9Nq/3Z1+cYQO7zU6eAq
p129MhpbACaxyJ4qMvLRh9tp//KNybBpQl393WxiZuiq1Phm8rU//RqqyUwuwuPxjDj/o//t
f0GTLwz2hSQIHjHWdWWNGe2Orl9bA0jEkVkvv2VcxqBRY1hDbBnorWKz+crX//M3mRlQuhkZ
Yphoz6Z6+Oq+nn1osdPN2/3BYe4EbIVeOz/SPtxckB/81Y+60+mydaaTZ6h5DDYJKQhTAIhM
CRQECC2zSwrEnLnMW/JVXfvAENhiUGhEDYAYTCEuuezE/cf37V/6/rWLZzau1hDijMDHpCZ5
ZUEGTIIAYBGMJEhRQqhjbOpa62g9ECiQBkjJgMwD0CGwkhWmRBXLJNOpNjGn1soBaGU3treG
O9stNIsuzwO4qB1QpNBYFEOZUi9gIcxInqDSpKDRmDq3lFFeaVZHC0kZAdQCOeGkWpGBogBX
kNe8YaPoi2xY+vXNYSOiQoioxigbmxvn8tzlpm9YPkzJ5q/A753Rlx946Vl86tQfvPD5J9/j
lnnp2R9ns7wTA7v1x7Mv/MEpeFcvule+DH+o+sXbR7s9oeqlZ/Gpk0/AmZc/Bzc7E+vnTz9x
8rnnvvDS5z5/+r2SDxLVmZtBt/P66bPw5B19S/P+tO+pBX6bcP5+PvXmg6p69oUnTj536qk/
eP6M6gNnX3ji5HNPPfs7P87p+SDXAAA88LmXX3zznemffeHOiw9nX3jiqffM/Q+f//Kd84he
enbeAPDlzz3w0rP41J0HfNfGwUvP4lPP/f4LT8+34Me3n33hiZO3DXunbXqXa+9dKp09/d65
vH/r5/JT8KLqk/P1eX8I9c56vvuz/lHd9tjjp8K9DCnVpKBJUlXVZ8+f+9a3//6BB0585jOf
MUzS+MxyUwe52RNMCG+d7VNQQCQyhqxlJtSU9FZ3GSMJFFjRCdTDUZhN23luGDUJBlDBSGQI
WKEALAkiaySNIkWet1qcrHVRCSXl0M5MZrNZLF1q9q/0r7z92v4H7rMrCyuH9kfCONoNGKmX
xVEFw+0wHrlZaQ7tb63sswYnG3WokyMiESWbFBFIkyJQbpwBLGzmycak7U7vviNHc6ZmNJLh
UACVCSEhAcg8/YUEU0SNEDU1DvKFTod6eRF6o9BsTyeXzpyNZfXIsfs7rs1NmI6m2EQTdbK1
69PoQrnhu+0Th49PHJw+/TYPzBO/9On7jh6a7viNy9f9uHFgCscex1jgY0/94m889tAYdmKY
RR+qiBPB0CStfF02TRXrMtR1LMvYeKnrOGt8WZ2dlfWsbADY+4AK2pdP3G8zl+e56XTzVjtr
t1utNhetYFistZYsASmoYSIGdsgWySJwBPIZ5C51WEhMiNksmjJDHkS3HqeX2qNW016AjJVY
1c7rX4Am1UgYESKQElQ+VCrCJjYh+hBxfpZBDaEzjoOKQEKMAjbPjTEQ0/7l1cFsCBtXNSkm
TSHMQpO8uqQF07wnYJSYGp+aOjZeYlJRUJR59jmSA7BBokpIKQIKYWUosk5Ixphct7WwshQZ
Lq9d29oZ9fJs0O5ar6RCCBGEUawgCxkEYgQEjdEpCcEURNt51irQB6rrLIpjSqrzMlCIiMRk
LTmbfNkMpxTrLHOJ02Bx8MhqNyExGEISQ8lwXhRM7Ix1TPVs9rN5CnwgHv+9px+A2/vNvvs1
92enfmx63Eq2Apj7fN78wu+feuWZF/X2N9r7R3vpz079WAxPfv75x0899+Wvnv3c0wAAcOrP
XvriF2+GiOZv5tslP9n40c1mwe+uBX4H4Xy2c+/WLb3fywe55oPw7nvfNfe7eV1uX8/5cr5y
xwHnu3X2hSfw5gXv344Hnv69x5975ZX3DXvbNv0TrJVBcosjAAAgAElEQVS7fZEef/7zT8Kt
muen3vcNu4ued5vyP0+3Pfb4Z3EvQyoEIWZF2Nrc/Nv/+vXVxcX/+OR/GPS65WRsQUWRCRFg
fjgceZ7BqwgIhMaYvHDOGlaNCEhEgEaIcZ5BAiQwHY5i2fRW+8Y4ALWiQTQSBksukYsYAiRk
zFyNsXOgX97Xu9HjKTBFEV9XIJZNMmWr3jp4oBcS755/awVgMhztL9q2yMiyWkixMaUWdQOX
13bXbpj7DnQP7e8R1aIaI0YhZC/ACFEEEwy6C51pM5yWJYmSU3Y1GOda3O3Xo8k0eQLIDSvY
lDyBEqmSAqEABIwevXGQ58Wy5kuOl5tqazyWWXn1zDnsLbaa4He2OEYLkvlwf9EdIV5T6XBX
0JYxy5KbzDT17eL+lXBou8oqjEzatAuVQW07lVuY5WZMMLOgAOzRJEBAh+oQCJQgKSInAVFE
4tL76KPxoo2AQEoSYwwxVbNKJFkmY8kawyyU1WRkXvhbQUQ1SRBQlSAaJc0PwjEGD7GSBA3U
EUqRpNAS3wbfOIWC2EUgAVYlRRUFQiAKoCXoLMUYpDEaOEckH2NIosbOu64wmIwt+4QqAmLz
dqvTmokMb9xou4MHlvcvXLoMk90uZUYl+lkMyqoCHGNoyir4WsuKUiIBQkTAAFAZZUUnaiJQ
VKtqiErGKenEpIqUWvnS0kKWZVVdXTt/Nczq44P+oN3BJsWmKtAQQSBR5k7CLDEyiSUPaBAz
AcOEjqSXtQiw9LmXAo1FDKBCKKoCqIbREIFCM031LjOiY850cGBwYGWRrMtdzsYk5sSQOZNi
0pgya8T/TJPN/yW839/zDq+cOgUAr/wj7UDmI7zy3El87h3hm2fgc+9udvLMi/rF90s+gEdq
bszcLnlP5O6mGi888dR7nGd3Ef5ceOD9cz955yvvGJ28EzdPJT7+/Bl9+cyPTyje7fa7bdNP
vGfe66fPwrs9gnfU8+ei2x57vId75UgBMyBtrG9+69vfVpVf+7XPLg0WqnJqDROiSASCeY2p
+emn+REk1cQIuePcGlKZFxlkImYmQlLQJOqjhuhnFYXkkJgNkXHCrJgIgiFCNo3gtDZAptOK
mYkk21q/Fod/Ezb/Lg2/5Ud/nybfxtkPdXYxld/dvbrRRdctysluv9cdj8ezsvTRhxQSJHTA
GZATkGpy7fL697+3e+GCTqaZTy6pS2qSZkAUBULTy/IuOytIiVLSOmglGF0B7R72F2iwFLLC
k1F2glaAI1ACTooJJEiqQzNtZlNfAWhu7MHVfUcPHFrq9BeK9saVq+sXL2MT29Za0B6bI1wc
9AbWJ5uXty5d3Z4EFmzf2BytbQ0bAm45w+iQOybLFNgnDCGlOkETpA6hklBjaFACaC1apziR
uIswBh0S7jBvI9zIs+0832p311u9y93FG+2Ftc5go9VbN8U1ztaUrxFvIG0IbIlOopRRypBm
Pk1CGouWIBWJpxitiBVxqYE4TTIRLJEDEWRgCylsaBlfuFhkkpkERpQEEUCJIlKNWhJMQEaa
dlKcIoSMQ4bJsViOKk2KdQxeJCIF5mQs5blttSLSqKy2d0cbG5vTnTGWvohka5FpjUEMEzIC
iqaQfC11QzEZAatoFVkAFUAhqviUfEpe1AN4hAp1gtIUnB1aXrj/kLTsubUrP3r7XCibw8vL
R/pLRUCuQ5utYUZENtZYa6wxSBaZkRQUHHnWwJoNOiYzPC1bZdMFaltrATOgHA2zEctiOanE
upbpLEfMDAJp0e3su+++wyeO33fk8MGDB1ZX9y2uLPcWF9rtotvOrQGUyCQ/q+fAv5QHHnz0
7n985sUXnwE49Qf3TFmZj/D482f0Nr44d1G8rKqqLz4DAKf+7KU7ST4or58+CzffuI9/5P0m
yEvPzlv9viscdEfhz40POvf5es7ney/mrsQX9Z3ZPf6Rk3e//e7b9BPmvTbcXfT8uei2xx7v
4V6GlE8wHM9e+da3r9+48au/+u/uP35/DIEQUCGmKAqgBEDIBokUSIEQiYittdYyEhAB07yQ
DiLArSa4QIiECEkI0BDHkNTlSQWZ0LAgIBMFhUnjgmatVsps3BzrpZ1rPzh//vTVN89cO31p
6+1rw3Obkytbk9NvXLh48fo3v/eD9fGwc2il/8Ahd2ipZN0e7lTbQ1c2RUpKWuZCXdsx6Jo6
jkbT9Y3Z1lYYT+J0Bk3tJLoYjK+z6POQTBNMUgae+VAJVMhjxdjq9g8f6x85hp1+YNcoV0JB
OZKpo1ZeZ1FmUXaqelhXw3K2ubu7vr3tU8yyzNdNxmyRQBRTItEWmYHQAcoHYq+dv3L+wrWY
TLcYbG9NL6ytX69mpcFIhB65gsLbnhR5Q4XYHI0BcZwyDDaVNnmbIBPMFJ2o1WTUG6k5zihN
rU4NTL1MGqormlVclqZqXAi5NCY1LDONniAaTtYmy4lRCERVBQgYhVUscwe0jVqoGiSjSEII
xBazHNpt6PWoD7WSoKQkqgqsxIkoEHmmqequyJhwxDRxWVXkQ40bTVlnTP0W9tt20HXLfbPU
D51sB+IOpoqgklhJ9KTD6eTMW6dPv/Z62Bm1E7QVspBaSBkjaFQVBmUVVnXGMM49apgAVIRS
0hi9yoRkx8F2QZs5TbtZ++Dq/mOHFwYLO9Pd7106e2Fn11tcPrSSt/Lt3Z3Nnc1pU5YaxlLv
al3XdT2a7E6no6aczKZ+NOYmiqbSqfSyrHBZ7dsz349QIFkAB5QDGyUiBmOUGRSgDtxIBhbR
qsmz/lIldOX69tr14ZWrG5curb195sJbb519843Tb7997ty5S2fOXrhw4erP6DHwL+fJ33nH
VrqZUnzrHfz4R04++cUXn4FXnvv9e5lST/7OMwCvfPmrt6WTP/vS7VnqJz/yOMDjHzn5fskH
SjZ/8vPPPz4PB908v/f0A+/6oFvhu5sRplvcUfhz4w5zv5sNe/t6zvO77s7cIDv7wh+cgltZ
+He7/S7b9C7uaVbflVee+8I7WrwnQe3uet7+WR9Etz32+Clwr9Be6eXV7/3w7bMXH3vsEx/7
xCdTiqpCgEkVjVORpKCISJAAFZAQjDFElGWZMRYBEYGQRBUVbnWKhfmRKUlJAYjIZq6JiY0t
lheHN6aMiOBlfvW4tLt13smnC128PllKemCSdp3M8qIh1Ax83Ux9OSw3Tny8Wyx2/vr1V3eP
HH1wsHrwFx+hi1ea89fo+jCvE2lkCwETGNNlUiUvqgrqfWh8RKPGkiQypKgdSUsGMl+XSoC0
tb194doad1rWZoRmwrZY7LetCaNWs7M1GY1iWULwolojNi7bjhoSdjo967LRdOxns6JVZICW
2brM5ABVEwUyNgjAoge6Cw/YbH17c1bXBhDKsLa+NupQ95ETee5MZhBSj23b9SB5M2zZ7V4r
ahTOIGVIKWHITe2QCZFEMIBGIBFIAhFIgkZCD2ANgQKwoSZI0sBIgslyRmpy6lg2qgFFDQAp
giiIaiIh54EBswZiFJAUAbGJKUZJKVGUFCnObD2pd254wMwD1gSEKqIxaQBtFHd8HKJ62yqd
88YAUiWeCMGa5EiRFJghQ8pCLUEjNMK+cSEE5ElT7oyH46qyzMtF0SlardwOy9KngAAUY24z
R2QQiDCFpJrmjVWiSlJRxaASLFWEjWXJXd7rLiwvMdFkOrlx+fL1nSEYuH9lcGR5xVSx2trN
6liAYSRUBOYEmosSYjCa2HAQJ2CSRCA36Nh2QT64SdOPmLOZdyY0RAQURYCIcqvAJnoL7BQM
uUYh2aIxeTX13dUVX/kMaHFxwWicNjOTIqla4swaS/f67/kh48kvnnn+9ZO3wio3/TfvnIub
Z6w894WXPvfFu4+gLwI+dSswcytkNx93Hst6/Pkzn3vgAXiv5O6JQrdzM6EbT80Hf69/6ebr
+bbQ0OPPn/lDuIPw5ac/0Ir8NHjyfasB8MDvPAOnTj2F7618+uQXX3zm1FPPncTn4Jnnn38c
7mxLza+6GSt85vnn4bnn5ln4t23ou26/yza9e9DbVPrDDzq3x5//yJ8hPjWf2PvGvLuet0//
A+i2xx4/eVDv3s/rf/1PX/lvX//6xz/26K//+mcZJfkaYkKVeZFoVAQEBUgiKoKAxMgIzJxZ
aywDgCpYUVWdMSJinoBDSChOZF/lJ9/+wdar3zt54oR2Onm/j1JffvWbLvpckRJ5wJFFc2Sl
c+zAtCl33jofZ3K2bP5qd2Njqec7rUCqsc6buKpZMegff+whrTfj9tZHDx396KFj93X6tD6U
01dbV3Z4ayKS6gyBNE/KSEqkihCTJsF5mVCm5Ejz3J588LuT+m/PXlxDN85agW3H2BO93gGJ
/bq6r92+b2Gx78SAD1U5Ge4M19enw93prBx77/atbluUxUG+PJBmGnzl8twypVnVCrIK3G9S
Nq24qkyMlqkG3DYmf/TRM47+9O/+v3bROnno4NWdy7EFRx84XF1bz9d29k2aw1is2DaQlosY
9ztc5pRKmc4KcISMvUQLYh1nlq0z1hI7QqtkmQwKiaBIqpAEmQQpiMQo4/FsZ3MkHtQDJSI0
oMQKlJREVSQqickqk10dVRtNGvo4ixA9hCY2te8WneVOz0XV0SxuTW2JHWjnKae6gXKqSSSq
l1TFVAEMY9z0jW+3U6cdDEdHWcf1Ox1rKImgZVFQAEzghFuYwbSm9Y2Was14dbh9fWcrpti2
WTcrulmGKNvDrQRCzhjErnM96zJRaDykpKIJIBElhAiQUDxibTB1i3xlqbW4EBFGw+HuxtZo
c1sbWWq3ju8/sNjp+kkZdqcckhE0QESIhpUpqSBoAgkWmJliyhMgI/UKs9CS6HF31p7GdgIA
CIQKaACNmgagLuyYpPGxraYlBBLBupINHzu01cvj6tJv/+7/6EelTutevx0LoxY5pXpWzptX
WoTf/t3/6af+CMB7PQT2+DfBh7xE+IesAv4ee/wTuddP3u+/9qOPP/aZ//j0k4OFzmR3R2NO
CgSKoKSEAIoASEmSiKgkAEBUhlt1CUWAAFEpJcZ5fA8Q0RiDIKn2rsgTQhWanlkYjXZXDy8V
y4N07XpOrmYUMk4EtmfNwkQPdPjIgrs0eiT1LhR+J4RR8qiUJxigXUUz2djc+mH8lV/4iDXt
nYtrp3fK+sSx/fuXO4sPp/t26jPX4vpQvMfgyUdDiGwAiRANkQHkqD6F6KEOZdhZb3PO1dQH
aFpJO73dsr44mTSIh4xtk3OunjG4Alqdvht0snYxXVvfvXZ9rWrQh9QblGyb6TTsrh9aGfT2
r0CSoFqPy0njcyVnjBAlRGRM1rS73d5S/5c//rDZNzj7xhvVjXVpSt/O11MqFpdJZZyvbexM
uOblYtFNzfR8MFsOfJyu7dR1yYoCNVAgAiS9GWi1BBYpIzAIjOg4qReICcGnEFSAeTycXDo/
RJGFIrfoUIkSmqSUBFUEKZrMre4bFe6769PTk3ITcJJAlBg55+z4/n2wuJyXHkcTmrZ6jNlC
3+UsbsqtHEQAgKPE2s9qX5XTiQ+zEFFB2bFhY3NVrmuvmkAIDZNlYAJgL1przU0pKY017pbj
CMJMJGpSIu+NwV6WoTWunSNgpkohJO8hRkJUxqjagETEZDkUBvJssLzUWVkpVW5sbV25traz
uQtJWxaOHFk9vu8gT+qN9Q3wgaI6IjUcVQmBrCJpEgBVFC0iZ8DKnDIwC22XGWw8jWs7ix0l
C+pREiMgaiIEyiwHw1VdxxAHLnNRI4qXhjtZb9/gtbXLIz9+6/yZ/Z1BC2G4szPlkPXahXWN
byRGx0Gs/Rk9Bvb4t8fNjjNz4+nMm++v57DHHnv8hLiXIbU1Gn/2wQf3HTww3tkyhslklBLO
yxsogoIoKMzrNwsgiogPzXhW+rqezSZ1XWuSnNlZUxMggk1CKQZJUpfHbfFgv9/qd4ebG8v7
Dje+rAH6+w8O1zYVMGmCBIVyOakmN7aypXzx4GqzWQ1m+MTqfWW1/Xo5rSV1EPpeOyL7u72s
Jnv62qP3Hx8vyJmrl68ZrDNo9bqLxxaXVhY766PJ5bV4dQOlTDHG6A2RQUZVQMPGZKiWVUJs
prPuYqcDAJOp9+p9tMZNVdZ9ACKbYquVx0BNM3PE/SLHdpuWltK0nJWz4Wxm+92gSi7PVlfc
yqBxlqP09+/ndomjmSZoxiVWVds4k2XZ8oAXB5OqCpubv/jRRw71u3/94osb66MZVHFp4aH7
7m8VptNFdNOtK7WW464btBSKkCyClTqHYBBAC9RFiSmkmDQhARAIQwSJmpSRDYnP1YOqpNgo
CGWOZqhnt4usc+i+YwXlqW6sepKgKYFKIqychWqg+VKBlrkiY1CENKACoCM7YLOQYJqaqY1C
mUUThcpkozeGFYhYRecenRSsOuuThGkpTQCA2YZKSqrBOgJGsJi38oVe32adyaTcvb6RTSel
xGFVTZqKmB1xx3E3Mx3rMmfbGc/TsVAVY0p1nVIiolqSGmoYxLBtt/N+26z08m43hLh24/ra
+vrG9mhainW4uNQ9cnC/NXR27Uo5GqcQUckxGSWKggrEBAbVUBTpeVhJxrgC+x3u5qZfoCOd
VXY0a5UxV57XNHckShhE2HJSk8hMp01h8rxtrPdswBsSx8VKHwtzee2SDPprVy9Jd7xUtPM8
jxr9xM9E5v4wCdDUH6Y6Unv86+I9B/zeW89hjz32+IlxL0Mq67Ypc2wAIV0+f2Y22o51pRKD
jwIQk1aVTxFURWJkIlEdj4a7w50YfFWWPnhD5FzLFbmx6EgJhBgJCWZVvbTv5Cceaw2643Ob
TWgy1xqt7RxcOVguX58Mt21QFxrDTlDD9gTPbaUH95sTRyrZWJ3FX+8utHZmp5uyQUDOG6Q8
wKIUfji7cuHC0eNHH+qe+NH1y1de3RocObKycmCQtQ8dX+kdXW5d3nTn1qcb6+JnqW5ijAVn
dRAABiYQYTYSUhvtkU5vbWNIVA93g2Z5RCxT3AwhzXYllPv7C60sc22ufVAQzIregdVWCuvb
W9PhjhG1Igv3rxYHVslHqBq0LliHS/0aqFkfxrq2noq8o91lLNpptNu8edYlXF5eXnj0o7s7
G5Omytcn54dvLMTmo4t9Xiom4xtpVjemKgR6ntS6oEWOLWU1Jlc0kBgCQsJ5kIaEjIoBcDbT
pHVKAuIAMuA6RRU3M6JZ3hAoo5nXTlAWkISqrMmotwldamz0RjymIDfb7moMIaXJbLSdsxMv
6FkbauqikrbBWsMYQVQsGUYbLEPuXExF9N43PtUhVjFw4zGmCATBIBEQJIUZmVEi5pRM1Uiu
M0zEuq/b6WZ529k2W6MAKYqfcYyUxHoPIspMhmtDE4m1Q9ttZf1uf3mh1etG0FkzW1u7vH19
a3tjPKvBtej4kaWl5UEry8KomqxtplnVQkPsVJCFWJhFCZGQE0EgsAaWRFcN41JbVju0UGAI
uD1sj2OvgUyNAkYkQswEbO19ZneZJtaWNeXQXYmmUJ+4rrI4zdqlK/btX74y3J7sDPd3e/uc
I1/fmI77iwutXltiIlVGZDSIKOln8xDY498m/3o6+LxTeWyPPf41ci9D6pGHHiwno/HuyBm6
eP782R/9UKNHUBGxrmDjUkoGyTAToDXGZqaw2F5ZbLWKzFkiJGZEi8RMiUmIlK1lZONlv8kS
04ET90+urV29euHkxz61c/Fi3WsvPHJy7ZWhVQKLNaVC0UbduH6jbsnq4cOdY/uG52/0yvpj
7QVN8ZqGytGmpt0w2dopcwfZZLg4nRw/ebx78sH1jfVvnTvn1m+cOHr4+kJv0Zn7jvVWDgyy
8X3+2lq4dkO3xvUoODGGXVIJKskYkKadJr9wdHmhY65M64vDycjXZeUhhBwIK7k6Go7b3cWl
5cGg73IjGhNhJREylxWtpom7N7bLa+tnL51bXOqf2H/o8Oo+2yMu2lNpElPnwFJdVte3ZhNI
WO6s9oxXv762NQZZbX/iY49+/NUL53504fRkOuHCUIgbidKs0uHsvqxXE0GUIEREQk5JiZGt
JTIpJlSNCAAwbzYHwCklbSISMZGCclJAQFQFJSaylJIECcpGURvWRCqihKAE8ysZgEXQJ8QA
iIpKwMmHzY2tajgqJHVUC+8z5UkTIWET00wVCAOpQqhjrHwsU/SICVBSSpJi8h5EIJKoSWyT
ZgA5YhEpR2wxt7tF1nZIgKIGiUQwpdj4m7drEhFFEIsRqAEVC9zptLrtxUHPdtuc2Ub89dH2
9vbOcLgt3kOAdr84dnRh3/Jyxm42mk43d2BSdqIg58hklAwSAkfEZAkMIagT7Qj0wdIimUHR
7XYcI0xq2J3ZWWhFMEqAGAgVkRCDwg6a0F8sjhza2t6q17f71uQSSJKiQyi8ouu2IbMXzp1u
mtoR91rtfr9/5dq1rc3NBQ1FniNzkCQJsixn3vNI7bHHHnt82LnnsaBy0sKVMBk5Qw8eP3Jk
uW8RYoopiXO5dZmmyBpwfsLupmuAmNkYA6opehUQpJREUwPilUDZJgGpU/KwNS0fWFruHzq8
feXK+MaV/upg59q1fSfu3/foo9e//5oV5ZAAkyVd8jI8f2PaxO7RQ8VDh5oLN45uY7994Ad+
/A9pdF0ri9ShQsSaAMXW5Fq8eP/DJ489+mk3Gf7grTe+9g/fOXpw8aGjB69k467tH97XO3jg
wf7Jg3RlW64OZXsWa1ERw4qUWhKy3c1+0dq/3G0OrOxG3Sr99nS2tTMe7o7+f/beJday87rz
W2t9j/0659xn3VvvYr34kChRsmXJppxuIHYnTTkDA0Y6DWSQBAikUWBOetbDnvVEGooBknaC
IIGRAAbSMY0g6U4HLTmx3bJlyRTJKhbJuvW47/Pee3+PtVYGp0hRLaoi2y1aBu5vdPGd73z7
27hn7/3f6xn6mLPGdr43n+w/csP1ga2KllNSSIiYwSZwXTYhHB/NHr/1/sP6nd1zWxeuXrzy
3DNbF89BadXp2qefK7u0/4O3isV4MHOWSELcu3PvKOZJWeaDadHJqCkvrG3QdPHw4Wk3W6yx
ObdpvBjN3GKuqARDwpFVMQBI1g969BKRgqIiERFBThlEySGSgiICGiAk9Gi8sSGHnFbeQEyE
SZFYEQAAjZJl8mJ8RhMzAhBRBkYyJBRTWnQxCgfJa6glunEbegPIIGgUMUDsclqGPnDuU1r2
XR+jgjiySjHj0oAUiGumGBg7csVGWY6KokRjRAwazEm6xCKZJTHHnMGSIGSVqMyoyUAclFL6
ajhcW98oBgPwtothfz6ZHk7ny+l0OumXeVDShYsbly5eGdUDs0x60ur+xC/CRrZQbHQ2R05E
pIhqSIkEEQkcYqlYKwzJVtaG3TpvFkUv/nRhT+dVVAekoILECA7JAAYLx6By4cozX/hiSNkc
HO+SjmQBhFwNeqpnIkniZlO3Xfvg0SMWLZu6rKpmMLh29erjw8eT8SQ1g9FghEDCTCTuLEbq
jDPOOOPnnqcJqV958XbhbTzd7/q24FCWVlJk5iiZkkKOnEIf2pSjtVZVc4yIaIxR1Zwz52yM
AaLMWVIkYCWKQFkp9bLpqsHW7hJp59at5eHBo3tvP/crvzxqhu3eQX396kZMk7fuUMtQEHMa
plwkd7h3+Dh3a9eunLt5IZhjf7z4DAyly99tx4+0PbCxKtZGrihdrcGMH83P7cAXP/urO+ee
eeveG3/x/T/eu/uHFy9evnbz+VgWx9xdXPcXt58pb14sphGOFv3RJEwn0rdFTzDp8qRNIGBo
q2zWvb+5VuettR6vdQyzxXJ8POmX7SL2y9DOF+OQYy8aFRdtaLvkTLHhCwY/UzZtbo9meyHu
Hx1tX929+dytqxcvRm8zxTwYjMYzPV1qXS3m/Xfv3Nv/zl9M0HRElYEakkuL+fHJctHVZb2+
senKwbxLNqRNX1FRFHXlEZEjrTpGgzICIBpDoJpSRlBPRISZWYRWVb+AwAAaMoWKJYqiIhlI
ERVRUYVWtcWQjKJhcIoOkFQI2CApK3A2iJbIKK8qsPaSjlPollpZLIEsoxAmzS3HkFMWFpFS
uFYorG2KSrxj4xvn1opq3ZYVoGelzJCz5JA5pZQhJc0iqmpIiNhyRAmo0RDUVb02HIwGsrXG
ZZEFjheL8fHjZddOptOuXYa+B4HN9fL5Fy7cuLDlSh9FF8uZjBf2aLmxpC1q0JtxDG1KKhkF
xaAComAlWiQZII2quhyUNCyh8b6ycbrU04WPWis5UgBkRF3V91AjAOPCmeuXn/+lL3WLsP9v
vndh0q9HI4rL2s4Gbqw47WU4Gviqvnv/vfFkgtatbayXZUlIg+HgormQHuxNxqcgZjAYqqrJ
4vwndBc444wzzjjjr8zThFR/vDftlieIqe849RaBc5LMKWVjrSEbc0o5iLAxhog4MwAURQFE
nDOLEBkiFGXNGVWAKJMNYHOAalTZtVEHdvP87vnnbt7/zp88eOMHz7z0+fHxSXv/we7NZz3Q
g+9/d8RaKmbVwHmUvHswC4t7fONaeXsnD/21o2790Jy39k948oMwczxpBsMy0dAVA3U+e2zd
bD9c3X7u5t+79fi9tyezce1GvhiZQk6lj9DXA2i8G61vNTfOlW3XHR3Gx/N0HHLX5r6lHN1i
psLCwEi+qAbNcMf6YvccqnaaA8g0h+Pl7Gg2n3b9xJvWR1Cj1sC8lz4gQEVkUVF4vFj8xcPD
/uaNq5cu5y60e4fmsLUNsth3lv2dENpiUG5sXxxtjIg+tbt7cVC9h/fu6uN6WFeuXLZhFua1
cFuUQyRBjTEaZc3s1IqqqgIgsyAiqmgWJSIABEUVWLn8FCyAIhhAb2wrrJxVRFULASeISgii
ooZAOIJEkGg0O2JWsSoqSgBWgDRbZqOZQFS4zzG2uRceIFlnjcEB6dCjd74kUxs3dMXAOk9G
CdSwBfKKGHNu+9j3IsygApohs0gywBYSYARJFo+7/X8AACAASURBVHPpoaqqjfXNzXWoygTS
Mx+0i8eP7p9OZ10Xu5gSM6dcGru5tn5uY3D+3Ob5jabkPvTtZLmcjOc0i5sJAcuub8k4Km0C
DaJlVdSCJuZKtXbeFYYKUw7ramOknrrQ4uPJcNEbBSMgIkzWIOXI6jBb14vOVdNo+6XPvTw9
Ob77//zh+mS50wmxhrWNbuhOjCwwq4Fy2LQhPny8n1mGm4ONzc2iLAEBVEbD4eWLF5h5Op0W
RVWWRcrZpr8Ji9STbK8P+Gslpf91su5f//E2IH/t9P1PoArAz3mhgTPOOOPfPU8TUnff+K53
tnROmGPoDKIxpKyZs7XWGpeZE4iKaMqIBACqmhSNdcwMCmSEszIzKaMIGVCHSmQKS4VvRQ76
OVi7e/tmNx0fvrOnb7915frN8d7+8dtvb1+/qk5P/vT7HkhLxySY+JxQSP1Jdwee2V3b3TVF
VZMtJm47Dp7TxYFrxaKB7C3P2tneweHm8y9mO3j44PGLz13/+7/+W2Vpovbz6WE32S8RiWOK
kV219DJPWZvs1rfKG1ft0uN8ng4P24NDms7csrd9LEI0bbCnc0OUidiS8bbxri79xc0N2TkX
FKJqYmm7sAjx3nixP5mHEPoUF32nC6bSO14slm+9+Rd3QbXI6Kgau3z/0aP7BMNf+dIzN24+
/9yLm7YaLNrPbZ/Do8Pjvft5Nm1zP1ZyXegxq8FF4bd4HRQ5i0UEEQEGWEU1AYAqANIHHaRR
iRARSBUBCJRUObMx4GnVDU5UxQAQKwCBqoBEBEBhTSrRSlfkthI0xhAoMoOIUSgIrKrJubI4
KExhKgtQG9gsnffOOmMQnTEOwDJTZMficgLucxZWMMZYMgoQBdD7ABA4LTlGoOzMwqFUrho0
5aBpBjXVNTu75HzQ96enB8enp5PpbNn1MUVGK0CCpqgG67ujUVWsD6rKIOc0PZkAqatsU1S5
ZImkoh0Z6x1nmeXlPLWV9yM0xAlyqgpfDb0dln5U2cKnGPujBSz7QeA6KyAIIAEZBlRwYLOr
jmLHW2tf+PVfd2n0g299b3L8zhYsC+46Q7q2MRuOTj0u+5nlflT6uvR3Hj8+OD5Roo2NrZ3d
XVEVzqvW3utrQ6Br9/cOprOpdVvGmGXbfiI3gY/wQUGfJ+Uq737jy7dfvf1l+Bsr8HNWW+iM
M874uedpQmrZ90CNosQQYxJDBEkIkRmJxZgsIgwKKmSM9946Z6wty8r5AlZPSu/QWDLGqDgE
IlRrE1gyftQ0kmVv7/G5Z673IV3+7C8smI4ePK4ybl++PJ1NDt95Z+fq1eIL/uHbb/N0USoS
UDCgIsNxzN2DdLqkKxfd7a3BaXn9weTcUh8O7RG3R9PFeN5N7XxcN9ck3Hzp05tH54YG5ien
1fpoczTY2XQBII8f5+msH4/HbQ8Gi2HjBmWncArJ1sNmvakuXx8sL9pFD+OZHJ7y0VgnS+0T
5xyRoxGQTDFAQJ2AEgKRd67yxYa3UDc3d8/1oiwacm5DZME2pGUfFzF2zCHn3mAcrRcXdhoH
Vxq3eevWhavPbDYboyDweP/uW39++vYbk+M9n1voApGzIsYwKjEnVfHOA1hlBnzS52fVhgc+
KLGIHxTuUiQEVVAChZXwzWytMYSogAqr+CdSEQFBZQQ2wFaMhRJ5neA8SqkCiI7AWvLGFdaW
ZAqExtqSqHLkyRhQEnaaAUBFNDKnDnKCLCiiABkQVVVJifocGTEZSh6iwQ64FVFfFoOmaio3
rHLt0boo+WixmJ4eTZfdbLmczuaLtk8pA6gzUJdFOVh3dUOuBLIi2oW2PTwiTpWjc4PCOpPm
8y7FnKRAyw5nwmRLL04XCWNqmNZJ1VndrOuttaouPSJ2oX90DF0egkW1FhFsRkWrSEpIJOQi
2RPOlz/30t/5h//x+sbWt/77f57u3NmuhBwuHOm54bIsFygRsAKsgTaK+nSx2Hv4sE+pWhts
7+ycP3/RWksqoqwhqcHRsLl4affBg8cHh/tlWQ1Hw5/19f+jvP61r7wGL3/9dz6ULn978r7O
OOOMM/6meJqQ2r5yva4bYx0oWOsREQCttcKCKkiGgC0pEXjvy6rx3pM1zpXWucwCiMYYQVJA
p+qAEZURGQ0DNs7PHu49PtxfXNzZGG72k/bGZz/3KNLh/fuocu6ZK9Pxyf79+9vXrlz94i8f
vfE2vffQYZ5DEiOVt75P8eHRbL7g3a3mwvl6dNEceHNyuM3ViPCRt36w5tZHzplL57dMO/HT
yYBjNx5HU48aO3TQrG2DM3OAw/nDB++83y8WZeE3N9fKc+uwNZLBaI4mAZiBa7Z2ihu7ro0+
sMyX3XgSJvM473Lbu6xFBgyRohhk7DnrMqmgAUWx1njvC3JD48iWWAx43fZI0VpqKlhbl91L
xc45uzk87tqQZWOwqbOuf/Do/nf+KBy859Ns3ad2YNWYxniXBCUUgCDSt30patSAMKBmFQSE
j2ipJz0NiRBRVNCSUQIRA6QI1tmIZJCMIUtIgKqaCZmQERNiMpINOMwlyTPrw61qkMGCNURs
UI0CMVNKmCKmRJwoMmYGZlFY+RhJwQCCsKgKYkYSwgCaRCKFZDWhBkNcFdCUWhdSuKoqXVVl
5UUfjmM4PpqdjMfT6bQLMWUJXZQkksEAjAbV2tpgfatGZ3vBTnDZ9fMuZmbg7IhLg+RtJDNL
WfpgyYyKRkVzjhF5uRiPwA3BXq1GTVXRRj3cXPN1gSiy7NJkYds0jOjZIpCqslVY/ZQB2Zil
pVMDy8p+9kt/5wtffPlk7/Sff/N/6t67s14CqG3ZyfratDFz7o32ZWCrUNTDFuze4/3TySSz
rg2H125cH22sq0riZBBBRVnIw2g0uHr18r133z0+PiT6ZLP2Xv+91+CDHnQf9/HX8Cuvvfzy
y9/+9pMyjz/iBPyI7ejD8a9+/es/vsKPTf8Id7/x5duvwv+vFeqjB/6oE+0njP/E/Xzcki9/
9avw2mtPfHOfwCmfccYZf+t5mpC6cO3Zqq6LovJF4b1nlidCSgRAidCoeEhPipYjKQCLkjFE
BnNWBEIDgAIgKlmAWBhUCCIHZ8CVPsb+6NHja5++0GMsDVz5/Iv7JR7cvRfm0yvPPTtRfrj3
8Nz5qxd/6Uvtxtunb7+pQTzZLi49YS1UnHTd9NHidJGv7jRXNm4Mq248r2aLc67SS9fcjWdr
xg0BrMrFo6mZnVatOGyWaX4ik9GG39kZre/srtVrO4Ot/mSyODgYv7+/uHvfDFy5uWE2NnFj
o9zY7KpyYTAP1I2MP79uYb1JZr1TWHZ5ssiThc67NFlAyCYLxUw5iYhoUOY8XzIDoVEqmHx0
XkfDwfbFjWeuVefP52ok1mrKbt5OD45h+Q6Pp3JyfGExtrUjqfLgwo3dnaBqlYBlLgkVasHS
IhliD9yxzWpVAABwVURKAXDVc2XV2ZBVFZVFOGZkRURDTkEQlQiQSAlZVFAzSEZlBEVEBQ0h
TyYVQ40FS+bUc+xAmDlLzpAScSYBI4IspIgEYF1GypkBgAkFMaEG4ICaDZrCY1mJE6qgLovC
O6ks1hWXDrwz1keR6en44fjwzfunB5O+D21kYAARMIaG9WBnNLqwtrZdDQpPVGsgOZouFyez
rkspiagiAiEaMgaNAEldFk1RqqEEYdFySg7RWhraYtcPNrfKYnPQDYxjxvHCzpe2S5bBggFV
ArJKBKaH1COiJVZoCeeFCU1RbW8vUvsH//P/MvnB3iDToITglUxhqrVxXI5ni7KEgQHMHdoy
l/a9yeTewX6bkm/81atXbt+4YVFBWUXQEgBk5hyiGq3r8pmrVx/Qw/l0+gncAj7k7tvfB4AX
n33qs/7b8A/u6LduAcDrX/thLNDrX8OvvPqffeM3vvXbt1bjK9Fw9xtfvv3hNz/aBuT1r+FX
/qoew5U0WR347je+fPsrCD8UPR87/vH7+dE1P5zz+tfwKz9vp3zGGWf8XPM0ITVa37LGrNoP
xxBFWEBDQgRAQjLkCQ0iKSAgqIqIKqgIQwJYPdeZUBGUQVXRKBklAlRnO+5GjSsJjt9+z1z7
bNEMu7TMJu38woto6fDNO/mNty7euDnc3p1Oxl3sBy/eWr927vgHby3e3xuIayI7FUUq0MLx
YjnveXsyPbdWXBpduri+Po1h3ua3H9Q03Lp8tRwN962a6Xg9ap4fZm59gyj0aH8fogz9cLS2
uX37cn7h06fzo8Xjo3Qwmbd9CBEnp/nhomqKwdpgfXMoJUaTubBtUYAlHVq9sI66boUKIZ0F
nva2ZV3GNO/m3SK3vWlDlbRQNEJBQat6dPX6+gvP6cZGRDLLrjs6PNm7P3u4F44PYDGzkg1K
YY0SJlFkHjI3oGpQrKkIGZQQYTaNIWaIYNQCOragoAqioiqwipRCVQUiYwAyg4oyqBKDNQDC
hGop5NxrjqiGACCrZBRBUVpVSogxznpEBCQEMCn7kEU1oUYAAWFVAIvGKkESVcJsMRBLQWBR
nVFnMikTQOFMacUaIEISb9E5o6qCgikIh27OyyQdcx+jYSnZ2z66jFVl3bDywxH6alAONn01
zNmHvo4JExDoDtjhxrnTIh4v20XXIWnpaTQoB5WzHri0pqwwss77AqEBs+aKnc21zXLQFKVV
1K4t92faRexTwerZGlm1N0JBVauJODrqUGfMSRHQGKa1pcD84PTNPcewTUaNxnLEg3oOeRxP
Ww2Fx4qcZGXjofTTbvb+wd5hmCdPly5ceOH553c3N50qczJEIsIgigCqRtEqrTUNnN89ODj8
RG4CH8tHbS8fsaX80GC1KqJ49xtfxifTvv3GHYBbHzVr3fqNf/Dyq99effoj5q5VC+Pf/d/u
/vaPyoofcyX+sFnwB7tYrfP1f/TKav4//uqrX3ntn3zjH73y7FPHf3w/H+Wje1tt7aNzfran
fMYZZ/yt52lCyhhLZOSJOUNERFWttURIiI6sNwSS4ANHEsAq2PmJg2nlagIVEAaDuhJfDAqK
AEBkDF24dOnu//mt904eXbp5w06JsyZLO595YXDu3IPvfP/en/7FtWefPXfzmWlcnNy5N9rZ
uvSFz80vnFv+4M50/8gRkTNRsmXc6Si8P2kfn3bnmvLiueHG1iiXsy5N3rtXX9rcef5aN9zA
ehO7SQ5LDYsSrAs0Ozpujxch4SGS31rbef765s1LW9euLRF3Uo7zZX80oUXXMMi4ax/vZcPQ
WCkMeO/qylYFFEaczYXNxuDQmytNVsNJIcraIlCXoY/YB+5CDClFWQSe4/xo/x6NG0O+znDy
YO/0wR7Op0UOHhWZSTL0SRQNiwhnyaIiBEqkoMKiIg6oUrQCDmwgaJ0oyAf/BUIABRAQREFS
FWUFRQULAIYRM+QklIlsXUeAFrgkogwgqwaKT/oiwir8SkFURSWBpAKZMBNkVCYCQIMWiVbK
i4nEGSmsEgIBGSJLDtGiIoCA5pxzDCopxCx9wD5AiJxzL9pRKi+d33nuxpXz283ubuDm0f70
vffe39vbm06nwGgWXTHrLJAKJNUWRIGjipaNq4c1m4ohAwgKgSImtMYbM2QcBqY+ecaNwfrG
WrHhq5qsYUld180XuOzrpMRKQAYNEjICoBVDTMSkjLCUPhtqmhEkjV3UxKpCxqAt2Wt0YKtK
bLUMYRKXkbgsvDdIAsyqtliE9N7B44PxaVAYrm88/+Knnnv+uaquM2dVzbISvqIKAKo5sSoi
NmVx8fzOz+iy/1huPfsiwLe///ZdeOXWh5rmY3LnnvDkk5e/fke/defDWT/JrLUa/6guAoA3
7gA8XVX8mDvs6ev/9ON/2TkA8Amd8hlnnPG3jacJKSXUJzlfZIkAQJljCERYV6U3hKBPqnGq
wkcCdH74NwIBgQEhBDAqyiKEQIggKgbPX7n2p/n/euOdH5y7ft4XVDZ1CiEIrl+59qnB9v6f
ff/9e/f85PDiZ57bvbw7eXxIx7R28eLo39s5OHg0eXePjme1mALERnbWZNR+f6KPpzA6cpcu
Di9fyhvlbHm0213YsmvjXB/s36fZaWE5LeN8MeNlV2Vj1SnY9v7R8WzBjw+Lq+d1c7NYG62f
vwqXrmNm6HoYT/rZJLZtFxbT07F2E84xajZVUYwarB01hRnWOBhQVZqqoKH3WxUCZpXIiTmJ
giRxbYZEKUKYLyHMl6y6YTbXLnvd5cUizafQ9Ryi9FkDx5BEGEhZWEUBVLMYBSuIiSELAGSW
BByM6KoHIiERIIDI6huiCizMIgoAgKoQRHvWbA0Nm3PDBoK0zAEYPCgZVRV4EqlurFF4EmwF
IgG1L40QIgGQEpAiBGEFAAI0RhBEARWRiAA0c2rDKthcUs4xphByTBozpkyZK2OGRemKqtwa
bly/uP3CzcGNS7IxWlrVsnrhM9cPj5+7+/beO2+8PX73/Tr0vm9tDN5VQPVSYZK7eQ7cdZg5
CUDfNcqOtSGzkXAtUa1YBW6sGfhqY70ZkbMhYZukm6c+SEyFSClUZqOgQEYNJRUmVGMyaALO
KpnVD+pRXUHQnAIxOCrR2B6kd8QD31mZxdBOZkCGvPVoQIXUASAr9IkfzyYPT8edQDka3Hrh
2Rc//9LGuXNAYIiAjQgTAJFRfXJ1oDAgOIvO1j+zC//jeOU3vwqvvfZTGk1e/73XfhiIdAcA
4OVP3f631dgPWY3/9cOEnr7+Tz/+l50D8Dd2ymecccbPOU8TUs7ZlRxaFWlcWZlInTFUOGcU
VAURReTDr3yMlgJVATRGkdQoMCAiqaJCYl0/f/7SzZt7d95evvgcbG5xgaje99pPe2ft5b/7
y+Wj9x6+8cadP/p/L56/tH3rtho6ebRvfblz/tbOhWdO9x4u795LpxNjkSU1ESt0WVI7HbcS
0nQcN7erkOnarXpjd7w2bqsHdiYQeyvRxOBi8gKFKZwvGiSx3nZJ3j5I8TCsNfnKeXdxy6wP
tK7sqCn1Qhn6op0P2iXNF7Roc7tYLmbL41nXt4kTGwDvbF35QWPLqhqN3HCgwwpLW3hi5xw6
WasselKLGRFs28eYUs4ha1LOlLLERIlNFA7supj7kGMPOUkWENGUJXGMWfoIMSOrJEZNqwAb
YVYRIlARZlZhUFUQZghIbIw1npxXZ9latiamREhx2YkiCAfKTKK4smchEAKiqAIhISpqFOmV
iYUyY8omRRVVYeYMoIQgrJS0SCgpxxQlZkoMKWtmUigNrXlf+GI42Bw1w7KpiqZ0taemKDcG
ZljZ0YBn3ezoFBzAutPRYGOw+0tf+sVf/NwvxKP9xd0fjO++cfz+/bblXiUlbBGSdyigzJR5
JFIiDRG2bLlbDDd81Tg3LAoPRJlxmbRdpEULIVnWyjhgVEWLBhQEUIjUGjWWQQNz1MysgEhI
FDX0C4m5cKWvBkk0KkhVZYf7i+lRN08EVeVL78CQckJGICMAjHo0Hb+3fziLbAeDa7dvvfj5
z+1evJiRUdQg6iqJchVhyILCq3KoqKAKKvDJ8so3f/+rr33l1dtfe3YlFu5+48s/wRz1hNd+
7/VvvvLK3W/8k9fgAz/XR9XY6//0hz6yH1FpP6nY0k8RbL5a59V/+vpvf/MVWB35q//4t28B
PHX8x/fz42s+fc7P6pTPOOOMv/U8TUiV3q28DapKqqiaOBNi4ZwFABEiZOBVsj38iIMPVVVk
1W9DSTWrigGDiIrKoqJE1Ke4NRh86ktf/Jf/7I13/vS7n/3VL1tXIRmUDMYucncyX2xcv/Sp
y7uPv/vn779172D/8Pyt25tXrvUhjff2R2V1/vxVvnJ58Whv8u47cDI2MaE1idQg1GJxESaz
R/085QvXy2c/vXbrdjCZ3kt68hhmY8/eZ6SghJBB/NrQbg6zhxyhLJtyY51Zl3v74X0Ga4y3
5bCqNoduMHDbW2gRQYrlshifDttW+8jLZe67lFIfwmLRjQ9n434PQYVUCMA5U5WmLMGXrqld
VVlbSlnk0UAKKyVFNVFB0Bo/UtYcswFLSpySUbGgwiLMRtEgcc4pBMnMIhwjpWA4SQgsSqCs
zJz60HFKKhJjBAPqnJC1xgmZLuVF5mWKXehBxHLTGFcgxm6ZU2CRLKwqisAqqgqIopI4c2ZN
jF0oulS0yXbRsiArMBthzYyZHZNh9GidMd660lZ145111prhaDAYDozzvipNUUSVBBJJMumy
j5BiGaN1TudTBC52HbWj1GgcQrG+vXVrd2sjb23Jxnb56P3Tdx/MeSkj0XWkVZn1oraNs2tN
uVaXW8N6WFoCMV200w5jjn2vMRmBEsiAQVBaKURARkqlUWOi5owamdvQswqiIUCDVFjrmRBs
9kUkagm49B3quF+ejmfz0JnCN4PSlo4cAashQ+AEsMv5YDZ77/hk3HXUVM/cuPnS5166efN6
VRewshaKAMCqypfKyqyrkvOHLyMfXlCfHK98U/U3v4ZfwY/op6/+/sfpmpXoem0186tf/zq8
+urKKPPKN+98/fu3X72Nr8JXv/71l+EDYfHKN/X3Ab/ygaPrrywpXvmm3vnUl29/sMUP1/nJ
4x+/nx8/mafP+Rs85TPOOOPnGnzKzfpffec7K30kzJKzMKMKEVVlaYlWIeesTzrUf1RFrYTU
asQBomprIRtygD4rpqwGs4PQ92vGb4H9F//sv1m8++5/9B/+xrWrNxPLMsRMyJxyaCX2w6oY
rq/3J9ODN++evPeortYu3bw5Or+jIYR2gSTFuXUtcX7y+Oj+fdqfNJNQRgZvcLM55bxA9+y/
/5XyCy+fiBy++1bx1r/ZNFxAkG6e+24xW7Qdd4HLqtlc2ygLl6yDZlAMhmKwDSH1UTNzTkDq
B4VtCmiKuFnTelM5XxgsnUMiyAwxSEwp9H0fYgiyXGjfSx/jsutn83bejsfTWdctYkikGSmR
yUURVdgglB4qz85iUZii8GVlrCNjEoF6qpvGGYuq1hjrHQAkkaScVRIzpAChXxWOUlTmzBwz
x5RCSoklgyqBRyUQTVkXXZgs2zYFRHCghYjP7BObECxASCmlhIRZJDOHGFNKvLKPiKQQ6wTr
nQ6XeS3RuisdogMa+KJALJWc8+SLsiidNYTgnbPeKQE4qgaDYlCBpdbALPfdrMXEpXGF8Zaw
Cz0D+0EJzhiP5CKtD5fru6f1SLc2h1sDL0s/n/Dx5O6f3fvOH795etg26ga+HDb1sCmHVdkU
rrKWJHMKqev6voU2Nl2yoAREQKiqrCpqiJAIFAE0EHWWeuYYc8yJhUXVkC2ct8YaMt4YBMMM
wZpYugnyYdceLGbL2CNAU/hh0RSF1caKIwlMTGRcVHg0ndzd3z+YT5qNtQvXrn7mpc++8NlP
nT+/A4Cco2RGg4SgIgAKCiLCzKj4JMQQAQD/k//81Y+9Nv8d8uGL0BkAcFaa/IwzzvjL8tQY
qcQAKiqSc05JVQrvvXe0eiSJgMKq1M2HN+J/K1jqiU+IRUQZFZGcKiEigLAY7+d9Whs0X/z7
/8H/+s3/+k//+DvnB9u+KW0BQTNLLAAJi24Rl+3RZjO68YVfunBz/PDNu++++T1/z+1eurS+
s02eTh89XCqv72zceOkX9EafH5xOHzxqF3MM2Re+DDx5883zt58f7OzawXo7wf1H9/0augGa
gTfnLpRi7DLqNPTzOZyIKUsZhMV0hk0JZWEcRmHWaIFMyjTP0nU8m82Bp6BV5ZumNI6KujJ1
Zaq6WF8rnANrhFIIPS977LPOuzBbjI5O2+WiCyEwT7vlcrbkRZpPZ9N22TG3nDvmHnWZYgBt
UxA0g93t4eVzAJhDSH0IfQgxAGFRlaYqwRklzO1S2s4X3jnDyopaVH5tfUjWpBzAQLto+0nn
xKbIMSY0znqfRYxklSwxpLblPjpDWPgUYk65KApHUFgjXU8qzXDonEuSYduuqbkQi/WFDHuu
wSoLApSuqJE8A1rDhSUiQLDW1E1N3gZJbLD12psExvQFdR44U5VMzbbI2I8X/XTCFiANi81h
Mxw1fmigGrIpQuzGY+JQlDYtZHK8zF28tDa4WA0b3wzrelSWnhnaHroFz1uM2cTskg4BV/UY
AASQZPVzJAAkRRIEZo4p9znP+5xZCuuNqAVTlKWxjgyhMaIaVYNCb00PcjweH3fLmcSEqETe
kHPOOSKDWQAYLXlVmHXh8XRy7+ToNHVuY3j7My/cuH7j3ObmWllZAM7ZAjACKuiT6LfVG42o
qggQrpIxCD7iKD/jZ8iTDMWVeLrzxrcBvvqbZyrqjDPO+Cl5mpCSmISFlUUyqK6S6q2xACAI
5OzKWgWweqP+oZb6oXXqSbTHKtKKFEFQHUJmQUOAgIU/Xsyfv/Xsl37t1/717//B9p3vffGl
z1ZKmgIQBZQs0JiBZllO2iUum1F981d/sTs+Pnnn3qN333q4985wd3fnypV1qrrDyUk69s2g
vHx5+/nnw2Jy+t69ND7xSSYP9oZv/KCpB8Xahn3m2tHBnoznetJZZFMXWNXNaM1vbeIWQM6x
C+1y0S9znqIrqrqqKmtRwBFa0MApx1z2VAijIc+S2nYpcYJqi6Koa1uUaB14z6UHY713hR/g
9shsnatv3qQsqJxyXi7mi8Vituxz4hS57frZYjnvu47zMqdp303axbzv2Jmd3fMXdna72WL/
/oP5eNomjCGS5ALVohHQZatpHlMviaCPvS1MaTcKU9bDgSL3qe+6GK3N4IJgTDyoa9/U62VZ
GxygNCn5rqW+N0jOO+tszDmlbL0bDkeCumzbruuc9+WoSRaboNsLqA6XcDKHJAmhV2arYC0L
FtZai6HvEbGwJYloSqqZs6YckL0prQFTKZM1lS0oyqJvF/1yGQJmwCmltk+ni0lRskFbOy1o
EeMkpmxdEkSWCujm9rZN2WQDKfLRJCw7CsGw2CReyaoBMCqoKNkYBZSVlZRIELNKyLkPsU8h
JVYla1xjfGm997UBQmsEtWPuJPaalinNNd9vUgAAIABJREFUjD8RmM3mOQtZY9VYs5JnICrJ
CDiDxhDYHPN80R3OZu+PTyYay631z/zi524+8wyFvF7Xw7KgxJKzcRYkyypASgUQ4Ul1elBA
UQQEkQ/qgp3xs+bWb//O13/39quv/dAteGaNOuOMM35qnmbV/9//xb9kkZXrgQxZZ+u6cs6v
3B9ICKAoT7rhAgJ8pKA2rFQUIbFqluhRnEFQE7NdFU6sfJdj6QrtYkVybWPt9f/uf3jwJ9/9
e7/wiy89/wIbaIF7VsPGdwpdbimrUeJEkutB7deGuW33335ncTCWwL4e7ly/Wm0Ped6mWUQl
3GiqrSZrkNPFdP80oF//1KebW9eMhMmbb82/+93i0f4GM+UcLc6ctgOX1ksdFHVZrvma0Apr
DCl0IXVBU0ZjTOmzQUbdIFcZysQLzT0kGvpiWIMlFUkxKYsjS1AKUEYVa6F0VBTkfMw55eys
K1wBhZ05Au/NKjEvSxTJir3yIseImEBn7cIY2t7cMgCQObRt33YxhLbrQoqiGmNahjZqLgrv
q8JY0wybwVpDFlc+PhURzQvWPsti3oYQ0VDb996aQeGGBuucqettH8qskLMKpxBUdG009M53
3bKPgayp69qVhTXYBPUnXXj3gI8XBbkg0qZI1pWuqIwtFaHvUt9XRTWqGwLInBmBLai3WHlX
l84ScAa0vqxUYTFbTsaT+XgmKVXOEWvMqWvK7E3lqC4sAYEtyJZZiVPGECAstV3aIEYUVEDE
IBAiZ0FAQCtIqiQASoAIIho5x5xCzjFGVmVQJLTGFcbVvjBoRZHIC2KXuOM847Tk1HJsOU0U
lkioaAQQVuHgoKBgwTeFbwrnTUlFCrI/Hj88OTns2hnI2sXzn/nC589f3JUQLeuzt29euno5
pRhiMMawCKCKiIACAhKpiAIC2NUViUiq+p/+F//Vz/wWcObaO+OMM874a/A0i1TM3apbmzHk
nPHeOWdVGVRQddUnV1FW0TnGGFhl3ctqZNU4FzMJWyE0lhlEM3NE8M5KZi+IicnRQ257tr/2
W7/1r0+67/3R97aK0c71i650AgkkZ8paiJUsnFeujm62aKfzsqiu3HgWb+Lk6OTg0cN7b/x5
46vh5tbGxparB5ri7M4D8JbW17ae3w2oXT+e/PlJPRyt75wf/YJbju7NHz+kxdxwqIX9oo9t
m4hFdeELU1fQVHZ9rdiqAerEyklSZA7MIY3DfCESMUcSMyjrwkAnjDHnpJwtgrUuk8TMLALW
muy1s33mxbxb9gnUolKv2pWk1gAAojXWsTFdFrbEziaiqJpBGfX0aCGgtizUoPfeDku3tlEI
IoMlAq9sGZ1B4xg0cw45Q06xTWnRWxVH2Fg7cH5jWMShRNA+hxCDii4FFhkzU+Gq9cLUZHLf
MfYVmWwsJ45CtqjrpiLvJEcNCTLGnOeconApJFlESJljjgyhA7Qi3nnji5xEUkZVtMaCJedV
PAYSJhCHSKzCwjlkFXBVIYWjoiirqram8SiOVoUepE/QJl0uXZdczJaBWFEQvVVkAQJQAFBA
sqRPym4AKojCUjhlzilxyivDqgUorbPGOGu9c8YQo2aVoHYhaRp1GnXBHBR7xsSWFVGlAhAE
RVXQjKAErEKIHsmCwYxd3z4+Ot5bTve5j01x5ebtTz//6bW6WR6cVmVRDweuLlcduxFNZFZS
UACDH2oYAVAVZ5RZAQBVhD/ptL0zzjjjjDP+sjxNSAEAMxMhonHOWWtXuXgffor4QVGEj4RJ
rcxRH/AkkX4VAAIsIoIAoj/CiKrlZDnb2v7l//If/t//7f/4r/7w27+ev3j+2uWiNgtKuVCh
bBKgkjA/MX+p9vNFmM2LshpsbWye2267/mj/0fHh4eHhQ182m5tb65vnbFHEkKfzgwRajgbb
zYiTCacLbwu9dtFuN9DOw/Gxzuc4a6ue10AzSkoq3TIczZI9parUwqKzrixcXZu1UqiOkDOw
YbZ9yCG1pzM7tSlnZvFl4ZoBlF5BUbJDtEpOrFHkpJXSCCjH1LedxJjLQowRkbaPi74LrL2C
OJ8NRYSeOUhWQkZS59m6NouiVQUQ0JgdYGUNFVoObTFoyHkxBNa40nlnhAW4MN4a7wyLZc0q
KokMGC1KZxTAIoGTZHpQCcDLGMg7AOxE1uoGOPcEWZIDKKy13hFIi842nKuG2ixqOTJkQVUE
JIUqpHq+xJgyS2p7Teyt9WgBkTPntkuq0UG0CACEZJ01hanqjRIRQAkpc459hFlHIcXIHCJk
MQCYgRgsoDWGCFmEUXXVnwgICQFBsqoCc04ph9BH5qyqoIhYWGuKwhpDxqxaEBKhiraoC+VF
iPPYzxmWTAvWTjQDioriKqYPDfAHVteValNUMGgQTOjzMnb7i+l+O5+kONjd+vxLL127cjXO
2vbklPtgi2I4qAd1iYSkZKwVAM5ZQQlp9R4CAIgoosxCRCISU/rEyx+cccYZZ5zxl+ZpVv3f
/z/+QESMMWVZFkWxevx8GAKFq+rXBB9Kq4/69T5k9dCHVfs3ZlEFBWesqKiqIQMihiijjuP8
yu72ds9/8ju/G75/90vPvXD5U7eW0nYehNB0ahIwM7CoKIgYJGXhGFW0LEo7bGBYGJFF2588
fDzdP3Zo6sHaaLTRrG8Y50LOqWfHxlQ+DGyxWWGBIfe2D3gyNYcTOJnTtI25W8JSOXsyfhUF
AxqUl5qCJWoKrJwWFgvnfGELb0xBxrAgR5aYURAS5JwDxqQMCL6gsrDGUc4pcWJhEWThlLUL
EJKwyKzt512fyATFXmXadVFViJTFEWawkbwUTTHaunL99oXdC6cHh2/82Z/NDg9z3y67sSnV
1xVYmwnUWt+UvirQoLE0GDR16RtnqsIpIaPaypnCKQlzBBFCkMRowAyLqBlZNSaO0RnbLpeG
yBWenM05R812WHtbWCHLYNGQGiS11lTWeUCKaWPcrr23P3nwqFssTQaPWBhXOEfGGu+Mt2DN
zENbgCVDkTUmk9WEjJFNYogZImvKVsTIB21TVAGRyDKACAPSyu3FKqtfoWRNOeXMMcaVFhFQ
USVQZ8g5Y70n69SQIGURBhDExHnRtuMUTkXarD1DQpfJZaQMyKvjagJUQAbMqED6xNalCoiI
zjLoom1PlvNH1HWVe+aZZ1568aVRVU8PTrQLBVlf2Hp9uH31wuXr14q6FtBVdl5KKedMRB9e
TavmAf8fe+8WdMd1nQd+a+3d3ef6X3EleBEp/JAMUZYdjR0FGCfO2IlDKJNwqqaUuGpmVHkw
OG/E1JTe9Kg3vRCPxMNU5WkqephSUiXiJVWeOBYsx5Jl2TAlAT9JkQSI6389t+7ee601D7vP
+Q9AECQlmXYm5ysUef7dfbp39zl9+uu1vvUthnPOqWqMEUb/6x/8n39zF3/CIrW3wAILLPCL
4HERqRSCcs5lWTb7xZ+BiAATkVlq76GSvYd/nQnE7FId4KzKD0aAGOoo/bx3+84WH1r+0h/8
/vf/3f/zH//oT/9hnDz7mefAsu/MKZEyw8Bk1nREMVVy5BwFqcvtcdiNnOedbv/Tv3IKGxu7
W3vl3nBn5/6tW+/1V9eOnDjRP35krIiDUdifyNYgc2j329Tt0fpyfvjpEKt6PJbBbra7FUfj
MKkmg6GLUoC9xJ5wOygGokMpPWo2NYLz7At4l2V5XhTULjRzocWUFeS6mUYzgYZJqOOwNpGD
86NMxs57ByV2GXKHgrO8lbe6RG44GFbVpCrLUFqRF63uysrhYvlwd+XwiaeeiWW5v3VPR3uY
7BZhskKUTSCjcW1WmwQVIapTQZ/KJPfOO2Xx7cxlGWfU6Xc7vY6xGmK321leWWoXRZZxuzYQ
VZMK0TLfYiVrLZsiRpWoUVjUUxWcqVbCCgZpjBlTblQYtYO4spZY7yLYWm/lyGpBPjNkcA5E
gCZ781jn+8FXdQhR68giLObAHuQUEHUGR97DmKFTwiSmyWzdHCsaziExSIx1SJwkqhoRO+eL
rHCZd84zgx3AZMwCCkQRXJPsTibDshrX9XAyGYiOXCHkzOXqfCQnIIDUhEgITKZEBlNqiicI
IGUSojLW2+PhsCwraOf4sc9/4XPPHDlebu3cu3Gnw1lmMJvk3eVWJ28vtZGzqBBzetJIGfB0
mcwiUgQCEGMkIu/931JEatpl75N2AJhv8ftzvnv6x8K8IGFh5bDAAp8EHkekYoxFUbRarcSo
MOdukHJ8BmOmFKlKwFzVXuJeqmpk1NhLzWJTDZFSVTYAloFcMHbFe1vb1frSb/6bf3X9+KHv
vfaHpdMnTxzrdQsyEhMmMFBLJDMC1CTZfgJEUC9i4/Fof1jeudfq9ItOb+mZJw89faKs6hDE
t10okHHWKpZaQbkKUgeUYbK/Fci05V23yNvdvNdrHXuCo8Qo9XBQ7e/G8UQnpQ7HqGoO0cXY
FuoqsZhXdSbMUWlSsZYZJplVOZnnNmWFy3yeuTyjLNN2W7NcvI8+F5eVokXe7nLu8oLzlvk8
gJVd3u4E1bKqzDAaj+7fumGhPvbEieVDx6LSaFwNd+//7Kc/Lt+8dmi4dcyNcqqZ8ypSqDXG
KKYCVTUW867gzAMkgcSTVrCqUhXaGZakLmNiTAqXLfeLtbVuv3NoB23CYDiKdei2e5nzMUYD
J4ZjMBhlcIwIIyJSIgHA5iC5akutBYLzEzF4x2IW6rqOZV1LFUzEgsJURbu1LQWilF0jB8Cm
3mNNrz9D9ERMAJuqGpsiiIqIiIQQ6xBiCGq1c0TMGbm8nXufuSxjcuw8sTPTCETPZYxRJQIT
1XGI+2W1O5kMqiqYGVH0hbq2IVNyAjZz6YvpYGxgVTaFGBPArICYRbIa2C3L3VCORIqV3sZn
Np589mlI3HvzXTcql3yeWTCo77Xb/Xbeb6mHOjMmMswuFmYWaWzYiEhERCVpppxzzCx/K4Gi
ze986wrOnDlzJXl4f+BqF89+89R3/27coB/kC5sXz26co6sfm5H9XTqiBRZY4L8iPI5IMbNz
zjmXbnLpp3/Gk2wmUJkjT7Pc33wSEI2oafYuprk/zcyrOGZ17Dx30N27v//G7ugz//R3Dz/z
9E//4x+98xd/9fwTTx956njFGkM0NccwAySlaNSM0rayCDIQeQGq0UCHw4FzWafNnW5nqVPk
PmahavsQMb67V++PfLQO5R1fMIPI1RMNg2Edo5opQ9p5a3m1dewoe+dDrCelDMc6GMpg7MfB
lVWoq6ghWlQL0EAkHbJuFFRRVEpMlLg2CEicN1dI1uL+Ened7/Z67U6nv85HV6nb5aygrKXE
VdS81cqd11h7IGfe37p77/Y7vV43y/K6LPPxILfdzpHss72ning4l5JjPQIPxZkqRMlMBUxs
RmrEPiP2IqJmTMYwMmUzJmSFA0wsKqvz3kVkqFsFtzs52rnzXqO6PG/a+RgU5gRFBEeFQKXp
Yi0aLEaqK1fVqCoI5cqiSmpMRGIwcyCIkpojR8YCaIuYyUAwgwGqjsg55wyioiqlSa0qwSQE
kaiiGs1EVYRBzK7Ic5dlPmPnPbEn59l5BQXVoKZmalaqbo/Ho6qc1KESnYQwjrEWC2BxDuyU
SOEJrmmIY0RgT0iNmiGRTEwlJxCojhIdVbBBrLfKwcTU9XobG5/59K98htltv3vD9od9n/d9
gRCVTFuuWO5wO6M8a7XaNCWF6cpIxRnpSSMlx1Nw100vMVWdirI+UWx+51tXcP61r+PcuQ/u
PjdlLp/05B6Ny9+8cGXOsqBpt/zx8HfriBZYYIH/msCPWZZlWWJR84NmJiI6dT1IQQKdw2y1
2QvDjFHNiNVBV+Pkxw1vgcVUl1Csa1u2yjfeutn79MY/+F9+f+2Lf+9719/4i6t/PRkMOq0i
KzIioybtkYJbaiZmogYCTzXwSZ0V68H+4OaN29d/eufN63vvvjO6eSPs7fT67cPPHOscWRlw
dXvv9u3bN+6982Z472Zrb69bh46iq8jHZXVvd/Devf17O4OyrltteuJ493On+2d+c+m3frP9
D349++8+X3zhdL7xaTp2HIeO+tXj2dJR3z3iOkdc61DmVnJayriXoZejl1O3l6+sdg+vdNeX
e+vLvfV2eyXvrlDRVdeK5I0LWDbZG2FStwOyURnv3Pdb20s726O//NGd//yfRt//k9abPz58
78anquHpwn221znZX3luZX3j0KHPnTjy/JNHf+1TJ3792ae++OyTv/7c07/63FO/vvHs2S9+
4Xd+++zv/pPf/r0zX/zvTz7za0fXvnhk/YtHDv+99bXPd3qfb3d/tdN/vtX7jMufM3+sotXd
sLYb1/ZkZbvsbY3at/da7+0UN+77d+64N96zN26Um29Xb74d3nxH3rqBt2/x27f4nbt88z7f
2cm3R51hWB7L2gSHSj4c/Vr0K+aXxHUidSP11S+ZX7Ks5/JWnvrGeHIcHdWeRizbWt2Jo1v1
8L16eG+4t7uzu7+/P5qM6xBMzXvfareWVpZX1tbW1tdX19c6K6uut6StTsiLMbkdsXt1vBfk
Th1vjMbXtrZfv3X7jfv3397dfW9/cHc02q1DZVCfwWfMntgxqJE9wciMYAQxaPONJSU2dhxB
taIy2imrW+PBnXos3dbTpzd+63f+4elf+exoe+dnV/+ahqNl73OJ1WRYxnHIDEuFX+9Zt+A8
7xbdQj1r+lo2jyjee+99ejiJMTYtlaaJP1XB30JEKvGoF1944cXzuPKt72xOxy+/RERnz54l
Ijp79qsXrgC4dG469NLl9OaLZ4no7MXNubekP9KShGblB7c5G5wtSEObF88ebPHRSL2EH+2g
+f45pB3NzefsxU1sXjw4ovetMJtbs4GLs0UvXU4zfegADgbnZv6BB/u+yTxm2mn8pZfOPrTT
Dxp/aBcPrP6+dRZYYIGfEx+ikUo/9LORxJPSwzQAoLFlnuk8mtEPjkjBTA2cSBQlcwXU3kqK
eW3tyDm47Xq9Tmtcjm7+5N319bXn/6cX1z+z8ca///ev/8VfHDtx4viJJzKfGZGIAqogAJwC
PxmpTvdrRjCLyoouMRPpcBJ3h6SuYj/MXLG8kh9e7x0/1Dq6FuqKJpMw2N/evRvqypiZfStv
99rdpbwtGq3UCZcjplGecaftuwUOL2XHD7W8c7XUZSnDiU0qqYJUkc28WaeqnYpCDeAsQyun
ditbXdVWQd1uRaiJc58XvqXkjD2TKwhVQJtiDo3b2/d+9lZ18+18+04+2ON6nEM9IlSYODpX
w5kvFKwWgQCVdB82gzBHdllvqR2ZkbuiGNx+7/5b18rhqAXmqE7MgUxF0u2ajYi9qEsibVWA
cqbUVgUGVYNIJCszIyOvcMpsrESBoERgZjZWgxGUiNk8K7NMrbtT32JHRAYxlVpjjCHGqKKq
dYxRo4GMLH0v2kaOc+e9y7xzzjEbmBwzs4JUJJqVcKVYEI1mZZRxjOMQJnUc1XUtGmIUKNgU
bM6ZGZMDkRhIAFPnQMRCFEym9acAQ9XUlFgdQdRUYpQQQtyP9a6VbqV39Mljnzr13HK/d+/2
nXd+cq0Ar3c6BXOMlYgQE7WcW82LI0tV2xFZj11hWVtd5SDcRJtmEVznXEpaEoEb3XeK9TJ9
8hGplNf7ygaA02cwbbg7xRV85bp99+QDubTNi2c3plnA669fAXClad/77UvAma98+WQjX0qZ
t5R3w4FqZ7ZNXP52M4eL37iEM69c/4hJts1rVz/2UV78ajIy/9q1sxsXLnzz8suv/ttXvnWQ
HXzshC+9fsrMNi+e3bhw6dw3XrludnLz4tmNC+deetFefeEBpdfll+jcxlkc5BgPDvaxk3n8
gV+5hNfMXkg7PXt6tvUPGgeAk1/+ypkL048pfTKnNz7uaVtggQUejccRqSzLgAeK8tJDM00V
LfOxpfcHrtILMhCBk8dg+s90RWrETaTORNURe7BGldw4Yz9Brnm9NXhzb++pUyc/9b+/9M4f
Xdn8/p8Pd8snDh/pr/SplamnEIMLkqvlAgXVZqICgImgcGKkMMRAUDLnOVczi25SVcPB4N23
o2ne7bR7nd5Sr73eW35iTZwNQlXtj+thvTXe0/v3WCgv2lm72+t2OVcbhfp+0Jy03a46bV7u
ZUfXOs910Wqby1QtjspqMCwHQ52M6uHYqgpRWI01+r0dGlGrahsZguB+z7e7nLfgMiKSqvLD
EUFDVY7v3C5vvqNbd305yDWQRg+FCsyMTAPUOWIWBUfLDCJKBmJVImOLpBr2xqN69PYNMZNq
zyZ7HUNGzkLM2bFR0vyLoSFLakTeOY4QMyNmTQrv1FWXyEjhU6TPkTKDOX2yZEQgEEEEGjMT
MiMRpN51IiIaBaqQxN0asRRMHbEnboO9K7zz3jlHDEAdq2NiTrlbgQWzaBZiTE0Aa7OR6CRK
HUIIWsYwCSGY1QoBK5P6TKCqgYjYCNPMHYyITB0ZcYQJYI7EVGHKbMTKqqaeTCTWk1EMYdfC
vrfuof7TJ547euLY0nJ/sLv3+o9/EEfjw/3ltsu4imWohYwd+cK1V/u9I6vcaweYxejYsWOA
U6ArnQGd5sods3embCJiygIxIxBIqXng+ASReNQrXz4J4MtfOXPhwoNM6sxXvvy+TF+6RV+9
tokXrn/7Es6fP3/p0uvXAVy7OuVR37wEnHnlay8AwMmXv37+wrlL37j4tRdefuQ2X//mVy9d
Of+aNXv9ufJ0Hw2Xvn351VenW58PeSWi8f4JfxnANPZ18tTzwJX3n5AD+gjgha+9cubS/Dl8
1Al8xGQej2ZmzcRevw6cfOw4gAeYFOanuMACC/zi+JCIVIwxhDAzPkiF2cCBNArvC0c9nN0j
OJu23qNmxbSooWVmWU0ZMgXGHnkG1ZCR+o4r1PV8Mdzdu/XDnxZHlo9/+YUnvvilN/7Td3/8
+o97d7aPHFtfXu11fW4onSlDRdlMXSIBhqhmzJoiIkwGjiY1RUfEnrIIDqJBdG+n3Lo3EsnY
tduduNLVlaV+v9968hCTQ1CppS5DHI6He1sSRCSyF85c7T2c53bL9bqu33MrS/nhQ/7Y4fzp
taI4BgNEtaxlMNb9kYzKcmurvntr8t7tvcl+oTFTlphN4OGcEYNJYbGuSGqKAaFuh5K1LDIj
MCwzMxjMTEUJlDmw1QpTZ1Fh7BlklM4qMoLFoHWtqgQrTJmy9HGBMzMzYngmwE0jhWBK8aDM
NTFINQNT6k9tZpmZT6omAGraTCalek1VNUrQWlAbLEW2VIwUMCNQE1gyx857RyAiZufdVHuH
plkKIGa1UQSiSYhai9RmpWolUqnWIlWMVYzRGvm5Kamaqmn6apHCACMxEwcYHMyBQc6MRFVB
BlamaJTMDpQUpqyJ25lIGNZ1sBhiUEd6dHn96SNPHjm25PPy3tbPNt+Q4XiJXK/db5ELoQ4x
xsxZkXGRtVZ7S4fWslYhCm8AuN1uW5FNYJokamqp31IyvyIix+Q9mQVVTYo0UxPIQeT3E8Lm
d751BTh/6iTQUIWHY1KPwMkp4/oyruL81199EZe+cW1z89qUkm1+5yqA5099tPv2lUuXAFx5
rM79of2feh648tHWnb7lwZYw51+zV+fCMynE9VEn/CDSe69c2KC5ZtMPcZoPn8zH0btfvbaJ
R63/8PiMSX3t9NUFj1pggV8qPsSQM93kZnm6WZERmqK8psHe1FnzgFfN0JhOz60zjWY96I/Q
rA1R0SjeeQMFjQor8gKErZ3d2zv3nnjmqad///cO3/z1N//4e2/+9fWjN/Ojy8tLh1as3Rpo
GUWccibEBpjCWXCkAifoRHRqC7D9DJoe9JNjjxGYlCnUphLGw/1sd+Cr98ZMk1buex3qdqzf
y5b7vSdWJM+V4AgyGMn+KExKmUxkd1CFOwApOWHmVivvdFynkOUWry75bp8yl7VanU536enj
aLkAre6o7e9hUqvWZqxEYEeOneMCgCnBnCNYJg7iMjVtTpcBgAMAYzARk1kehUWnpx0qSsxE
ST/WkC+oAIq5k/7ARzKHpmwuGaiaSdDaNFVoms5rzAXSbEU1VR2QmpoJYJ5B5JzP4InJOXZI
RMo5GEWoETETiBRmhlolxBhERCSKBInDUFcSo0odNMIiENQCTEBCJCmuk+aS8olgsaapo6EJ
fnqDj0pMIFZG8p/QJMlTzo0LxZit9sTMzmAhSoxVrIdSlZmFjs8PHV594uih9TUXZf/W1s/u
3adx1eVsKWtDBaE2xwwFBIxWt91b7reX+1krNwKzK+vY6XTa7Y5qisTOfP8PSjTSJZaivzFG
gJvjmmXGPzEkHoVZyzlglqd73NsSk3r9O9+5euXMVzawcfrMlW995zvPz0Jbp54HrnywcP0h
nH/tNZxrQlYf7V7/wovncenxJYbvn3MT6Lr8Ep27dOnbl1/92tyyjzfhB7ebAlUfr2LwfZP5
OETq+VMngesfMD4fZ0uRqqvf/ubV2SezwAIL/FLw2KbF0+BT0sMmFjUff6LUVfWDf+0bIciD
g/PhqySTSi2PFU21EqdUTqwLA2c+cpV5/0R7/fb23Vs/3bzb8sefPPFr//pfDm7fufGDH/3w
6rXe27tP9VfX+30qOLIF0lJFGQzXNpAIi3mz4FEbkblplR+IwT65Mag3BrOJeROfmarVk3oy
3gumAVDnrFX4Vt7p9dq9nq30sn6rf2g98wWULZhNaqtCrEJV17Ecx/Eg3q7EBOpUwcS7YFZz
ppmJg5AEWIwpcJMYEHMS9hMsA1hFksckR0vdDNPJTGzBpudflVWdxIduz/OfIACVaDM2Nltk
Nm8xP+9slPYw25Sm5C0MainXx+Q4ZzRCbRCTc47gzJQgTEyJf4Ob4FRqxAuIamWoU/gqxirU
QSRKjCJVjEFiTL7kFiNMzYzYmEEsoJjCayAxFVUITNXMGC59MVOyObnpgygzaweALDgNLnkr
EDyZmVIzdYbmakHisJ6MQxWgkjt6YrshAAAgAElEQVRe7SwfWVs6fsT123uT0e233qZ7+1ZW
nSzv5V2OIuMqanS5gwNnzrdyv5R3Dy0tLS0hcyAFXF3VpsjzIi9asBTeg0y9xGYPJ2bGzEl+
rjqzT3840PsJIPGoORKQlFCPYlIPRoESk7pwATj/9ZPTvw7u1onoNOKfzYvfuASc//rLj779
48zpjRdefu38pXMXvnrxy999OcmP8HhikjJojUZpOu10HBunzwBXXr8OnGxoIjCvgdo4fQbA
6Q2cxMERfdCEH6d4n83lxfO4NI3jfSQbp0dMBnjktBtcSTNrJnYgsv+g8YfmdgkLHrXAAr9c
fDiR4qmFIAAReUhUjoNf/oYhzaw7D56n7WCF+fsH5u4WBlO1VA1PjgTmMj8eDGO3v3r0yM57
7+lw/FxndbLD5fZ4uP3Gnc7Plk8+9cQ//0dP/uO/f/fP/vqd7//k/js3ui10Vrqtfo8dsyAX
5GIqJKBJhok3EyqiS3rf5qabOpM4YxCYg9ZjFySjZGDN6lrKLVGthQYj3h+5uzul2k5hVS8v
irbLiqJot1vtbq/PeeFWirzodlyjbSFVFuJKpJIYJHGXgCiI5uBhvhSoGRIhEBOJVWUhhKik
CjMS9jWQJmupF4/CtLEjAplZtFBbeCSRmjvPTQnYQ0EOOzCpb7olNkarszWYmcmIzBTJ0YCS
bySDCAQzVpslc8nUQJkCqhZVRKKYRpEoGlVCjHWoa9XaICJBYxQVM4MFlZhokWowSW74Seyu
ImpRU5IQaadmpkS+ma01yjuFJYERACKIQ0UES4lH8kQOrAYxKEkkFaCOUmpZwoYssla0Dq30
15a7Sz0mDHf3hu+8Uw3HndqWKct926nJcBxCMDJkDM+1A2fcWe72ji35bhFZvHMAJmU1Gpfd
/kqRtRiO2Ik0dRCqmmUZTS1FGteDJo1nyXzjk45FATMeNZfzSdmgxKQeWjndk89RI6tOKz4g
H5rb0guv2vXTZzemga4PT14lZnThm5dffvUjKaJPvvxdO/USHUTSZrtoBEPn6BLOv/LKGVy5
kiZ0/ZWrGxfS+mdeuf7ySeDk/BF93AnPz/1Vew10bprb+wgH+4jJ4JHTbnDmldPfJjqX1p7b
+AeNz+3qxfO4dGmR11tggV8yHtcd4rt/9ieJRdG0fwWmN91pIs4MTRpi3vr8gSfpqbIHwPym
5u/3s2QfYEzwQGZogUb3tpby1sazJy2Gu9ffKKJ0jOKwrOtqEKqtOAptv/bksaeefc7Mbr+5
eevqj/Z+dqNb2zNL6+utridSRgUrLdaIxvDqfCSYNU5DEIKxitWVhmAxmkiUGCwC5ogceVKw
gRSIAjFTE5FoMGIQCQjpBVMgqHNc5FmnnRW5z5wvct9qcVG4VpvbLe8zNnIEnsZ5XKWkBknd
e0WjxKoiFYhZjFIHG411MICImUDhOGXdhAAmYhCRCmmgmOjEtKOKJULROJUaFAZTBtm0O+J8
7ClJdii5exExMQgGEJMeRKcUBu+cJRdUhaioQURDjFGiqIpobPyeLIpE1ahax5hqAwUwVTFT
pN4vDTc0IknkyVKzaxNViKSuvWqkplMm2Hh/pyOSqWUlKdTIiERNiJQo1Q4GZ6UXMnixIpI3
JoGo1bAaVpJOVEuy2PLFynL/6Hq+uhSYxuV4uL032dr1o3qF8w55z0wEiVGjkKmZUOFdK0Mn
o27RXu33VpbynqeMYkytIP39+7tw2fqho08+/cwTx09kRTuKEFlVVWaW5zmmDuazyypFpOqp
U3tiV//mpUWLmAVm+CDz949qCj9fjrjAAgv8svAhhpw0NTFPmLkIztaxOdnHLAM473VuevB4
/WimBfBB7410mzcRFcC38sFgdO/2rSPPPLNy/Njdt97YD7GX562svTqw5UiD7dHd917/Lz/6
ycpzTx4/9akvfu7F8f2du3/1k/f+6vqt2zf7Ll/u9/JWXmRZLqxRTCEiSPZBABlZ0igncqEg
gzdmc7OWJGpQAhyUYJ5EzSK5SnNJ9z8HKAhmpIBY1GGN7RERBQ01Q72LzOKceed85tgVWd7K
cucy85nmucvzIveeyQhqoMxRJMfksyLvkPa6ttR1lFzcTSRYsulWNRU1SS8y04YzJd/4xFEA
zJ/5JCEy02QKnz6XKZVKthSpko/MDKQioskKs0n9Jc6kajFKkBhFTAFCLTGRIFURoIZJClEx
Tc2RXDqNQmSUOjQC0yaMBphzbLNImKWDhYLM2ECWejrKAZGCAZAkYjcYNSE04sY3oEndGnJl
MmOBRq1Vo9gYOvY2cFZ1vF9a6h1aW1tbIYNWYe/O3eHWrozKTOwQfNe1czhEra0KZARjIuc9
+dwyaO6yXrt7ZKW9usTtvNYAgYhOxrVqXdba77eZXJG1mLO6DsSkqim4myjUfLlrunCI2Hs/
u9wW/GaBXyouf/sSPtBya4EFFvh58TgiNU+bZqrztOhAPXPQpOUgtZdeNAQLB6RppgiZfwgm
ZqiSwZJvTuprZgSidqezvz+6eeu9Xr/XObTW2dvev3NvEmvnc2t5sZhxfqy9ure7u/+9v9r8
89fzE4dOnN44+aUvyZf+/uDmrbs/fePWm+/y1lahttTpFJ12E2IiM7JoqjGaKAHJjpGMTcwI
kmrVwY5AU2VScm5kQGDBoWJt0n9TIysCOXYOIDUzNSaQZXXsKlgJSqomZDXZHlEki0SVd+mQ
XdIUMTvHUMpcnueFcxmY4JgYbqrpITWTaBY1BBUxFY6RRHjqK0EHQUCbSsYbTbhN1c6wpkni
wRpmoqppkGBmohKjqKmaAWQNcWs+N52qkUBNHRwxg12ERbPkbkngpo2MQZHE7wRAuUkI69QU
I2UKU7rVDKkdsREZsSGxNVhiX0mWl75Oje3qbCaYmmuaAQK4QPkkGqwmK2FjsqpF2u/qUnvp
8Gq2tiQZh7La2dqebO2GnUG7xjKyDrKMvCMKUWutlUzIDMaOHZNzDE9CEGgr951+j4qsjIGc
DyEMB5Ph3pjNt4pOkXezrN1qdZ331WSS5V5VZkRq9lgyg4gAkpKpqUXMgkgt8EvDtBfhgQP8
Agss8EvC46L63/vz/4KpUsqmtXhp0SwTJxZhcMw8ZV3zSh1VZSKmg8Yy874JmCUKzciIHCvD
SCHKJlnUnrGOy7A/Wu33jn3605PbtwY37uj+MFOQiZGFWJnEQkxHk3J3sDcsR6y03l8+9fTq
p59srfaLOu7fvLO7+fbkxh3bGWRSL+WcFznn3phUkSyOuBbUgjoiKlIXFZ0er2kKF6ScmapI
FBGd1b4ZkTWla4lBchMYIhMVUnVGZDDRlJtSkIKMQSAmaEymSIKUqEt2TuTYZTCOcIG9ph56
JjDzBO+YTEkVKVFGUyo0rdMDTTVDeiAwNwAKTb4FTRAq8ahGmkREKjpdDDExg6ZGQMluGwTP
apbYowFGpiBJzkfMCkokqvmsQWoQTcys8b0AKLIYK9AwKU6JRjGzpt4ghhgZShRFxQSzPCRT
qm9Icw8Cg6XOjYmxR4MZRWg0U0OAlSQlWygclrp+bTnrdYpOL/eZlvXw/s5kZxDGI5uMvSFj
33ZZbg6aiCPELAneKWOfeQdiJmYYqzIkp/7R9fWnj8ecS43qi5390Xg4qUYhI7/cX+31+ocO
r3/q0892+t061C5jU5szLldmmr/4VCVl+VJ8MD2HfPUP/o9f8Ar/UCxSewsssMACvwg+xP4A
c6IozFWBNX+amYEds3MESr1imJhTaXu6Ac/9RDfUai6ylSAwJmSAU4ghqJlzkVRN+q2iHWwy
mISyzHo9396VOmBSkZiF0PZOSAzK3aLr3EoeR4PR7p39nVs/uPVnP/RHl9c//fSRp5589p/9
I6rC4N3buzdv3n/77eH9e0VJSz7vkO+Qc8JmVkslrOIFarmSmaVuyqZmmvQ/pmQpQkNGbC5F
RIwUBAMrjMyMUkE+cnaqEMAYASokibewEYNd4x0AB3jnATYTA5RYiUCpgUhhnAU4E20yejHC
IomqRpiYJhty02Q9MK3BIzTOXclpMxErAyWP8oNwVWP3lPpPA9OwUXqlSJorGNimRXyqBuaD
DSYVVjKBAjVOCwcsGdN83fRLwAYznmbijFMaT4lgZKINhYiAikFTFLCxSABx47veiN+RIl3p
GymmohxMa1BNJJ7Vuapw5VonX+4uL/VdUYiplPXw7v24O9S9cVFKPxKbsM9c5gxJYiWcOzWE
KARk7BqLcVJONApQYgGM2OW5mKmIdzypopaG2jJQ5jxBDdJqF8wEAmccU9hV0vefiSDSFHNM
rw5ihnNZukqSRdZHv5IXWGCBBRb4W8HjiFSq0ZuVF+F9NV8A4Ji4oRwAeLpeusM6kJhGlRlv
en9Ft5lFMpA6o0LJKdQQmSzPg2istM3eQcb3dlaPHsp7vVFVSQxs5JQ1iDMisKgIQwrrtHvd
2F4bDMaj4fD6nZ2f3rjbbbljh5Y//czac8+snfzSUXd2vL29+9a7+5s/2711Lx+O2qI97/M8
R6ydJybEOqhICsYkrY1CTJXUHOBggdgOdNsMWDLNBIOnwaHETRwBBgaJsZqlmIORRUdJe5XS
nICqMcgkWQY4py7XLCfvVdmcgA1BDUTmRLSJj7lki0AkzNOgDxOriZnSlCcBsGnww7TJ6CVp
+TSYY0Dqi2JEsNQSrunWAkCapj6cLCMa+4VGjGXGUwqVzBsk0SIghaIaKkWKaWCMVKdZQTIz
NQFzI42CGpmRKpRgRNw0/VE1YsDARNQI9BGFiCLzxOKItPQci0LaBXc7Wb+bd3r9drGU+1BV
9WA0vnm/2tnHpCoMBcgbkePm1KfG1wR2zcGRae4JTMmPK80BxmZQcgISR5RnrsiFTNnAjBqF
sRopU5G7PKc8p04v5wzsjIxUheETl02fuVljxja9NBhASrGqahLlf7yreYEFFlhggU8cjyNS
M/KUUk7zqb1mhcQiAFWjWSE9aBZ5agY0FevTfNu+GVL3DjUNKhBygGM2EBvlBlcHDtJmP9zb
a/U7rW6v2t2jooi1kHPURGQIIGWedP1IzTO5frtbZL24tFrWo7re2rx9+8fvbvV+iLWl/Jkj
x049u37quSe/+KuxnOzeur3z7q333r3FW4N8wFkdOmQMhWdmMtMYo5oasaboh5pRzMxyS41R
yIgBJwQ1J01Si4mpco2se+rIRACIG90YNwJpIwM1RlpIOTOlad5NdabHNqRiNGJL3CQJtGBg
eh+5/ZuDEYRSqKnRiuvUA7ORe4NZp18SMxjYTNGwKDKYWU0qTpDKHU2dwic3cmFTg0KURs5q
EkCZU/DHqQJqKpaUXLWagmLmJhkP86xeKnS5i24n63SydkuCbO8P6zv37f6+TiqtQ6bUzfLC
tagOpEYyU/55kDVkb/phUfovmhKI5sgOdO5EBJd0bSBHToAooSxLMHvvRCMx50XusgyEKtQR
5pwjYW3a6tHMVWSW9U5I0v5ml5+sj9QCCyywwAI/Bx7BbA6WTfF+CpVgUxPpeV3U7MYwjXzA
ez8Tz86UUjPw9H/GpAxlJueYiAFn5IiIWAEzGQ/2TWK6/zrviBmOmyyPgYk8ZY48OCPnldnA
vVb7SKd3auXw88uHnwn+0I0d+3//8q3/6z/84NV/92f/93/42Y9+jKL9zG/+xhd+/38+9b/9
6yf/1b/I/oez+yefGqwtD4piJKS170qxJu3V2Opbq0U5+1zzlmQkTowiSBnqoC1BW6wXqBfc
Us29wHlkp+SEM+FMKBcqhPKIVqS2UDtSofAKZ8ZqbMpqbHBQMiM1UmtSp0SNDAvTfnEgAxsY
cJZk7p8IFClM1MRJDsjTnGbcAOGDf+YgDCVKPgcGMpA5E2/qTZ2ZJ/OkTJFICEqscAr2wnlk
DohB6yhjtQGwQ7Ttaa+Vj5f746NrW6eOb33uqfD5Z1ufP7n03Kf664e5tvLde/d+8JP73/3L
8gfX/fXbK3v1kZqfQOsIFb0aeRnySLlQRq5wWcsXucscyIE8iNNptelpNfAcY8XUsiG9dJnP
fMbsTKka1+OyrCUGiSEENbBz7U4nLwpyzpp6Sp5K2BqzA+eczZXEztOpxLTcJ90j5vJLzUX5
0uUPHAE2L549uHzPXnzIpvLxSx+xv0du+Zc1OJv/B0zpgeVzm3p42UOLFlhggQUO8LiI1OyX
/TGPyGZGzqV8SLoduAeL+2Ag1zx507Ss76GNOAUTOSYGE5EyABLVQJoXvjTUVJLScH+v6Fu7
XdShRu5V41TyDBA54X4AYir58+ZI8lCLqMFA5LNuJ1ut28+MyrKsB/fr/btv7fzozbudXPrd
4tjhtWefXjlxfP03frXz219ELCf3tyc3743fvbN7ewvbAx5MspIzszYxM1V5O7haVVVUxUgs
xWFS/C3tkWHZ9M5oaOwklVOsg42SPadna2rtUqe7qTLbkmootYvBgcho/hXNvf4kwIC3plgO
UyLVeEdMJ6as6iw1bWnUUtSkOWka8nFq7dqIWEHJU6oyjWpiqiBxiEANjkzRrHakmY9FXnuW
dqFFrkXu2y0pfJVTFSoe1dnuSLeHujPIK+lGeEVm5DgnpiBRG3OtFMpj8o6IQWyc/E2jm36v
bdryCI1gv2m2DSMlIoOmtB+JsfM+c1lGcBLjeFiV40qVQwhmurbW63R67VanyFowct7DLKoU
LuNpecd8IGquDsNmMsS/VUz7o2xeu4ozZ85cOTCEnDoWpabCmxfPbmzQ6zNfoscvfQDJ9fvB
ka8mG/CNi2c3zp09ff27L5/8RQcBzJu1X36Jzl3YOIvmDfO+45sXz26co6vTVTcvnp05Ll1+
ic49sMEFFlhggTl8JCKVQFOnqPmoEjtOzkPTwq4D086UzbLpe2kuUUhz/uZqClWGJvufSCYA
EQmMHFek5jJ2ysNoZTXi4Xp/iUJQUExZEG1E1s6QmRIoKAksMsxlBh8ia6wlBInRuVj0xff8
cvBLdRYmoRrG8d7e8Mb2rR/+5N124Zf7fGg5f+LQkadOrJ58dv3XPuezjMpyfOtuuHlPb+2M
bm/F7YGMJxY9AGIjVrhoMIKqKVlsYhZKEIgqGE0lIExNDACLkbFyFgGlRDdBlqy5CZzsJokd
VMDJYCuSRZVk+q3JKEkTzbLmDGMq55812Zv7LPH+kz9Pi2d3dzRqt0ZHPlOlU/L/VKKk1Gos
q5q0bWKBouJUCwpoPEAJRqompkxMKa+p6tRI2IBoWsFqsglQEWqHyKSOhP2o5UM793lhmVPv
UWSS5eMYWstLq0eO7o6H717fxI27vVpXW92ukg3qQnIHJTY1FdYIhRoRfBJ8OWL2YCcE0aTm
I1GDaTbrxJz64ZkZEQOUfCASoyKa2p2mc+GyLAPIjIaD8f7uoI4AXCirLM86RcsRxzrEus5a
OQHMhGhgMDet9JL9RAr3TnN8qZhDvfdmliwSfs7L+hfBmfPncanps3v99Stnnj9/YKx9+aVz
l3DmlX87ZRRNw92mO8vjl85jWos/j83vfOsKznxlY64N8ssv4xcbfJj4JMv01Hvl8jcvXJkz
A5g2vGtw/fUrM8elF141e/UXP7ELLLDA/0/x4T5SD+EhdiUqAIHIsXPOKYvKgV1CE2V5MHIy
vwVr9M4NIVCeam5EnRqn3mTORzKnwirDwb6HdfKCRKUqlRjExAw1ZZnkBpgIRCHJmgoUzZg5
b7U11OMoQxYTKZgK77Iib4WsHWxVrI6xGodq7154627gt27mf3695+TokntivXP80PqJo2tf
eLb/m59ruYyGZbw3qLb29re2Jrs7493dam8Hk5Kq0oVYEDmABC1zedKQEQzJYdTETJHq/EjB
wSxltKZnIFEgIQAkJqKEqdGCuSZFZjAxqKk2CvIpDcA0JmZNA5k5N4p0uucG56IgiS2kMsuD
woK02eQPNotFsjZ25ERQsqRI12TCRVAiNsoCbKqUTt3wJBUZQkRVgZJpkkHYxHF0HByHjCXz
6Z95NueQ5UQ+mmrUqqwmu6N6Uq/2l3/rs79xaO3QH//xfx5c31ofx1Y01ylBVEqcMAULyuY8
GMTkPFGmbKLRVJMFFSmIja1JNiKCGp+q5gs5FUsZzY2CyTj5OoAM7EDms4zIVZN6f3c4GZXE
OQBH3G11W3nLGb/zs3d/8Od/8U//2e+tHjkEM09eYqQ5wxEimu9vmB4rAIsxvv9C+wRx+vTz
+Na1TbxwcvPaVTz/ldNXp0suf/sSHmowkrrDpE67j136wkNd886/Zl+7Nh+UStTl1Ek0TWYu
vX4dwC82+JgI0uVvf0BHujl8zE7ICyywwH+b+PBee/NIUqf0mpJGGs1Pvs7iQ1MkE4QmevGA
nFZn928AZiB2xDCQkJmaE8uNOWgLVO7tD8pqpdvxMIWJ6mB/0FrxzQyYjIidh7FAa5JkAGWS
LJvIO5d3sqosg9ScZW3XaVsWEaLFEjrSCFZfsDfy4rvCfS00SAwxQsq9arI7HF+/MXH0prNr
ncKvLnUPrfaPrGdPHM0+tbL6/PFjWe5FuQ46HJe7++Ot3XJ7Z7i7U+0Miu2xL2uJkVRMiQis
IGMy9Sl8R0S5U1NOGvbkwqSEVEGWCtaiNqxGFCIkgQ1MpmRGZlBDalMzve2nJOs0U/VAFHB6
xulBQ+3pR0mEhn+ZHmyqGbfmPUhOAIn6wYQgDIGJaYQmTyciUuJoJIQAig6RKRIrkzk2x8OW
n3Q8eU/MCguqygBRVKtDqOpKQmiNNJtoqKpYhdy5w0V7qdVeK3nwh3968+4Wbe9+NvPIfYUY
YlXDaqgRU0ZJ3pSZy8BMZI4DEDXVHRLUnIdjZgNUk+Xl7Ks5zeNhan7ajB9YVzWUE+SIMs/e
hUlZjisV5MwSlUG5yzzYg6rB6Pt/8qfe+XP/4z/v9LuTUM+7h6Qz/1Cam4iTydTscnvM5fk3
h1Onz6SgzfXXr5w5/XVMidTmtasAnj/1AD1JnfWuXtvcxOOW4gE+8sKr9gKweW1uKG18HrNt
/tyDeJgEXf7mhSbQtHnx4fUfxAuvvnb+Uup29xFaryywwAL/LeNDWsQ8NDIv7ABSjRZbU87d
yGZxYGXZ1KIBB/bm82r02e1EGUIAAwZSeEFbsUQt3RtuXbvRyfK1Z/qBeMQo8nw8HNZ13W93
YllqHZB6gjCZoFBNLpgWSRSm8N63eh1r5RImlQmPsDRx0EJYI0mgGDlEiUGkTslAhrajdIIT
7UVaC0SBELgax8n+qLo73v/rd3YtDjp57LfzdqfV7XTX1nqHDnXX14qVlc5nTyx1u8d9RkRu
MtTRYDwYjHb3xjt7cTCYbO/YcCLjiUzKWFVUhbbWTpXMvCmZOYDZmSoTZezUYBpZNREYkEy1
U0iBquY0AnEWkUpxjsbWqdEyEagJutCB4IqadzQhQwasKbic5v6aXslTdmFkZMGiEMQgMCES
kBCCIbIZwZhjlk18EZmUSZiD4+A5eh8cR8cRFpEsGBQiWgYra61qK2uuAgdFiEUUilIotdg5
58DkgQLC1fD+zj3E4A3kpCYuBcbsgLZxz5gqMMGS5SkBDIULhmjGzhM71Siq3iwjcoCJQk3J
ajI9CD7NZOVT33QksVQyJ1UjJgY7dp4USNbvElRUABR55kFSB8nyajxZ7vd/+P0fdHvd3/kn
v8tEMo0Fzi4Ba5hqs59U11HX9ft9zz9JbJxKIamNa1fx/IsfqW3wR8BDubNPEFcuTFsIY9bQ
97EieABNQu/yS3Tu0oUNurBoULfAAgt8EB5vyHkgtGma4BHNXLObW7EagXhazq0iSpoyTOnx
vYlOAcQ09dQ2oOmskp71Yx1dK4+qzlA4bxQLde3t6t6Prp0oivXDx2Il2XIvI6I6OuPx/rDn
i3ZRxKqyPM98Ecs6xNobs5k3ECjCKrKJRclBayuFXx2P9zHMy/3aJpVF8ebaWUaSm8RQ18Ki
KlFiFJHEahxHAjtmhWu12iHkImudpRgkVrWOy0kclGYTfvMuU+ldbBXSaVG301pd7qwst9aX
2v3Oyspy5+kTRz+zkTPn3jNxHaSajKvRWMfjeP8+JmU5HpeDYT0YaVVrGeKk0iik0FpiKFkr
ajy/AbOMHKe7MJk5ZwSngEUwRaiYMJOJOkdkCBKNiYgdpjWADmamZpGEALjEmVSVIrUUPlk/
KZGQRYIm2wXPRBwdVZ4jk1LivqyO4DmgsTcwQnBUEUU1FUkGrSIxlpWIxhCjhBhjVoW8FguR
gzpDZsiVcqUMKOByl7uWC6yBRCQCAImWwyCRQYIYCK5gwAo4NiZVNnXJmQlUqymTeU76f+cy
okxUvPcaoGXpjChZuUOV1Wg+JjWf4pvjVgRNxlzcuG9lPmNAJVaxLmNVSa1iLZ9n3mfM3kjK
+r13bqx1lrb3dr/3h3/8xPrRz33h85ERJDrnRATT3DfmFIQqSk2skvHYrgN/s9hIIalTr185
/+ILuPaNZnguvHQQoJmFqU7icUs/dJ9p4/Mjs23+3IPAdQCPDCm9f3ePxAuvmr26efHsxoVL
37j4tRcWgakFFljg/Xis2Bwy+yVPJdyNqeNMiQMyUQJp8qM0M4CdO9DfUNOetlEsJ5/BZEmF
RmPLRLkvRKwWKbx3AINpNLn70zdbe+NnN469++6N7Xr85G/8mi96Ug06eTsMhyO3v7K0lBdF
9Nbu9qSttcaqKn2EV3gGoEZWs5ZaHTp87PAzT+f724P798L9LR1NZFxxWWNcOouZgbxreU9A
HaOFFgtFiUFijaisSsJkIIZaZAXUAbkiN79MrCBRi4ZQ1eWkrO9thbfejsC2czVzdIRWzt0O
ddpZv+d6nVZ/qbu00uv1fLfFJz/lc9/K8j675PHAZhai1CHWdV0F1JVVk9F4VKsodLy3X/9/
7L1JkB3ZlSV2h/fc/f8fcyAwJ5ADALKSWSSrxFazkDKT1F1aZFIy0UwS1TJJRpnJDLlkbmrH
hRbccaHMJXMjKzMtZLlim6kdk4wAACAASURBVKwyZSZ1S91tCatikU2yiByBxJBAAog54g8+
vPfuvVr4j4/AFAASHItxVsAP9+dvcvfr9557bhNcUokxpRBUgoivom9EECJoFRoSIzMSA7Wg
Ap4VwYK2/hAiAgRRMQQjQGxT68yQEnlBVjRFMiYhqFUjqKApkmBL8AIxaIsZi2oylRhV1aJI
jJIUU+QYMCmkiDFxFE7qY8qTelEHiGaM4JgZiZkZqa1hTIgmamqmIYkmNUGVJGiWu4zMRNo0
RQcAKIDIqhBVkqaI0AYNjRCdJ2JqBaLYiXPO+7Iu0zhxUkxUzBAhEURUBGO947sDo1bvQBEn
xpTh2GGIRK3R6JlNTWOKMYYYFIwYVU1ECKnX6a6trm2srPWmppZmD6yurf1/7/4/s73pIy8+
H1JbBga9c8ysKWlbO4jQVGSc7zhO5PidsaROfes7Z9/+5E146+yLfwVwJwD3yrfPwVtv3UXj
vvQ3b09Y2bv+eunNl0+//Z2Lfw1vnwc4++Jje7UuTLhZ7VkXn+7HPdCO5bFIUC3T6/wjSFf7
2Mc+/lixd9YewF0aUXf+MvlHm4UEBm0WEuwqdbyL9jwO/E0O2NHmMWImIlSV0MwURVChkI43
2cY/XEqrW0dPn+iv3Bx8/FE+XeDg+XxxLuVOIhlhWVVmmrkMJVZlVWR53s0lcSs+YNqKMxmj
QJRmVBnq4tHDcwfntRwNNjaHa5vNVl/6IygDNJHqKHUAEWbOyFECl5DRxixlM1WRcTactHJI
CgpgbUYdKeTIBfkeowIYoSIkixEkGYQ6pno7bWxH1SamkULFbtWgQhtN5dERZd53ct/tcJG7
bu66ORU5OXZ5RkWevMMDHSMDxPyZRVQBAS/CpmTKIhLINI8pRTBp6hClYJaQUoiqKgSJMASN
QdVUTVU1pQgIIhJTEBE1M42kQ0tVTDG2xfMEmqZJMs4vExVsNKuFUpt1pigKohCSV2AxFkUF
VvVmhEAIDMQAjEzG7MB7dkRmmkCFzcw0iYQgmlrxdxpTvRARCmJ0GfqxnFJKqQnBWsmI1jdH
1oAIqRkQtFV+HfsMiRVJAYmYvHOdgjNvBFVVJhEwUFVqvwUAmB5kqtiYUL+bUgaILXEJkZjY
k0PF2EQN4oEJgdlp3OE2Obd8e7lpGlUriu7S0oFr16793d/+7X98eL43N5NCIERkNjMiNgMR
ARG4W/FzEvj+XeDUmZfOv/r6eTj3zt2WQ8scev30a2d2NAO+uyv1bfzX7775rfe+973vn3v9
1dOnASb883vI5vehzaj74CJAm4D3xrdOAZx6uh/3Qnu9SUphK4aw47xq0wrH4bzdxuI+9rGP
fdyLR8gfTOriTfjj7Z8mmfOTqAQijt+4OxbV7sPGR+5oe2JL5Nm5ipI6D9SEgvxCcvWFa9nN
7WNHDpXV1vrlDxerYQz94a2b+XQHEMAxZz7UIW2HuZlZBqzLMoRACD7LzQAkWTKHkIuhYhVl
tN2vRoNs2seCqTPVnet1nzmSRtVwbb1e344b/bDZ5yrkCWIdmqpmVjCxJGji1CBJioHA0GFS
SaDiUIlUhFRNxMTAEiEROSQCQ0TzED1oy4hvJTTRsaI30R1hR4x9SCpRQ2P9ABrRIllEqEAj
GhJWeV72esdPPX/k2ZPgefv2559+8FGqKgYkZmNMaDX6CnzWLYywqmtLQmIkikmRKYIFTWgR
LJpqS5rWFFVFYhJJbU4exJDHQEk0KUlbDAc7CtaaR4ZggKJ5MqdjOXuGVo4JHJJDdugdojlU
3pGYahMyzZKImphKkqAioqIgrbBARsTOtxuFiBiJmIgYFYicYUvGsggmoAIt+6tNuEPsECOC
AINjdA49cqbMznklKptQlxVq8ilLbdE6BCZqa7q0xQJBUR8iDH8XjY/QdvyxCEjoCJ1Eqeso
dXBCjASKvsizPGfHw+FgfX2dkVRke2vr0OHDiwuL7//qwuLJI9/4i286x0QkoirJO49MhCDa
VpohoHEVgVZq5Ivd1U+P0y+eBTj/AK/OKz+yiy++fPpVHKsXTESjdv91FyXpCXDqe++98wG2
LZ97Z9zqU/74iOvZmddwMpTdNKhXfnTxjQunX989zH2G1D72sY8HY6+v3vd++h4z40654km2
HdxtP+1+5dyh0O4cNrHACNuSv+OkJE88/uwGCxwZZarEeeima2ujX1w6OT/fm8mvfPzv3eba
dDUMJhtf/Wr2J3+SeYci2oQ4rLxoL8sKn0eVOgUAc6zWBKgCR3FJNUlirAqq5zoHvna6OHmk
QgVNqgYIDOTAOFnqj8qV9WZtS/tlGlWpHFloqIlYBhxWWNZQNdoEz5QVWdIURBJBADURFDQ1
FUsihMzkAEnHeW4BW3NhHApFEeF21GKAwORIGWHshbE23Ikoaq1nQ0WGiP08P3zimYPPncTc
1+trH/3yQj0YtApIAhBII6Hv9r768jd7hw7V6+v9lZVPP/pkc3mVDRcOLJahufrZ1RnW2YwB
gIkdIZiFptYYESBzDgGDiJg3ZDRsRb3BlI1QjY0IkYxAlU3RcKzEgKAIgqBoAqCmBpjAAoiY
iqiYSBvMtXbSARkRoDArEJnYOYdMSNRy52As4D5OUUCkaBZBI2gZm6gyFghHYiIkBFYRMQUC
z5QDevB5ACiTjFKoUlJURmAm513GLGVJIeVqTscVbhKqjHUQAHZCe2MdApzoyiI5VgIxRWZw
5LpFZ3pa2ZVN6JfVqI7kfDHdzbpFRm5pYbEajn7yt3836g+J2Pts8cBit9u7cv3qzNGl/+pf
/DeHjhwBppBikoTMnrn9CGmzNIDuSuj7H//n3zg/+zft+nr3NfzBvpjlPvaxj3+8eETW3sQw
agNzuwU5W9iuHPvWQmqDfbstrfGROm6kPRLVnHM4DuaQmOTkOmujlcufFTN559DM1qef+JVN
TzFQSAwhlD40QaPznjIHCCoiTVQgdkSETQxgxkjoWVVDMkKQmIwIRcrRKNNEjsDQTAXASIMZ
sfmFqZn5Kff88dAflVvbg+3NcnMrjmpXBjesZb1Pw0qHZR1C0mQqKSUG6AGZISikZEk0I4fk
ACmKABgyq7a1QdoiL6CmhhzaZDhGVRWJzGxqaEhIbIQGjAxqaMhAaNAB9HU9+uTjSx99oKBE
NAO4AGNzw5gikSCyGF66LLdX5rrFCweWXvgyX1aqyvLEoUO31lbLOhyfPXCgM9NqnzKiiUQK
wAkMSMFUo0pDoG0ywU4pEzMDT2PhTbVo0JAJmplFU1EV0KASW9EGhGRmoqw7BggiOXRIhB6M
gICIHZNXZRFCAgQFi6JICITSbpA23ZPZ2IRQkCOg5l6FCJCJPBIBOENWQfRAHJWQ8mHUzX5/
EEMDZnkOGYOBN2MclwgWVVC1VmxirNylbUfv2flm4wSKXYqlrefKiBmRyqoxr8NRMyqbqJB7
7k5NK6ogAOHaxnpdN+0NoDGV/eGB+cUjSwev3rr9i5/+7J/95V9S7g3Me9/SzgiRmZOlJEmj
tm7g3Q7gP2i0Ugf72Mc+9vGPFY8I7e12OO0ON9xjTt1jM00OvutIRB7zQkhVySZuLSuUM0MX
0tbV690QD586OdpcHd7+vFtVsSujjIIjn+eFdwE1aMwp48xRVFSQJiIwkTpGDQkQiRkyUFVU
VVNNpk2MwzJVNXRzMG0LqKkZmiJjQq1DYIb8YK97cKojh2PdxP5ItgZhdbO6vdasbdmggLIO
IaamRo/dKmEVQZIpeqSCvCmgoRFlY9ElSsCtc6VlUKuKoiUzxdRmPrKCT2ImBmBIoc2rQ05J
DYDZKQDFVKRWV2osM65gMSm01C0mRVPMo6abW7URUua7RZGRw4agoZtXb29s91MNZQ/6CqCW
JJmYpkTQqoUatLQjVRK11ocEpohALGAJLakmMzVtyEYMSkCAqIbWsreVEBmRmQHMCNi1xaqB
kMZ1WABQDYDAABICmjAGEwEzQsw4tUFHJiQyJkAQUnBkjJVJNAQuut53nC9c5gW0DlLVVCfH
WZ10fWvQWKgBt5oQEBOBhEoDkAghzc3NMLsUg8aUjTViDcZ6UQj4uN4YRDYkQxIDA2hC6JdV
VCCXcVYoQhWaqW4vStrY2EwpEYAjlijVqJQmHDt0ZGO0/dGFD86cPvPCl89UdenyzAAFzFTa
kOvum+ieW2wf+9jHPvbx+4lHG1Ltv1uLKqVEu3hOEy8UAEzYVKp3qsTAzvsAd/6P4+gMgo5j
OAg4VeOsuNHyZn978/mD83ms+pcv++EIGQQ4ZrlM96ZnZy0m8KQIioAu886oSRqDAVlB7Agj
q1oiAMeWMapa6+xIIsOSqsZNFbWNyVlsxqaWQEE9IyAES1GSGbki57woFhdmjh+1skrbw3Jt
Y2t5pd4aSFU3gwH0614jICk1Yk2yIEyeAcAUCQmpze+CsbKjEqghphSjipoaqFIr09SWb0Pg
1rBBJVZIAhAQkgEjtEZTVE0moIqiKIpmwGhASuAjYNkYs3l2mS/L0K8bFEuSmhQblWo42vSI
HFVSiIGQHBGN5SmAiQBUQFUtiYilaNrG9xKCWFur2IDBkDwyATsgR+CMmMAxglpb6URNBU0Z
bBzSBUM0NQNoKxGqmQIEwujB2CGjMRmRERqhMQEjMAECFZB3M+7kORMyZXmeE3NSGDVpa1jX
QaROhuv9zbVBuVHV1JsVn9eM7HxhQKAd53LnVMUBa1ICZCRsSeTUqmtBq/Vw/85Hwjs7GMZS
7mMyGYAYAGAZwrCuuegUnU42NaVEQRIR9QeDzc3NducToiJJShsbG9Oz00eXDn1y7fKnFy8e
f/YEmqWUDNupGTv94K7cjn1Dah/72Mc+/gDwuIKcagrjWmDj39s4XatxcA/NYreBdSect6N6
OCZOtYxaQwZc1KJYHd24eMN183yhGz+63L29Dkn6DoMp+6nO0mGX+Vu3b3aWFovZGVXLvHOc
CMTEjJIoMbte0S3rutYIzjjPGMnM1MSShP4oDkZufio5MzNWYwVSQxsnrkcTRfAIRiyCiiBE
VHjICHp559Bc70snLcQwqjaXl5vl5TAq61EZhzXWKRPiBNCI1QmSopqpihoCsBqrkSayZBpB
IoogKKIZkKEDNTAlxzmzImrCDheqEJMgOe8JMktgETRqUhEy822OJJgyJrQczVtSpwlEMYAx
cpIU0aFjRUe9WS7YOsjkXURGMHauDbXamCBECaS2hMzOM7dBLCTBVjCBbKwfxiQ8Fldto2Ng
RiDOFFEZk6KpODUE01aMilEMlMCYW9UrNYicxCVFREecefBM3lPmfJHnvW7R67gs4zyRE3OU
TOsYNEQR01JUG9EGIDlHoyRbKW6kZhuUQEJTMvpCtUCe4WyaMudcyDiZNq21jbij9o6GZm0o
E+6S3YexSOld2RI22bc77qwoUoUYJOVE4DjrdpRCe/L6xkbZVM45S8nUCAGd629vb29uZYX3
SJ98+NHpF7986JnjQUWptesmqvLjW2ZyHz35Hb2PfexjH/v4rWJPQwrJxoXIDA3NlJHakmUT
2ezJL2O6bpuZDgiA1goRoKlDFGVDBlQAQRU0IcuJGMxHm4q6eu0zkXRy6Rk3rEa3VgqLwUbC
MMydTXen5w7wMNj1W6Im7DDvoCvEKbKxGluykDSJZp6dd9FSiITm0XG3p2hV2bckMqwxCHuf
LCgmZSUwEuyvbJfDOqSE3h06fjRkZpkUoSKxSC74qWi5x0Rcqq/dtD98+ISkI7FupKrTYFRu
9OutUeiXMgpQJY5GETAkjIKmIhpixIAuklUGQVUSICCRqnpUUTEzUEImNmIgAlSAzHlHjjMf
wAjUm0nr1jEDGctztqVa2AkVSo7aHDckBIAYgqqJmSJMdbs553Pz84SU6kZCQAUEVRRDAAJR
RQ1EgIimigiIHNTEzJAcewSzqIgkulMcBkDQBCEhBrREao6MmA1FxJjEsTAbozAJmDAIgzg0
B5plruOns85UVnQ7Hd/NtfCWOcqYjXIDCyFRHSDVoUqWVFRJggKiQCvtCpkHzi3NOJEuUlNX
IXig6U7RJe6xm2KfESmaiiqaQ4wirUC8mjKM6fStydSKxbfhZwCzXS4qHafpgaEKkjIYI4Kh
mgs6xRkT53nGha9HQ8c0GPY3NtdFEioiQpDoiJEwatoaDqZ5Ns+6G6ubn3169eDhI2DCGbUq
n219ytaq3p23sY997GMf+/g9x16G1Fg/BwCJafyGbT0RQETseOx5upd9bhO27FgRhwDE2MDj
WBQbmUAFFJxA16jZ2NjYWFk4dnDGdXR5OYXYuKSk7CB0yR+YoaxI125PbQ1gqpeKgudJ8hyQ
mRmFwYSTisowVp2p6dmsqEejWFdqigTKqGCgFoYllpE8AwixIauqSrRPP71x6/P12fmDkOc0
A3OHZ52O8iZmuW2yVFwYO5+ULYALlTbBPDBpx7liqjPb6x05AKapiWFUp2GTBnXYKtNWGTf7
FpOKacpYjISoCtiEVI1Mk4KlutYkmKEABEhEBOwQmRyCIROnZESaTHYy2QAmccC2gq4KAiow
oDNt1RcQkFSVyBND3pLS0mo0hYxBDdBARUJiUDBDJERSHEvPc5tAp0DMghpUElkEJSQmBoDI
kNAUQQmVSRmViZgdoxEKmKAmVGGOjs07ynKX+Y5nX2R+Oueeh5zFIbPrcVYogJmQNR4jSSPR
acu9DoJRQBGUVE0TJgl14mjeCM3H0Ogguib1oiK6Iu9FhcznnazwOyE8tWRAzrkmpVYKjBEJ
oVUGAwBAQzUDoDs2FLSs+bsk0xARjcAMVJHGzKooGFKXnc+KvNMxsNQEj7C1sTHa3kY1UXVI
aorIAipko6bq6lyvmB6Oyts3bleDYd7rSBQjFSAFEzUwdbs8wfigsOM+9rGPfezj9wp7pQW1
UgWTuAYRtUJQ7Uu9rQkzPnSnzutujjnsMKXQgMavIzTE1qXlkDBJBliQu3XrVu794vwshmZr
fcVMUkrEHAHQu5mZGQhNtb1ZsHVQcolYVxgrBwpkSqYEgjgmITWBEHrdTifPzSw0oRwOyRBE
R/1BU468qlfNACkRRI8wfeDwmUPPfT0//KUrlX/nF1c/vFbJaDbDxZRYMbGrOtmo60eZDrxF
UiTIM8mL5ChREogIFUgq2C/0po7NL71w5NhXTp742vNHzhxfOnFo7vD81HzPdbyyioeYgfVy
munydMcvzBSHl/yRxezwYnZowS/NucUZmu/ZVC5dHzJqctKMJ5M/WRFAHOs0AcAuPs1u1YkJ
1QYAiLANsDIRMSM7MY1jGW1r1TUB0ABFwQAFoDFNjpoMmw4Pc6yn8jDfHc3m9cGpcHC6WZpq
lnrNYq9e6Axns+Gsrxc6cmjGPXMge/5o58Vn5//sS8e++acnvvn1E9/46nN//qcv/NlXn/vK
l489+/ziwaNTM4tznZlZ32GihqByVrEmSAbCpOJkk5qNQgZOGkvRksSIdaIqZVXKay0iZEG5
iVJVVjeZwhS5hayz1JtZyHuFOadEQkQ+K7p5b4qKnLwDJiBs69iQ4Y7Ew6TU85gsZdhqIdxj
vrSOop20PQNRC2qJED1nReYdh6pCUa1CudlvRhWaoQEBMKLuJEDGEMBsZmoaAZaXb68ur7TK
saIimlqKP+wUrPwdeaTefQ3vwctvProo3UPa+WKn3sGlN19GxNfefapG4AGDevom7+3eF+pq
e9LTzvavZaofhl/XEvyu8Mj+/0Zn77ePh433D30d/wDwCLL5PS/pSXbeRBDh/o9mRNxd5R4R
zNTM1AwVBMEMNErGLlPssUubg7I/PHnwQI4wWlsZbqz0pAHVGCVmwN2O91zf3oRqNF1kdVNi
3UHnkB27AkmNwBQEUIkIsCmHEqtOXuSZ81g0taYmkoFTkCo0/bI3P+UcQK3D7YEGLvKimy12
p2TA3s3D1WtXPvvJhW88d/xreSzKm9mULZ48CowekuQSU52Za1Y3RLA33RUHAlEJogZI6o2c
IhuCRbHQyTgSmkhO5Lpdl0OqYlOWKhFMVSQhVGgJTc0MyPnMuSxFlZgy9qA4GAwsSU6Eavrw
N+tOlHXHgbGTa5kk7VhgqKKmgJ6ZmYvcJKUmIiIStjRzAGgr+SIhICRGnC6mZ6c2UxCGkGUB
UBlT5oCIveM8yzKPnjnPfJHnnYIzT8zqTHJISAKggIQMSiEptkX3wMZEtBQDaHJgBGRCklxU
RE2spSVxQBFQzEWjWqFKVicKRhGgimlQQYjODAAc8FjBgEgVwMg5T94nxtpSHeJAQlSxNkSN
1CobtLwnMpMdYU4wVGzlzHdP7J2ae6aAhGikCgmsFk1M5FwCpRRjaDKDUDcQUi8vmroR1dbQ
VTVgQMAUk4RQdDqdPC/7g9WV28eePaYqbcaiIgHSPffS5D76reJOZbp3X8NXXz/9MvwjEIGa
aG1eevPl068iPKoC8aU3X/7hmfd+Cxqc+7WQ9/FHjt/avfabvPSeob1dgPvysVNKE6mbe3wh
7Yu85aGDKaiagSoAmYIhMBt4ww5wJ8LKjZUMqZjq5Yb927co1iGWXbAoJs7xVE9NQ3/bp8YD
1eXIdbt5t4sSvMuEQMgUQRVN0SEqWairVFXdTjGVF67oEmGqRqhgycKw8mXIpwoTCdtNM4pD
jWtrzVB9mJru5P7w4uzl/tqv1m50muq54TYM+8NPhzbf6RzqLBwtOuysP+p/dOXWcHv+9IkD
xw9LhhFF2VgABTRak1RBgSxHldhgWbrGOuBJMI6qut9Pmtr8xsRYeaQiDyHUdT2TFTPz8zHE
7Y2tOgXPmaIZwTjo9CCxeAAwANqpfbuzRmCq0Iqg7qxOq/cIikjsMjYx02Fb7lAktvaCjaXC
WMiEATIfHNXez504OrU4J4i+KMyzMZJjdo6cA8IEqXUxRtMgSS2KpQSQQI1IgQ2JGABU0RKC
gIBGdiFCimSMxE2S9c3R7TUPOHf44PRct1aDyFa7uF3rVqPD2qpIyKSYqkbrgEnELImgwwSs
TC7zBGxKjVrVlJWmBjWiAZO2wTKhVkGKcEd/s50waAVU2yAeGNKYrjS5C2xH0twQFIAwIVQx
NpoMDEPg0DhRbzgqazKY6vZiHcaCWC2TzQiJQDWUtUMqnBs05db6RmqCufHSqmlr1+5mmv+u
Q3ttEZXfVY25U997z379aqSnvvf9c6+/uncF4rZazLl3fgfd+/3CH8MY/xjwe7yOj3Wv/QFc
eq/Q3kS2YOJ/krFU9VjRYLcUwsTeEpFJlRhTNTUGcu1LHRGJFAwAOWoX2FUxrm/Pz8zneRHK
Ub22kmECjAZqxMll2dy0glpZMoJIimXV9LexaWQ00lh7T4qmYGrAyIzApoyIZHU12trYkBBz
cl2XZYCcTIaN1UaBMsgOzi4W4JcvXWlWV+YsHbBwmJsvzWaHtOwNN+Pa2tqttc9XR3/7i8/+
/ue3Lvxy+eIvbq5/dGv4/tXDVTpVTE/XKa/CjFHXKBPAqBq1UR2hbXne6vpRjtj1vbne1HS3
auqLlz/9yT/8/FdXP71dDq/3t9+/fvWXH3/8wYUPP3r/4yuXrt68cfPWreWVWysapNfpSkyO
GNUYsa0rjIjMzG1pwvFcI43rqdyxliYC9K0kfbtSRGSqKSU1ReKkRkzIzojETBVAqWUGEZKq
AVECi6aVJen6xReeWTjz3NyZk93jB4ulhWJhlqe6mrlAUmlTpVhLaDREjYpGYD5JFyAHcylQ
qiGVICVAQE7IUTFEqNQF4ESWugh5k9LyFq6OOlvS3ZBiudFr25sfrXz2y8+u/OzytV9cu/nB
zfWrG9s3++VGWfXrumySSBRJTI33TZENHS3H+vN6dG2w+Xm1vRrLIUnwGBkVAZmRSE0dMRiY
2Ji1D2hICmjoAFkRFWnidLU7UTYFREMwbZW3KCYbSarBSglJIoaYq8WyKstSRLCNEhIqghEY
IRqgGiGm0ISqRFVNaWtjfTjsqwiiMWO7XLRTamm3pO3vC+4KRe0Kh+z+/b7oQRtXG//8yBZe
fu21B8fLxiGYN1+77zoPPncvvPLtcwDnP7i4u4O7e9TWDwR469Vxaw/u9t5hlLsbfPkJ40dP
PNVvf/ehK7AzxvH8PGT2Hnzm4y3BQ7s+OWSPc++f/z3w8IE8YiR3L9a7r91z0N/ssa/u9OwR
jdx72p3hPGCMd5+9+38PnJB24C+/POnjw3bIg/Hk6/jg9n8Ne+DOHYGvvXv/vfaYw3/k1fbq
f9vOyy/fe+kn3bp3YS9DatIq3K0TqKoiAnfrI0xwTwQK2wQpAEAwamWTjBA8kEuWNoauSrNT
U96xrKzkEpwGScEAjJA6heVZSgFDQ54jYUjSjKpqq98Mhk1ZmggREBK1uWcpAqhjZCITbZpq
1O83w5LEnAJFgSpAmbyQM+wW3Z73o82Njds36rXbB1gPaJitBl9O8PWaDwmWphdj/WFKV0v4
/PPm4k9vXPy7z9auV6urqboW6p9vrfy765t/e1M+HBZXwsIyLm7Q3Ab49ciDlJnXbh56rvJY
oYlnnO7IdGeLbRV1GdLN2GykWFcRgxbkp/OeVxxubi3fvNnf2EwxMRIBxRARkRBVVXZeruMZ
nrxk7a45t7ur9MA42AqmYwVvNVAkdo6dQ+ax0SCy8x4HsjaPrdXkxEa1MqnAStTGJKiKjY1X
Q2Rug4HKoATiAAvLfOQssY+QGWaAnCRXczG5mAojj+QUp8DNqffbElcqavLZuZOQHfj0s+1f
vb988aONz28MtzZVbbozdcTlB0YNr2xUK+vDwSikYBrNkbO86KvdGo2ujwafh2qF4mjKjwoa
oTQgSsCOEAkNTE1FtY3wGYwrAIIm0wQgBgaM5AzIgJC4nb3xx4OKoAqRIgI5QxdVI2AijmYx
RKtq14R6OBzVdVIRACNsg5gTkarWcJMYNUYCILO6qkaDgUmKTRNDVBGR1DRN+x0SYwwhNE2z
x+35m8e7P3x9Uqz3XvhOCQAAIABJREFU3ddOv37+3DtmZvbOOTj/+nfbp9Ou3985B2+9etdD
/dKbP3hrUqZuzxbOvnHRzL4Pb51/eHfOvw1/1Z4Lb/3gCc+9Dxc+uTQuTdyebu+cO//66Zff
vASnvvfXb5wFgHPvmP3olYd2+8HYqeJsF984C+df/+Ejzbq3Xr3vgf/kU30evnPxnpm5f/be
gu+b2cU3zk5O3XPtHtzI/UuwG+++hpMWL75x9q1Xd793Hrh8D5j/R+GBA9njugAAp771nbMA
b/24fVf/+C2AXWUkxx17yMzcWYI9GwGAB6/+g8f4yrfPTZpq2zr37VceMSHjVX7iPfmgOXzU
Ou7R/tPsgXtx3732WMN/SFfv7OG9+9+289579176ibbuvXiEITV5H0+8ULDLU/WwE20X1BTG
9eYIABQMAR2xQ3TJyvWtKfSOfQjNaGXVibAlx0AIBsydLuY+hAZiEORB0lGIZdWUg5EmjSGY
KRM5ogzJmTECmIok1QQIjhnUQtNIHSwkDMJBwtpatbZarW2kskSi7eFweXmtPyiHW6OVz27T
KP3JwolnswOzUIBagDR7YO7U6VOHFo96twAzJ274gz+pOp/okTU6tRaPX73dufDzzV/8v5fe
/78ufP6vPoo/vzl9fXhgtZ5eLzuGiJjKptkeUbCF3vzzz7wwP79U1mFzexATFvnU8cPHMvKb
G1sbG5vbW9ubG5tVXdcxRElRhDKOqgbARKqqIhM3CexmlMO9Ub/dccA2wCeiKoroDEHBAFFg
XDYaAQnZkWsttjHjWg0MHJKpibSLhq0yp0Er0dlKWgIBmCRLYyVOUN7YjJevrg4H2vHzLLmL
vtBO2qrj2gg2Yloe9K+sbVy8XV7akCuj8tP+9o16c5T/aq352cA2Dp/RL/3T/CsvH/jq2aUX
/3z61EudE2ey4y8Ux5/XmcXbw/rW2nZVB4xGyQBclWyjbrZEQifD+dnu0YM80wsgMdTOpGBm
IjQChVaKk5CY2RCSqSCaY8ozJA4ptX8XEWmp93csUTMEI0LnFDGqJkAhSmZJFVQ5KdShrMpg
SZnMEThSGieoTj5HGNABEmDus8z7FGJTVqDWrimoqYqYTFZw8rny28b510+Pu/zqHSsIXvmR
mf3o9Jsvt7/v+HTuvAPGh+yiU33ww+++fv7cOzu/7NXC2e986xS0scSHd2x8UFtP+QnPfSB2
n96ef/7tv7n3MfmQbu+Jt3787qnvvbfz1D/1vffumpe7MH7cm92Zuyef6vtn5n6cfeOv2s58
f8cht9faPQR7X6idz/Y64wvteu/svXwPnf/HHchDrwsAd1lSd111V8d2NfiQJdirkV24a/Uf
NsZdltQuO2rPCblzuS+yJ++ew703zJ7tP80e2BuPO/x7T7lvDz9O/5+k249xfz0iaw92MsUm
j3Vmds5Nihnfg/ETYeffAOMXGLYaPnfKc4AlsZDKze2ZvBOqOjSNNI2TaCm2JfiQGX2mzCE1
qFrHtDUsB2U1KsvRqJQkANZW2UNARqKx3tVYYMGz8+wRQWPSlCwmaSI0Tb262r91c7S5Phr0
2We+08W8S763tR2uXVlZWx5dXx/83cXLN26tLvj8pbn5r01PLcWmx9g7eqTzla8v/PP/4th/
dy7/78/V/+J/Sv/1/+D+y/+295/957P/5D+FxZMb/fTpB1eu/uLD/uXrsLIJK323WeVbTbFZ
F5tVsRVmKnxh5vAhN90Z2RJ2D3APkzG5qe70wvy8L/JRXdUhAhM6biypYwU1NWIeF4gxUxmz
mHf7C4nawJ9rY3/tWuyIc7cOLdkh37RZa6hqaSdBDGC8RgQTQwrQ2hovhmqEwIAIoABCIG0k
EMepg0jETJLS2srKxcuf/fzj67+6dPvy55tXb22vbzTlULZu969/eOOD8xc+PH/h0598/OlP
Pv74vQ8uvvfRZz+9dvPCyurtZjXk8diZqW/+M/v6N7eff2lw8itb80dWfXEt6cWq/iylYaeH
CweCLxo1EGSBwhwINdGaZIkQOrkVfmM4GJRlSpFUs2RZFIbxDDCSQyZCQ1TDBIjs8qmpqZmZ
ufmFo0ePLR085PNsJ3LdzjAwo/NOkYyJMy9qTRMMyecFOAeAmfceqdzarusamI0xmiqitHpU
OM4EaJnkzEyI3rEj1JRCaAzMEROBajIV2gmXE1Ebyd3j9vxNYfxFeNe7fcflfvrt71xsv8xa
XPrkwkPbOf/WW+fvfHXv3cJLZ74ICetpzn3pzKn29IndePr18w98Mzyw2w/D+BN7x8/0hRKl
nnyqnxwXPrn0620Qnnw5Hm/+H4HJQB5x3YkRdOmTCw81gcYNPnTRH9XI/av/8DG+8u1z7cXe
/fFb41f4Y0/Ik+3JJ8cXb//pb+cn2g8P2cNP3P+n6XaLvQwpZpoIO7cC5iJJtf1KHn8xmwi2
tFoVEzEVBKM2jodICKyCEgNKDZJUOSFF9AJFMNcfyfYWczIpddTXuiIBTIDGgpQIlMwUSAmF
IIFGE4UyxM1qNEqhSmLoyRVirJQD50nVcGwQQEv58R69AwBMyk1Mg0pGob6+jjc3YWWrED15
4jhIc/vDD/offrRUNt2Nrfr6Nd1eL7f6ReQznaVT6Gerfoqb6zIYTGXlwen1Bf/5vHw+HdZ9
qovcHT3BL/6p+4v/aOYvX1k4+8/r3vGrn9WfX+pXn/dhfdQDyMgyghyhMJzH7Pm5pX9y+szX
XjhxcmlmyfsT3ekz8wdemF44PbP4JweOLfliypiTZeyYnSqoKAA5zpgyAAfAAKytTjehoRma
kRmOBTjJ2prDrY43mCE6Z4jJ1BAN2iQAcJ52uOlt2p4giGlEVEBFQlPdWXsDULOIpIQJMSJG
wwgoyGYAqDjcHF381aVLv7y8cqsa6vzVQffdTwb/+/vL//Li2t99vHbryuZwvV7dCh9/PvzV
B1sffLTx0Y2ti1vxCk1tPPvl8Gdn+T/8T9xL3xjOLG00qb++uX7jxrWby5/XOOwurfm5DTez
mmx1uIlWzWRQkHEkScUQuwNjQ3ZqMBhRf8ijJgs4z7OLMDfbdKdDQQJiKgKCzvLO/NETC8ef
jVluvjB0UkWto4l4x7n3jmlc6QhBDQAZmQEIAYlYCCtMNas563qaBpwx6AFraIZ1mRAiYDRC
zmJMDIgKLOAVvZpTYwNi4IwsRVZgRUwQGwlJoliTUkgSU6rqJiYRtSSafidZew9E++X3zm4v
yIunAU6deenh55x7564wzl4tXPjki2Sgf7FzJ0GZ9vS77Mb7IgcP6/Yenfreezshhd1m5BP2
7wmn+onx0plTv94G4cmX47Hm/1GYDORR123dDBd+/MO3z+9lR8FLZ049fNEf2ci9q7/HGF/5
qzfOnn/7b9795MLEJ/aYE/LEe/IJ8RTtP/3t/ET74cF7+Mn7/zTdbvFY5eVbK4q5zTazVs4A
EVphKSLklvVMyESZ944JxhVEFFTYLKFFVFUjQ2+EAh0gGI6cBMCEUstoKE1jaghoitqWoxmH
Y5waKkBtqTIp0bZSGGqqogzrkBdTzEUyp+CJHLYFQNrISFsMlgmIGClDwiAU0Y3EbTR6e6u8
vZJJ6oLhcJBWVnx/m4dbU6E52CnQ+aFQzb42qSne3F5ZHWybmdV1V+vZzeX8/V9W/+bfXvuX
73z8f//bG1c/u1lWN42289mZI2cOHn1psEEbVwbl7VoqiooNSIAEoN5sjv2hTrFU8LHpzskD
C8fnZ49MTS24bMkXJ2bnnpldXOpOF8iWEqgSoqmCmCNmZAIidIitFCqYgYIaqGiKGpKEJFEl
gSqO/X5ggMDOEMS01TEnxtbngdDaW9imVyKoWQLcqfhmYKpqoiYGSSSCJWAzUiMxUiBBMkKQ
CNtb9epKSW5ufv5EkR3g4uAAZzZo5laJN29urV1f3Vzd3KjrG8NyhN2p48/L4lJ16HD+51/j
P/9ac/rU8ODSWkzrN2/d/sUvr//rfz366U/mmrpLtHzj9qVL1xuhqklrq6uGqrlfJ7c8vbCx
dKyZnY/sCXmOsvmEU2WaTTiNfirrcJYHxIaw1dtMIiLWmZo59MzJ2UOHhF1q9TjrGIbD7a3N
5Vu3lm/dqssKAYmZ2RGzAaggEnXzjmfXaAysDaWgjcUmT2mGXG5WN01jWqvVIZHzTB4ECJAU
nCIbkAEbMKCiIIKJkppTgGiInNTqGBvV1v6NKSWRmCSmlNLvIrT3cLRmwaU3f3DHW95GKH6w
w1TZzRA9++LpV350D0PhYS2MXfgtKevx8QXOba997vvfO3X36bsJufc9nh/Q7YfgDrn+9Itn
xw/wJyebP+lUPw7GlJ3x+L/9ylM3eD/a+RwTw3ZP9J7H3z//X2ggj77uK98+17pJ7zGB7m8Q
AB626A9o5E7PH7D6e4zx1Le+c/b866++fn7S1JNMyOPvyS+GL9j+Q9diV1zs0t+8PblVd99r
X2A/PHwPP7r/91/68bfufd3by5Ca5N/tvG6RmQHG6o6ww79pA3+0A5xIQVpbqY3vFIEFBAQi
05SIsSxHnhlNJcYUQ6s4hdTK/Whb2HdMcWGQ3I/QSrTGQUAA57K8KAdD53y3O4WIhoTkCT0h
t04ZUFUVRCBHyGQEZkIpOYM4KquNweD25mh1e35mcfbAoZr9ta2tK2vry8NyI+itJpy/+un/
+Q///t9cv/LxcHsthpmp+XnoHKmLg6ta/qu/v/i//R/h/M8OrG1u/vLn137299Jf9yjBpI+q
8zPdZ06UVb56vd5aFU0ZQB5FRCKaOoAsaVYr1knBhMEcokdgMDLnCAl8xikF78k5VI0ikRCY
cFw+FwAVQBXUUKDVmBpTzncRz3fiU8hErU9psrK4G7sqyo1jsruOAzVQI0MCVINEqA7bLAMy
bQOECdD35uaOPZcfPKGu6HlYKnC6Gc5sbx4Lab5uaDCAuqoHW6rlwuHpqSNzU88cPvTiC70j
C5FiSsPR8o1b//Dz7Q9/tdDvn2J8TlJx80r50U+3L/6svn3JBmsuRVAuobOazayfeH748l/0
z36Dnj82e+xgb2Gx053p+W5H2EdxFhstt2i4ORUGU9FYCZUtOdBUjj6/9unlTz4I1ciBegSH
gGaEliSGUKtKm+EoKY03vOPpqZm86ISUkiqyU9PQhKaqQS3zmSlUTahCbEJIInmeiyaDcSnJ
MYmMEIiQ0TvHRG1YnIiaFFKKk8B5Sqm93UREJLV33x63528Vr/zonXPjmMXp1196442zO99v
r/zo4htnW3f8q2/BHUbUznl/tUO73aOFd86NHfo/fvEJeU6Pfe6E090ySsffua/8qOW03vuH
9vH8KuLLb55+WLcf1qGWs4x4+vXzZ9/460eyjnaTzdsH8xed6kfg7Bsv/hjH3Zrw3p6mwQfg
lR+Nibr3TvTDj3/g/D/5QB7ruq98+xzA/SG5Ow3CuXd+9ArssQQPb2Q8nvtXf48xtpHC3U09
3oTs1b1fB56q/YetxZh49Coi/hC+c+dW3XWvXfoC++FBe/hx+3/PpZ9o696Lu4oN34P3fvLv
cFfKXnurt8963LGiQDWlhDv59m3adusQ8t4TkTWNqVQFK5KLmAsRAIdwTNzmT/6h2BrOTPeS
p7q/Ga5c6Q77nlLj1AFX5MILR9OfHAsrm8WvbkTVy+vLJYp2OlR0Xvraf5BRPhpUx48/O794
cH19XVLgZsQaTUQlgiRQNTW0cWU6kRTMqCgKQyeSmAZAHy+vB+4o+O3t4dr2+jCNcl/kVFRS
9sOgxlBpyDpFlnefPXH6z776jWOHn71x5cZnP/9JZ7T1T7/+jQOHj7x/69rlpt87ukTI6zdW
MenSkSMUgv39Bb18da7nFmd84Uwl+rwIqoCceWbQFIOQGSK1QpCGYhABGrAyhZFK3usObq9n
w5S7PC+6phCDtCaPajJTQAPTZMlQzQwUCAgBiB0AChogsc9W19evXP302LGjhw8dRlNTIUNN
MZSViZhoihFBFEVVkFxCrgjTTEcOzq1A8/w3/nT+xOGGtDYNDIjqRFkiJYUEIWK/hO0SVrab
rVJkVE9nbiR6dWUF1Y4TLm1tuq21leHqqpf8yPGimF+RMs4Xx184Pb90LPNT/fX+pZ+/32nC
V48cPeqo/vzGaHV5o9q8tr2xBW7beGbh4JH5xbixkao6P7A4/9WvyrPPbaRo1z4eXbk4WF7v
JVhA7onOdnIiK+uy0po7GXvGxtBAk6GBhhSbgCaASGBmgmhGZEhqpqZELGoigkjIxETO+azI
GpGRSOW5zzZUiSFlAjOUzfkOiKxvbayPtuvCB8Sjhw+Hqlq7dTtD9gDOyBExgEPKvJtfWPDO
jUbl1mC7Oz978sUvHXzhhBYevVMwS4KAzN52xG+Z+X/5wf/6JLfzFwHiXg+B3y52pF2+iEjl
05z7jxs7yVB/8LqqTzuQd19r37ZPtUUe0MilN1/+Lvz1H/z07uOLYy9BzgkmTqkJl3kcDdrJ
tN9xbcDkv8wMY4lzM5zk6xsimCoRpjpK0+SMkBI51hioTQgjQAA2dIYhioopkTmX12FaSclG
MUWfil5RD2JIcWVtZebkSZJpK0ca6rbMMpGpKhKApXHv29rKaGKq3iuhz3Ir4/b2qM+pZl+T
xtleuVVRigudfG5qehGKKE3djEDAJz8/bEZXr1+8uVE18SunnwvXPl398P10/UaWmvlYLX/y
6XYdb61tufl5WlzqDwdF01/sWsxS1VSZceZcjM3Nja2NpuksLLipDmicUsmIiizL0Hl2mfNd
75S5X5fD1WVFRUYDVRMAZSDBMTG89ROSoQEQoAEpKSGBYbs8u91PRGhmrYCq7dRGbB1RuzLT
wGh3cRQzM1QDUxFBM0iKpmTIaJmCaywMKzZnNWwtb99Yr0eaQT5H5HymPjYHp90M0PygzmPK
s8wtLc11/BAplCNzIs53iu60YnVjefTR5YXbayemZ+bWN/qjzbK/nlIFw1F3JE0v24C0HYcw
ZK1GU1wszR3MXXF7eW1LGo21zM32Fg8d6M5lTWxW19b7Wz1pzMA3VoCxYABDRNOoMYGaw4TQ
BjGVCBUgSRJEU1MzcAitQEJbFgbBVMuyDGLUKYLp5nCUMueZc8q8EoiKai2xNiXvPGGWudFm
AwbeEUZlQhpzrICc6xRdBESozRCQk2hMamItNx3UACBqhF1Fgf7xo7V/xi+mix+c3x1d+U2e
u48/Orz747fgqbfI/Y1cevO7b3/nr9/bt6L+mPGIEjGt5TQR2Ewp3dHLboWwEb33rSE1Sctv
T1cRAEBDURA1Q/VABICAbJDqylIkNQIxBW75zgQAaiBgRAYaI4Dx/8/emwXpdVxngueczLzL
vy+1owACJAASICSKWiyJlMZbj9tkd4c9M9EeT8eMHX6hHskX9fRE+GF6RhEzEZ4H8lF8mA6P
Z4lWz9jqiDbZ3W63lxYhS6JkLuAGgACx1V7/X/92t8w8Zx5uVbEAFoAitdESvmCAwP3zz3vu
yfzvPTfPye+LAyaIcz9FQSFZbijTmIqjipEMM5f1++utxcX+las6rvnx2HobEJFSXFgCEGRB
QdneRuiBJTBZXoRaYYigzdpomFarObLYPAz0QrU+xWIU1CqNaqChsFJ4B0qCijZmK1L1uXar
Xp2MB8O1d3tvXbLjUXt6OiZaGYxqKryWDTffWwq6tXGyOYVZrMLYS40JMnEARcIbo8SElaTI
kXzTZcZbg6oWxJUgNMqgUg7FgngUVCiqJN9i6wqjQxVoYRHHOxm5MioAL1zuXiwTfyLCLKAQ
BMp0FRKVC4eCQKiQBWiHDr0cOwGBshB9++veexJBESkceUBmTQKOiRkyO1jvDVY2K6Y6GhWj
YWHHvtqZadWbeTG2+bqXgjidbAxUL206CaqxDiLlrUpSLmzUrs905mfChmyMkwtXpycu4CDa
GK0n18Z2qDrBsJjkvYm3iiPjDQ2KtJ9napLPmGpl+UY2HFnCiiYmK0FYm52uzB2yLHmltvLW
62o4rqVplGRtp2NBCYS5XITbVifarg1DLNVfiJBBUAEBgTAjGK0QSWBb3BAAlNJeqcJbJ2LZ
h2EUgtEph0GQSJZ5r6uxFY6jqiaVjEbVINAMIBAQIZS85liN40oUe++ZvdZaQCaTyZQwEubO
aiDwnpCAiHZeVz5Gqb0fH44//YfPfuPEM88/ic8DAOxkV37s372HnyuU60g/5BTZv5PjT7/0
0g9t3z383cZdJGJEbtUnLtN27x8s5UcAdqujbhEII5FdDoSyA0IgBAIh9lw40EjIoSKWskqH
EUAJEDPnjplNGIJRxmBcCZQ46sS6VV/2w8WpQ0l/M0a19N67rYXZ+lSXeWSB/FicTSNSAkAA
DEIgIqgAicUxW/GexFurg0pnaoqTydilBXCQZ4cq1fs77WqeJ8M+9BOtg5oy4NAiSj2MZ2br
i3OTRjzxvLnaHa1t8GSsFGdcbK31t7ZG1OqaZuPqjesPLHyi3ZnyN645Jkfx0CJhxEGUU5FL
niVOYppd6HaqSgskW8PRcGyzwhB79miIFWXiyXswmlXGwjlb9kBgUJARuKTWBAGAfXd24XbR
PYAIEpWLUuVKlLBAqT1TttxeXwSBMtDYFUMu44oyEGUCVCwIIpbTQTJYH5EPexvDa1dXQcf1
WtM4N7x6cTzcqlQUR7J69SqvbZmgUak3htWoQIaCFDvKbZACbGa9YnW4tjlF8aGZ9ii9no/T
rPA+jseI4zDGpjE+oJhm2zFlk7XNdW2tQzNeX9UbmzNxpBU5gCGZ3nqWTJCmOhtJMQwMK8kU
N6PABMYBBKhIHIso2aZrwPeZt7ZpsBTuqLLgtnChlPGp7Fb1oUd0wg5ZyChSPvcGjVg/Gg5B
k6oEaZLMdKdGvZ4CICcaUCsi7402ilCAG9V6o1pL08w71toQqVGSJGmmjbLCoBR5QWQAcADE
jHs0K3+28cMIWHyMxS8+Pnji6yJf/2kb8aPAD3EhPxIf/Mw48h5+1LhLIHUL96NS6n1CSCJE
ZOdgD5v2PsUW+IF/CbPzwB6ExXsF4J2Ntc5LeiNgRCAQDeDyLM/ysNlQ1TgbDM1svdFqj2rk
mtWkobKmcjWabE3Q+v577809/MmN1aGJ4kBcMbJ5PomNAVegK1M1KCIGkB2IdajIWmdIptrt
en9jddhTmhbqtZOVZsuDQWUq1dFoMBiPIKzEuoo6shR4Hatmh2vhZpa7ow/EQawPL1TyZPW9
S9eSNcvKxDKBfHWwZlaXT9W6RUOugx816ocOLaIJk8KvIiynw4bBThy06tW4XbVZVjHBJMmT
0WS2M2UQ0ShLgN56UjoMLAiIBy/MEgbKqEAck1IiBLJb9/9Br2Op9YYAapuT0zOzViTgd6tw
cIfoqGzPuGf3wU7xubfWFVYIGLwoEAATVaqN7ub13qX3VvvrWyyDQ4d1sx6vLl9JN4ZRtZPF
TImtBFHh/eXhYGSTsc2jwseTHGxRCauwNukPLEWmemR+kBZrATXnpzvhQoHF8qhnxEb3x/W4
3kbJwcnS1XHmyoCRCKoBdQyELg81jcPQ5enajRs2mWylW1Lk4K1SaJShIBBBBi7pHVB2LrT8
U1CoXKtkpPJqS3IIEBGUMjEMAIBCAOCZrWcHAAQMrFCD5/Fkkue5roVjdpVqJQ6ClcEoQBUQ
BIABkHinQQwpAQpIKaW89866oBJ5AOe884xclvLR9tkEhNnfRjbgHu7hHu7hHj5uuFMgpbUu
V5t2N+LtqI+JAMgOAzQA7Faa365qdY+cCQASipAAinhnkZnZxqGBslIHoNyJr41i72yWYqup
qpVJhFE7jo9NiclHBvNYtWPAejC4vmp0fXzjuptbbB47unzuDZ+msVZst8MILGujABnAAAKj
K7ypBk6szQpCHSsTEmmAxUptMYj91iDXiIGWKEo8FyA1Im2CxMlklIUZ2NigkHCcet3bGtnB
Kmmeevh4iypiSTa2sF5Zfe/G6wmEVhYfONr61Gn/wFGOgnQylql4wluUDWuY97fWxr6eZ1kA
lDrvvE+dr1erDphBtDGWvQCLAW+dc2xFo1FISpRISeTlvGfP4gkEeDtXte1lBAQSYREgIkIq
iSYRFIvHnbTg9k7KErK9p2A78Cw353nY3qRmtAXLRjQFca2uuHr1Un+Q2MRKFGrrss3Va5Pe
jbmw27SGyUnYcHYySCbr4pZzmdhiGsMZoVoU16NaI2y4VjU+PG1b1fF4nGZTkhSb/VExGltH
veFky611Os375xeqnud1SI1OfzxOx5OtoGiF7QzFBFDAOPW2sCbdcrnNSApKhnFWTBnVIl1X
gSAWPvMCDAgAVJbolSEiigAxkADDnoUfAQGhW+auZ7HelyxlAOC9eOeTSeGTVBQJUW6Lxbkj
k61BkWUV0hpJMWhEUpqZFRISTiZjXFPjJBGRMI56eUKhKhnYyh2mwsAKmHf2Tgrf+hZyD/dw
D/dwDx8/3CmQItxVydspnNl+GJc8QyLMJQmC7Kgaw81v0iJiCTxzWIAiQFQZSQBSUagLZwon
RQY+dD7jkIwOPBOSEaRUIaFUclAbKU7JeKGZrWo1SW2WXBn23waa8PDL9zfPtJupvF0MJ1lc
Wbq6dOTxU+HU+to7q7W5BmNkJ1nDxuw4RZdqh5AHDrQHy0oIGMET9PIsVVJVulb4luVURgUm
iJoK7bwZFGrsC1CDFuiKDmrDUXuchVGcCOpKVU/PNiqBHSxcPf/GhYsX0W488olPn3rg5Ejw
9YvnX3vn5QoE9WYrT/W6C0dk0kg3jix8tn+arlzWSU/Gicq5q424PE3HztthkYQmbIYxiBvk
wwn4zkPdhcVpRTAeJ8kkyyeFOMu52MTbxBaJt5kVlysv6CroA/EWOCUQBRWyVZYQwTufOpUV
Nk+lQDSWCVlYgQVw3uH2CgwEoD2TBbHEXinHoj0DS56kPvOIiMorKaBglKob+Y2N9X4+yDVn
SG44qSgfB2To8sAjAAAgAElEQVTQumTsUjdhO7BZ7j2jm3buiKHZOG6pamyq0czh6OgDON3I
Kmozn0wY8kAnSRrGRpkGJGktmvJ54MAORpudwMw2oljqdY3rW8OscCtpnpEJA9pK8lExSthA
kIU+rSEG3gccVhwFSkGag0YxO+EIl5scQQj89qQWJV4AGVU5VwVKeTwC3GaZLXPNjnRC4kHA
M3nQgRmOxsoXxgDFQZqOWrWoW6neWFo1nkmc1kGgFLEQoEZlgBQqSLI8tXmeBgEW5BIsjAo9
MjqHRK5MwvrtTa8CUopS/7h+9/dwD/dwD/fwI8IdJWJ2asx3GLJxp858m08b96SU3hcb2UEZ
dTEKagoAQ0FCsMQFOCAwRIHWRVF4a71zoNAEhpFQByzoFHjkyPqon/I45UasO61ilODKADcm
a6v9yxuTC9d7FDRm5+cKzjY318drG72rl2ZPP9hcWLixtF4LahFVnJBVygca9K7CGrCwMBMD
O5/YPPHWI4RBEJtACTjnRkU2yIqEovaDn5j/4pcbjz46bjcu9Teu3riysXJ93FvjfFirUHuq
0T00f/+ZT3zxV3/987/69+OFxe9eujBkOzXVnm5WgwhsMdi8dP7KX3/ryp/9+cbL37HXL+ti
bG2+tLbVGxJiN4Zq1eqG14dr7SNTU40gROeUdRFjVO4160jw2Wb0n3Xaf2/u0G8cPfbbJx74
7eP3//YDx//J8Qf/24ce/t0zn/y9Rz7xe6cf+u+OHf/t+4/8xuG5X5+e/s8b9S8F0edQPUpw
xtuTo/yBHjyQFrOTUbM/ag2zqTSfTiet8aQzGrcHw+bWsDkYNgf92tZWddCv9rfqg0FrOGps
ZbU+NhNVm5haouIJRKNcr3M4cTz2kqPhhNO+ZD2EFQ+rFnus3ytG70J/wye5uDCIunHt/kr1
c9XmFyrNEypoealgEHWnYHbK1WtWB5mXLLdhpXro/mMnP/3JBz/3qSOPPLTw0NHTZx5++OSp
drNBKEA+jFQlCgAxYV5Nkre3Nt8YDK6nfugwCMKpWtwiH2WTulLNSh11OMqKrWRixYGiUjiI
BEl2efq39yfittSNKumeyqqpslYMBUuSBBYuSApFSWGJVYxBPkoHw1EB3hqw5D2740ePZMPB
uNcLCSOlNaBCJEACMEoZJA2oWKDIiZ0JFCse5GOKtSgsd1+CACOVPBae2Xv2zP4nXSO1R3r9
fV6jO2MvMd2BORV/CnYeBB/6Wvboxe9p+f7Ru5r1E/PYTefDH5578yeEn7R/9p4VER9/7sUf
vwE/jms8SJ8/Gd/+dEbwR3TeD9HJXXbt7f4dibZrxQE8825p+fanO0VUNxFyAogIIWmjFAiB
IsSSetyj94GmejVfLtkSNJAy1QoMDSgvTjSAAWBEOxn79V7cjGV+dri2WV0fH6lAK0u3YrNy
baM/c+LI4dPLq4PR6uZiY3104Y3WoenFz36GJ0X/vfVDncZWkCeUoLjQEXHolLfIwKxzL56z
Ile5w8JNxG8FylfiBV8vONhAb6MmLR6JTj4YLM4zc//ypQsvfee9K5elHszHLs6iMAjTNLty
7era6rJGbNXrDz76SQWQTibra9fnus1Tx45feedCmmxMlgSHl+Olmp5t2G47qMXzn/8i5uFk
Y8tmq8PB5qwOYwAsMlMJBZCds4Y8A1tOnawbrww7LIh8yYgJIopBgTaoFZeKcHHIFXIRSgOl
icLIgctDa73jJM/aD2wds2OuRg2FWqESz8hsbW7z1FnLzjvvGdABeGYPyAReUVivFex1FDXa
1ktekC0U2mExuLSCWG3Xo2oYOIICg8TTxHnxDjhBHMc6qpvaVBDOBVHDO5OnaIVJ56BFdG6L
SToMgiqGKtJU63RUYd0kubJ0Y2NjdXltJUknM1H1SLPdqUQInI+K1MsEIdXkopANpS6PrO4K
BEJVMp2oakyQBkm/1+NAh1p5A4VjEReDEillmhFBWOTu+bKd3RG7M1lQpVYAdLfVydK8v9EP
vBhDjXptfbB5+L4jjHL96lVDqhZGSoC8oLAi1Egat38SLOzYqzCgOB57P07Sw1EMiN57EEAW
gW1yi59yjfkuQ87F5x4/8STCx5WY6cPaefG5x//g5Es/wmt58Ss73AvwFXzyycdPX3jp6eNw
8bnfLdmsTjz3+Imdgx8TvPgHz5yFXR6mH7lDfkaw10sXn/vGT9ucnyX87E65A+3aK1GuOZWU
m7tRFO4Q3tzCJlX+nQg9gHfOsrAIIzEwCXgkG+hwuuOvXC+8qFAneVFptWDQy9PEECnvlfgA
SdnC9ft60vb1WjjdDVfW5pgPZe7qaLBM6o3VYeeBo9VD9+cbP8g2rlQjXaxcj46dXPzUF1fW
vrW2uh7NEqATlysxJJEDAl1Q7sl70DpPspjUfFwfihv4Yr1IT0NrVjdiwkmt46YPjxrtzMTj
zY2r71yqWj7dbMHqcr0bVyuVLMmuLi+9/Mab569dG2YpE7XbrV/+0uNnTpzQ3ilrP3/0ganE
Xbv67mi83KDKfOIry9nW1RVoz1aPf6Jy6rSpVvtX3ui/+oM0SVvKy9a6zTNlTOFsqCIFmiyh
Qy+gtFKEotj5DMgRimdRSE6IvTCUDJtExqF4kJwESDxHHhCVJnJxNK91YZV2zmUU6HI/WijK
+Eqp0AcgDskDeBb2LMyCpZ4iODthU7BjjaK8aChGied+Gk+yeYo61epQ4mGOictyDlHyiuaK
UTFazu2kGBnShgwyBhQoiMST80yKxeZaVBvJp9nytavXLl1aWr6e2iwVO0rGw7AeLFjqtgJ0
HlHiKDBhQwd2PEmsVYbEWnRSM2E9jEIUn6U2mUiRJYVwGIRG6zgMArOtfCOiEFEQSlaIO2Pn
8x12fnRAqbW1ahMdJP0hFa5eq3fbrbX1lWqzOjUzdfHSBXEu3iZCZUNlOg8UkFakCVlEhIUI
gjCuNzaXroDScbVWFkcJe9wphhek/Tdt/ORx/Onff+qZJ5//2nNffeJgscBPaQvdQezcYeu8
S0cfyv7393Bd3JW/OH7xT79xFh77rRMlb/Uzz3zjTy8+/fGJpN7HgRxyD/d2hf6Q2OPAn+iU
+0kP3J1Se7uB0W6ZeRlL4Y5GPQDsPb5bKYXbVctEpBSAeGYBD8AiigEBPZKrRHq6K9VKUjjC
IMucrte5EhcICKQFybMSDkTCJKPNAYuY2RlfqTTYnKLq9CAfjkffuX51Pap1j57oHFnIZBJk
w+TSJRinen5h5rEv5HFlsNYPJ3ngHKOIUorJOAgEvXARURoCKGyruK0q4HBlPF6yWQZgPCgk
NEHUaEIUTdJsuLqm+1sPBcEjRJ3L74VvvV2/en12OHmk3f2Fo8dmKtVksPXmudf/+I//30vv
nVeBINj5MPjU/MKxqekkz1ZHY4em4aOTvnZiCMnrb7716kvLsKU+9fD0r/yyPXpkVWvqdgut
R2kmIqEOYhXGEoY5VQtVySEqICqk4rHiVeQo4iCwYehrNZwyrhJzGAFqLrTnEGONYaBCjazA
KWFinEixhZOBGm7pXp82t0yvpwY9PRrFxSDI+mbcN6MtGm3xYCBbiR5m4STBwVh6uRnZcJLT
OKexhYnmIix8vrq1+s71fGU0pao1jxVra+wr7Cvs51X0cNA6TtEMUMB2nE+Ws9E1m64Sbik9
8C51DmwOg36+dINvLEUb/cFb5y9/53u9y5chSdFa5VwjjlVk1tPR5fXl673NXjrZShOHGEYR
CWLhaqhjpDiOq806omysrS1duzrYWAsB2nGlG1dbQVxTgRIk2d5zQNt8sDdtcNw3WkEARCql
WrAU2GGoVhtREPVW1jC37Wptqt0cjQdk8Oh9h7c2NpL+oGGimHQIFKAKUWkBLWRKHjZEVf6U
tDHVGhvTG46DSqyjcJv9szRSoFT42c2S/9TpD0oFqh2dqj0Jtf0zQ7eIju0uie/9176dlCpZ
jz+Ou//b+e7NHX1UOy8+97vPnIVSDuIrL8JNObk9dtxuJf+uYnl7uEAvvHl2R0b++Mkz+6t8
7Z79K3+6//H3rbrZpPe9safpvobt39WTzwNsK+qcuNkhdx+XW9yynwH7enX7Cp7bk+98/1w3
XdjjX/nKXXKid52BN9vwfi/7Ht3WZnvuK3s/2Oulx5+7uNf/N3vjQNd1e5tvNwfuMoi3M/vD
93lrmztMgH3HZV+XfnBW7DrwA7/BW0590zS7m9NuMmXfxh/5XnQwN34AdwqkypBoR/zL4w6z
Oe5ZqVJKbXe0J6lXPgy8d8I7WRWlgQgACJQi7YiKOLSVsDLdLSyHcQNV6II4mpnzJnQeNGoC
AoAAKcidXd4MCxd1m+l0B0x8PGifobjq83eWLr1y9V3VbMWLi/rQTJ5spVevpJcugeTBycWF
L3yG2Pi11CQApHNbKOGo4Ih04os8AFcPc0MmCBZqnam4uT5JX0v774buqimGNYXT9SyAwghU
DVYVK6vJNQ3WbdYp3HQh95vKZ+YXP3P4vjNzC0cajfl6tRaoG9fe621tohaLGXA+EzfbprvU
s6+uDi/nwLo2ReFDlag7Xn3nb//qnd7qciVaqkbnssnbefLuZLyapIVDYmVYBaxskmtGzcow
adbKG7SB8nWybe1mskFj5YrYQWxclSxFKlISamiQbxZp4AoFPgBbCXAmVHPELZtHCivCGiU0
VA90R7iuqW10B7kWuGoVmhWpqyJQPlDeKAiAFUDgGAWNE+0pmrC5uplc2UxXEr/peD2f9LOt
ie0XfhAaO1+LjwT1RVWdwbCJYWSqGLdWLV5M8svi3skHb0+WN3tLZn29PRgG15ezNy+4i5e7
mVswwWxgZsPwvkbr+NTMTLetIpXadGs0WOttrq2vry2vbi6v8jjtmHBKBVUPjUZdV6I0z8bj
kSLpNOpTjfpUtVpFVUPdMXEshJ7BsYKSjFO0orJOquQrK6mltuumdkEEO3LO5VSOjDEsm8sr
aF0tCJq12LpsOBkcu/8+Q7h57UbsMGKMkIyXkMgABkoZpFCpgIhEVPlqEQTVdmtjMEisrzYa
Joy8iGPvmZGFnXfO7X0t+ZgQcp47f/F9dQ4RKfWw7lhlU6pYfXP7fvXNHTHYO3ZyFn7rgoi8
9Ie/9djud1/85vMHl56/vZ3Hn/7DZx8DgKdeEPn6E2VO7qkXpGxzk6ryR8CLX8Enn9/JM148
f24/q25uX8rEichX4X311v2tKuXYbvVG+VB66gW5sKNkuM8pPtDV0y+98BQAlKe+cLND7j4u
N+Vj9jPgjl59/s2TL22f9Pknv3Z6x4Lnn9zzTD37PPz+7vEPTq+DzMA9Nrzw1E4vL34Fd4+W
emp7T/oN+GrZutS+3eul/ZKyt3jjLtd1O5tvNwfuOoi3M/vD9Lnv3LvTBPjguOzv0jtOy1t+
g3dw7B2dts/g3mVWfMh70UHcuA/uEkjdVCa1o6+3F7d8pXwSlYFX+UAoJW8Z0COAKBJiLwXC
GGWAXJ+by6yzucRxuz/OmnOHdK0BZBA0gAJQSlBbUcPULi0ze3VogRvtOpsvtqZOBxTnWy+/
8t1N5/TikbTVlDjM+hu9t98aXX5HKA8fvH/+0c9lRThct8oqg0LgFCCyV+zBezaYRMCtSlyv
T7WmvQpeSXp/Nlr6ttu6QMWG5gn6oU1cXQ1qdC7Z+EF/5ZJN1wkybZyg957TrCbyqfsWf+NL
j/83v/5rv/zpRyFN337t3KWrV1bcKIkxjKoR1cYTePnqyl8uLf0gG74xWFtZWZbl1eTt8++8
eu69tdUbNluOzfd6m5dzu5b59Y1ROsxDCogoKbIMoFBYKFUoKpSxJkzBOF29tDT8o3/57//w
//m3/+7Pz61vBkBzuWsP0+ZfvXTjX/zRd/6n/+WPf/+f/8v/9X/74//9//jO2e8MN5ZrERym
vINZI5SuKhqYN4qtih1UJWlS1o78DKVNnNRi7oa+o4o6cU1xFbmCPkKpgMRClTSI1pgujIs3
J8WbuT2XJ+/KZAmS9aIPynZb1UYcldyeiEq8Hgz9cs+OqOmac/0gvMHpe9n65trVcGWl3R/W
1jfj9Y15wAc77WOt1tFm66G5uVMzc0ei6pTWnSBoh1ErDBtBWA/CCJXx0DBmptqYiattY1yW
9Tc3J6NxvRLPdTutalwNdEQYKFQowh4FSAAFyrrvUrr5IL8KEQGQknyLtFYio7U1laXN0LQb
MSle6y0/9IkHm+3GpfPng4IboiukA6Cd/zBACog0kkJCAXYOEONG3SIs9zZykXqnHcQV69l5
Zr8Ndt5am+d5mqZZlllrD/o7/nGjfIZvq6uWSsTbKu37Y8/da8+9646d7HywJ3a46Qs/Ojuf
+LqIfP3Ec48jlkzVdxG3P/70S7LvU3W3txfgyYMWcO+1cFu19k5W3cEbz3/zxeNPv3RrjPMR
LvBg4/JB3GzAHU9aLtcdP3nmDh0+9uxXn4AdddsPWHyQkX1/qm2b89LTx7e/WPa9I537td2h
2u7xxG5y9i64xfg7X9ftbL7tHNiLO/rzg2YfpM/btbnLBPjAuNzRpbeflnfErvfu6LT9B/fO
s+JD3YsONDT74C7F5nizaPHeP/fNPuBOyRTupv9EAJgJRLDcE4VKWyUp+Nxm9SCYmp4b9cYz
p06ubVypdnV7bmG8OfTeo4BHEKKACAvfu37ddOrh/NG8zbSZHUP6e42mzQbvrFz61luv/Mbn
v1CZvS8fnYfhQJauhJVKrV6D+fngkYfnc3/5r/8quD7oztTGnHkkBVIVwonXIV4db04kbGC9
3WzOa35l88pKNugUxeH1Tb26VdWNokIOYByYZc4319auDOOpen02tHVlgIDBO/Bic0yTSKta
tTIxZjhJl64sDRrRTK3dPTw9E4bz7eDKcP1ysrF1cbPjVY2MUzhqtjbz85UHedJb80JZGGe5
BM6PJzxSiQ7JKp+iz7zTpFlAwDM6UsSeQUlYCawvhqMxYKWwFaRGMhn/X//y3/ybf/3yYMAz
s+1jD8yu9rLvff+1P/mTNw4v1P/RP/r0Fx47zTBgWwjTf/izs1sbxcL8fQsLc+125Tvf+XaW
ZUGgP/nomflD0y4fMRKgBykXbhAAEQBVMLHF0nByI1GoKmIChxa56ITVxUprlnSQ5ilbVjQG
GFWrZu7I7PRic/6wUtS79m7DgBptVIsi6PfVOItzG3uxGobOZWIDcXZiSauWMk2tHYNzohhJ
BxIEqcMQbWFdNhmmNucilVxXwziqNyuBjoxG9gSlQrH3XkphoFJsRSEqoIMFUeCdK0VavPNx
HKvA9EZDHA+n6tVarZL5fJwnx08fb8123nrlXDaadCmqY0CWlSIQ0EiKkJAUEVLJB8UIEseR
qVZWJ8nmYBRVg1a368RnWaoAERWiIhBAYH6fj+1jUSkFcObk8XKh5ewzJ/CZ94+/eQHgtiHO
E7/5FDx/7vxFgG8+v33PPWgnx//Bbz32zNnnv/niV0+f+zBx1J3svGlNa1vp47FnL8hLF7ZV
P344PPGbT8Hzz3/jTy8+ffIMwE2vsWWabxelhbccvJNV+3rjZmmcD6jwfrgL/PCDe6s2z1Mv
yNef+BF79dz5i7Dnog5i5AeXA+FODv+x43Y2H8ykjzKId+7zdm0+1AQ4d/7iRdi/n/1mxR2N
vr2Rt3PaR7L8Q9yLPvJsucuK1L4H9wJvBgCUwnxEpLUmUrBDkY3b/NuIpWxLaArmfjZpHz5i
rcvGabXR6g+G9VY3rjWcB6XC0jxCZRiqhS+WN4phUZubr81Mxbb4VBB8ududi9XLb732g3cv
teePqW6XahEkA//uBbnyHk+GWasWfeYzR878gh5D79oNa9NCsWeOLTQy7uhoVOSX8sENl/Sy
SRBXa/Mzrlbbsrh8Y7N3edUvD3B1BIPce5BKM680t3Rl1anzafLKcP2lK+ffXLlKVTM7P9Wt
x4HNgqJYqDcXWlN1ivyoeOPixZeW3hnMB+1PHlJ1NAGm6HuBWq/UNkzLQT3eHBfvXnVbI+v9
2PmJk7jWqtfbmkxeuMxbxx4dG0YjrMQFUCiXVpUzfhJIeur4oYVufP71v3n15b8Ut2nT1Xad
P/e5B371Vz7xz/77r3ztn3/1f/4f/9nv/e5vxYH+3veu/+EfvfjmO5cq9ZaJahu94fkLN0Zj
Nx57hPjs2XOvvvbu0Ycefv3yyrdefmPsvNeBJ+UQgXBn/xqRgLZOW4ZChNGJyq2wpZqqPtBZ
PBpNd4c6HovN3KCwfU320Fz7sc8v/MovJjNTb15fWr++2hi7h7B6Mmq0kCLnQ2EtzheZK1Lv
cm9Tsal2Rcy+IzTF1AVTBx0xKsdSFGmSDEaD/qif+6zRqB3uTB3rTh+qt+raKOfF+cLmaZHl
7HLiXAMbEhGE93k69g9MZEe9qJyoSpWLQ1EYIuFkNOEi7zaqzUqEYkfj/sLi3Pyh2TfeeG08
2KoHYQV1xBgiaSRDpAkNoiHSRCXrLIgopSq1mhNc3tjIrGtPTTWazaKwLMKey+Vbt02wuk2B
izdv9fipYDeXtP3KXS6E7+DOd8knvvrsY2e/8acvnj+3+4J30E7Kd9xz3/yDb5w9WBz14ews
X0tf2LPGdNDk4V2we/8t03kXz5/7YNelhR/I993Bqn29Ub7zl7md3bzFR7zAjzC4+xjwo/bq
LQ+zgxhZtrnV0Ns5/MeP29l8IJM+0iDeuc/btflQE+DMyeO3P9cdp+XBcGenfTTLD34v+siz
5S7F5rKntJx3UhCyhy9q+2OlcGfvHjOjALAICwIIoUfUXgLHBljQpVhYsAoAQN0YDNxU0zfC
9dUr7XotZ87CKJ47nJnYWmraIM7F+qyAYmYcdd5L7dXL43iQnKr2FmLM7Cdt8HkIYf3Gt77/
1+cGm/jAJ8zcfVFMbnxt6eK3061rSOjjdu0Tn2ucOJURDQebftjHyUTlljzHFM2HnSmuhhxM
PK6mCYA29fo4gM1iePnKm72l82a8jhsrlVHSpXC+Mzc1sxi1pmy1WTQ6RaPZCxTNdGcefODo
mRPduU6AvolypBIejcMZpafr1dFo850L5268eyHInBbCRrUfcp8SxPSQLn6hQcdtb7q/NLO5
ujDcOurzwxFEOitwYl0WWgzHPvYYiCAXSNaXJAhsuUijgElNLPW6c2ZxsQNsOs2pf/yPf/Wf
/g9//5l/+uUznwpYRlkmzIFHp0PwooMYN/pXl5cvA+Mjj3zm8S/94tLq0l+89Ocrm2sqiqJK
xF4QlSIlvC0Zw6wEjLAWVmU9tHOkdR1NmEpesKUMZiTqOqezXoZpQpKDkrgdzx1vHX046szb
PNt4983Ju292i9GcohqrCIxnnypcI7ngknch7ddJzVYac812u9qp16pBAOJAI1ejcRBuetqy
sDHKNtLJmBwEtDAzc197qmGEeFy4XsFDi5lTIkHoQBcOisxxnikuDBdavGdr0bEm0bStPygM
wIxMzJFnzSIkXgNrAEAlqkpBTCabZGlemECHkU6huDbqTd1/ZP7w4cuvn6fV0YIPZlg3UWth
Q6QBNYJC1uACsMYXyhbsHCsD9YavNVeLdHk0KBBa3akorHDuFSsjmkB7xtxz7rhg8QzM4P0B
mBp+rLj43NeeB3jq958+vlPOvb1qfjB6leP/4LceO/vMk8+c3X36H7yTJ37zKTj7/PMHi6MO
YucHbsPlXb786t1yOvsXm5e9b9eM7KYMyizBmxcAtvfv3Wr/Xgtf/INnbi7C2N+qD3ijrJL9
yovbuZ19n7B3ucC9Dvnwg3s7Az6MVz+A7aqa7eH8zVsehwcxskzifG2njKccnfKL2xU7e+fK
jx+3s/mOc2AvPoQ/D9Ln7drc2bcfHJfbuPQu03LfUOjORn7AafsP7l2n7sHvRQcemltwp9Se
Ukp29uvBTlKvhOyU4pYSJNsCZruBVykcJiIiDMIsAKIFEYEVMnpSxIWLo/j6YLjaHE4fnr90
7vVmnnYqzXScTR29L1/ddKtrhUaX2yDEQkOYUyVxKysrk2mpzrUrDx7O7VLcy3/RTKHxry7d
+Is3vsePfvHEsUXKt+I+rF67Qq+dW4gOSaClHcWff2RmHq+8chY3Ro0gtugLQeVct1K9NswK
sByEiYcic1Ud6ao1k7EdbKxffEPS+ahZm9eqFkUVUnEQ3lhdSrKsXq9QHI/HyZWtzQcfPWVa
0WwA442eslgkeUDSdVxH3WnM5IxLq2u9fgFxBGF9zW7FNltU6j5XLPoorNU2UC3115ud+QZq
6Q+Ii3q3E9Qjl48zBskNSsgSCDgRtiwaQhDdaNQe//KX7z/5YK1Sb9WahbVkLKkk1EWt3nzn
rbX/+4/+5OrldHVtyEK/+Etf+I3/4rEjR2lp5QeD/vp9h85oDcurV6bmKkfvPxRX47/+6299
/3vfOXFs+pFHjgeKPCpXlhiVxEuCIIQiIamAOTZVbTLMk5BgJqjMURCNR7YYFUohVapRuzK3
mN13Hy8ellrd9vvQ61fTJNCUGbWuhF1ufVHkxXo6WR71K9ONE4uHarUgAHaTTBVUMVVUOHL2
yvVr791YqZhat9EFRYU4JoTIMIEVj4bDWIWmynk+nqS2cN469kSEhIAERAjOArAAM5ZM/cjw
/sKUIDCJAHoED8LC4NmIqQYRMY0Gk9wVECiHUoBfHvQWHji2cOTwxXfOp+v9jgR10JEoAmaF
BABIQF4BKAASL060CixQEFejzlTPu6X+5iBPq+3mwsJiMk5d4TQqKbe+IjjY3i4IIswAcBPh
7U8OOyvzAHtX55/4urwA+OTOUvhBlu3LpNTZPbHEwTspc2V3jKM+nJ1lh0/i8489e+GlF556
/sntrz/17LPwzDPnzl+EJz7ks/X40y+98CY+uZ0deOzZC19/Ys/hsvMX5INP7Ce+fuHZcyee
OYHPwFPPPvsYbN+sn/j67a261RvbXZRtH3v2wi0nuVNX7zfa65APO7j7GXCbkx7coY89e/qb
iE/udegfP7MAACAASURBVObN57y7kXt8CwBPvfDS08cBjn9dLpx+/MTOdPloGaePhtvOxv3n
wM1f/ZCz9AB93rbNHX27z7g8sb9L95kVe98+bppytw9lD+A02Du4B5m6B78XHcSN++BO6YPv
fv9suf4kIuV+vZt0YES8CItsL1gh0k7iTyHtbvdjBCDSAloQRCyKJSCjVe5bnn7wb/+yK+bX
zjz69t98pwA+efxEkSZRJcKN3sq514nyCjJAzuhVoQttNmNJWlF4ZLa7uJBtJfatpUrfDRX8
J+m9bNJDp05/6eEzRwpLb1+hK5tQmPlHvxh/7nNFiEU2CZPJ4Ac/SP/2FdzsV8LI1cPefPt6
t/GtleWVzKOuAGoPgmIr1nYn3MqhQsobpFoQxMZPUnBOh+bi9atbW4MH7j/W6NQGSd9j/pnP
PXL0vsVkOPBJykkx2NjK1wedvsPMjUQmqRsOknGS62bNV8KrGysqy0+1Z6fE1KMYokDq0eq4
f/3atboJ64i1wKDyEoA1MJnKTv1Xp2Yf7I6h74LEQwYCGmMp0FsKwqoQAQj71MkEyCqlBCve
V29czf7iL15dXyvOn7+2vjY4ferM419++JOfrke1DXajWtS1eZClLq5oUGKCoNcbJaOsXq9F
kcmysaAjhFJfDkCJADMZ75tM577n/uRfX7+0Wmxl47rCh8LWmaDa9an4cR6GBVTBdPyxo4Nj
i8mhWa9Uurxy4+zfTN56u+YygoKVgC80CirDWg2ySbVd+fxjn67VtbhcrOPCkxj2XDj/ytvv
vPP2xenq1Hx7epCmS5O+r+rZmemZSnUqiuN66LggYJdlvdXeZCszEik0QWBIMZCPFFKWgnXC
TIC4rQMjzAIIjChQknWCLwUkmZWDwFNApij8xBVQiXINfU7HYE+cerBSq158+52iP26iaYiO
GA0QAAqIIkJCJCEsef8BBBwjVerh9FSq1JWtzbfWV9bGo5NnTp359KMr66sEqAGJkRCYQJBK
cZodSilAxH/1xx9iC+5Hw8chh/hBlBUiP8mH3scZP+ve2NlC9TEiL70HuDcuB8edVqR2uTd3
s3tlnAS7lVLMgNuyq4i4u2a12wwRCYGIkMuMinhhRCXOIwhp3T00987ffP/0kSNHTz147jvf
X1/bmJuZTZdXGkfnapCsvnEODFLqK4RJwN64jlPN5SzPNicS4uIMHj80efN6nNhHw9oomVx8
/dxZZ2tf+OLMQw9GvCSXVzfffLXV1vEjp4uKsdLunPpcnkH/tdeKyRAmqVtzjVrQRO4R5+SF
UYOm1IZF0pZgBlUbqbD5+rXVerfZrtVFYQGchKoa6BmlQsJ4qj1zdH7M+RCt1EMMyNTiTi1y
rcbyd99evX6tWm8RmKTIE2/dZGQCqUQBOF+vVUITXR2NlpeXlt5Yrzar6WgSAz506LAP9crS
NStFd74r3nIxQqmS5CTWoxNg59ioUJFiHoiAUOExVdoUGWVJABIyFIePBv/175wJg861K/b/
/Bd//uf//uWz3/7WP/mdX/iHv3G6Wq1alykttSYZYzObZQVW61GlUrM292BVwJ594QrAUmAa
REjQkxJ0NByMB+NxLkoQAuAmUkeowyhajTWuIdh2O5ia0VFNsXEIicAG+J5ykXfGuzgIGu2u
gM/ygj27IOiPx+v9zbgxw4oFJa7X8sylk4JCU21XatUwAKuLvBOH7YXjfqoWt+qTrcG7vZ4U
4Hwx3ai3q5Wg1WI/4VRUqdGMzMIsLOJBGBB2WA/KwvmSQhwEgAG8MAooJ8qxcYieLWc5MAc6
1zzg3Ib46KOfMUq99crrbjBuq7CJJvAQIFI56xURg0JEJCERYWEBIjImatTZ6N5osL7Vn6RZ
pV5bOHJ4nEyKojCkhEEDApEAOmBhUYQoQDsVhz+vePGbz8M++Z2fU9zzxj3cw8cadxQt3mE3
2K0i392jtxsqwU68VbbbkdXY6QGp3G1WMvcIAAEKC4sQKC88e2Txpb/4q3OXzv/aZz4/e/jQ
YGWjU2uadru3ud44NL+1tZFsrLdNjM6ycsIcFhJYnfbyzQurrI05NFXEODh/NVxPvgx11eut
vnbx2zk8/tnPz5++TwJMlldWX/n2AxVVP3TSSgW6c/FnP6Pr8ebfvlL01yup0/20PSo2RHQN
JYyV0OxMq15M4qWtKa2bjoMgmjZNBVSzVhN5rbpTU0m9OclTHvFomFy3SePwTJLkURSowJBi
XxQ2wsuR+3fvvXli7r5jM0eWkuH6aJwpyNeLiGjaRBdWrtssu1qMBzYjpGBcVIwhwZXRVm/L
2jwzGl3moBCxZCjEQqOECAIgzOJAjAIRDyUXkhiS9rXLW//fN/5scz07erL5D//Lz953vMY+
WZg9ct/Cw1Hw3kZ/5bVX3v2VXz1Zr8WOc4+FIkyyRBvFYrzPxDskzyLWWURUxCwCQAgABOjB
I2Qhrdp0rUgz03CAAqgDjczIAgDe+qyiYX7KTE+HYZRlTkjG4/FmPhloX9e6TaFClQsAialW
feFa1Q5Ilqc5CZggyPPEAzOJRyAFhw7Ph1Z4fVT1VO200np8LU/7m1mWFaktRpNJHAU1UhXU
OUvu2DApohBJRFCJIbQssL11tCS+BNiuPEIEFBFiUQBomawPBFHQMTsFEpsJuoEU0VTz/vvv
S9PktdfeCK1MBXHFQhVJeaayUpAQADUB8fZqrRNgEUbVmplJEQdZNsjz3mScszu8eKzRam30
NkGAPXvrEJVoxQIOQFBAEQo7fv9393OH7e1K8NQLP6PrLx8K97xxD/fwscedAinYGyTtFpLv
vCsTESDyzsFyHWA7wJL3WdGhLDwHYBZGYRBkNEqjFwtcn+52jx569cKbnz5+fOHBY3Ywuf7u
5SOfPiO9STJM506dXnvlNdcfIKBJisAoQJgE4sFXR2n21jXrC3tfC87Mq7dg7sbkl+pzP9ha
f++7r70M8KnPfapzesbXnFnZWnr5bxcnYTR3rNBKGpXo9IPTcbD2/Ze5N+hINO/Cy/31bGSl
6ZuN9sKh+a4W4avh5qjGEKBvhlVyNvJeO28LiYgLMqlRg7wYpckwmRxamA8s2SJz4ipGs7Wi
1cyjD+tzb71+4cpUZyFqt/JsPBGPiI1GY67aMIUnrQ5XmvfHM6EJNGqDFAEaFrC5ata0VqR4
UGxlSUJCRIbZeUQihUrYeidMCgRYhNiFHsz09KFuZ/bsfzr7ymvX1lbHX/zy6ShsLl9d+tZ/
PJ9OJp02PHx6sdNsiUuFRWnyNkfl2TMwoQcAC1IwiyYQYRZBFmEEUAieALzAUAXrPh/4XAxm
TkbCy3naVagCw2y3WBLRBtWgSDaz8VgRBDTeWDGTSZd0w6hGHE/Go2GvV49Md6buSU+3Ot1O
lTmxqeWCnWWfZyIUaoMC3UqtvbA4ypdwlHOSrW9tXskHkzioNdtkIraFB9UbTrQlscwgCkQr
FSgCj+iEAKwAljtGBaDkNxdQggDAIsisnDeMikGcEKFlzgznodp0YxvruWNHmjNT/eXVlYvv
RYWfjuo1r2IkIyiEAIBaISIwE0sYmJx9bq0KI0GoNloQRuNksuWLlWFvq0grnfbJUw8NxsNx
OmHrA6UNbGfAnRdPSApL4g/5QFXizxHel165h58Tb/xcXOTfQdwbl4PiLqm93cipXH/SWu8N
p5CovONvy/ABALMgKKD32TsFGEQARGFZS6tRkUBZsMKGHvrMIy98/7uvX3jjlz/72PypYxde
eeP6OxePPXhidflak2Hx/pPX3nrLDscdH6F340jGkSeWmuN4M++/lRYw3zy2EJ08IrxcWxv8
wsxcO9m6duHixZCPP/zQ3LEFj8bfGK289dp0nteOHHFhJTGm+sDRLsHqq29OCqyqms7WfZFm
1keANs8pqppalTdHjCIKAVlr0E60E2JBAHIOEciENmqITVffvpKN8qnFWaUhT1NDypCan+v8
+i//0t/2/iwaJyEqShIAZwwd7bYeWbyv6ryxFr0Fz856b/128KnQB6bkkXTiEgUsOaBTRkxE
Qtp6q4BQMYpncCwFSCBoGF2tHf727zx6+pHuf/wPr77z1vXXXn0vCijPMQqrn/505wtf/sKX
fulkGOTWJkYZYI9ixeeCBELIAGiBLW6zLTEyKEDPwmARSCF4pTNvC2s1i8u8ApqgvJtN0EBP
kQI/8ejIzqb5eDRY9WkRh5S4bOl6ZzTqsqqAzopi4vJYwVQQdMNgmCerV6+4pDUz13K5S33u
rSMVsHPVQBsF4jIejYPcKitJkUUKQ9CpI5dLniQqDLzlQZEaVoaZtBbHyhUKOBSnwJMDLLn1
Acr4iQRYWLwvk9FKBBmUYxFhhAKhiHGscdNOwm7t1EMPIcK7b709Wdloq6gdVSsea6gNEXvv
CIVQqe0snELwzpJRCoxlqXenVBSvbPUnitbHo5XRwBt1/4njtUb9xrsXPfty/ntmBAFBD+IB
GZBBaHfl7OczkLqHe7iHe/g7hbusSMHNbFK7+/K2a6S8t86VcZVSqlTXA0SC7QbsuaRD9AC4
kycEAO+cQeXY5+IXHjg6d/TQ33z/2488+GBrqnvoofuvff+tzcbaTGtm6+q1YK4z+9Dp/5+9
N4uxLMuuw/be59zxjfFijsi5MrOqsqYe2ewqkRRpkUIVaallwBBswCbgj27DgFH9Q0Af+uQf
YbgL1oeqABuwZMMWYYGEKbIokxIlqru6mxTZQ01ZOU+RMb/5jufsvf3xIiMjq7Kqm2q32W3H
+okX555z732IOO/tu/faa917++2JqxIgZY5rJRBiDdC2xrW9fF9K0PUl89xZvnWPNrYvha2T
XO3f3i4wmD5xobGyErQWJrc3tu59v1cO5k88q504D8ScPdW04fh+PxpnZnvLT4eE4Eejuj+S
IEZPQRhzXZbgAgvMgqJCqqAeFC1YBaPQTZocxBvD0Y39K1sbOysnFucXu4X6pBmHrnjp4hNf
+Dvh9be+560xF0/tSlGWWWIrcQPnvXhnnIspiG0AoN6zKJK1TlkAAjKGEg3nr7wzfvv2n8aL
sZmzQTtsdZvNOEiCOA7RBoLEQC4Imciyr9pz9ud+6cRnv7C+eS/f3MicF2Ncu0srK6udbuxl
tyr3AxsChypKUIrmMEskAqvWoo7ggRSk6qz8RSqz/wMLYQRx6nSBo7LCcWAyq1tGSy7veIlB
ANPEc1gMIENFj1JwNg3G/Qa7ZWNilaz2wioAUVW4QT9AE1oUkHGRZ16sIQJDKuA0zEpQ7+sa
J5mtOVL1AhFRAuGgrPM6V4YkjFhqp1KiOgNoAdJQ85q4IlRLM4s9gAN9c0VRUEAVZQEFQhAV
EPUiDCKGcisj9HlArVPra+tr1Ti/f/k6TcoTttE2YaomZG1YI14cIgTkCYjIAhKgigdCr0pR
2G63bZpsDUc5yNjVm8P9HGTp5IkzF87fuXe3qmsASOKYa4+sQACKSrNnEJwxuR5K3x7jGMc4
xjF+svFJgdRheQ6PWBcfsshnr40xh5U+OlxzpLSHAqqiAMIqMxKuMahqrHGgNftmM/3CF3/2
99/5H7/9/b/8W1/8m53eXHHxzNbV242TZ7or6/s7m93VhZVnn91697vldNRQaueM4HPyk6AO
IWzvZXl2d1Sxv7iSnl9upKg3N5uZX9F0uJ3fy67aU2vLayvJhZX63p3+xg3o2+5T52QpLuJQ
V5cb7YUTTKdBNt797riYjvf7d6/faqnpTgoq3WK3jVRNiqFBQYtABF5ZxXtmJBC2ahfa3ajd
3pqO+tn06vvXBsPe/ELXcW0rnJPw9FKPFzrDqlo+uw5rXQ0x8gzjUZ1NGIxjWznVrMTckQCp
JWZVtWQIIFSwVefm2+aP3/5WFpSdE3PUDB3XhqgZR43UNFq22Qxa7Wp+PozCbqPRajTRBONW
O0hbnUvPr1ob2MgrTtSXnqfWlmEzcqWEQcCsnh0rEZASiRJJIIgsOvsDzhS2EUSUZzkcX/l8
lJlJvWLT0scEVChk1m6LDpgj0UAlrnK/u9F0UxPFRiWsqwWEMDEhSErYhXDFWqaEiSd1XTnx
IDu7uQwUSCwZgyYNm73mXCs05BwUhWaVcYJeIxvGhO0k3Z/IZFo1m53ARIUwGDFxEpChkNFB
5RxVLiQMaabqRIoKqiCgLCCqqqSAiKAAXpwyW9LA5uwm4O1859Tp9aCRbG5sDm7cnatwLWpF
ipaxYWxkkViZSAkYZ//kZBRIgRFMGBJolCZBmt7b28tZamt2BoNxVVEaXXzu2dJVg8EQDBqi
uigDQBXwCkDICqwAiubBkwvhAwnbYxzjGMc4xk8wfjDZ/LBNDx5kpGYGxgfRlYh58HrmsToj
SwHOGqWAUVVAEUAB9SBTRdZWKk4YKjHGnHzhuZWnnn7ve+9/+oln28uL4fnFMBvevHX1iScu
duYXNjf3Fk6url/61NZ7bxdFQejBgQKhZyVRkDAv5z/YGA/GxTMnmqurSdTOr9+TURFNq1Ze
bA/36/Hu2oWz4dNnsDsZXbuVX95ZqM93l1YLbA5iqcL4zEu/sNfuXrl2ZW/r/pXbm5xXZ5ud
TlU3G9Ha4rxGQZGPsrJSrw2hxEMBUIIwknOVYdtKIpxrLwXd3dFwb7MvmV9bW0pTYcw3+jUS
NBy4nentcrITc7PXTEKzX7ul9fXlxkJQ1TDeH1y/geM6hQDrQGrSmguojCmndaMhaRws7kD5
zHMvL6+f2doa3L63ebMYd5tJvrdn9/L+zU2Z5mmrDQa9FlHoYytJZOIosqFpdpLWXBoHEBmx
KCFBgAheLCEZLatcyYAJxHOs5GpXi4qxdS3M6GrnvPMitbjc13kupm7YaTPoNBrdOJCEJ/kO
awHqyCB4Ei91le87U2XKTM51A7NC1I6CpuMGagQeiL2IQzM2OALODWXMhWcTWGWsvauFxzjF
UZ1WZcLaEkzBoFEGHyOdbDeXVldu7/en3k98VpY5kcmGLgnJeKiyPPCC7CfMUTNh8YJokAAE
lYEBWBEAiVSAQYSotmYYSu7ruBWfOv1Eq9sej8ZbVy+X/dGpMOk1ooQxRDCzOrYFL6qEiITe
Wa8BKTEgAsdRQdjodCWwd7Z3nSKF0U5/f2+a10G4fumpaKF37/0rxnsQJCIQFUQHwEgwc+A5
KKqiACCAIOr/24HUI/67ANdee2lmE/pYmvObR2wrPr4x/+isv1r//idf/fGzj47MVh4Z//Cp
PvRuj0x9OPj4N3k4erQz/HHLf9TBYxzjGD8N+KRAqq7rQ+GDh+W8I7Lms8HDeOtQBn22/GAE
EA3hg18PJgOwCouIyGgyTZZ7L/3Hv/ov/7s3vvXNb//tv/uKkqw+fXanqDeuXz9x/onl1dW7
G/fXlxdOPf/Zm9//Dld5M2xjPk2VSZVRBMDWVWOHa+er0yv25IL5zMVqY6e+uzsv1EJ784M7
26xzT5xuL7aajXjnxtbo+vunPDcWT1gwmUqr2770/PPzq2sbd+5u3ri6fe92tj06kSbEEMbp
fGcuCuLCZvl4PHWFAUEy08rXILUBKAHQQRyaMFxeXk4bjTwvxntDryVEkfRzm/sWxnv9Ihvn
14rdHS4XVlfiXnvj7p4f3joz3zm/2DaLnUm+U06yWAMDgQ2MktZQUQLBNHM8GNXjzf07aXeu
3Wp/4TNnK0Nnnzxb5sNqtP97o39RtuFXfu1lRs3LrC6LOpv8yR//yyrLXvqFv5GsLkEc/Ltv
fp2LSWwt+pqrgl01HQ93d3dX1lfQBKNp5rkui7LdmvOEKydO7mzvS8ULrY6WtHX/PpIKgvOS
Bnh2tbfUbpgwTEzUS+NiOtXa16UngVixBdCtfVqURTGBukobrV6atgVS1VjEgKh4J4oKDWsx
DJMoaKKpRI0NFan0VQ0KIFyLeCBFq4gCrKoIBLDQarfPnT355PnLd+9+78Y1x2zB5qV3tQSi
kuWROPFVaJCRQAnACzM5RseBoFEQVQdSoVak3uKE/MTCyqn1U+sntarvXr2R7ezHrMthaz5M
TO0M66zLQVEFQQ2oCiqSqDITWWMMIwiZ1kIvd257Z0fIijX7k3F/OpnU1cmLT37q85+/cv3a
cDBIkIiIH+ymQyohHmnigAdmNR81Bf+JwbXXXjoUNnrzK/jKKy9d+qjYzKOh0LXXXrrwCr7z
YxKlOf/qN/TVRy47s3i/MPOjfx0evcmPxl3XXvv12b1eeO2lC7OZ8Ng3efVQWwdee+nChZfg
6jdePf+45T/y4DGOcYyfDnxSIMXMQRAcKJg/2q93+BF/9LN+VuYzxsxY6och10xoSuGAQisq
KjKL0IgMAO6OB6tPX3j+l37u3Tf/zbnvnL74mWdLqk4//9Tmd67cvHJ17anzq6dPbN66vdyb
O/uZz+1ce3+0udEzYcyePDNJGWIegqB0dybFtJxOR3RujU7PJ/MNf2sv3s0+FfbINwHTPeVh
DO3nTst+cXXr+vx0uLx0qtuaHxWiZOzKUtBpLp1YG9y6vXPzxtb+Pk+LoD9xgt0kTFuxTVv7
k/3+uI8lM4uENiNX1EXSCIwBLqeWsNlM5tKwmmbTvbG7N5rLpMU2UG2iPUWtKrHFYKcuwsXo
xHg8urWxYY1bW51bPXmynvqtnfs6KUOJI2uiSNBWTvPGQvLFn1/rFeGIv/Pu97+b0tx439tk
bvPK2cSGTROVw5262Rpkgxp0YWFxPlqJTbi3n4c2/PTnPx81UrBWzfLyQu/GlSt7O9vtRnzh
3Nnf/Z3fKfNbL/zi31tcXh6Ox6xw7/7mhaeeytnd394698WFU8ur5f4g3+3/0e/9i3w4iGPL
rmINc27sl8ZPJ7WZ1CFJLKGRECAW7Eow72FRfZewDiNR6AXBPGIkEopaUARiMIpaq7KIkAFC
MiYiIrKsEKD1rnTeAQIFVlidB0QBJIyCKAm9wu5+v72+try8nG5tjqsaA6LIOOedrwOjPOs3
BDKC6ERrZwVDxQCsAWUVB1obmBrNrLjIxHPdSydPpEG4fe/e9s3btvDLaasbRYlA4tGwIQBF
YougaFVBRBTZ+4gsJrEAiiETBGmS5LXfHQ1zZZsmw7zYGO8PXLG8vvq5z35md3unv7k9C+MO
hWsPXSkPYykC0JnVN8DhU8dPJK6+99YjVuyPa+1587e++taRrv0joc6PE9de+/WHl314a9cu
vXhgsnH+0SzZg2UzQ5cLByLIX/3t37/26qtH7vjQlvhX4aF38MOZ8LjlP+LgcSR1jGP8lOCT
AqkHopsyq9nNxM1nhw6jq0Mq+mGO6nAhPFDZQYHZ7NmAJaOIh0sVuKr8pJE8+8ov3r585Zvf
/NbptRPJYnsM2eoLT+5evnntypWlc6fOnjl//9Ytr7py8el+Ek9u3UaviQiSsUTWaA2QGIwm
5fSD+9lo4p5YtEu97tMng2RU3N7D/ansDLunFjKjeVWEXTM3t1xujW7d/O5ic7nZWwu6vaiR
+GZkoihtdubOnL5z/cpgZ+deYF1WdsdZLwrac63m3EpGwe7O7qiubWQ5TnISaaQUUV04rOta
XASgzoVxagcurqqEAJUbCEs1lbZxz7av7GRkJ2WdV54KpK1JFrbb6cp6ulur5bA2VtUYrwac
7p97qvnKy39ry46HOh7uD9zIjftumEF/dDMb1/W4eu6JYlRt3/vuzWntbtvIQpiGTc282vjf
/B/fmJZV1GxXEp84cQFE8v3d6Y6X6XTj1k1XlP3NnWbScHk1nUxle1REmyUyldnEuQ92Nq+8
8/bW7VvlcJCGQekFA6IAPdQVRSXR1NW1FwowBExQO4SLiIuG5j0vAGgYiTERalK7kHlmHacE
isiGapESwCEUzAzoPTiXIyEaYmEABWsZhEErZjQ2CMMgTTQJN/NiUpVjFYniENAyG+d8ntVl
HSIagFnyySqaWkzNgRoLqCgOOQd2RkqCzHCd2GhhfnV9JU2Tyc7Oxq0NN54uBY1eO0oZE4+R
YGwMBQGzq0CByCgQAyqpgpBhE3gATxikCcZRVvqt8aggpla6U043x4MRV7aVfvaLP1OV5eaN
W9FBevZhtc4Y470/3DIAwADK/GPY5j8evPG7b77+8scWoR76zz0WR3NCj6twfflrX3v0dA+j
n4cFtWuvvXThq3A0yfXmb331rQOH90dwNPKbXfE3rhxNSs2OXzw0NH3jQx7yh7bwV3/3LYAX
Z4MPZ8Ljlv+Ig8eB1DGO8VOCH8yROhThhCP1u8PBw9dH5aaO/ooAhhAU4IC3DgAqqnCQpVIF
Ved29gera6uf/09/7d/9o//5W3/8J7/0pV+1abxbVr1nzkgAezduRxN/4uTJ7dH2PS5Wn34y
aDb67112zoTijADWYlWRNQmsRTF702lZ8eaIT65HJ3vxSm9nc8/f3Yh39hcW5qUdjUI3CjNa
pOZCe3enf+fWdrd3cu7kmWajk7Xm+0G9G1kfXKxOrza898Pxzr3t4f5wFanVbXa7y7VN7uZX
x1keRR2JosxLPh3FFhdbTVbNi1y4Tm3QbjeiUsT5KDJS1x0Il010otm+OxrcunIdjaZrEUBw
4+6Om6tXoxY225KNkcEqhjY01mpoSCeNeLo8VzVoenZNsGJLSQFBoYFCWI2LmK2fVJOidGiH
o2k+qbNxUWVcF9m08JOyGudb4yLIRpeTKGybSe3K8bY+fWZae853f+dmlhaOXVY2axhlUR1A
3E0n/VLRr8/L6QXtNBphaOI0iuO5yb69fXtcelVqqiTKKrkPwLe9zvt6yeh6SD0KUxYwoAjk
mdQbAVHwiB6pQp2CTkCn3tW1ahKrIS9agYCCeu/YeXFAgQlCZa8GjAk0CCkMa2MmZTVWvzEa
TopyOs1aiK2oKarjqg4AI9Wo8ilDakwkaJisc058iVKFWJBk6p3FZq97+sR62mjs7e/du3YD
BpOGk/mk08GwoSYxRCpkEQCUwASRQc8yE/FAQgQgJKOEGJiwkTiDW8NBXrgaIei0duri3mh/
z96R1wAAIABJREFUUBZBI/n8i18Ukq3NTRLmsrRkVHUWPBHRh3JOqjrbIUfVbv8acNTD7mNx
xAnsYxwkZpHHx+Co5cms5AcHBbQLX31rdsJrr7104dH5Bxd68yv4yoOC2kdP/LtvwEcd+j7E
hnr5dX0Z4NqVT7zbR43NZv6lX/7Sy/AyHKSmXn31/Exx/J0r167BY5b/iIN/ZfO/YxzjGH9N
+KRAytpHjorIoVHMbGT2ZQAzK5gHPI/D8dkhYVYQBJx9ZTxwZNWZ0dms3pdAUHm8Ox2d/dyz
z/zyL1z553/U+eY3n/vln5Om3R0Xq0+cTIVGd7Zdni89dXYnG167dffkibW1hfm9t9+tB/1I
xIqE7AORErkKMEJqj6UeDif7xfb5ZT2x1HxyJR043hpVtzdsOwkT4AZAN8mDKlpvtRYag82t
6Xv7a70TJ7qri1HjZLu724pv7W2Krxud5oRg7/qtLBu1sUaDttXuPXGx3N+Zige1UorXIEkb
ttUC9mBMyd6VVZKYbi8Z7+0lolFghCujsNJu/ML5U99+7+qN27fWwhNa47U799y60EoSGhJU
Ug4VLASBmMQ3W9yzk1YcGgwrUhcBMedKCgExYNTRgL1tc0tVTXDaRMIJqjEQgloPWAo4UfLg
8rIuSwSpXZ3nmbXkvHeeAY0CWDQpRCJcETsUDNEQWEALYIlsaNUAaPfKe/S///btvftDhpPq
2+RMojJn3ALokuWlkObANBANwExHHAmNggAIYG2oAJiIDJlzSyMvXjVJk9zVihR1mqiiIFZD
8B6MdWCKalwhVCLWOVMhQ5jVbnu0P6kqdtw0tp00WhSUVRlUPgBMjUnDKEVshiHUrq5KFscW
puhGIL4RJfOL66vLjSgp9oc3PrjmJ1lX7TxFzTSK1MRCoSCBIqEgikFPMzNjJWFgBTRARsGo
sRAYNpS5epxXk6oyYQzW7kym2/V0XNcUx89/9tPd+e6tm7dcXbMrULwoEJmj9gB4hCwFAAhA
xhzuOP5ryU59mGz+MdNefl319Te/gq+8MTP4/KswpGfhzkHa6Pyr//DLX33ljd987Tdevngk
DDqwGT0y/yA8evk3vvbiGwfFrw+XCx8fR8HLr6t+6SuI+B9oG/bmV155Ax6WCw9dTV988UWA
H9og/hjHOMb/R/EDSnuzF4ciCEcjpBkOO/gepqAeaE0BzOImVREi84AsJaBAoiyCKghoiCyC
KoynxaTZ+tTf/eXxnY1vvf99XWic/+zzPg7G42L+7Ikgjjbv3B2+/f1TF863k3T79lZrsbf8
uc/1r12f3LoT1XWCBNYJMjFYUMMaq0pWlVfuFbsDs7zQ7C0mZxYGeZaPJ0HmWyMXDIs8AGkX
tt1dXE251N3Brf7u3abtpotzF5YXl3vzgzLzqt2zdjwe3rx1vc3p4uLiaNDfneSVCAZRaBOy
2EyjuV5TSXLJmnPzaaNRjwZ54fqBMS7hoqq9YmApCZd67Wc/9dzZC+f+1Te+7oFFTNToZCbM
02R3t58QI7kUozhOQjQkNrtl3vmjTdernR11Qr/YjIMg1kjDpqEULYFa5kCMARFSIQJUFhEB
dEpACMZqiL7Z9JKIgldQ7wJRFiEQEFbPTJY8VB4hIVVQFDaMlgGcgiKIeECvvq54NB7lZRzG
ZMh2mq0nlntzboq7G2mpFrCovUVjLCEK6My+FwW0AshQx6pj5cxQYewoDDQOh1U5LvJ2q90M
rEE1llg9qIK1tdfA9iKmcjBl50LVsiz700mWF6KahmFLoYvQ8D6qXdsE7UYjCax6x74Wqabe
e6jU+goZm8nS6kJ7ecFE4Xivf/f969KfdCHoUrNFpm2tRSLAmdoqAiIRGGRUVnbCJGoB0Fiy
Vq1hMA6xRJ24eliXuTDaQAI7cX57NNxzuUmCFz732bX19asfXBYvxpqyzIUVLIKgiiJ9uHXj
4AUAH5Ed+evMS/1wePl11devvfbSha++8Zuv/cbLR2OU8xef/bgg47BM9lcaf+uB7/sMjyt+
fUwcNbvVLz3MJH306Efv9vA2Zrz6Q+N7OMr2evMr+Mpbz148fx4es/xHHHzMezjGMY7xE4kf
HEjNqg+HZb6PqzscyqAHQXComKCqiGAO0liiKsyCiIAzdxkFUAVwisjYQdrf3I1XFl76b/7z
P/jv//FffPvPm0H4xKUnq5D28lHzid56L9y+fHv7nQ9WV06cXV3f6vc3htPlJy40F1b6H3ww
GQ+N1dBp7MUoeGSHDA7nK5BJXe5m4/m+Ls0lJ5bTzmKZZVwU4/1BIMD3B1FvHC0v5M0A1gx5
M+n39+7fT7fjTmu+SxYbjWiu13n+hbeN2bh1Owa78vSTtLN3+86d/v5odPO+DcPFpTlXuM5c
GsZhbi0rx72etYZLn+fO5cOFKIk6rcbCXBga3x+cPb369//+l/7yL77/je9/d+TqwruGrxuL
HQdcqi9qX4k3iFZNsV0OsqJMeDjea3rZ0iBQ4BioRaYB1jBYkYgoDWwaBGkQRCaOTBCACTCK
DEQGIqwMB3GEEHlfAfqQicWJ54M/kjdkggqAEQgJFNCr8aQeFYNa7UQlAxXbnUyhqhbJNgWS
IA5OrvXWe404q8rcogtrCZE8GaZZulGUQFHVi+SqU4aByqCuOU0rQ1MKIIomXCXzvaDbzarS
glqLjgEtOYI0DG2EReYm4rmuLMK4KAbjCQMaawkpsSYykJL2GqkJI1LhPKt95UFK5BqF28Z0
0vZcZ3F+3igNd3b3t3Z1UswxzptmL0xiNahsDgwFQQ0iEQKKAqCSaCBAihbJIqohsbYmKFUy
kbGrJ96LDWwYO9FJWexOJpOisEnwmS/87NLqytXLl11eddvtcTZWYZ3xn2aNrvCIBfjROjge
CaR+SnCQO/pwZDOLXB5Ho5pFLR+tXn3y+A/OJn1cIHYEnxygzC597co7AC9emhUWH/T9PfbS
D2defdzyH3HwGMc4xk8JfkAgdZiL+hCLHB48Rs/mHJb2jjLQD/QOVAUUHmjkGGNEFEARzcyp
z7Mg2YiC0DkBvtPfObmy/Cv/1X/2J6/9T9/9t2+1gmDp7KkJ8Z1it5WE5z/33Oi7N4fXbhd7
k8WLp7NQ793f6i3OL33hZ4dXLk/vX1dkAiAVUiBQAAaBlOJGIcXdQb2fVZvD+sQ8npzPe11d
btlxFW6Psq3dfDTUxaYstqatpq7GzfkG7ZWDretYgoBtzM0vLcy/ePLsTpBWVR0nTXqyt7i4
WGTV1vbu7Tt37966//Y773bm2088eWbt5HIjadiEcnVsIVhdEYknWZ0BthRCDHbv3JNyvPTs
U08/+eTbd26/d2un7vfDhfa5hYUAtN1o0c54Z3fUxLAbx4FURsLYJ1S2GyU1ChuyqUG8qR0U
TjyXuTJDSN54MUxGjGEkbwI1MVFsNKS808xDUxR5GJper2tIxFV1VYpzhlC8KIMqguIstlUB
EeMg0KS5z3h9PN6qq5r6o3HTw3oYp1mpzaaNUsnLXV8OrBEII4J2GodRUKrUzDVwTYgq7Ku6
Zsm8G3k/VM6qysRxZSyL2kZHw2BSlGWeobAxwOLDKIzDwES2mEynm3s8zAKlejhw4g1qM06i
JO6myYq1kfNS11rmrnYqHkgzrnxkTCftzrWi+Ta0ApcVW5tbbnsQTMoe24Wg1Y6iFK1hRAAx
4AwDIgCCKhACkHoBLyQaAhIFBkEIasIKZcJ+5HyhWrA4QGPIC0/KcjvL98siaTc/9TOf7fS6
1997r8rzpfkFr1yVtfeiRKCK9CARdSQjdYijOiPwkyV/8KiUwVGO06zh7TG08lkN7pWvfOmh
/MEBAWoWY331t9589fWX4dprv/kGwJf/4avnAY5kjWaspIMzHc0mHb2TR8nmV99769EY5Mgl
z/8A7vvsZt+7CjDroPvar56HBzW9R7nrD/la8Pu//RZ8+Q9ePQ9w/nHLf8TBYxzjGD8l+KEC
qRlDFh4IHxwNsA4DKXhQ+4NDDfSZruAsakKa+V8gAOJB47cKghKIkAGWihAMa7E/uV3Vnz5z
9le+/Ov/59f+8Vtff+vnEZfOnqiQtyfjiuDEpVNRp71/5c69t99tnltfXJmbDCbsp/NnLrVX
VvavXc729kE8CSACg6uBBTgF0/LkJzApi3KwaTaL7lrPLHTqJNYLPTy1ku31y/4g2M3sXMPN
p0UUNnut9uK8H5fZ1mC4uz265wO0DaBAhG/ChaefhfXlvTy/dOHUyL1wb3f3nfc+uH3r7tX3
d269v7841730zKmlU0vYjrmzXJ4aFxvbkys3T43y+UpxnGWT8VYti8+/8JnPfOb93Y3duhhs
7V/eHS2Dbc2vxHNhPq3yLHe1dKPQ1qZlQ2AD4jXGmr0KWEUDiWrIJlSt0IPyzGt45jMiouJw
lhIEMdaxv39/L0zi6GwHRLkULlgdI4CosjgQVAABYFBH6gjqMDBL6T2PV4d2o7ZTNDlUFATW
MJGEhi2IK9lPJSi1acgGHlLjohAhMBgTCAJ4EF9VeVFO8nxaaAE4UioqDwh+WsHuRIXBOxCx
SAaUEIw1KXAGDM5pVcdIcRi2U9uwcTOKYjJGIWKJhiOo61x9Br6wUkXoQqRmq7PUa3Y7HqQY
T+p378hwarJ6CaOVqNOlMGI0CgQqBgSESdQAIhDM3jwrexQgRSADRIAAJnAEQ6gHwEPwBXhC
EsAwjAvv+2W+VxUbXAWLrc9++jPzC53L3/9eNZ2uLC8HBvan5bSsCiFEEwCSZzQ0e+CYbS04
FDHXGYHwkd6OH9/O/5FwwBM6pKUfLXw9xPlXv6EXv4IPyeuHRKqXX9erl1668ODIkfFD/tGX
v/a1F+FBLPXy6/oHgK88qO39sISs869+4w/ew1cOaoKPv8kPzcU3Zud/9TwchHhHi4ovfu3q
N159/Q++/MbBOR+e8jHLf+TBYxzjGD8dwE/4sP7Gt/4tHHksfsh8OsKaeuzyR9Q7VVEeVirw
QdMfM6soqApzEBkF9o6d83meqfKJ5ZWzSysbb7/7f/2T/63r5D/6wosLJ1cz9ZOiCCGYD5pJ
qRs3b+/s9xu93srqWqSmzAu/3Gp2Gm5ne3L9qgz6YV1adqDeECkCqxgKEUO1QQFaRmTmW7TY
CRY62EodgSCMx4PR3Xu2qsJWE9pN7aTaTEySQOXr7aHZmUZ7OfandUadxfXWidWqnZTthus0
syAY1n6cF/3+ZHdrdzgYFfU0XWjNrS8trszNBYHZGeqNjW7ObRv2s/G1rXu7UkVnT47Qf/eD
90ZV0e31lud6Ue1apU/zMs7LpkDTBs1G3AjDXrM53e/7Ig8toQh4IEVVYhF2pfjyQB5bZsKR
CogKqIBEBg2BsbX463dvY2DOnjuDqpKXUhTiHQIKqogHBlAUBAHxyJXRIrS0un5b6N2Ru1Ob
iTG5OlZPAJZwdb53dnklKCvf7+to3AJeiJMkSdUEqgzAxoAJrQPN2E2Zx5Xbz4r+NC9rrj2U
vnZcO/YsHglCohDRCgSAEcIcc1PVBCaKbCOOmlGUkoGipKIIWa1jqV0tvgCXWfXNEOZSaqfN
bquZJFC64cZ2tj+wlY+dtMOwHcQNMLFgKGgUCVEQGNWDAgEaFBYVBVZQVAFFBGPAGCVUoJJg
KG5Yl1N2jhDAeC8UhDn73arYLbOBr9Olhc9+4WdaUXDl7e9JXsx3O61Ot3B+sz/anUxyRQUK
VQMAMuZAOApAVOmBcNRHQUTfee/af/jm/uHwcbv4GMc4xjGO8cPgkzJSxhg48lhMRDMWFBwJ
qg7p5x+qR8wmP1j4iDICPOrHqgiOnRphEoua1Lx/bzMTextg/YufesnwN//X33nr23/+N4rn
T549lUbJZja6bzlMwoVPnUs3eztX7tzbeXduZaV39tSE/fbusNVbmF9eyW7fzm/clP1+4uvA
siPnLYNWjUoJESME9W6/z8Oh3ottt629Ji91oqXe2lJX+6PB1s50c4/2bLw4B+1GZSjstTpL
i2HOOpj6+4N8e3v33duOFQIbttq22ZpvNubSZC0K6tV03AvG5dzY+Wp/UhXlTuXrnf1Wzml7
oXH6bOmrvaL8zs3Loz//M3Heepm3Uc9AG7jIsv1JNq7KeQqiIM5EM6rmg7BBodgY0ZEHIwQi
CqgAMxNo1INsHxyY8YiKKiLMEoEICh4JFZXFeWWjKuAZGFQAcVbNIwBSIKGZ4LYo1IDIBGJq
XxVeSzQeUdEKADPvjzKU7QYgFaVlZlCtfYM4sLHz7JXFgGFl0pxd4TWvJWfwiraqg6IMQGqU
ElzFFbJGCE2yrSCYC9O5KO4YSgEixEjQ1AzTgl2tKh65EO8BXKxVhKbVbvQ6cSul0KpqMRj2
r93T/UnH00kKExNFzdAagwrgxCKagIRFiIBACRCAWLH2wPKgydBqQEwIQSCEFbu8rve4HqkD
VisYqWGgHGjseCOf9MVlBGvnn/j88y9IWV393rvZeLy0uNCYm3MA+9l0MB17z4aCQ0agPkg7
fVQ6BI48t/y0MaWOcYxjHOP/p/gBgdTsg/4oEWp26OM+4o+SPB708iHJjH+iqgdmewffHw9O
pCiAouLBixTVzt2NaVE8t9x779bNCy88343b/+r1f/Kvv/X1F4vnz1x6stdu3K+mYynrslru
NC688Ozg7ubm3u7GaO/k+qnV9dXdcTZA7p07N3/ydHbj5vT2LRkPAjSAbACsxdqXngADExBa
DXBa1qMcdmO9t4ftlJeawULj1KUnfVVPhuPNra16czdttbDZmKQJRiGtJMHphi2X2oOp3xxW
G/u0v0H3VUUwCGwaUxyEQdiUZNGEFUqp7L3HWqzSZND/xqB/zxd3XYbzcz1pNJ3OeRvlrqWJ
TOFuP3fq0YYujEdOfVkgk02b80CKhsAa0WCmGz8LmuDgJx0EUoAz/XhEARUVFAQEZgZrCVBU
SBVEVURZUARo1kImM7nMmSWiUWMUiKzBEBAqgVz8lK0nRCJhAcUyK0fTIhSJxTdRO9YWaFqV
S3wpIozKwqpS+TIr8tp5V3nnPNYsZWbUNYNAVFR9ENh2EHSjuGvDNplUKQR0vnKupMpbj7Eg
itbqJyjjUIt2QIuduNdqN6xBNDXzeFrcHvnhOMjdAti5MOk0kwgNqZJTmPGTjBUABcDAABAA
oAiJoFf0QECKpIbUWg2sJ6xAcl9nZZnV9cD4ymBKNgIrDDnzrrpNV+5ypXF07okLT58/X+zs
bN+6yVW10ptvdboYxVk2HRVl6bwAAntUIkMKMHsaOaSZz1RFjm6fT95ixzjGMY5xjJ8o/OCu
vaM6N4eDh5IHhzYXs6+Ew0Dq4BQIKspHhXMeHFJQmKU+DJExjitTe6l4XEzvT4dYZc8Un23H
4ebNjVNnz3zpH/y3v/uP/oc//P6ffa7On33u2VONzv1sXLpsUFeq6fy5E9Fq7+69u3tX3is3
785fvNjpdYfjLIvizqdeSC89ObpxY3jjGk6nNdfOVIZQAQKvgKpVZcg0yZpCbOXc9mBwc7tY
jN1iGxfa6fLcxVOr+TgbbGwPtrbU2EZvzraSUcRstL0Qz6+fj55/gvfG043dYnfgB1MpBjDi
EGyCgZoAk9ATVJ5VyAsWSOOqMlKnxFVdIusiJusmbkXR0tyC6aTsy9vTPWehCMmDZ1brXFlV
ohqagFmtomVVBaZZLKWIqEQoAqLwsBUScNaMJl4ARRHJWjSKasioeBDFWbe/qoIigFOZFZlQ
SRCVFMkoIBrUmTg9KaMBQCQENYLi2B+sVvW1G9dVQydzJkQSVmauRJ2K57pG72OnLSUSCGKb
thqNMEyAIoYmUSRq2aNz6gr2noWJxKrUwgMEjciHBtMIkihqpo1WioF1zNO9fTee4DBvFDzP
Qc8k3cacBTCG1KADZdZQSB4QxwhJCRGRHYsXZCFRQwQ28qpsQKxlS7WhjP2wLCZ1WbJnAkFj
gEQxE85F9qXe5GqL83R+/rmnnjkxt7xz+cZ0ZyO22G7ErU4nTNPt8XhnOMwqJ4gqSgKG4IA4
eCRUmiV9j+RuHyZ9j3GMYxzjGD8V+KRAakYwPwyn4NGqxFFe+aEa54ccjmfmq7OvhYO2vQf8
dENGgVUAEb13RtQI5s7vZdOtYvpLn/+Fkwsr441tBr5x8+bapbP/yT/46p/+L//s61//91z5
S08/uT7XGmvhvd/Lh1NXNrq985cu6FJ769qtu//+W83e6tyZM7TWLvKKk7DzM5/vPPv8/tXr
o1vXxqPttK6bThteIwAhLUk8sUelwMRoOp5oNy/2J1W0VbRiO9duLvbOnljTs6fHZb7T7w/6
uygQpXHVoi3JODKwkqQnLoSlb4xLOyjq7cF0Y9tNR64ecCYIBAwhBQ0MezZaSsJzEI1UJmky
cBJ5mvMQom/UWYThcmoGhfiAoggZAZ0EzqOr0VcBMHtHZEAZhFGUCEkFdZb1E5VZmySAKqsA
ACAoASpZDAEwsMY7R6peBAUONJP0oByoBAyKioAqAEKgRkHqMAgT4hR9LWjAwkFjm6KIFbbK
hr0VT6qg3lXl1BdxaAJjYoLIUEwUJVGkcUo2IRuSUYOiEngOKm+cN56FPYOvrVQopRVHqspC
4CLDzQTnmkGngcZiLZpVfmcfhjnmVa92EVEziJtR1DJhiEgsszIlqxpQINSAQFFBZ8bHyqTM
6sQoWCBU8orOkhqj1nrUTHlSFqOqzNhVwmIIyBgyKFCwz7zbcfku13lsls6eOXf6bMPR5jvv
y2CcWKAQk3YjSZNp7frD0WicKRCCEV+LKBhFNIoHu2C2xQ6fPWY43FwffiA5xjGOcYxj/KTi
kwKpj3q/HG3QO/yUP4ylZjjadgQPdHEOiedyqEAIikgzno9Bawmdy4uy3NjctFH05FNPh0GA
IrE1eVXdeP/qqbXVl/+L//Kd5dN//ntv9qeT5597prvQK0LImDPvqvG+Rew0w/VPP1PtjYc3
trb/8rv29nzz/Jnm6TX0kpw6lT7zzOLW/f573xm99wHvDjCvua5JWRHZYA2Sc05kjDUhBWkN
NGXdGVfBJGtu97uRLnaSlfm1pYVTZjkv/WAyHY6mrihtK4EgLK1ESHEjSpI0Wp5fevYJzEfF
eDDqD6bDgRvlVeEjV+N0bIRaQdw2QU22bDTESBJT0DSKjuv99QShndSEaqAArZWxykxGkDUs
mdB4YafsgBmU4IBYDjqrjiKCAiHMRDBZWBSIiAwqoBchQO88e4+iKooKhwlCBLCEAgg6E/hS
QVHhUF3qpVVk7aI2FAigiooIqRqFQCEEDYRjglYUpkGYNGwT44gotDYmEyESM3o3M/ARLrn0
wIqsIAysoupJa9LScGGgTgw0YkqioBmhQYOAhOy92x24cRkU3hS+4bFNYRsbS1EYoGGDbMCT
ehAxSAAG0AqSigJ4oyzKwsKgIgCEQGgMIikaEXGG6iR0Ck4lc9WoKseuqkWYSIMQiVhVvGa1
m4rb52KPy2Cude7UibXlFZ6UOzfuNSpJCOsQ4vlmc67jEKeToiwcCqkCKQGSEqt4IIIjRb2P
qofAkdDqaBr4GMc4xjGO8ROLTwqkDunk+CCfBI/GVUdnHv1iOHrosDp4sOSBvxgzCwsBkiEU
yoqCrHFe8/3x+aW1U4urQjqNUMtiIU6rcbbXvy1rSz/z9/5O78yJP/yn/3T7ra9/5qmnTp45
22m3JlmhFdcV75R1K0mb8/NLy6tur791+869v/wzut5ZvHgRAeJmI37+2bWnzs9/7s74/fdH
l98rN+7GeRYLh8IhEivPwj2jNrKhReW6RmGdOCjqam9aXd2ERhI3G7TYW1qeP7W0mvt6MJ3k
lS+5yo3hOHaRtbFBwLi1RIvzrSegZxCLstrdL/eH1U6/3BuVFRv25Ku4PyVEtliiKgk6M092
sdUQNF6gNpFPG3XdUuYU2QIXnBeVC1SRlZRICWYGLIQEiIBECsI4q+whAiqiqshMHyoOo0mZ
ERok9KqzfvuZuxuCWgIFEQBGRFBFCZhjrltMc3WZO5YAIguB0cDYyIQRmYhsYjBUSRAiQqMC
6lENeaFayLN6p86pCis7ZSYAAkFwJBWJi4BDK1GASURRGARBbA0oqIh33ueVZAUVdVj6Zslt
E7UoTEyaJIElYxQjNgcUexFRJUAEQUQEQlQEUhCdNYeKkAAoAgEQMaE3Fq1RBGdMhjrO80me
F+IqFU+kQTCj6rNIVbvcy4jdvi/yEOZWV0+fPZVSML55z/cnLQCL6lGDuTRcaGkY5NMqq0oA
smg9M7PMVD5VRUBA6UMPIR/dOx/iIx7jGMc4xjF+kvFJ5YO3vv2nRIRECDAzbT36QX+4UERm
rnwzu5ijSSmEA3GjB6dUmtF4ZgrpB0awIFJXdclOrr975c7lqz//sy/94st/e7/OtgfbdX+4
rkE6qieOxzHiQuP0k2en9+59+5/98713rpxeWX360rOdbtdlhct9IVSxGINJGs2lccPY6d5w
9/7WpHI+TbpnTs4/c6lz/iKtLkDIbrQ9vHp5+u5lvXY32NhvlByLkmJlsYoIAlICEImUQq+m
8sTCwA6hIp2EtkpC22mGC20z35FWWsVmom5S1aV4NIZMaE0YBSEatQShIWI2LEHltCh8VtXT
nEZZsjmoppN+NS24ZvZQ+8Bh6MkKEQVIViz6EIR9QKCuriZZgEReLAMpWjAoWIJUpIasIQRl
fSCIKiACB5U/pojVTsp8a2/r7LlzIOzzgjwDewBUVFUxAAgoiIwoCEriUcJu10XJXi05RYIQ
amFUCIhEiQW8g6om740wiVNmFvIUKItVJEDPXkDFUE1aGvEG2ahY4shIaCUJOArEGhbBmoNa
TO4or01Z26qOFEJjQzSpsY0gjNEGgKRKM4shAGeIAQ+ZXgAH1ise1D8IoGZaVgiIYpAMgxFr
JDTOmlIld9Wwrkauduw9iwIAEiN6BQb0wlVdF7Xf8/XYou02Fk6sLC8v1aNJtrGVVj7KVlFz
AAAgAElEQVRWFGUmqS0Ea910uYcQDAfF/v5kMqnYq3Ou9jWiEqqAAhAgweM68g4TvR/Ce9fv
/j+74T+KI3t5prP5CD7Gee+ozPhHVz1WhPyHnPYj4hGpUPioeudbfyVTwEdP++O432Mc4xg/
/fhEjpSIQaQjspyqs276Awc9VUVDDwmzogoqMpPehANCucgsriJEQlURQgQU9p5g9pzuQSsD
rpxmo8G+WLPy9MUJQQUAjCGaoi6NkYi1WWt/c++DPDt34dyv/ddfeftff/0v/vCPhm9969NP
Xjy5tABNUzFL6dBLNan642kVxY1G6/RTnbLId/Z2hu+/k73/QWthNb1wuvHpJ5tPnlr84s8t
fOpny2t3q794Xz+4Vd3fDspaXaFSaaU2CRjRGZRQASkQsGBCFsM+KpWLmge7/u6Ojy3H1nST
3nx7cWHOJ3EmkvlyYrLSIyEioSGaUZSiRhjP98LABixU+aBCU9aUZcVoUgzHfpzhtIKsrCYF
l5XUnutKygpVLWAAaNEgqwVjAUiExBNgZFTJs1bKhA++qQGQQEUURBRRtQZ0iWVTFW7Yb7Vb
ma9BVUDoQNUbYLZOhRQAlRUMoh+M4P9m702CbLuuK7G19zn33vdetj9//gYfLdGRgCiprBKr
bNAlV0iqUpC0I2qkCDvCUQNHgGF7QHigmYaa1QQcCjNNNbAVjhBph8shO6SCrYaiRIEd8NkA
+AR+m81r773n7L08OPflTzT8IEuShSq+BQQiM/Hy5s3X5F1v7bXX0vlhCFnEzGC58GAzYmBs
LMYqIyCSgq4EHpUqpuJx1Ku3QjYRTWQdJWhN1Ea6e06+WDBlaXvtUjRvTMYaxyFuN5NGtdIQ
SuwShe4sa36AqKqG2sVJB1gpVQzMZmakMVAiosMZTUMAgluQUCPWWWXpedb309zOu9WJpzYy
qgbR2DOSURQ5d54XakdpNbPcbjeTa1cffuhaIzq9ccuOpqNkY0Z1E429mVRax5GaGplT37fL
1HeilcMQxEXNKYMcpmUUXpzm73sbw/e91v5BRns/ngW98PIbJSry+pc/+8wz8q3zlOQ+P/nq
F+XzLz3zxWc/lK/cP3i52WfxU3GT61/+7L959t99NBH6QPPfBhtssMHfGx5EpGKMZ3/Wz+Qo
0q34mM+9qx72j9bvtpnNzIn7QwqSosM7X1lv7A/2KQHcxW1+enJ8dLS9f3h49Wrr3pmNx5N2
PkvwVMuk97rt9yo5vTf/1uyvH3ri0Z/73K8+9Oln/+R//l/+7Te+9tzVhz/11JO7+9sTrVbz
hZtDqpNVe7qcqYbdnb2HH3/y4cuPnB4dHx0fT/+vdyd/9s2LDz+88+zT8elHx1cOx7/xz/CP
n5t/41vz71wPp6eYnnjb586iCMQopMoyRgZklWwcdToyCdTAVC+TL/v+3iz94FaqK92eHO7s
XNkadxcnaRQp0jKvLHfqKcq8CotRHbZGoalEglTSjKr6wsWtRy7tkZrNVj365G3fr5btcmmz
VT6Ztotlaru2T2nVssuanV1ShkCBWcipcgE0aIiiAGkIogESAafDmShZgOxj1CEh9hKzklSK
qirFwQwaAJQMBACg07IDBhjo5paimMIBVmISXcWiDiKWgiJdJauRFPtbWSkQQRiSVz13nWXT
Noc+I1t01EDjUkNHMhqPqnGIlYRQoq/AgPLDKXQVQVBCvdB4uAZdUzknxZ00U0gMMUJpnmge
ImKkaA7aQzpPp6vVLKWptS1SFlJklBTGIKIhZHKV88z6E++nOaVxHO3vX7t6dWt3d3k6v33n
qE4+IWqXgCFrlhSNVVONgsS+Te1y1batmysCCEEA4DSsK7zlXPn3eeb0oWbEjwe++sXPv4IX
Xv69NTV5+ku/9/LvP3O//eU9KH0uH2zMe//NfuvlF1556QMNfQ/AWlH6SW776tA/cx73u4Y3
2GCDDf4O8dEeKQ7sB1LSAssu3jo5MLvlnEMIIQQRodO8TJdcVM/sUMBgqiLgZsU6pSplpy86
mGx6cppzfvbZZ3d2d6ftimYhKFV7oVcx1wxJJ4Za4ryz22+8ee/e3Sefe+a/+p/+x2//yZ++
+r/+bz/8i7/4Tx56+NFHH97Z3Trpl6vUIbHJFlKY37o702mzvTO+fOnxKxf6o3m+O1+99sb0
m9d1f0uv7I4eurjz2JXR/mR+aeJb2H50n6t+dXTanS60T5EKo7iSqlFUqq5hEoh7MKglTVYj
jBxYOVZz3F1SxdXDuKm3JpNJhckI282qwkK7pax6PWnhPRyxkhCrqqqbpm6qWAVt1GvHTqVh
P8iF2qVa0brezQPEU86rdjVfdMvW++RmadXlVWerPq3a3PU0R6Y6JBmMlUaYLxfLHs4qoqrn
V/atqU6CrRrAEEIMIQjgoDXRysNMoMiJgIMAiqKWlKsaSUGBCwp5MhUTZNKE7gi0YEktI7lm
08yYWWXGZGKsKJWEiggidTWq61iHWEEiEIAAUYeSIHtI1pLMDiGFiKoCgcJJo2d6h2ygEOKi
1OiopCqaqYQgQVSY1bNI75z3/Wm7muVuBcsqOcBEM3OdZZKi0VOUhfhJ6k6tm0teBK/3tg8f
fujC4WFou9MfvrWczxuNI1fpXSkAKSWvC03T1M0oG9rFsp0tvTeVIE44SzotAcDPJunnfeUP
EKU+Ljapr/7BK8ALv3m+/60UFX9oJfGH3PonwPnB4Zm6VXSwF1549dVX8cILL7z6KoBXPi+v
vPgV/tbr51v23ocXX375tZc+cG7nR3uDwvbyp1966ZX3/Mj7J/LCiy/ilVfeOwr8wy/K+2//
4We+wQYb/AzhI+IPhn2ic7MGgej93T2EEEpKQvmSw1HmF+vbnL3zdlLoZi5uRURQERAkPOX5
0enRnbt1VT/73KcMzJ4BuHNrZ3veLxe0PNJGa0xX1covhbDluuhX1+9+/cLj157/579y7Rd+
7s/+8H//+p987YdHR48+9djlKxebKqJPnHdgIoJRTmYnpyd39moZX9wPFw7HTx3waNbdPpFv
383feL0fRWxV2KkxDml/Z+dgv97ZSavW2i7NW1u0bFNI3nQIiMtoqaJqVFGBENnczLxcW0NA
CLKHKrU5LY9NYFFyENRx0lQ7W6Nq1CBoJ5yHtiPNLdGWQlO6wqJIHbWqqAREtIkhCCSIxBCr
rVBP9mrfAYeGYVeYI/fZUkJ2untnzBZcrE/L+cIWCzNz1ZyzzbenZhqCYDeIMlZSV6G0TUdx
ABxiFEhayjlny9nd3OluYAadDpqruTqCmThpgBPmo5S3cwpOmEciutQaRhoribXGOoaoEVEY
1oSN5ZkAAYo4WboZVTSIhjIOI4QuELhbMkeJdqAEKKiUICIUikKCqTLGHEMW6dzmqZsvl/O+
W3luyaRgVSU3pwOoqIpoIS49HffdsXdTyUtltbv10OHh3sGFGOLJzbv5zp1J6g9iFDpSriXU
UJpRxUUNHqo6hGo2nx7fO1ktWoUoNDvgFJWyXCkQDWruJEMIxUp4nledzxkBUN6Z/H297n8a
XH/9NQCffvY9jOXpZz8NvPra69dRiMMr9xv1cF68+rEopcSlRfi8sen6lz/7zOcF9xnJq/jN
N/jvnn5fb/L11x908Ge/9Nsvyu98+bc+94UH3erV38fvkb/71S/K59ezwK9+8X7L8Rfl8x96
++tf/uwzL33+s8+vb/Rjz3yDDTb42cCDiJSZFf8T1qYoVY0xCKRcA1SLzWmYVnipIiZVhoip
nPP9LSQROkWExS8V1hVjJf16vjo5Pj185PGHHn10lVKfDFSI1JMJptWyXdXNeN7Ox02oHKH3
/YzJyrYCj7751td+dPPqz3/qn//r//bOZ1742lf+j6998/UnfnjzmWtXLx7uY68+WszarguW
Rx4roy36434xj9Ls7e5e2r1waS+crvKd4+5kmk5mnIoIbDydjRuZNHFr3Iy2m/EuLrq3PdvM
ZeerbtKtuOydJkIRpVTmMNIJAdxycoiWYVGoVIX0ztBnTHvcWaiGIFpHqRqRugp1RNAsyPAE
7zx3uevNzHMrXEZQBD4Q2RhCgDhpOZe7N4tkEYhUMdZ1HWPd1E3TNMV9Ppaqmuxo0JTybDaL
UjtZxZhzdjPx3vqut+zmKaWzDQB3B2l9Nss0F4KkutdmwV0c0RlNoyMYG2rFUIkGotJYh0qD
aB1iUNVw3kENwtwgEBWn0ekCCVqWDgSAlBgsBIkKlbKKkB1OgYN0ESeNJBiphKhq1IgQsqhp
6IElMM/dtGsXfZfNstMUXgUTGN3MFBpFYU7jCnzHV7Oc5kyr4GFv++qVSwf7+9GlP5kuj2ZV
1+26jDRappvFENUJZhKEuhiCatS+T7PpfDlf5uyikURpzRECYFDlevX1zCN1frR3fkr+DzzX
u1/RO1iafqLvep+k8z4H1Yce/Kzyt2hYL//W5wDg6S/99osvDcSm3OpDxa2PmtN97rde/p1n
/s1Xv/D8A24zHPmZ518AyozxvJo2zB4/ePvhDL/1BvDGjzvzjT1rgw1+hvARo73yRx/rHIRi
jy0ol9vM4V21mVm2s9QofKDkeDCIoMgo1BCKWmVucK4Wy/ly+YtPPFZtjVbLVjU4kgBOr7cn
p8uFdratdSt9qtkEjHuE7ONVuhzD0a3l28d/efuhS49/6vl/9d//D+/81WuvfeWrf/rN1w93
x489cW3/0kEzzv2qtW65tFyPmorYT55u3ZveOpo2dbMz2X5sb+fxiz5b5bun3dE0na7SrGOl
UsV6PEYVPUgcN/VkFHbHYt7Ml7pYdV3X55Qs59QLIgl1CAg63VwpKiEoIAQCoSKgDOZuUrKx
zaar4ZEQRGEFbgUJQSUESGWBK0slQLw0PeecrfCenCybWfkMABzM8OTIQWcimRRVAtldSUuW
UoKbEolu2YbZFKQSUWIr2bqzT0SgFC0xCJCijRDUOmqABAkiUUKQEEsiE0IQVcBLejhKqL37
WmQZFjlLxhVY8gkQ1AkCvva7s9ALQM1RZMwi2xRvnQgVRKSIQOgMqhIqhpCIlftp105znlpa
0FZ0E48aPbCM4YwOIoaYk5nRjTnnO5JvRO8qjHe2rly6eHDxIELae6fLk5nO290kIwYJNJqK
UoKZORAkuJIKc5Mqhjo4uVy2bZfNGAS9uZkXXqiAqLg7zf3HlwScf92d16j+/8YHzeb3xaf7
Xz4nU73x3u8f+MSHm6Q+zMn+oYrX3xpPf+E3X3jpD/7w5Z/me36qM3nt9evX8fdx5htssMF/
YHgQkTrLLi/0qEz0Up9ADpt6ZhDUdS0iOWeFaHnnnc3NuD5CufyDXi6jxStVNA866eza/vbN
O7FpHn/qSQ/a5lyGOEHQ5xTrkcZqdXe6N9nr0FsTlin3niZjiZAx5DJjM12eHr353ddvHD/1
+FO/9POP/pOf+86r/883/u0fvfvaG4/t7l27enjh2sFyu54vlimJtLlyNhpqSOrT6s69VTia
VvXOeHv70Sv104/Zqj29cy9N59obpzMximqO0jURo4g6zoUylrCzpSEqpHam5cpXrfedpB5m
aiURwMUJWZuK4DaUBEvhGaDL4NwfpAvxQi3oKAGSvqXmTh0MZ4QE0drFLRjFPcCjl8bdDBpI
MpPJPRerGpkHHoZMCGIZqKIWFZG1rVlFojHcz9nGsJ7pPqiHEIBaOhNBh7qCKiZwwoAMNwqA
dc9Mca1TgABEigIKJWniOcjwzBr2OymqEAVpbtmscihgXjqFVGIoIz9qgCpFqAKJK2eb0nLV
zft2kfoWvnJvQQ8BMVKQTchcfgSJbMw5J2CV0zKl3nFakfuTRx+6crh3YRyr2d17x3eOqt7q
5KMsI0iEZIpJBOmihIAwEQ1wFUPWIFUdM7ls+1WfQojl3vKhE8mBEq3+3qDadVLU+2jT+774
sUCxj//+H17/0hkHuv6Hv382mPvb4ukPI2p/B4f90m+/KJ9/6aNv+O95Jp9+9umn8fdy5hts
sMF/WHgQkQJKr1rRFwDA3bUkZ5Plq0WmWscx6yBVrd9KD1eI8o2CoIGkUlWgopbN3M1tNp/d
Ozq6+tDVqw8/sso5uYtRoQEQCS6+d7A/P1kcpWUVNObsKSnNg6QGOfXofUfCdqV3V0dv/+X/
+9Zb377y6Z977Jd+4b/8zGfe+vOvf/OP/u83f/D24e3bj149vHTlMFfstU+rLvcWjRV1y4Mn
y6v23qy/rfdk1Ix3d3evXtx/7Jquun46XR6dWttqTjZb6oyxrrtKch3r8aQaxWrUVHWzc/Gy
55yXi7xcptXK2lbSEp7tnKBSulecFIe70yzKumN4vX2GstUoxTtGEMiF+MCFpaAQg4cZQg0A
RIUGUS13eGkhUaJYtSnmRhWHezaUChU6rNRRl8+Hb1pXS4MkBEIUL1GhAmpsXNRhpAGFKzjh
hTRIWG8lcG2mFjhEFAIvNXcQki5ahM3SvUyWDuFh9qWu0SHBRcq3FJdUoKqEYBIS2eXUJT/N
3SLnVepb5o5MdAZ1UREJVGQKyaAgYQY3N+vMl/QZ/cRyP2rGhxeuXjm8eLgXss3uHN++exS6
tO2IDiU0ooe37kIBogms0CIApAiTOwOqKkDYdm2XssYQqsqhFGY6CQlBB7HtXKztedPhe0d7
7/v444HP/e5XXnzlXKbB9S//65dexYtf+XAz0E/tNi9Ebdizu/7l33kFePG3v/QBravQnOGT
619+gNl8fdwPDOd+sjMplLGYuM5jWAUczvBffQ7AjzvzDTbY4GcIDywtdi8hPuXjwo5CCCDs
vuAkZja4qciSyVkI1nAQUtxBagga1N3cAYG7040pWddPZ/NZ1z//5NP7Fw6OlyuFOF2D9l1f
1zHlFMbj6qFLd2/ePvR6O7HuFW4Z1ot7RFCNvTC1u3tV7Mc3b9393jt/dOu737/6/HMPf+YX
fuM//Uc3X/v29f/zj7/7+g/feePGlSsHFw4v7Ey2vLFu1XWr3mkxVs4SSkSbL9rpbBZCXVdb
4/HWZLzz+ENBNaS+nS/62aJvO12l8bLH3HpZpaqKo6YfNVVdN1WcTHY9Nrletb2a98jm5mIU
GowKiCMIoMGp5gYSAtKG+xderrY+NLWII0AoEFnfpWce/sGtLCISxAWAEEqhU4p4UkZyDJme
i7RT1BGK2MDFytXdBUncQAzTvXLVF4HLcBixaAsZnOgCLTNLJgegWM/niKLaOMCgDIMUl+gO
GLO5x8zaBUBYb74BwBD4qiIqUTwkqlHUXDKZKG3ObUqrnNs+d6nvHLOgnQgFpuoiBg46n6/p
HGk5mVkyS8yLlOawlUo/qreuXH3o2rWws121q+7GrdXRqSQbm1UusdibBBaFAZmqLuJwERd3
qIIgszPDNIYqRs+e2gxqXTfU0LZ9MvMiuPm6/pAMGjAQsTVjPTcBX2tRAM5WNT42jOpzv8s3
nv/sM2eO8rNIqTXeYzbHi1/5qZIr33f0H7f7VmjO5+WVF15+4/d+kuOW3cKfikkNlFFewosv
v/wC3uuRevn5P5DiQF+TyJ/wzDfYYIP/mPGgZPM/fvWPhlCDUu0iCuB+/EEYisPK1t5ZWRgG
XYTDgpIjOl1gikwz5qAa3CVl6bP2+eTo6Gvf+PYbN37037344jPPfepoNs2e3QzulrMKVBAA
Se3pjXe3O9vuvW57ycngLkIAmehNcxJvVaXr07TrOw02GYW93e2rl68++cTOwcEP3nzz9T/5
4+XX/oKL5dWrDz106bAJVaXBUl4tll3XxSqYW5tT0BiIkKmZgRq1ik1db2/Fvd1mPIIIFq3M
VovlvF0srevoLrQgUodYx1hpKCFO2ZNbolMhQhd3uoNQUBVG9mfi3dBCSC9J8E7CBVCHsIzV
IIOGoQotg9GBdBXVpjwsRWEqB3RfK0NF7xII1oIhB01xDQp7uJ8N2krIt6iXDhkRIUzM1Bzu
7gJRqIh4IQwgWBjf2jRXuJjA3HPOoiDgdHeWX2otWipERSNFCBVoGXr11iX2nXNpeW55QV/l
3DvMxa0kc4qFQChBoxvNAQnq5Y5UyWYp955zLzy2dOypHzVhd3v78ODg4GBrMumX7fHdO7h7
vLfs66CiSnOz9fO7nJsWcugkCTkbz1Fhnk2AIFu7O/Vk693T1bvHU8aQBMu+TSkVM1dQhYoH
4ZoWnd/UKy+c8topo9sybV1/Qb/x3bf+/V7YPznkAxEMGwDA3yoJfYMNNvgZwkeM9s4uuvTB
5BFE134d0JltyD7A0HjGIWzw3BEGzcPd6UPYOV3pMEt9v5jN7hzdO7h85ZHHH2+7XgDQQ5De
rAx7RAOAEKud/f32nZtjKINaX7bfYU7SBRBRUDy5Stgdb2XRnpLn7eytH92+8c721SuPPPPU
r/83/3X/L3/tW3/+56//xdevf+eNS83kkQsHl3d3D3Z3U+oW7dw8bRtGHcuifw+uJDt7rhY+
P9J30cRmPJ5MxqPxeDy+dngxxL7r8mLVLeepbVObum6JIgWJAR41xKhDUhOhgLubZZHiM9P1
cI9RAKFKZOEapEJcmWkyzAULR6IXIYclL6BsQ95nRWsehaKEDA+i61rJGsQsUt7DpECB6PAz
ynxQYiz2K2gh3ISCTroqfDgbEcKHyFYTrClHOWaRmihlu1GgVKe7iotKafCFOIWiLuLOZLnv
c8ppntLKcyI7sgd7gQEOdYISGFTIQIrQQapYSVeHkPDsZjl3ac7uXuW5Up/Uo92DK5cu7104
iCHOj45P3vmhzRdh1VXZquJt92ECi4GdlmcipHi9ilB0RrEgiaAwhAjIcrFYTFeWXaqYLSfL
2SyevfEobrMHvtDWA76SegAA5RX24JfnBn/HGDKhCnl641t/ZzawDTbY4D9mfETX3v1PfBgk
BdWgw7zP6eYu6+56M3P3EpBTAg/cPDgqSqYnuMOgDCKwLMnQpe50/ub3rv/pa9/5J7/6a1/4
whfanFbtyml937s53YNK1BAETF1o25M33560tpUZ+n5IhzanixiUTmst9cWcnJ2r3hiDTEYt
fUWXqube1sGnnnn80ce4WLz17e+8/fW/uv3d78bV6tL21sMH+xe2JuLsu9T11vd96juaB4iQ
nozmUVWBPqVW3KpQV/VkMtna3tlqRvV4EjXWgpxyu2pT26blIvdtznmogSNABkUo9iVSwOCu
HJzdChTpaZiElntdkYJDBDLYvIG15DNoRcOjc5bKtP7gPpsludaa1lTM32N8Ll83GuGDX6oo
Rl7GsBqCigjNrW9BL8sCpT1ORNacahjU6VAaI8VrXrQtaIAKBXT0YAogmd377Gbe5dxb6rN1
KZkxgwuVTpUDuVERyU4vDc1SZncMJU3dKQ4zz2TrvvA09X4J9yr049Dvj7f3dy/uH4xiY223
Op4t7x7HZE3ykTE4ndn0XEim6Nn9WAaV5cEbOFTZQhRApLXkQWJTh7qaLhf35jnHRsfNyWo+
a5d0VqIRGlRZFClBXMeC4D5zkg9TpO6bzf/m9Rs/9Wv6p8RGkTqP9zQKbkZ1G2ywwU+Aj9ja
M7NBf1KImZPgoDxJubq/95p9tn+kqkHDMG2ilLlIuTgohM6yspdyfzKdahWf+dQnDSxUzN2L
U2e4OoMOhBCoOtnbXy5uIvt2UKXJ4GyiQKBCqsQoRroHwwSSe7O0GMcwisFTt2jTjaO/fmfy
3YOHLj3y7NOf+KWfnx/dufH66z/887/8sx+8vZf96mT/8GC/3p7UZO67frlsVysxF/EgFmhB
JUTRummD5Gyns9PZdCoiIcSoVdOMmslka7KzdXCglw/VTcws567rc9+lrmuXqzalECsRep8r
sqQhBIiCcIi7rtUkVR20Hi13xyAKOtcZ2VrCBDCoKcDZyvy5B4Wkg1piDcoMkeukqOFeLqM5
zygDKJZ/xMq6vioQBQK39YRXGILLYBWHAHASIgotm5vluVF0OaVKpmazlCxlW3pqmbJZX7Ic
iOye3a1IbEFcwjIgqQRHcFGDGmuHAybMgpIS7pbdnUbLlrKvYFPmRaPLSbS9SX35ws7e7lZV
sc/tdHF6cgvzVd37lnHkrA2VU9xykD6E4qm/P5OUc+rdmlW+ZxJKlNDQ7OxTWrR9bxabiaom
y2YeghYTv4Nl0j1MaM8xFvkAyuN21l/8cUk2/1nCpkdmgw02+GnxICJVCNMgL60323Hu/TQ5
5ASV2xSdQ0RKY8w6u9mL8TbTHK6FODgtJXFbrVb3jo8uXrl05dpDBnR9n7OBVi73MtAtikhx
wDQ727a3O33n3aYeV1Al3Ug6NBA0DJ3K4h4gqmpuKRtJS8mBsVa7i7w4Pp3eu/Nn3/t2deXC
5U88du2Xf+m5f/qfzd+98+bXv/nmX732gzd/tOO2v7tz+fBwa/dgdw99v+qWiy5189wRDCHA
2YSoVeNmZVEtp9x13WrV8viUeJduTR3HTT0Zj6vxpKnrajzZvnDgIjRnyjmlbDl1K+SUu75P
Weih3H1mQ/oAAXPFEPMJFqrFMoErEeSDnXmtmKyXJYcR23DhZkkxENGyaOegr3sPi8+5OLFU
7o9y4cIgGoo6k53DRp9SBCUMK6CkjZeng0IIdfiC5vTsNKcRvXnbp0xkZzbrs/V0g1kRx1QM
MIipihaLPExIGJzqUNdIaJkaF55RurFJJG/d55LngYutsGqU453R3u7h/oVR3Vg2m3fdvTu2
bNmnLUdjiM4I0gykKVw8DzVF5X67n0TAtTls4KOy3mwEhAIdpp8U7VLqLMe6acajpSWSbddV
MY7rmoPlnS4gJJspztOm9+dwAqIaVOX8yWywwQYbbPBxxgO39tY6k7vrWTsYJKgOO3p0fGBI
VPiTmZ2pVj6oHgwCFYVbOXI2O5lOZ4vFP/rlz+xeuDCdTY2Df8izD3OuMvCSEGPIOXdsxxf2
+tPp6elpVY3HECWNdHh2W38HVYo8BVGW1j8nEl37xU62rRAWS4x6LhftjbduvjJemlAAACAA
SURBVDUa7V+9eu2JTzzzhd947vP/8u5bP3jnO3/z5t9867tvf+/Aw+V6cjCe7O/uy15YIi+Q
Zu28XqbosQSQxhjcXGIVhGZGpxPm3rbtcrG4bXdBxBhCCCFWzWg0Ho2q2NRVFZoq7O+NQxAi
9clSx2x0Miealbmp9Ak5n4U3CV0EKsU/Llx3pch6rkdAhJYH67oWKxJQ2AfKZp/o2RV8iGVY
l+tKmSmWwVw54Fp3LH6mnuaAeQnbpAOOYG7ZzZ3Z2HnuYOZmzmTFF47sMIIqJDKsJEJ5yW0A
cpGDVEzAgTWyBmp3GAUGMIln9wy2zMnZe+6BVuOyknYU7cIkHO7u7O1sbW1hlXC8zO+eyOky
LvuJMaoEiMLLcgKBRM8KCzAIiPBe3/2HvA4GguoD9SyjPoc5RDQ5jRpjA0Hf92UUuR5zr+M/
ym/lQxNgAT9QTlw+O/u/P+FreIMNNthgg39AfLTZHMNVt3hdtARI3t/8OveWutjMzwZ/w3I+
vcQT6TCpKNMT0RiX3ezu0b163Dzx5JO95cVqGVGWp/JwGV9TqSGWOgTUtYa4d/ng7tHRopuP
6nEtcKcxaVTk9eFBouxgmSso2YM43WDuvVImFsZZMrwFltGmp9//y9evh4t7B088cumRq099
4dee//y/OH3rR7e/872b337j5rt35J2TkcvF3d2dyda10ZXYWO7blDPdLVt2M89SvEwBWi6B
plSJ8X4WvGU/PT65l01DLOMejVDVGGJdxaau66qKIaAY0aIKgbqWTDEnnW6FurgT6lpSvst1
GrzPpEipoCXEqFyZCXhheIP4ciaJrJOcSLqKDoFVLFlV4nCD50xzd/NE751GZvcygzXSvBTX
DZaqRGQVLwb3wVMkLmUdYIieIkCI+xl9GYLO9WyvQShu7pbhGdLTWvdOvKN38BwlxDrX1Wx/
K+ztXNjZHVUVcuIsrX54g4sVFl1jHFErl4qiLuW3cXEXZrhFUNU1mENg4YGMZVi1G/5zlkqO
wvCdQgSJNUJo2y7lNB5P6vHo1q27q66fNLUOq3/DOPABOPc6EhEF7utVG2ywwQYbfGzxEYoU
QVnv3WOwyxhdjebuTmp4fzpziZKSs2QEQkVIhqCqQpqIEubu8/ns5Pjk8uXLjzzxWGcpxkoI
dTMDhEKIDNKIuwugIWgM8+Vibzx6+NFHbn/zjbhoL+7uNk2VPSUnnGFYryr2FGeAgVlMqqhV
tB69MGSrMqNzlBndR1G2THfUZ+++u7jz7vTr4ze39vavXn7oqcee/NVfefLXf2U1nd66/sM7
3/nerbdvv/3O3bjq9yfVpNHxZDIZj5TUUUPPOaXU9dmNTgjDoI6VPX8hJeesRB0iYiCRc2aC
iPfou8VqxiEGiSzZk1oUoRBiSWwSUMAirihdhAooQBVXIc48OILhMF62/dbU9n72QQmnWN+M
a9NUeUzLVM4NzNlKDENhxKRQYomcGDJDRShqZNkgdFEDAL2/AKc+OK5wv1SOw+2H50x5esAI
dwybnW60zm3l1iqWFZZRV6K5qcNkUm9tV+PxqB7VW42lbKfL1elxNVvVXb6Qs7hDgKBZmcQR
RAQUpw4qHqACFQk1wzAaHtgdsA6R+HAIhuissl6nYiLZPbkbSctdMgomk/Fkd3u+WM5PZxY9
BgdC4bxx/X7jzP9UPj4/61MNg7X/nPVwgw022GCDjy0+omtPyhZXqbxwImcQEFMMq90lRKo0
xpQ//WW059kGQUmKjaZkpAvNS0BUbrvZdDZdLJ7+1HM723tHfadBU9utZ4UigpKAPlzmRQjG
USNtlVPevbB3+bFHpj98+52bN/cP9qrJKPd5iFQqvTNCB1H8NKBUEmIdm7FMuJou+tTXBrUu
wINx3PsocD8wk9SqO5nPb5x+5xtv9DvN5OrFg8cevvLMU09/5h8z+cmtozdff2P2nTeO3/7R
9NatkG23aS5MxttNFYWjca1BzM1y8kVizsktm1MEqsW3JRQlYQwQI8qeOyEOChCiurnTU4ab
OU3Eih0KbuKm7jQTN7iVfbmsTIGlmkVk2P/jmYIoCtA9e+HE7uZWRpCD5X/tTi8EyQmnGOkC
c0JUgqqW7r9oUBHlmjoTMLOyTCeixFAtrBAMG4Ali50lHZyCLIC70LWwmnIKVkicJ3M3S8C0
lnmlHqKPap80rKvJpAl1o1UNou9TOp3LjduhTdJbNBu51ECAUoVCqKjA4L2kkkyKdflgCBXK
jDQjOpxDlOnwnP+QyAHBIMMOtTflaWUUJ3qzru/6bEzuIqGO2Uw0XLh4YT6fZ7fIWAnMTSAi
AQOn1MESdZZ2utaiQrg/+Pu7epFvsMEGG2zw94cHEak6VokgGdZr26W0GEO8k4agnlJpiRER
SxmkqFSlFcSdECkzKQgoOZHZ1U3N2sXy3tFRoj769HNERFrmrrWcSk+tDMW50DKsokPEA0xF
t0azk6mJX3z0qpB3rn//zq07B7u7o6ZxEa2qxGyWi4dFqTAixNRp1rD98OXdx67cvXvn+Oat
k6PT4GkfNZadmAeKmwtMfbnjeaeKrftstVzcO7rx3e99r6nD3u7+I49ce/KZT/7KfzH+9X+R
Z9N7N26888b3jr7/gx/dvt3fuxu75USwO2p2JuNxFbfGdcAow9tknSdzmoPJkN3pESIQU8lw
gWgIQtCYDKCSCkI0BhitEwxXcEhpWRnsS2UMZ5aMuYheROmDUwlBNAhElDKEMDlR6uxgwyPs
Q2gnAXfAIWX/L6D8FFGWWjupAXUqJQyKF4fsqEIFi4ur1AoLUknEEpIQc4Ku0KKDwUnr4Yl0
zzTCKInSivRB+6ry0HQxHO+OlttNU9dNXQVBdMKyLFsujmXVjZNVhpjTQIFEEMQg7kPCGRxl
24CSaF7FmiIxBoxqDyFbzqvELsUhr2Gdwr/W9HzIFheSVu5o8aKild1Jh7gGRs3ZumQpZwRq
VcUQRMJ8sawnW3EyWS6XTH3TxNjlqCohFGubFr+gqrvrWpsLqjEG0pmdAhUN8aManDbYYIMN
NviHxoP+UpsZzi1pn7EorCcUAOK5v/XE/dxOlLQiuhbXC5UcpjY0c8vL5fzk5PTipcPHn/yE
uRXhikOco9Ddhut06esNFBeoqMS6tqbu+rTIae/KpZ1mdOt73799797uZLI72QolvkiKGUjd
3BUm6GlufU7Lnd3xYw8/t3vt0unNO9O3fzS9e1JbiPBAdyM0JLUV58EDiC3h2NRc+2zLWTt/
891v/+lfX9/aGl+6sHvl4qXHHn7+N/7zWP2zVbuc3b13+tY7915/8/Ttm8d3Tu3uQuEx6DjG
7VhPqskoaKwgY8libU6dJYKanDl7NiMUAoU4AaqzmJpN2GmmgEq6C8lAj3AbvMs0hCTBZahp
Kb+8qJgKdW1MEiLKkG1QpKKB+QhdKAIYrA9wMRAYQhakaFhDUxxdS0uvDBlRLsigAVnoAgOd
DLSqtLoY4S4u6hSzInCJkdCZ+mmAlR5olVbRN7Ef1Wlc50ltTY0QxsaLlj1nn7fWJ6YkKVXJ
qszKPDoDJYQ4POswFNsY6eZQ0RBEAyIKnwwaUsomQkebu3XjjgJ5eK5+mBmJ71Gn9OwzF5iI
w3vzPqdsZmBUVFXQGFZ955a3d3cODy+++YNFVqacq0FowjrOdp1qW+IkytsGnMUeDK+4TfzB
BhtssMHHHw8iUjnnwp8AnPnKzz44c8IWfw3KhX3tiME649rMPCdKrEIAKEqn55y7rl8ul88+
+ez23oXZauVmIUaImJnlXH7K+fSeoOqEGWMIo60tN0t5Ne/6/Z3JI88+E380nt65m+7dG9XV
aGsc6+juBKNCFFQo3JiW3fxocVIdbtdX9q9e3rvw2OXpjVurd26vbt712UJz+cWo6kg5uEdz
NREKVS0EVHWf+8Xd5fTOu+++oW9Xkdvj8aXDyeVLBw89tP/UJx/5xX86is3idHF85869G2/O
bt+6c/P223eOZD6vVm1j3gQ2UauoGrVS3YoqWqGuANI852xudLfgNANFISNX9/shm05kegn0
FtLN1REtDMyxZGCWizFFXYuxR2nEupWFoiYkaEpjGdEp1SWZlLzzEqokcAokUqMwqGbxlfYe
yrhW1SUWH1qpq3EnKE4kNzLTM9xUEtDBc2SOcBGDzcbNfDxCjFpF1LXHYAobuprh2aTtdLWo
u1ayI3s0Bqc6qnVsKYp/jGDpcIGU4HcJDDEWU1fVhGp7y2rtU993aWX9KiV2dHoIMUJ00M9K
SeCAdUkO3hccBayLpIeIThiRs1t2AiGGZjxyIFuSEAnrU7+3t7u9Pclt2/c5VBXB8B7/uIhI
6bIsWxrFxKb6HhPVBhtssMEGH3N8hCIVYzxzI6OkYp7Ly8aZd/gDi9z3IaUEhVacTpZVvOtW
946PupyfeOppR2j7ZJ5o+ewHqWoQCaplMc3d4aCKqsS61hjaVWtNzslO235SxcPHHp/s7M7e
ent2cjxrF1tb4yqGAARKhGoIDmr2xelidu/uxWuXs2IlHi5t7+1PLjx+pb9zcnrj3bs33snL
VdMzzrP2OSSvDJppORsdVfC60lE1jiGKeAa77NNZurHo5O03qyptjdPBDg73m6uXdi5eePiX
P/1E/MU6BFieHx3Nbt0+eeed6Tu37t28ZdMZFytbtcFzUKk01LFqVCNRB1WKQkPhQhAXBMLp
8NKW6xVqt1yCpBw5qa/Uh+hz0BEIgyAEiEDUUaxu62R0mNsQM2Uu6z4+ugBKHbK8QYogiBMm
npCF7vScDZkE4FJiJ6TkpRrEnI4+yDJqC+sEfRX6SvompiakkeYmJhULIqERaejuTjFX89DZ
JFnVetVZTB7MTFeUFF0qajCWKj5AXEp5sLhIDynrD7r2HAXA4LGKWlfmPl9MF3PklJugKpWI
mXusGlWIO2HD002GkAKuIyHOu5PKF4UgdPDUA1B1Smmn0Rirpqrqqks9gaCSzVPX11W1v7d3
t12V1UWVoEGL8y8ECaGUwJQdvcEIWEyGPLcMu8EGG2ywwcccDyJSJVQTQAmFGkr01vzJ1xtf
sr4OFZCUEDBISsVpoypSumVz39Vi3Wp56+bN0WTryWc+iRChQmPZ8ivOm+GCM1iChtRpOgUR
IqFuWEULgU2dsy97q0Sa3Z3tp544ubt9+86t2ydHDWWnbibQ4KoaR6pV0JT6/tYRH59zUnuj
S88a0ezV1fjw8PLe9lPXbt14Z/aDG32XrM9IOfYWegtQUUk55b4Xb5rtrVHVMGMIFBeYe+q6
PqXlyeny+2+mgDsx/Ch6tTPavniwfeni3pXD7YcvXfrkJyJCcPFlNzs6Pj26d/f47nI+n987
Pj4+ybNFWHWxTbrqY/IGUokAruohaCE4ERJF1EWlUil+6qwhBO2Lbc0J95I3AQFZRnXOIeNx
uFijmM4pLsLSCKfmgZVQHGLCXAQpkQzmwtg8E5AsoGRhAhOYVJJoVsmqrioa2iYsJpFBVQOi
ughCIWTuIOjiVs/SaNUN1jenOjVZnfIocZQxMgnEUpFDrCRGFzGWUmaDJgUEBu+cFj1WqtAA
iaJChqamaqIlcGndven07mwlxKSqtkeTugpNVcUYzTNtMNsLhnD4tZa6DiWVtRF+HX0g5R4H
CJioC7o+r/okVWzG4+zJSVGYZTMXDX2Xdnd2T++d5NyzaWJdiZQ59fnMT2CwnN/ffj0zHW78
5htssMEGH3880Gxe12UpL8Y4hEKdS+nkOQBlK6908nqZza3LXobkIAVBV2HXrqbT6fHJySc+
+em9g4vzrrd1JHoppXF3DeFsU384mVhluNHbnDWq1rXHPsOCqCiZvc9WCbavHI4O9mbHdxe3
7/XTBdscEkFAA4OmSWzHWN66u/XolZXkHDyLt5SorCc63rl87epFe+bJ7u7p0Y9uz28e2Wwp
S4vJo6EqWUgK1UCKB2aVXGQ0obiP3Leyxexh5e68E20xhd24dcf9ZlCva93eids7o4OD7QsH
4/3d5tKVx558soqx1gA3a7u0atvprJ3NlqezxXS2mM76+bHP7+Vlm/qE7MxZUoZRSXEGUQgq
y1WfAWoIEC0h5hDQTUr6Np15nUewXhYgseYRA0c1qEuRfGAiWZCVpsFVXOBAjtrFaCEgBotq
IVgMfdA+immwIK6aFZkOp5iHzrTPVee1senztiE4kb1JqbEMuEnJmnIlBIxCjTCFE3VoApVO
ByUgkRQkYRIwKKomRJq2VRPHsa4kVAjITogRy8XiaLloLRG4uLOV2yRm0W17tNWMRy7oU1fC
MlSt2MrPVRmWWNKBPQ0VOKJgaReEqbiqC/pkq65H0GY8NqJPiUJRsCSwu1vKo8nk4sGFd999
t+vTaNSoSvl3iPwS3P/K/QwEOSsF32CDDTbY4OOPj/BIYR0NxTOF6b2jvTLs47pcbwi8zvk+
ySpx0nQoSvdL33Wn05M+dU998tmqGa3mp06hu+I9vqs1RxtGe4nZFa4K9+zMAIqcoqGMDklJ
0Ow5xLh7eHl3ssv50qdLdn2/6rqce6c2wVM/O7p3+ImHO83SNB1NBBWCOhIglWLrYPTwxYef
fwLzfnnrePr23XQ0zbOVzxexN3VPffKcVYWCUGpXSC9xS8ETPAd3YhSDmquhRi1J0jL1R8ct
jpd4652APgaPWjVNrOt6azza3a53JmF7Uu1s5d0mXN3dm4z2oqr1oV/lvm/brm97S7lru+V8
YclS6pd9n7ocF8u4aj1byTWwbCUVtcRkOuluNHmPuMFCF0qieBCBgSmqa7EbCQGooAoll8FI
glmwUMkiIMXL48LcZ1CQzTvPOWu2STY1D47grB119hLcVZWULxLMVAPWjdmlcFBEVEAxJaB0
ON3VEYIrejhVPKoHSAj1qB5Pqkt7B5NxHRNs0fWzVd93Ofnte/eOFzOrag/q5qNY7zbVOG7V
VcVs3enUlBJjHWtIhPSWyy93TvwRkTBUIaFE0RIiGkRFxJSEJGebeyrqUSMhpJTMPVQqKuIU
EbrnlNzs8PDS0dFR2/aTicVRFBUJA1ta+6P0LFNKS+DHe82IG2ywwQYbfJzxEYGcxWleMjbP
G6TOblA2+zDYmIa38eUaUNrszYuPCu7mOTGnPvXHx0d7e7vPPP30qk8+xJfDzM5miFj30pAs
mkm2jKCMSqL07Zb+k6BVFQJMPYC0lK3zFAVVHeu9nfHOLlLuc6rMepEQcmgcUQhv6goC1aAQ
Tx5EgwhETC0zaxTdb3Z2ru0+cjVPV/PbR9N3b7dHJ3mxZKex66vsmhmyi1EdYCBCRkziSUgw
pDzOFGhUhRBBKsVEuG80GlPyhH65TM6etoAfwTuVDp5j0Mmo3pmEujIVVnXd1KGqEKPUlVY1
t/coWoUQg2SjebGduzqFhJlbhjvNy6IfLA81OvScs2UzNwLuZmYlFcn9/2PvzXYky7IzvTXs
fQabfHaPISMyIrMyK6uaLKmJbkCNvtIL8F7Q+0lQP0DrQmhAQutGLZFik8WasnKIjMkjfDC3
6Ux7DbrYZp4hkp0pCVQjCdgPRMDcw2FmYWbHzzpr/ev7RUAcMovTMOO+hoTqrmqirgLmJBDM
wJzVOdvA1aMBG5ABObAKy7Dtp2wp6gCuZpYZnOYumc7pwIbsO1olgCEagFIOJ1YoSBl6TBIA
qoAFl5NqMp1UdVVVVVVHDurDsLlZdmkNIMg2bBpijDEOkoCL2XRySBwNI3JkNqTOYNm13dAr
QKRQhVjURV4R3XZQfUs+Bd+uNgIAOQGAmolbSjo4DADmFmJEYhE1dyAgBnBzNQI2wJz8Mx6N
z87OXr961Xf9ZFQS4wdlE28xH54N85gLqfvpeT769tprr732+inrR4CcHy7ubVfz/t6F8r1H
SlTAt1XX93EwRM6IoqBCpqppvV5f3y6fPPv05Pxi3W7cFFAJXLbpbUDEvovw813wCTEBo6mZ
uzLFImx0MHRHQ7AAnmONmYO7iyZ3E4XeFMEtkAQWRAXgAGmw9bop6hlmio9aCRwxLO9Wm3UT
JwWVRICRjDhagXBcTo8fTp+fD6tmdXN79/4Gbu5g1aW2t36gXmmwIM6GpVPlDAauCgpi0Gsa
vDfyvD9HAIEguIOouSNjGTLGOuSFPXf2DnAYaCmq2pm3TslUwTXjBhCNSRyAyZkMoGcaIgMA
0jal53t7T+aGA3BZiEHqBxNxdHTo+k5UyBHdyABTCtKRq5vjlhkF5JA91QSI5sFxRIFsa2pH
c9oi2xWSoXsAdFQDQdryEdS2dXIOWN5iFJwJmWAX44e78DsiJVACQccQoCCPFKs4ORjVh6Ny
MipHVVkWZiIpiUmXWpCkICEgEBpCwTyJsQzFMaARlTEU6oTASBm8YWaEhCY2SOeDIDEzB4bv
uQRouaGXP9QODph0q+QOIXBRuGurgyCWzO4uSSCwE6uaqFIuzwnBfOj746Pj9++vmr4/ci9j
vI+KoRw+jR9sCG7pYNt9i+ym2muvvfba66esH9na8w9gBx96pO6/eT/aM9X7f+WdRZ2QnIOY
sQlLxybrtpnf3rWCH3/xp0ZFShtPHbkCOCLkaGTa5Ym4u5iBAxEksADE7gQgbTuajBpUK4CQ
2CWYs3hunMXcUVA1U7TgmtzREdC0QB5EBYfN9SpOxlgBB8xBcoa+ulp//ccXMq51VD84PXr+
0ZFZEusxYrLk0aqLyfHZ9ODTj3DZtdfzu/dX7XKZNq2uO2g66oUHDUYk5kkgABigcKHog2hK
990HJwIMwMAZlwWA4AE8ECGiqoEbmDE4g9UgELctOlVVMe9h+yrlqasZoBuA+i7w5YNVSo6R
A4/OzgaH9WKRhh4Rq1H96u2blORgPOakIyfvB00D7OhPnimXhKoARJELwGCWK0QQ1wSewAZ3
NXVRcC+IZceXygAFAkJkAEDD6AiA4OjuRqCAhq5oHsgDDgi9m5BipFCVRVXgtC5m4/F0Uo5K
KpkiZttRq60nATMwV3UUYCdNPqw6axOKR4WSmTGAOSX0UEhOm5GhM0uQ8bKxjOyqaOBJ2qFD
5F1RlwOV83ANLG8ounLgEAMBaYgWQiNdE1gYExqJoCmHKpkPSdCBAAIiIZSRNaWqKo5PT96+
fr1pmtl0oqbMaKaAArCDSDk6oAGQK2wH5TmZcK+99tprr5+0fqiQyuRl2JVHf78X9eFob7th
l/mbO20ZVGqZ7agprdeb29s7CuGjj54kEVUB9H94PWlrm8GMCQJCcwPzGCMGRsSyqprb+bSe
wKA25GLCzHMNQEDADq7m2W2dn78DIaSub++WxXIceWyKq/nSepvUh9PJ0cUDbaF+udj89Xq+
rmcPLw6mVJRDW6o4oUHXuxhDOC7C4cXJs1MYUrfetPNlM1+2i9VysZJ2UBFOYdJJ0YMLcAKM
AQbSfsggSABwcMk2JnTbhrRsURGwS312MzDNAb8AwABkzg7mW+AoIgIFRzV0BXffMU2zdRoB
CTkQM/umicxHIbZDGpKUBjyISaoJYwiFAhB5jA5077Gy7KEyMFHZQhgofbA+gGAE4G7gwMyB
ODdwGACAEDJOYQtaQMrWKzR3YUqEwAgBPbCSY8l1VcZxOZqO6/GIq8JGwcuAhA4insAEbXDp
UVLh7jIMbcLkhUcYXDY9tCkqGoAFBiYlcAUD33RrAU9uA6gxA/PWbI8EOcsYqMJMOM9P1W33
2iKgIyGCg1H2bzmYe5JkIvlYSGmgpGUm+4uYaiAOzPk4GdIQiyhJzk6O726vl8v1wWQyno0J
wFwdAdzgnqvugICBg7sRUQhh75Haa6+99vrp60cKqQ9vfOg0/1C7rT2ALbTQ7/3p2Ueb43UR
KKmt1+ub29uTs4dn5xdDSu7biLi/c89mxkQcQt7jUxUGIkIxdTN36Pt+Mh5vbm7FtAjsrEZ5
Rw0MISACITgRgwE5OLojkZqRWXDyfvA28Qg4clRarPp+MW/btGlTPTkbV7Pf333313/xN58/
f/CvP3p4ITYl0sLX3lNFkCS5ewhORpHK0aw+npzqY2uHfrVul5vF3by5W/iqTd1gXcI+8WCc
3Bv0wUw0owCA0O3v7In5dq7knknvau5uCN9jIvPyo6na1jCeA0e2kzwHgBz0TLittNzAxNpW
VdExmJtqv15DSkPXDt1oPJrWZVBEVXE0shBc87uqZqqWkqWkZg4cMDCB8zaymO5fcCd0RHEF
ACI2dwdUQAV3RguoSE4AxBhoiHGITEz1uJ4eTOvpOI6KUHEoI8VgroMKkBpnFoYCKHoiS2ha
mLEkaRpZNWildj2s+nJAdpIk6uYhSMBGpRk6cTMFMfVAXEQM7ICuZgy5pHZGdOftm4C7hija
B0t8eeNB1cxBkARczFIa1AU5EGNgKmKRwMyFCELgEAKYI1F+31RlMpudnJy+e/16uV5PDqbE
GCjuBtk5D5HIKTCHQLmQygSE/y/H9F577bXXXv8Z9UOF1O6M/j3H/O8XUt//rt/VAe6+Q2iD
myESZ7exakppvWmarv/T55/Gqtx0naOqJN42nXb35B5CIMQ8zMrPwcxiCMA5hA4BfDQehSKu
27Yqai/YBgTNxGs0gGzl8S1uPZuZAW2bNJtWna0aGo3J+GJyGtP6r/7jb5fLrpocrJt5Xx8d
np6/vVr94TdfHb2+TUUJ2oQHo5NPHsQBJoISfKO9m4FTLtIcIBRQHo5PDiePHx6nvmv6brPc
rG/u+vka1x22gnVJg3gvOogmMdEcxQwAH9ai23YPYE65MwC0HFrsOSkaTTyJqFjmO4EaOeZ6
F3cxuLTjRjgIAgK5eeAQmMFdEY4m44KpINYhdTqAiLgaqLu6SnY0uTs6MQAQcSiwKAcCB0Tb
5vxZjhtGM0RFGAA9NzIRnFAJhQADeyRBwCKW43Fd15ODcZiNy7qq64oCWc5hBmlN1HtDI/Io
A7ZdEmWOMRTghBq9t37V9PNFWq+tlxDMO6UmFQOQgCsY80ZkJX3jagVPQD8I/AAAIABJREFU
RlMfrLm7kySTqggcRFTNUDFvHWKOFUREJPv7dI8t/8AJwdHNXQEEbDBxcFADl1hUgRkRU0oI
WBRFICbwrZE8hPyGiqST0+Pb63dt15l5jAGJzIQQiAMBORAjE22x8rBjSv2/PZ732muvvfb6
z6wfH+3d/0LP07r7L+9v3EfymaqrMbPft6OYVTUgiCYCW6838/kiFNWnn32GxDlw193EDAws
R84SmSgREaJJphygu5upCDAz+nYdXVXHs+nd67cdBw5EZSDX7eArp8llj7Obg9OOnoQKTACD
rt/NXbwYTybTo9piN1+/u7yenuizz5/Us0IKbJZ1urmN7frOlvPudn1J47dXZ0X9ZxdPji5G
dbTbxfL9za0Tnz18WJRFOySDBK5UAVbIMJ0ejQ8eHOtdu768sUVjy043vTUDKMgwpLY3BQZU
1ZQSAIC7qhl4fiWzTdsBHZAJCQzVwB3VQQzNyA3AgxqYESLa91PY7TuYy99MNEBk8wjATOIO
aXCRUVH6IAGZQgDHQdRUwZ3yEM7dTM0gxhiLMiHmvBwzIyYFc4ABTNCFQRmNWBE9BAzsjBBD
HJdhVMVxVYzHxaQeTcYhFimYB3eHwcU0GZipIAGwqggSESjM58vvXq/a/uTiYZydbDbDcr7c
zNfeq7eD9+jJI65YrVR3KAhZAjWa3q9XDTlP62I06s20aRgwUCgz+NWUDEyN1BGAIRuRzOH7
Ksp2iHHY1fduDmhiJuxKPIhISmQaioLcA5OqAHgMxIgESExMZOrgTuRM5GZ1WR0cHl1dXd/e
3T3+6BGCa3Lm7abefeV7f2zBP3Tdstdee+21109NP96RyjfuIVK+w0fR1hmtGdeZF+kpL2zv
rqQRCV1UEjmkrl8sV/O7xeHR8YNHj9RMVM0BEUwtw7PvHyWlhIhglh/FHZh4V9DlPpN15gfH
R/N37ztLEYpIgMSO5mC73D/fshZha11HpOxeMQNZtxuztOpCB6EaH1TVi75rl8vU3r6//g6m
1RdlrI8fHm709vKtO767u7n+7VfnXA/Hb794fn7yYExdW7Sd1aVwU1/UFsoBPbkpCiPGXqJY
gYQFhjraUMggkRirypJpEp1C3/WaFFWDGbqbmfaJEdlMRczcXEWSI5haAAxE6EaauY+euQcB
nHf17S7oORvYt/97BDAGdXU1c4cYENiTSN9TfjcB0DFn8jgQujMAbMOAAX0bnaKOFqhNA0ZG
5l5dCY15AFcGiiEURSgjj6tqMqrGdTGqqnFFZSF5aZFJwBOIUXIffFsQO6gxGiOAWUEAYNL1
7XxTeD2rpnc3w+u3bwflGOsExexgGg84bdrUrIa7N93itkCejUskTgQLlzVgQoqKYXBQj0BV
UYF7MGBxcGRDVQcDBOCt9/++hQr+Yb7jB4eCuyOhMyUwNXW3KhaUt1PdkgpzJmtujX2MEAID
EYdIgQgdwE9Oz65u5tc3d+cXZ7PpNCG4Kzo4KOQZ9875tjt89h2pvfbaa6+fun6kkIJd5ykT
Df5OaZWVO1LM7LBrWe0w5eAemFPf6zA0bds0XdsNn/zi8XR2sO77lAYkh+2OFOxqJiek7P65
xyhgzt1DQNtGsugO4Hn+8OL9t99Nj0ZpGBgBt04p9N1zcKLt6n2OtyVGNBdFJGu6oZPVoLPj
s2cXF99882K+vH371ZebtqGqGB0edZt+3llVxLBq9Gp5WIazs5MX1+/m795+/vHTOJlgNYow
2lwqDE0cx7oqxzGaD6oDmqCYp4QiJcGAkNwHSexOgUW8kSGMqr7v3SgQgUNAZPWh6xAgMg/9
0PYtQQQ1WTfuCu6sgDtk431y7w5Jmhcqt28fguM2PM7BANDQTJIgFsAlI4Kqq7ADgjGgOkQk
QnZTUHNwN8M82qVsvHJx5XEdxtVa+hRLrRjLoqqKajwejcd1VVHFXhBGIkRDNwABFxckCBzd
1NwCZO8/oDk5BgzkoL2AQkAa2mZxs1qv4qbh3jBOD+vz00k9UePV3XretGnTIJVljdittOh6
xAWREUIZ50O3ISqKijFyTwUCIrNnapYj3bMFdrnaCNt8mB+WA6IisQccVJIKEdZlYQ6BQ2tJ
3QouGJGRMpAMAZkQmXJioqk6+OHR8dHR4bt31/P53eHBQVlVIr1JDqg2wsBMjpS7uXuz+V57
7bXXPwn9yGjvQ8tILpK2X+76VfeLeyEEB8/jud1lNTqCibmZAzBy27bu9uyTT4uitG5AJPMB
JKFtmZy5eMJdjCt+MNqwLYgTOCAhGqADdMNwcHw0v3y37pqDIrBCXgwDcDdHd3J0xNzpwhzH
S+wAphIDBiBL0i8Xt4MkDJMytAKL129ORkdVjL7qr2+vreuenl5cjCcHH3/aFeaz8nJsq2L2
7unnzDEliUggGi6bkLqg3TjAQRVH49hVrZNFw8qZirKPwlUpyPPr+eWby7v5AjgePXy0btvU
DzIMlrQoC1clpNl0ejw5Gk3GaYXL1SoyCgGzU2ZWglseAAEguhMIue8MbYSInjELsHsB3RGA
ckKiuyuDEaKpuAp63G3sAyEzquRI39x9RAQkQHLMznEKo6pnkFH15J99TgdjCYSRiZkcUF1B
FMSyXR0sc+1jYDftNpsihpKCdClQYOKuaZp1U0BoVpu76zkmjcjaD+s23ZTnePb04OwiTqcb
87dtu2pWIZYdtr1244JtsYrrVU2MzDQa1QcTL0JZBHYrKY6xKJKVhBCh6daShAM7groZgfAW
uUkMCIA/VkkRgBk4u4MnEwUvy5IAYwjmYOYcAzEx5FzibZbezuJk7vl6gIn54aNHi8Xdu3dX
Z6cnhwdTAFVXQiT37DrP1y5bpNW+kNprr732+snrhwqp3clgl09seVK22wyH7X4ZEbmDZm8N
U16Id3czQUez5Gw22GK5urmej8azJx8/X7dNMnEEFESlD+7HHYyIM5wSAQjp+2cDbojkwEQO
oGpc0DCki0cf/eE3fzt++BEJkLoHM1OKERwNzJ0tJ/cCgiPlwBMzd3NDMwtFtVk3b+aLoqwe
Tg+u3t3crlcxDaODaajHFsJNv344Ph6XszZt5kO/jAU+ebz+1S/rasxOpFw6QBq8XTU375p3
75eX87C6jGOpp3E6ncI0mLpK0nbDnR1S9MPjIen1ZgNNs2raoe3bplXVsqzLWDBZt24WfRqN
RoQgALEM0hEkJ3NOgqqurnlZ3t1dzAfIrjDcvmV5Iyx3qmAHynZAJzM0QAMCA1fXgtAVkFjd
nBAimpihADpYhhqhMRhBAugQMHiLWtST2cVJX+KgA4AJmKIxuoL0nBgomEfDQMEFu1Yur+5u
15vZ8cn0aFaZxaHtlov3r1/fXd2i03q1Waw2on50dHJyfFodP5p9/It0+mjTdrdv391dvYd+
OJyM3XXx7u1icXd2foaIBRDHKs6m1ZOPyqPj27aZK606i1WdMNIgXbMk1k0m5quVIC6KiLiL
0DN0NKfdENO3N3Y4ivvLBkQFdgdLRgIVMyM4OEXqZQDwMhYAhoBEuJv3bZujCEAEzAgMktLh
0cnR0fHbN+/n87ujg1nkIqBJSgQOQL4rdc1ADfeJe3vttddeP339IJAzSYgREXJ3B7833gIz
b5PzCNws57CqqruBm6gAZKQUKCS1bui7+fyu2Qwff/bZ2YMHK1NBNTc2ICU3vfeu510pNAd3
QDTVPI7LkR2AIGBo2aJrIKoOk6Oj6YOLd+vF8fg49Und86ags7uDIRoiZIuVOZmxAwECoAEo
IAILwt2qmw8ClQ2jek3eLFfV+25Wjyz1m275anlTTEZxNhOK4sVERqVHF8ZQNZ1s1DcGVo1H
P/vs8NHT4tXN7N2Nvn+z+PpyVc3PH0MZTTY9Nz3eNbH3Q2aZHWIsbtab5d2ijNXJ0UmM1WK5
Xi5XRRnXbU8Ek74fFRGTel0JA7qWBpWYiSQwJQCjYM6OBOyZIa6OmF1kAOjElKNg0GmLOfVA
HsAZKDhixm6xGzkPlpsm0YkNASFXy4BABmxORpgQlWAgLFQt9RKZ0FhUwHo2QgAXI2MzSD6K
0/W8/ebr16uerjp9N0A/H3q+foTpqTel+vK6v7pcNV3nZTGUo6X5+LOfx08+h3I0hOr162/7
d+/HbXu8XExE6gXdLhe83rCBjA9irKWotIyTswcyOX7T420Hq+KgfnJyeH6KmjbL5duvfoeL
FUsqkINDBI7MmgYWZwDcgvLNtp/rDPSibYm9/bijgSugAYM5m9fAkbxzozIMqAMqB46EZsRE
9yHEu26uIzmxEzmiOzoRnZ2fXV+/v7m5ffjgwcF0ima9IxOqCgAYs22x/rbvSO211157/fT1
g6M95jyqA9hapb5P1nPL3SnfmaVyl8dUEZ2RKBJmMKeaqy2Xy+Vq3Q79p59/HsvSm7WbqSqq
ETjC9tr9nvx53wzYDTi+xyTtsOcQY8zPZxiGZ88++eP/+R9XsBkXQVUIwDQxoaMjIShuCzH0
7XwPMSfHkZELVHFU1tPl4nLd9jiqHEmA226oIVSM01jfzufvL99Pj4+AimI8OXr65LTvLaXr
t9+8+fJlWnTVaLQmgWl5cXZyGis/OhxFZuLu7vrd1zcHwQ8iBasHlKQdqV5gOa2LWu+OTiiG
MucLF0zH08l4Opnf3i6Xd6MYD4u6V+XkEYiIjMwCiFuiJGCggOBqbFRtV/Uwv5jqbgCogNmX
zuZkzByd2YjNQnQiLUlKxpJUyGN0N2E2cavREEzR3ZwcC9TaIEazEiwJGyM7AwYnMvecvwIA
CDB2rlsnwMjjt69ufvO7b9eDTy4+8oI3N3fzzQqqqm6WxfJmHEOzWs03q2IyOvjowSZgLMrZ
0wdpWjRdu/ryRf/Hlx+fHJ9ORn2/abvlYnE3qCAYIzSrK2fCzR3DJMxvVzfzpB4ARwpUleu+
j5P6ZrVITNEtcqhCURQFOd1HD/0/Ue7CWm5RGUBADoEIBh2wKoSgGToOoSxKAGBGDpxB//ee
wvx39qEDkSGaysnR8enpydXlzfXV9dHBgbtzCASgqiGw7QL+cJ+1t9dee+31T0E/VEiFEIZh
yLdFJEMC7+GckE1UH/w8uhMAEjEgIpqpiJgoqKVh2HTt+HD2/LNPUxo0iYmBZU8Pmdv3OKq8
N7WtzAw5ZNY3YN4dg+w7J0IRVdUYQ9u20+n04+fPLv/6t09PzkJdtYtVycElceYw5VkXulNe
aoOA5GJMRBCGzngSDk/O46ZdL5cwqKdhjPTkwcWMCPs+htGT4+NVu+n6TkV8vcZXbxdFubG0
vJnX8/ap1796+nyl+uurN8Nw5R9/9B500zSzo6OD6WR4+fL66t3gaTqKqq6RWL0UnRE+P5p1
ooGjiJqi1JWKmfvjhxftwTSJsCoBoFlEdiYTUwZxVEADR1BUc0QDz36ojKVyxDxeBQBCxgBq
AqBGSIEtgKA5gKIIJmV2TUiwA5EbEEJAUTMDdXNSJHFycCwEKOU2XnAIhqToaLv4OLBgTorq
oUm+1oAHZ6Mw1lhZnw7rcX9zHUQfF8UhxPX7OUp/VNTVbOquw6Y7mE2PR3XXrOev3slvvn06
+EkR1q9uUt/00t3dXWtReOCyKJfrmy4Npcpcems2My4OuYzMFMv53fr6+pJPjuZtI836UL0I
oYpFYAYBywhUAvB/YDdv9znejq/vHXrZkWeOBp4QrGSL1Kl4oKIqIwd0ByIK/HcYtvl4Icp4
dEQwdC1ifPTo0fW723eX7x89fDidTlRVVO6pIvfrenv8wV577bXXT18/6JHaMaLuXef3zaf7
ZOLcpvrQGL4dF2UjuhmoSdf3/XC3Wp49eHhyftalBA4BcmaIJ9O/s3F+T+A0M0fz/OjbAgsA
LLuyUkpEJKIxhvVmczCbHj64eHdze3F8jDGSk6WEhrTLgGUzQ0+I7kCQY+WgKIqUvG1TKKo4
msqmNU3HZXwymz0e1RO1BKnvhoDldDzhyQRSkpQ41PObZkP9g0cPDi+w+/LV6ps/cgiPLd1Y
t/j9Zj70X1/fXjx68MWDB340Ha7frOZ3j3UcowO7BSfykjiARxd2ADcOAYksAiKZ+nhSi2iv
0qsYEpI7gBgGjwhITsEGVitVzQ2525qickVjWxISAzEyErQhtaCMIbJgiEJsrAO3PbUSELUn
VgM2AAOxKOq9+KAE4g5BMJpzcGEUZtctsRtwu/JG6EYEHgCdcAjUJFh0qQmVjWi5HHTdTutx
DXpA9SjwQdPWA464prLGIqy1Xy27s7J8Oj073MjLN++ru83IeWrp5ttvmr4JdSFs9cEB1qVQ
4KqGm5u07BgBzTlwFULFAzmkbpOaTgZppacYR0jjWAaVoesBBwhFQeTbpioYwj9YqCAgbtOC
c6sUEByIkKmVoQXAyaiVvnet61GMBYgVFIzRP1gyvV9x/X5kbRYjIAKYnp2enl8cX767efvu
/cnpCSKkvmeMSYb84/uh3l577bXXPxX9UCGVUrrPqcg3VFVE8m3fhplAYDZ3BCfEoRc1jaHI
Abum5km71eb26mqxXPzzf/VfURml7VEN1UAUtwRyYyDYVWy4i0lx9/xwuxgNAABwMNuWRsys
KvnLTds9eP7x1+3merO+ODxI8yUTORoxu5uqOJjnkySamWZsj6oBYhLZmCZwiiGIntXlWcmj
vhmZOEgDdrdYhKKo67IEm5QFVIeTZ188vBgNhTRX79v31bcvL4erGwwcD2cwdDQkatN7kaNZ
JbLW1H5ej4oEYw6991Bz2/eLdu2x4CIgQohRHYvAaIbqqM5eYEr9IAA2QOvM6u7AoE7OpADm
7EAmbgOi0H2LxZ2AANDRySAv8UWMTsTAwRktZjhUDQX0GqsApowcHQOiKZnmUB92M3R3NTIH
MAOViANzQiQn8c6hADc0BzEKQAbDpjcMKeH11eKby8Vg1WxyAuBTQFcrRHB11/dtLMLk+ABM
2tR1m7aTdPhoio28fv+NCZ6PD4dyGKypzs8D2LxZDS6hKmcHB0VRdE2bij7Upv0GVPu+W0Xo
EVPfLTctUFEUVTApBwxoI2AAlSEpMwQ3z6xWyNl/W4r8BwlF90xO961TPE+CgWkAbTTxbNKB
tZKquirLigADUKQg6EZ431XKgznaCRGJnDBzJMDdzy4eXF7NX755+/HHHx/MJoFIZQAsxPQe
LLLvSO211157/fT1Q4XUhwUT7NhRsBtb3OcZ59YRgds2J5fA3cxERFLyvvc+bdbroqo+/+Uv
jDCJaBJPSg5IIKLgSvR/i4hBxMz5hC1f2wCciGIMudgy1by4lwssYpaUOrIHn33y9d/8puq7
KiAwkpG7gSE6ugM6sIPn4ReAoQGZOnVDtxaVoSXQEn0aQuUiy1sxJQpIVcJw13u0/mgUxxwI
YoxTrycb7rqDQ3jyOMZyfDjFtunXq+7yNbTD08nFm3e3r+Ifzg5HRVFhN2yxmXW9SbLp/c27
eVeGOCoZoQoxIIzLsqBQMhdFZKBoxZTqpsFN07oixAhDxme5MGxk2GiKUVxVIPvnKXPQs+0M
AJgpEBKzOlpCdDIOAUIRoqkUXkUpgxZDl7DYNWEMXUAHVCNTz7B0HBQIBh7Wdd9XA9Z1PSrF
1qhjVOWEoAqIzWKz+O4dJG4Gu7tdj6iajcbQbppNs2GCvrm6eqHdZno46aoqWpKu69tm3fUp
sLXd+uoq1LPD0zMt6qZpeBR7Tatm04XY9jbMV4fJT2eziuPp7LB0XC511SxvN81SO45kboY2
KuhkMh7FUXO3Xt7NhwhljFUsqiIWoTBVR9zlQrqZmTt+sBu33UvdJvbkbpQP5Mo279piNh6Y
ms1mPBvXVRUoBvWAiBQYPRdO+X5yR+q+kNresykQmgERHx0dHR4dzm9uv3nx3b/8s38O5iop
cN4w/U8mMu2111577fVT049ypCAHZtwTCuD7qZ3lSGIVc1cx3bGq0d0RIBAZURJb3S0Xi+Xp
xfmT588SurghYiDiEJIbKCJQHhNuEzl2HnMicrUPIv/QzHMYGSKZmoExEwC5OQS+bZuD8fjj
L37++//jL3/+4HGJ0TeNm+YiIwfGECAjA4KCJ3AkUQymRqZVgIqhMBrFelowFBtrmk5hHYt+
ethxcd2u56oToWqxOlnclecjokKhljgdxoIGUEZFiycnM8GNlJfN8u7m9mg2GtXTvu1uDbiO
ejxuUbwc7m7bhSUWHlXxeDwGHVoZvN9ASmWIJTKaY4gtmmMEJoU+kWG0Yso//y+/kANftNfS
bSTpkMzMVGxL7VJwczN1teRAwJ7cepVkyRKiOQ+96XLdNNSHSaBajcwDKORQGJWk4uaEHItY
l+V4VFSl1nhyTH5yUI0OvMGeBjAqnHGwksN8sV68vYZG2kX73cs3q6Z7+PjZuBheX77abDY4
HbmnqoIe8baZDxvDXmhIEZnLuhxP6+PT4uSsPrmwcnLXD92sxs6Iq3o2C31/yBTRWRKkZKZA
CqwWKTGLBVVCwEAxANY8gqTr1bU0bc1WF3VVloGZCN08b8YB7BJhMivr3guF91SpPJIjV3C3
IcCddDyroCq6vq2rsgwFA5EaGJh7AuEQcgvq3j54bznPVkI0izEg0qCO6FU1Oj05vXp/+913
b/70l78cVSWYmKaCSUTuJ+n/fx33e+211157/SPpB/EHqvejinxiYObsiMrtqMwud3AgSqam
lusfd7t3zEpKd7e3Xdv+2S9/UVTlsu1UldUZyM2TJASIIVKep+SayUy2NJ2dIX0HJ1TNPYU8
mSE3vf++E1gM16vlxfTg53/yy1d/+7tPTy+4CJoGIMQc0iFOQIxsCIqSXIHJyEihAJqUcTqw
9LZaDYlG54fnWrUNRDx8ND5/Vj160mha3b3/7ve/vrx6j5ffHRwGOpgd8tgmEYrj4Wh1e/t2
noRmiUM1Gh9dWFJb/f79VXW7eYr1Jx8/PfziuV4cMGL/4lI2cvPuRTTiclqcXdRoOHTt4ubu
3XsDKqrKwUxJYZc1HHrjZqBWpnDwZwfjz2dNcSS2RjAzB0PY5gpu4+JMFdTACIHcgw4uvZio
JU8Og9vDzUdd2xzPZqMQPYkX7uSBgqEDoKMpOgSCgBAQAyHFpLGjCFpsXm+W75eOyloEYXKu
Q8mzo+vm5rfvXyyb9ago09CsL1uaLx6PRtx3bd+oC7VdN7TsXlM4KCfjYlxOpjCdzi6e+MmZ
Tg/et8NCbDKbTqdlWdRd1715+WpzdVdHPpqOxiVRSkmEagzFhMhCW2ACTjCOxayuMQ39aq3S
VnWYjMYl1YwsKiaZ1b79g4jbhOd/4IPvu1aQg4OK9gF8WlNVbNouxjieTtwdHYmYAiOAW56h
fm9uQtxeF2w/ug5AQGCqEkMtDoj48NGjV28v7+5Wv/vyy1/98udVVfWduVtm22Z+2z/qwb7X
Xnvttdc/vn6okNqeIQCYKO91by20udzJ/4oIRLkTZYjbwDE1EVEZtO/Wq8WqaYD4Z59+Zor9
JqEF8MzeFjNFAmYC85SSqpoZ7aaH7k67Kiqf2QjRAd09mcYQEBkcXQEMANEZYl3erBYfXVxc
NO0fv/z645PzUEQ3JXdGBPDg1iMKojtHcXbAkptgYhrH1SGHuSwvN5uAUl6cFFUBWM/On6eH
H/vZOWvbNzdcUCHru69+zbY+fPqExpOiGnVBnYvzg2eHDy6uX3z37TffTmBz8vxhDYd/eXV5
dXNtNOKHF21Z+nTKsZQU6l/ao6KG1XUcuvX12yUODAidrtdOTBBLLtDI2SANQwghjguwDoXa
fnN1d907boo10FqlJ0Dz5JYQkTCSF4ylKZgKBwUUCIU7gyEBFxQLhxLsIIyWi0UkDhA1kXJi
drWBAqWUHFgEkgoSJUkK6sagUxEquNLQJU3UAyoAwpDacV2ForpaLi6bOyNAiO9Xi4pCqGKH
7kMamlaatiT6aDzBiBOONZYuBEIMk8pq9SI5F0yTcVF17e3l1dvXb+e3N02ztmGYjeu2n83q
mk019WbaKZoDIwaCQHZYxZp9tVmjyfHRQVkFVwVVcet1QCImQgZPefoJiOBAhmq7dGICQAcx
BwdnEoQetCdLMcbpeNNsDGE6mQTK82FgohgCAogKEn4/2nPY1k47g5S7g7ODG3ggyhHdk/H4
wflZs1y+/O7F588/Pro4MxXtO951s+5pI3vttddee/1k9SOFVPZF5d/sIgJmRQj3xilVVQBE
UDMDJA7b/XlwNwEVT8N6s543m5OLi8ePnkoPqIEQDVRc0D1nxCE4IOWxCOVreYA8OgzEiJg5
Rb4jSDkAALlv84zdAJHBnVSR2UJ4dXvz+PmzdTu8ePv2+fG5m1onJQdmIJKWUJDcaCQUkiU0
Ablsll2sR+PDKVe3t9fftktd80mcTatRMT6G6ayreLFaffPdb/z9t89GPBEd372fltYHXGj6
7YuXb+YLqMajo7PR0fHxzz6teKXtzVTpV4fH35Q3zbp5/e23N307evRwfH46ms3CwQS94ERl
UO67MnrEQgc6xklIkLpkdUEVFWamQyBLQZLrxGPZRr2xEqcLbYITSgVYgq0NLEZ2iyLjpLU7
x9IhNIPeyrBGKplGDqXSSMTcTIbevEoGSLF3K4qpk6bUgStSmXojqoPTsEllMRbtksNA6JRI
XWywXguPgcrBWrfeN8la81ZCUlMSTRuwlZMiqiqDTwAPJ7PDqhwFK9lI3d0HBuVARaXOQ5d6
u5uMYtrMv/vt31y/fNF2DYCWaOU4VIUM7WI5bNBQBzE1UyLRKoTJKKT1ouuvpPPJuK7KiZlL
lwhINBmCohEjgIMbIbJDzg4SQAU0dIKcHOTgxkhilhBSwNvUx9mIJ6Nl0yLSeDIKRWTMLVFg
xMBE+cOHOZMQmLOD0HN/irYO9O2uK8cCCAsOgghgjy7O379+tZivX758+fDsrIgFgouqqQFA
oD1Haq+99trrp64fH+196PzINKl7s/nWhA5galvUoZOqgYqbg8GhDhKhAAAgAElEQVQw9F0/
bJrun/2Lz49PTy/nC0fXJIxgoiKDgRGTirqr7RpdWdtyzZ13+ce2wzHgds98y5zaJnqYay8C
aVTX7ZCul4tnv/j5K4Kvvv3uk7OzwDis1xGdA2JeEQSAiKou5FSW3vD1alkqz6aH0/Ls5l36
ann3xtbnx/EZ9DHCCoa+ZD46XL4iBTicjjAlff1+WpayXj9Z9rDq3l7dvPjmjzeq1fH0T371
xecXD881nhzy9FP446vvVpsrfLme3N0cvDkIs5FX1enJyM6eiWy65bxeLA8HP0JOY03RhioJ
DaiBrQgGoEYOiEiOCNyumoghYjCHGGrz0sECMjikAf/D//ZXv//N648//vhf/MtfHZ3UbjPi
ZMYEZ25Hf/WXL//Hf/u/tE335MnZn//5f31wwGlYxFD+7m9fbpphdjA6PT25vrp+8+qtiH/y
/JOT4wPRjrhAEUwQIRTOtYeRWqUwbBobVm4dVKOR1Rdx+pgPBgIl6lWHQaxL0XQai0kRRpEG
6bwzwIAEYRSFeaWdt7ezNE2Lbt2uqoLby9fy3csLIi/HYqmqiqIIkFukomBIXChYB+rEnoZ+
3ZHqdDyqQwgIBGAI6qaiagCMjg5GSFuGVE6wdgBHIANyoExxd1d3A9VIrUuTdHRyCGVYDj0H
GtejoiiYODAjEQIwbHkHRKxu2elHxABAtF1BvbdM5eYUIBORAzKimx8czM4vzr/+48sXL777
2fNPjo8OAdxhgL3TfK+99trrn4h+hCO1cyDp9wRO9y04KhNvXMzATFUzAx3RXdXAzUzbzebu
7g6Znz571qekJohsroyEbuqS3Uu7/N3vIVV5YfC+GWZ5hvihN+uDLcLtGccsMALh0PdcFr3Z
6+Xt4y8+j5G//vKrp0cnoapEW3JDMAYAQCBXt+QSYj2ejKHXjZul5GAym7So14v2ZvG+vXl1
cnrQe1nWANXoatOVzTpU1fl4NinG5bp/ZPF8enJeVN91y6+aOS5v57eXL//AD8L47PC0HtUP
L05VNq++Xfri/QTloxHSzbJx9qOLdH5ePPw8hrL7+kX/x6/MGqPGuSfQolX0yEwiruiIGCkS
JHTezJfaJh4xAoOjGhgEAENHQHKP767mi2V3eHz6J6NPi2pKpESx7+v/6d/+h//+v/t3t1cp
Dfqzz/Rf/ys6OZ1i2SxXt3/xF39IqTo8nD58yP/rv//fHz18dDe/++Pv/+K/+W//HNCTDMyR
laNDMI6mcrfZLBF7JO2RRTfepgHu5BQPNixtBJVuSP2I4KyeHtU1uSt5I7ZAWgCn1MemFddW
lLsGAc4PT0frtdzNnw3dg8lBQmv6RgDregQIZo5AggJIkUKXhiR9mzrpuum4nkwOxnVFpujm
5jIMknoRI4rgDIhAW8Yl3hPyMefoQPDth8jcBVyZNiANajycwqi6Wy+Rw8F0FmMkRAIKIRDk
ZO2tjYmYtp6rD/Q9pn93KBWxUHPNGd4cMBASPX788PLNm9v58uWbN6fnZ9EKc/vw6Ntrr732
2uunrB8c7e1GbPnCOpPN778pIqoqkmBb03BgjsyqottFP+2H7nZ+d3Lx8JNPP2u7zh1UExOg
GxISEjJCTnzz+7S9LXZzt/HEpiqiyN8vqW93BuH+XJW/awQeY9FpMkRn6qR/eXf79LOfxRDf
fPnVUVkeTkeyWKJCVAd1dANQNGfRWVFPiuE62WYY1GUtgvVInW8WbfrmD8N4VJ4e4aTUtlOF
q81GVour1ea8mpxUoxLJzTXAuIyfTM4fP324Uul6u3310pvVw8OTCDKJ8eHR8Zfz6y9v33pJ
T8YHD0fj1PXfvnxz2dvki19Uv/ovmpPpzTd/O21S3SxpsYk9l7EsJwWQtegECESgUFBoN12/
asJxUIpiiqiEmDFZsSiePf/o9at3L15c3ly/E3lWUV3FyYuX1//m3/wP//5//vKT53/y4EH5
67/+ddu3m27V9v7q7V9PZ8Unn300dNW333yDBBcPzs29qurZbOoABIwxmLsDqnmmyzfLlV52
M5yYds6y0PXt2m7mfW+0kLTSYQBhtsNJ+bCajhzFoCW/kfalAx7PjsZlRWDzW7q5q7u+ns9D
O9RD4q5FHa4ptZjAhBk1NVwUHAIhu7snMLW2a9b9qrOeGeOsLkYjIayKOgAMbdumRlxCYEAk
Zqeta8kdtot5uaxCQMAAYA6CIAQJvHFJkeqDw/+LvXfrkSw7s8O+y977nBPXjMysyrp1d/Wd
7OFQ1PSMNOO5YSBLhjAGLAECDPi/GdCLDD9aL4YNGH6xZGig4VAckk12s7v6UlVZlffMuJzb
3t/36WFHRCU5QLdGImf4EAvoQmZ2RGZkxDl5VnxrfWv1ni7n11z4vfHEowvsHXPOmiXC7blg
ZkyEjm1Dm25zqe2qRF7h4+1cbBMMe7i/f3R09MUXx599/sWbb705mwxTittx7w477LDDDr/h
+GYidfvjrca33WtiBLnlo1IRJESErm27drWYz5er1dsffG8y21/ELodtZq86mDIhECmCJjHV
dSo3ADtnZus4UIdmxpxLX207qSIi0HVaFRkw5yGXahTvfSRIZq6oUoxPz88ev/XYF+HJRz81
52fDyhatB0spqiZA8AxFF/d8OUF30iyjIDApEDD4omCvcnGx+uzntDyE2bRarA5cQZMZkCzZ
r1I6Tst+uQKR6XRSDcsoEet4NB41FBddd3163F6f75Wj0bAKfNdWV08vLy7Ozl90+C0qh2lR
DKfz58efduLeegij0AyLvRs7nOvgGsd+NCVfRKl8qFNDgZXIOUciWqf2qi7fGJsxYgRqAZTW
C/1wdH/6P/zzP14u2oPDg+GwFLH/8O8/+9/+zf9zcjb/H//8z//Vv/pf/o9/+39+9NH3u17b
flGEfe+KENw7700Xi/jG2x/szw4l0unJRfCDyXhE1Bv2YEnBzFFCTWZcuj51FlMUJ7HpKZ3U
9Yt5mkeswRrWZVoFj4eDMMQAmlQ4Kc8l0vjg4ePHxWuvD/q2+/LzwjVHe26gWBhh3ypZH9Kq
Xc1JbBgKP0hm6NlXJSL3bbq4umkX7bAYCGkLMQWqJuNY+WVAR3TdrUZlGcalpDISaLQhBsfO
LKFaDtnKlF0BkomYOUYRi2DouVddatLC82jYoq66thxU1XAQ2BfgGJBgveaXdy9e5f4DmObo
VwMA7322im9v4Jwzy8usQIiwtqRD6vuyGrzx+I3nL05Pzq6eH5/cv/uBpB4AYowppV/HOb/D
DjvssMOvEN9ApH7JqLE2mItkLy0RADjDiDmwM6mq5pwBM5nfXJ+dnRrRG2+/Y+QMOkAkQrG8
9ydqCmoGqHkjfc2j1oGc63f8mXitB1WvcPtTMFM1RjCwpImUmB0TmoEhmfVPXhzfmU3f/r0P
P/2rv4qiR7NJulqSaup7IADRAmDP4YEvX4ZYIyqzo5AsOoWpC1NEd/qSl1dpNBwZvFGUrQMo
uY391XxeFK4Hv5g3973+ow/e1bZZnJ2n1Lsgh1Xgzjd1O786P2UOk8mDDz6Apy9OX5x90vdf
PH1yJy49lWl8NL+8XM1Py6M9W0Vsvcxt0OGLrn5Q+QF7QupBlb2RA1BIpk1qb+qR7kUkMAEW
NENAMzSInuHwTnl4OAAglfr//3d/+W/+9Q++/CJ9+Hu/df/uox/+p796efzlaORjl370Vx97
6h4/fjQdknF/eFcQLMaFRnjjjdKUTBtVMetMOwNQQGBLZuTIWMRMNQJAk9LpavWsljYU0VOC
VJbF0bDcJ1c10rV9J67zXvePjt56e/jOO6f16vlPP4NnJ48cjTkwSGtdS3hj3Ul9pgU8fHD/
YDYDpjb26MiFgIbx/OZyeVlfr8qj+6Gq9iYHYTqsRsNl3byYX4uqQ9wv3AiLuakQVM6poYER
EgCgrm1Q60psQkCIpkomxKvUriDRsCr2JqsU5009nkzKwYCJC3Ie1vH6yDmyE2STt5kP1Cgp
T223m3q5wsg592q2qqAqqgkUEJmQnWMz3T84uPfg3qdPnv30k59/8P6bw8Go769CCM59g/K+
ww477LDD3zu+7i91NiHhrRinGOMto7ea5iYyhFxbZgYAkpKkyMx9119fXk2ne++8+14UjVFE
RSWZJsw5n6qQm08MCF79lOyF2i7osXN5i2nro8qPLTe+5cUoADAw9g4IAQyiZo0lay3C/Ozq
8uBg/70/+f0vf/DDz15ePZ7s62qFAURaNHWqFeKsLAZ1XasqAnt/MJkdGJXPz+50ErpWunqx
uAB2h4PxCtnYnS+WL89PaW86O5wFB01c9di/8/4bF9NwfX3J2rhevaahD2zW9f3Z8fMyDEOb
xuxrgwb0wvtCperm+xYPLqS/uRKxKkKfoBoEB51A3fYFuyKiGSM4BiJGxgSr6+VdYe+dKObi
EwTMAao+cIwrEfNuyMSlpw8+ePTue2Ox7i9/8H+p4HIhBBZ7+/5f/Pjs9Mm/+Bd/tL//0CyC
qkEP1oMJIhH6aGSqaIqQ8nySQE0JufDODITAyHtU7djVAdvgoraFpANfHokbtIJNigm7svT3
Hw0ePaa792orb85eXJ7fDGJaEUrsNMVaYlfgjfYnfT2shm/t743393pJpIEcdV0vMU7Hg8eP
7t3w5Z733nuajC76+uq0FsTOMAEEcperdoVRRR26ziRKBHTsOWdrUT5KLfcWI2Q5D7HTOLc4
vDOjQXG1WpH3d4/uevKeHCMQMCPBegUvf0BEqmZ5x1QRiBgQbrd6O+e89zmgX1UBAQkdOwRG
NVEVEe98Ug2hvHP33lfPT05Ozj/99LMP/+F3B4NB13XMu629HXbYYYffdHyztLcuDP7Fzgoi
VM3FdQZAogYGSQTAJPax61PbXF1edF37nQ++d3j3aNm2MQkQphQpx0bnIB+z7DdHXX/zHByl
pogEZmCGuWFj86iypKKqjHR7NAWoCoamkAANyHC95efIl6V6PKvn1bB6649+/+Kvf/b5j588
nIxzHDilBG1CSME7x6CqbeoTu/Fk/Hg4DXWanF0OmTjQ3EGHGDzOfDCC2XA4nB2Yd5NikMpC
ndVXZ6uj8ej+2O3xfhPjqm3dIkF9l2ifi86PzlfLk/pmmOKMvVB42YivvFlz5N0d1q7pL5rY
OVpiLIO9UZKd3yTEpK6B6KAkRkBCI0ZezZcWkwvs2QEqAIHhWr4SdaQMRCCa6A9+/8Pf+VDQ
cTJLCRn3fvCXX/zr//Xfnsf04Ye/9T/9yz+c7Xcic8eViTdA0giggAqoBICIzGQAIGZoCOJM
qlBUgRQiqpAj7wNXZYp9j+AR7hTlQ64OVhK6COz68dgfHOnDR3D/kc7u1rG9VDnTFuPiqmtG
qAPvxBQjM/EBjfQmxevaplMCRDUy8KYqOnBh9uj1uRvWZzcS4/KyuWyXgsRVmQCSmFAUYl+w
d4UDs77tus5IApfO0FRpM/bU9VsBU4JlaoXp8LUHPcPZ4hqd25vtjaoBiTnFQAxImntkcgVS
Tio3RDAiAkRTccFtDVJ5W8KHgABqto2S0jxqJQLQ3Pm31awfPLj/6edfnZ2cfvzxx++9/eZk
Ok0pNU3z6znrd9hhhx12+JXh64jU2siNqLnbTnW70W1qzIQAse8EENAhETGQJBVxplc31zc3
N+SLt97/Njkv1jKTaAJYO1SMENAhEiZdN8VBzvUEMMgkSTZFe6aWaZ0k2ZT7Gbn1dGorsiAY
qoGCqpiBoTFT5nujwXhRL1eL5lnTP/6tD0b7h09+8INhTEdUlV0rdd+TFmMMQ+tX0lMgNVz1
7LUoq96zNz9weBgCG6W2FycCaeph/3BWJ2naZoUJxoVqWnXN3nSfyQblQMcjPx2tzucvf/bE
r3RaTBBLKMaxxFAUiAzHJ0NXHI4HE6aZL4rRaK9pjq8u7x3OimYO54uhGxXTg+TJ92p9Ym8N
t+giKdklXD9VeTBT5oJaQBIAJAYGsMQkMTVl4cmAyIoBtakNRADeO/2dD+8X/p/1jU0mxYP7
bCjIxGRmaujNRilFA5JERBjFVLhvuybavMem1f5mZeeLk/N6SCU4UmUDDDT0qNT3Y8K7ITxy
/g6QZ9c6f15VaTbDo3vNdH9FFFO3XFy1EoGJQ8kOFawAdMCB4O7eQRE8k2t6ZaauFWiiUwvg
IAn06g1YRUUGAabOX65W87YNo0koh33XEgfiYKZd36pZjzYgmAJ6Q1Z0hmZiCMLQkdaaFinx
ZDTen940TdfG0Xg6GA49OxCtQkVqkpJzzM6Zqq6rY9bqc9jkIDgjcm7dNoPgmJlYLSvdyMwA
KJKIIdsEiZGcY0NDgCh9ioPB8LWH964vz54en/30s89+98MPfRn6vvu7+Suwww477LDDfzW+
Vtpby2MAtPEx2yYSU9Vyi4iJAAJ4QjQ1i5E0auq7erWs63K89/DNt9qUDAyBsjEcAKLlFAMG
QzBDAUBdq3ZqqpqXy3M2VY6zyskHefTEyOwgu09UVUDNFMFhLoxBU0yZlAmYqWAUERlyMPJN
237y8vjBo3uP9/67l3/5o5vj6wkMfTGcq5Wqdwb+81WbkgaBxfn1hTCwg8HAmFkFUypASrbe
Okm9CZTAJeHctE3x+PhieO+OKLZNjCliGcxRn9Cm1fN2+eLnX/z2w/e8cn1dL1HTMLkyIKWJ
H97dO2hXq2fXN/P2uMfUp/78stkPlRvPFOD04tx5KieVJmEPvVdn7BPXx/p//+9/+UNcQqVj
tulsX53rUqIiGGjwOBo4T1J4BUi9IfuiLEszDWFRhFDuHZVTaJvm+z952bStpN5gFQIrIIJf
rHoVqhupmxh76KIsl83NvGWaXJxenT0/Lur423dee3+yt0wAAhddf9N2asmhlMBD5wOBI1XT
Fliqys/2muCvm1XvIi7n3dlxkVJRDKeV12Y5n19OB+VkfyyxX1lPoXBl2UaztmubCCkFQwXg
GG3VyKr1zGR6wIwIYTS+MmiQVCwlbCVhVDJQTWDgCD2jEllaj6GAUD01EK9To5Uf7B/acHDd
ttF0NJ6Mx+PALvuhzMwIOQRyjIySS3hgfVYQAmKOUADK9vEcJbuRnw0IeO00V1VUZoaUkpkC
MCCoCQByYIfoGB49ODp5eXz84vyvP/r49cdvHexNGl7+mk//HXbYYYcd/lvxDdLetjCYAJB5
214MAKoKmGdUOctQVVVjb31s6+ViuTy/uP72d793dHSviSm71E1UzbKBHDTXurBzrCYqsvU/
mRnqKyVx63n/Jad5NqPg+h1/RvaskGXehZBERXozc8GHEMCsCgVY+uSzz+4d3Xn3T/7R1UdP
Pv7xz8fIo2owZX6nH85BPu51kfoT6XtIh+zumA6rsiLiflVLrdr5TqqeyAjRBQMEheAU6XpR
P/noZ+PD2f7hLHatKwIAzGaz97/1/vnnx09Pnz9+9DgFPLu5qdPClnAnuOs0r58v29Wq7brG
onri4FppEoG4QIsl3FwNyjJUh84PyChZBYaiIj09O3n+/ZOf14OJ6OC733v9waM3blbt6qx3
vqiqCs/09OXzi5OXb731xmA6WtRLYjo/vzh+9unDhw/fe+/d2MdPPn7ivQc1MBwMRtfXly9f
nrz73nuA1fxm3rRtNRjuHxz0Mc4ODvbuMDfdRfOz+cmLUfQ2maQ46k3qrn1ZLy61aUsQ5b6j
FdI1c1SNCnMDDcEz31xfdbxMyPXpV9rcTFiG7MZmfR8LhaJPRYrJdLFYqsRVM2dmSaISh0XB
oUhmqW6p7byaJ5QEoDSYHL57/2hVFp+8OP78xUsTYR9URCQCqEtpkHSA6HIsp6cmJSVroZ+n
rtifVPvTZYzLxcJ5fzDbHw6HBMSILm+n6vqoUzAQNbDt4bdNMrt9WCLx7dwn3BycuE21VTED
VQOUHOXP7KIYgoLJYFA8eHB0fHJ+cnr5xRdfvvaHf6C7rb0ddthhh994fK20d4u+AACaYe7a
M9tofDmAEwANDdCUEcWsbpqLi0s1+/a3PyjLajmvRURS0iRmmYcZIpmBqSD88m7g38Q2iGEb
rrNNYci2dCISEUTIbcaw6Zx99UsYmFlgBwZ9lPFwcn51vaiXb3znzTdev/fF9398fnxxlAYP
/dBPwh3f/Wx19Vzak/bqmvjGHJWjYTXaG44xYl9HjKSEIEACrBCAhsz3xqW39uXNzfObm/OX
J4dHdwaTYTUYYrDHb7zOf/xHn//kEwOhQFy5zvrKud9++PCAfWo6HY1j35N3CWHVNTDF4AKI
QlWwn1ShTCIYwfthq0nVGAhA9yejB3H8XGNn3dnl51yqK0aIOByOp9OpJk0xnRyfT6f394/u
7QNUZTUeH4dw99HDh2U5Hg753n3en+0fHz+vV82dw0fnpx9dXVxMho/ffPPx9c3lZ59/ujeb
jEfDxfIm1m0X5+dP/1Nsr999Kzye3psmU50L8SKtbvomlmTMfeqXas8Wq7r0Y+fapHPpq7Yf
r1Y9dwBAInD6bBLrseNS4ygBe/Z70xh7bNrGUmkam1WzNGQHZgRKXTTXcEohSaVYAjg1NkD0
6EoSHBfV0Wz/xcmpalugtxi1bwkgiO67ckSBFAytd9iCLWOrBc8ePRTGi+VCkabj6Wg8AjPP
ziHptqs7dxXlRQq0zfLdGvm9wfbYAwOE9d4DbPZbt+6oDe8nAAIUy0zfTHKev0DUyER3Dw9n
0+rmuvnxRx+9+87bB3vT/6qTeocddthhh787fHNFzPZiYABMtL4qIGm+BOTVJVA0IjAA61UW
i+Xl1dVgNH78zjt9SqACKqZqpmaSWQ4zgpqaiSKBbd/X334At73k27aN7ddzIigAbCdSYkq2
tXGtWSAz5/hEMkgiiFwVRQFWBT9v659+9fRgb/r2n/1B9+z02X/8Ub24OgiD7474cLz381j/
9Pr6olmeATsTYHttWO1Xk9IVQovrVWNtpF7QKBo0XYzODwbD+4Nq1dfLtn755bGvikcPH4Xe
RuQ/eOutu1w+/+pZKGb33370rF1YjEPPQ3YK4K3ULnoiFQUqVBUNgCBV2AM5KsT8Ilrqxftg
sVZoU1z93vce/8n3/vhYXqzsqoup61fRmr63Lr2UZK5wY+4OB4ui+Pf1ZejExaII2r/9gOr6
06cvFkmSxHjZDu5Mx33Zyerj917T1w6hwr9YnP0otvW9cc3ArnPvHs188KHoxn+4vz96uOcP
Dvnwk//w6Y//3ceiM/TBMThB1yMm1xFcmiy7nrtOlJR9WC7GL44r59iilzSrb44cDtEqgzEy
ADlCKTky3BiOh2Vt2im5UAGBiSCI9h2aETGBgRoT+6KwUF01dX3SLi9OTpdz7Lr9UFZIsRcW
GlVFibgPgZnqFGtNLUhLWh3tl9PRMnV1XftBuT+ZDcpBzjCQmFSVnUMAtyl/BFhvQNw+9rYZ
m6/qkiznuzJsA2OJ8pG5PUpVNR/7RGgIaklSMgMESn2vauPh6OGDB9dXn52cXHz62Wfj7373
V3GO77DDDjvs8GvEf1FQTUpJU/Lem1mu28sfJImGiVwAMJVkEi3F5WJxfX05ny++9Q9+5/DO
vT4mVTFTQvOOUzK1RISghkjMYGabSoxXjTTZ/5S7aOzW3tO2qYaZ+77HWwAAUXOIUcVUAFFV
UQ1UTfPGOgpAn1JhVBB5Icaq8f7i4ubmZv72/Qcf/PmfffXXHz37+cdj6fbK0XdCeZcOn81X
54u6uVl81fai+81oPHKD4rBIk6a5WsSbWtukhomxaVvEVE2G4/FsNJoum3qxWl4en7pZLIoq
Wl103VhtFDyV1Xg6sso5Smd1jaIVu6DQLRtYttYkFAmY3fzkqHLgCZyXJBJ9sL6ug+8DrKah
+e13Dr+z51suDE0NosEm2zTLT4aofR+7xNE85Jz6XOij2jRNEkFCImYCj30ZWMViH5kZdMzE
aGCKzIxAQKU5tmiuB67r0dgSNX1HAuPxeIDmQ1LvYWFSS9eamCiasoAt51WSpEmbRWA7CG6K
OEAuFVh6MrMEHaoqDELonTMCNAb2wBAl9r0YURGKto+A4NgFZA7FyuF53V0vVjdd26ZYumLP
uyGyOV/4YlCWJmJJOkmr1NUofm8ym00j22VbK9re3bvD0ZCNPG7WFnK8QQ59yqnnOWsDzDaR
s9uKJABg5kyViEgUVC07omKMeRwFm4noOjmWOO+/EhESEHnJEqICqgkoIN65e3/w5fPVov3k
k5+/+/Y7v/ITfocddthhh18tvo5IZf1ivSMHANshkIFuMgjIIQHEPmmMmKLGbrWYL5fLJPLm
2+8OJ7Pr+UJNQRUBiIkUIUcpoCICGpqB5uXwTWxVvjjd1vtucynYCHy0ueytbwNGTH2MpkKM
lP3pogDr+0JuRCYwQ0vASqUZonODaR3bn33x5cHe9OjDbx28d/fFRz+8fPJinMp3wvTx8GDu
946X8xer+cnp6dVyMZ1N9saDyWiExXCBZ5enF31MFEJv5gH3kFhA+uhDOXbeUr+4ujG5UmO4
qceq3aq5Xi7PY/tSmu7OAEdl8P7u9GDIbv/OJJ5cdS/PCyQVk65VEwDwAAiqqsBolgwSkwSN
zdnZyU9+ZocQfUMe2TE5ItKy4lB6YmQHhmYFSZkEewAwAFsvDmAfnQETc0rJVBwmws4ULCGY
WYLU96nX1GmMmvoUW6gbji3Ferm6Wp2/EPX3OvVd5GSuYH9AFNBZsmW7kn4FFgPgEGwo6aBr
ByouxQnRvulAMACQKoICYUKMaq1qbdgm7RzVKaaUjCBZEhVCMOedoYpZsl7BK8wRz9v64urK
TKuimDBP1MbMVJamihLb2C9Sr47K/fHd2bgjmHdNn2wwHk4m06oswQBE2TYeJ0RGAshBBZsO
4ryy8It5sFtCvz38kFDVtkLeL2VpYo5dQGZel22bqBkwIiAKGjCAQkxy7969N95446cfffzV
Vy+ePPniV3GO77DDDjvs8GvENxMpyKUWG5FiE3+gqpblDfdjIqMAACAASURBVBERSSYJUmqa
1enZyeXlZTmo3nzrbVGomxbBVMVAAZRwXRtL9sqZS4j6i3U0tuknzpel216oLZe6PYvKiQyK
ambMzIwpJVH1zGaKiKaaS+IK7w0gISCSGTpCAnUOByO/XK0+Xl7vTyZv/MGfPXx3/sV//OHJ
8fkeFjP209nktdng+Wr+dH718uV1vZg2k9loOMbxLHVyenWpYFT5oghNiphSARgQHBsSiEas
G1j1B0oDcCyG3iGWVzfLFy+uBw/v6sT96Kun9fX177z7/qPJcGR4+emXuqyHIeTICdCEpmoK
BbEj7wImK2zUvIw//H8/alEBARnJETtANq64KHwoXTEsiiK4QFAk8opM7JiIkVBVU37lpHXe
mwoaxi7FPsVeYpNSl5Y3Td9KbCX1ErsYO6rrSrECN7hZ6VJ5BffqgCvrekloHSbtIBhVpfdI
gTCNzPYUx1HvmO0F59wgsFWILGIgAiYEPVlt6UL7JWFHrkVIiFA4RVQCBQJEZFL2IAYCfdO3
y86iXqyurxcLMJiWg71yMHGhAiRJopJIG40L6/zBZHq4b4yXy1WvUgwH0/GkKquiKAgJzBAN
VWnjE8dN3Gs+FM0sgWV5G25pzXBr1LTuikFSXZfGqGr2mOc3AAAAwGbAFFQlSZ5XrUeGAOgY
yQDZklkSvXfv/rMvv2rq5icf/fRXdJrvsMMOO+zw68I3JJvnCwMSmYiktM1uVjNmIiaBPvbJ
1Lz3KfXNclUvVsvF8vW333vw6FHdtUissRMRM0EEkWSm6+tSBhrAqxU821SY5U9zucdmiPJq
bQpgG8jwyhFvgETIjlXWZhdC0qQqggrIyMBOsQNtzYSBiAKgVyyMBuTLcrjoYH5Sf352dXj/
8J1/9k+vXzx78eMf2dnFJMEe8WQ6uDfky667avD6+eW5XvCg7D3LaHjRNZpi6agEHHrnB0NB
TamLbR1iVzoMpedOMWlJRMzm+cHenecXF/OLWrt40SzY40nfTQbjh3cP3OXldb1s25X0XeFc
kcgji3MBCmaPrtIGA5bSaOxjaVj0appExVABVUlqbGsCY8jNO4kjkGVGkp88MSPHV5dXy3r1
8OFDE/M+NKs2dckUUh9BgJRAHKkn5MLQoSqBhFDuH8xTe75YXWCcI/YM6B2DaNf1vSigx1RC
KkkOgI6cmwLsmY0NEchS9IxAFE2FoUW5EV2iXpDUzonnJVhSGYXgnSNGJTQEIVbno1hqUheb
rm5YsdNu5MvxZDYORSFWqlGKUWNtfc1K0+H04C4Nq5vVqm1aVxYHk4PBei+P2dAjscuJ50JE
ORQ+e6HWM1E1UVEwU82H/fb4zIReNrb0zIfyMZtblIjysOnVsQqAIoJIRGy2jkMDNCYkZHIA
wAWRkts72L977/4Xnz05efnyV3OW77DDDjvs8GvDN1XEZHe5yNo/i0SMrxiMkYMiqRCbp6Ta
rhbXYGjqXn/87mA0fjlfAKlZzJEHIiqqhGSG0YwJ1u0upph3o3LlHmG2mwDktHSA9eZ5HoGB
qplppleUs6URzIyRwMCiqRoZszHkOB8gQ0IgNetip3kx3YCZDVABHELfJwUbF+Oxd12nT18e
Q0lH9w6+9c//+/rZ8flHP784Pi2bZgI484Ml03VpF/PFydXLlWkiY4DWtG/bOBjguBxWQyJk
NhgWq2vwbTt2JrFlSSW7xqw3G4/869W9p6vFp58/S5W/98bDk/mqXcV6uj8cT8NeR/MFsVMV
R8RKwCSaemU1Q2Qw5mSlcEAoJMK6/sQAwJIagKEBYaaZYrIuhQYEMwUQ1VAUq/MervtQjR07
Z8RNkWJUgDwIAgATXe88KggqUmqUkmmPdhO7a7UakMoQQiAGc2QpWuoD0UjcNJV3nTvkMAhW
5BAwJQSOKAlTB1AbLBVvjGoOS8IlkUKRvDOm5FlBO9FORNlIxcceVw1ersom7QNXRC4MiZAc
RZVe4jWkhKl34sfVcDbF0tUp1Zc3PoTD2d3pdFqEQkRNFRQCO8esooDAzkFe8YSNTLweuSpi
lvogC9nOuUyhtiEgG3HZADnPmZzzKSHRL4jRRBRjBFACQrRcDGCmAKhqQAoASEYgIE3li0ev
3Xv67Ktu845ihx122GGH31h8A5GCzVvqLGmAmQlI7g4DA0VCx8hsKTWLxc3Z5cXZfL4IYfDu
ex8IgGDUlEAiwLqJj3MeOaKBido66zzXnm1q0CBnnFNukDEkorWjCgBBXz2a/F0NEUHMTCmv
75mhEQCYAjFTLpklAgCxpJqY2UPOT7f8O3VmmklcQvLmB7A3Gq667qsvn50WxcODg7f/9GF/
enH+s4+vv3jKy2Xhwx2m2d7gtUl5sWpezG9eLJaXMc4BzunqpT99Oh7f39vf35+U+xMeh75Z
LVddoKWezcuYmArvaej4Hzx6+LsH+//fX33/yfnpCMOiab9YXXmiO+QEUrA0IgQFx94RAFMy
AQZxEDEVag64BFKQ6F5tNOZaHQDI3mnIq4tAa05lYGCsQEhsXPqKYaWJ2HuMyYFZtq0xGmS6
qptWIBMwU0ByiiSBhQwUPLK2Gts2Z60i4nRSPhpNR22k88sQBcwSQksWQYksdy53mBqlheiN
wEKxBexcaJFEyQRTTPPFXGKXTM0EQZ3qQG1KPFXeK8KMg0tqZlFT3cYOpGHrPPCoGs72OLhl
08TlyrG/MzkYjcdFKIiJgDwzOkwxuTxzAiMkdq7v+ywK35btMhNiAAVIkjJ5ygbzGKNtqrVh
0zZjYCLqHBJlMXo9uNq0GAOiqUZJshGp6baSqKaA4NAY02wyunt48PT5yX/b2b3DDjvssMOv
Hd+wtUebDywHjmcjrgICEDITmSoBoGnq+vn1vO/7pu0m0/0333676zoTi330WZm7ZS7JLt2U
Ut4BzF16uMlMz06srR8lb0upqhmIvlL98hbhOilgLWH9QiSVbdpkMGuG6y+i2VZzwXVA4vpb
khlYMkIDg2EoPHOM8fOnT186fnR4+Pqf/uP733v/yY9/cvHzL6tGHLiBC3vD/dervetZ/6Je
PqsXJ6l92a4unp7H4+OLyXjv4Z3J/f3JcHQ92OuKKXE5WLb7XMyKwaOq7BIPsPif/+yf/vT8
5C8++9n5zSU4u0or9U6KiEM5VL6XipIrr9BaEkSALAYBgK4jI2EdC7mZkcCtZsINMP+DgLb+
D8xMneMcppqf+Lw/SYjb4darJzMfDwzESmxEgEyoIAYpv7iK7Hg6rR4e7u27QFfzFJgduTIw
E4LmB71heSIxdV237PrrFBvpo8WYIDbaxdjHjiB5sEJhiLznyv1QjdhVRMEBqiaNHUiP0lrs
UKAsR7PJwWSUGOb1KrZdNRwc7t8pi6oMFWwCCADWwvRWI94uMeRsAtzoxdttBgAwUYV1lkGm
TdtG7dtHWr69iqSUtvcFeJW/v2VpsJljbV+yV7I1QOFdTFqVxbvvvPni5fnXn5477LDDDjv8
vePriFThvW2288w0LzOZKKgSZwsIJElg2UEbY9vHXufL1cO3vhWqct5FSwqiOSEzX07gF5PK
nXP5f+Sf+ItyCWwveJkP3TZWbUlSJli3vcC3L05bl3pmErbmHK/MWLhJVcjcbnN7ADNCHoRS
2Afn2r775MUz5/Hw7sH9P/29Rx+8d/nx5xdfHfeLvuiXLuEBh9ne/uuT6VVqXy6unyFdN938
/Pzi5vrmZDo6urt6eP9wtl8OpzivL88W1uqRd9r1T3/2ZPLma4/uP7wC+ez0+HK5EAa6e1jd
O0hN1S1jf97UXVcgoWcgQzSXJUzIT5pteNGGPhngJhYSNlxo/exsZlJIiIoqErxHtaz7maht
xk+4vt364y2lQlAEBRAwBQAjYPKiyMRJRZlcKJm4bhqOfSiDGsigDFUhoJJT5wEFWcRS33bN
quNV31OUFCWmPkrX+6QDsuBwFIqZL2fgpuAn5HzuIIbYkzYmPWsMxFU1Ho24KiPAou/6NpFz
B7PDg/2DQVFaDioTMTPnHDNbju1Iaet5AgQ1JWbnnAFISgZAzJbFu/WwFOkWi0op3d7dWx+o
RMF7IcrvDXATN+Wcyz90e2BnRnV7heKV1Q8wGYIJAd87Orp/dPC3Opl32GGHHXb4u8c3m81F
BHTdWp8plZnlv/0iGlMHsYXY9XVTL5umbSPA4/ffRe+17QiMkSSl2+/+M0naBu3kbAW8xaK2
q+N5kLBmPLRRSDZXIFFRUTPLQUjbCdZ2nQo26kz+btnukn3BtrGvA6BZLhUERFTRZAkIEYF6
I0ETYcBBKMuyaKT+/MXx5y/h3t7e0Yfv3/3d76yOz04+ebJ4eaGLZWjMEz8ifjTZ/+5oeprq
08XietFeX/eLxcubqzbdOzw42GfHjXSxrgmByvIa4NNPP3Wr68nr9+8/eP3kJ39dXzXXOO/K
ELxz4xHUtDib10nLYSUFspkngrUhyowMITOkW3znF4PiTQ1wy6UMYT2TU1NHbFlU2tr5N/wJ
AMg2z1eeZoGgqUHe3FQzMyQBSKpoIKYuOHRe0EnqpRMXxULB7FdqyVRMzfKypEaxOsabPi5j
7PvOYh9ExoolumHBo6IYlX4QfIFMXYSu1xQbkMb6FqVHtMoVs0k1Gah3MaZ5WyO5cjKaDUZV
NXDsQcEUvAtd7HDT0IKbbAK4xbbBUNczOUAiIMxR+1nVznfcHrFbTnaLc28aYxAzV9uqdbiJ
pEopbQnT9otb8gSbNwDMbAaxl6oou5gA4OHDB3/7M3qHHXbYYYe/U3wdkUp9xJyswwBqaMCA
QGQbi5OiGESw6MBi2zXLVd205XD0+L33JAsVYqhmAMy8FfK2Q6bbCottFsVxg9sr6MyMmcaJ
bF0pWcuDVzod3J4w3R4ybaQTQ0A1SEkRjZkA0TQ3I4NITkhQZVQw7aXyBRqQoImxQ3DENAhj
bmJ7eTk/P7uoyurewcHr/+QPrGmvvnx+8ukXi9OLKqYyGoMdeN7f33MTJz3edPGl9JcvrubH
L69SV3R9Y9inlvDO8+Xyk5fPll/+3P14UK+aoqUqwtBTEbHp6zNYdfN51aYRORG1RB4pkyfO
lnl41VGIvzix25IGsPXaI6oRc+48EUmZM+XUynwbzLQyz6Uwi50IAEZoagCIRGaZJQAiwsbU
nlSICQ3my1qj+D5CE30niKpRwDBpfvUMVGOMdVN3sUuSJLYDsIHjvXJwWA1H6HxUFsEUTfpk
2khsLEaCFgWGvprMRtMhMHeabmLf9Q07Nz6YTSZ7RVkxeU8+9UlMxVD6mMMvstcbbzXfbVxK
+bfFdeVLSlvOBABIhOssqXWgGgBklr99wvOoKet3tknpzMzJOee9jzFuk2PXp9VG+9uOWrfs
ChGDQxF1hKDw6NGOSO2www47/Kbjv6hrL/+bh0aSEhj4EAgxaY+EzJRWXXOzjF1aLFf33nn/
3sOHrSTI0QVMALa9Dm3nSbCR5wCANl+/PS2AX4xAzGMQ2txANa+BITsmwHzpytrN9nqGm2z0
7VDBbo2ssptITfMdASDXeiARkDpyYJREwcyT0yQpdkZYhjAsAvBARebt6rMvn2LgalDdf/vB
d//hB/Hq5uqLZ6sXZ6dfPe0vr0fGQwyV+WHBd9249njWLlc9JsfaNKt6/uLy5svrqyXEFiSd
n4+L8oGFMtKsIW2lvr6+gdSxTQxjknrVePOVjUti5wJKQgCHvzR++gYgAAKogYmKKTqmXFNo
YEktE4jtbA820RJZEzUUAzBCIgMTkdhJRFNkArSkUVNXr66SlAYjtaHBdey6tnaeo/SSkopm
IkaWJsyDqihHg9JzAeZVgyj3DcSECmKx136laeVMRgFGZTEZU1UowWXTpLZmpDAcToaj4XAY
QsnImF85VSR0nlU0SgyvCq0BYL18t51RAQAaARPkcdSGiK/9UJssqC3d2R6T+bi9HRZlm1W+
PFLd8i2/0cdfvQSIuPFI3aZfZpZSJPSAwMjOk/P+b/Pa7rDDDjvs8PeAbzCboxnddi7rJgRh
rfWgmFjqV/P5zcVVU9dt17/z7W+NZvvn1zeYlAzR0Ahjkm2EJvyN1HJkxo0Yt1VGbnMgMwNE
JuJbtRsma48UEhMREyPhbaN6Jkbbn7IV+/LauWkea5F3Lv9mCACGKMYIjGRmUQyRAMnATBnQ
0BCMCu8EZVRxNRqvYnOzXP7k8tMnRTjcm9175/XDb7/zWvu710+eXTx5evHVM1suK8VCiSMe
AtwpKghVCgNBN2vTXuFPmvm8b5S5VJyyG6DbK8oa0lLSjfYSKiaWLmLXlKCTKGNfEgVTJVRC
UjD6G/byPM3Lgl62Nb3yoBuAWVZGg3eOOcaIm2zJTJtgvTFpr14qMzBQMTVE4PXISxTJHBkQ
mqkkIQQmwL6HKIQAJn2ThKwo/LQoqjIU5AKiR3RorEKSrG9BkplFSZ1qUotqvUswdDycVINg
ZdAiNJq6vhEFH/xsb288nAyqsYgBgHcBETVJUkkxIRExAZtDBl0fV977LSvaanMA6wAO3Yww
86+V77JVn3HTBmObTb2tWrfFeqf1VkB/vm82ReVZaWZX23cRW7H7NqNyhGKWTFNU+aYy7x12
2GGHHf7e8XVEiuGVHgG5mVXVe09IiGgqkK+UXVcvl01d920bfPj2tz4QyDHj6gBFLRug/uaa
EmwEDsesItvNpo0R69VVBDcVG7fvmG+VLTz5jtvLm3NuOwOzTYMsrov8spiTeUL2rFjXdUQU
fFBQEOEcAYpkBMk0IQCjIBKCIqBpo5DIHAdU8RqOBjMaWNM085Or8+cn5Nxsf3b39cdvv/3e
W83i8vmzyydfdKendLO0ZeeUAwU2QpSR4dF0rz6YriA1bWtN73qBKEUwNg2uV5GIUCuomjNw
BgKM7Ig9YiRAZgRNiPZLROqVERoRzfIiXt6YyzKfqogkwpKIU0rwywuPuVHaVBUQYDtHZAJg
sJwLQN4xEqmCiBAIoxGCAx0UPC3dHtLYYeWwcBQYi5yeFBMmdQYQo/SdQwDTXlOjqQaRIuCg
1CLwtNCBUwBJqkmla0NRDUbDsizL4cgFB0YOvCNERAJU0ZQE1ByzCy4rzxy89oop2WbkY2be
e0KUW8qyqGbOhESwZV1EtDkU80Jf/g752Ash3B6yMrNtiPt2jmtm2dW+ZkjOMXNWotdPphnx
q5JjRPTOowIxkQEm+VtOG3fYYYcddvh7wNdKe5s/5Aia9Y0c+6RgZgymakliaprm6vqq7utF
vZpMD2d7h6lTUxCNxEk0at7qkmT5Qr7xqmPWQRDzQGA7PYKN5fyX1MCtvLjWAYnyp0ooYGaa
QBVzOI+hIwIQU1FlWispomoI6DiPD/JF0kQMEQiTqYERoEhmWJvpmWwuukCmoGYCpqC9JjJg
cKXzjnlQjLu+Xdarumsvzucvzm7I0f50dPe1h2+9+UZISZd1c3x6/NlXly9Pu1VLXTNqxQEO
ghsQmhH5AlCtBCj8AKh89NrSZKmgMcUG+4gIEvuFDl0qTXtzit4MRIw2caawroUDAlWD/AyR
GZFmg5gJqoIKpM4RSeyJIWoyBCRCtfVuZuaZmyU+MANb3xnASCWkONDYSS+WCLHwfhj8ILgh
wgiwBKwASzUvyiYYVdoeVABATVNKUVI06U16ssSQKofDiRtVXBXqnZh1KqpAxK5wo3FRhCKE
QlWAuCgq51wSUdHgAyDEGEUSoBEjMRGiGYiaSHTsFVjNkClP2pzzIskMxRQMHCFv1xgAclQp
ZQeVKiGmNWGH7PDjjVa43cjLR2bahtZudOR1OtRGBDSzJKJm6/cmhKpmKkgIhppDpQiVzLZZ
qrojUjvssMMOv+n4WrO5yHpjy0REVIRpK5khgMa+t6ht21wvb9rUXy0Xf/iP//jw4G7dS+qT
Y4wS+9SB+awxMTNs7CYIYAaopqaai/I2etzWfrt1pWQjMGy6O9ZAULBswDI0A4MNtUoqsE7Y
RECwV4t+agRuq+ast/iMHSPmK6utRTE00Jz4cMscQ4iKpooqDiFLfkAYbbM2x2EwcuVgpGZ1
bJu+uZ4vrq6uCWAQwmwyPnjv8Xvfec+6/vr8/OLps/mT58ubRVw1nMSpUZSQC+C6riJ6zZWm
ZgxaYZpOO5Mu9YxYmJjTRKkHDJKD2227h2iAOf7AYE2E1TAZAAIBODNWNUlJRBXIWUFu1Ucz
MISUX48tgSKA9bcCUxSEaAoghcmhc+9NJ72hDyEgDolLQI6RtUeV1HWQBBVIt4V0EtF6067v
xaEEVOegGqTS03gQykIcdSpt3zOZYz8tZkWovPfOeyYCRJGUK1aYPQBBzuIAAQMBUTRiYmJY
jzmzn08AMWc64WYgmiQBADtWUxExgK1evBY3AQAgx/ATE9M2TGKzQLqxSW196Lixrt8euN7e
Tl0f2+vD0bJ7L3Ov9fIE5iONkDE/Bkdovzxn3GGHHXbY4TcOXyvtMcst97Gqppi2kYYiEmNv
GmPX9zEt237ZdN/64INBNVjczEGFHEU1NdCYAMB775xTkVsSkpm9MvdsbSVwS+zYXoRuUyjb
XIpuPzYzy7Umaw0xcyKD7fUsX+UI11IL3NoZvB2xuHWu5MdEtr7KbtxXuQ8EANGxM9AcaNn3
CbbWbCRHMMZi5AM67CUtV8u67+YnL7548ZwIxoPB3mR68Du/9fiPflfarrlZLs8uFifn3cVN
fXEVbxZWdy4hJfUpDYAJwDEWjIKootg0IEqK60ACh5a7cdaWJlxHRm0yHkGNQAmUABwSgJgh
klMB7wvnilT3KWo0TSYAmZ5CfioR1xGfhkRgFQKI8KqZkR9RCUioBn2i1DmNFKXFNGcRU0RQ
0AgpmQlDYlTPUAZ0AywCBl8UATyLxNYMCJiwcNV4OBlWgxACUjB0uI4IN1Vl8sEXucxOVQkZ
GdbpmkCZBPF6roT4n9u7th1LjuMYmVnVPbfl7lpcS5BIkZIeJIqyDcP//xV+MOBX27QEk8vZ
uZ3TVXnxQ3bX9K4AEoZtYQV0PM2cS59L10xFR0ZGUhbNtqnbz5muawDBWF3YmDp2mQi9921R
brGcEfms2EVSjUCEJHyjoJzgHT9LymVrKBXGwhvyanZNwh00dNl9gNeBAwcOHPhI8SNm8/xf
L5IX3Njc4qs9tjC383J6elSPt3ePrz792edf/uZpeYowYdLWtJuwBMWogyAiNyF3X4MjiRCx
9n+PZKltNGz2lGFjdcOYMtrFh5DAzGpGAO0GfVCOQ44IrMU62kV9Yutgz9caNprhFx4aA55b
CDGmJGdmZXiAwzwQWygAwEQCQThDuDCubvz6WsMWa623b8+nP/7xm6nW1zeXr1++unnz8sWb
V6+++vXk0IfT6bvvH/703eO3bx/evmvv7tv9Q3s6WWvcwWbiUVkmcCXK8pUzOUtEUHgEIYLX
gTuDpAY4nCIY4WEUi7sy8zzFPGudrNbmQerVBocCgHAQb6N5CAG7CIK6PTw5uAYZ0N3MVEnP
3lWXJ8SDsAuJFL4qKLMJ08VUry7ionSBMiy4mzuC2ahO08XF9dXVXCZymkUu6wzHYqY0vvMg
Gl749VwAwVyS3AA5DJgBNvNkJswCMGCj/8622HFsoVB7Op68aj8wO73hexIWO6vfnpMNpSqb
JPYsClvKOW1UPV9r/BXQlg07Dhi7sNn/xZ/2gQMHDhz4S+BH4g9y68p/76UI4VmeMdUwbY8P
D3d3S/d3T+0f/ukf//bnvziZuSuFm3sEMbGUEJZMJoTZaOoOrIN0kwrRzg41GJVtJnTbpCza
coCwPT42bYnGWONtpg2zPM9MyWlnW1zncKDTLmmdRor6thHGLuBqx/PgDkvXF4GDzSzciYmZ
mMkBIbaI1rqHq1sGNgbH5dXFzdWLrloj6tJv/+3bb/q/dxhNRaZ6Oc+Xr+brT7985V++Nqfe
+8Pd+e7udPuuv31Y3t722/vl8UxLR1e4EYcEyWLkwfnxU5JCkKciRRJuCJKAkzktGme1ILqa
L6TOImWuk7tD0gSWgVFETEjBDUEZbA/T6EFQgwV6RKc4s7XqvUbMhDoHC9HFPM31YqYi6u7w
mIqW0imUQaVUKjO4lKnOU611vpiLVHKEWqgZKJM7mZhZMrNCJM7nJafXMScFMRF2d1VNTWia
ZjNdp96tVPt5HtFedhKRaZpypaX4tM89H49JhSkP7u6t9QgfzCkfkER80wI/aA4Nf78RFUx7
nXWf3rm1po5rFYwnHjhw4MCBjxk/6JFSBeAe3RsAFoEbAZbVlohwXR7vlqenx9OiVH791dde
pr48ZMc9IYiTM2lrbWxjtHWVRyAHcXg4vX/dD+CDzrtsgKItSHpcstM2ODb9LtgFcq40K3uy
AsLsEfsAT96S1seRh1CBP2v3G7tswn39dZomokBwhIc7kSAoQI2iR5hqxruTgwNkLIWX5XxJ
PJdKIK7MdVKOxr6E3Z7P3z089rbAfOYiE7vo/Hq+/vSzG/DrxXjpvvT+dOrnUzsvy/nc757O
d6e+LNpamLlqqMKDzCM8zNmtgsXQ1UqtZ7KFo5RyYr+uoS/mU3v4L31q8eRkGaWQSfFBpKrE
xCCL8ELLBaGUwnWNAxAyCp6lTtxLuCCCvZODFoCJQdVB9WK+ur7hWqRWqbUqFScQ6ap/iXVn
Fq61RdcIngjmjL32w6XUca7d3SzTx3ia5h1lYZGyZ04jxmlw4vwhOdCyrLnnwzYuIrVWbNHk
SeW3GE9mLgCSe2WNOxehiPCWcVC364R9CO1KwbeViR1JyjF/q/C5dZIeFOrAgQMH/lrwY4oU
kutYRBBQS4lwM8v+euvt/vvv2vl0e3d/9er1b373h2bR3dw0PLcueISbD7Up9568iM+EqthK
huN1cwOjXUQntkTp/RHGfjMezLspZhGxVbqeC3QEdNWkRxksNHbKiOi9DwEjd9x9gWavLphl
+3xs5I+eJ894ijexmAY82/M58xYWY3cxliUKcwSdREVd2gAAC4hJREFUhM4eqkZMUssNz1LJ
uEOuKDy6n9rprp8ezd56SkKebXkQlNeT1KuIEIspmEypNe2tAqFdW/PetZ3bsljXCxPqfjo3
KuXptCxLA0DMpcA+uXr3yVJLTX/ONM1SJTMruMgaRg/y8F75dCkoZaIiIPIgy5EysaYJCEkR
elGYqEid6zzVmUAiZSpTJqMjQiIqiVOomcIRwiROMIsgCcaiWrkU5uRHyYSep+Nt4Ux5TlOt
VFUiLuU5fQA7iXGsgEGdk0hlSXee55SmcmGsOmgEARmooVvi+cCziy4Z1a48V2tNCrWuwZ18
5drHEQaf2/M2Isr3uq8PHjhw4MCBjxk/4pFKxBalk+7XpBS994fb29P9XTufv7u9++Kr3/zi
i181D9Ue3t3UHA4Kc1MtIsiq3M4bTsJFiqoGPftIkr58sJEMLYG2eJ7cnHiL7clfR4UvDwVG
kcLM4UHhY0PNg+eRhxVmL1pgv7Ftutd4oWwCc1+9NePeiLX/kNgD4QwSYWF4WDg7iFjqxEWg
3iKW5bwUc4RQETA3E1cG3DoFai1U6gWXV/VC3Y1cwxfvFt7dT+fHx6X7ublZtFYCAFSbdWUK
oYhwokBlKjPH/EjidTqf2/d37/7z/t25n9+8efPLz37hptaW6XwhQAWHKhGD4Koi7KDe2zZS
xpyog1kqprnUCRYcdHNxTY6wEOZpmrmQogWlL06EJRxZ4YVToeJhCnUGEaf3n7hkehMohAxE
Dp9qFWYzLyvZ1d6VCKUIsN6i3UAEMIIQzCzTNHXt2hWEfHXQhy11yZMADCFqM1qt9GukkX2Q
b47d4J0RhZCcW3vXrWaXrC4L2XkobNEJtdbxWmM1ns9nM5vnGUBrjeXZwyfvx7IfOHDgwIGP
ED9EpIgI4SCsBm0P3cYPI6K3dnf3bjmf2vl8Pi+/+/0frl++enf71szZPNzCycEUEOFSCjFb
79isIbFZayk77MYQWeDPfxh2chFpranqNE1DFRiVETUT5tgqceFhsIhVJGKP7EfL7TNFjtQz
hlEmidGeqOH9nr5856WIeySXMvOyFn0MMAAEijAEONJUBmIRKeEewmezx3AIaYQYapAEFQQM
cGeGELfwc2suNBPV5qbqFFzkSi6JIyhinutMBRzaT1ftiVRNV+eatnAjeJjaeqOLC7wu5/P3
//Hdv/7zv3zzzZ9+9atf/u6zL17c3Jzu31Gpob2bN5JpmkTE3KZpUrPWGjMTs4cLyqSVmKfL
y4vLqyBSD6rVDaoWgVomIfP24J5fD5FRRsO7p5RUADYJl6BgBBMgLN6DQaVyW4CgyyKI6P35
LLiPAuvqecuU+zzL2DnYmNamgYxuJ448y9jNhxla0WpvAgBM04SIDOekbcxLcqak1MOrnvcm
ZzKztYtiw95FPnI715WG5zU/KnpDQ8XG5EZd8ijwHThw4MDHjx8s7S0nMEVA83o9K1Ru0Xth
xOnp/vb2qevt08MnL2/+8PVvtZ17awQ2BzzYXSKIglmQE9aYTTUZTK018wn32wneH50xJKjY
PWyYfIe29Gwb34J3hqDl4abu4cSUBvTwGEQqHzb0iaRNa48aEUtam3KwbTb9pW+HPRwcBHIE
KDRcQFwKLJNCiZwqUUQyKxAJKOfLOMIYHholwlsXESIGURAMUbhYWFrY2ckRDbB1KhxIg4jC
PBQaAMNdiC9mjomc6hq2JCKFGQhzNTNTqywIXPLFwycP1+W6PfHt9/bi5Zef//xnj+++jXbP
oU1d1ZKIqFu4E6hrTzEyIsIIBpYipRCJAywEMBFKFRJhIu/umLkQRXhX23LnIxxE5tZbdwqI
gIkCTGuYeHa+JV2KCNsUxNFNmQIQgGQ/RFSqpCXOVLNPsmsjZhbSTQ/1CBmVPs44BxTKMYsR
GZYWwSJTrR6R04yzyGtmxFxqTfrGIrxSNCPm5G7JgLgUBrI4iK3yOLhdvuGMj4cHISPPWERM
zREgpli7KCx8TDXe87MDBw4cOPBx4oeIFMPCOf+ZswgJw4yZhKI9nc/3d3o+N9X7p9NPf/bl
Lz//DG4cgZSwWBC2qkVEI7TJt+Yp5H3AGDY2ii/YzZAZF+X+furB8xy07Ibb9Zn7LuzH3OGe
M23Sab53rA9VzLfZapnNmFkMoE3D2CIZI5MAmEwjYh19k1kB67vNOGwWbGGYuc0bzEVy2EyE
VxL3noZ9EIHgTCC2cA9HUJFCRG5mrkZEZS3xEMjd1YOINbyFE7MIQbupl1I4SgRzSHgWMxEu
jHAKh8pcr1988vL1Tz796enlm5+XF5/Szd+gNUMPa+YuVSiiWxr3OQcLCqOwBNCpL2RSCkRA
jFWncWxFKDVVc+ECYURQsCDM3AMjeMnDS61ciptHeKaQp8l6JcQg8vBYhcPUcqZpWt1aW72V
ciRg1l5XJ1qe0IzzAhhCssVdrlFkKSPFhjG62N1tO2ZqUa01c+dtUp5uebClFGzdfCNbn3Yu
+FxO463mu0oZdZ7mKsU9ryZsnQ/jkWtmzc7wGArrQaQOHDhw4OPHDxEpd6irRYDTRUwkHL0j
wnp7uL9rvS29Pz6dvv78i5sXn7TWzI3CKK/gmU2f03FGP13SoNWVshVlxt4GgLZ0n3H7KIXs
dSns5r9+4GfKJ44mu1Xr8tWtMwwxIzhqVAnN3SNYODvefYtS3L+92PmoIkf5xrOdazi01JS2
TPb1VeC2ZWcH4OHP/pst7d3NiJDUIDsjWVJGQZai3D0QhSVd2LUWQpy7mppwkVIIa2ZEYJ0a
4+ZAuGs4FSlVyjxPN9fXpj2H/xAETsLsOr5SMtOc4FdrKWVKA9jExFKIcmg15SRpz8/l4Tnq
bjevt7XGTPnDoLyZPx5bEY2329cWTiAQWaWlrRuAdmEBwxvn29LKvoE87dgEofzGeu/5XNmm
2olISkf7Yy7L4u7ZKJr9oYOy71dX7KITWmvLsrTt+EncZTfzEZu5UEQuLy/HyiHPSX/AjiqZ
W473HjgUqQMHDhz4q8CPESnzQAgLwlWDrLfTE2lvy+nx/u50Or29vVfHV7//PTH1pRNRmLkp
PItutKV4vpfPlDtNurz3TVUfILacw9yohkNlbzEZWtTeez42s3zYWjckIKC9x3Pb13PqD636
SnIQ35cX95mNSOFq25jHexhSxChHpsgR7rJLbE/nu9m60eZ7Hp+LeVh5lDZvP5mnymJm1hWb
l0skOyjdVF0jYxyyY9A9zBzIjn0yNTiAUut8Mc1w6NLgYa1PZQK4d4cGBVQ7wkWKFHG32GzR
+XN4MAsRZzNlbvck0CBTdQ8RKcI8wghWYY+Y2c7ntUMtiaOtE5Q5R/ytZvP1G3ZAtqZRIcom
OBbBxqqTRWHHofOJ2X8XO1/dWA+5ElYmui22fN1SSpqZsCVr5Kl5Po9bhlnsoqfG4/NzJXFP
+rWdzeeeiWfda6sjD4I47lq/ADCRj77C/9Ef84EDBw4c+MvjR4YWC3O21AUiVNv5EarF7enh
/vT00LvePZ5e/uTN13/390TcerPekkKFqhP5tsmNXW00IuXteePQn2gkkm/3DuUgbStp8lXV
oTRgR3GSCY2n7w+Y9zrem5U2xK01jmF1AbPvxnqMzW/PvfLV97v4cC7zLonRzJDzQ4qkVyl3
x57FR8lAh3XMcE7yY5YAZRd86m9uto0KISYhojD0sFIKSNqy9LYQaNrSHNKIrWpmChQRZhaH
A7iYLi+nSwKF+unh4fH+galSSPKZ3jsCwqn8RZGirhERnl5yX+ucIJEiwsSMgHkwS50k1cd8
w7HlVkgp+QVeXFyICK2zgSmHQzOLbNNaiCgrvvmpR4PbWlxzH4ZxZAl1i/vCTtQcy2ycncvL
yyEOYSv+jh9677ke5nkegQVJYmwbQhxbeXFkl++V0fWE9p6rYp+qkIn5tNNHB3kakQe5nvNz
5Qowt7F6/0/+wg8cOHDgwP8r/hv7Ci2Yb3aygAAAAABJRU5ErkJggg==
----boundary_998_71d08529-2a97-4e03-a5fa-4d4ed47c58a4
Content-Type: image/png; name=logo3.png
Content-Transfer-Encoding: base64
Content-Disposition: inline
Content-ID: <1@localhost>

iVBORw0KGgoAAAANSUhEUgAAAbwAAACWCAIAAAAqkqpsAAAABmJLR0QA/wD/AP+gvaeTAAAA
CXBIWXMAAA7EAAAOxAGVKw4bAAAgAElEQVR4nOy9OYxe15Wou/Z8xn+qgYNEipRIs61Wd9/7
roB3r/wg3HbQgGUFDQdOHAjoQIoECQI6aKADBgY6uIAgQZEUNKBAiQPBAW0CDtyAAAsNPL5+
3X1l2aUiRalI1vyPZ9zzDaqKVUUWi1WcJJnni6rOsPfaa52z9l57/2cv5L2HhoaGhoaDgb9p
ARoaGhq+SyD47xCdniIdkZvCc08CzhCJMeeGeumRAeSxMTbLJ3VReKtYIlq9HkNEDmu1VvAM
IYMR5TrgGZbe1WA0aAMArSNk6sxMPNW79uVKvlLwgpoCQCS8O62c0apMnA1UIXVWgTUhoA6N
ozRBQcgTyuOvv/xSl1nYCyBEbCrsnDyiCSijkfGBw8yBsUY6VRHrGYksK/6zn9+Y9J5q9ZJW
VRQsFGErcgGx2Feydg4oZkRRlat6pKWsqnrcOdJup6k3ztc2GxejsvA8IBUiHtmAGVNCXWME
jgLvkpNPzs5G3clqtnZjUFXe+hDzhIZpVdegzOzM1Ne/uvRNW7OhoeGhQ1kqOEJgfSyCoBUG
SajLarI2FEE7jWPsCDgoijJ3DsAFYZAkccw5AoCAsU4aBsQr5DwhFBXOehpyz72unNGIQqGq
fLhWTnKXa299EiU4jGtrTF3FYZAgb+rMGk8CHLRDaIfIo6KWRVbb8bIel8AAEcpDzkXICScU
0ijGDiNtvLK+LpEyguAwis1IKSNxjMMwYlwAIBbyIA5LLyd5RkPOuKCIYY69x1J6a5FoJzwQ
iFFKkQPDtAkYpmGMC++k1gIDiYkioGqprXOudnKkJ8NqPCxqbxATDHPksdVWYbDKym/alA0N
DY8C+sxTJzxxNTIaLLMo8RiJKIhUzEVAqFXOIS8IcAIaOeJtSmknDLKytF55BhYTT8Ea5ygO
qPAMBFhVK62UC1BNnHG19hIUSCmFwJgwZ7GvasYZp0hKrbVjLZH0erYV5mWpZOm1dlmOgACn
jlBHiHXe1YZxgrRyFnkA75wxVlW1zhUutc09IiSdjuN2F2NKOaOCWooBUdFJWBoaBFmlvTMo
wlYT5xA4MAgcpxgj6yyKKHHYescTbogzzOMkDnzgy0wXuQ9RzZ1m+YjmCnvAYElhPfFGWVcj
zjVS37QpGxoaHgX0aGeqkEUI2hKnKultEcVR0u5hj+tS1VWJEKYIIkF1hUylvawF8pk3DlsI
mPFYeVNr7TwmLCAMcUIpCSVzOMQQccGwOAaeWDS0alTVE41oyBBFSqraqEp5BICwRUR5rwBc
IGIcEog8+AJqQzFCGDnwynJMtdJlWeZKemeMUaqqsNZWGGLCdtxtdTpBGCmtEBOOgTbKC4ha
LR/iSstSaUc9w9Rpa0trJ5WOQwiZ9rYuFQ5xSOhgMKHYG6I1OCICTwXlTnCHQrAxqUWNjpKQ
ECKRLLTMx4BLIJgEzBH9TZuyoaHhUUAL6caZpAJ1Om2LrDK1V9gR7BHWCFlCMCbIAxCKCAOK
lIOslsYDjiPKYwaB4ZVjlXNOecMVEMETFkvnamucpSwNuqfa3acFm/ivP7txY2FMiQ1YgMHq
WnrnEQEjbTaYKIkdRpSFQRyEGBlvSwUuZKwdC0EZDzGmhCDr8iyfGCWRdci5BNMUBCMR5WnI
UymllHUQhxgjDNgjb2pprcfgEkJRQJx2hZPW1hhjShlQLFVdm7oVhGkcWiWxtIAtIt56pTCj
EQtp7JC21PjEd7uto6e7iQqX5levzq1aYzHnCEnjvmlLNjQ0PBLoRNnlfoacQ150uillocNO
GaPBWsxpxBFCVVmW2lpEwiTwjI+KqvauQoggHYoAkjiOY/AoG45QWfPSxxSxEnliXYfEQRr1
6MzMVMeltbbLw1zWChwlgCjGjAnnVF0owBkYjpKQEHDgNcXSesNZ2GolvS4mXltbV3U+HtVV
4TwgwghY4gxRzsgqDlNBuBnX42JY2hKwT0QkGOUBlbqus5oYSxjFGMq6LqsSWZOmLYxQXhTS
lk5VDrwI2ZEkcF7XHhyHDFsLCHiAvVNVCc7GbTGTJudax6dsKxyr/herIwmUKKd8bZrwvKHh
sYD2r1+bLI+gtHSU+9le2m7xmDOBCy2d1xiBMcblOSsKZlxIKJSVchUhiDiPyhIiSZhAhFLC
4hYLCZxGwREeuiAYQl3GMUpSG3pPkTRGEwcUvDFKl5wTRgLMhKoMKIsMClkchDH22GurvS9l
jbALqRAksLaWpcmziamlt55pMLXCykYeulSkJAw8qybl6ngwzMeGWs2MIWkY8YAmicGscFBr
TJ33yKkqqA2mrB0lRa3ycSFC1OY8UAqNxh0RMkoNYYSDchVgzzj1SttapSw42uskztO6pjUK
iyoqIc8BWeBtJ039TZuyoaHhUUD1V/1n2uzYdK/FWEqDFHFOOGK4QF5jjMAb6V0SkSSmmGKM
tSk8FWk7QYQCYkD4WMvlbFzUMkTkRCf9H1Ozfz51jFq9VI3m/fiqnKxpUWVFPhqNhyPwjlFC
NBFABWESFLIgGI3DpJ10o0AYpQpVj6pS5QUNRMfQpAKpwVisLUUGBUgkPIxou0uDKREFHtVl
3S/d8mB9fTA03kIbSl3oiQpqKow9HqSno5npdhxQVhm1IvPVVpk5V5S2tBXCthO0nunGrVIh
pWIqAsItJgTkep5ZRhhB0hhi/BOt6WefPI1Hk+kx7Ukkom7wtB9LlHlYk/l6nn/TpmxoaHgU
UDJ0//P/PvU//+v/FRnqKkkxVqALWywNVxBh3XZCnPfaCMqiIAKELK6jDu112976qpZS2+uT
0Wdqci2bYAdPxcFf9KK/mk4jD8sF4P5weXlZlkGZdkwOtjTYeYIRwxQ5cNZp7yxCjLNWlLQI
B2mQNsI7J2tQspu0ngza2NBhWWtKtKdgcIigh+iptPNXJ5955ugJleW/n/siGy7k48woy2dZ
9GTiE1JUuatqQ8IO6z7beeLpdMYrszTpgzMGnHV6lOXIOeJBIPy9maPfTzqsUlgZX9jCujpf
myvrioC3Vta1IDT2Aq2rjhGzhp9gaXrqyP84jiQVC5PqX/7j96CWvmlTNjQ0PApoEAGhJOJ8
Ok48lWBsrnxdK7k+CALaTZKIcisdlEY4zbigcZeD0Ct6OBxnWaa9z4w0E0A6ZGE4VuQ/ri5P
lsseC0uvFsb59eXsOpvYaRfxFracAHEetHeAHGJYRlxhTRMBaVh6rVXNKXYIuMDYi16StHho
rANpqqqWqkYeTCWzMh+ObMm7JOrx0vDKcqWJtohD64koOTWVgaRjFZWESu3XMwqqHg+Xl1b/
uLJ4TWZ9oXMOY6NtRKy1+XBU8mHK2kdFAk4ZrEbGpLXpAMaYqryoJgVjbLQ2/qIYnQ7S2ESz
DE4kx9K4NwE88v3AEiKbr1EbGh4LaOsk+t3v51avrX7/6JOp59hZA2aQDyfFYKabZMGIUEE1
WOPHwxqJ0GEzmPS/vrG0Nli3AESIWqAhtXXS7kwdnx+N/+3yfMugJ9IpRPGSnsxnpi8cygZJ
bK30nEfG+9o7CAiOqQbsNbhY6G5Y1BlmPurGKs9jYI4RADMaDRBmWumiLhRocEbXpZ7UxTDL
vlq+9p+fpySUxiQMjiVCUcPaWEUmVxWJcAcxfL0aD5b643BS4/mvr/1hbbDivZoCPMN1RDxC
iKEyy68sfXbk6oqbPhIYzyxV3tf99alWJDrtr/rrulAV9+Nh4Sq+psasGIesas/Qgsuv+4Pf
r66tLa6O18bftCkbGhoeBVQc6yzND9f/sP7l3KhLSUwxpjCplKAOGzSDkyThrbAlorDUdmL8
teXs9/OLq1kxVhVQTLhTFDJqbcfLlp+UaDHTsq9SW1GPnHAT7KoAUC1lp4jDOG11wKHBcJLL
UipwoQDOIOGWgrSSemux4xwHSGRqkhUjbRwXsUSAGHVGlrJCVk6lFJxeXDEyX30iEb00PTl9
LO3GSF5fdcqCTNtBiFUrMx3lO5rQ0lpDHBUuIFZrZQCUxe1AUw8Mh1EQ1RpXyo5y77DzVGqd
rxeVQX46Fq2IzyisvAemazIa11NWBHFLkNhIW5fKKW8qY6tmpNnQ8FhAUfeob9vB4mSwbgJt
Ygycg0fQCaEXIJuGQdiLSY9CsDRc//erV/7zy5Wv1osMfE4dCBRyzQOGA4Yt146QpMOPHy98
f3UtIxVEDjkKKCSeMsRDHISMMmQ9Z4QaTQOKEqGs5owKcGVe1aryoWjHAaYoQ3pY5S7LMRW0
08UBK412Aosk6Uz3hDSr/3t5ZdX0Qj5z7Nizx550WMpJ/aleKsrJTGd6SkezRB8J417NqUQr
dT20JuOgCBBGPKYae+UNxixpp8cIO2JEO0wTFqhS2XpQZW6gFD6aR09O8zgStWOelcPSVnal
KhbsGow1cbjCJIhihDDyzdYnDQ2PBbQSoTg63a6YERoyV1Q6dyqgIJyXmuuKFwOLi8IgeWVx
7d8+v/bFelEQyAWqGQDxEwxdRtohY5xao0gQJFNtAl4zwjIdOTKWuYoi1EvCdgzO6LoOPe5y
2uLtCqHBaGSVlsN86NHk2gAjn3b1sXanP1xmunLa1RMDXrJQ9DqzHscKUNji8VNHhLVsPCn1
RMake2TmWNIWplztTF8er1/rlyLIT7CpP0t7T7qIgL9yfbiwtvplma1gbQJoBwFNQ9kSjtc0
opGIEssjyZkIiAhBg/ROal9oT4oqIriVtNI2VYUfrBfOw3pe/H61XDQ3Ys6DbteEqUMI4cZp
NjQ8FtAxzfg06+Iem3G4QMWwGA76qtYWAAQyXvUHKyuVUZjeqIqcWJcCbfGoE6AQLNYM40Bw
QTERCrnMVWVo9Ewr6qWdjqNUui+Xl64FXomQI6LrPDTume50q9NzBq4Phqi/Oh7ocgQEwdNR
cvbpo//l+KkopVfGg9np2dXIX/aTfr8ikyrsWMHFRFchZ0CxxS6ZSWFYZnk1mIxrHLaQeypt
nTPdan0x9pOnnjz6X7rHpmk4zIrP5fJKNlo3RgZAAhakkUhi0Y1IEBDmsARd1lqjitra18rZ
woNFXlBEGacIe2WMtxhYxBkVjHFsKlt7h73Pxv3JYKw8ca4JzxsaHgtoyQYWc9rhOBC0Zpgr
61E99C0ErOWjLqYTPc6GQ2nWnCkjq0PkejQ8loQtDtjHDLUZ49Yhr6lw1WRMKn26e+wvZp94
Mu3pSsYhDPpLxjnqDNbyKA+fP3bsybhd9CfLQI/V+vfXFtfX4S9OdF564b/917/6i6gTrOc3
Thw7xTrJ0LJL0Zef1vODceWW+7gdhAHEjHqjjTPdXjeYNZP+6h/mr5w6otNOp5cGf9U5asZj
IuE51n22fcRKs2ZGmaxyb0wA0MK8k4StmHCijOaYEIxtVVWZNIRbQhVABT4DcMQnEQnjFBOR
5ZPhOO/EvZkkcnnZ68WzHZwaDxavjOtikCvEtWy+o2xoeCygkpfIWmXBaVtJU6i8EJq0gSAI
e6I3245DWclyab3fr+slaScJas8IMRPxVoCxbQWsl0QceasqmWVymKXSnSToe1Ewm4YrTjpf
GJCIhUwgR5zwuo3dcc6UYO1Wi5VKpYMzjPw/f/ncf3/m2cCywcKq0YMjIZ5NOoa3+AT0av35
ymqlLMaYxlEQCoKBINzutNNjuPh6cvXa6AvgUxT1tHiqPeWnjjOAZ7tHZ+LkuhusFKMVWUwA
tAAUs6AT8XZkic4n40DEMQ8CRIQnIQ9FnEjvsmo4klIbhy0mCojyUNSqP/SGBpgzDE8fmT4R
Rn6Sj4ZZKCEKUVlqb5uRZkPDYwH13GmtjeCq0KNyopTCCYtaPAYctCORiBBwmMa0rhxohS3q
Bq0nZ9ozPcGQVgWniIbEC+SlQkDCgBwz/FQQTjlvxqOV4crV0crQVpZEBBMEsqyL0dqNiVRc
OqRdKvBfPvtMO+we7xybu3L58tUvJTWnn0qfPkKioiQmepom650j66vjPwxHWphwqldqhTQE
BBtvwjhoHessrVVflOOZOj6j2m0Tnox6Kacp0HE+uZH1L2drCzIfIzAMgGMcMZ8yawwpbJfw
I3G7pXhXFMSTopTLslhcW1+fTIx2SIHNKlaoWOMAiVRprqsnovi5E8dnKFvXdT3WiUDthOXa
CWK/aVM2NDQ8CihhWFYy18gjpKiCFg3bqTCGac/jIEjjbsAciCwKv1pBaGXVU8RClrbimJGq
9AgZEEQLrzyQkEVBMOuCHhZqmK2VxVfFylKWywCUKV3tI6RpSDDzUhU615UFxsLpI5268v//
/B+vXb1+9cYQYshsJ/StCNEw0BwlJ1pTUyJW/dUs09jE2bggJBVBkOd1wFq9U0fH43Jpcbxg
ymmjyNiFzgcIrS6tTQo7tzq+PJksaasYIhEhESMRMaGv8to7HSLUCeJeEqQBNrkbrPcvry2t
9ifOmV4iUBpgFiY8BMooi6bCiBv7VLvzdLerBn1flZ0oTDppMDbjcj1kzYYdDQ2PBTTEgUNO
e+05wTMMHKUC0RIoQtZZ4yCIWlMz0RHMpqsyXB7KXNOqosjgWIRxCyNnsC2c9IBRZbHGpkZZ
prTzq1U5rrSuMLKOhdpFHnMWhKkhvPI44AIcNoj2h+MvLi9evbqaV74GcBLo4qgV4NoO4sAY
FE4MQ54wRLy2siyNU/mqFiKMLbWtkE8lwam0yCZ9bVbGY5mZFnImSpSrrg3rP66OFyayRBiY
EwFNe1HYwpJUBZS1KUtVAcaA0FpWDa8Nx17mrm4h2mknM0e6rSNdMpPQmRQhzy20gx7VDFdV
trg+WF6xte+0pyBs5arwZq1ZO29oeEygAQRI4NoryxwRzCtndU4FFV4UeX7j+nKQeOy5k5CS
qAfCm7plPQVbEY0CShEdjgdZXcSIuVzj0o0nctWUsUhyR7RjVCOqHE2w5QgzJg1dXJ8gEs+G
qfRoeZxdXVidu7a+XHrJwDMgBK1YPDewEzMRrMomdanImtHOADKg8kJQnK2PGMnb3SOWa9XV
aJbzmVhOUH84Vr4uMdS1lz6cW5z8/sZwTRrghAsIA9LuCJrSoS0kURJMaZTxUCjTHw5hbUBC
2pvtPJkm0xHv9uLWTAulXGprwTMiUInrsVtcWFv+es6q8ex0xynan4yXlobFpNJNtouGhscD
6rQHB5hii8FjC96B9xg80r7Mq4X+oiVFyOIKEVMp7nzEiSDEg6+sqcrKSVkUhbGaIMc99h4b
42ptjK36sipU5Z0TFAOitbYVshUxlfE5VKSCUurVrByNi0p7I8C1CIQhFcJ7f2M4GfcnnIKS
Xhk0clAwoB2GHVCg3lujoK6tZ9rXigmRdLsdT5j3oLU2PivUsJLLK6PBsJTUI85anTTshJhR
zDF1zDiDEGKEYYe0MSwKjj59ImnHhPie4AG48TBbGQ1r5GvnjUUYcWfDKrfj/no1Wk0jaCUp
yqrV/nh5cV0qq5rovKHh8YBKrZ33HmOEAIH3BGGGGeZYMuNMXhZryiA3yi1aUmVRVZYzZa3R
qrYok4XKC2cNo8R758E7561z2hhtXV7mla2VdRX23tnKOlvrmmDvRcB4h0Uppw6Hk5GkpCQc
sek0PDrb7U61qnr8//2nrnycQJgEuaajWhPqCE/SJO7MtuqwUHk9NqReHWNZtGJ2Omo9TUJR
VK7EILXRiAJ+stfWhM4PxyV2cRyF3Z4TxFiw1iODQhCxD4QjjOCoFZ6YafemO8Zol5fr/dHC
ysq1QT5UXhoEhiFHOcNhAAEyrYAn3ThMUi54HJlWq8blGDXxeUPD4wGtlUIEA0beg/MOASKU
MRIQhhHxVFAMXJc2L8pxnVXSKe2yumZSWy+s9VoZRjHnAhvvjUHgNpwvwkgIxpT2HtXGIe+t
97XSY2kzp1An7nZSEkQKyNeEIAAiCG+HvBPaGDsH3W4yk7gnj3REnC7nvl4fjWTmgPZasyee
fKKW9fLS2srSYH21H4xh5vjMmSemT7ugsP1JZZXzxjgC9Egv5u3YR3jRSUMxE4ETfKyyspLE
Uay8nihVSO68N6oyozLwnpDxeHJtcfUP18dX+m5iwAIQ4xkYimwc+ZkWpl1iwBljCCGM0TSN
o0nF62ao2dDwWECNc5ggDMg556wjgAhjHAWMAWFABU9EiyaMJLIYM6HWK+el8wDII+wBEAAj
VFBKtPZGM4RCzuMwCsMkJMgUo2A88dKCRwiw1a4qrPIWt3ESxjxJ+1JhioEAj2nSS3E7KE0d
+vp7p46douTk0SMWcb8yWZKKurq2jhOaBBENAppVhoyth8iS01H3VNJuZTKvlVbe44gIwjAI
QdIwTWfTz0eDy1LKsg66sTNE15YC88qUWV2WNbNW59mNSTUaDT1h+VBeX55cH7s1By5kwDgz
hBpwSFZMjZ0ZKzPrQowxp4xTYJRzzilvnGZDw2MBxYQRghECay14hBHmhFHMMAXKEBMionEi
WsL4nJNoko1Q7TCxCFtjtTYYgCNCAWMHznnOWBxF7Va725kyQaDGIr5xDVfKaQAKCOE4JB3e
bnd6YZwA4wZ8rW2pwQLQkNJUWOlDR09Mt/6s1TrabvfHdTwuAoEpdgAA1hpZG04Qw1Ecum5r
1tiehvLacrY0WloeDyUQEUVh2A5JwnCail6r41O+/OVXN1bWcSvhoaCeOesAUetBOae95eAI
JmDQZFQUYys1swJ4R0A7oSLgteNlzRlQgZiWWFARR61WKxKRgUKUChME6Ju2ZENDwyOBchZQ
Ct57j6wFzBAhQBCA90AYj3EsUOA8MsZ6571FBpwyFrSti1pJHWMWMMExI9g6ygOKAi6iMIzC
QFJKEKZAqAUnwVLPgbXjtJtM8yjMlRwOh1dvfL20ng0zgI6NVBkaCaARMoyZbovFAkZQB8yH
AjDSBGHOvQdFCG9FDDpxp1Tpyur4ypW53Kh1tZb5gUWKIs75k9Pxn58+koBgtj4WB0c5W1pb
k2t9fqTLkSi0AoQluMqrGLlOEjzZTlo8vX59DdVV5XwibN0OYSrGBOFJgUFZZDxHghERCsYD
a7yy0hnHMaW42a6joeFxgQrGKcHOWg/GeU888dZrbbQCQIIwbg2qiyqvZF1Jq512TtaKGKtr
hxxqRXEriAEMpoyKmJcKI4QByrJYL6uV9aVyXFMFRCOvMKYEA5HarAxGo/XxjUH/Sn91pBRL
gaeEU+qVMnUuiyzzdZFSXE2y0RgbxFztdU0RCUOEmQkZhhBCjmjMA06D0YQbigLqpc4q3y+8
K2tFbScPxASiCkgcP51EgwlfGfYdx0KQzCHtjazqpOjHkSAhdMLgSNxCtREm8FQuynIovOVa
Iw0wJKaSxnuNaAgWIWNslhXKIYepoFQwysg3bcmGhoZHAuWIMUIdGAfKgcMeee201rL2UuLS
Sy49SEQIj4KEU+Yq8AYIUAIooLgbt2LOqjInHpIoImNrndVSjif5/I0bV9bX+ytKaogjyzjB
mDjrKqXHGojz/TIfKwUBhCGQwButfFkga7C3ta4nZVZUsr82caRFjOLei0QgZpTOIxYIWTNZ
HRHBiVMnusePq5FfWMkXzFqlShkiHNIsQFcG684WJ1tRLOXpVkfp6X9dWhyvD/iRGUFDz53M
+oXMocUod8iXzIaziWBdUlovTG5dXWujUeV1iZXXChz2QwVDW+ekVI4yTBHHFCOOKW2Gmg0N
jweUAxmtDa1WnDHqiJFSOa89gGfr62PVH53g7dn2LAsFoRoBNgqM9NgibIECSUXcScK1us7G
/XJYtMfmSdQx3le1KsoqzysjwRmQeWUClkRRIELOQkEEWIMZj5IEsLdaGka1VsTQKODCC2lU
P8unqCBUYOUjzFLBLOdAARFfjUZBpp+g4fei3ven2ikmV76czA+vDV1/gqFknqRQtcSqqUVV
Pj07e6zdOnJkdqrdGpT5v08mOk4JpRgxZHxd5Gg2TWMstGyH1iNCNRpKCgNdjGolAh8Q7zAC
iwPAAM4D5yxNO920i5WplHXeO60FabxmQ8NjASUOy6xSVY3i2Htw2hImBAmM0aO14XDJstQg
wz0u1uoqK5SuoS6UzGuEedSKIxbqUo76o+H6oF4ZIBeizjRQzgLf6c10jW3V4yJ32hitPaDY
Y2ys0+Ct1EBYuzftVTkYS4QpZYwyFoWhl3JlOIyl756YSjots6458gxTY8EC9ojoSXEM4v92
/Jk/i2dmI7K+3l8aVV+NysVK5QxUSiFhRcqpIbm1lodPHD1+otc5htHYmPXP5/6YlYYHRiov
fTmSppJpGPUiPtWOoEag1ZRyBKAuJXjBwo5LCYUiCqAuVezIM987/f1T56Ydk+NMrw2KyUhK
RXGzEtTQ8FhAwcB0e1rxWta1UjpgIhFJTONKrnqM2r0EoWBpfb2oTIZQKTVYUJmpJ1Wrm3ST
djHJv75y+cbKDU48AfCMaYyVA2AibndTpcU4B+W8IBAQBWZS5auld4hj40gYpK3WcK2qK4NT
G3COGbUYao+XBrUoyXRL9XjbE6FkKSUyntbS4VLFnh9NZ0/3nprFcTFa+f3c1//vH7/+YjXr
WycTJI6ltJdY7yvlJnn95eraszNTR8Mwabf+4pmz86X++qtrw8K4wkCJrPN6BOxIHIpUG8YI
J4x4YgjHiAIQxJKARCKcaiXIrl9bRgpPHT3aO37cr+fcESbtZHlZWWd2bqe58OHPfvLe3M1/
z73+8UevnDyUWe6/BACAT84//9aFjT9ffvvS+Rf3O/pg2VHHQWvZbPKDEumba3vDnzoUGZyI
xIGoHLXUEYLBYa0tpkFvdvY46qaSjZaHWpcWISIIUmBKg7Vth4ku6t/PX1lYuA7Ct7sCAdQe
hlKGeoI8K62tHVQOFAOXEpwKiLnVqDJV6XxCWMADxAKCqPdgnDcA1mqrfEK4QsnSyMbXJ8dS
hqWYlFBr6pwoamCKBKQlUCvrqz+sDL6+9sW/zn32b1+NFp2QIcEtynpReKTljEelK5X74/La
LGWQ5bMzU3W7O8k0P7EAACAASURBVNvqTkeTUZXXI2s1hVy7AatFMOK0UBVnfjSsFgdFLo0H
AOJRTKI4iLzjRS3WuVPq+tr6Z+4KyWSEWaXqCqHS2Mpubg23y11sMPfeT57/zcHd3v2XsAfz
VxfgxZMAAAtX5++1kINxi8MHuPDW8/BNeqhH2PaGxwEashADMhglISGMSlkXkxKDjyjDgdAK
58ZUAJYxC8hg7AGIQwmPAhbe+Prrhf+4bkLbOdY2yPAgKLVfHA6zcoQcs5gOlNQYo4Swbgjd
iMQcCqtJVSpFAcuqMOBqY4Egi1CldVXUgWEhaeOwt7q2XEwWrwfDlHb7tSotop46R52hRuPl
lcm/rv1hMHf9+trCl5PJYgVl4lknYdOYRpRELBKhnxhTlMsafvvvN4aLa92Yu1bLzBxrsTCy
1aSwYLHG4GuxeF2qOiuLnBBqjR8anVsDAZAQ8RiJLuXeIo96M20g5f+eu/KF+mo25G2RYEyc
oAXAZjLKT85v+LvtYcymC5l77x8+/OuDOL37L+E2zp07Nzd39SuAkwAAX12d2zgyd5fb7pGF
f/nNHGwPjje6gAvvf/h3L967y793Hm3bGx4LKCUcOW+9AQQYUa2hVkZwobxfz/Phct/3rSkt
EKwoyZT2AFHAW2GrzMob1xeNtbMnZoOpaGV1iUXJeFDPr+R4zWODMKM2xFawIGZ4KjUdYbGr
pbZW55leN0qjzIREEysBeUDe2NpIrXUiYgR4pPziulywMuaVC6BsicADyauyGi+vF3OrDl/T
xfIklybHUCVguWURbs10UEoRgSBkSjkbcZjqfLGwNskMViXw0ZFzlW51IK9orZFHrJUw1vny
xuiPiysTWSpAgmKakAnTOAQeeMZqzL21Rvuy0wqjMJn/ty/XvtIzXRB4BI7OHotz5GoMAAsf
vn8BAM69/vH2sOrkK//0+m9+8pu/2TVM3DUY2xl7338Je3D27Nm5ua3h1ie/vQDw8tmztziO
XcPbe5wN2JMXf/gyXLgA247rwHXtbOLuq3aPxfeNs++j7ZuHX37749Pvbwmyjxy7T20L//Lb
l374282SbukHb71vo7xzr79+9r33LjxYMzQ8QPB4mGvtCGXOo6Koq1o5TIAyjXBu7GqeXRtM
rg+KxWG2kk0KowBBKEJO+er1pao/io8lraNdDWA4MYJWBI2sGWi7XpulSb0yKQtrgVOecCRQ
hXSOTIbNmjZfZ/KrUb0wKlaLukZgMQZEkMflKFtdWc1ljeLAJ2SC4YaSK1aWzGnsvfP5JLu2
tPL59ZXPh4MF7weC1RGzIbZeG1eHadjttjkBq0ptCstQcKyHT8Y3BPlCwdUKvri+vvDlohqN
qdXUmSRKaZguF+oP6/l8Zecr/UUmrxTlBCzhwIljUFOoGbVKl3lV4DCIj3ShhdYMXC/1wqj6
Ynl9Aq6k22Osv/nrXU/6yVc+urTz4f/k/PO7wte5937y/PlPAOBBlLAnP/zhyzD3m39ZgM0A
9eUf/nDX+VsnBObe+8l+xe3PydNnt2R6/vwnAC+ev3Tp0qW9ZhX3q+uWJs6995OffbgAAAAL
H/5s9+zFhbe2Tu3Ffbf9wls7BNlxdp8bd/VoF966Zbblzk3bOvDeBYDbn4KGbwt4eXltnOUe
CEJUamOcx5RrBJ7xuNttH5uNZyIcY0Wg8t4hAATe47Ks+stroBxthZXSo7pk3Y4NGeu12sc7
009FUyeSoMdrhMZlVWuJkPPYSa8V86gjyAxFPWJ7WEao5gRFERWB4EHEQpfL0dJ6Yep4Njn2
TDp1mgVPUJgmvkUh8GkSdttpZ6YXH0/ZqR49PcWfbIujIY4xEACkndecEYZAZmNVji0Ylwbt
75/Cp4+QZ1J+sj22bH2l8HXNiSfMgTCZKydMZSmWRwN/nMke5BHYyPOQRJxHnicQt1gXO7a8
ur42GXefmpn5fs+1sOsQMs1yICWlmtMtlZ49vd+zviP8vnTp0qVLH79+DgAuvLXzTb3/Em7h
1OlzG2O9jQD19KndBf72wo7i3n4ZAODCb+/Za754fqMIALjw1vPPP//8Tod+sLo2B9xbl116
+2WAuff++RO42a9sntpo+0bT7sB9t/3c6x/vOLl1dp8bP/nnm2PMbRkP0LSbbJxsRpnfVqhU
ejAYIQthEHhAHjBgYgBxQpJeImiHSibliq41JswShIzUWq4srRV1hUPqAWVVZQSIMLTOhZ24
Y1tB5LCidJBnq6uyqr11jBBCEUhrGSa9hPGA9ozXzihDMYvSDg1YGIQR4hWJ8jKvSA6doNVO
oulAaD/RxoYMx4CR4gySqQREUpTCSWYnhcoG2rqwQ+OuKLIx6XsWUpA1qBqRqCaAZ2cZ7QXH
EBqVw/l5Aop4TwIwzOd2dGPydR3W/gkKUwI4QRK4s4KhMOItGoUmElXCrPAFmww1w6M/e+74
DGFrdY0QiUSkqxojTOBgidU2lyJefntr5HXylddefu+tC3Dht5+cf/HUfZewd5x68vTZjfWQ
U1fnN3zy1R1nXzx/6dJ52GMF51558fylS3+3o7ALbz0/vxlqHqyur67Obd63c4y20cDTZwHm
bp56+e1Ll/ZfYbrftr/82qbz2pxouOuNmwY69/rfbQi2ZZ+7N22rxh82q/rfamiv1RsM+utl
f2qmBxg5BwgjSjimlJM4tAiRdeMAMyFaiWaYWFmbarK4iKZE0OsQhrGgELIqr5DBjiBVgS40
A+QpBcxA1QKFcZBoapH3GowSoSDMBxZbj6WmjIedlveOBTzlXLa71UqphnIUjEU3irpxypkx
ymDPONK2dkZqinzMRBghF+REq0IBcbMnZpJOa2FxuSjy4yePEIuwR5Rxi4jDJGy3gkSMs2vW
QxILTxQKALiXUtXrqyIIaMhcj/gQC0tFqZj01FCUE22cHtXWVqOFHIZQgS6VpVFE08AY62Me
JIwBJujm7zS3F2v3YPOd2cmp0+cA5jbe6/su4Q63nTp9Di5c/QpOX52Dc39z6paze6zX3zcn
X/no0ivbZc+998+fvHL+xYPVte8i94t/9/q5CzuD3wt3mdZ8SG2/442bBtoRLmzaB+BA6/e3
DoYbvm3gk1NPzsRTrlTj1aHKa13WdVZSYN3WDHZsbWmwvNRHmHamp9NuN57q8F5kQmSIZ5TH
UYSpN7r0VYlHVbgK2R+Hi5+vXruyfuXyjes3VlSmQYNVOBvWOpPEYmNtBbrmCJKQdRPaEdDC
dIYGM4IluNVKAx56472E8WKxvjaW2vN2HHXD7nQSRNgRYElM44AnlHFZV4uVXoKoFtNB1O3E
SScSkczk2vXBZFjXNaorZyodWDuFwawtTRavI+9RGPpWB1odIyKHqQMma80YjgLKKDJWY8yE
i4sb5sb/Xl/8bHDt0vWFT6/kVyeQgynt6vX+qD+OotBTp7F0gfcR2BDg5F//zTmArRm0bT45
vx2hnjp97lYT7HjJ7r+EO7A53Prwtxduv+yWcH8j0rznN/eT87sj8hfPb4an81cXDlnXVlx8
kw3HePKVj3ZEywAAcOH9fWY1H0rb97lx00DzV2+KdHs/d4emNXwnwLbQCQ4iInxtdFEbqU2l
dKmqrFpb6l//emmwMgZLuAgIZ0EsaBridkiTEFnkpJam1roU1rYNZWtaXs2yG2U2kON+ng0L
VzlkaV278bA0uaIWAwKNjMROM+w5BgaGaiWUj6xj1iGnpXXKE8qcRMN1eePaoD8as5A8cWLm
6NEpHgUijRGnytVaDo1cx7yk05R0AulRrWws4jhIdOWKzGS5q2tPHHbjiVq6Prz8x3p5EXsr
2p0jz5x74vt/GU8f9yIOom7Ao4ALrJUejUFbRgQzAc6EWkPlipl8PSmvZ7giBAvwrBpKmcs4
DoOY0xBQ4FGEIULbXnPnWsLmusXWasXmMsn2DOTmHNdmTHb/JdyBF3/48uYKw62X3RJNbszV
3Tsv/nBjeu+mG9uchISzp08etK6bWtic6tvpiDf//tmHp85futV37iPSA277fjfeXAnblH7L
PndtWsN3BLowf5mHQRxE0qlclh4jwkU+ySajgVmr9TAHDarWk0nmcUijiAZBRFp1NtG1zjNl
vQzbop2kXuGV4arNPXYUY++IB+SpZdpbq50znnnMEPIYA0bGOWMldsR5q70uZAEsoA4meZZn
BWiEEMEUrPaD9YmdJd0TrV6vXebEWosJUUYPhkMmbZqE3bidSYlBYEKVVJTQ2elZ632/yAvr
wiAOeThaXcPjCinFCTJGGetEEsfdlsjHcaqnul0rs1KPjKw77SSNO25kvfa9uBczMpJSuZJw
EqctS6nDFiPCGRcRdcIBRUCRYMGGNk++8tHbV59/68KtU1YA517/p42psRfPv/3yhduuuDlD
ef8l3IGtEPG2YdTJjUnCufd+8vx720d3/kTocGyGz7eUt+leDljXzWnaHU3cLGHP4u+y0Pzg
277vjVsTCFvSnzt3Mzrft2kN3xHwYHm5riouBCGkrqUzlhJSVeWkPyiHQ11XAGCMKcuyriql
NUYoikLOuHNWVqXVhhASiIAyWpallNJ7hzHGGG0AGHlrrLUOPCCEMUaAnHPWWGstIOQ9KKWd
cwBQVlVVVqC9yWvnHBeYRTgMA86Eda4oS4wwpcxZVxeFUSqMol5vKo6TdqeTpimlJAqDmZmZ
mdnZNE3jOErimFJqjaaEdKenWt221XptZeXG9es3rl0r8rzb7R47fiwIgsl4jBB69i///KnT
pxAgY20URWkrAQTeGoxxEAZRGCGMrXUY40AEaZrEcRQEQRAGQbjpN188f+t6KZx7/eOdS6Ev
nt99wbnXP94Vnd1/CXuxOf7ZI4a/GUADALz89ubYbUd0eUhOvvLRLeO/7R8cHriuF8/vKmPH
TxZPvvLRbu3c/eeMD6Ht+964S8SX3/7otbMHa1rDdwPk/cGWfRsaGu6JzQX2xjv+qUDvfklD
Q8Mh2Htdvfmt+p8MzS6QDQ0Pllvib4AmBv/TognPGxoaGg5BM9JsaGhoOASN02xoaGg4BI3T
bGhoaDgEGBY+/NltHyXsdexOPIpvGjbr2IufnT//s/upf6upW8Xt8z3evbFD9p27/ex59KHw
YKr65Py93r2p4YfTzAdvvrs/z/upovnC5zEAw8lX/unWvcU297a620cm33k+Of/887dsUTP3
3k8eguPcZPtH048u78LWR9IA9/5B+cKHP3vwO3rcP4/afADfWlU0PEowbHzZBXDze+HN1+zb
9G3X5ka225sTbm948NH58x/d244Ht+5LebPwuff+4cG/d+fOndux7+Nm3oUHXcmda3/9429s
h8bNDTYedAf8aM3X0HCTjTnNzX1j5977509uJlz4p5tv2G0hxx1ikO04cHdnvyu4vmUcsCu8
uqcxwq7o76ZkO8rdu9Q7JZY4d3tKg70EPHhFm5w9e3Z7qPnJby9sHtnFPdW1j3W2h0Vz7/3k
5j37mWPnuc1T2xtGXnhro9j9n4eNf3724YdbO2t8cjgDbZ/Yq6rDmu8BPGC7RLtFFbfU0ETl
jwVbC0GbbnNzb/+dLvOA7NrV/2BZAQ6dtuEQwuydomCbgyWWuOdcCHvw0PIuHIr9kzTsOndb
HobDsCNnw6k9Tt+pLfsnitjmIeUFOTy79yC+8FYTuj8GbK+eb2cpuAeXCdsR82aQdICsAIdP
23BoYXanKLiduySWuNdcCHvykPIu7MP21hnnXv/40qWPXjl59yQNuyqZ+82/LGzvPvHy24eJ
svfP2XCHHBL7JYrYgweeF2R/blfFYQVu+FNg22lub/t3a8qSA3EzK8DWljIbbMxGnn9xI4bZ
2Q/vmbYB4L7S09wmzObujvfInYW/l4pu5l1Y2Mq78PDquhN3N8fNSs7v5/Puzv45G/Zuy16J
Iu6xeni4D9gdqrg/gRu+I2w5zYUP/2EjLn99I0p/YEHM5qzU7QlY7pS24X52JTs0d6nqjsLf
E1tDzb0Gmg+6rjtxGHPcD/e0UH+HRBF3Zl/zPYIH7PYq7iJww58Cm05zI8w49/o/vfLK320E
MfslEDg4d88KsJO7p214YBwkscSDzQPxKHNO3InDmeORc5BEERs8tLwgD0vghj8hMMDNd2kj
ZtoMMXb8cGPz0diKanZt378/B8gKcMi0DQ+MuyeWeMB5IOAh5Zw4hHUOYo6tznJzTXjnUtCW
b7j35+Eu7Jso4pZLH1ZekIOxtdvwwQVu+NOB7vG7zM18CnPv/cOHf/3RKye39va/PQPDXdk/
K8A9pW14YNw9scQDzgMBDyfnxCGsc5AkDbtPbs497rjv3Osff3TPz8Pd2DdRxC1tuc+8IDt3
BoY9dgm+vXEbp29RxS0CNzwO4JsR22s75vxf3IjRt0abu/b2h5ffPkAyq62C9k0ncC9pGx4k
d0ks8aDzQDyknBOHsM7dkjTcmodhyxqbz8M91HhI9k8UcQsPJy/IXbhFFbuTbzxIVTR8a2n2
02z49vIIE0V8cv759083OwU3HIAm3UXDt4dvLFHEwofvXzj3Nx83HrPhADRbwzV8e/imEkV8
8s/vwT190dHwONKE5w0NDQ2HoBlpNjQ0NByCxmk2NDQ0HILGaTY0NDQcAgxw8TWEXru4+/Be
x76dXH73B2ibH7x7+QGV+xA1sFvknTxyld9Re5ff/cEDVOYetT6wwg/UhMsXL95DdXsZ6gfv
Xj60/Devf6ha3VnboWz6SKT6U+K7PdK8/O4Pzr753K/9Fr9+7s2z335nf+aN320JPP/OC/DC
O/Nb/77/o0cpxzemvTNv/M7/7o0z91/Q5Xd/gM7+4qfz2+qE7SbcrOXyuz84+/Mv7lHOHfz6
VYAXfvrjw8t9U5IH1vA7sa9C7ipew8H4TjvNy7/6xaev/nqHp/nR+79+FT74edNrHoTvvPYu
vnb2TXhnfsf7fuaN382/88LDacLF11764IV3Pvw2O5dHq5DHl7s5zVuG7rtCnr3CgI0LLm6f
2tHLXXzt5rUX3/3BxpmN69997dZgYp96d/PZF7uO/uh9v9lr3qmEA0r4xW03PgANHJDLt928
W4iLr+0sdnMi4Z7qvaP2NvjVa+j20vYN/265+vK7P9hxycXXbglvD/u07OLiLz+AV//xVie2
PWy6WfjZNz+FT988i37w7uVbJl0u71nwXlx87aVba9tTObu0s1MLt4XnD+T1OZRC7iT2Qczx
sGbBvpPc80jz4ms7Qrv5d1749M3/taXhT9/8OXy4dfyDlzY0f/E19NJnm4HoP37+0pufbhf1
6Ztvwq9vBhOH8DFn3vjHVz998+zhrXgACd/84EFr4IDsDJrn3/nsJfTaRYAzP/7pC5/+4lcb
jbz4yw9eeOGFLX93+YvP4NW//dHh672r9j598xfPzt+upb1bvafYcOaN3/361c1LLr720gev
/vq2OPCwT8tNPX3xGbzw7H7fpgMAwNk3frc1B/K7N8786G9fhQ9+uaWWy7/6xadbqtuXy+/+
/APYNSrfWzm7Jjzm3/nspTs+lw/w9TmUQvYUe/cFe9Z+R2kfRzad5gcvoV28dDeXcfns3++Y
gjvz45++sGPUcrO72z6+qxf80d+/88LOwrYexzNvfHjIWOJH729Z8eyhxnWHlXAPDquBA3Lx
f725HTSfeeMfN1/yMz/+6Quffj4PAHD5i89e+OlPn9v0obtf/MPVezft7VHanVp9B7FhQ5Uf
/Pzdi+/+/IPdbmefWg5qi+e+d9hgeafXPKjPvPzuK29+epvse4m9UwmbndLe3uWBvj47uLtC
7vqEHMLojyubTvPV7eWAm3Pe+3LmzBnYETGc3dn17dHd3dILnvnxT1/Y8/oz33sONn3Dwdkx
X//rVz946QDDzsNKuGeth9PAIdh599lnNx/QMz/+6Qsf/PIiwOVf/QJ++uM3/vbVTz+f33jx
t6+/l3rvrL29Stun1XuKDRtdIbz50pvP7eky78sW9/Dq/ujv39nQ40F95t4u805i7xp+vPTB
nUR8kK/PTu6qkLs+IYc0+mPJfYTnCG1HDPN3HZg9Gn70/vw723Hsw+URa+DM956DD355EeY/
//S5752Bs8++8MEvL17+1S8+vacF3T05iPbupdXzn38KsCMsfiDcsX+9+Nq+3eZW73Mwn3kH
l3lHdvwUYoM7LEs/hIfnXhVyAL6dL/s3xr06zYu//OD/tHf+uokrXQA/Xt1HIOEFTCRv4AWM
cq+UzqSJVle06UykFNCki8SHlG4bU6y0uNuWIkoT0yHtjfALQIIU/AIkvIO/wmN7xp6xPeBs
kt3zU4owjOfPmZnDzJmZY2ImAoBwWIhJNigTn/qCNtDlUur5sswS8pCUgAS02lo9uuGayzg1
YfE0vLXNUyMc+18ft9aZ20kvo9aCYhNbpu+YhY27xdqCrPsTNfCG13b2KjWQ3LCIzpRTmepB
HQr+YJc5fCK2FUjZpf3tyVOaTEfwhmc9ziDxhs1cG6hxSp1mmXTY+LEd/axHenJGvnHZgp0M
euQHKVx11WIppEoYDmuqhGVJoCDGJWVnDzYgwqFtnJpur2eTBZR6UAfbtreeZ2ZLLwNurYXF
DlTmyABjVFxtZvYWqgY/LGBq4A2btbSWS6oc9aSt271ers6UnWUSIZwlDwvwKW/4RBQVyBa8
Rlf/sOTONI1RcEBWURSl9ngVGjuNkWOG+we1cXvlmDm2SGMUHJ5WFEW5/mzRNlPdsqAVGEvq
TmhwFuTLTpCMkR/HCouSl0JWCU1ikqJLWJYECqJ2Zytr0VKSAoFg6MTHq9lPxSguPQHCWvOL
HRxuvCQNcln4KEFGb6FRu7M4nqIotR5YnBoEv4fxNldgExTozFhEq0c3vUWaudOodmeURFsL
5swkW7+Shk9ivVBQIJK8Vlf/sPhvgmMS60/iSkwRVpYp9wBC8RGlF/WWslhZemrvk/62ZBFt
0cmzoQXyERv0g/OrbgQxS5XdzCyrx0V55frj+BDSK7G38PDuxhBOfzm8ioh2tCqKBfIhGvQ3
45epZ2bPzaROykr+CDumeI6A5PFRpMfvLWWlm9njyhURsebsnKRIIB+lQX8n0HM7giCIBB/a
YQeCIMivBpUmgiCIBJ+yfOLmHH4O7lXJOvGhsmOTj65pFThznbrkkKhE/KUgO278uATiEyap
M+GCYgtrI/pCNh3whs18Qe0mbVkpCVuhWKlEj8uGU1+IKs02rGSb8BPkJ/0KfKg2FYaL2iL6
loqdNTAl27podTLJvBWVvUUT2Lil7NArS49TdUzm/ygzOlycMRuDfypFmF2RUyy8YiT3EQTF
pvNltrpE1aQirSw9EqowneBzXg12lraslAqdDZJvFMlwx4wEmKh03FsLNFaW8Oms0rzm1szH
alNRuKgtqAKIc002qkRbl3N8LbF7LqUH8yKne1oiJPooChdmmpQp/+CdONnsg3phPjyVqevs
sOKln3g2alhheaimpz+J0gk/MTV4BWnLSik/fkapxI/LhbPpszKWaaxM4dOhr6gkP3qbFhwg
yXlF5oxNMNfwi7R1werkUcimyXOsmmC7pXrIir1CneF/f3Jrm056crx6dKU8/OTFDy4CshcK
veFZr+78aBcrNnUuL3L5I4qfcGLDOH3kpQMQCKLoJf0EhaUtKyXZViiYnVy4qDpJB8s0IiGL
wovBLCrDdWAQSPkcDofNbp4U3mub8sloC8EA5xdPvq3LqU5ajSd+OdMrEGYOTCLzfnI5P3f8
6TpR/3FOeb8FyZQdE0DX9dTjWasDXnwmvXTJTYf3q5UuNmeSAqQnCOPTZQg/C9Ih/ydKXb60
ZaWUFz+zVOLHZcOTRRR+wZ2VRELOED6bTBLTiWrJ+4IaKrnz1A/fpmJTTEae/IpnJhF+l9PW
RauTQ57S5BjxIEtR+tzuwkSLW5bVvsxwyLY7cKb9whVFOrvs+D5P/HEUKmtxsekRRsQRd+l0
fJ7SjJYx6XR82qD5etKWlVJefBqJRpENp0sD/CHGPioSslD43HrwZJ+Ixuaom6YuGLe/TZtm
hnNzoEI5sQV6sXBby1Qngzylybd0x2NdzqbJ6YicH/WMHw1xyslcBP2+yHwklTst3ITSFBab
Ms045BFhfNFMU5AOfxPo9aRdUEo58XNKJfW4RDi3MZMxuULOCE88mCEGevywo1WkYGk+fJvm
tLWw+fkDPK94hdpaJkEB253T3NbHyeSWetWK2v1hBXai2medezeXtqVuY+4RZFfkKdpc6N2N
3cjbTfiWrs5EWGwAIK+S8H3fn3VDE4owfpa9jJMOwOoRCng3elVpp6QkIpnsdo2yNexbgSYd
ruMfrpAzwgtAqk27UYuFHnmklHbM/LHaNLOtBW2xQ/EKtXUJ5CnN2mfuuqTccoROyWn0zzXm
vdNZL2ZOvGMwfHy7+On3UzGvvw5/A0eGsNgJm37kXV0YP+HwMeocgnRgcmvv4P+hqLQlpSSK
v2sjlhAelnnYbNmmkyiFSMii8GJ4d2OXtezRNW3ZAKbjmLEr2R15p20qht8W2Y8UKl5m+rKK
QgSrDAttBLE2vUz7ZhLGnJ4wGDPWLSmbZtqKybEtJrPjxQ+jFbXNiz6srOisJfVvxsNJ2zm9
okmlk94EErCjtGWllBW/YKl4j5cULqxnVmPxwlN1yTKzpYcNtUsgspRm8bHaVBCeO8J5ESSK
l9nWBaqTQ6FzmozelNoU4kAlxpSY6ma5NRHZejiPy4bntSdPX2enn95S4lZT8EA6mGfQFLKj
tGWlJIpfMFqZjZgI526upAzHIlunqGdndXvKILqKjGsJPUlt4ORPFbaotezjv6ZNM9tCVBZR
8bZv6+00po9ejhAEQaRAhx0IgiASoNJEEASRAJUmgiCIBKg0EQRBJECliSAIIgEqTQRBEAk+
ibxPv4L/adp9XAFXch40Fci++uUNQVFAyYsW0VFIfPLX4WQ68YrmXj5vkmkx6F6S8qn9yq7K
f1OkffUjb/bAjAAAC3dJREFU74N3PNNUYeZD9i2rrz0wHfDzogHApAOKAuCA78d/1oJVuB40
a/BUOPc/h0mHuUTt9mo4nndk0lGuP4d+QeqRQL1hs2WHl3zaY5TzeyRSmulbDaXfcw/cH5SX
rAcLgCJXR70htBaw8iGRd3cGvgO9GuBEKZPJrQ2Ji9Tu+A5H8/Z4w2s7vspuXFp6INDJ156r
Wz+CH2u1e2W6va/YOd8bkjPN0O/0sBMv4SOXJ/EijeuzWriUE4RHa9Xgnwk0w2V1Z0JmhS5A
rxavsodN3rrbg7MeODNQAWASR/CGoDTBM8AxodVhEmwO83Ln4sVxomiBASGaLnQUKvHQtqA0
gVFAd8lE0vBryg1PlT8qQE51khA9SZw1cDwtkFbvTIIekfZOHjZ03D2YzAUvCEi4NOfFCqJM
2Fd4URGpd4QViJbIhLtM3jEpWZ//yLtC5DJT7ORQDO2RgJOW6K664ALvytfBt1bkH9D98GKp
D+A7dATf933f0n0wOf87pq9bwX8+hPGDREicla8HidMJ5uYuKi31MaweySj6h6QZphOXtlhe
dO3MMBehBHhp5leHhW523u3s0KUjxykF7dOQ13sSbnQ53zBS5rs/YV1lkw7IfDSd4tFY1xB8
Zx27JUX8RqSvtif8YjL+LpD3wnZKkx4C2f5a6KEk6eCDVVtxVDY8HHPs4Kc+mmEcWr8EH4ky
9X0zrYVzc+eVll+d4BGHipBZo5y8RAUQSYCXZn51UnVgfwkTbnpNJ+HuldIp1MCne4zPdCZW
Bwm7h8hbc0KxcT7GvbRAtFW+p4jdkqKHEakHxy3QNk6QkF+A2KYZ2P+4m6aBDzriarN+oAb+
/BIET9L7BzuRb7vUIY5SAx3gyQPwYKHDiQowARsg4cG0flBe7ipcmdDibsqr8MOCXgvqDtD5
x2mqUAeI/Gnm5LUCF4DvTZMrAV6a8j4E1e6M1pxur0avrBfXZz2X9k2onrT1YEHvPS0STguj
12IZpyYAwOLJI1553V6NdLWg2yRXpt7TAiJv0GG0MFLCMaLIT2KBaKqqAtXzaz2XL5PdktKt
VWRiN05D46Ux8h2TVPEMrix9O4ePyGuy8+55uk0zfVa/Fd4Tq1M8WEQ6dAI2/VXBBIex5bE5
BAAwRmRT3rRJeKRWVo8AAPbtztV4WwJzJlGdtEZzXdB1AGrXItKad48uyHjvpeG9ICA159vO
CW4mk46iKK1FlqOx3ZKS9fmPvCvylGbUgoW7Z5bP6lfFhXhDl56OuZAeepOv4IaKcngN5hXI
jjy1Gx9dmnWZr0aB6gQgrrQn0LLB8cG0mV2XWCPQGjwXdgrJIJLAjjBvneWuKkxnNvth6QD2
dRSLaM3xIvmbGcUJNuWhfqAWfEGAelD/Fdv2k1tbt1ZRd4/eTVFeUrI+/5F3RaQ06VUPb2tT
ErLSIuuR7B3AMg5IG2Dp0PtKPg2vAUwwgKx8nzxQT0B34Wu4o92yAeqgAgyb0KuHR5HYZbIc
E2ZqSR+H6rTAdMAAGDlgt+I40f/DM3BNKHoSS4U2XdNmaA0QSWB3gmV0Yu2cGsxq98oEcHtn
ZMwTrem6qfe6hCkF6ViXRvhwohOmN62NS0uPc4BJ57UOgEeq2Rs2d7QwcZMyLi09/oGJDyCp
J209nK97w7OeS72BHHkvlH+4Xe3OUj6rt30RmwzdGTmsrijQq4M/IuGXFvS+Aqgwc8BugaLA
GcDKArBBUWDcjmMCwKkJdot3UygXA3wntGkqoNSgvYKuCpMO2DpcGiSOpUMrTNyySHy6tCKG
TWIEyKipKHxnjFFiyaBbK86ywxg5JrVGD7Rm2oATvBwn/D9MxxilXhBAvqJnX2p3trIgVN+t
Bbccu2KMHDNU7LVxe+WYpAMnDj/tkpTana3aY1KP2rgd1kPtzkKbZq1Xd8o/LI3szh/hub2j
wMJKLqJjJqC0wFr9wvs/HjRDrfo7M+koLZtSsMHFIhNVAfKhecfXKMtj5EO9x7+i3lFAaYGD
NybLxxte2/AeNgERpFT+eusCCFGU8tPs1aDHC2+9Ql65uILCvAnlrje8YTOwZtPnahDk9+CP
WJ4jCIKUxR+xPEcQBCkLVJoIgiASoNJEEASR4NOyMxg07zdU0Gb4faAMBp0lHXbfTISwePff
le/3782HlbdcknPFb1G89ykTBEF245N22gD35SUO2SzH64bZgMWG0qQvLy5UP+/9+vJtj3f/
vfaTVEE9OvfPj/DoC4IgO/MJantVmC/jC4AvL25171JruA/L+E0wyzlUD08qb1JEBEGQ98Nf
oGqH+vThaQNGBQBgspzrhxfqHujrh7vNUbcCAJunZ9APNTJT29w3v02JD4NG2/+iRYnd3Qx6
cwAA/fhidlQJIz+0L867EP4T5HIzaM0bTv+LAQCw7AzGcHy8mO4Wod0fhWXx7r/XpmuAdW3w
YF2cnyy/1x4OV+dHcP+99nDoHD60puugAmEKScIUgIq2GX7/1oPjVTBpXd4o4+cocWt/mqw7
KxNhXpQ8TaoK3HAvWf4qKUBedRAEKYlPABWtXV2Pl8FKdnk7r7a1ShD4SFbtL4/raluLlOC0
3u77/b7fv7Cex7E9dD0d7134/b5/cQzTb8QAWjma9c+7FU4uevX5Kfi02SygcXq0c4RYe4N6
dL46rkL1eNUnSjZmPb2Gf0n5q/PWDcdSG+imVT+oZn91/Nwa3Eyg0j1vm+vp1yUALDvjudkO
E19Pe9BO1p2WiSgvWp7thj2+maTDL44X4wGdJlX+de/b4CyvOgiClMcnAKgc7Idmzc1mAfsH
FQConBxW7eUSAGC5tEkgTP6buo12OBuqdP9puNP/wqV94yqYYVWOfhxX7Z+JPZDKyWE1zqV6
2N4nGtBbPrgNzSghQkHCckLl5LAKzxv+Xs1+JX6Fy9G5T2Zw2uVx1f55P7n/acdyAICGE8y4
k3XPyctbPrjV48sgHe1LmAsr58rRVQNIW6TTLFgdBEHK4RMAgKE1gsFGax+1sh/YOr3NM5DA
zdMzwHysDAbkbzwHCKd7lNpSK/uwfkl4NlK1Qz1IcPkAh1pXa7gvLwCbu4e1vrdXSoRCVPfy
3bAf/WOG1UycGVCP/rVg2pruO5Rdgk6TqXteXquXNa2daega1fYobZhIs0B1EAQpj+DuuaaZ
4593Gw0e1uY/oS7QNBN+Pm02T3RgymYXUGh2U6nUYXq7/HL6sq5rFdjb08fLyd+b8bra/rdS
ToQt2Qy/f+sRA2bVujjvVrRRvz+CwHA5sIG23r48rgFgfrv8YmiiBBEE+V0hh9v3PlfXjy8v
j2v6XFE6sHKwD/SuOgO1MPQ2z7wZkHbagMXm/nbeONUCK+f89r8XN96X3z3CdlS658R86Sds
oNqXwNoYHTCY3IztRpuxPwLQ02pqYp4PM4VkoeWcMSFFEOTXQpRm5eSwuvj502a0DyfQ+PtY
X0/Pws2fyc0gPr8dhW/uz6Zr8x/OuUhDa7jTqU30aeVgH+z5PN6XLyNCBNdEUJDJzUChdlTi
34DlTWvecL5ooH1xGvSuS/h/UHet6BRU1Q51srPE3CAI5ByFX8+heJoIgrwmoWs4tbLvruf6
8b+09lG1Q5hOmcDK0ewCmt++KVMAAKiS8zceAFSP2y/flAEAfXRmw5w0Ak0zYb4IdZyhNWD+
TPbloYwIdHaaZsK4NZib7f6lpFSML33nZhDUharmsjOe68cXxA3738f6t3FH618CQPXYgnGy
7iLoQhJ5DgLfdPGzonAEQd4YdA23M+R8Et44QpA/AuV/8L+3LgNSMn2//9ZFQJDfFpxp7gzO
NBHkT4J5zwMqUARBkGzQnyaCIIgEqDQRBEEkQKWJIAgiASpNBEEQCVBpIgiCSIBKE0EQRAJU
mgiCIBKg0kQQBJEAlSaCIIgEqDQRBEEkQKWJIAgiwf8By1kDu7MtyrwAAAAASUVORK5CYII=
----boundary_998_71d08529-2a97-4e03-a5fa-4d4ed47c58a4--

----boundary_997_2870cf01-1463-4ef9-9d5d-023701733bf7--


From cmaiolino@redhat.com  Tue Jan 19 04:46:22 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 63DF67F37
	for ; Tue, 19 Jan 2016 04:46:22 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 54020304032
	for ; Tue, 19 Jan 2016 02:46:19 -0800 (PST)
X-ASG-Debug-ID: 1453200374-04cb6c7b8711e90001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id EgexISHtVBWZGtCQ (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 19 Jan 2016 02:46:15 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id CDE8DC0BF2B0
	for ; Tue, 19 Jan 2016 10:46:14 +0000 (UTC)
Received: from zion.usersys.redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0JAkDSr017092
	for ; Tue, 19 Jan 2016 05:46:14 -0500
From: Carlos Maiolino 
To: xfs@oss.sgi.com
Subject: [PATCH] Minor whitespace cleanups
Date: Tue, 19 Jan 2016 11:46:09 +0100
X-ASG-Orig-Subj: [PATCH] Minor whitespace cleanups
Message-Id: <1453200369-20171-1-git-send-email-cmaiolino@redhat.com>
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453200375
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

While investigating a quota issue, I fixed a few whitespaces.

Signed-off-by: Carlos Maiolino 
---
 include/xqm.h | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/include/xqm.h b/include/xqm.h
index 47f58a0..697cad4 100644
--- a/include/xqm.h
+++ b/include/xqm.h
@@ -55,14 +55,14 @@ typedef struct fs_disk_quota {
 	__s32		d_itimer;	/* zero if within inode limits */
 					/* if not, we refuse service */
 	__s32		d_btimer;	/* similar to above; for disk blocks */
-	__u16	  	d_iwarns;       /* # warnings issued wrt num inodes */
-	__u16	  	d_bwarns;       /* # warnings issued wrt disk blocks */
+	__u16		d_iwarns;       /* # warnings issued wrt num inodes */
+	__u16		d_bwarns;       /* # warnings issued wrt disk blocks */
 	__s32		d_padding2;	/* padding2 - for future use */
 	__u64		d_rtb_hardlimit;/* absolute limit on realtime blks */
 	__u64		d_rtb_softlimit;/* preferred limit on RT disk blks */
 	__u64		d_rtbcount;	/* # realtime blocks owned */
 	__s32		d_rtbtimer;	/* similar to above; for RT disk blks */
-	__u16	  	d_rtbwarns;     /* # warnings issued wrt RT disk blks */
+	__u16		d_rtbwarns;     /* # warnings issued wrt RT disk blks */
 	__s16		d_padding3;	/* padding3 - for future use */
 	char		d_padding4[8];	/* yet more padding */
 } fs_disk_quota_t;
@@ -73,7 +73,7 @@ typedef struct fs_disk_quota {
 #define FS_DQ_ISOFT	(1<<0)
 #define FS_DQ_IHARD	(1<<1)
 #define FS_DQ_BSOFT	(1<<2)
-#define FS_DQ_BHARD 	(1<<3)
+#define FS_DQ_BHARD	(1<<3)
 #define FS_DQ_RTBSOFT	(1<<4)
 #define FS_DQ_RTBHARD	(1<<5)
 #define FS_DQ_LIMIT_MASK	(FS_DQ_ISOFT | FS_DQ_IHARD | FS_DQ_BSOFT | \
@@ -87,7 +87,7 @@ typedef struct fs_disk_quota {
  */
 #define FS_DQ_BTIMER	(1<<6)
 #define FS_DQ_ITIMER	(1<<7)
-#define FS_DQ_RTBTIMER 	(1<<8)
+#define FS_DQ_RTBTIMER	(1<<8)
 #define FS_DQ_TIMER_MASK	(FS_DQ_BTIMER | FS_DQ_ITIMER | FS_DQ_RTBTIMER)
 
 /*
-- 
2.4.3

From prvs=2826D68DDF=Margaret.Hoey@gsahs.health.nsw.gov.au  Tue Jan 19 04:53:47 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id D6E3E7F37
	for ; Tue, 19 Jan 2016 04:53:47 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id B472F304032
	for ; Tue, 19 Jan 2016 02:53:47 -0800 (PST)
X-ASG-Debug-ID: 1453200823-04cb6c7b8712180001-NocioJ
Received: from mail1.gsah.health.nsw.gov.au (mail1.gsahs.health.nsw.gov.au [202.59.20.70]) by cuda.sgi.com with ESMTP id ouGp7sjWMVLMV28n (version=TLSv1 cipher=AES128-SHA bits=128 verify=NO) for ; Tue, 19 Jan 2016 02:53:45 -0800 (PST)
X-Barracuda-Envelope-From: prvs=2826D68DDF=Margaret.Hoey@gsahs.health.nsw.gov.au
X-Barracuda-Apparent-Source-IP: 202.59.20.70
Received: from SRVQUEMA05.gsahs.net (10.124.0.156) by
 mail1.gsahs.health.nsw.gov.au (10.124.0.162) with Microsoft SMTP Server (TLS)
 id 14.3.210.2; Tue, 19 Jan 2016 21:53:40 +1100
Received: from QBDCMBX-SMS005.gsahs.net ([fe80::e5b9:3ae:6efd:b8c3]) by
 SRVQUEMA05.gsahs.net ([::1]) with mapi id 14.03.0210.002; Tue, 19 Jan 2016
 21:52:44 +1100
From: 
To: 
Subject: RE
Thread-Topic: RE
X-ASG-Orig-Subj: RE
Thread-Index: AdFSp4Xd+0rrlsGERDuNzwDeAFW4CA==
Date: Tue, 19 Jan 2016 10:52:43 +0000
Message-ID: 
Accept-Language: en-AU, en-US
Content-Language: en-AU
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.124.0.47]
Content-Type: multipart/alternative;
	boundary="_000_CEB6438F03002545A23C8605102907600CF99A2BQBDCMBXSMS005gs_"
MIME-Version: 1.0
X-Barracuda-Connect: mail1.gsahs.health.nsw.gov.au[202.59.20.70]
X-Barracuda-Start-Time: 1453200824
X-Barracuda-Encrypted: AES128-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.02
X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, HTML_MESSAGE, NO_REAL_NAME, THREAD_INDEX, THREAD_TOPIC
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26262
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.01 THREAD_INDEX           thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig==
	0.00 NO_REAL_NAME           From: does not include a real name
	0.01 THREAD_TOPIC           Thread-Topic: ...(Japanese Subject)...
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 HTML_MESSAGE           BODY: HTML included in message

--_000_CEB6438F03002545A23C8605102907600CF99A2BQBDCMBXSMS005gs_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable



I got a donation for you, Email me for details via::   mr.pedque@qq.com
Best Regards
Mr. Pedro.

________________________________

** This email is intended for the addressee named and may contain confident=
ial information. If you are not the intended recipient, please delete it an=
d notify the sender. Unless explicitly attributed, the opinions expressed i=
n this email are those of the author only and do not represent the official=
 view of NSW Health nor the New South Wales Government **

--_000_CEB6438F03002545A23C8605102907600CF99A2BQBDCMBXSMS005gs_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable





























I got a donation for you=
, Email me for details via::   mr.pedque@qq.com


Best Regards 


Mr. Pedro. 

























** This email is intended for the addressee named and may contain confident=
ial information. If you are not the intended recipient, please delete it an=
d notify the sender. Unless explicitly attributed, the opinions expressed i=
n this email are those of the author
 only and do not represent the official view of NSW Health nor the New Sout=
h Wales Government **





--_000_CEB6438F03002545A23C8605102907600CF99A2BQBDCMBXSMS005gs_--

From macsaimelinda@gmail.com  Tue Jan 19 11:34:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,
	T_DKIM_INVALID autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id CFEB77F37
	for ; Tue, 19 Jan 2016 11:34:30 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 6E345AC005
	for ; Tue, 19 Jan 2016 09:34:30 -0800 (PST)
X-ASG-Debug-ID: 1453224854-04bdf05f190bf40001-NocioJ
Received: from mail-lb0-f195.google.com (mail-lb0-f195.google.com [209.85.217.195]) by cuda.sgi.com with ESMTP id RnFftzKH5Cob0Lqq (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 19 Jan 2016 09:34:15 -0800 (PST)
X-Barracuda-Envelope-From: macsaimelinda@gmail.com
X-Barracuda-Apparent-Source-IP: 209.85.217.195
Received: by mail-lb0-f195.google.com with SMTP id dx9so9097107lbc.2
        for ; Tue, 19 Jan 2016 09:34:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=vbzR5vGiIG8DsqrgpzbxuwC6qJacFgwxFKwT5BBkux0=;
        b=cb8Sx/KdU03bzfpFfd7/LZvCJRCXDYrgw7+AP0QxDyuUPn28ZvH2fQnlEdb95lAdZB
         bEMcfuqLtapbwlQBH5+cfCTBoEsn7LQIg7NPJWG+VJXUKXvFbR7vcY3QUP9ipuu9xCKj
         Y4w818FTqODoexY+W6SHbBlKPNNEJ4xf9bfNr8d9rCfDeXZ9l3J/w/AITrqM810QYhYt
         QW4fYT03F3LhFdYEuIYr9ShY0K98KDDwXz3MkwAyHhBBPTusqSB2WAT9JG53svGDziSq
         eGk4PXKQlLCgZU2bPZQdWH2AUR8hl8l234nbCGKSaqynd2B2M1JT/M9R7IAatwPrMru9
         qjcw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=vbzR5vGiIG8DsqrgpzbxuwC6qJacFgwxFKwT5BBkux0=;
        b=XEhkvOcG3EWHwy6/oUSZ5azyi+Fo1WCPMXBuAOIGsNuuePgicRPhS0iJx69Tyzx7ZL
         2sEfoPwe10Y5U2iUOJc3z6+VgxUioqg3POEvDWyeYJ5UGmaBX53KK3+gEJgDtqsVTbS0
         kaAiTytxtcP6nT5zBgrphdUcoEHR09CruYbomQKysGkB5Sq+djs00MZ7gk226/H58ymH
         uA2U8RkDlJaU2vljpaLlZ1eNk8uvz+tuyDe7JNKRpBksT/CwAeMznhtU7asrwmRmvE5V
         e9JIKJpRho8Sog5+3LHW5ylFckWMyMea2bopNOoyI5/CeKxVW1PUGj8deMeCFNX5vOg6
         bdDA==
X-Gm-Message-State: ALoCoQnYStqpRI3VONl20I4ncu4T2VAK7sHtKmP3gEHAU4ioSNlRc2OdZmhzPPpBvLHRRVjAsypA2AGtGdJL8xK/ntLEGoF7lg==
MIME-Version: 1.0
X-Received: by 10.112.173.164 with SMTP id bl4mr9203680lbc.144.1453224854104;
 Tue, 19 Jan 2016 09:34:14 -0800 (PST)
Received: by 10.112.227.75 with HTTP; Tue, 19 Jan 2016 09:34:14 -0800 (PST)
Date: Tue, 19 Jan 2016 18:34:14 +0100
Message-ID: 
Subject: Re:Payment
From: =?UTF-8?Q?Melinda_M=C3=A1csai?= 
X-ASG-Orig-Subj: Re:Payment
To: undisclosed-recipients:;
Content-Type: multipart/alternative; boundary=001a11c1a43e3536090529b34763
X-Barracuda-Connect: mail-lb0-f195.google.com[209.85.217.195]
X-Barracuda-Start-Time: 1453224855
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26270
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	0.00 HTML_MESSAGE           BODY: HTML included in message

--001a11c1a43e3536090529b34763
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Good Day

I have been directed by my Supplier to release the following payment to you
directly to avoid delay in processing my order ,Please see below the
invoice paid for,and the scan tt remittance use in making payment please
confirm receipt of money and ETD to my supplier.

*  VIEW/DOWNLOAD: INVOICE AND ORDER CONFIRMATION

   DOWNLOAD WIRE REMITTANCE
*

*Regards*
*Melinda*

--001a11c1a43e3536090529b34763
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable


Good =
Day

<=
/div>
I=
 have been directed by my Supplier to release the following payment to you =
directly to avoid delay in processing my order ,Please see below the invoic=
e paid for,and the scan tt remittance use in making payment please confirm =
receipt of money and ETD to my supplier.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0=C2=A0


--001a11c1a43e3536090529b34763--

From david@fromorbit.com  Tue Jan 19 15:42:42 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 398647F37
	for ; Tue, 19 Jan 2016 15:42:42 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 2B2FF8F8052
	for ; Tue, 19 Jan 2016 13:42:38 -0800 (PST)
X-ASG-Debug-ID: 1453239752-04cbb00afe24c30001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id RlSg5MUaTIdt6UJj for ; Tue, 19 Jan 2016 13:42:33 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2CPDwAHrZ5WPGwtLHleKAECgw8iMG2CZIN6gXifGQIKDAEBAQEBAQaBDYpQhUKEBBgChW8EgUhNAQEBAQEBBwEBAQFBP0EOAYRBOyQ0BSUDBy2IGqBQn2EIGIVailmCEQxBgTYFh2aPNIVIiA6PCAKOXIIuDAGCRio0ghCEXgEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 20 Jan 2016 08:12:32 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aLe2p-0006xa-Ht
	for xfs@oss.sgi.com; Wed, 20 Jan 2016 08:42:31 +1100
Date: Wed, 20 Jan 2016 08:42:31 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [ANNOUNCE] xfs: for-next branch updated to ee3804d
Message-ID: <20160119214231.GS6033@dastard>
X-ASG-Orig-Subj: [ANNOUNCE] xfs: for-next branch updated to ee3804d
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="pf9I7BMVVzbSWLtt"
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1453239752
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26279
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------


--pf9I7BMVVzbSWLtt
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi folks,

The for-next branch of the xfs kernel repository at

git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git

has just been updated. This contains a couple of regression fixes
for the recent 4.5 merge, and the XFS_IOC_FS[SG]SETXATTR API vfs
promotion that I forgot to merge into the original pull request.
If nothing comes up, I'll ask Linus to pull these in a couple of
days time.

-Dave.

The new head of the for-next branch is commit:

ee3804d Merge branch 'xfs-misc-fixes-for-4.5-3' into for-next

New Commits:

Dave Chinner (7):
      [334e580] fs: XFS_IOC_FS[SG]SETXATTR to FS_IOC_FS[SG]ETXATTR promotion
      [e7b8948] xfs: use FS_XFLAG definitions directly
      [58f88ca] xfs: introduce per-inode DAX enablement
      [4c931f7] Merge branch 'xfs-setxattr-promotion' into for-next
      [3e85286] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
      [85bec54] xfs: log mount failures don't wait for buffers to be releas=
ed
      [ee3804d] Merge branch 'xfs-misc-fixes-for-4.5-3' into for-next


Code Diffstat:

 fs/xfs/libxfs/xfs_format.h | 11 +++++-
 fs/xfs/libxfs/xfs_fs.h     | 38 +------------------
 fs/xfs/xfs_buf.c           | 10 +++++
 fs/xfs/xfs_inode.c         | 60 ++++++++++++++++++------------
 fs/xfs/xfs_ioctl.c         | 92 +++++++++++++++++++++++++++---------------=
----
 fs/xfs/xfs_iops.c          |  4 +-
 fs/xfs/xfs_trans_ail.c     |  1 -
 include/uapi/linux/fs.h    | 33 +++++++++++++++++
 8 files changed, 147 insertions(+), 102 deletions(-)
--=20
Dave Chinner
david@fromorbit.com

--pf9I7BMVVzbSWLtt
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBAgAGBQJWnq3HAAoJEK3oKUf0dfod9ikQAIFmmb19FqS42E7u74IJaPLq
13CuYWnqULrJwzfhQUoeJeavIrcHWj6whV756QwSNG/c9c2ODp/H8l6e+vPw0Ggm
fOPav3T0vufWz+trgVZjMY/DiViYVas+wM9paliFC15XTq9tGgaP98CCCTpcQ4JM
9cbKegcbM3jcR2clq1Z3biXPzdQjVMJFYyapZiO9Z1jHE8ziC+hPFD0T74YbbU6P
t5FqVRbfGmjLANlicbaPoty8Eis+noLUEykjPIlFVFYLcHdsZd45KX5dV1ESVcUD
mmiAxzmRnZ1GlrejX5iGRdaOprfQjZPy6S/BGZMydelV85/X0327bImXbXzLL+8j
dTde/VPXKi23eyqGe869VmIqpSX1/HLUbIupeRKbLbvfM9NRv4bfGdZxD4KXhiPr
gSPmjsTpGq69fxMwZpcA+MCuowWNhOfySM0UQ0htm3MMytcRIHy5VmhOYC45Dvx6
fqgvUVCtZoWIk+wL5ZqUSpgzARqKmWeJGIm/QvUBikdQ/ct/zGSJ6kmQhQVjJWkJ
oXr1lsMh/KMYK22m5mLvGL+uXQaCUZoRQJhlcA0NF1kbSMLaE19s6TdXO+3SDHPF
9YJDjNhnfCusXMCXdl/VUV02Cyotj3zPjDrez94m5QNnkdKwzYlM0qwGAUldmakN
uBCmRXlWKe5p2UEpO3Fk
=mIFI
-----END PGP SIGNATURE-----

--pf9I7BMVVzbSWLtt--

From david@fromorbit.com  Tue Jan 19 18:47:20 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 70FDF7F37
	for ; Tue, 19 Jan 2016 18:47:20 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 4E34B304059
	for ; Tue, 19 Jan 2016 16:47:17 -0800 (PST)
X-ASG-Debug-ID: 1453250831-04bdf04d6a007f0001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id xQCW9ooHn6ZBW51P for ; Tue, 19 Jan 2016 16:47:11 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2AICQCe2J5WPGwtLHleKAECgw+BP4Jlg3qBeJ8xAQEBAQEBBotdhUKEBIYJBAICgUFNAQEBAQEBBwEBAQFBP4Q1AQEEOhwjEAgDDgoJJQ8FJQMHGhOIGr9mAQseGIVahTyJMQWXGog8hRqPCkSOGIUBKjSHMgEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 20 Jan 2016 11:16:40 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aLgv1-0007Fc-Ac; Wed, 20 Jan 2016 11:46:39 +1100
Date: Wed, 20 Jan 2016 11:46:39 +1100
From: Dave Chinner 
To: Christoph Hellwig 
Cc: Felix Janda , xfs@oss.sgi.com
Subject: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
Message-ID: <20160120004639.GT6033@dastard>
X-ASG-Orig-Subj: Re: [PATCH 1/6] Move from __uint*_t types to uint*_t and likewise
 for __int*_t
References: 
 <20160112195935.GB568@nyan>
 <20160112212405.GL10456@dastard>
 <20160113074859.GB21939@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113074859.GB21939@infradead.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1453250831
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26282
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue, Jan 12, 2016 at 11:48:59PM -0800, Christoph Hellwig wrote:
> On Wed, Jan 13, 2016 at 08:24:06AM +1100, Dave Chinner wrote:
> > I can't apply this straight off. Most of the libxfs code that is
> > changed is shared with the kernel code, and so the definitions of
> > the variables need to be the same as the kernel code. There are
> > reasons for the kernel code using __[u]int*_t type variants (e.g.  I
> > think the endian conversion static checker requires the __ variants
> > for host order variables), and so before making sweeping changes
> > like this we need to ensure that we can make the equivalent changes
> > to the kernel code as well...
> 
> There is no functional requirement for these types.  I'm not sure if
> they are an IRIXism, or something that was added during the early
> Linux port, but they certainly aren't nessecary.
> 
> I'd love to kill them in favour or either the kernel __u/s types or the
> C99 types which are like the urrent ones just without the __ prefix.
> 
> Maybe we should get an agreement which ones we want and ask Felix for
> a patch to the kernel tree?

Either way is fine by me, but it's not a small patch - there's ~600
variables declared in fs/xfs with the __[u]int*_t types.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Tue Jan 19 19:59:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 148097F37
	for ; Tue, 19 Jan 2016 19:59:13 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id CE16B304067
	for ; Tue, 19 Jan 2016 17:59:09 -0800 (PST)
X-ASG-Debug-ID: 1453255144-04bdf04d6c022f0001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id NpBW2O1cAm6tYW7m for ; Tue, 19 Jan 2016 17:59:05 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2BkCQAv6Z5WPGwtLHleKAECgw+BP4Jlg3qBeJ8xAQEBAQEBBotdhUKEBIYJAgIBAQKBRE0BAQEBAQEHAQEBAUE/hDUBAQQnExwWChMIGAwMGQ8FJQMHLRuHf8AJGIVaijkThCEFh2MDhxCIJIg8hRqBZ40jhW+IbYJ1G4FxKjSHMgEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 20 Jan 2016 12:28:53 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aLi2v-0007ZY-Gz
	for xfs@oss.sgi.com; Wed, 20 Jan 2016 12:58:53 +1100
Date: Wed, 20 Jan 2016 12:58:53 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [PATCH v2] [RFC] xfs: allocate log vector buffers outside CIL
 context lock
Message-ID: <20160120015853.GU6033@dastard>
X-ASG-Orig-Subj: [PATCH v2] [RFC] xfs: allocate log vector buffers outside CIL
 context lock
References: <1453177919-17849-1-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453177919-17849-1-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1453255144
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26284
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

From: Dave Chinner 

One of the problems we currently have with delayed logging is that
under serious memory pressure we can deadlock memory reclaim. THis
occurs when memory reclaim (such as run by kswapd) is reclaiming XFS
inodes and issues a log force to unpin inodes that are dirty in the
CIL.

The CIL is pushed, but this will only occur once it gets the CIL
context lock to ensure that all committing transactions are complete
and no new transactions start being committed to the CIL while the
push switches to a new context.

The deadlock occurs when the CIL context lock is held by a
committing process that is doing memory allocation for log vector
buffers, and that allocation is then blocked on memory reclaim
making progress. Memory reclaim, however, is blocked waiting for
a log force to make progress, and so we effectively deadlock at this
point.

To solve this problem, we have to move the CIL log vector buffer
allocation outside of the context lock so that memory reclaim can
always make progress when it needs to force the log. The problem
with doing this is that a CIL push can take place while we are
determining if we need to allocate a new log vector buffer for
an item and hence the current log vector may go away without
warning. That means we canot rely on the existing log vector being
present when we finally grab the context lock and so we must have a
replacement buffer ready to go at all times.

To ensure this, introduce a "shadow log vector" buffer that is
always guaranteed to be present when we gain the CIL context lock
and format the item. This shadow buffer may or may not be used
during the formatting, but if the log item does not have an existing
log vector buffer or that buffer is too small for the new
modifications, we swap it for the new shadow buffer and format
the modifications into that new log vector buffer.

The result of this is that for any object we modify more than once
in a given CIL checkpoint, we double the memory required
to track dirty regions in the log. For single modifications then
we consume the shadow log vectorwe allocate on commit, and that gets
consumed by the checkpoint. However, if we make multiple
modifications, then the second transaction commit will allocate a
shadow log vector and hence we will end up with double the memory
usage as only one of the log vectors is consumed by the CIL
checkpoint. The remaining shadow vector will be freed when th elog
item is freed.

This can probably be optimised - access to the shadow log vector is
serialised by the object lock (as opposited to the active log
vector, which is controlled by the CIL context lock) and so we can
probably free shadow log vector from some objects when the log item
is marked clean on removal from the AIL.

The patch survives smoke testing and some load testing. I haven't
done any real performance testing, but I have done some load and low
memory testing and it hasn't exploded (perf did - it failed several
order 2 memory allocations, which XFS continued along just fine).

That said, I don't have a reliable deadlock reproducer in the first
place, so I'm interested i hearing what people think about this
approach to solve the problem and ways to test and improve it.

Signed-off-by: Dave Chinner 
---

Version 2:
- this one doesn't crash and burn in generic/324
- fixed handling of order items when recycling shadow buffers
- correctly set up log iovec pointers in all cases
- fixed moving current log vector back to the shadow vector when
  they are switched during formatting.

 fs/xfs/xfs_buf_item.c     |   1 +
 fs/xfs/xfs_dquot.c        |   1 +
 fs/xfs/xfs_dquot_item.c   |   2 +
 fs/xfs/xfs_extfree_item.c |   2 +
 fs/xfs/xfs_inode_item.c   |   1 +
 fs/xfs/xfs_log_cil.c      | 249 ++++++++++++++++++++++++++++++++++------------
 fs/xfs/xfs_trans.h        |   1 +
 7 files changed, 193 insertions(+), 64 deletions(-)

diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c
index 9220283..3f39d96 100644
--- a/fs/xfs/xfs_buf_item.c
+++ b/fs/xfs/xfs_buf_item.c
@@ -833,6 +833,7 @@ xfs_buf_item_free(
 	xfs_buf_log_item_t	*bip)
 {
 	xfs_buf_item_free_format(bip);
+	kmem_free(bip->bli_item.li_lv_shadow);
 	kmem_zone_free(xfs_buf_item_zone, bip);
 }
 
diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 9c44d38..4569cc4 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -74,6 +74,7 @@ xfs_qm_dqdestroy(
 {
 	ASSERT(list_empty(&dqp->q_lru));
 
+	kmem_free(dqp->q_logitem.qli_item.li_lv_shadow);
 	mutex_destroy(&dqp->q_qlock);
 
 	XFS_STATS_DEC(dqp->q_mount, xs_qm_dquot);
diff --git a/fs/xfs/xfs_dquot_item.c b/fs/xfs/xfs_dquot_item.c
index 814cff9..2c7a162 100644
--- a/fs/xfs/xfs_dquot_item.c
+++ b/fs/xfs/xfs_dquot_item.c
@@ -370,6 +370,8 @@ xfs_qm_qoffend_logitem_committed(
 	spin_lock(&ailp->xa_lock);
 	xfs_trans_ail_delete(ailp, &qfs->qql_item, SHUTDOWN_LOG_IO_ERROR);
 
+	kmem_free(qfs->qql_item.li_lv_shadow);
+	kmem_free(lip->li_lv_shadow);
 	kmem_free(qfs);
 	kmem_free(qfe);
 	return (xfs_lsn_t)-1;
diff --git a/fs/xfs/xfs_extfree_item.c b/fs/xfs/xfs_extfree_item.c
index 4aa0153..ab77946 100644
--- a/fs/xfs/xfs_extfree_item.c
+++ b/fs/xfs/xfs_extfree_item.c
@@ -40,6 +40,7 @@ void
 xfs_efi_item_free(
 	struct xfs_efi_log_item	*efip)
 {
+	kmem_free(efip->efi_item.li_lv_shadow);
 	if (efip->efi_format.efi_nextents > XFS_EFI_MAX_FAST_EXTENTS)
 		kmem_free(efip);
 	else
@@ -300,6 +301,7 @@ static inline struct xfs_efd_log_item *EFD_ITEM(struct xfs_log_item *lip)
 STATIC void
 xfs_efd_item_free(struct xfs_efd_log_item *efdp)
 {
+	kmem_free(efdp->efd_item.li_lv_shadow);
 	if (efdp->efd_format.efd_nextents > XFS_EFD_MAX_FAST_EXTENTS)
 		kmem_free(efdp);
 	else
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index bd9808f..45a882f 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -649,6 +649,7 @@ void
 xfs_inode_item_destroy(
 	xfs_inode_t	*ip)
 {
+	kmem_free(ip->i_itemp->ili_item.li_lv_shadow);
 	kmem_zone_free(xfs_ili_zone, ip->i_itemp);
 }
 
diff --git a/fs/xfs/xfs_log_cil.c b/fs/xfs/xfs_log_cil.c
index 4e76493..f5567fb 100644
--- a/fs/xfs/xfs_log_cil.c
+++ b/fs/xfs/xfs_log_cil.c
@@ -79,6 +79,148 @@ xlog_cil_init_post_recovery(
 	log->l_cilp->xc_ctx->sequence = 1;
 }
 
+static inline int
+xlog_cil_iovec_space(
+	uint	niovecs)
+{
+	return round_up((sizeof(struct xfs_log_vec) +
+					niovecs * sizeof(struct xfs_log_iovec)),
+			sizeof(uint64_t));
+}
+
+/*
+ * Allocate or pin log vector buffers for CIL insertion.
+ *
+ * The CIL currently uses disposable buffers for copying a snapshot of the
+ * modified items into the log during a push. The biggest problem with this is
+ * the requirement to allocate the disposable buffer during the commit if:
+ *	a) does not exist; or
+ *	b) it is too small
+ *
+ * If we do this allocation within xlog_cil_insert_format_items(), it is done
+ * under the xc_ctx_lock, which means that a CIL push cannot occur during
+ * the memory allocation. This means that we have a potential deadlock situation
+ * under low memory conditions when we have lots of dirty metadata pinned in
+ * the CIL and we need a CIL commit to occur to free memory.
+ *
+ * To avoid this, we need to move the memory allocation outside the
+ * xc_ctx_lock(), but because the log vector buffers are disposable, that opens
+ * up a TOCTOU race condition w.r.t. the CIL commiting and removing the log
+ * vector buffers between the check and the formatting of the item into the
+ * log vector buffer within the xc_ctx_lock.
+ *
+ * Because the log vector buffer needs to be unchanged during the CIL push
+ * process, we cannot share the buffer between the transaction commit (which
+ * modifies the buffer) and the CIL push context that is writing the changes
+ * into the log. This means skipping preallocation of buffer space is
+ * unreliable, but we most definitely do not want to be allocating and freeing
+ * buffers unnecessarily during commits when overwrites can be done safely.
+ *
+ * The simplest solution to this problem is to allocate a shadow buffer when a
+ * log item is committed for the second time, and then to only use this buffer
+ * if necessary. The buffer can remain attached to the log item until such time
+ * it is needed, and this is the buffer that is reallocated to match the size of
+ * the incoming modification. Then during the formatting of the item we can swap
+ * the active buffer with the new one if we can't reuse the existing buffer. We
+ * don't free the old buffer as it may be reused on the next modification if
+ * it's size is right, otherwise we'll free and reallocate it at that point.
+ *
+ * This function builds a vector for the changes in each log item in the
+ * transaction. It then works out the length of the buffer needed for each log
+ * item, allocates them and attaches the vector to the log item in preparation
+ * for the formatting step which occurs under the xc_ctx_lock.
+ *
+ * While this means the memory footprint goes up, it avoids the repeated
+ * alloc/free pattern that repeated modifications of an item would otherwise
+ * cause, and hence minimises the CPU overhead of such behaviour.
+ */
+static void
+xlog_cil_alloc_shadow_bufs(
+	struct xlog		*log,
+	struct xfs_trans	*tp)
+{
+	struct xfs_log_item_desc *lidp;
+
+	list_for_each_entry(lidp, &tp->t_items, lid_trans) {
+		struct xfs_log_item *lip = lidp->lid_item;
+		struct xfs_log_vec *lv;
+		int	niovecs = 0;
+		int	nbytes = 0;
+		int	buf_size;
+		bool	ordered = false;
+
+		/* Skip items which aren't dirty in this transaction. */
+		if (!(lidp->lid_flags & XFS_LID_DIRTY))
+			continue;
+
+		/* get number of vecs and size of data to be stored */
+		lip->li_ops->iop_size(lip, &niovecs, &nbytes);
+
+		/*
+		 * Ordered items need to be tracked but we do not wish to write
+		 * them. We need a logvec to track the object, but we do not
+		 * need an iovec or buffer to be allocated for copying data.
+		 */
+		if (niovecs == XFS_LOG_VEC_ORDERED) {
+			ordered = true;
+			niovecs = 0;
+			nbytes = 0;
+		}
+
+		/*
+		 * We 64-bit align the length of each iovec so that the start
+		 * of the next one is naturally aligned.  We'll need to
+		 * account for that slack space here. Then round nbytes up
+		 * to 64-bit alignment so that the initial buffer alignment is
+		 * easy to calculate and verify.
+		 */
+		nbytes += niovecs * sizeof(uint64_t);
+		nbytes = round_up(nbytes, sizeof(uint64_t));
+
+		/*
+		 * The data buffer needs to start 64-bit aligned, so round up
+		 * that space to ensure we can align it appropriately and not
+		 * overrun the buffer.
+		 */
+		buf_size = nbytes + xlog_cil_iovec_space(niovecs);
+
+		/*
+		 * if we have no shadow buffer, or it is too small, we need to
+		 * reallocate it.
+		 */
+		if (!lip->li_lv_shadow ||
+		    buf_size > lip->li_lv_shadow->lv_size) {
+
+			kmem_free(lip->li_lv_shadow);
+
+			lv = kmem_zalloc(buf_size, KM_SLEEP|KM_NOFS);
+			lv->lv_item = lip;
+			lv->lv_size = buf_size;
+			if (ordered)
+				lv->lv_buf_len = XFS_LOG_VEC_ORDERED;
+			else
+				lv->lv_iovecp = (struct xfs_log_iovec *)&lv[1];
+			lip->li_lv_shadow = lv;
+		} else {
+			/* same or smaller, optimise common overwrite case */
+			lv = lip->li_lv_shadow;
+			if (ordered)
+				lv->lv_buf_len = XFS_LOG_VEC_ORDERED;
+			else
+				lv->lv_buf_len = 0;
+			lv->lv_bytes = 0;
+			lv->lv_next = NULL;
+		}
+
+		/* Ensure the lv is set up according to ->iop_size */
+		lv->lv_niovecs = niovecs;
+
+		/* The allocated data region lies beyond the iovec region */
+		lv->lv_buf = (char *)lv + xlog_cil_iovec_space(niovecs);
+	}
+
+}
+
 /*
  * Prepare the log item for insertion into the CIL. Calculate the difference in
  * log space and vectors it will consume, and if it is a new item pin it as
@@ -101,16 +243,19 @@ xfs_cil_prepare_item(
 	/*
 	 * If there is no old LV, this is the first time we've seen the item in
 	 * this CIL context and so we need to pin it. If we are replacing the
-	 * old_lv, then remove the space it accounts for and free it.
+	 * old_lv, then remove the space it accounts for and make it the shadow
+	 * buffer for later freeing. In both cases we are now switching to the
+	 * shadow buffer, so update the the pointer to it appropriately.
 	 */
-	if (!old_lv)
+	if (!old_lv) {
 		lv->lv_item->li_ops->iop_pin(lv->lv_item);
-	else if (old_lv != lv) {
+		lv->lv_item->li_lv_shadow = NULL;
+	} else if (old_lv != lv) {
 		ASSERT(lv->lv_buf_len != XFS_LOG_VEC_ORDERED);
 
 		*diff_len -= old_lv->lv_bytes;
 		*diff_iovecs -= old_lv->lv_niovecs;
-		kmem_free(old_lv);
+		lv->lv_item->li_lv_shadow = old_lv;
 	}
 
 	/* attach new log vector to log item */
@@ -134,11 +279,13 @@ xfs_cil_prepare_item(
  * write it out asynchronously without needing to relock the object that was
  * modified at the time it gets written into the iclog.
  *
- * This function builds a vector for the changes in each log item in the
- * transaction. It then works out the length of the buffer needed for each log
- * item, allocates them and formats the vector for the item into the buffer.
- * The buffer is then attached to the log item are then inserted into the
- * Committed Item List for tracking until the next checkpoint is written out.
+ * This function takes the prepared log vectors attached to each log item, and
+ * formats the changes into the log vector buffer. The buffer it uses is
+ * dependent on the current state of the vector in the CIL - the shadow lv is
+ * guaranteed to be large enough for the current modification, but we will only
+ * use that if we can't reuse the existing lv. If we can't reuse the existing
+ * lv, then simple swap it out for the shadow lv. We don't free it - that is
+ * done lazily either by th enext modification or the freeing of the log item.
  *
  * We don't set up region headers during this process; we simply copy the
  * regions into the flat buffer. We can do this because we still have to do a
@@ -171,59 +318,29 @@ xlog_cil_insert_format_items(
 	list_for_each_entry(lidp, &tp->t_items, lid_trans) {
 		struct xfs_log_item *lip = lidp->lid_item;
 		struct xfs_log_vec *lv;
-		struct xfs_log_vec *old_lv;
-		int	niovecs = 0;
-		int	nbytes = 0;
-		int	buf_size;
+		struct xfs_log_vec *old_lv = NULL;
+		struct xfs_log_vec *shadow;
 		bool	ordered = false;
 
 		/* Skip items which aren't dirty in this transaction. */
 		if (!(lidp->lid_flags & XFS_LID_DIRTY))
 			continue;
 
-		/* get number of vecs and size of data to be stored */
-		lip->li_ops->iop_size(lip, &niovecs, &nbytes);
-
-		/* Skip items that do not have any vectors for writing */
-		if (!niovecs)
-			continue;
-
 		/*
-		 * Ordered items need to be tracked but we do not wish to write
-		 * them. We need a logvec to track the object, but we do not
-		 * need an iovec or buffer to be allocated for copying data.
+		 * The formatting size information is already attached to
+		 * the shadow lv on the log item.
 		 */
-		if (niovecs == XFS_LOG_VEC_ORDERED) {
+		shadow = lip->li_lv_shadow;
+		if (shadow->lv_buf_len == XFS_LOG_VEC_ORDERED)
 			ordered = true;
-			niovecs = 0;
-			nbytes = 0;
-		}
 
-		/*
-		 * We 64-bit align the length of each iovec so that the start
-		 * of the next one is naturally aligned.  We'll need to
-		 * account for that slack space here. Then round nbytes up
-		 * to 64-bit alignment so that the initial buffer alignment is
-		 * easy to calculate and verify.
-		 */
-		nbytes += niovecs * sizeof(uint64_t);
-		nbytes = round_up(nbytes, sizeof(uint64_t));
-
-		/* grab the old item if it exists for reservation accounting */
-		old_lv = lip->li_lv;
-
-		/*
-		 * The data buffer needs to start 64-bit aligned, so round up
-		 * that space to ensure we can align it appropriately and not
-		 * overrun the buffer.
-		 */
-		buf_size = nbytes +
-			   round_up((sizeof(struct xfs_log_vec) +
-				     niovecs * sizeof(struct xfs_log_iovec)),
-				    sizeof(uint64_t));
+		/* Skip items that do not have any vectors for writing */
+		if (!shadow->lv_niovecs && !ordered)
+			continue;
 
 		/* compare to existing item size */
-		if (lip->li_lv && buf_size <= lip->li_lv->lv_size) {
+		old_lv = lip->li_lv;
+		if (lip->li_lv && shadow->lv_size <= lip->li_lv->lv_size) {
 			/* same or smaller, optimise common overwrite case */
 			lv = lip->li_lv;
 			lv->lv_next = NULL;
@@ -237,32 +354,29 @@ xlog_cil_insert_format_items(
 			 */
 			*diff_iovecs -= lv->lv_niovecs;
 			*diff_len -= lv->lv_bytes;
+
+			/* Ensure the lv is set up according to ->iop_size */
+			lv->lv_niovecs = shadow->lv_niovecs;
+
+			/* reset the lv buffer information for new formatting */
+			lv->lv_buf_len = 0;
+			lv->lv_bytes = 0;
+			lv->lv_buf = (char *)lv +
+					xlog_cil_iovec_space(lv->lv_niovecs);
 		} else {
-			/* allocate new data chunk */
-			lv = kmem_zalloc(buf_size, KM_SLEEP|KM_NOFS);
+			/* switch to shadow buffer! */
+			lv = shadow;
 			lv->lv_item = lip;
-			lv->lv_size = buf_size;
 			if (ordered) {
 				/* track as an ordered logvec */
 				ASSERT(lip->li_lv == NULL);
-				lv->lv_buf_len = XFS_LOG_VEC_ORDERED;
 				goto insert;
 			}
-			lv->lv_iovecp = (struct xfs_log_iovec *)&lv[1];
 		}
 
-		/* Ensure the lv is set up according to ->iop_size */
-		lv->lv_niovecs = niovecs;
-
-		/* The allocated data region lies beyond the iovec region */
-		lv->lv_buf_len = 0;
-		lv->lv_bytes = 0;
-		lv->lv_buf = (char *)lv + buf_size - nbytes;
 		ASSERT(IS_ALIGNED((unsigned long)lv->lv_buf, sizeof(uint64_t)));
-
 		lip->li_ops->iop_format(lip, lv);
 insert:
-		ASSERT(lv->lv_buf_len <= nbytes);
 		xfs_cil_prepare_item(log, lv, old_lv, diff_len, diff_iovecs);
 	}
 }
@@ -784,6 +898,13 @@ xfs_log_commit_cil(
 	struct xlog		*log = mp->m_log;
 	struct xfs_cil		*cil = log->l_cilp;
 
+	/*
+	 * Do all necessary memory allocation before we lock the CIL.
+	 * This ensures the allocation does not deadlock with a CIL
+	 * push in memory reclaim (e.g. from kswapd).
+	 */
+	xlog_cil_alloc_shadow_bufs(log, tp);
+
 	/* lock out background commit */
 	down_read(&cil->xc_ctx_lock);
 
diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h
index 4643070..74e6819 100644
--- a/fs/xfs/xfs_trans.h
+++ b/fs/xfs/xfs_trans.h
@@ -52,6 +52,7 @@ typedef struct xfs_log_item {
 	/* delayed logging */
 	struct list_head		li_cil;		/* CIL pointers */
 	struct xfs_log_vec		*li_lv;		/* active log vector */
+	struct xfs_log_vec		*li_lv_shadow;	/* standby vector */
 	xfs_lsn_t			li_seq;		/* CIL commit seq */
 } xfs_log_item_t;
 
-- 
Dave Chinner
david@fromorbit.com

From darrick.wong@oracle.com  Tue Jan 19 23:13:00 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 8F2F57F37
	for ; Tue, 19 Jan 2016 23:13:00 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 3B2E3AC003
	for ; Tue, 19 Jan 2016 21:12:57 -0800 (PST)
X-ASG-Debug-ID: 1453266772-04bdf04d6b068b0001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id 2QCi9wAldW4CwsLt (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 19 Jan 2016 21:12:52 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0K5CoUR027404
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 20 Jan 2016 05:12:51 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0K5Co0g024958
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 20 Jan 2016 05:12:50 GMT
Received: from abhmp0008.oracle.com (abhmp0008.oracle.com [141.146.116.14])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0K5Co55025606;
	Wed, 20 Jan 2016 05:12:50 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Tue, 19 Jan 2016 21:12:49 -0800
Date: Tue, 19 Jan 2016 21:12:48 -0800
From: "Darrick J. Wong" 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
Message-ID: <20160120051248.GA5744@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
References: <20160111232657.GA7831@birch.djwong.org>
 <20160115205801.GB5757@birch.djwong.org>
 <20160115225219.GP6033@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160115225219.GP6033@dastard>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453266772
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26287
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Sat, Jan 16, 2016 at 09:52:19AM +1100, Dave Chinner wrote:
> On Fri, Jan 15, 2016 at 12:58:01PM -0800, Darrick J. Wong wrote:
> > On Mon, Jan 11, 2016 at 03:26:57PM -0800, Darrick J. Wong wrote:
> > > Check the sizes of XFS on-disk structures when compiling the kernel.
> > > Use this to catch inadvertent changes in structure size due to padding
> > > and alignment issues, etc.
> > > 
> > > Signed-off-by: Darrick J. Wong 
> ....
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
> > > +}
> > 
> > Perhaps this huge function ought to hide away in its own file?
> 
> Yes, I think that's a good idea, and I think it should only be built
> on debug builds, too.

Ok, I'll make it go away when !CONFIG_XFS_DEBUG.

--D

> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

From mhocko@suse.cz  Wed Jan 20 02:48:01 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 2FDD17F37
	for ; Wed, 20 Jan 2016 02:48:01 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E51778F8033
	for ; Wed, 20 Jan 2016 00:47:57 -0800 (PST)
X-ASG-Debug-ID: 1453279671-04bdf04d6c0b160001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id DVinrovMapiAG4ag (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 20 Jan 2016 00:47:52 -0800 (PST)
X-Barracuda-Envelope-From: mhocko@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id F4125ABE5;
	Wed, 20 Jan 2016 08:47:49 +0000 (UTC)
Date: Wed, 20 Jan 2016 09:47:50 +0100
From: Michal Hocko 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, jkosina@suse.cz,
	Hendrik Woltersdorf 
Subject: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Message-ID: <20160120084750.GA14187@dhcp22.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
References: <1452661968-11482-1-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452661968-11482-1-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453279672
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26291
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed 13-01-16 16:12:48, Dave Chinner wrote:
> This reverts commit 24ba16bb3d499c49974669cd8429c3e4138ab102 as it
> prevents machines from suspending. This regression occurs when the
> xfsaild is idle on entry to suspend, and so there s no activity to
> wake it from it's idle sleep and hence see that it is supposed to
> freeze. Hence the freezer times out waiting for it and suspend is
> cancelled.
> 
> There is no obvious fix for this short of freezing the filesystem
> properly, so revert this change for now.

We had a similar report opensuse bugzilla just recently. I believe the
proper fix should be the following:
---
>From ae910a86ada86804c34cc8136afebc9fefa15813 Mon Sep 17 00:00:00 2001
From: Michal Hocko 
Date: Tue, 19 Jan 2016 20:28:49 +0100
Subject: [PATCH] xfs, xfsaild: Do not block suspend

Hendik has reported suspend failures due to xfsaild blocking the freezer
to settle down.
Jan 17 19:59:56 linux-6380 kernel: PM: Syncing filesystems ... done.
Jan 17 19:59:56 linux-6380 kernel: PM: Preparing system for sleep (mem)
Jan 17 19:59:56 linux-6380 kernel: Freezing user space processes ... (elapsed 0.001 seconds) done.
Jan 17 19:59:56 linux-6380 kernel: Freezing remaining freezable tasks ...
Jan 17 19:59:56 linux-6380 kernel: Freezing of tasks failed after 20.002 seconds (1 tasks refusing to freeze, wq_busy=0):
Jan 17 19:59:56 linux-6380 kernel: xfsaild/dm-5    S 00000000     0  1293      2 0x00000080
Jan 17 19:59:56 linux-6380 kernel:  f0ef5f00 00000046 00000200 00000000 ffff9022 c02d3800 00000000 00000032
Jan 17 19:59:56 linux-6380 kernel:  ee0b2400 00000032 f71e0d00 f36fabc0 f0ef2d00 f0ef6000 f0ef2d00 f12f90c0
Jan 17 19:59:56 linux-6380 kernel:  f0ef5f0c c0844e44 00000000 f0ef5f6c f811e0be 00000000 00000000 f0ef2d00
Jan 17 19:59:56 linux-6380 kernel: Call Trace:
Jan 17 19:59:56 linux-6380 kernel:  [] schedule+0x34/0x90
Jan 17 19:59:56 linux-6380 kernel:  [] xfsaild+0x5de/0x600 [xfs]
Jan 17 19:59:56 linux-6380 kernel:  [] kthread+0x9b/0xb0
Jan 17 19:59:56 linux-6380 kernel:  [] ret_from_kernel_thread+0x21/0x38

The issue has been there for quite some time but it has been made
visible by only by 24ba16bb3d49 ("xfs: clear PF_NOFREEZE for xfsaild
kthread") because the suspend started seeing xfsaild.

The above commit has missed that the !xfs_ail_min branch might call
schedule with TASK_INTERRUPTIBLE without calling try_to_freeze so the pm
suspend would wake up the kernel thread over and over again without any
progress. What we want here is to use freezable_schedule instead to hide
the thread from the suspend.

While we are here also change schedule_timeout to freezable variant to
prevent from spurious wakeups by suspend.

Reported-by: Hendrik Woltersdorf 
Signed-off-by: Michal Hocko 
---
 fs/xfs/xfs_trans_ail.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
index aa67339b9537..d6c9c3e9e02b 100644
--- a/fs/xfs/xfs_trans_ail.c
+++ b/fs/xfs/xfs_trans_ail.c
@@ -520,14 +520,14 @@ xfsaild(
 		if (!xfs_ail_min(ailp) &&
 		    ailp->xa_target == ailp->xa_target_prev) {
 			spin_unlock(&ailp->xa_lock);
-			schedule();
+			freezable_schedule();
 			tout = 0;
 			continue;
 		}
 		spin_unlock(&ailp->xa_lock);
 
 		if (tout)
-			schedule_timeout(msecs_to_jiffies(tout));
+			freezable_schedule_timeout(msecs_to_jiffies(tout));
 
 		__set_current_state(TASK_RUNNING);
 
-- 
2.7.0.rc3

-- 
Michal Hocko
SUSE Labs

From cmaiolino@redhat.com  Wed Jan 20 09:18:56 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 1E9787F37
	for ; Wed, 20 Jan 2016 09:18:56 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 10C7130405F
	for ; Wed, 20 Jan 2016 07:18:55 -0800 (PST)
X-ASG-Debug-ID: 1453303133-04cb6c0d6014730001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id B2xXIMnDDWKEsLsX (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 20 Jan 2016 07:18:54 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	by mx1.redhat.com (Postfix) with ESMTPS id 6E8AF8E74E
	for ; Wed, 20 Jan 2016 15:18:53 +0000 (UTC)
Received: from zion.usersys.redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0KFIqDh002799
	for ; Wed, 20 Jan 2016 10:18:52 -0500
From: Carlos Maiolino 
To: xfs@oss.sgi.com
Subject: [PATCH] Split default quota limits by quota type
Date: Wed, 20 Jan 2016 16:18:47 +0100
X-ASG-Orig-Subj: [PATCH] Split default quota limits by quota type
Message-Id: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453303134
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

Default quotas are globally set due historical reasons. IRIX only supported user
and project quotas, and default quota was only applied to user quotas.

In Linux, when a default quota is set, all different quota types inherits the
same default value.

An user with a quota limit larger than the default quota value, will still be
limited to the default value because the group quotas also inherits the default
quotas. Unless the group which the user belongs to have a custom quota limit
set.

This patch aims to split the default quota value by quota type. Allowing each
quota type having different default values.

Default time limits are still set globally, but I don't mind to split them by
quota type too.

Signed-off-by: Carlos Maiolino 
---
 fs/xfs/xfs_dquot.c       | 26 +++++++------
 fs/xfs/xfs_qm.c          | 96 +++++++++++++++++++++++++++---------------------
 fs/xfs/xfs_qm.h          | 34 ++++++++++++++---
 fs/xfs/xfs_qm_syscalls.c | 15 +++++---
 fs/xfs/xfs_trans_dquot.c | 15 +++++---
 5 files changed, 114 insertions(+), 72 deletions(-)

diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 9c44d38..23f551b 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -92,26 +92,28 @@ xfs_qm_adjust_dqlimits(
 {
 	struct xfs_quotainfo	*q = mp->m_quotainfo;
 	struct xfs_disk_dquot	*d = &dq->q_core;
+	struct xfs_def_quota	*defq;
 	int			prealloc = 0;
 
 	ASSERT(d->d_id);
+	defq = xfs_get_defquota(dq, q);
 
-	if (q->qi_bsoftlimit && !d->d_blk_softlimit) {
-		d->d_blk_softlimit = cpu_to_be64(q->qi_bsoftlimit);
+	if (defq->bsoftlimit && !d->d_blk_softlimit) {
+		d->d_blk_softlimit = cpu_to_be64(defq->bsoftlimit);
 		prealloc = 1;
 	}
-	if (q->qi_bhardlimit && !d->d_blk_hardlimit) {
-		d->d_blk_hardlimit = cpu_to_be64(q->qi_bhardlimit);
+	if (defq->bhardlimit && !d->d_blk_hardlimit) {
+		d->d_blk_hardlimit = cpu_to_be64(defq->bhardlimit);
 		prealloc = 1;
 	}
-	if (q->qi_isoftlimit && !d->d_ino_softlimit)
-		d->d_ino_softlimit = cpu_to_be64(q->qi_isoftlimit);
-	if (q->qi_ihardlimit && !d->d_ino_hardlimit)
-		d->d_ino_hardlimit = cpu_to_be64(q->qi_ihardlimit);
-	if (q->qi_rtbsoftlimit && !d->d_rtb_softlimit)
-		d->d_rtb_softlimit = cpu_to_be64(q->qi_rtbsoftlimit);
-	if (q->qi_rtbhardlimit && !d->d_rtb_hardlimit)
-		d->d_rtb_hardlimit = cpu_to_be64(q->qi_rtbhardlimit);
+	if (defq->isoftlimit && !d->d_ino_softlimit)
+		d->d_ino_softlimit = cpu_to_be64(defq->isoftlimit);
+	if (defq->ihardlimit && !d->d_ino_hardlimit)
+		d->d_ino_hardlimit = cpu_to_be64(defq->ihardlimit);
+	if (defq->rtbsoftlimit && !d->d_rtb_softlimit)
+		d->d_rtb_softlimit = cpu_to_be64(defq->rtbsoftlimit);
+	if (defq->rtbhardlimit && !d->d_rtb_hardlimit)
+		d->d_rtb_hardlimit = cpu_to_be64(defq->rtbhardlimit);
 
 	if (prealloc)
 		xfs_dquot_set_prealloc_limits(dq);
diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
index 532ab79..1bcb733 100644
--- a/fs/xfs/xfs_qm.c
+++ b/fs/xfs/xfs_qm.c
@@ -560,6 +560,54 @@ xfs_qm_shrink_count(
 	return list_lru_shrink_count(&qi->qi_lru, sc);
 }
 
+STATIC int
+xfs_qm_set_defquota(
+	xfs_mount_t	*mp,
+	uint		type,
+	xfs_quotainfo_t	*qinf)
+{
+	xfs_dquot_t		*dqp;
+	struct xfs_def_quota    *defq;
+	int			error;
+
+	error = xfs_qm_dqread(mp, 0, type, XFS_QMOPT_DOWARN, &dqp);
+
+	if (!error) {
+		xfs_disk_dquot_t        *ddqp = &dqp->q_core;
+
+		defq = xfs_get_defquota(dqp, qinf);
+
+		qinf->qi_btimelimit = ddqp->d_btimer ?
+			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
+		qinf->qi_itimelimit = ddqp->d_itimer ?
+			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
+		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
+			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
+		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
+			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
+		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
+			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
+		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
+			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
+		defq->bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
+		defq->bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
+		defq->ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
+		defq->isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
+		defq->rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
+		defq->rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
+		xfs_qm_dqdestroy(dqp);
+	} else {
+		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
+		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
+		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
+		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
+		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
+		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
+	}
+
+	return error;
+}
+
 /*
  * This initializes all the quota information that's kept in the
  * mount structure
@@ -570,7 +618,6 @@ xfs_qm_init_quotainfo(
 {
 	xfs_quotainfo_t *qinf;
 	int		error;
-	xfs_dquot_t	*dqp;
 
 	ASSERT(XFS_IS_QUOTA_RUNNING(mp));
 
@@ -614,47 +661,12 @@ xfs_qm_init_quotainfo(
 	 * Since we may not have done a quotacheck by this point, just read
 	 * the dquot without attaching it to any hashtables or lists.
 	 */
-	error = xfs_qm_dqread(mp, 0,
-			XFS_IS_UQUOTA_RUNNING(mp) ? XFS_DQ_USER :
-			 (XFS_IS_GQUOTA_RUNNING(mp) ? XFS_DQ_GROUP :
-			  XFS_DQ_PROJ),
-			XFS_QMOPT_DOWARN, &dqp);
-	if (!error) {
-		xfs_disk_dquot_t	*ddqp = &dqp->q_core;
-
-		/*
-		 * The warnings and timers set the grace period given to
-		 * a user or group before he or she can not perform any
-		 * more writing. If it is zero, a default is used.
-		 */
-		qinf->qi_btimelimit = ddqp->d_btimer ?
-			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
-		qinf->qi_itimelimit = ddqp->d_itimer ?
-			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
-		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
-			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
-		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
-			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
-		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
-			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
-		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
-			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
-		qinf->qi_bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
-		qinf->qi_bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
-		qinf->qi_ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
-		qinf->qi_isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
-		qinf->qi_rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
-		qinf->qi_rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
-
-		xfs_qm_dqdestroy(dqp);
-	} else {
-		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
-		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
-		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
-		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
-		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
-		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
-	}
+	if (XFS_IS_UQUOTA_RUNNING(mp))
+		error = xfs_qm_set_defquota(mp, XFS_DQ_USER, qinf);
+	if (XFS_IS_GQUOTA_RUNNING(mp))
+		error = xfs_qm_set_defquota(mp, XFS_DQ_GROUP, qinf);
+	if (XFS_IS_PQUOTA_RUNNING(mp))
+		error = xfs_qm_set_defquota(mp, XFS_DQ_PROJ, qinf);
 
 	qinf->qi_shrinker.count_objects = xfs_qm_shrink_count;
 	qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan;
diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
index 996a040..45e2c36 100644
--- a/fs/xfs/xfs_qm.h
+++ b/fs/xfs/xfs_qm.h
@@ -53,6 +53,15 @@ extern struct kmem_zone	*xfs_qm_dqtrxzone;
  */
 #define XFS_DQUOT_CLUSTER_SIZE_FSB	(xfs_filblks_t)1
 
+struct xfs_def_quota {
+	xfs_qcnt_t       bhardlimit;     /* default data blk hard limit */
+	xfs_qcnt_t       bsoftlimit;	 /* default data blk soft limit */
+	xfs_qcnt_t       ihardlimit;	 /* default inode count hard limit */
+	xfs_qcnt_t       isoftlimit;	 /* default inode count soft limit */
+	xfs_qcnt_t	 rtbhardlimit;   /* default realtime blk hard limit */
+	xfs_qcnt_t	 rtbsoftlimit;   /* default realtime blk soft limit */
+};
+
 /*
  * Various quota information for individual filesystems.
  * The mount structure keeps a pointer to this.
@@ -76,12 +85,9 @@ typedef struct xfs_quotainfo {
 	struct mutex	 qi_quotaofflock;/* to serialize quotaoff */
 	xfs_filblks_t	 qi_dqchunklen;	 /* # BBs in a chunk of dqs */
 	uint		 qi_dqperchunk;	 /* # ondisk dqs in above chunk */
-	xfs_qcnt_t	 qi_bhardlimit;	 /* default data blk hard limit */
-	xfs_qcnt_t	 qi_bsoftlimit;	 /* default data blk soft limit */
-	xfs_qcnt_t	 qi_ihardlimit;	 /* default inode count hard limit */
-	xfs_qcnt_t	 qi_isoftlimit;	 /* default inode count soft limit */
-	xfs_qcnt_t	 qi_rtbhardlimit;/* default realtime blk hard limit */
-	xfs_qcnt_t	 qi_rtbsoftlimit;/* default realtime blk soft limit */
+	struct xfs_def_quota	qi_usr_default;
+	struct xfs_def_quota	qi_grp_default;
+	struct xfs_def_quota	qi_prj_default;
 	struct shrinker  qi_shrinker;
 } xfs_quotainfo_t;
 
@@ -171,4 +177,20 @@ extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
 extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
 extern int		xfs_qm_scall_quotaoff(struct xfs_mount *, uint);
 
+static inline struct xfs_def_quota *
+xfs_get_defquota(struct xfs_dquot *dqp, struct xfs_quotainfo *qi)
+{
+	struct xfs_def_quota *defq;
+
+	if (XFS_QM_ISUDQ(dqp))
+		defq = &qi->qi_usr_default;
+	else if (XFS_QM_ISGDQ(dqp))
+		defq = &qi->qi_grp_default;
+	else {
+		ASSERT(XFS_QM_ISPDQ(dqp));
+		defq = &qi->qi_prj_default;
+	}
+	return defq;
+}
+
 #endif /* __XFS_QM_H__ */
diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
index 3640c6e..31830f0 100644
--- a/fs/xfs/xfs_qm_syscalls.c
+++ b/fs/xfs/xfs_qm_syscalls.c
@@ -404,6 +404,7 @@ xfs_qm_scall_setqlim(
 	struct xfs_disk_dquot	*ddq;
 	struct xfs_dquot	*dqp;
 	struct xfs_trans	*tp;
+	struct xfs_def_quota	*defq;
 	int			error;
 	xfs_qcnt_t		hard, soft;
 
@@ -431,6 +432,8 @@ xfs_qm_scall_setqlim(
 		ASSERT(error != -ENOENT);
 		goto out_unlock;
 	}
+
+	defq = xfs_get_defquota(dqp, q);
 	xfs_dqunlock(dqp);
 
 	tp = xfs_trans_alloc(mp, XFS_TRANS_QM_SETQLIM);
@@ -458,8 +461,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_blk_softlimit = cpu_to_be64(soft);
 		xfs_dquot_set_prealloc_limits(dqp);
 		if (id == 0) {
-			q->qi_bhardlimit = hard;
-			q->qi_bsoftlimit = soft;
+			defq->bhardlimit = hard;
+			defq->bsoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "blkhard %Ld < blksoft %Ld", hard, soft);
@@ -474,8 +477,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_rtb_hardlimit = cpu_to_be64(hard);
 		ddq->d_rtb_softlimit = cpu_to_be64(soft);
 		if (id == 0) {
-			q->qi_rtbhardlimit = hard;
-			q->qi_rtbsoftlimit = soft;
+			defq->rtbhardlimit = hard;
+			defq->rtbsoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "rtbhard %Ld < rtbsoft %Ld", hard, soft);
@@ -491,8 +494,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_ino_hardlimit = cpu_to_be64(hard);
 		ddq->d_ino_softlimit = cpu_to_be64(soft);
 		if (id == 0) {
-			q->qi_ihardlimit = hard;
-			q->qi_isoftlimit = soft;
+			defq->ihardlimit = hard;
+			defq->isoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "ihard %Ld < isoft %Ld", hard, soft);
diff --git a/fs/xfs/xfs_trans_dquot.c b/fs/xfs/xfs_trans_dquot.c
index 9951701..c3d5472 100644
--- a/fs/xfs/xfs_trans_dquot.c
+++ b/fs/xfs/xfs_trans_dquot.c
@@ -609,17 +609,20 @@ xfs_trans_dqresv(
 	xfs_qcnt_t	total_count;
 	xfs_qcnt_t	*resbcountp;
 	xfs_quotainfo_t	*q = mp->m_quotainfo;
+	struct xfs_def_quota	*defq;
 
 
 	xfs_dqlock(dqp);
 
+	defq = xfs_get_defquota(dqp, q);
+
 	if (flags & XFS_TRANS_DQ_RES_BLKS) {
 		hardlimit = be64_to_cpu(dqp->q_core.d_blk_hardlimit);
 		if (!hardlimit)
-			hardlimit = q->qi_bhardlimit;
+			hardlimit = defq->bhardlimit;
 		softlimit = be64_to_cpu(dqp->q_core.d_blk_softlimit);
 		if (!softlimit)
-			softlimit = q->qi_bsoftlimit;
+			softlimit = defq->bsoftlimit;
 		timer = be32_to_cpu(dqp->q_core.d_btimer);
 		warns = be16_to_cpu(dqp->q_core.d_bwarns);
 		warnlimit = dqp->q_mount->m_quotainfo->qi_bwarnlimit;
@@ -628,10 +631,10 @@ xfs_trans_dqresv(
 		ASSERT(flags & XFS_TRANS_DQ_RES_RTBLKS);
 		hardlimit = be64_to_cpu(dqp->q_core.d_rtb_hardlimit);
 		if (!hardlimit)
-			hardlimit = q->qi_rtbhardlimit;
+			hardlimit = defq->rtbhardlimit;
 		softlimit = be64_to_cpu(dqp->q_core.d_rtb_softlimit);
 		if (!softlimit)
-			softlimit = q->qi_rtbsoftlimit;
+			softlimit = defq->rtbsoftlimit;
 		timer = be32_to_cpu(dqp->q_core.d_rtbtimer);
 		warns = be16_to_cpu(dqp->q_core.d_rtbwarns);
 		warnlimit = dqp->q_mount->m_quotainfo->qi_rtbwarnlimit;
@@ -672,10 +675,10 @@ xfs_trans_dqresv(
 			warnlimit = dqp->q_mount->m_quotainfo->qi_iwarnlimit;
 			hardlimit = be64_to_cpu(dqp->q_core.d_ino_hardlimit);
 			if (!hardlimit)
-				hardlimit = q->qi_ihardlimit;
+				hardlimit = defq->ihardlimit;
 			softlimit = be64_to_cpu(dqp->q_core.d_ino_softlimit);
 			if (!softlimit)
-				softlimit = q->qi_isoftlimit;
+				softlimit = defq->isoftlimit;
 
 			if (hardlimit && total_count > hardlimit) {
 				xfs_quota_warn(mp, dqp, QUOTA_NL_IHARDWARN);
-- 
2.4.3

From jeffpc@josefsipek.net  Wed Jan 20 09:40:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 0ED8D7F37
	for ; Wed, 20 Jan 2016 09:40:41 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D86608F804C
	for ; Wed, 20 Jan 2016 07:40:37 -0800 (PST)
X-ASG-Debug-ID: 1453304434-04bdf04d6b16890001-NocioJ
Received: from josefsipek.net (josefsipek.net [71.174.113.7]) by cuda.sgi.com with ESMTP id vs33or1nXAaLkNgI for ; Wed, 20 Jan 2016 07:40:34 -0800 (PST)
X-Barracuda-Envelope-From: jeffpc@josefsipek.net
X-Barracuda-Apparent-Source-IP: 71.174.113.7
Received: from meili.valhalla.31bits.net (meili.valhalla.31bits.net [172.27.0.10])
	by josefsipek.net (Postfix) with ESMTPSA id 17DE4554D8;
	Wed, 20 Jan 2016 10:40:34 -0500 (EST)
Date: Wed, 20 Jan 2016 10:40:36 -0500
From: Josef 'Jeff' Sipek 
To: Dave Chinner 
Cc: "Darrick J. Wong" , xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
Message-ID: <20160120154035.GH1366@meili.valhalla.31bits.net>
X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
References: <20160111232657.GA7831@birch.djwong.org>
 <20160115205801.GB5757@birch.djwong.org>
 <20160115225219.GP6033@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160115225219.GP6033@dastard>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: josefsipek.net[71.174.113.7]
X-Barracuda-Start-Time: 1453304434
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26298
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Sat, Jan 16, 2016 at 09:52:19AM +1100, Dave Chinner wrote:
> On Fri, Jan 15, 2016 at 12:58:01PM -0800, Darrick J. Wong wrote:
> > On Mon, Jan 11, 2016 at 03:26:57PM -0800, Darrick J. Wong wrote:
> > > Check the sizes of XFS on-disk structures when compiling the kernel.
> > > Use this to catch inadvertent changes in structure size due to padding
> > > and alignment issues, etc.
> > > 
> > > Signed-off-by: Darrick J. Wong 
> ....
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
> > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
> > > +}
> > 
> > Perhaps this huge function ought to hide away in its own file?
> 
> Yes, I think that's a good idea, and I think it should only be built
> on debug builds, too.

Doesn't this turn into nothingness as far as the generated binary is
concerned?  Because if it does, I don't see a reason to keep it debug-only.
Afterall, non-debug builds need to be correct and these are pretty important
checks.  We certainly don't want another ARM structure padding & alignment
fiasco.

Just my 2 cents,

Jeff.

-- 
Only two things are infinite, the universe and human stupidity, and I'm not
sure about the former.
		- Albert Einstein

From sandeen@sandeen.net  Wed Jan 20 11:30:15 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 92B707F37
	for ; Wed, 20 Jan 2016 11:30:15 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 806A530405F
	for ; Wed, 20 Jan 2016 09:30:11 -0800 (PST)
X-ASG-Debug-ID: 1453311006-04cb6c0d5e182d0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 9kRFORuB0rn34RVp for ; Wed, 20 Jan 2016 09:30:07 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 83FF165F01BC
	for ; Wed, 20 Jan 2016 11:30:06 -0600 (CST)
Subject: Re: [PATCH] Split default quota limits by quota type
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] Split default quota limits by quota type
References: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <569FC41E.2040300@sandeen.net>
Date: Wed, 20 Jan 2016 11:30:06 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453311007
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26300
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/20/16 9:18 AM, Carlos Maiolino wrote:
> Default quotas are globally set due historical reasons. IRIX only supported user
> and project quotas, and default quota was only applied to user quotas.
> 
> In Linux, when a default quota is set, all different quota types inherits the
> same default value.
> 
> An user with a quota limit larger than the default quota value, will still be
> limited to the default value because the group quotas also inherits the default
> quotas. Unless the group which the user belongs to have a custom quota limit
> set.
> 
> This patch aims to split the default quota value by quota type. Allowing each
> quota type having different default values.
> 
> Default time limits are still set globally, but I don't mind to split them by
> quota type too.

Hm, I guess it seems like it should be done; otherwise it's a weird
caveat, isn't it?  "Default limits are set by type, but timers are
inherited from whatever first default quota is found across all types"

So yeah, seems like it should be done for timers as well, IMHO;
grace periods can be set for each default quota type, so they should
be honored.

> Signed-off-by: Carlos Maiolino 

Really just minor comments below; there is one code comment that needs to
be removed.

Thanks!
-Eric

> ---
>  fs/xfs/xfs_dquot.c       | 26 +++++++------
>  fs/xfs/xfs_qm.c          | 96 +++++++++++++++++++++++++++---------------------
>  fs/xfs/xfs_qm.h          | 34 ++++++++++++++---
>  fs/xfs/xfs_qm_syscalls.c | 15 +++++---
>  fs/xfs/xfs_trans_dquot.c | 15 +++++---
>  5 files changed, 114 insertions(+), 72 deletions(-)
> 
> diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
> index 9c44d38..23f551b 100644
> --- a/fs/xfs/xfs_dquot.c
> +++ b/fs/xfs/xfs_dquot.c
> @@ -92,26 +92,28 @@ xfs_qm_adjust_dqlimits(
>  {
>  	struct xfs_quotainfo	*q = mp->m_quotainfo;
>  	struct xfs_disk_dquot	*d = &dq->q_core;
> +	struct xfs_def_quota	*defq;
>  	int			prealloc = 0;
>  
>  	ASSERT(d->d_id);
> +	defq = xfs_get_defquota(dq, q);
>  
> -	if (q->qi_bsoftlimit && !d->d_blk_softlimit) {
> -		d->d_blk_softlimit = cpu_to_be64(q->qi_bsoftlimit);
> +	if (defq->bsoftlimit && !d->d_blk_softlimit) {
> +		d->d_blk_softlimit = cpu_to_be64(defq->bsoftlimit);
>  		prealloc = 1;
>  	}
> -	if (q->qi_bhardlimit && !d->d_blk_hardlimit) {
> -		d->d_blk_hardlimit = cpu_to_be64(q->qi_bhardlimit);
> +	if (defq->bhardlimit && !d->d_blk_hardlimit) {
> +		d->d_blk_hardlimit = cpu_to_be64(defq->bhardlimit);
>  		prealloc = 1;
>  	}
> -	if (q->qi_isoftlimit && !d->d_ino_softlimit)
> -		d->d_ino_softlimit = cpu_to_be64(q->qi_isoftlimit);
> -	if (q->qi_ihardlimit && !d->d_ino_hardlimit)
> -		d->d_ino_hardlimit = cpu_to_be64(q->qi_ihardlimit);
> -	if (q->qi_rtbsoftlimit && !d->d_rtb_softlimit)
> -		d->d_rtb_softlimit = cpu_to_be64(q->qi_rtbsoftlimit);
> -	if (q->qi_rtbhardlimit && !d->d_rtb_hardlimit)
> -		d->d_rtb_hardlimit = cpu_to_be64(q->qi_rtbhardlimit);
> +	if (defq->isoftlimit && !d->d_ino_softlimit)
> +		d->d_ino_softlimit = cpu_to_be64(defq->isoftlimit);
> +	if (defq->ihardlimit && !d->d_ino_hardlimit)
> +		d->d_ino_hardlimit = cpu_to_be64(defq->ihardlimit);
> +	if (defq->rtbsoftlimit && !d->d_rtb_softlimit)
> +		d->d_rtb_softlimit = cpu_to_be64(defq->rtbsoftlimit);
> +	if (defq->rtbhardlimit && !d->d_rtb_hardlimit)
> +		d->d_rtb_hardlimit = cpu_to_be64(defq->rtbhardlimit);
>  
>  	if (prealloc)
>  		xfs_dquot_set_prealloc_limits(dq);
> diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
> index 532ab79..1bcb733 100644
> --- a/fs/xfs/xfs_qm.c
> +++ b/fs/xfs/xfs_qm.c
> @@ -560,6 +560,54 @@ xfs_qm_shrink_count(
>  	return list_lru_shrink_count(&qi->qi_lru, sc);
>  }
>  
> +STATIC int
> +xfs_qm_set_defquota(

It'd be a little easier to review if this were simply factored
out in patch 1, then default quotas handled in patch 2, I think,
but not that big a deal.

> +	xfs_mount_t	*mp,
> +	uint		type,
> +	xfs_quotainfo_t	*qinf)
> +{
> +	xfs_dquot_t		*dqp;
> +	struct xfs_def_quota    *defq;
> +	int			error;
> +
> +	error = xfs_qm_dqread(mp, 0, type, XFS_QMOPT_DOWARN, &dqp);
> +
> +	if (!error) {
> +		xfs_disk_dquot_t        *ddqp = &dqp->q_core;
> +
> +		defq = xfs_get_defquota(dqp, qinf);
> +
> +		qinf->qi_btimelimit = ddqp->d_btimer ?
> +			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
> +		qinf->qi_itimelimit = ddqp->d_itimer ?
> +			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
> +		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
> +			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
> +		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
> +			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
> +		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
> +			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
> +		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
> +			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
> +		defq->bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
> +		defq->bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
> +		defq->ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
> +		defq->isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
> +		defq->rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
> +		defq->rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
> +		xfs_qm_dqdestroy(dqp);
> +	} else {
> +		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
> +		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
> +		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
> +		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
> +		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
> +		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
> +	}
> +
> +	return error;
> +}
> +
>  /*
>   * This initializes all the quota information that's kept in the
>   * mount structure
> @@ -570,7 +618,6 @@ xfs_qm_init_quotainfo(
>  {
>  	xfs_quotainfo_t *qinf;
>  	int		error;
> -	xfs_dquot_t	*dqp;
>  
>  	ASSERT(XFS_IS_QUOTA_RUNNING(mp));
>  
> @@ -614,47 +661,12 @@ xfs_qm_init_quotainfo(

There is a comment above this that should go away now:

         * We look at the USR dquot with id == 0 first, but if user quotas
         * are not enabled we goto the GRP dquot with id == 0.
         * We don't really care to keep separate default limits for user
         * and group quotas, at least not at this point.


>  	 * Since we may not have done a quotacheck by this point, just read
>  	 * the dquot without attaching it to any hashtables or lists.
>  	 */
> -	error = xfs_qm_dqread(mp, 0,
> -			XFS_IS_UQUOTA_RUNNING(mp) ? XFS_DQ_USER :
> -			 (XFS_IS_GQUOTA_RUNNING(mp) ? XFS_DQ_GROUP :
> -			  XFS_DQ_PROJ),
> -			XFS_QMOPT_DOWARN, &dqp);
> -	if (!error) {
> -		xfs_disk_dquot_t	*ddqp = &dqp->q_core;
> -
> -		/*
> -		 * The warnings and timers set the grace period given to
> -		 * a user or group before he or she can not perform any
> -		 * more writing. If it is zero, a default is used.
> -		 */
> -		qinf->qi_btimelimit = ddqp->d_btimer ?
> -			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
> -		qinf->qi_itimelimit = ddqp->d_itimer ?
> -			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
> -		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
> -			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
> -		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
> -			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
> -		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
> -			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
> -		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
> -			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
> -		qinf->qi_bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
> -		qinf->qi_bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
> -		qinf->qi_ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
> -		qinf->qi_isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
> -		qinf->qi_rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
> -		qinf->qi_rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
> -
> -		xfs_qm_dqdestroy(dqp);
> -	} else {
> -		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
> -		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
> -		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
> -		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
> -		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
> -		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
> -	}



> +	if (XFS_IS_UQUOTA_RUNNING(mp))
> +		error = xfs_qm_set_defquota(mp, XFS_DQ_USER, qinf);

Hm, I guess the rest of the function looks at RUNNING not ENFORCED;
for some reason I thought it would be ENFORCED, but *shrug* I guess
it's right!

> +	if (XFS_IS_GQUOTA_RUNNING(mp))
> +		error = xfs_qm_set_defquota(mp, XFS_DQ_GROUP, qinf);
> +	if (XFS_IS_PQUOTA_RUNNING(mp))
> +		error = xfs_qm_set_defquota(mp, XFS_DQ_PROJ, qinf);
>  
>  	qinf->qi_shrinker.count_objects = xfs_qm_shrink_count;
>  	qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan;
> diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
> index 996a040..45e2c36 100644
> --- a/fs/xfs/xfs_qm.h
> +++ b/fs/xfs/xfs_qm.h
> @@ -53,6 +53,15 @@ extern struct kmem_zone	*xfs_qm_dqtrxzone;
>   */
>  #define XFS_DQUOT_CLUSTER_SIZE_FSB	(xfs_filblks_t)1
>  
> +struct xfs_def_quota {
> +	xfs_qcnt_t       bhardlimit;     /* default data blk hard limit */
> +	xfs_qcnt_t       bsoftlimit;	 /* default data blk soft limit */
> +	xfs_qcnt_t       ihardlimit;	 /* default inode count hard limit */
> +	xfs_qcnt_t       isoftlimit;	 /* default inode count soft limit */
> +	xfs_qcnt_t	 rtbhardlimit;   /* default realtime blk hard limit */
> +	xfs_qcnt_t	 rtbsoftlimit;   /* default realtime blk soft limit */
> +};
> +
>  /*
>   * Various quota information for individual filesystems.
>   * The mount structure keeps a pointer to this.
> @@ -76,12 +85,9 @@ typedef struct xfs_quotainfo {
>  	struct mutex	 qi_quotaofflock;/* to serialize quotaoff */
>  	xfs_filblks_t	 qi_dqchunklen;	 /* # BBs in a chunk of dqs */
>  	uint		 qi_dqperchunk;	 /* # ondisk dqs in above chunk */
> -	xfs_qcnt_t	 qi_bhardlimit;	 /* default data blk hard limit */
> -	xfs_qcnt_t	 qi_bsoftlimit;	 /* default data blk soft limit */
> -	xfs_qcnt_t	 qi_ihardlimit;	 /* default inode count hard limit */
> -	xfs_qcnt_t	 qi_isoftlimit;	 /* default inode count soft limit */
> -	xfs_qcnt_t	 qi_rtbhardlimit;/* default realtime blk hard limit */
> -	xfs_qcnt_t	 qi_rtbsoftlimit;/* default realtime blk soft limit */
> +	struct xfs_def_quota	qi_usr_default;
> +	struct xfs_def_quota	qi_grp_default;
> +	struct xfs_def_quota	qi_prj_default;
>  	struct shrinker  qi_shrinker;
>  } xfs_quotainfo_t;
>  
> @@ -171,4 +177,20 @@ extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
>  extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
>  extern int		xfs_qm_scall_quotaoff(struct xfs_mount *, uint);
>  
> +static inline struct xfs_def_quota *
> +xfs_get_defquota(struct xfs_dquot *dqp, struct xfs_quotainfo *qi)
> +{
> +	struct xfs_def_quota *defq;
> +
> +	if (XFS_QM_ISUDQ(dqp))
> +		defq = &qi->qi_usr_default;
> +	else if (XFS_QM_ISGDQ(dqp))
> +		defq = &qi->qi_grp_default;
> +	else {
> +		ASSERT(XFS_QM_ISPDQ(dqp));
> +		defq = &qi->qi_prj_default;
> +	}
> +	return defq;
> +}
> +
>  #endif /* __XFS_QM_H__ */
> diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
> index 3640c6e..31830f0 100644
> --- a/fs/xfs/xfs_qm_syscalls.c
> +++ b/fs/xfs/xfs_qm_syscalls.c
> @@ -404,6 +404,7 @@ xfs_qm_scall_setqlim(
>  	struct xfs_disk_dquot	*ddq;
>  	struct xfs_dquot	*dqp;
>  	struct xfs_trans	*tp;
> +	struct xfs_def_quota	*defq;
>  	int			error;
>  	xfs_qcnt_t		hard, soft;
>  
> @@ -431,6 +432,8 @@ xfs_qm_scall_setqlim(
>  		ASSERT(error != -ENOENT);
>  		goto out_unlock;
>  	}
> +
> +	defq = xfs_get_defquota(dqp, q);
>  	xfs_dqunlock(dqp);
>  
>  	tp = xfs_trans_alloc(mp, XFS_TRANS_QM_SETQLIM);
> @@ -458,8 +461,8 @@ xfs_qm_scall_setqlim(
>  		ddq->d_blk_softlimit = cpu_to_be64(soft);
>  		xfs_dquot_set_prealloc_limits(dqp);
>  		if (id == 0) {
> -			q->qi_bhardlimit = hard;
> -			q->qi_bsoftlimit = soft;
> +			defq->bhardlimit = hard;
> +			defq->bsoftlimit = soft;
>  		}
>  	} else {
>  		xfs_debug(mp, "blkhard %Ld < blksoft %Ld", hard, soft);
> @@ -474,8 +477,8 @@ xfs_qm_scall_setqlim(
>  		ddq->d_rtb_hardlimit = cpu_to_be64(hard);
>  		ddq->d_rtb_softlimit = cpu_to_be64(soft);
>  		if (id == 0) {
> -			q->qi_rtbhardlimit = hard;
> -			q->qi_rtbsoftlimit = soft;
> +			defq->rtbhardlimit = hard;
> +			defq->rtbsoftlimit = soft;
>  		}
>  	} else {
>  		xfs_debug(mp, "rtbhard %Ld < rtbsoft %Ld", hard, soft);
> @@ -491,8 +494,8 @@ xfs_qm_scall_setqlim(
>  		ddq->d_ino_hardlimit = cpu_to_be64(hard);
>  		ddq->d_ino_softlimit = cpu_to_be64(soft);
>  		if (id == 0) {
> -			q->qi_ihardlimit = hard;
> -			q->qi_isoftlimit = soft;
> +			defq->ihardlimit = hard;
> +			defq->isoftlimit = soft;
>  		}
>  	} else {
>  		xfs_debug(mp, "ihard %Ld < isoft %Ld", hard, soft);
> diff --git a/fs/xfs/xfs_trans_dquot.c b/fs/xfs/xfs_trans_dquot.c
> index 9951701..c3d5472 100644
> --- a/fs/xfs/xfs_trans_dquot.c
> +++ b/fs/xfs/xfs_trans_dquot.c
> @@ -609,17 +609,20 @@ xfs_trans_dqresv(
>  	xfs_qcnt_t	total_count;
>  	xfs_qcnt_t	*resbcountp;
>  	xfs_quotainfo_t	*q = mp->m_quotainfo;
> +	struct xfs_def_quota	*defq;
>  
>  
>  	xfs_dqlock(dqp);
>  
> +	defq = xfs_get_defquota(dqp, q);
> +
>  	if (flags & XFS_TRANS_DQ_RES_BLKS) {
>  		hardlimit = be64_to_cpu(dqp->q_core.d_blk_hardlimit);
>  		if (!hardlimit)
> -			hardlimit = q->qi_bhardlimit;
> +			hardlimit = defq->bhardlimit;
>  		softlimit = be64_to_cpu(dqp->q_core.d_blk_softlimit);
>  		if (!softlimit)
> -			softlimit = q->qi_bsoftlimit;
> +			softlimit = defq->bsoftlimit;
>  		timer = be32_to_cpu(dqp->q_core.d_btimer);
>  		warns = be16_to_cpu(dqp->q_core.d_bwarns);
>  		warnlimit = dqp->q_mount->m_quotainfo->qi_bwarnlimit;
> @@ -628,10 +631,10 @@ xfs_trans_dqresv(
>  		ASSERT(flags & XFS_TRANS_DQ_RES_RTBLKS);
>  		hardlimit = be64_to_cpu(dqp->q_core.d_rtb_hardlimit);
>  		if (!hardlimit)
> -			hardlimit = q->qi_rtbhardlimit;
> +			hardlimit = defq->rtbhardlimit;
>  		softlimit = be64_to_cpu(dqp->q_core.d_rtb_softlimit);
>  		if (!softlimit)
> -			softlimit = q->qi_rtbsoftlimit;
> +			softlimit = defq->rtbsoftlimit;
>  		timer = be32_to_cpu(dqp->q_core.d_rtbtimer);
>  		warns = be16_to_cpu(dqp->q_core.d_rtbwarns);
>  		warnlimit = dqp->q_mount->m_quotainfo->qi_rtbwarnlimit;
> @@ -672,10 +675,10 @@ xfs_trans_dqresv(
>  			warnlimit = dqp->q_mount->m_quotainfo->qi_iwarnlimit;
>  			hardlimit = be64_to_cpu(dqp->q_core.d_ino_hardlimit);
>  			if (!hardlimit)
> -				hardlimit = q->qi_ihardlimit;
> +				hardlimit = defq->ihardlimit;
>  			softlimit = be64_to_cpu(dqp->q_core.d_ino_softlimit);
>  			if (!softlimit)
> -				softlimit = q->qi_isoftlimit;
> +				softlimit = defq->isoftlimit;
>  
>  			if (hardlimit && total_count > hardlimit) {
>  				xfs_quota_warn(mp, dqp, QUOTA_NL_IHARDWARN);
> 

From sandeen@sandeen.net  Wed Jan 20 11:35:46 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id C7D0A7F37
	for ; Wed, 20 Jan 2016 11:35:46 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 96FF6304067
	for ; Wed, 20 Jan 2016 09:35:46 -0800 (PST)
X-ASG-Debug-ID: 1453311343-04bdf04d6a1a060001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id hdZfRqfYE19Xk7HI for ; Wed, 20 Jan 2016 09:35:43 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 9243E63CBD0E
	for ; Wed, 20 Jan 2016 11:35:43 -0600 (CST)
Subject: Re: [PATCH] Minor whitespace cleanups
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] Minor whitespace cleanups
References: <1453200369-20171-1-git-send-email-cmaiolino@redhat.com>
From: Eric Sandeen 
Message-ID: <569FC56F.6050902@sandeen.net>
Date: Wed, 20 Jan 2016 11:35:43 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <1453200369-20171-1-git-send-email-cmaiolino@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453311343
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26300
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/19/16 4:46 AM, Carlos Maiolino wrote:
> While investigating a quota issue, I fixed a few whitespaces.
> 
> Signed-off-by: Carlos Maiolino 
> ---
>  include/xqm.h | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/include/xqm.h b/include/xqm.h
> index 47f58a0..697cad4 100644
> --- a/include/xqm.h
> +++ b/include/xqm.h
> @@ -55,14 +55,14 @@ typedef struct fs_disk_quota {
>  	__s32		d_itimer;	/* zero if within inode limits */
>  					/* if not, we refuse service */
>  	__s32		d_btimer;	/* similar to above; for disk blocks */
> -	__u16	  	d_iwarns;       /* # warnings issued wrt num inodes */
> -	__u16	  	d_bwarns;       /* # warnings issued wrt disk blocks */
> +	__u16		d_iwarns;       /* # warnings issued wrt num inodes */
> +	__u16		d_bwarns;       /* # warnings issued wrt disk blocks */
                                 ^^^^^^^

If you're going to fix it up, might as well get the spaces-not-tabs
before the comments on these 2 lines ;)

>  	__s32		d_padding2;	/* padding2 - for future use */
>  	__u64		d_rtb_hardlimit;/* absolute limit on realtime blks */
>  	__u64		d_rtb_softlimit;/* preferred limit on RT disk blks */
>  	__u64		d_rtbcount;	/* # realtime blocks owned */
>  	__s32		d_rtbtimer;	/* similar to above; for RT disk blks */
> -	__u16	  	d_rtbwarns;     /* # warnings issued wrt RT disk blks */
> +	__u16		d_rtbwarns;     /* # warnings issued wrt RT disk blks */

and on this one as well

-Eric

>  	__s16		d_padding3;	/* padding3 - for future use */
>  	char		d_padding4[8];	/* yet more padding */
>  } fs_disk_quota_t;
> @@ -73,7 +73,7 @@ typedef struct fs_disk_quota {
>  #define FS_DQ_ISOFT	(1<<0)
>  #define FS_DQ_IHARD	(1<<1)
>  #define FS_DQ_BSOFT	(1<<2)
> -#define FS_DQ_BHARD 	(1<<3)
> +#define FS_DQ_BHARD	(1<<3)
>  #define FS_DQ_RTBSOFT	(1<<4)
>  #define FS_DQ_RTBHARD	(1<<5)
>  #define FS_DQ_LIMIT_MASK	(FS_DQ_ISOFT | FS_DQ_IHARD | FS_DQ_BSOFT | \
> @@ -87,7 +87,7 @@ typedef struct fs_disk_quota {
>   */
>  #define FS_DQ_BTIMER	(1<<6)
>  #define FS_DQ_ITIMER	(1<<7)
> -#define FS_DQ_RTBTIMER 	(1<<8)
> +#define FS_DQ_RTBTIMER	(1<<8)
>  #define FS_DQ_TIMER_MASK	(FS_DQ_BTIMER | FS_DQ_ITIMER | FS_DQ_RTBTIMER)
>  
>  /*
> 

From sandeen@sandeen.net  Wed Jan 20 11:59:12 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 12FF17F37
	for ; Wed, 20 Jan 2016 11:59:12 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 01AE48F8052
	for ; Wed, 20 Jan 2016 09:59:11 -0800 (PST)
X-ASG-Debug-ID: 1453312750-04cbb026fe19f40001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 8ZZY4V4GJpVu0AWd for ; Wed, 20 Jan 2016 09:59:10 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 3551263CBD0E
	for ; Wed, 20 Jan 2016 11:59:10 -0600 (CST)
To: xfs@oss.sgi.com
From: Eric Sandeen 
Subject: Pass mp to kmem_alloc and friends?
X-Enigmail-Draft-Status: N1110
X-ASG-Orig-Subj: Pass mp to kmem_alloc and friends?
Message-ID: <569FCAED.4050306@sandeen.net>
Date: Wed, 20 Jan 2016 11:59:09 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453312750
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26300
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

I had a request for the kmem_alloc deadlock warning to print the
filesystem involved.

Any objections to passing mp into kmem_alloc() and friends whenever
it's reasonably available from the caller?

It'd be a big mechanical change, don't want to embark on that unless
it seems acceptable & useful.

I think we generally know the root causes of the most common deadlock
warnings, but it's a warm fuzzy to give as much info as possible.

Heck, I almost wonder if passing a descriptive string in, for at
least the problematic cases we know about, i.e. "extent map realloc"
so we'd get something like:

XFS (sdb1): myprocess(123) possible memory allocation deadlock size 12345 during extent map realloc in kmem_alloc (mode:0x250)

I dunno ... too much? :)

-Eric

From bfoster@redhat.com  Wed Jan 20 12:54:43 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 03CFA7F37
	for ; Wed, 20 Jan 2016 12:54:43 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 8B76FAC002
	for ; Wed, 20 Jan 2016 10:54:36 -0800 (PST)
X-ASG-Debug-ID: 1453316074-04bdf04d6a1bf80001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id x1SuEUX7GJDoEMC6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 20 Jan 2016 10:54:35 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26])
	by mx1.redhat.com (Postfix) with ESMTPS id DFEC6A452B
	for ; Wed, 20 Jan 2016 18:54:34 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-215.bos.redhat.com [10.18.41.215])
	by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0KIsYOq008881
	for ; Wed, 20 Jan 2016 13:54:34 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id AA772120055; Wed, 20 Jan 2016 13:54:33 -0500 (EST)
From: Brian Foster 
To: xfs@oss.sgi.com
Subject: [PATCH] xfs: clean up unwritten buffers on write failure
Date: Wed, 20 Jan 2016 13:54:33 -0500
X-ASG-Orig-Subj: [PATCH] xfs: clean up unwritten buffers on write failure
Message-Id: <1453316073-9585-1-git-send-email-bfoster@redhat.com>
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453316075
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

The xfs_vm_write_failed() handler is currently responsible for cleaning
up any delalloc blocks over the range of a failed write beyond EOF.
Failure to do so results in warning messages and other inconsistencies
between buffer and extent state. The ->releasepage() handler currently
warns in the event of a page being released with either unwritten or
delalloc buffers, as neither is ever expected by the time a page is
released.

As has been reproduced by generic/083 on a -bsize=1k fs, it is currently
possible to trigger the ->releasepage() warning for a page with
unwritten buffers when a filesystem is near ENOSPC. This is reproduced
by the following sequence:

  $ mkfs.xfs -f -b size=1k -d size=100m 
  $ mount  /mnt/
  $
  $ xfs_io -fc "falloc -k 0 1k" /mnt/file
  $ dd if=/dev/zero of=/mnt/enospc conv=notrunc oflag=append
  $
  $ xfs_io -c "pwrite 512 1k" /mnt/file
  $ xfs_io -d -c "pwrite 16k 1k" /mnt/file

The first pwrite command attempts a block unaligned write across an
unwritten block and a hole. The delalloc for the hole fails with ENOSPC
and the subsequent error handling does not clean up the unwritten buffer
that was instantiated during the first ->get_block() call.

The second pwrite triggers a warning as part of the inode mapping
invalidation that occurs prior to direct I/O. The releasepage() handler
detects the unwritten buffer at this time, warns and prevents the
release of the page.

To deal with this problem, update xfs_vm_write_failed() to clean up
unwritten as well as delalloc buffers that are beyond EOF and within the
range of the failed write.

Signed-off-by: Brian Foster 
---

The alternative to this approach was to just kill the releasepage()
unwritten buffer warning, but this seemed more consistent and cautious
in that we can retain the warning to catch any other potential problems
in the future. I'm open to either approach, along with any other ideas
of course..

As it is, this survives xfstests with 1k and 4k blocks without any
explosions and many iterations of generic/083 w/ bsize=1k, which was the
original reproducer.

Brian

 fs/xfs/xfs_aops.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 7d6e368..c8c74d0 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -1783,14 +1783,22 @@ xfs_vm_write_failed(
 		if (block_start >= to)
 			break;
 
-		if (!buffer_delay(bh))
+		/*
+		 * Process delalloc and unwritten buffers beyond EOF. We can
+		 * encounter unwritten buffers in the event that a file has
+		 * post-EOF unwritten extents and an extending write happens to
+		 * fail (e.g., an unaligned write that also involves a delalloc
+		 * to the same page).
+		 */
+		if (!buffer_delay(bh) && !buffer_unwritten(bh))
 			continue;
 
 		if (!buffer_new(bh) && block_offset < i_size_read(inode))
 			continue;
 
-		xfs_vm_kill_delalloc_range(inode, block_offset,
-					   block_offset + bh->b_size);
+		if (buffer_delay(bh))
+			xfs_vm_kill_delalloc_range(inode, block_offset,
+						   block_offset + bh->b_size);
 
 		/*
 		 * This buffer does not contain data anymore. make sure anyone
@@ -1801,6 +1809,7 @@ xfs_vm_write_failed(
 		clear_buffer_mapped(bh);
 		clear_buffer_new(bh);
 		clear_buffer_dirty(bh);
+		clear_buffer_unwritten(bh);
 	}
 
 }
-- 
2.4.3

From darrick.wong@oracle.com  Wed Jan 20 13:54:06 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id BAE457F37
	for ; Wed, 20 Jan 2016 13:54:06 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 8A22A30406A
	for ; Wed, 20 Jan 2016 11:54:06 -0800 (PST)
X-ASG-Debug-ID: 1453319639-04bdf04d6a1dc00001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id XgxkDwfjM4X1oAi8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 20 Jan 2016 11:53:59 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0KJrwLP006540
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 20 Jan 2016 19:53:59 GMT
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0KJrtfZ002287
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 20 Jan 2016 19:53:58 GMT
Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11])
	by userv0122.oracle.com (8.14.4/8.13.8) with ESMTP id u0KJrtmI005892;
	Wed, 20 Jan 2016 19:53:55 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Wed, 20 Jan 2016 11:53:55 -0800
Date: Wed, 20 Jan 2016 11:53:54 -0800
From: "Darrick J. Wong" 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: Pass mp to kmem_alloc and friends?
Message-ID: <20160120195354.GA5500@birch.djwong.org>
X-ASG-Orig-Subj: Re: Pass mp to kmem_alloc and friends?
References: <569FCAED.4050306@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <569FCAED.4050306@sandeen.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453319639
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26303
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Wed, Jan 20, 2016 at 11:59:09AM -0600, Eric Sandeen wrote:
> I had a request for the kmem_alloc deadlock warning to print the
> filesystem involved.
> 
> Any objections to passing mp into kmem_alloc() and friends whenever
> it's reasonably available from the caller?
> 
> It'd be a big mechanical change, don't want to embark on that unless
> it seems acceptable & useful.
> 
> I think we generally know the root causes of the most common deadlock
> warnings, but it's a warm fuzzy to give as much info as possible.
> 
> Heck, I almost wonder if passing a descriptive string in, for at
> least the problematic cases we know about, i.e. "extent map realloc"
> so we'd get something like:
> 
> XFS (sdb1): myprocess(123) possible memory allocation deadlock size 12345 during extent map realloc in kmem_alloc (mode:0x250)
> 
> I dunno ... too much? :)

Not enough?  What about putting in just enough macro madness to report
the name & line number of the calling function in the message?

XFS (sdb1): myprocess(123) possible kmem_alloc deadlock in
xfs_eat_my_data.c:5135 (size:12345 mode:0x250)

(or maybe a tracepoint?)

--D

> 
> -Eric
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From david@fromorbit.com  Wed Jan 20 14:32:18 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id E5C097F37
	for ; Wed, 20 Jan 2016 14:32:18 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id C8BDD304053
	for ; Wed, 20 Jan 2016 12:32:15 -0800 (PST)
X-ASG-Debug-ID: 1453321932-04bdf04d6a1e9f0001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id C8nmxD3E7sHKHlRG for ; Wed, 20 Jan 2016 12:32:13 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2ANCQBL7p9WPGwtLHleKAECgw+BP4ZfgXifLAaLXIlGhgkEAgKBR00BAQEBAQEHAQEBAUE/hDQBAQEDATocIwULCAMYCSUPBSUDBxoTiBMHvggBAQEHAgEdGIVahTyHOkOBDwWXBI1WjwlEjX2EXSguhxIBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail04.adl6.internode.on.net with ESMTP; 21 Jan 2016 07:02:11 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aLzQJ-0001F5-4N; Thu, 21 Jan 2016 07:32:11 +1100
Date: Thu, 21 Jan 2016 07:32:11 +1100
From: Dave Chinner 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: Pass mp to kmem_alloc and friends?
Message-ID: <20160120203211.GC20456@dastard>
X-ASG-Orig-Subj: Re: Pass mp to kmem_alloc and friends?
References: <569FCAED.4050306@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <569FCAED.4050306@sandeen.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1453321932
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26304
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed, Jan 20, 2016 at 11:59:09AM -0600, Eric Sandeen wrote:
> I had a request for the kmem_alloc deadlock warning to print the
> filesystem involved.
> 
> Any objections to passing mp into kmem_alloc() and friends whenever
> it's reasonably available from the caller?
> 
> It'd be a big mechanical change, don't want to embark on that unless
> it seems acceptable & useful.

I think I hinted at this in the configurable error handling patchset
I have so that we could have configurable ENOMEM error handling. My
comment in the current commit message for that patch includes this:

| I'm not yet sure how to hook it into the memory allocation calls -
| that will be done in a later patch; this just demonstrates how
| multiple classes are configured and initialised. It may be that we
| don't configure specific errors here - instead configure how we
| handle specific types of allocation failure e.g. GFP_KERNEL vs
| GFP_NOFS vs allocations inside transactions. Either way, we are
| going to need to plumb the error config handler into the
| memory allocation code in some manner.

So I think we are going to have to plumb the xfs_mount into these
calls at some point in the near future.

> I think we generally know the root causes of the most common deadlock
> warnings, but it's a warm fuzzy to give as much info as possible.
> 
> Heck, I almost wonder if passing a descriptive string in, for at
> least the problematic cases we know about, i.e. "extent map realloc"
> so we'd get something like:
> 
> XFS (sdb1): myprocess(123) possible memory allocation deadlock size 12345 during extent map realloc in kmem_alloc (mode:0x250)

If we want more context, then make it an error message that can dump
the stack when the error level is turned up. i.e. if we pass in an
xfs_mount, we can use XFS_ERROR_REPORT here...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Wed Jan 20 14:35:10 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 4994F7F37
	for ; Wed, 20 Jan 2016 14:35:10 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 2AB968F8040
	for ; Wed, 20 Jan 2016 12:35:07 -0800 (PST)
X-ASG-Debug-ID: 1453322104-04bdf04d6b1eaf0001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id kuQCSDXHmQGfO2DN for ; Wed, 20 Jan 2016 12:35:04 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2ANCQBL7p9WPGwtLHleKAECgw+BP4ZfgXifLAaLXIlGhgkEAgKBR00BAQEBAQEHAQEBAUE/hDUBAQQ6HCMQCAMYCSUPBSUDBxoTiBq+CAEBAQcCAR0YhVqFPId9gQ8FjTCJVI1WjwlEjX2EXSguhxIBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail04.adl6.internode.on.net with ESMTP; 21 Jan 2016 07:05:04 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aLzT5-0001GS-Fj; Thu, 21 Jan 2016 07:35:03 +1100
Date: Thu, 21 Jan 2016 07:35:03 +1100
From: Dave Chinner 
To: "Darrick J. Wong" 
Cc: Eric Sandeen , xfs@oss.sgi.com
Subject: Re: Pass mp to kmem_alloc and friends?
Message-ID: <20160120203503.GD20456@dastard>
X-ASG-Orig-Subj: Re: Pass mp to kmem_alloc and friends?
References: <569FCAED.4050306@sandeen.net>
 <20160120195354.GA5500@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160120195354.GA5500@birch.djwong.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1453322104
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26304
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed, Jan 20, 2016 at 11:53:54AM -0800, Darrick J. Wong wrote:
> On Wed, Jan 20, 2016 at 11:59:09AM -0600, Eric Sandeen wrote:
> > I had a request for the kmem_alloc deadlock warning to print the
> > filesystem involved.
> > 
> > Any objections to passing mp into kmem_alloc() and friends whenever
> > it's reasonably available from the caller?
> > 
> > It'd be a big mechanical change, don't want to embark on that unless
> > it seems acceptable & useful.
> > 
> > I think we generally know the root causes of the most common deadlock
> > warnings, but it's a warm fuzzy to give as much info as possible.
> > 
> > Heck, I almost wonder if passing a descriptive string in, for at
> > least the problematic cases we know about, i.e. "extent map realloc"
> > so we'd get something like:
> > 
> > XFS (sdb1): myprocess(123) possible memory allocation deadlock size 12345 during extent map realloc in kmem_alloc (mode:0x250)
> > 
> > I dunno ... too much? :)
> 
> Not enough?  What about putting in just enough macro madness to report
> the name & line number of the calling function in the message?
> 
> XFS (sdb1): myprocess(123) possible kmem_alloc deadlock in
> xfs_eat_my_data.c:5135 (size:12345 mode:0x250)

Problem with that is the the return address will often just point to
a higher level wrapper function, and we are still without any
caller context. e.g. if we find the caller is xfs_buf_alloc_maps(),
what operation was being performed that needed a new buffer
allocated?

IMO, the only thing that will actually be useful for debugging at
this point is a full stack dump....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Wed Jan 20 15:19:50 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 4FE097F37
	for ; Wed, 20 Jan 2016 15:19:50 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id D2E48AC002
	for ; Wed, 20 Jan 2016 13:19:49 -0800 (PST)
X-ASG-Debug-ID: 1453324786-04cb6c0d5f1e030001-NocioJ
Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by cuda.sgi.com with ESMTP id Ea4ZrzdbTBNpXH4U for ; Wed, 20 Jan 2016 13:19:47 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.141
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2APCQDT+J9WPGwtLHldKAECgw+BP4ZfgXiefgaLXIVBhAWGCQQCAoE+TQEBAQEBAQcBAQEBQT+ENQEBBCcTHCMQCAMOCgklDwUlAwcaE4gavTsBAQEBBgEBAQEcGIVahTyHfYEPBYdMhxiIII1WjwlEjX2EXSguhVkEH4EWAQEB
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail04.adl6.internode.on.net with ESMTP; 21 Jan 2016 07:49:45 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aM0AK-0001NR-Li; Thu, 21 Jan 2016 08:19:44 +1100
Date: Thu, 21 Jan 2016 08:19:44 +1100
From: Dave Chinner 
To: Michal Hocko 
Cc: xfs@oss.sgi.com, jkosina@suse.cz,
	Hendrik Woltersdorf 
Subject: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Message-ID: <20160120211944.GE20456@dastard>
X-ASG-Orig-Subj: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
References: <1452661968-11482-1-git-send-email-david@fromorbit.com>
 <20160120084750.GA14187@dhcp22.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160120084750.GA14187@dhcp22.suse.cz>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail04.adl6.internode.on.net[150.101.137.141]
X-Barracuda-Start-Time: 1453324786
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26305
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed, Jan 20, 2016 at 09:47:50AM +0100, Michal Hocko wrote:
> On Wed 13-01-16 16:12:48, Dave Chinner wrote:
> > This reverts commit 24ba16bb3d499c49974669cd8429c3e4138ab102 as it
> > prevents machines from suspending. This regression occurs when the
> > xfsaild is idle on entry to suspend, and so there s no activity to
> > wake it from it's idle sleep and hence see that it is supposed to
> > freeze. Hence the freezer times out waiting for it and suspend is
> > cancelled.
> > 
> > There is no obvious fix for this short of freezing the filesystem
> > properly, so revert this change for now.
> 
> We had a similar report opensuse bugzilla just recently. I believe the
> proper fix should be the following:
....
> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> index aa67339b9537..d6c9c3e9e02b 100644
> --- a/fs/xfs/xfs_trans_ail.c
> +++ b/fs/xfs/xfs_trans_ail.c
> @@ -520,14 +520,14 @@ xfsaild(
>  		if (!xfs_ail_min(ailp) &&
>  		    ailp->xa_target == ailp->xa_target_prev) {
>  			spin_unlock(&ailp->xa_lock);
> -			schedule();
> +			freezable_schedule();

Oh, wonderful. A completely separate set of undocumented schedule
functions defined inside the freezer header files that are needed
just for freezable kthreads. Perhaps there should be some
documentation somewhere on how to write suspend safe kthreads?

This is too late for the current merge window, but I'll queue it up
for the next one so we can get some testing on it first.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From sandeen@redhat.com  Wed Jan 20 17:43:54 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 690067F37
	for ; Wed, 20 Jan 2016 17:43:54 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 586DB304053
	for ; Wed, 20 Jan 2016 15:43:51 -0800 (PST)
X-ASG-Debug-ID: 1453333429-04cbb026fd22730001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id khx3RyFeXMILjcfm (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 20 Jan 2016 15:43:50 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	by mx1.redhat.com (Postfix) with ESMTPS id 932918F877
	for ; Wed, 20 Jan 2016 23:43:49 +0000 (UTC)
Received: from Liberator.local (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0KNhmAm019831
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for ; Wed, 20 Jan 2016 18:43:49 -0500
To: xfs@oss.sgi.com
From: Eric Sandeen 
Subject: [PATCH] xfs: don't overflow quota ID when initializing dqblk
Message-ID: <56A01BB4.1050409@redhat.com>
X-ASG-Orig-Subj: [PATCH] xfs: don't overflow quota ID when initializing dqblk
Date: Wed, 20 Jan 2016 17:43:48 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453333430
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

Quota IDs are unsigned, and so we can pass in values up
to 2^32-1.  But if we try to initialize a block containing
values over MAX_INT, curid will overflow and assert.

curid holds a quota ID, so give it the proper
xfs_dqid_t type (and remove the now-impossible ASSERT).

Signed-off-by: Eric Sandeen 
---

diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 83ce4aa..5d92bb5 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -232,7 +232,8 @@ xfs_qm_init_dquot_blk(
 {
 	struct xfs_quotainfo	*q = mp->m_quotainfo;
 	xfs_dqblk_t	*d;
-	int		curid, i;
+	xfs_dqid_t	curid;
+	int		i;
 
 	ASSERT(tp);
 	ASSERT(xfs_buf_islocked(bp));
@@ -243,7 +244,6 @@ xfs_qm_init_dquot_blk(
 	 * ID of the first dquot in the block - id's are zero based.
 	 */
 	curid = id - (id % q->qi_dqperchunk);
-	ASSERT(curid >= 0);
 	memset(d, 0, BBTOB(q->qi_dqchunklen));
 	for (i = 0; i < q->qi_dqperchunk; i++, d++, curid++) {
 		d->dd_diskdq.d_magic = cpu_to_be16(XFS_DQUOT_MAGIC);

From sandeen@redhat.com  Wed Jan 20 18:07:39 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 96F097F37
	for ; Wed, 20 Jan 2016 18:07:39 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 2510EAC002
	for ; Wed, 20 Jan 2016 16:07:36 -0800 (PST)
X-ASG-Debug-ID: 1453334854-04cbb026fd23180001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id PGQPsOAedXh6YdCr (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 20 Jan 2016 16:07:35 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	by mx1.redhat.com (Postfix) with ESMTPS id 61A316E772
	for ; Thu, 21 Jan 2016 00:07:34 +0000 (UTC)
Received: from Liberator.local (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0L07Xsn014442
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Wed, 20 Jan 2016 19:07:34 -0500
To: xfs@oss.sgi.com, dwysocha@redhat.com
From: Eric Sandeen 
Subject: [PATCH] mkfs.xfs.8: Clarify mkfs vs. mount block size limits.
Message-ID: <56A02144.9030206@redhat.com>
X-ASG-Orig-Subj: [PATCH] mkfs.xfs.8: Clarify mkfs vs. mount block size limits.
Date: Wed, 20 Jan 2016 18:07:32 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453334855
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

The current explanation is a bit vague to the uninitiated;
clarify the text to make it obvious that while the mkfs.xfs
utility is able to create valid filesystems with block sizes
up to 65536, the Linux kernel can only mount filesystems with
page sized or smaller blocks.

Signed-off-by: Eric Sandeen 
---

diff --git a/man/man8/mkfs.xfs.8 b/man/man8/mkfs.xfs.8
index 1fe510b..980c7bc 100644
--- a/man/man8/mkfs.xfs.8
+++ b/man/man8/mkfs.xfs.8
@@ -126,7 +126,10 @@ or in bytes with
 .BR size= .
 The default value is 4096 bytes (4 KiB), the minimum is 512, and the
 maximum is 65536 (64 KiB).
-XFS on Linux currently only supports pagesize or smaller blocks.
+Although
+.B mkfs.xfs
+will accept any of these values and create a valid filesystem,
+XFS on Linux can only mount filesystems with pagesize or smaller blocks.
 .TP
 .BI \-m " global_metadata_options"
 These options specify metadata format options that either apply to the entire

From root@cultura.mg.gov.br  Thu Jan 21 01:38:44 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: ***
X-Spam-Status: No, score=4.0 required=5.0 tests=HTML_MESSAGE,
	HTML_TAG_BALANCE_BODY,HTML_TAG_BALANCE_HEAD,MIME_HTML_ONLY autolearn=no
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 36CF57F37
	for ; Thu, 21 Jan 2016 01:38:44 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id B96A1AC002
	for ; Wed, 20 Jan 2016 23:38:40 -0800 (PST)
X-ASG-Debug-ID: 1453361915-04cbb026ff2c6b0001-NocioJ
Received: from cultura.mg.gov.br (mail.cultura.mg.gov.br [200.198.28.212]) by cuda.sgi.com with ESMTP id DCg8IyJyoCqgVxcX (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 20 Jan 2016 23:38:37 -0800 (PST)
X-Barracuda-Envelope-From: root@cultura.mg.gov.br
X-Barracuda-Apparent-Source-IP: 200.198.28.212
Received: by cultura.mg.gov.br (Postfix, from userid 0)
	id 40541967A32; Thu, 21 Jan 2016 05:36:00 -0200 (BRST)
From: spam.quarentena@cultura.mg.gov.br
To: xfs@oss.sgi.com
Subject: Sumario de Mensagens em Quarentena - cultura.mg.gov.br
Mime-Version: 1.0
X-ASG-Orig-Subj: Sumario de Mensagens em Quarentena - cultura.mg.gov.br
X-Mailer: Prodemge_Qspamd
Content-Type: text/html; charset=UTF-8
Message-Id: <20160121073600.40541967A32@cultura.mg.gov.br>
Date: Thu, 21 Jan 2016 05:36:00 -0200 (BRST)
X-Barracuda-Connect: mail.cultura.mg.gov.br[200.198.28.212]
X-Barracuda-Start-Time: 1453361916
X-Barracuda-Encrypted: DHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.81
X-Barracuda-Spam-Status: No, SCORE=0.81 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=HTML_MESSAGE, HTML_TAG_BALANCE_BODY, HTML_TAG_BALANCE_HEAD, MIME_HTML_ONLY, NO_REAL_NAME
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26318
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 NO_REAL_NAME           From: does not include a real name
	0.81 HTML_TAG_BALANCE_BODY  BODY: HTML has unbalanced "body" tags
	0.00 MIME_HTML_ONLY         BODY: Message only has text/html MIME parts
	0.00 HTML_TAG_BALANCE_HEAD  BODY: HTML has unbalanced "head" tags
	0.00 HTML_MESSAGE           BODY: HTML included in message


Sumário quarentena


Sumário de mensagens em quarentena - xfs@oss.sgi.com


 As mensagens abaixo foram colocadas em quarentena por terem sido consideradas SPAM. Caso alguma destas mensagens não seja Spam, é possível que ela seja liberada para o recebimento. Veja as instruções no fim desta mensagem.


ID MENSAGEMRemetenteAssuntoHora


in.28450.2016011923503430press@albaath-univ.edu.sy GREETINGS23:50:34




Para solicitar a entrega de alguma das mensagens acima



Envie uma mensagem para  a caixa postal spam.quarentena@cultura.mg.gov.br com o assunto:

Mensagem ID_MENSAGEM


Substituindo ID_MENSAGEM pelo ID da Mensagem requerida na tabela acima.

Deixe o corpo do e-mail em branco.


As mensagens em quarentena serão excluídas após 7 dias.


Mensagem gerada em 21/01/2016 - 05:36 por qspamd - Prodemge.



From mhocko@suse.cz  Thu Jan 21 04:21:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id E676A7F37
	for ; Thu, 21 Jan 2016 04:21:40 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D62108F8054
	for ; Thu, 21 Jan 2016 02:21:37 -0800 (PST)
X-ASG-Debug-ID: 1453371694-04cbb026fc30570001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 4eJA9utaEZxp4mXL (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 21 Jan 2016 02:21:35 -0800 (PST)
X-Barracuda-Envelope-From: mhocko@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 587E6ADC6;
	Thu, 21 Jan 2016 10:21:30 +0000 (UTC)
Date: Thu, 21 Jan 2016 11:21:30 +0100
From: Michal Hocko 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, jkosina@suse.cz,
	Hendrik Woltersdorf ,
	"Rafael J. Wysocki" 
Subject: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Message-ID: <20160121102129.GD29520@dhcp22.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
References: <1452661968-11482-1-git-send-email-david@fromorbit.com>
 <20160120084750.GA14187@dhcp22.suse.cz>
 <20160120211944.GE20456@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160120211944.GE20456@dastard>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453371694
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26321
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

[CCing Rafael - the thread starts here
http://lkml.kernel.org/r/1452661968-11482-1-git-send-email-david%40fromorbit.com]

On Thu 21-01-16 08:19:44, Dave Chinner wrote:
> On Wed, Jan 20, 2016 at 09:47:50AM +0100, Michal Hocko wrote:
> > On Wed 13-01-16 16:12:48, Dave Chinner wrote:
> > > This reverts commit 24ba16bb3d499c49974669cd8429c3e4138ab102 as it
> > > prevents machines from suspending. This regression occurs when the
> > > xfsaild is idle on entry to suspend, and so there s no activity to
> > > wake it from it's idle sleep and hence see that it is supposed to
> > > freeze. Hence the freezer times out waiting for it and suspend is
> > > cancelled.
> > > 
> > > There is no obvious fix for this short of freezing the filesystem
> > > properly, so revert this change for now.
> > 
> > We had a similar report opensuse bugzilla just recently. I believe the
> > proper fix should be the following:
> ....
> > diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> > index aa67339b9537..d6c9c3e9e02b 100644
> > --- a/fs/xfs/xfs_trans_ail.c
> > +++ b/fs/xfs/xfs_trans_ail.c
> > @@ -520,14 +520,14 @@ xfsaild(
> >  		if (!xfs_ail_min(ailp) &&
> >  		    ailp->xa_target == ailp->xa_target_prev) {
> >  			spin_unlock(&ailp->xa_lock);
> > -			schedule();
> > +			freezable_schedule();
> 
> Oh, wonderful. A completely separate set of undocumented schedule
> functions defined inside the freezer header files that are needed
> just for freezable kthreads.

Yeah, I am not really happy about them either.

> Perhaps there should be some documentation somewhere on how to write
> suspend safe kthreads?

There is something in Documentation/power/freezing-of-tasks.txt but
freezable_schedule* is not mentioned there. Prhaps something like the
following would make it more clear?

diff --git a/Documentation/power/freezing-of-tasks.txt b/Documentation/power/freezing-of-tasks.txt
index 85894d83b352..4df53f0c05a6 100644
--- a/Documentation/power/freezing-of-tasks.txt
+++ b/Documentation/power/freezing-of-tasks.txt
@@ -56,8 +56,11 @@ calling try_to_freeze().  The main loop of a freezable kernel thread may look
 If a freezable kernel thread fails to call try_to_freeze() after the freezer has
 initiated a freezing operation, the freezing of tasks will fail and the entire
 hibernation operation will be cancelled.  For this reason, freezable kernel
-threads must call try_to_freeze() somewhere or use one of the
-wait_event_freezable() and wait_event_freezable_timeout() macros.
+threads must call try_to_freeze() somewhere, use wait_event_freezable*()
+and wait_event_freezable_timeout() macros or use freezable_schedule* to hide
+the kernel thread from the suspend if it is guaranteed that a later wake up
+cannot interact with the suspend process (e.g. there is a try_to_freeze call
+or no IO is submitted or any of the suspended devices is not accessed etc..)
 
 After the system memory state has been restored from a hibernation image and
 devices have been reinitialized, the function thaw_processes() is called in

> This is too late for the current merge window, but I'll queue it up
> for the next one so we can get some testing on it first.

Sure, no problem and thanks!
-- 
Michal Hocko
SUSE Labs

From chandnond@yahoo.com  Thu Jan 21 07:34:59 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: **
X-Spam-Status: No, score=2.4 required=5.0 tests=FORGED_YAHOO_RCVD,
	FREEMAIL_FROM autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 92F537F37
	for ; Thu, 21 Jan 2016 07:34:59 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 82BBB304066
	for ; Thu, 21 Jan 2016 05:34:56 -0800 (PST)
X-ASG-Debug-ID: 1453383291-04cbb026fe35810001-NocioJ
Received: from eguangchang.com (eguangchang.com [104.255.171.61]) by cuda.sgi.com with ESMTP id A2BVOrkwzM04LapW for ; Thu, 21 Jan 2016 05:34:53 -0800 (PST)
X-Barracuda-Envelope-From: chandnond@yahoo.com
X-Barracuda-Apparent-Source-IP: 104.255.171.61
To: xfs@oss.sgi.com
Subject: =?UTF-8?B?SWhyZSBuZXVlIEhlaXp1bmc6IGtvc3RlbmxvcyBkYW5rIHN0YWF0bGljaGVyIEbDtnJkZXJ1bmc=?=
Message-ID: 
X-ASG-Orig-Subj: =?UTF-8?B?SWhyZSBuZXVlIEhlaXp1bmc6IGtvc3RlbmxvcyBkYW5rIHN0YWF0bGljaGVyIEbDtnJkZXJ1bmc=?=
Date: Thu, 21 Jan 2016 14:18:15 +0100
From: "Heizungsbau Franke" 
Reply-To: chandnond@yahoo.com
MIME-Version: 1.0
X-Mailer-LID: 43
X-Mailer-RecptId: 25520412
X-Mailer-SID: 1322
X-Mailer-Sent-By: 1
Content-Type: text/plain; format=flowed; charset="UTF-8"
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: eguangchang.com[104.255.171.61]
X-Barracuda-Start-Time: 1453383291
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 1.41
X-Barracuda-Spam-Status: No, SCORE=1.41 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=FORGED_YAHOO_RCVD, FORGED_YAHOO_RCVD_2
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26324
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 FORGED_YAHOO_RCVD      'From' yahoo.com does not match 'Received' headers
	1.41 FORGED_YAHOO_RCVD_2    'From' yahoo.com does not match 'Received'
	                           headers

Guten Tag,

möchten Sie Ihre Heizung erneuern, kostenlos dank staatlicher Förderung?

Die Bundesregierung unterstützt den Erwerb von neuen, ökologischen
Heizungen – das sollten Sie nutzen!

Ihre Heizung wird kostenlos erneuert -
Sie bekommen zusätzlich eine staatliche Förderung von bis zu 11.000 Euro
zur freien Verfügung.

Diese Aktion ist für Sie vollkommen kostenlos und unverbindlich – auch
wir werden aus Fördermitteln bezahlt.

Bitte informieren Sie sich hier:

aktionheizung.com


Mit freundlichen Grüßen,

Heizungsbau Franke

From chandnond@yahoo.com  Thu Jan 21 08:24:25 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: **
X-Spam-Status: No, score=2.4 required=5.0 tests=FORGED_YAHOO_RCVD,
	FREEMAIL_FROM autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 74A457F37
	for ; Thu, 21 Jan 2016 08:24:25 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 031DBAC002
	for ; Thu, 21 Jan 2016 06:24:24 -0800 (PST)
X-ASG-Debug-ID: 1453386261-04cb6c0d6033f40001-NocioJ
Received: from eguangchang.com (eguangchang.com [104.255.171.61]) by cuda.sgi.com with ESMTP id wH44qAYPhf2b8PKN for ; Thu, 21 Jan 2016 06:24:22 -0800 (PST)
X-Barracuda-Envelope-From: chandnond@yahoo.com
X-Barracuda-Apparent-Source-IP: 104.255.171.61
To: xfs@oss.sgi.com
Subject: =?UTF-8?B?SWhyZSBuZXVlIEhlaXp1bmc6IGtvc3RlbmxvcyBkYW5rIHN0YWF0bGljaGVyIEbDtnJkZXJ1bmc=?=
Message-ID: 
X-ASG-Orig-Subj: =?UTF-8?B?SWhyZSBuZXVlIEhlaXp1bmc6IGtvc3RlbmxvcyBkYW5rIHN0YWF0bGljaGVyIEbDtnJkZXJ1bmc=?=
Date: Thu, 21 Jan 2016 15:47:24 +0100
From: "Heizungsbau Franke" 
Reply-To: chandnond@yahoo.com
MIME-Version: 1.0
X-Mailer-LID: 43
X-Mailer-RecptId: 25520412
X-Mailer-SID: 1477
X-Mailer-Sent-By: 1
Content-Type: text/plain; format=flowed; charset="UTF-8"
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: eguangchang.com[104.255.171.61]
X-Barracuda-Start-Time: 1453386261
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 1.41
X-Barracuda-Spam-Status: No, SCORE=1.41 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=FORGED_YAHOO_RCVD, FORGED_YAHOO_RCVD_2
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26325
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 FORGED_YAHOO_RCVD      'From' yahoo.com does not match 'Received' headers
	1.41 FORGED_YAHOO_RCVD_2    'From' yahoo.com does not match 'Received'
	                           headers

Guten Tag,

möchten Sie Ihre Heizung erneuern, kostenlos dank staatlicher Förderung?

Die Bundesregierung unterstützt den Erwerb von neuen, ökologischen
Heizungen – das sollten Sie nutzen!

Ihre Heizung wird kostenlos erneuert -
Sie bekommen zusätzlich eine staatliche Förderung von bis zu 11.000 Euro
zur freien Verfügung.

Diese Aktion ist für Sie vollkommen kostenlos und unverbindlich – auch
wir werden aus Fördermitteln bezahlt.

Bitte informieren Sie sich hier:

heizungsaktion.com


Mit freundlichen Grüßen,

Heizungsbau Franke

From cmaiolino@redhat.com  Thu Jan 21 08:33:55 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 15CDC7F37
	for ; Thu, 21 Jan 2016 08:33:55 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 925ACAC005
	for ; Thu, 21 Jan 2016 06:33:54 -0800 (PST)
X-ASG-Debug-ID: 1453386832-04cbb026fc37340001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id SIRiYtbju0aXJCqB (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 21 Jan 2016 06:33:53 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id 926CAA287B
	for ; Thu, 21 Jan 2016 14:33:52 +0000 (UTC)
Received: from zion.usersys.redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0LEXpJP019218
	for ; Thu, 21 Jan 2016 09:33:52 -0500
From: Carlos Maiolino 
To: xfs@oss.sgi.com
Subject: [PATCH] Minor whitespace cleanups - V2
Date: Thu, 21 Jan 2016 15:33:45 +0100
X-ASG-Orig-Subj: [PATCH] Minor whitespace cleanups - V2
Message-Id: <1453386825-9097-1-git-send-email-cmaiolino@redhat.com>
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453386833
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

While investigating a quota issue, I fixed a few whitespaces.

Changelog:
	V2 - fix also whitespaces between comments as suggested by Eric

Signed-off-by: Carlos Maiolino 
---
 include/xqm.h | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/include/xqm.h b/include/xqm.h
index 47f58a0..116ce00 100644
--- a/include/xqm.h
+++ b/include/xqm.h
@@ -55,14 +55,14 @@ typedef struct fs_disk_quota {
 	__s32		d_itimer;	/* zero if within inode limits */
 					/* if not, we refuse service */
 	__s32		d_btimer;	/* similar to above; for disk blocks */
-	__u16	  	d_iwarns;       /* # warnings issued wrt num inodes */
-	__u16	  	d_bwarns;       /* # warnings issued wrt disk blocks */
+	__u16		d_iwarns;	/* # warnings issued wrt num inodes */
+	__u16		d_bwarns;	/* # warnings issued wrt disk blocks */
 	__s32		d_padding2;	/* padding2 - for future use */
 	__u64		d_rtb_hardlimit;/* absolute limit on realtime blks */
 	__u64		d_rtb_softlimit;/* preferred limit on RT disk blks */
 	__u64		d_rtbcount;	/* # realtime blocks owned */
 	__s32		d_rtbtimer;	/* similar to above; for RT disk blks */
-	__u16	  	d_rtbwarns;     /* # warnings issued wrt RT disk blks */
+	__u16		d_rtbwarns;	/* # warnings issued wrt RT disk blks */
 	__s16		d_padding3;	/* padding3 - for future use */
 	char		d_padding4[8];	/* yet more padding */
 } fs_disk_quota_t;
@@ -73,7 +73,7 @@ typedef struct fs_disk_quota {
 #define FS_DQ_ISOFT	(1<<0)
 #define FS_DQ_IHARD	(1<<1)
 #define FS_DQ_BSOFT	(1<<2)
-#define FS_DQ_BHARD 	(1<<3)
+#define FS_DQ_BHARD	(1<<3)
 #define FS_DQ_RTBSOFT	(1<<4)
 #define FS_DQ_RTBHARD	(1<<5)
 #define FS_DQ_LIMIT_MASK	(FS_DQ_ISOFT | FS_DQ_IHARD | FS_DQ_BSOFT | \
@@ -87,7 +87,7 @@ typedef struct fs_disk_quota {
  */
 #define FS_DQ_BTIMER	(1<<6)
 #define FS_DQ_ITIMER	(1<<7)
-#define FS_DQ_RTBTIMER 	(1<<8)
+#define FS_DQ_RTBTIMER	(1<<8)
 #define FS_DQ_TIMER_MASK	(FS_DQ_BTIMER | FS_DQ_ITIMER | FS_DQ_RTBTIMER)
 
 /*
-- 
2.4.3

From eStatement@ebill.etisalat.ae  Thu Jan 21 08:35:03 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: *
X-Spam-Status: No, score=1.4 required=5.0 tests=HK_SCAM_N4,HTML_MESSAGE,
	MIME_QP_LONG_LINE,T_REMOTE_IMAGE autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 203577F37
	for ; Thu, 21 Jan 2016 08:35:03 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id A334AAC002
	for ; Thu, 21 Jan 2016 06:35:02 -0800 (PST)
X-ASG-Debug-ID: 1453386896-04cb6c0d5f34310001-NocioJ
Received: from vps.paullafayet.com (vps.paullafayet.com [182.239.55.55]) by cuda.sgi.com with ESMTP id 8r79d7bMYqQgiYqm (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 21 Jan 2016 06:34:58 -0800 (PST)
X-Barracuda-Envelope-From: eStatement@ebill.etisalat.ae
X-Barracuda-Apparent-Source-IP: 182.239.55.55
Received: from [154.118.46.189] (port=51164 helo=[192.168.8.102])
	by vps.paullafayet.com with esmtpa (Exim 4.82)
	(envelope-from )
	id 1aMEVq-0002iN-In
	for xfs@oss.sgi.com; Thu, 21 Jan 2016 20:38:55 +0800
Content-Type: multipart/alternative; boundary="===============1682289242=="
MIME-Version: 1.0
Subject: Your Etisalat Online Bills Is Ready To Be Viewed
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Your Etisalat Online Bills Is Ready To Be Viewed
From: "Etisalat Online Service" 
Date: Thu, 21 Jan 2016 13:38:48 +0100
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - vps.paullafayet.com
X-AntiAbuse: Original Domain - oss.sgi.com
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - ebill.etisalat.ae
X-Get-Message-Sender-Via: vps.paullafayet.com: authenticated_id: janice@paullafayet.com
X-Barracuda-Connect: vps.paullafayet.com[182.239.55.55]
X-Barracuda-Start-Time: 1453386897
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 1.46
X-Barracuda-Spam-Status: No, SCORE=1.46 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC7_SA578_CH, HTML_MESSAGE, MIME_QP_LONG_LINE, MIME_QP_LONG_LINE_2, MISSING_MID
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26325
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.14 MISSING_MID            Missing Message-Id: header
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.00 MIME_QP_LONG_LINE      RAW: Quoted-printable line longer than 76 chars
	0.82 MIME_QP_LONG_LINE_2    RAW: Quoted-printable line longer than 76 chars
	0.50 BSF_SC7_SA578_CH       Custom Rule SA578_CH
Message-Id: <20160121143502.1673C1296080@cuda.sgi.com>

You will not see this in a MIME-aware mail reader.
--===============1682289242==
Content-Type: text/plain; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Description: Mail message body

   =

 Dear Customer, =

  Your Etisalat online bills is ready to be viewed.  Simplyclick hereto vie=
w statement.  Regards,  Etisalat Online Service Team  ---------------------=
--------- -  Note : Do not reply this Email as all message(s) will not be r=
esponded to and will disregard.   2015 Etisalat, Inc..Telecommunications Co=
rporation, branded trade name Etisalat.   =

=20
--===============1682289242==
Content-Type: text/html; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Description: Mail message body






	

	3D"http://www.etisalat.ae/en/syste=

	
Dear Customer, =

	



	Your Etisalat online bills is ready to be viewed.



	Simplyclick =

	hereto =

	view statement.



	Regards,



	Etisalat Online Service Team



	------------------------------ -



	Note : Do not reply=
 this =

	Email as all message(s) will not be responded to and will disregard.



	 2015 Etisalat, Inc..Telecommunications =

	Corporation, branded trade name Etisalat.




3D"http://www.etisalat.ae/en/system=3D"http://www.eti=







--===============1682289242==--

From sandeen@sandeen.net  Thu Jan 21 09:04:49 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 8370C7F37
	for ; Thu, 21 Jan 2016 09:04:49 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 64C5B8F8035
	for ; Thu, 21 Jan 2016 07:04:45 -0800 (PST)
X-ASG-Debug-ID: 1453388683-04cbb026fc38490001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id eVCd4I98mVVHeKC8 for ; Thu, 21 Jan 2016 07:04:43 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 494AE63CBCE8
	for ; Thu, 21 Jan 2016 09:04:43 -0600 (CST)
Subject: Re: [PATCH] Minor whitespace cleanups - V2
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] Minor whitespace cleanups - V2
References: <1453386825-9097-1-git-send-email-cmaiolino@redhat.com>
From: Eric Sandeen 
Message-ID: <56A0F38A.80105@sandeen.net>
Date: Thu, 21 Jan 2016 09:04:42 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <1453386825-9097-1-git-send-email-cmaiolino@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453388683
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26325
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/21/16 8:33 AM, Carlos Maiolino wrote:
> While investigating a quota issue, I fixed a few whitespaces.
> 
> Changelog:
> 	V2 - fix also whitespaces between comments as suggested by Eric
> 
> Signed-off-by: Carlos Maiolino 

Looks fine to me - I'll leave it up to Dave whether he wants to merge
whitespace fixes :) but -

Reviewed-by: Eric Sandeen 

> ---
>  include/xqm.h | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/include/xqm.h b/include/xqm.h
> index 47f58a0..116ce00 100644
> --- a/include/xqm.h
> +++ b/include/xqm.h
> @@ -55,14 +55,14 @@ typedef struct fs_disk_quota {
>  	__s32		d_itimer;	/* zero if within inode limits */
>  					/* if not, we refuse service */
>  	__s32		d_btimer;	/* similar to above; for disk blocks */
> -	__u16	  	d_iwarns;       /* # warnings issued wrt num inodes */
> -	__u16	  	d_bwarns;       /* # warnings issued wrt disk blocks */
> +	__u16		d_iwarns;	/* # warnings issued wrt num inodes */
> +	__u16		d_bwarns;	/* # warnings issued wrt disk blocks */
>  	__s32		d_padding2;	/* padding2 - for future use */
>  	__u64		d_rtb_hardlimit;/* absolute limit on realtime blks */
>  	__u64		d_rtb_softlimit;/* preferred limit on RT disk blks */
>  	__u64		d_rtbcount;	/* # realtime blocks owned */
>  	__s32		d_rtbtimer;	/* similar to above; for RT disk blks */
> -	__u16	  	d_rtbwarns;     /* # warnings issued wrt RT disk blks */
> +	__u16		d_rtbwarns;	/* # warnings issued wrt RT disk blks */
>  	__s16		d_padding3;	/* padding3 - for future use */
>  	char		d_padding4[8];	/* yet more padding */
>  } fs_disk_quota_t;
> @@ -73,7 +73,7 @@ typedef struct fs_disk_quota {
>  #define FS_DQ_ISOFT	(1<<0)
>  #define FS_DQ_IHARD	(1<<1)
>  #define FS_DQ_BSOFT	(1<<2)
> -#define FS_DQ_BHARD 	(1<<3)
> +#define FS_DQ_BHARD	(1<<3)
>  #define FS_DQ_RTBSOFT	(1<<4)
>  #define FS_DQ_RTBHARD	(1<<5)
>  #define FS_DQ_LIMIT_MASK	(FS_DQ_ISOFT | FS_DQ_IHARD | FS_DQ_BSOFT | \
> @@ -87,7 +87,7 @@ typedef struct fs_disk_quota {
>   */
>  #define FS_DQ_BTIMER	(1<<6)
>  #define FS_DQ_ITIMER	(1<<7)
> -#define FS_DQ_RTBTIMER 	(1<<8)
> +#define FS_DQ_RTBTIMER	(1<<8)
>  #define FS_DQ_TIMER_MASK	(FS_DQ_BTIMER | FS_DQ_ITIMER | FS_DQ_RTBTIMER)
>  
>  /*
> 

From tdm@sgi.com  Thu Jan 21 09:53:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=RP_MATCHES_RCVD autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 219F27F37
	for ; Thu, 21 Jan 2016 09:53:41 -0600 (CST)
Received: from estes.americas.sgi.com (estes.americas.sgi.com [128.162.236.10])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 1DD89AC001;
	Thu, 21 Jan 2016 07:53:39 -0800 (PST)
Received: from [128.162.232.11] (porter.americas.sgi.com [128.162.232.11])
	by estes.americas.sgi.com (Postfix) with ESMTP id 846027000737;
	Thu, 21 Jan 2016 09:53:39 -0600 (CST)
Message-ID: <56A0FF03.4070804@sgi.com>
Date: Thu, 21 Jan 2016 09:53:39 -0600
From: Troy McCorkell 
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: xfs@oss.sgi.com, Dave Chinner ,
	Eric Sandeen ,
	Russell Cattelan 
Subject: oss.sgi.com - maintenance 25Jan2016 at 8:00 CT
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

oss.sgi.com will be offline for approximately 30 minutes on
Monday Jan 25th starting at 8:00 am CT USA for maintenance.

From dan.j.williams@gmail.com  Thu Jan 21 10:37:14 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,T_DKIM_INVALID
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id CA23B7F37
	for ; Thu, 21 Jan 2016 10:37:14 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id A60E18F804C
	for ; Thu, 21 Jan 2016 08:37:14 -0800 (PST)
X-ASG-Debug-ID: 1453394232-04cbb026fd3bb60001-NocioJ
Received: from mail-qg0-f66.google.com (mail-qg0-f66.google.com [209.85.192.66]) by cuda.sgi.com with ESMTP id QFJgAyfpmydqElJD (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 21 Jan 2016 08:37:12 -0800 (PST)
X-Barracuda-Envelope-From: dan.j.williams@gmail.com
X-Barracuda-Apparent-Source-IP: 209.85.192.66
Received: by mail-qg0-f66.google.com with SMTP id 94so3192257qgt.1
        for ; Thu, 21 Jan 2016 08:37:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:sender:in-reply-to:references:date:message-id:subject
         :from:to:cc:content-type;
        bh=XnYlUfPJlqgnERVUjvcfvMtgthtAAPx5ga/Z/XQKV5g=;
        b=YFBYsDcl6fARIqvwPLvpFV7CDWszHSrkSkRhG2RaNSY/OwAHxAUYi8q+8QOh3vAcYa
         RTPe4QM56Kpl5MylFuFwIHKDZGzSntr/ozfHtwxxKb9xmeBFisnquNoNt3EVWkKTFBHl
         S6yDFzceiwSsGS6kKjmgrUS/JnnLq4gIzZ5s2CVTzdICjD1lbMNreC30t88lNIxrzsVM
         JDs6YYHYbmlDIMYSy0TBSxcvS9NeKzgkkojMsCza3jl61DnWdz77aLz6bysqdKwbh5aZ
         jjqJtW+0Jko874v4ZsfeE4HKViVXXZkrrwBIYdXeNEh9jupZRUyzs8Y0FP5CuOsXAoq3
         cYCg==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=intel-com.20150623.gappssmtp.com; s=20150623;
        h=mime-version:sender:in-reply-to:references:date:message-id:subject
         :from:to:cc:content-type;
        bh=XnYlUfPJlqgnERVUjvcfvMtgthtAAPx5ga/Z/XQKV5g=;
        b=xAAKO6XApktXf2AeEyNxX2OqKHKOB7zzwH8GDsGwt7F5hfN8IEQuh0jT50FjimQJy9
         faApPECv8FhSir6/FBK6QCK3GVA92trWqrNX29y3T19VZ7hzn98xYY4rG9s3a2049K4X
         RzHDTBilOCc28u7STkhuNVrRxAriqqmKUmjAiDzaEwUU3l0/VekejC6yW33uofQO8/f5
         h7v5IO2FnMNvYebrC1LTVh3/Wv6VtnAMtwmLDwvhaigl6Mx3ulPgBDyYIJCovCDnnJUz
         fN9LpryDBlUemLpqAx3Wgt4/de/3zOQe/MoZ6btox8ed3iUAx/BTi3iNJ28k7BzWPX0h
         /1Mg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:sender:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=XnYlUfPJlqgnERVUjvcfvMtgthtAAPx5ga/Z/XQKV5g=;
        b=ftn7+WqIP0QemDCS+DkbdMnaYULqbM++Y01QI4AAf9qELA+RpzppKym6Dki1nnLglY
         ihhGct/Lnczjo0ZeyWfF/S8jl8yvvXuSyZmwhyvcDDIz68+AKbg8fJguxLEEDaws2cIP
         LsQLe6XQQXz6PQrZJGWRUfPd3k5jl6vZNzoN70PcQveZZ8i/OUUn365y3xcavZOTCEzY
         7iR6kmBBKvFp2fQG3c9afGsY2lMX4Df5VCQUe31mDxW4XKnEZlOlR9/P5f7DoVY/8M+R
         hUYMuNF7AnhLdtmJU6irGuvo/M+S4yqU8UZ1PCZOo3jVUP8JmYgBaKhV1Zw4cqXNkDPd
         +lNg==
X-Gm-Message-State: AG10YOSSnw8Ra9vMBr1rpnrI0AKmxt+zQe46Rj9USSCM8ta9TRDzFdy9i22ix2Q4OsVU8pI6yFfwDVLNzo1rPg==
MIME-Version: 1.0
X-Received: by 10.140.176.206 with SMTP id w197mr37417388qhw.59.1453394231980;
 Thu, 21 Jan 2016 08:37:11 -0800 (PST)
Sender: dan.j.williams@gmail.com
Received: by 10.55.19.67 with HTTP; Thu, 21 Jan 2016 08:37:11 -0800 (PST)
In-Reply-To: <1451886892-15548-4-git-send-email-david@fromorbit.com>
References: <1451886892-15548-1-git-send-email-david@fromorbit.com>
	<1451886892-15548-4-git-send-email-david@fromorbit.com>
Date: Thu, 21 Jan 2016 08:37:11 -0800
X-Google-Sender-Auth: _kxnfxcGWGr5lAFJgSoLG9fsyWg
Message-ID: 
Subject: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
From: Dan Williams 
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
To: Dave Chinner 
Cc: xfs@oss.sgi.com, linux-fsdevel , ext4@vger.kernel.org
Content-Type: text/plain; charset=UTF-8
X-Barracuda-Connect: mail-qg0-f66.google.com[209.85.192.66]
X-Barracuda-Start-Time: 1453394232
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26328
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

On Sun, Jan 3, 2016 at 9:54 PM, Dave Chinner  wrote:
> From: Dave Chinner 
>
> Rather than just being able to turn DAX on and off via a mount
> option, some applications may only want to enable DAX for certain
> performance critical files in a filesystem.
>
> This patch introduces a new inode flag to enable DAX in the v3 inode
> di_flags2 field. It adds support for setting and clearing flags in
> the di_flags2 field via the XFS_IOC_FSSETXATTR ioctl, and sets the
> S_DAX inode flag appropriately when it is seen.
>
> When this flag is set on a directory, it acts as an "inherit flag".
> That is, inodes created in the directory will automatically inherit
> the on-disk inode DAX flag, enabling administrators to set up
> directory heirarchies that automatically use DAX. Setting this flag
> on an empty root directory will make the entire filesystem use DAX
> by default.

When switching from page-cache to DAX, don't we need to flush existing
page cache mappings and remap directly?  Or, is the thought that
userspace needs to comprehend the presence of mixed mappings after
changing S_DAX?

From david@fromorbit.com  Thu Jan 21 15:58:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 0E2817F37
	for ; Thu, 21 Jan 2016 15:58:29 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 8283CAC013
	for ; Thu, 21 Jan 2016 13:58:28 -0800 (PST)
X-ASG-Debug-ID: 1453413501-04cb6c0d5f459f0001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id zwq8OUqNa4F3t7e8 for ; Thu, 21 Jan 2016 13:58:22 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2A1DAB0U6FWPGwtLHleKAECgw+BP4Jlg3qBeJ5sAQEBAQEBBotahUGEA4YJAgIBAQKBQE0BAQEBAQEHAQEBAUE/hDQBAQEEOhwjEAgDFQMJJQ8FJQMHGhOIGr5SAQEBAQYCAR0YhVWFOoRxgwaBDwWWdY1MjwKOPIJbggEoLocjAQEB
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 22 Jan 2016 08:28:20 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aMNFE-0004Lu-Bl; Fri, 22 Jan 2016 08:58:20 +1100
Date: Fri, 22 Jan 2016 08:58:20 +1100
From: Dave Chinner 
To: Dan Williams 
Cc: xfs@oss.sgi.com, linux-fsdevel ,
	ext4@vger.kernel.org
Subject: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
Message-ID: <20160121215820.GA6033@dastard>
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
References: <1451886892-15548-1-git-send-email-david@fromorbit.com>
 <1451886892-15548-4-git-send-email-david@fromorbit.com>
 
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1453413501
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26336
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu, Jan 21, 2016 at 08:37:11AM -0800, Dan Williams wrote:
> On Sun, Jan 3, 2016 at 9:54 PM, Dave Chinner  wrote:
> > From: Dave Chinner 
> >
> > Rather than just being able to turn DAX on and off via a mount
> > option, some applications may only want to enable DAX for certain
> > performance critical files in a filesystem.
> >
> > This patch introduces a new inode flag to enable DAX in the v3 inode
> > di_flags2 field. It adds support for setting and clearing flags in
> > the di_flags2 field via the XFS_IOC_FSSETXATTR ioctl, and sets the
> > S_DAX inode flag appropriately when it is seen.
> >
> > When this flag is set on a directory, it acts as an "inherit flag".
> > That is, inodes created in the directory will automatically inherit
> > the on-disk inode DAX flag, enabling administrators to set up
> > directory heirarchies that automatically use DAX. Setting this flag
> > on an empty root directory will make the entire filesystem use DAX
> > by default.
> 
> When switching from page-cache to DAX, don't we need to flush existing
> page cache mappings and remap directly?  Or, is the thought that
> userspace needs to comprehend the presence of mixed mappings after
> changing S_DAX?

The change should be transparent to userspace. In general, I don't
expect users to change the behaviour of files that are in active use
(why would you do that?). This patch is really just introducing the
flag, the userspace API and making it propagate correctly via the
on-disk format. We'll fix up whatever problems with switching it
on/off dynamically as we go, like we do with most experimental
features once the on-disk behaviour is sorted out.

i.e. I've already got a couple of fixes we need to add to this - the
DAX flag is only valid on CRC enabled filesystems, so we need to
check that in the ioctl (general problem with using di_flags2 field,
not DAX flag specific issue). Adding a code to sync and unmap when
changing the flag is probably also necessary in the ioctl - I don't
have code to do that yet, but I have been thinking about it...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Thu Jan 21 16:24:27 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 48FF17F37
	for ; Thu, 21 Jan 2016 16:24:27 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id BDF15AC003
	for ; Thu, 21 Jan 2016 14:24:23 -0800 (PST)
X-ASG-Debug-ID: 1453415060-04cb6c0d5f475c0001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id gZBBCTG4XHB8DBDV for ; Thu, 21 Jan 2016 14:24:20 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2CDCwBTWaFWPGwtLHleKAECgw9SbYJlhXKeYgwBAQEBAQEGi1qFQYQDJIVlBAICgUBNAQEBAQEBBwEBAQFBP4Q1AQEEJxMcIxAIAw4KCSUPBSUDBxoTiBoPvl0BAQEBBgEBAQEcGIVVhTqEGxEBAoNIgQ8Fh0uHDogchUWIB4IxjFFEhXeIAYJgGIFkKC4BhWkEBRcDgRYBAQE
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 22 Jan 2016 08:54:19 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aMNeM-0004Oq-MH; Fri, 22 Jan 2016 09:24:18 +1100
Date: Fri, 22 Jan 2016 09:24:18 +1100
From: Dave Chinner 
To: Michal Hocko 
Cc: xfs@oss.sgi.com, jkosina@suse.cz,
	Hendrik Woltersdorf ,
	"Rafael J. Wysocki" 
Subject: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Message-ID: <20160121222418.GB6033@dastard>
X-ASG-Orig-Subj: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
References: <1452661968-11482-1-git-send-email-david@fromorbit.com>
 <20160120084750.GA14187@dhcp22.suse.cz>
 <20160120211944.GE20456@dastard>
 <20160121102129.GD29520@dhcp22.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160121102129.GD29520@dhcp22.suse.cz>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1453415060
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26337
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu, Jan 21, 2016 at 11:21:30AM +0100, Michal Hocko wrote:
> [CCing Rafael - the thread starts here
> http://lkml.kernel.org/r/1452661968-11482-1-git-send-email-david%40fromorbit.com]
> 
> On Thu 21-01-16 08:19:44, Dave Chinner wrote:
> > On Wed, Jan 20, 2016 at 09:47:50AM +0100, Michal Hocko wrote:
> > > On Wed 13-01-16 16:12:48, Dave Chinner wrote:
> > > > This reverts commit 24ba16bb3d499c49974669cd8429c3e4138ab102 as it
> > > > prevents machines from suspending. This regression occurs when the
> > > > xfsaild is idle on entry to suspend, and so there s no activity to
> > > > wake it from it's idle sleep and hence see that it is supposed to
> > > > freeze. Hence the freezer times out waiting for it and suspend is
> > > > cancelled.
> > > > 
> > > > There is no obvious fix for this short of freezing the filesystem
> > > > properly, so revert this change for now.
> > > 
> > > We had a similar report opensuse bugzilla just recently. I believe the
> > > proper fix should be the following:
> > ....
> > > diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> > > index aa67339b9537..d6c9c3e9e02b 100644
> > > --- a/fs/xfs/xfs_trans_ail.c
> > > +++ b/fs/xfs/xfs_trans_ail.c
> > > @@ -520,14 +520,14 @@ xfsaild(
> > >  		if (!xfs_ail_min(ailp) &&
> > >  		    ailp->xa_target == ailp->xa_target_prev) {
> > >  			spin_unlock(&ailp->xa_lock);
> > > -			schedule();
> > > +			freezable_schedule();
> > 
> > Oh, wonderful. A completely separate set of undocumented schedule
> > functions defined inside the freezer header files that are needed
> > just for freezable kthreads.
> 
> Yeah, I am not really happy about them either.
> 
> > Perhaps there should be some documentation somewhere on how to write
> > suspend safe kthreads?
> 
> There is something in Documentation/power/freezing-of-tasks.txt but
> freezable_schedule* is not mentioned there. Prhaps something like the
> following would make it more clear?

When I did "git grep freezable_schedule" nothing showed up in the
Documentation directory. I'd suggest that the entire API needs
reference in Documentation/scheduler (i.e. all the
freezable_schedule_timeout variants, the wait_event stuff, etc) as
well as in the freezer documentation so that git grep finds them in
the appropriate places....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From dan.j.williams@intel.com  Thu Jan 21 16:53:10 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 9B6AC7F37
	for ; Thu, 21 Jan 2016 16:53:10 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 081C4AC002
	for ; Thu, 21 Jan 2016 14:53:09 -0800 (PST)
X-ASG-Debug-ID: 1453416787-04cb6c0d5d4a310001-NocioJ
Received: from mail-yk0-f174.google.com (mail-yk0-f174.google.com [209.85.160.174]) by cuda.sgi.com with ESMTP id pH6j0iwEB6cpyjnB (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 21 Jan 2016 14:53:08 -0800 (PST)
X-Barracuda-Envelope-From: dan.j.williams@intel.com
X-Barracuda-Apparent-Source-IP: 209.85.160.174
Received: by mail-yk0-f174.google.com with SMTP id s126so41702451ykf.2
        for ; Thu, 21 Jan 2016 14:53:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=intel-com.20150623.gappssmtp.com; s=20150623;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        bh=WRt1KTIo5BUq7WRyiFygKHFhLkTM3n0uzsjmiz61hD0=;
        b=owmuK5CHMqgI/okY4iM2RazPFFtFs8uOcfZyPlK0Gw2rRnr37gxCmriBCyDMuB1Kv0
         7Wfw69pcKW7McJbMjD4wlqspgds65X0mcZcoFoYPA2vLIYsAoDrOFBBI2d5r8/IViGUz
         uZPyAL08570Wmelf1t+DZuO5mkX2w6QAO0tMrIjTHiDfGslU1qn7NMHnRdPsAVUufvt4
         uoERi34nV+HjOGDjMHks7NlIFNsG1tjII5uM3hvkVCYCJoc+xM45vnOb/DETxAjGpw6L
         y5GbJ20xzSBK3jZwOmK+cJoSApjLl0c22bejCbT9aOT2jrEz8wOLajfvYWR273z2qna9
         MyeA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=WRt1KTIo5BUq7WRyiFygKHFhLkTM3n0uzsjmiz61hD0=;
        b=I+NMEt90+jSK/tecOP0xBsXnoOaIV4gZGrNpUaaXfJfcz/ubjYcJhVI7rDXnaQ7rDB
         KUd95Kv00LwzY73y+rOalu/crzc9CdMasPGj2+c7zWB48Nb0mWKlKGWUg75vQC+QnTkG
         vRCaj5xfMBX2eNirnRvFWBvgXhfOcrHoYK+/dnHn98yA/HlY1GAbSGZQszMVA1l7XVg/
         mo52kY8iLXRYfBVvCYlgQ0JV0qZLpcj3y2uyDc48L7bslG6arkfhpV2X5/YsnTD5gJLr
         FN20Ycu/2Hxn7yg6biaVpS+5hjZK2C8k8xIPY08SGnmBgx2LbeOAgJLtFfrZ5vjDgEfq
         fwAA==
X-Gm-Message-State: ALoCoQn1ya9luk+jlPJ93BiQ8qEnBhBIlCvVUpSLimObSV4SumPuoNTif4ki8DGLJfjl5g6UEt5Ez20+73pErV00ra1Y3m4OdBgpN9HW6C1uOJYh6zmPUZg=
MIME-Version: 1.0
X-Received: by 10.37.64.134 with SMTP id n128mr14529329yba.51.1453416786926;
 Thu, 21 Jan 2016 14:53:06 -0800 (PST)
Received: by 10.37.202.20 with HTTP; Thu, 21 Jan 2016 14:53:06 -0800 (PST)
In-Reply-To: <20160121215820.GA6033@dastard>
References: <1451886892-15548-1-git-send-email-david@fromorbit.com>
	<1451886892-15548-4-git-send-email-david@fromorbit.com>
	
	<20160121215820.GA6033@dastard>
Date: Thu, 21 Jan 2016 14:53:06 -0800
Message-ID: 
Subject: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
From: Dan Williams 
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
To: Dave Chinner 
Cc: XFS Developers , linux-fsdevel , ext4@vger.kernel.org, 
	Matthew Wilcox 
Content-Type: text/plain; charset=UTF-8
X-Barracuda-Connect: mail-yk0-f174.google.com[209.85.160.174]
X-Barracuda-Start-Time: 1453416788
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26338
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

On Thu, Jan 21, 2016 at 1:58 PM, Dave Chinner  wrote:
> On Thu, Jan 21, 2016 at 08:37:11AM -0800, Dan Williams wrote:
>> On Sun, Jan 3, 2016 at 9:54 PM, Dave Chinner  wrote:
>> > From: Dave Chinner 
>> >
>> > Rather than just being able to turn DAX on and off via a mount
>> > option, some applications may only want to enable DAX for certain
>> > performance critical files in a filesystem.
>> >
>> > This patch introduces a new inode flag to enable DAX in the v3 inode
>> > di_flags2 field. It adds support for setting and clearing flags in
>> > the di_flags2 field via the XFS_IOC_FSSETXATTR ioctl, and sets the
>> > S_DAX inode flag appropriately when it is seen.
>> >
>> > When this flag is set on a directory, it acts as an "inherit flag".
>> > That is, inodes created in the directory will automatically inherit
>> > the on-disk inode DAX flag, enabling administrators to set up
>> > directory heirarchies that automatically use DAX. Setting this flag
>> > on an empty root directory will make the entire filesystem use DAX
>> > by default.
>>
>> When switching from page-cache to DAX, don't we need to flush existing
>> page cache mappings and remap directly?  Or, is the thought that
>> userspace needs to comprehend the presence of mixed mappings after
>> changing S_DAX?
>
> The change should be transparent to userspace. In general, I don't
> expect users to change the behaviour of files that are in active use
> (why would you do that?).

If by accident someone tries to dynamically change S_DAX while
existing mappings are established I think the kernel should just
return EBUSY.  I was not proposing we support it as a first-class
operation.

> This patch is really just introducing the
> flag, the userspace API and making it propagate correctly via the
> on-disk format. We'll fix up whatever problems with switching it
> on/off dynamically as we go, like we do with most experimental
> features once the on-disk behaviour is sorted out.

Ok.

> i.e. I've already got a couple of fixes we need to add to this - the
> DAX flag is only valid on CRC enabled filesystems,

I assume for torn-write protection?  The CRC limitation makes sense,
but we theoretically could get the same effect by using a separate
logdev that does not tear writes, right?

> so we need to
> check that in the ioctl (general problem with using di_flags2 field,
> not DAX flag specific issue). Adding a code to sync and unmap when
> changing the flag is probably also necessary in the ioctl - I don't
> have code to do that yet, but I have been thinking about it...

Matthew and I have also talked about a modification of mincore(2) to
interrogate the effective mapping mode.  It seems we'll need that or
something like it given the growing list of caveats with setting up a
DAX mapping.

From david@fromorbit.com  Thu Jan 21 17:22:35 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 96C6C7F37
	for ; Thu, 21 Jan 2016 17:22:35 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 68155304032
	for ; Thu, 21 Jan 2016 15:22:32 -0800 (PST)
X-ASG-Debug-ID: 1453418549-04bdf04d6c4b230001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id J5PNrAOpvZej3ZGr for ; Thu, 21 Jan 2016 15:22:29 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DODAAzZ6FWPGwtLHleKAECgw9SbYJlg3qBeJ5WAgoMAQEBAQEBBoENik2FQYQDGoVvAgIBAYFATQEBAQEBAQcBAQEBQT+EdRwjGCQ0BSUDBy2IGr8GGIVViWqEVgWHVYsdhAOFRYgHgWeERIhVAoVriFGEXCguhWoBHwOBFgEBAQ
Received: from ppp121-44-45-108.lns20.syd4.internode.on.net (HELO dastard) ([121.44.45.108])
  by ipmail06.adl2.internode.on.net with ESMTP; 22 Jan 2016 09:52:28 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aMOYd-0004Z7-KC; Fri, 22 Jan 2016 10:22:27 +1100
Date: Fri, 22 Jan 2016 10:22:27 +1100
From: Dave Chinner 
To: torvalds@linux-foundation.org
Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org,
	xfs@oss.sgi.com, linux-ext4@vger.kernel.org
Subject: [GIT PULL] xfs: update #2 for 4.5-rc1
Message-ID: <20160121232227.GH20456@dastard>
X-ASG-Orig-Subj: [GIT PULL] xfs: update #2 for 4.5-rc1
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1453418549
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26339
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Hi Linus,

Can you please pull the changes from the tag below? This is the
second update for XFS that I mentioned in the original pull request
last week. It contains a revert for a suspend regression in 4.4 and
a fix for a long standing log recovery issue that has been further
exposed by all the log recovery changes made in the original 4.5
merge.

There is one more thing in this pull request - one that I forgot to
merge into the origin. That is, pulling the XFS_IOC_FS[GS]ETXATTR
ioctl up to the VFS level so that other filesystems can also use it
for modifying project quota IDs. This allows the new ext4 project
quota functionality to work with the XFs quota tools to manage
project quotas, which means the ext4 project quota code can be
tested against all the xfstests project quota tests that are written
around xfs_quota. The original branch I posted has been used by Ted
in the ext4 tree, so I have not modified it to add SOBs, etc. I've
added those tags in the summary for the tag for this pull request,
hence they shouldn't ever get lost.

Thanks!

-Dave.

The following changes since commit dde7f55bd000696acc38296c21241971e1840142:

  Merge branch 'xfs-misc-fixes-for-4.5-2' into for-next (2016-01-12 07:04:30 +1100)

are available in the git repository at:


  git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git tags/xfs-for-linus-4.5-2

for you to fetch changes up to ee3804d9f94c5a391c66386b70b9fe5a58775507:

  Merge branch 'xfs-misc-fixes-for-4.5-3' into for-next (2016-01-19 08:28:36 +1100)

----------------------------------------------------------------

xfs: Update 2 for 4.5-rc1

This update contains:

o promotion of XFS_IOC_FS[GS]ETXATTR ioctl to the vfs level so that
  it can be shared with other filesystems. The ext4 project quota
  functionality is the first target for this. The commits in this
  series have not been updated with review or final SOB tags because
  the branch they were originally published in was needed by ext4.
  Those tags are:

  Reviewed-by: Theodore Ts'o 
  Signed-off-by: Dave Chinner 

o Revert a change that is causing suspend failures.
o Fix a use-after-free that can occur on log mount failures. Been
  around forever, but now exposed by other changes to log recovery
  made in the first 4.5 merge.

----------------------------------------------------------------
Dave Chinner (7):
      fs: XFS_IOC_FS[SG]SETXATTR to FS_IOC_FS[SG]ETXATTR promotion
      xfs: use FS_XFLAG definitions directly
      xfs: introduce per-inode DAX enablement
      Merge branch 'xfs-setxattr-promotion' into for-next
      Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
      xfs: log mount failures don't wait for buffers to be released
      Merge branch 'xfs-misc-fixes-for-4.5-3' into for-next

 fs/xfs/libxfs/xfs_format.h |   11 +++++-
 fs/xfs/libxfs/xfs_fs.h     |   38 +-----------------
 fs/xfs/xfs_buf.c           |   10 +++++
 fs/xfs/xfs_inode.c         |   60 ++++++++++++++++++-----------
 fs/xfs/xfs_ioctl.c         |   92 ++++++++++++++++++++++++++------------------
 fs/xfs/xfs_iops.c          |    4 +-
 fs/xfs/xfs_trans_ail.c     |    1 -
 include/uapi/linux/fs.h    |   33 ++++++++++++++++
 8 files changed, 147 insertions(+), 102 deletions(-)
-- 
Dave Chinner
david@fromorbit.com

From darrick.wong@oracle.com  Thu Jan 21 18:45:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id C47A07F37
	for ; Thu, 21 Jan 2016 18:45:41 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id B2713304039
	for ; Thu, 21 Jan 2016 16:45:38 -0800 (PST)
X-ASG-Debug-ID: 1453423535-04cb6c0d5e513f0001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id zf43vGVYA0WiDXkn (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 21 Jan 2016 16:45:36 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0M0jXM3010463
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 22 Jan 2016 00:45:34 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0M0jXhP007351
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 22 Jan 2016 00:45:33 GMT
Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0M0jXH2004481;
	Fri, 22 Jan 2016 00:45:33 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 21 Jan 2016 16:45:33 -0800
Date: Thu, 21 Jan 2016 16:45:32 -0800
From: "Darrick J. Wong" 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, linux-fsdevel@vger.kernel.org, ext4@vger.kernel.org
Subject: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
Message-ID: <20160122004532.GB5500@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
References: <1451886892-15548-1-git-send-email-david@fromorbit.com>
 <1451886892-15548-4-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1451886892-15548-4-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453423536
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26340
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Mon, Jan 04, 2016 at 04:54:52PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> Rather than just being able to turn DAX on and off via a mount
> option, some applications may only want to enable DAX for certain
> performance critical files in a filesystem.
> 
> This patch introduces a new inode flag to enable DAX in the v3 inode
> di_flags2 field. It adds support for setting and clearing flags in
> the di_flags2 field via the XFS_IOC_FSSETXATTR ioctl, and sets the
> S_DAX inode flag appropriately when it is seen.
> 
> When this flag is set on a directory, it acts as an "inherit flag".
> That is, inodes created in the directory will automatically inherit
> the on-disk inode DAX flag, enabling administrators to set up
> directory heirarchies that automatically use DAX. Setting this flag
> on an empty root directory will make the entire filesystem use DAX
> by default.
> 
> Signed-off-by: Dave Chinner 
> ---
>  fs/xfs/libxfs/xfs_format.h |  9 +++++++++
>  fs/xfs/xfs_inode.c         | 32 +++++++++++++++++++++++---------
>  fs/xfs/xfs_ioctl.c         | 18 +++++++++++++++++-
>  fs/xfs/xfs_iops.c          |  4 ++--
>  include/uapi/linux/fs.h    |  1 +
>  5 files changed, 52 insertions(+), 12 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_format.h b/fs/xfs/libxfs/xfs_format.h
> index f28eeab..b4ae7ce 100644
> --- a/fs/xfs/libxfs/xfs_format.h
> +++ b/fs/xfs/libxfs/xfs_format.h
> @@ -1024,6 +1024,15 @@ static inline void xfs_dinode_put_rdev(struct xfs_dinode *dip, xfs_dev_t rdev)
>  	 XFS_DIFLAG_EXTSZINHERIT | XFS_DIFLAG_NODEFRAG | XFS_DIFLAG_FILESTREAM)
>  
>  /*
> + * Values for di_flags2 These start by being exposed to userspace in the upper

"Values for di_flags2.  These are exposed to userspace in the..."

> + * 16 bits of the XFS_XFLAG_s range.
> + */
> +#define XFS_DIFLAG2_DAX_BIT	0	/* use DAX for this inode */
> +#define XFS_DIFLAG2_DAX		(1 << XFS_DIFLAG2_DAX_BIT)
> +
> +#define XFS_DIFLAG2_ANY		(XFS_DIFLAG2_DAX)

Heh, I was gonna use bit 0 for reflink and bit 1 for cowextszhint.  Well, I'll
move them up by one.

"cowextszhint"... in which language is that a word? ;)

--D

> +
> +/*
>   * Inode number format:
>   * low inopblog bits - offset in block
>   * next agblklog bits - block number in ag
> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> index ca9ca5a..8929908 100644
> --- a/fs/xfs/xfs_inode.c
> +++ b/fs/xfs/xfs_inode.c
> @@ -610,7 +610,9 @@ __xfs_iflock(
>  
>  STATIC uint
>  _xfs_dic2xflags(
> -	__uint16_t		di_flags)
> +	__uint16_t		di_flags,
> +	uint64_t		di_flags2,
> +	bool			has_attr)
>  {
>  	uint			flags = 0;
>  
> @@ -645,25 +647,32 @@ _xfs_dic2xflags(
>  			flags |= FS_XFLAG_FILESTREAM;
>  	}
>  
> +	if (di_flags2 & XFS_DIFLAG2_ANY) {
> +		if (di_flags2 & XFS_DIFLAG2_DAX)
> +			flags |= FS_XFLAG_DAX;
> +	}
> +
> +	if (has_attr)
> +		flags |= FS_XFLAG_HASATTR;
> +
>  	return flags;
>  }
>  
>  uint
>  xfs_ip2xflags(
> -	xfs_inode_t		*ip)
> +	struct xfs_inode	*ip)
>  {
> -	xfs_icdinode_t		*dic = &ip->i_d;
> +	struct xfs_icdinode	*dic = &ip->i_d;
>  
> -	return _xfs_dic2xflags(dic->di_flags) |
> -				(XFS_IFORK_Q(ip) ? FS_XFLAG_HASATTR : 0);
> +	return _xfs_dic2xflags(dic->di_flags, dic->di_flags2, XFS_IFORK_Q(ip));
>  }
>  
>  uint
>  xfs_dic2xflags(
> -	xfs_dinode_t		*dip)
> +	struct xfs_dinode	*dip)
>  {
> -	return _xfs_dic2xflags(be16_to_cpu(dip->di_flags)) |
> -				(XFS_DFORK_Q(dip) ? FS_XFLAG_HASATTR : 0);
> +	return _xfs_dic2xflags(be16_to_cpu(dip->di_flags),
> +				be64_to_cpu(dip->di_flags2), XFS_DFORK_Q(dip));
>  }
>  
>  /*
> @@ -862,7 +871,8 @@ xfs_ialloc(
>  	case S_IFREG:
>  	case S_IFDIR:
>  		if (pip && (pip->i_d.di_flags & XFS_DIFLAG_ANY)) {
> -			uint	di_flags = 0;
> +			uint64_t	di_flags2 = 0;
> +			uint		di_flags = 0;
>  
>  			if (S_ISDIR(mode)) {
>  				if (pip->i_d.di_flags & XFS_DIFLAG_RTINHERIT)
> @@ -898,7 +908,11 @@ xfs_ialloc(
>  				di_flags |= XFS_DIFLAG_NODEFRAG;
>  			if (pip->i_d.di_flags & XFS_DIFLAG_FILESTREAM)
>  				di_flags |= XFS_DIFLAG_FILESTREAM;
> +			if (pip->i_d.di_flags2 & XFS_DIFLAG2_DAX)
> +				di_flags2 |= XFS_DIFLAG2_DAX;
> +
>  			ip->i_d.di_flags |= di_flags;
> +			ip->i_d.di_flags2 |= di_flags2;
>  		}
>  		/* FALLTHROUGH */
>  	case S_IFLNK:
> diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
> index 94b35eb3..478d04e 100644
> --- a/fs/xfs/xfs_ioctl.c
> +++ b/fs/xfs/xfs_ioctl.c
> @@ -945,6 +945,7 @@ xfs_set_diflags(
>  	unsigned int		xflags)
>  {
>  	unsigned int		di_flags;
> +	uint64_t		di_flags2;
>  
>  	/* can't set PREALLOC this way, just preserve it */
>  	di_flags = (ip->i_d.di_flags & XFS_DIFLAG_PREALLOC);
> @@ -977,8 +978,18 @@ xfs_set_diflags(
>  		if (xflags & FS_XFLAG_EXTSIZE)
>  			di_flags |= XFS_DIFLAG_EXTSIZE;
>  	}
> -
>  	ip->i_d.di_flags = di_flags;
> +
> +	/* diflags2 only valid for v3 inodes. */
> +	if (ip->i_d.di_version < 3)
> +		return;
> +
> +	di_flags2 = 0;
> +	if (xflags & FS_XFLAG_DAX)
> +		di_flags2 |= XFS_DIFLAG2_DAX;
> +
> +	ip->i_d.di_flags2 = di_flags2;
> +
>  }
>  
>  STATIC void
> @@ -1004,6 +1015,11 @@ xfs_diflags_to_linux(
>  		inode->i_flags |= S_NOATIME;
>  	else
>  		inode->i_flags &= ~S_NOATIME;
> +	if (xflags & FS_XFLAG_DAX)
> +		inode->i_flags |= S_DAX;
> +	else
> +		inode->i_flags &= ~S_DAX;
> +
>  }
>  
>  static int
> diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
> index 245268a..a1b8af1 100644
> --- a/fs/xfs/xfs_iops.c
> +++ b/fs/xfs/xfs_iops.c
> @@ -1201,8 +1201,8 @@ xfs_diflags_to_iflags(
>  		inode->i_flags |= S_SYNC;
>  	if (flags & XFS_DIFLAG_NOATIME)
>  		inode->i_flags |= S_NOATIME;
> -	/* XXX: Also needs an on-disk per inode flag! */
> -	if (ip->i_mount->m_flags & XFS_MOUNT_DAX)
> +	if (ip->i_mount->m_flags & XFS_MOUNT_DAX ||
> +	    ip->i_d.di_flags2 & XFS_DIFLAG2_DAX)
>  		inode->i_flags |= S_DAX;
>  }
>  
> diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> index df175dd..4cad4c8 100644
> --- a/include/uapi/linux/fs.h
> +++ b/include/uapi/linux/fs.h
> @@ -138,6 +138,7 @@ struct fsxattr {
>  #define FS_XFLAG_EXTSZINHERIT	0x00001000	/* inherit inode extent size */
>  #define FS_XFLAG_NODEFRAG	0x00002000	/* do not defragment */
>  #define FS_XFLAG_FILESTREAM	0x00004000	/* use filestream allocator */
> +#define FS_XFLAG_DAX		0x00008000	/* use DAX for IO */
>  #define FS_XFLAG_HASATTR	0x80000000	/* no DIFLAG for this	*/
>  
>  /* the read-only stuff doesn't really belong here, but any other place is
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From sandeen@sandeen.net  Thu Jan 21 22:07:28 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 9E3627F37
	for ; Thu, 21 Jan 2016 22:07:28 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 6CBAD304059
	for ; Thu, 21 Jan 2016 20:07:28 -0800 (PST)
X-ASG-Debug-ID: 1453435646-04bdf04d6c50620001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 5EBvY08jLJvELGk3 for ; Thu, 21 Jan 2016 20:07:26 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 12A9960DDB99; Thu, 21 Jan 2016 22:07:26 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 1/7] quota: remove unused cmd argument from quota_quotaon()
Date: Thu, 21 Jan 2016 22:07:18 -0600
X-ASG-Orig-Subj: [PATCH 1/7] quota: remove unused cmd argument from quota_quotaon()
Message-Id: <1453435644-32261-2-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453435646
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26343
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

The cmd argument to quota_quotaon() via Q_QUOTAON quotactl
is not used, so remove it.

Signed-off-by: Eric Sandeen 
---
 fs/quota/quota.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/quota/quota.c b/fs/quota/quota.c
index 3746367..ea66670 100644
--- a/fs/quota/quota.c
+++ b/fs/quota/quota.c
@@ -79,7 +79,7 @@ unsigned int qtype_enforce_flag(int type)
 	return 0;
 }
 
-static int quota_quotaon(struct super_block *sb, int type, int cmd, qid_t id,
+static int quota_quotaon(struct super_block *sb, int type, qid_t id,
 		         struct path *path)
 {
 	if (!sb->s_qcop->quota_on && !sb->s_qcop->quota_enable)
@@ -659,7 +659,7 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
 
 	switch (cmd) {
 	case Q_QUOTAON:
-		return quota_quotaon(sb, type, cmd, id, path);
+		return quota_quotaon(sb, type, id, path);
 	case Q_QUOTAOFF:
 		return quota_quotaoff(sb, type);
 	case Q_GETFMT:
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:07:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id D5A697F47
	for ; Thu, 21 Jan 2016 22:07:28 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 961FA8F8049
	for ; Thu, 21 Jan 2016 20:07:28 -0800 (PST)
X-ASG-Debug-ID: 1453435646-04bdf04d6b50620001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id ooopnpGfq1lPK9YQ for ; Thu, 21 Jan 2016 20:07:26 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 235CC60DDB9A; Thu, 21 Jan 2016 22:07:26 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Date: Thu, 21 Jan 2016 22:07:19 -0600
X-ASG-Orig-Subj: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Message-Id: <1453435644-32261-3-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453435646
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26343
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
will return quota information for the id equal to or greater
than the id requested.  In other words, if the requested id has
no quota, the command will return quota information for the
next higher id which does have a quota set.  If no higher id
has an active quota, -ESRCH is returned.

This allows filesystems to do efficient iteration in kernelspace,
much like extN filesystems do in userspace when asked to report
all active quotas.

The patch adds a d_id field to struct qc_dqblk so that we can
pass back the id of the quota which was found, and return it
to userspace.

Today, filesystems such as XFS require getpwent-style iterations,
and for systems which have i.e. LDAP backends, this can be very
slow, or even impossible if iteration is not allowed in the
configuration.

Signed-off-by: Eric Sandeen 
---
 fs/quota/quota.c               |   30 ++++++++++++++++++++++++++++++
 include/linux/quota.h          |    3 +++
 include/uapi/linux/dqblk_xfs.h |    1 +
 3 files changed, 34 insertions(+), 0 deletions(-)

diff --git a/fs/quota/quota.c b/fs/quota/quota.c
index ea66670..4bf8d40 100644
--- a/fs/quota/quota.c
+++ b/fs/quota/quota.c
@@ -33,6 +33,7 @@ static int check_quotactl_permission(struct super_block *sb, int type, int cmd,
 	/* allow to query information for dquots we "own" */
 	case Q_GETQUOTA:
 	case Q_XGETQUOTA:
+	case Q_XGETNEXTQUOTA:
 		if ((type == USRQUOTA && uid_eq(current_euid(), make_kuid(current_user_ns(), id))) ||
 		    (type == GRPQUOTA && in_egroup_p(make_kgid(current_user_ns(), id))))
 			break;
@@ -625,6 +626,32 @@ static int quota_getxquota(struct super_block *sb, int type, qid_t id,
 	return ret;
 }
 
+/*
+ * Return quota for next active quota >= this id, if any exists,
+ * otherwise return -ESRCH via ->get_nextdqblk.
+ */
+static int quota_getnextxquota(struct super_block *sb, int type, qid_t id,
+			    void __user *addr)
+{
+	struct fs_disk_quota fdq;
+	struct qc_dqblk qdq;
+	struct kqid qid;
+	int ret;
+
+	if (!sb->s_qcop->get_nextdqblk)
+		return -ENOSYS;
+	qid = make_kqid(current_user_ns(), type, id);
+	if (!qid_valid(qid))
+		return -EINVAL;
+	ret = sb->s_qcop->get_nextdqblk(sb, qid, &qdq);
+	if (ret)
+		return ret;
+	copy_to_xfs_dqblk(&fdq, &qdq, type, qdq.d_id);
+	if (copy_to_user(addr, &fdq, sizeof(fdq)))
+		return -EFAULT;
+	return ret;
+}
+
 static int quota_rmxquota(struct super_block *sb, void __user *addr)
 {
 	__u32 flags;
@@ -690,6 +717,8 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
 		return quota_setxquota(sb, type, id, addr);
 	case Q_XGETQUOTA:
 		return quota_getxquota(sb, type, id, addr);
+	case Q_XGETNEXTQUOTA:
+		return quota_getnextxquota(sb, type, id, addr);
 	case Q_XQUOTASYNC:
 		if (sb->s_flags & MS_RDONLY)
 			return -EROFS;
@@ -712,6 +741,7 @@ static int quotactl_cmd_write(int cmd)
 	case Q_XGETQSTAT:
 	case Q_XGETQSTATV:
 	case Q_XGETQUOTA:
+	case Q_XGETNEXTQUOTA:
 	case Q_XQUOTASYNC:
 		return 0;
 	}
diff --git a/include/linux/quota.h b/include/linux/quota.h
index b2505ac..ac79c18 100644
--- a/include/linux/quota.h
+++ b/include/linux/quota.h
@@ -327,6 +327,7 @@ struct path;
 
 /* Structure for communicating via ->get_dqblk() & ->set_dqblk() */
 struct qc_dqblk {
+	u32 d_id;		/* user/group/project ID for this quota */
 	int d_fieldmask;	/* mask of fields to change in ->set_dqblk() */
 	u64 d_spc_hardlimit;	/* absolute limit on used space */
 	u64 d_spc_softlimit;	/* preferred limit on used space */
@@ -425,6 +426,8 @@ struct quotactl_ops {
 	int (*quota_sync)(struct super_block *, int);
 	int (*set_info)(struct super_block *, int, struct qc_info *);
 	int (*get_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *);
+	int (*get_nextdqblk)(struct super_block *, struct kqid,
+			     struct qc_dqblk *);
 	int (*set_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *);
 	int (*get_state)(struct super_block *, struct qc_state *);
 	int (*rm_xquota)(struct super_block *, unsigned int);
diff --git a/include/uapi/linux/dqblk_xfs.h b/include/uapi/linux/dqblk_xfs.h
index dcd75cc..11b3b31 100644
--- a/include/uapi/linux/dqblk_xfs.h
+++ b/include/uapi/linux/dqblk_xfs.h
@@ -39,6 +39,7 @@
 #define Q_XQUOTARM	XQM_CMD(6)	/* free disk space used by dquots */
 #define Q_XQUOTASYNC	XQM_CMD(7)	/* delalloc flush, updates dquots */
 #define Q_XGETQSTATV	XQM_CMD(8)	/* newer version of get quota */
+#define Q_XGETNEXTQUOTA	XQM_CMD(9)	/* get disk limits and usage >= ID */
 
 /*
  * fs_disk_quota structure:
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:07:30 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 7409B7F4E
	for ; Thu, 21 Jan 2016 22:07:29 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 63ADA8F8033
	for ; Thu, 21 Jan 2016 20:07:29 -0800 (PST)
X-ASG-Debug-ID: 1453435647-04cb6c0d5f5dbe0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 5ni2yE5dLzzcDmzj for ; Thu, 21 Jan 2016 20:07:27 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 78FA560DDB9F; Thu, 21 Jan 2016 22:07:26 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 7/7] xfs: wire up Q_XGETNEXTQUOTA / get_nextdqblk
Date: Thu, 21 Jan 2016 22:07:24 -0600
X-ASG-Orig-Subj: [PATCH 7/7] xfs: wire up Q_XGETNEXTQUOTA / get_nextdqblk
Message-Id: <1453435644-32261-8-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453435647
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26343
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Add code to allow the Q_XGETNEXTQUOTA quotactl to quickly find
all active quotas by examining the quota inode, and skipping
over unallocated or uninitialized regions.

Userspace can then use this interface rather than i.e. a
getpwent() loop when asked to report all active quotas.

Signed-off-by: Eric Sandeen 
---
 fs/xfs/libxfs/xfs_quota_defs.h |    3 +-
 fs/xfs/xfs_dquot.c             |   96 ++++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_qm.h                |    2 +-
 fs/xfs/xfs_qm_syscalls.c       |    6 ++-
 fs/xfs/xfs_quotaops.c          |   22 +++++++++-
 5 files changed, 124 insertions(+), 5 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h
index f51078f..8eed512 100644
--- a/fs/xfs/libxfs/xfs_quota_defs.h
+++ b/fs/xfs/libxfs/xfs_quota_defs.h
@@ -37,7 +37,7 @@ typedef __uint16_t	xfs_qwarncnt_t;
 #define XFS_DQ_PROJ		0x0002		/* project quota */
 #define XFS_DQ_GROUP		0x0004		/* a group quota */
 #define XFS_DQ_DIRTY		0x0008		/* dquot is dirty */
-#define XFS_DQ_FREEING		0x0010		/* dquot is beeing torn down */
+#define XFS_DQ_FREEING		0x0010		/* dquot is being torn down */
 
 #define XFS_DQ_ALLTYPES		(XFS_DQ_USER|XFS_DQ_PROJ|XFS_DQ_GROUP)
 
@@ -116,6 +116,7 @@ typedef __uint16_t	xfs_qwarncnt_t;
 #define XFS_QMOPT_DQREPAIR	0x0001000 /* repair dquot if damaged */
 #define XFS_QMOPT_GQUOTA	0x0002000 /* group dquot requested */
 #define XFS_QMOPT_ENOSPC	0x0004000 /* enospc instead of edquot (prj) */
+#define XFS_QMOPT_DQNEXT	0x0008000 /* return next dquot >= this ID */
 
 /*
  * flags to xfs_trans_mod_dquot to indicate which field needs to be
diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 5dbde0d..3b93e94 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -686,6 +686,56 @@ error0:
 }
 
 /*
+ * Advance to the next id in the current chunk, or if at the
+ * end of the chunk, skip ahead to first id in next allocated chunk
+ * using the SEEK_DATA interface.
+ */
+int
+xfs_dq_get_next_id(
+	xfs_mount_t		*mp,
+	uint			type,
+	xfs_dqid_t		*id,
+	loff_t			eof)
+{
+	struct xfs_inode	*quotip;
+	xfs_fsblock_t		start;
+	loff_t			offset;
+	uint			lock;
+	xfs_dqid_t		next_id;
+	int			error = 0;
+
+	/* Simple advance */
+	next_id = *id + 1;
+
+	/* If new ID is within the current chunk, advancing it sufficed */
+	if (next_id % mp->m_quotainfo->qi_dqperchunk) {
+		*id = next_id;
+		return 0;
+	}
+
+	/* Nope, next_id is now past the current chunk, so find the next one */
+	start = (xfs_fsblock_t)next_id / mp->m_quotainfo->qi_dqperchunk;
+
+	quotip = xfs_quota_inode(mp, type);
+	lock = xfs_ilock_data_map_shared(quotip);
+
+	offset = __xfs_seek_hole_data(VFS_I(quotip), XFS_FSB_TO_B(mp, start),
+				      eof, SEEK_DATA);
+	if (offset < 0)
+		error = offset;
+
+	xfs_iunlock(quotip, lock);
+
+	/* -ENXIO is essentially "no more data" */
+	if (error)
+		return (error == -ENXIO ? -ESRCH : error);
+
+	/* Convert next data offset back to a quota id */
+	*id = XFS_B_TO_FSB(mp, offset) * mp->m_quotainfo->qi_dqperchunk;
+	return 0;
+}
+
+/*
  * Given the file system, inode OR id, and type (UDQUOT/GDQUOT), return a
  * a locked dquot, doing an allocation (if requested) as needed.
  * When both an inode and an id are given, the inode's id takes precedence.
@@ -705,6 +755,7 @@ xfs_qm_dqget(
 	struct xfs_quotainfo	*qi = mp->m_quotainfo;
 	struct radix_tree_root *tree = xfs_dquot_tree(qi, type);
 	struct xfs_dquot	*dqp;
+	loff_t			eof = 0;
 	int			error;
 
 	ASSERT(XFS_IS_QUOTA_RUNNING(mp));
@@ -732,6 +783,21 @@ xfs_qm_dqget(
 	}
 #endif
 
+	/* Get the end of the quota file if we need it */
+	if (flags & XFS_QMOPT_DQNEXT) {
+		struct xfs_inode	*quotip;
+		xfs_fileoff_t		last;
+		uint			lock_mode;
+
+		quotip = xfs_quota_inode(mp, type);
+		lock_mode = xfs_ilock_data_map_shared(quotip);
+		error = xfs_bmap_last_offset(quotip, &last, XFS_DATA_FORK);
+		xfs_iunlock(quotip, lock_mode);
+		if (error)
+			return error;
+		eof = XFS_FSB_TO_B(mp, last);
+	}
+
 restart:
 	mutex_lock(&qi->qi_tree_lock);
 	dqp = radix_tree_lookup(tree, id);
@@ -745,6 +811,18 @@ restart:
 			goto restart;
 		}
 
+		/* uninit / unused quota found in radix tree, keep looking  */
+		if (flags & XFS_QMOPT_DQNEXT) {
+			if (XFS_IS_DQUOT_UNINITIALIZED(dqp)) {
+				xfs_dqunlock(dqp);
+				mutex_unlock(&qi->qi_tree_lock);
+				error = xfs_dq_get_next_id(mp, type, &id, eof);
+				if (error)
+					return error;
+				goto restart;
+			}
+		}
+
 		dqp->q_nrefs++;
 		mutex_unlock(&qi->qi_tree_lock);
 
@@ -771,6 +849,13 @@ restart:
 	if (ip)
 		xfs_ilock(ip, XFS_ILOCK_EXCL);
 
+	/* If we are asked to find next active id, keep looking */
+	if (error == -ENOENT && (flags & XFS_QMOPT_DQNEXT)) {
+		error = xfs_dq_get_next_id(mp, type, &id, eof);
+		if (!error)
+			goto restart;
+	}
+
 	if (error)
 		return error;
 
@@ -821,6 +906,17 @@ restart:
 	qi->qi_dquots++;
 	mutex_unlock(&qi->qi_tree_lock);
 
+	/* If we are asked to find next active id, keep looking */
+	if (flags & XFS_QMOPT_DQNEXT) {
+		if (XFS_IS_DQUOT_UNINITIALIZED(dqp)) {
+			xfs_qm_dqput(dqp);
+			error = xfs_dq_get_next_id(mp, type, &id, eof);
+			if (error)
+				return error;
+			goto restart;
+		}
+	}
+
  dqret:
 	ASSERT((ip == NULL) || xfs_isilocked(ip, XFS_ILOCK_EXCL));
 	trace_xfs_dqget_miss(dqp);
diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
index 8901a01..193a9c2 100644
--- a/fs/xfs/xfs_qm.h
+++ b/fs/xfs/xfs_qm.h
@@ -165,7 +165,7 @@ extern void		xfs_qm_dqrele_all_inodes(struct xfs_mount *, uint);
 /* quota ops */
 extern int		xfs_qm_scall_trunc_qfiles(struct xfs_mount *, uint);
 extern int		xfs_qm_scall_getquota(struct xfs_mount *, xfs_dqid_t,
-					uint, struct qc_dqblk *);
+					uint, struct qc_dqblk *, uint);
 extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
 					struct qc_dqblk *);
 extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
index 3640c6e..961420c 100644
--- a/fs/xfs/xfs_qm_syscalls.c
+++ b/fs/xfs/xfs_qm_syscalls.c
@@ -637,7 +637,8 @@ xfs_qm_scall_getquota(
 	struct xfs_mount	*mp,
 	xfs_dqid_t		id,
 	uint			type,
-	struct qc_dqblk		*dst)
+	struct qc_dqblk		*dst,
+	uint			dqget_flags)
 {
 	struct xfs_dquot	*dqp;
 	int			error;
@@ -647,7 +648,7 @@ xfs_qm_scall_getquota(
 	 * we aren't passing the XFS_QMOPT_DOALLOC flag. If it doesn't
 	 * exist, we'll get ENOENT back.
 	 */
-	error = xfs_qm_dqget(mp, NULL, id, type, 0, &dqp);
+	error = xfs_qm_dqget(mp, NULL, id, type, dqget_flags, &dqp);
 	if (error)
 		return error;
 
@@ -661,6 +662,7 @@ xfs_qm_scall_getquota(
 	}
 
 	memset(dst, 0, sizeof(*dst));
+	dst->d_id = be32_to_cpu(dqp->q_core.d_id);
 	dst->d_spc_hardlimit =
 		XFS_FSB_TO_B(mp, be64_to_cpu(dqp->q_core.d_blk_hardlimit));
 	dst->d_spc_softlimit =
diff --git a/fs/xfs/xfs_quotaops.c b/fs/xfs/xfs_quotaops.c
index 7795e0d..d576443 100644
--- a/fs/xfs/xfs_quotaops.c
+++ b/fs/xfs/xfs_quotaops.c
@@ -238,7 +238,26 @@ xfs_fs_get_dqblk(
 		return -ESRCH;
 
 	return xfs_qm_scall_getquota(mp, from_kqid(&init_user_ns, qid),
-				      xfs_quota_type(qid.type), qdq);
+				      xfs_quota_type(qid.type), qdq, 0);
+}
+
+/* Return quota info for active quota >= this qid */
+STATIC int
+xfs_fs_get_nextdqblk(
+	struct super_block	*sb,
+	struct kqid		qid,
+	struct qc_dqblk		*qdq)
+{
+	struct xfs_mount	*mp = XFS_M(sb);
+
+	if (!XFS_IS_QUOTA_RUNNING(mp))
+		return -ENOSYS;
+	if (!XFS_IS_QUOTA_ON(mp))
+		return -ESRCH;
+
+	return xfs_qm_scall_getquota(mp, from_kqid(&init_user_ns, qid),
+				     xfs_quota_type(qid.type), qdq,
+				     XFS_QMOPT_DQNEXT);
 }
 
 STATIC int
@@ -267,5 +286,6 @@ const struct quotactl_ops xfs_quotactl_operations = {
 	.quota_disable		= xfs_quota_disable,
 	.rm_xquota		= xfs_fs_rm_xquota,
 	.get_dqblk		= xfs_fs_get_dqblk,
+	.get_nextdqblk		= xfs_fs_get_nextdqblk,
 	.set_dqblk		= xfs_fs_set_dqblk,
 };
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:07:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 039347F37
	for ; Thu, 21 Jan 2016 22:07:30 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 7983CAC001
	for ; Thu, 21 Jan 2016 20:07:29 -0800 (PST)
X-ASG-Debug-ID: 1453435647-04cbb026fe53e70001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id xxzUGXmQhgAi6iEr for ; Thu, 21 Jan 2016 20:07:27 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 65B5160DDB9E; Thu, 21 Jan 2016 22:07:26 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 6/7] xfs: Factor xfs_seek_hole_data into helper
Date: Thu, 21 Jan 2016 22:07:23 -0600
X-ASG-Orig-Subj: [PATCH 6/7] xfs: Factor xfs_seek_hole_data into helper
Message-Id: <1453435644-32261-7-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453435647
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26343
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Factor xfs_seek_hole_data into an unlocked helper which takes
an xfs inode rather than a file for internal use.

Also allow specification of "end" - the vfs lseek interface is
defined such that any offset past eof/i_size shall return -ENXIO,
but we will use this for quota code which does not maintain i_size,
and we want to be able to SEEK_DATA past i_size as well.  So the
lseek path can send in i_size, and the quota code can determine
its own ending offset.

Signed-off-by: Eric Sandeen 
---
 fs/xfs/xfs_file.c  |   82 ++++++++++++++++++++++++++++++++++++----------------
 fs/xfs/xfs_inode.h |    2 +
 2 files changed, 59 insertions(+), 25 deletions(-)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index ebe9b82..5dc7113 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -1337,31 +1337,31 @@ out:
 	return found;
 }
 
-STATIC loff_t
-xfs_seek_hole_data(
-	struct file		*file,
+/*
+ * caller must lock inode with xfs_ilock_data_map_shared,
+ * can we craft an appropriate ASSERT?
+ *
+ * end is because the VFS-level lseek interface is defined such that any
+ * offset past i_size shall return -ENXIO, but we use this for quota code
+ * which does not maintain i_size, and we want to SEEK_DATA past i_size.
+ */
+loff_t
+__xfs_seek_hole_data(
+	struct inode		*inode,
 	loff_t			start,
+	loff_t			end,
 	int			whence)
 {
-	struct inode		*inode = file->f_mapping->host;
 	struct xfs_inode	*ip = XFS_I(inode);
 	struct xfs_mount	*mp = ip->i_mount;
 	loff_t			uninitialized_var(offset);
-	xfs_fsize_t		isize;
 	xfs_fileoff_t		fsbno;
-	xfs_filblks_t		end;
-	uint			lock;
+	xfs_filblks_t		lastbno;
 	int			error;
 
-	if (XFS_FORCED_SHUTDOWN(mp))
-		return -EIO;
-
-	lock = xfs_ilock_data_map_shared(ip);
-
-	isize = i_size_read(inode);
-	if (start >= isize) {
+	if (start >= end) {
 		error = -ENXIO;
-		goto out_unlock;
+		goto out_error;
 	}
 
 	/*
@@ -1369,22 +1369,22 @@ xfs_seek_hole_data(
 	 * by fsbno to the end block of the file.
 	 */
 	fsbno = XFS_B_TO_FSBT(mp, start);
-	end = XFS_B_TO_FSB(mp, isize);
+	lastbno = XFS_B_TO_FSB(mp, end);
 
 	for (;;) {
 		struct xfs_bmbt_irec	map[2];
 		int			nmap = 2;
 		unsigned int		i;
 
-		error = xfs_bmapi_read(ip, fsbno, end - fsbno, map, &nmap,
+		error = xfs_bmapi_read(ip, fsbno, lastbno - fsbno, map, &nmap,
 				       XFS_BMAPI_ENTIRE);
 		if (error)
-			goto out_unlock;
+			goto out_error;
 
 		/* No extents at given offset, must be beyond EOF */
 		if (nmap == 0) {
 			error = -ENXIO;
-			goto out_unlock;
+			goto out_error;
 		}
 
 		for (i = 0; i < nmap; i++) {
@@ -1426,7 +1426,7 @@ xfs_seek_hole_data(
 			 * hole at the end of any file).
 		 	 */
 			if (whence == SEEK_HOLE) {
-				offset = isize;
+				offset = end;
 				break;
 			}
 			/*
@@ -1434,7 +1434,7 @@ xfs_seek_hole_data(
 			 */
 			ASSERT(whence == SEEK_DATA);
 			error = -ENXIO;
-			goto out_unlock;
+			goto out_error;
 		}
 
 		ASSERT(i > 1);
@@ -1445,14 +1445,14 @@ xfs_seek_hole_data(
 		 */
 		fsbno = map[i - 1].br_startoff + map[i - 1].br_blockcount;
 		start = XFS_FSB_TO_B(mp, fsbno);
-		if (start >= isize) {
+		if (start >= end) {
 			if (whence == SEEK_HOLE) {
-				offset = isize;
+				offset = end;
 				break;
 			}
 			ASSERT(whence == SEEK_DATA);
 			error = -ENXIO;
-			goto out_unlock;
+			goto out_error;
 		}
 	}
 
@@ -1464,7 +1464,39 @@ out:
 	 * situation in particular.
 	 */
 	if (whence == SEEK_HOLE)
-		offset = min_t(loff_t, offset, isize);
+		offset = min_t(loff_t, offset, end);
+
+	return offset;
+
+out_error:
+	return error;
+}
+
+STATIC loff_t
+xfs_seek_hole_data(
+	struct file		*file,
+	loff_t			start,
+	int			whence)
+{
+	struct inode		*inode = file->f_mapping->host;
+	struct xfs_inode	*ip = XFS_I(inode);
+	struct xfs_mount	*mp = ip->i_mount;
+	uint			lock;
+	loff_t			offset, end;
+	int			error = 0;
+
+	if (XFS_FORCED_SHUTDOWN(mp))
+		return -EIO;
+
+	lock = xfs_ilock_data_map_shared(ip);
+
+	end = i_size_read(inode);
+	offset = __xfs_seek_hole_data(inode, start, end, whence);
+	if (offset < 0) {
+		error = offset;
+		goto out_unlock;
+	}
+
 	offset = vfs_setpos(file, offset, inode->i_sb->s_maxbytes);
 
 out_unlock:
diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
index ca9e119..ed7e933 100644
--- a/fs/xfs/xfs_inode.h
+++ b/fs/xfs/xfs_inode.h
@@ -437,6 +437,8 @@ int	xfs_update_prealloc_flags(struct xfs_inode *ip,
 int	xfs_zero_eof(struct xfs_inode *ip, xfs_off_t offset,
 		     xfs_fsize_t isize, bool *did_zeroing);
 int	xfs_iozero(struct xfs_inode *ip, loff_t pos, size_t count);
+loff_t	__xfs_seek_hole_data(struct inode *inode, loff_t start,
+			     loff_t eof, int whence);
 
 
 /* from xfs_iops.c */
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:07:32 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 13FE67F47
	for ; Thu, 21 Jan 2016 22:07:31 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id B77E8AC003
	for ; Thu, 21 Jan 2016 20:07:30 -0800 (PST)
X-ASG-Debug-ID: 1453435646-04bdf04d6d50620001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id V1UZi6xqkVpLtHAY for ; Thu, 21 Jan 2016 20:07:26 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 014DA60DDB98; Thu, 21 Jan 2016 22:07:25 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 0/7 V2] quota: add new quotactl Q_GETNEXTQUOTA
Date: Thu, 21 Jan 2016 22:07:17 -0600
X-ASG-Orig-Subj: [PATCH 0/7 V2] quota: add new quotactl Q_GETNEXTQUOTA
Message-Id: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453435646
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26343
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

This adds a new quotactl, Q_GETNEXTQUOTA.

Q_GETNEXTQUOTA is exactly like Q_GETQUOTA, except that it will
return quota information for the id equal to or greater than
the id requested.  In other words, if the specified id has
no quota, the command will return quota information for the
next higher id which does have a quota set.  If no higher id
has an active quota, -ESRCH is returned.

So if you ask for id X, you can get back quota for id X,
id X+N, or -ESRCH if no higher id has a quota.

This allows filesystems to do efficient iteration in kernelspace,
much like extN filesystems do in userspace when asked to report
all active quotas.

Today, filesystems such as XFS require getpwent()-style iterations,
and for systems which have i.e. LDAP backends, this can be very
slow, or even impossible if iteration is not allowed in the
configuration.

Patches 1 and 4 are just small fixups that turned up along the way;
2 and 3 add the actual quota plumbing, and the rest are xfs-specific
to allow xfs to support this new interface.

For non-xfs quota, this does require a new structure which is
able to pass back the discovered ID along with the quota info.
For xfs-quota, the id is already present in the structure.

I sent a regression test to the fstests list today, and will
send updated xfsprogs & linux-quota patches soon.

Thanks,
-Eric

From sandeen@sandeen.net  Thu Jan 21 22:07:32 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 962077F54
	for ; Thu, 21 Jan 2016 22:07:31 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 842498F8033
	for ; Thu, 21 Jan 2016 20:07:28 -0800 (PST)
X-ASG-Debug-ID: 1453435646-04cb6c0d5e5dbd0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id ilSaKGMbyxc9WMqK for ; Thu, 21 Jan 2016 20:07:26 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 31F1860DDB9B; Thu, 21 Jan 2016 22:07:26 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
Date: Thu, 21 Jan 2016 22:07:20 -0600
X-ASG-Orig-Subj: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
Message-Id: <1453435644-32261-4-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453435646
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26343
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Q_GETNEXTQUOTA is exactly like Q_GETQUOTA, except that it
will return quota information for the id equal to or greater
than the id requested.  In other words, if the requested id has
no quota, the command will return quota information for the
next higher id which does have a quota set.  If no higher id
has an active quota, -ESRCH is returned.

This allows filesystems to do efficient iteration in kernelspace,
much like extN filesystems do in userspace when asked to report
all active quotas.

This does require a new data structure for userspace, as the
current structure does not include an ID for the returned quota
information.

Today, Ext4 with a hidden quota inode requires getpwent-style
iterations, and for systems which have i.e. LDAP backends,
this can be very slow, or even impossible if iteration is not
allowed in the configuration.

Signed-off-by: Eric Sandeen 
---
 fs/quota/compat.c          |   27 +++++++++++++++++++++++++++
 fs/quota/quota.c           |   31 +++++++++++++++++++++++++++++++
 include/uapi/linux/quota.h |   14 ++++++++++++++
 3 files changed, 72 insertions(+), 0 deletions(-)

diff --git a/fs/quota/compat.c b/fs/quota/compat.c
index fb1892f..80773a4 100644
--- a/fs/quota/compat.c
+++ b/fs/quota/compat.c
@@ -19,6 +19,19 @@ struct compat_if_dqblk {
 	compat_uint_t dqb_valid;
 };
 
+struct compat_if_nextdqblk {
+	compat_u64 dqb_bhardlimit;
+	compat_u64 dqb_bsoftlimit;
+	compat_u64 dqb_curspace;
+	compat_u64 dqb_ihardlimit;
+	compat_u64 dqb_isoftlimit;
+	compat_u64 dqb_curinodes;
+	compat_u64 dqb_btime;
+	compat_u64 dqb_itime;
+	compat_uint_t dqb_valid;
+	compat_uint_t dqb_id;
+};
+
 /* XFS structures */
 struct compat_fs_qfilestat {
 	compat_u64 dqb_bhardlimit;
@@ -46,6 +59,8 @@ asmlinkage long sys32_quotactl(unsigned int cmd, const char __user *special,
 	unsigned int cmds;
 	struct if_dqblk __user *dqblk;
 	struct compat_if_dqblk __user *compat_dqblk;
+	struct if_nextdqblk __user *nxtdqblk;
+	struct compat_if_nextdqblk __user *compat_nextdqblk;
 	struct fs_quota_stat __user *fsqstat;
 	struct compat_fs_quota_stat __user *compat_fsqstat;
 	compat_uint_t data;
@@ -66,6 +81,18 @@ asmlinkage long sys32_quotactl(unsigned int cmd, const char __user *special,
 			put_user(data, &compat_dqblk->dqb_valid))
 			ret = -EFAULT;
 		break;
+	case Q_GETNEXTQUOTA:
+		nxtdqblk = compat_alloc_user_space(sizeof(struct if_nextdqblk));
+		compat_nextdqblk = addr;
+		ret = sys_quotactl(cmd, special, id, nxtdqblk);
+		if (ret)
+			break;
+		if (copy_in_user(compat_nextdqblk, nxtdqblk,
+				 sizeof(*compat_nextdqblk)) ||
+			get_user(data, &nxtdqblk->dqb_valid) ||
+			put_user(data, &compat_nextdqblk->dqb_valid))
+			ret = -EFAULT;
+		break;
 	case Q_SETQUOTA:
 		dqblk = compat_alloc_user_space(sizeof(struct if_dqblk));
 		compat_dqblk = addr;
diff --git a/fs/quota/quota.c b/fs/quota/quota.c
index 4bf8d40..85d8f7b 100644
--- a/fs/quota/quota.c
+++ b/fs/quota/quota.c
@@ -32,6 +32,7 @@ static int check_quotactl_permission(struct super_block *sb, int type, int cmd,
 		break;
 	/* allow to query information for dquots we "own" */
 	case Q_GETQUOTA:
+	case Q_GETNEXTQUOTA:
 	case Q_XGETQUOTA:
 	case Q_XGETNEXTQUOTA:
 		if ((type == USRQUOTA && uid_eq(current_euid(), make_kuid(current_user_ns(), id))) ||
@@ -218,6 +219,34 @@ static int quota_getquota(struct super_block *sb, int type, qid_t id,
 	if (ret)
 		return ret;
 	copy_to_if_dqblk(&idq, &fdq);
+	if (copy_to_user(addr, &idq, sizeof(struct if_dqblk)))
+		return -EFAULT;
+	return 0;
+}
+
+/*
+ * Return quota for next active quota >= this id, if any exists,
+ * otherwise return -ESRCH via ->get_nextdqblk
+ */
+static int quota_getnextquota(struct super_block *sb, int type, qid_t id,
+			  void __user *addr)
+{
+	struct kqid qid;
+	struct qc_dqblk fdq;
+	struct if_nextdqblk idq;
+	int ret;
+
+	if (!sb->s_qcop->get_nextdqblk)
+		return -ENOSYS;
+	qid = make_kqid(current_user_ns(), type, id);
+	if (!qid_valid(qid))
+		return -EINVAL;
+	ret = sb->s_qcop->get_nextdqblk(sb, qid, &fdq);
+	if (ret)
+		return ret;
+	/* struct if_nextdqblk is a superset of struct if_dqblk */
+	copy_to_if_dqblk((struct if_dqblk *)&idq, &fdq);
+	idq.dqb_id = fdq.d_id;
 	if (copy_to_user(addr, &idq, sizeof(idq)))
 		return -EFAULT;
 	return 0;
@@ -697,6 +726,8 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
 		return quota_setinfo(sb, type, addr);
 	case Q_GETQUOTA:
 		return quota_getquota(sb, type, id, addr);
+	case Q_GETNEXTQUOTA:
+		return quota_getnextquota(sb, type, id, addr);
 	case Q_SETQUOTA:
 		return quota_setquota(sb, type, id, addr);
 	case Q_SYNC:
diff --git a/include/uapi/linux/quota.h b/include/uapi/linux/quota.h
index 9c95b2c..38baddb 100644
--- a/include/uapi/linux/quota.h
+++ b/include/uapi/linux/quota.h
@@ -71,6 +71,7 @@
 #define Q_SETINFO  0x800006	/* set information about quota files */
 #define Q_GETQUOTA 0x800007	/* get user quota structure */
 #define Q_SETQUOTA 0x800008	/* set user quota structure */
+#define Q_GETNEXTQUOTA 0x800009	/* get disk limits and usage >= ID */
 
 /* Quota format type IDs */
 #define	QFMT_VFS_OLD 1
@@ -119,6 +120,19 @@ struct if_dqblk {
 	__u32 dqb_valid;
 };
 
+struct if_nextdqblk {
+	__u64 dqb_bhardlimit;
+	__u64 dqb_bsoftlimit;
+	__u64 dqb_curspace;
+	__u64 dqb_ihardlimit;
+	__u64 dqb_isoftlimit;
+	__u64 dqb_curinodes;
+	__u64 dqb_btime;
+	__u64 dqb_itime;
+	__u32 dqb_valid;
+	__u32 dqb_id;
+};
+
 /*
  * Structure used for setting quota information about file via quotactl
  * Following flags are used to specify which fields are valid
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:07:33 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id B87777F55
	for ; Thu, 21 Jan 2016 22:07:31 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 57C88AC001
	for ; Thu, 21 Jan 2016 20:07:28 -0800 (PST)
X-ASG-Debug-ID: 1453435646-04bdf04d6a50620001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 0CzJktgl96cgxzeq for ; Thu, 21 Jan 2016 20:07:26 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 4357B60DDB9C; Thu, 21 Jan 2016 22:07:26 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 4/7] xfs: don't overflow quota ID when initializing dqblk
Date: Thu, 21 Jan 2016 22:07:21 -0600
X-ASG-Orig-Subj: [PATCH 4/7] xfs: don't overflow quota ID when initializing dqblk
Message-Id: <1453435644-32261-5-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453435646
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26343
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Quota IDs are unsigned, and so we can pass in values up
to 2^32-1.  But if we try to initialize a block containing
values over MAX_INT, curid will overflow and assert.

curid holds a quota ID, so give it the proper
xfs_dqid_t type (and remove the now-impossible ASSERT).

Signed-off-by: Eric Sandeen 
---
 fs/xfs/xfs_dquot.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 9c44d38..c9c7c2d 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -232,7 +232,8 @@ xfs_qm_init_dquot_blk(
 {
 	struct xfs_quotainfo	*q = mp->m_quotainfo;
 	xfs_dqblk_t	*d;
-	int		curid, i;
+	xfs_dqid_t	curid;
+	int		i;
 
 	ASSERT(tp);
 	ASSERT(xfs_buf_islocked(bp));
@@ -243,7 +244,6 @@ xfs_qm_init_dquot_blk(
 	 * ID of the first dquot in the block - id's are zero based.
 	 */
 	curid = id - (id % q->qi_dqperchunk);
-	ASSERT(curid >= 0);
 	memset(d, 0, BBTOB(q->qi_dqchunklen));
 	for (i = 0; i < q->qi_dqperchunk; i++, d++, curid++) {
 		d->dd_diskdq.d_magic = cpu_to_be16(XFS_DQUOT_MAGIC);
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:07:28 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id B86467F3F
	for ; Thu, 21 Jan 2016 22:07:28 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id A7D3E304059
	for ; Thu, 21 Jan 2016 20:07:28 -0800 (PST)
X-ASG-Debug-ID: 1453435647-04cbb026ff53e70001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id F78jcd8MGQzIZWOy for ; Thu, 21 Jan 2016 20:07:27 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 54A2A60DDB9D; Thu, 21 Jan 2016 22:07:26 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 5/7] xfs: get quota inode from mp & flags rather than dqp
Date: Thu, 21 Jan 2016 22:07:22 -0600
X-ASG-Orig-Subj: [PATCH 5/7] xfs: get quota inode from mp & flags rather than dqp
Message-Id: <1453435644-32261-6-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453435647
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26343
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Allow us to get the appropriate quota inode from any
mp & quota flags, not necessarily associated with a
particular dqp.  Needed for when we are searching for
the next active ID with quotas and we want to examine
the quota inode.

Signed-off-by: Eric Sandeen 
---
 fs/xfs/xfs_dquot.c |    3 ++-
 fs/xfs/xfs_qm.h    |   10 +++++-----
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index c9c7c2d..5dbde0d 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -464,12 +464,13 @@ xfs_qm_dqtobp(
 	struct xfs_bmbt_irec	map;
 	int			nmaps = 1, error;
 	struct xfs_buf		*bp;
-	struct xfs_inode	*quotip = xfs_dq_to_quota_inode(dqp);
+	struct xfs_inode	*quotip;
 	struct xfs_mount	*mp = dqp->q_mount;
 	xfs_dqid_t		id = be32_to_cpu(dqp->q_core.d_id);
 	struct xfs_trans	*tp = (tpp ? *tpp : NULL);
 	uint			lock_mode;
 
+	quotip = xfs_quota_inode(dqp->q_mount, dqp->dq_flags);
 	dqp->q_fileoffset = (xfs_fileoff_t)id / mp->m_quotainfo->qi_dqperchunk;
 
 	lock_mode = xfs_ilock_data_map_shared(quotip);
diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
index 996a040..8901a01 100644
--- a/fs/xfs/xfs_qm.h
+++ b/fs/xfs/xfs_qm.h
@@ -104,15 +104,15 @@ xfs_dquot_tree(
 }
 
 static inline struct xfs_inode *
-xfs_dq_to_quota_inode(struct xfs_dquot *dqp)
+xfs_quota_inode(xfs_mount_t *mp, uint dq_flags)
 {
-	switch (dqp->dq_flags & XFS_DQ_ALLTYPES) {
+	switch (dq_flags & XFS_DQ_ALLTYPES) {
 	case XFS_DQ_USER:
-		return dqp->q_mount->m_quotainfo->qi_uquotaip;
+		return mp->m_quotainfo->qi_uquotaip;
 	case XFS_DQ_GROUP:
-		return dqp->q_mount->m_quotainfo->qi_gquotaip;
+		return mp->m_quotainfo->qi_gquotaip;
 	case XFS_DQ_PROJ:
-		return dqp->q_mount->m_quotainfo->qi_pquotaip;
+		return mp->m_quotainfo->qi_pquotaip;
 	default:
 		ASSERT(0);
 	}
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:27:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id A97297F37
	for ; Thu, 21 Jan 2016 22:27:13 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 32DB0AC00B
	for ; Thu, 21 Jan 2016 20:27:13 -0800 (PST)
X-ASG-Debug-ID: 1453436831-04cb6c0d605ef00001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id mPTQ91zBvm2hDt2E for ; Thu, 21 Jan 2016 20:27:11 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 5AA6E60DDB96
	for ; Thu, 21 Jan 2016 22:27:11 -0600 (CST)
To: xfs@oss.sgi.com
From: Eric Sandeen 
Subject: [PATCH 0/3] xfsprogs: use Q_XGETNEXTQUOTA
Message-ID: <56A1AF9F.80303@sandeen.net>
X-ASG-Orig-Subj: [PATCH 0/3] xfsprogs: use Q_XGETNEXTQUOTA
Date: Thu, 21 Jan 2016 22:27:11 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453436831
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26344
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Patch series to make xfs_quota's repquota and quota dump use the new
Q_XGETNEXTQUOTA interface.

-Eric

From sandeen@sandeen.net  Thu Jan 21 22:27:47 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 4A60F7F37
	for ; Thu, 21 Jan 2016 22:27:47 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 37F0E304039
	for ; Thu, 21 Jan 2016 20:27:47 -0800 (PST)
X-ASG-Debug-ID: 1453436865-04cbb026fc544f0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id C4VbGMC3wsGGpXK4 for ; Thu, 21 Jan 2016 20:27:45 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 8DB0760DDB96
	for ; Thu, 21 Jan 2016 22:27:45 -0600 (CST)
Subject: [PATCH 1/3] xfs_quota: define Q_XGETNEXTQUOTA
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 1/3] xfs_quota: define Q_XGETNEXTQUOTA
References: <56A1AF9F.80303@sandeen.net>
From: Eric Sandeen 
Message-ID: <56A1AFC1.5090403@sandeen.net>
Date: Thu, 21 Jan 2016 22:27:45 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <56A1AF9F.80303@sandeen.net>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453436865
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26344
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

This simply defines the Q_XGETNEXTQUOTA quotactl in xfsprogs.

Signed-off-by: Eric Sandeen 
---
 include/xqm.h |    1 +
 quota/linux.c |    2 ++
 quota/quota.h |    2 ++
 3 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/include/xqm.h b/include/xqm.h
index 47f58a0..8357c9d 100644
--- a/include/xqm.h
+++ b/include/xqm.h
@@ -32,6 +32,7 @@
 #define Q_XGETQSTAT	XQM_CMD(5)	/* get quota subsystem status */
 #define Q_XQUOTARM	XQM_CMD(6)	/* free disk space used by dquots */
 #define Q_XQUOTASYNC	XQM_CMD(7)	/* delalloc flush, updates dquots */
+#define Q_XGETNEXTQUOTA	XQM_CMD(9)	/* get disk limits and usage */
 
 /*
  * fs_disk_quota structure:
diff --git a/quota/linux.c b/quota/linux.c
index 8a73922..74dba01 100644
--- a/quota/linux.c
+++ b/quota/linux.c
@@ -49,6 +49,8 @@ xcommand_to_qcommand(
 		return Q_XQUOTAOFF;
 	case XFS_GETQUOTA:
 		return Q_XGETQUOTA;
+	case XFS_GETNEXTQUOTA:
+		return Q_XGETNEXTQUOTA;
 	case XFS_SETQLIM:
 		return Q_XSETQLIM;
 	case XFS_GETQSTAT:
diff --git a/quota/quota.h b/quota/quota.h
index 416f335..2bbc176 100644
--- a/quota/quota.h
+++ b/quota/quota.h
@@ -43,6 +43,8 @@ enum {
 	XFS_GETQSTAT,	/* get quota subsystem status */
 	XFS_QUOTARM,	/* free disk space used by dquots */
 	XFS_QSYNC,	/* flush delayed allocate space */
+	XFS_GETQSTATV,	/* newer version of quota stats */
+	XFS_GETNEXTQUOTA, /* get disk limits and usage */
 };
 
 /*
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:28:25 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=T_FILL_THIS_FORM_SHORT
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 97ED07F37
	for ; Thu, 21 Jan 2016 22:28:25 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 17C8AAC002
	for ; Thu, 21 Jan 2016 20:28:24 -0800 (PST)
X-ASG-Debug-ID: 1453436903-04cb6c0d5f5f030001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 4YtxwcPJlMpEClts for ; Thu, 21 Jan 2016 20:28:23 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 1CF0360DDB96
	for ; Thu, 21 Jan 2016 22:28:23 -0600 (CST)
Subject: [PATCH 2/3] xfs_quota: make report_mount() & dump_file() take an
 "output id"
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 2/3] xfs_quota: make report_mount() & dump_file() take an
 "output id"
References: <56A1AF9F.80303@sandeen.net>
From: Eric Sandeen 
Message-ID: <56A1AFE7.8040604@sandeen.net>
Date: Thu, 21 Jan 2016 22:28:23 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <56A1AF9F.80303@sandeen.net>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453436903
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26344
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Allow report_mount() and dump_file() to take a *oid pointer,
an "output id" which will be filled in from the returned quota
information.  Useful if the quotactl returns an ID for something
other than that which was passed in, i.e. GETNEXTQUOTA.

Also, when printing results, print the id which was actually
returned, not the id which was passed in.

Should be a no-op change at this point; the next patch which
wires in Q_XGETNEXTQUOTA will make use of this.

Signed-off-by: Eric Sandeen 
---
 quota/report.c |   46 ++++++++++++++++++++++++++++++----------------
 1 files changed, 30 insertions(+), 16 deletions(-)

diff --git a/quota/report.c b/quota/report.c
index c77b24f..3085a9e 100644
--- a/quota/report.c
+++ b/quota/report.c
@@ -77,6 +77,7 @@ static void
 dump_file(
 	FILE		*fp,
 	uint		id,
+	uint		*oid,
 	uint		type,
 	char		*dev)
 {
@@ -87,6 +88,10 @@ dump_file(
 			perror("XFS_GETQUOTA");
 		return;
 	}
+
+	if (oid)
+		*oid = d.d_id;
+
 	if (!d.d_blk_softlimit && !d.d_blk_hardlimit &&
 	    !d.d_ino_softlimit && !d.d_ino_hardlimit &&
 	    !d.d_rtb_softlimit && !d.d_rtb_hardlimit)
@@ -94,7 +99,8 @@ dump_file(
 	fprintf(fp, "fs = %s\n", dev);
 	/* this branch is for backward compatibility reasons */
 	if (d.d_rtb_softlimit || d.d_rtb_hardlimit)
-		fprintf(fp, "%-10d %7llu %7llu %7llu %7llu %7llu %7llu\n", id,
+		fprintf(fp, "%-10d %7llu %7llu %7llu %7llu %7llu %7llu\n",
+			d.d_id,
 			(unsigned long long)d.d_blk_softlimit,
 			(unsigned long long)d.d_blk_hardlimit,
 			(unsigned long long)d.d_ino_softlimit,
@@ -102,7 +108,8 @@ dump_file(
 			(unsigned long long)d.d_rtb_softlimit,
 			(unsigned long long)d.d_rtb_hardlimit);
 	else
-		fprintf(fp, "%-10d %7llu %7llu %7llu %7llu\n", id,
+		fprintf(fp, "%-10d %7llu %7llu %7llu %7llu\n",
+			d.d_id,
 			(unsigned long long)d.d_blk_softlimit,
 			(unsigned long long)d.d_blk_hardlimit,
 			(unsigned long long)d.d_ino_softlimit,
@@ -129,7 +136,7 @@ dump_limits_any_type(
 
 	if (upper) {
 		for (id = lower; id <= upper; id++)
-			dump_file(fp, id, type, mount->fs_name);
+			dump_file(fp, id, NULL, type, mount->fs_name);
 		return;
 	}
 
@@ -138,7 +145,8 @@ dump_limits_any_type(
 			struct group *g;
 			setgrent();
 			while ((g = getgrent()) != NULL)
-				dump_file(fp, g->gr_gid, type, mount->fs_name);
+				dump_file(fp, g->gr_gid, NULL, type,
+					  mount->fs_name);
 			endgrent();
 			break;
 		}
@@ -146,7 +154,8 @@ dump_limits_any_type(
 			struct fs_project *p;
 			setprent();
 			while ((p = getprent()) != NULL)
-				dump_file(fp, p->pr_prid, type, mount->fs_name);
+				dump_file(fp, p->pr_prid, NULL, type,
+					  mount->fs_name);
 			endprent();
 			break;
 		}
@@ -154,7 +163,8 @@ dump_limits_any_type(
 			struct passwd *u;
 			setpwent();
 			while ((u = getpwent()) != NULL)
-				dump_file(fp, u->pw_uid, type, mount->fs_name);
+				dump_file(fp, u->pw_uid, NULL, type,
+					  mount->fs_name);
 			endpwent();
 			break;
 		}
@@ -291,6 +301,7 @@ report_mount(
 	FILE		*fp,
 	__uint32_t	id,
 	char		*name,
+	__uint32_t	*oid,
 	uint		form,
 	uint		type,
 	fs_path_t	*mount,
@@ -308,6 +319,9 @@ report_mount(
 		return 0;
 	}
 
+	if (oid)
+		*oid = d.d_id;
+
 	if (flags & TERSE_FLAG) {
 		count = 0;
 		if ((form & XFS_BLOCK_QUOTA) && d.d_bcount)
@@ -324,19 +338,19 @@ report_mount(
 		report_header(fp, form, type, mount, flags);
 
 	if (flags & NO_LOOKUP_FLAG) {
-		fprintf(fp, "#%-10u", id);
+		fprintf(fp, "#%-10u", d.d_id);
 	} else {
 		if (name == NULL) {
 			if (type == XFS_USER_QUOTA) {
-				struct passwd	*u = getpwuid(id);
+				struct passwd	*u = getpwuid(d.d_id);
 				if (u)
 					name = u->pw_name;
 			} else if (type == XFS_GROUP_QUOTA) {
-				struct group	*g = getgrgid(id);
+				struct group	*g = getgrgid(d.d_id);
 				if (g)
 					name = g->gr_name;
 			} else if (type == XFS_PROJ_QUOTA) {
-				fs_project_t	*p = getprprid(id);
+				fs_project_t	*p = getprprid(d.d_id);
 				if (p)
 					name = p->pr_name;
 			}
@@ -425,14 +439,14 @@ report_user_mount(
 
 	if (upper) {	/* identifier range specified */
 		for (id = lower; id <= upper; id++) {
-			if (report_mount(fp, id, NULL,
+			if (report_mount(fp, id, NULL, NULL,
 					form, XFS_USER_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
 	} else {
 		setpwent();
 		while ((u = getpwent()) != NULL) {
-			if (report_mount(fp, u->pw_uid, u->pw_name,
+			if (report_mount(fp, u->pw_uid, u->pw_name, NULL,
 					form, XFS_USER_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
@@ -457,14 +471,14 @@ report_group_mount(
 
 	if (upper) {	/* identifier range specified */
 		for (id = lower; id <= upper; id++) {
-			if (report_mount(fp, id, NULL,
+			if (report_mount(fp, id, NULL, NULL,
 					form, XFS_GROUP_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
 	} else {
 		setgrent();
 		while ((g = getgrent()) != NULL) {
-			if (report_mount(fp, g->gr_gid, g->gr_name,
+			if (report_mount(fp, g->gr_gid, g->gr_name, NULL,
 					form, XFS_GROUP_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
@@ -488,14 +502,14 @@ report_project_mount(
 
 	if (upper) {	/* identifier range specified */
 		for (id = lower; id <= upper; id++) {
-			if (report_mount(fp, id, NULL,
+			if (report_mount(fp, id, NULL, NULL,
 					form, XFS_PROJ_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
 	} else {
 		setprent();
 		while ((p = getprent()) != NULL) {
-			if (report_mount(fp, p->pr_prid, p->pr_name,
+			if (report_mount(fp, p->pr_prid, p->pr_name, NULL,
 					form, XFS_PROJ_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
-- 
1.7.1

From sandeen@sandeen.net  Thu Jan 21 22:28:58 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 32BC77F37
	for ; Thu, 21 Jan 2016 22:28:58 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 0FD07304039
	for ; Thu, 21 Jan 2016 20:28:58 -0800 (PST)
X-ASG-Debug-ID: 1453436936-04cbb026ff54560001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id sUvuHav3HCTlHrek for ; Thu, 21 Jan 2016 20:28:56 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 4458260DDB96
	for ; Thu, 21 Jan 2016 22:28:56 -0600 (CST)
Subject: [PATCH 3/3] xfs_quota: use Q_XGETNEXTQUOTA for report and dump
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 3/3] xfs_quota: use Q_XGETNEXTQUOTA for report and dump
References: <56A1AF9F.80303@sandeen.net>
From: Eric Sandeen 
Message-ID: <56A1B008.1020801@sandeen.net>
Date: Thu, 21 Jan 2016 22:28:56 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <56A1AF9F.80303@sandeen.net>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453436936
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26344
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Rather than a loop over getpwnam() etc, use the Q_XGETNEXTQUOTA
command to iterate through all active quotas in quota
reports and quota file dump.

If Q_XGETNEXTQUOTA fails, go back to the old way.

Signed-off-by: Eric Sandeen 
---


(This stuff is screaming to be refactored and made consistent,
but my first attempt left me screaming too, so I'll leave
it for another day.)


 quota/quota.h  |    1 +
 quota/report.c |   84 ++++++++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 71 insertions(+), 14 deletions(-)

diff --git a/quota/quota.h b/quota/quota.h
index 2bbc176..4bde351 100644
--- a/quota/quota.h
+++ b/quota/quota.h
@@ -74,6 +74,7 @@ enum {
 	DEFAULTS_FLAG =		0x0100,	/* use value as a default */
 	ABSOLUTE_FLAG =		0x0200, /* absolute time, not related to now */
 	NO_LOOKUP_FLAG =	0x0400, /* skip name lookups, just report ID */
+	GETNEXTQUOTA_FLAG =	0x0800, /* use getnextquota quotactl */
 };
 
 /*
diff --git a/quota/report.c b/quota/report.c
index 3085a9e..8653134 100644
--- a/quota/report.c
+++ b/quota/report.c
@@ -73,20 +73,27 @@ report_help(void)
 "\n"));
 }
 
-static void
+static int 
 dump_file(
 	FILE		*fp,
 	uint		id,
 	uint		*oid,
 	uint		type,
-	char		*dev)
+	char		*dev,
+	int		flags)
 {
 	fs_disk_quota_t	d;
+	int		cmd;
+
+	if (flags & GETNEXTQUOTA_FLAG)
+		cmd = XFS_GETNEXTQUOTA;
+	else
+		cmd = XFS_GETQUOTA;
 
-	if (xfsquotactl(XFS_GETQUOTA, dev, type, id, (void *)&d) < 0) {
+	if (xfsquotactl(cmd, dev, type, id, (void *)&d) < 0) {
 		if (errno != ENOENT && errno != ENOSYS && errno != ESRCH)
 			perror("XFS_GETQUOTA");
-		return;
+		return 0;
 	}
 
 	if (oid)
@@ -95,7 +102,7 @@ dump_file(
 	if (!d.d_blk_softlimit && !d.d_blk_hardlimit &&
 	    !d.d_ino_softlimit && !d.d_ino_hardlimit &&
 	    !d.d_rtb_softlimit && !d.d_rtb_hardlimit)
-		return;
+		return 1;
 	fprintf(fp, "fs = %s\n", dev);
 	/* this branch is for backward compatibility reasons */
 	if (d.d_rtb_softlimit || d.d_rtb_hardlimit)
@@ -114,6 +121,8 @@ dump_file(
 			(unsigned long long)d.d_blk_hardlimit,
 			(unsigned long long)d.d_ino_softlimit,
 			(unsigned long long)d.d_ino_hardlimit);
+
+	return 1;
 }
 
 static void
@@ -125,7 +134,7 @@ dump_limits_any_type(
 	uint		upper)
 {
 	fs_path_t	*mount;
-	uint		id;
+	uint		id = 0, oid;
 
 	if ((mount = fs_table_lookup(dir, FS_MOUNT_POINT)) == NULL) {
 		exitcode = 1;
@@ -134,19 +143,30 @@ dump_limits_any_type(
 		return;
 	}
 
+	/* Range was specified; query everything in it */
 	if (upper) {
 		for (id = lower; id <= upper; id++)
-			dump_file(fp, id, NULL, type, mount->fs_name);
+			dump_file(fp, id, NULL, type, mount->fs_name, 0);
 		return;
 	}
 
+	/* Use GETNEXTQUOTA if it's available */
+	if (dump_file(fp, id, &oid, type, mount->fs_name, GETNEXTQUOTA_FLAG)) {
+		id = oid + 1;
+		while (dump_file(fp, id, &oid, type, mount->fs_name,
+				 GETNEXTQUOTA_FLAG))
+			id = oid + 1;
+		return;
+        }
+
+	/* Otherwise fall back to iterating over each uid/gid/prjid */
 	switch (type) {
 	case XFS_GROUP_QUOTA: {
 			struct group *g;
 			setgrent();
 			while ((g = getgrent()) != NULL)
 				dump_file(fp, g->gr_gid, NULL, type,
-					  mount->fs_name);
+					  mount->fs_name, 0);
 			endgrent();
 			break;
 		}
@@ -155,7 +175,7 @@ dump_limits_any_type(
 			setprent();
 			while ((p = getprent()) != NULL)
 				dump_file(fp, p->pr_prid, NULL, type,
-					  mount->fs_name);
+					  mount->fs_name, 0);
 			endprent();
 			break;
 		}
@@ -164,7 +184,7 @@ dump_limits_any_type(
 			setpwent();
 			while ((u = getpwent()) != NULL)
 				dump_file(fp, u->pw_uid, NULL, type,
-					  mount->fs_name);
+					  mount->fs_name, 0);
 			endpwent();
 			break;
 		}
@@ -312,8 +332,14 @@ report_mount(
 	char		c[8], h[8], s[8];
 	uint		qflags;
 	int		count;
+	int		cmd;
+
+	if (flags & GETNEXTQUOTA_FLAG)
+		cmd = XFS_GETNEXTQUOTA;
+	else
+		cmd = XFS_GETQUOTA;
 
-	if (xfsquotactl(XFS_GETQUOTA, dev, type, id, (void *)&d) < 0) {
+	if (xfsquotactl(cmd, dev, type, id, (void *)&d) < 0) {
 		if (errno != ENOENT && errno != ENOSYS && errno != ESRCH)
 			perror("XFS_GETQUOTA");
 		return 0;
@@ -435,7 +461,7 @@ report_user_mount(
 	uint		flags)
 {
 	struct passwd	*u;
-	uint		id;
+	uint		id = 0, oid;
 
 	if (upper) {	/* identifier range specified */
 		for (id = lower; id <= upper; id++) {
@@ -443,6 +469,16 @@ report_user_mount(
 					form, XFS_USER_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
+	} else if (report_mount(fp, id, NULL, &oid, form,
+				XFS_USER_QUOTA, mount,
+				flags|GETNEXTQUOTA_FLAG)) {
+		id = oid + 1;
+		flags |= GETNEXTQUOTA_FLAG;
+		flags |= NO_HEADER_FLAG;
+		while (report_mount(fp, id, NULL, &oid, form, XFS_USER_QUOTA,
+				    mount, flags)) {
+			id = oid + 1;
+		}
 	} else {
 		setpwent();
 		while ((u = getpwent()) != NULL) {
@@ -467,7 +503,7 @@ report_group_mount(
 	uint		flags)
 {
 	struct group	*g;
-	uint		id;
+	uint		id = 0, oid;
 
 	if (upper) {	/* identifier range specified */
 		for (id = lower; id <= upper; id++) {
@@ -475,6 +511,16 @@ report_group_mount(
 					form, XFS_GROUP_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
+	} else if (report_mount(fp, id, NULL, &oid, form,
+				XFS_GROUP_QUOTA, mount,
+				flags|GETNEXTQUOTA_FLAG)) {
+		id = oid + 1;
+		flags |= GETNEXTQUOTA_FLAG;
+		flags |= NO_HEADER_FLAG;
+		while (report_mount(fp, id, NULL, &oid, form, XFS_GROUP_QUOTA,
+				    mount, flags)) {
+			id = oid + 1;
+		}
 	} else {
 		setgrent();
 		while ((g = getgrent()) != NULL) {
@@ -498,7 +544,7 @@ report_project_mount(
 	uint		flags)
 {
 	fs_project_t	*p;
-	uint		id;
+	uint		id = 0, oid;
 
 	if (upper) {	/* identifier range specified */
 		for (id = lower; id <= upper; id++) {
@@ -506,6 +552,16 @@ report_project_mount(
 					form, XFS_PROJ_QUOTA, mount, flags))
 				flags |= NO_HEADER_FLAG;
 		}
+	} else if (report_mount(fp, id, NULL, &oid, form,
+				XFS_PROJ_QUOTA, mount,
+				flags|GETNEXTQUOTA_FLAG)) {
+		id = oid + 1;
+		flags |= GETNEXTQUOTA_FLAG;
+		flags |= NO_HEADER_FLAG;
+		while (report_mount(fp, id, NULL, &oid, form, XFS_PROJ_QUOTA,
+				    mount, flags)) {
+			id = oid + 1;
+		}
 	} else {
 		setprent();
 		while ((p = getprent()) != NULL) {
-- 
1.7.1

From jack@suse.cz  Fri Jan 22 02:55:47 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id BD23B7F37
	for ; Fri, 22 Jan 2016 02:55:47 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 48A73AC001
	for ; Fri, 22 Jan 2016 00:55:44 -0800 (PST)
X-ASG-Debug-ID: 1453452937-04cb6c0d5e66140001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id wMhiRuF7EgVeEzxM (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 22 Jan 2016 00:55:38 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id C7812AC01;
	Fri, 22 Jan 2016 08:55:35 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 2F144823DA; Fri, 22 Jan 2016 09:55:49 +0100 (CET)
Date: Fri, 22 Jan 2016 09:55:49 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Message-ID: <20160122085549.GB16898@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
 <1453435644-32261-3-git-send-email-sandeen@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453435644-32261-3-git-send-email-sandeen@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453452938
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26348
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu 21-01-16 22:07:19, Eric Sandeen wrote:
> Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
> will return quota information for the id equal to or greater
> than the id requested.  In other words, if the requested id has
> no quota, the command will return quota information for the
> next higher id which does have a quota set.  If no higher id
> has an active quota, -ESRCH is returned.
> 
> This allows filesystems to do efficient iteration in kernelspace,
> much like extN filesystems do in userspace when asked to report
> all active quotas.
> 
> The patch adds a d_id field to struct qc_dqblk so that we can
> pass back the id of the quota which was found, and return it
> to userspace.
> 
> Today, filesystems such as XFS require getpwent-style iterations,
> and for systems which have i.e. LDAP backends, this can be very
> slow, or even impossible if iteration is not allowed in the
> configuration.
> 
> Signed-off-by: Eric Sandeen 
...
> diff --git a/fs/quota/quota.c b/fs/quota/quota.c
> index ea66670..4bf8d40 100644
> --- a/fs/quota/quota.c
> +++ b/fs/quota/quota.c
> @@ -33,6 +33,7 @@ static int check_quotactl_permission(struct super_block *sb, int type, int cmd,
>  	/* allow to query information for dquots we "own" */
>  	case Q_GETQUOTA:
>  	case Q_XGETQUOTA:
> +	case Q_XGETNEXTQUOTA:

IMO you should require CAP_SYS_ADMIN for the quotactl. Definitely doing the
UID and GID checks for GETNEXTQUOTA looks strange to me since the returned
structure may be for a different ID. Or did you assume that existing user
will have quota structure allocated so we always return quotas for that ID
in that case? I'm not sure this is good to rely on...

> +	ret = sb->s_qcop->get_nextdqblk(sb, qid, &qdq);
> +	if (ret)
> +	        return ret;
> +	copy_to_xfs_dqblk(&fdq, &qdq, type, qdq.d_id);
> +	if (copy_to_user(addr, &fdq, sizeof(fdq)))
> +	        return -EFAULT;
> +	return ret;
> +}

So how about passing pointer to 'qid' to ->get_nextdqblk() and return the ID
that way? That will also force you to fix the issue that you currently
completely miss user-namespace conversions for the ID ;).

I definitely dislike mixing d_id in the qdq structure with arguments of fs
callbacks and that d_id doesn't get filled for most callbacks. That is going
to cause confusion.

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Fri Jan 22 03:28:17 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 343C87F37
	for ; Fri, 22 Jan 2016 03:28:17 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 162AA304053
	for ; Fri, 22 Jan 2016 01:28:13 -0800 (PST)
X-ASG-Debug-ID: 1453454891-04cb6c0d5f66d80001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id CMCbY3D8hlluWbKF (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 22 Jan 2016 01:28:12 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id CD374ACD0;
	Fri, 22 Jan 2016 09:28:09 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id BDB3F823DA; Fri, 22 Jan 2016 10:28:22 +0100 (CET)
Date: Fri, 22 Jan 2016 10:28:22 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz,
	linux-api@vger.kernel.org
Subject: Re: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
Message-ID: <20160122092822.GC16898@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
 <1453435644-32261-4-git-send-email-sandeen@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453435644-32261-4-git-send-email-sandeen@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453454891
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26349
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu 21-01-16 22:07:20, Eric Sandeen wrote:
> Q_GETNEXTQUOTA is exactly like Q_GETQUOTA, except that it
> will return quota information for the id equal to or greater
> than the id requested.  In other words, if the requested id has
> no quota, the command will return quota information for the
> next higher id which does have a quota set.  If no higher id
> has an active quota, -ESRCH is returned.
> 
> This allows filesystems to do efficient iteration in kernelspace,
> much like extN filesystems do in userspace when asked to report
> all active quotas.
> 
> This does require a new data structure for userspace, as the
> current structure does not include an ID for the returned quota
> information.
> 
> Today, Ext4 with a hidden quota inode requires getpwent-style
> iterations, and for systems which have i.e. LDAP backends,
> this can be very slow, or even impossible if iteration is not
> allowed in the configuration.
> 
> Signed-off-by: Eric Sandeen 

Same comments as for XFS version apply here. Furthermore:

> diff --git a/fs/quota/compat.c b/fs/quota/compat.c
> index fb1892f..80773a4 100644
> --- a/fs/quota/compat.c
> +++ b/fs/quota/compat.c
> @@ -19,6 +19,19 @@ struct compat_if_dqblk {
>  	compat_uint_t dqb_valid;
>  };
>  
> +struct compat_if_nextdqblk {
> +	compat_u64 dqb_bhardlimit;
> +	compat_u64 dqb_bsoftlimit;
> +	compat_u64 dqb_curspace;
> +	compat_u64 dqb_ihardlimit;
> +	compat_u64 dqb_isoftlimit;
> +	compat_u64 dqb_curinodes;
> +	compat_u64 dqb_btime;
> +	compat_u64 dqb_itime;
> +	compat_uint_t dqb_valid;
> +	compat_uint_t dqb_id;
> +};
> +

Is there a need for compat version of this structure? Everything is
naturally aligned and the size is a multiple of 8 bytes. But these things
keep surprising me... Added CC to linux-api in a hope that there's someone
who definitely knows.

On a side note when I have API people attention: AFAIU we need
struct compat_if_dqblk (defined in fs/quota/compat.c) because its size is
not multiple of 8 bytes, right? But what is then then reason for get_user()
/ put_user() when the structure is copied into userspace (see below)?
copy_in_user() should have copied everything...

               if (copy_in_user(compat_dqblk, dqblk, sizeof(*compat_dqblk)) ||
                        get_user(data, &dqblk->dqb_valid) ||
                        put_user(data, &compat_dqblk->dqb_valid))
                        ret = -EFAULT;


								Honza
-- 
Jan Kara 
SUSE Labs, CR

From mhocko@suse.cz  Fri Jan 22 03:52:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id BC2327F37
	for ; Fri, 22 Jan 2016 03:52:40 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id AC02F8F8049
	for ; Fri, 22 Jan 2016 01:52:37 -0800 (PST)
X-ASG-Debug-ID: 1453456354-04cb6c0d5f67610001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 6CF2hhwhkaGDrEQI (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 22 Jan 2016 01:52:35 -0800 (PST)
X-Barracuda-Envelope-From: mhocko@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id A5242AC01;
	Fri, 22 Jan 2016 09:52:32 +0000 (UTC)
Date: Fri, 22 Jan 2016 10:52:33 +0100
From: Michal Hocko 
To: Dave Chinner 
Cc: xfs@oss.sgi.com, jkosina@suse.cz,
	Hendrik Woltersdorf 
Subject: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
Message-ID: <20160122095233.GA19465@dhcp22.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH] Revert "xfs: clear PF_NOFREEZE for xfsaild kthread"
References: <1452661968-11482-1-git-send-email-david@fromorbit.com>
 <20160120084750.GA14187@dhcp22.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160120084750.GA14187@dhcp22.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453456354
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26349
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Wed 20-01-16 09:47:50, Michal Hocko wrote:
> On Wed 13-01-16 16:12:48, Dave Chinner wrote:
> > This reverts commit 24ba16bb3d499c49974669cd8429c3e4138ab102 as it
> > prevents machines from suspending. This regression occurs when the
> > xfsaild is idle on entry to suspend, and so there s no activity to
> > wake it from it's idle sleep and hence see that it is supposed to
> > freeze. Hence the freezer times out waiting for it and suspend is
> > cancelled.
> > 
> > There is no obvious fix for this short of freezing the filesystem
> > properly, so revert this change for now.
> 
> We had a similar report opensuse bugzilla just recently. I believe the
> proper fix should be the following:
> ---
> From ae910a86ada86804c34cc8136afebc9fefa15813 Mon Sep 17 00:00:00 2001
> From: Michal Hocko 
> Date: Tue, 19 Jan 2016 20:28:49 +0100
> Subject: [PATCH] xfs, xfsaild: Do not block suspend
> 
> Hendik has reported suspend failures due to xfsaild blocking the freezer
> to settle down.
> Jan 17 19:59:56 linux-6380 kernel: PM: Syncing filesystems ... done.
> Jan 17 19:59:56 linux-6380 kernel: PM: Preparing system for sleep (mem)
> Jan 17 19:59:56 linux-6380 kernel: Freezing user space processes ... (elapsed 0.001 seconds) done.
> Jan 17 19:59:56 linux-6380 kernel: Freezing remaining freezable tasks ...
> Jan 17 19:59:56 linux-6380 kernel: Freezing of tasks failed after 20.002 seconds (1 tasks refusing to freeze, wq_busy=0):
> Jan 17 19:59:56 linux-6380 kernel: xfsaild/dm-5    S 00000000     0  1293      2 0x00000080
> Jan 17 19:59:56 linux-6380 kernel:  f0ef5f00 00000046 00000200 00000000 ffff9022 c02d3800 00000000 00000032
> Jan 17 19:59:56 linux-6380 kernel:  ee0b2400 00000032 f71e0d00 f36fabc0 f0ef2d00 f0ef6000 f0ef2d00 f12f90c0
> Jan 17 19:59:56 linux-6380 kernel:  f0ef5f0c c0844e44 00000000 f0ef5f6c f811e0be 00000000 00000000 f0ef2d00
> Jan 17 19:59:56 linux-6380 kernel: Call Trace:
> Jan 17 19:59:56 linux-6380 kernel:  [] schedule+0x34/0x90
> Jan 17 19:59:56 linux-6380 kernel:  [] xfsaild+0x5de/0x600 [xfs]
> Jan 17 19:59:56 linux-6380 kernel:  [] kthread+0x9b/0xb0
> Jan 17 19:59:56 linux-6380 kernel:  [] ret_from_kernel_thread+0x21/0x38
> 
> The issue has been there for quite some time but it has been made
> visible by only by 24ba16bb3d49 ("xfs: clear PF_NOFREEZE for xfsaild
> kthread") because the suspend started seeing xfsaild.
> 
> The above commit has missed that the !xfs_ail_min branch might call
> schedule with TASK_INTERRUPTIBLE without calling try_to_freeze so the pm
> suspend would wake up the kernel thread over and over again without any
> progress. What we want here is to use freezable_schedule instead to hide
> the thread from the suspend.
> 
> While we are here also change schedule_timeout to freezable variant to
> prevent from spurious wakeups by suspend.
> 
> Reported-by: Hendrik Woltersdorf 
> Signed-off-by: Michal Hocko 

Hendrik was able to test the patch so feel free to add his Tested-by
-- 
Michal Hocko
SUSE Labs

From sandeen@sandeen.net  Fri Jan 22 07:57:26 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 873547F37
	for ; Fri, 22 Jan 2016 07:57:26 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 5A6CB304032
	for ; Fri, 22 Jan 2016 05:57:22 -0800 (PST)
X-ASG-Debug-ID: 1453471040-04bdf04d6c5dbd0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id PtZAkaFpNmBpCCs7 for ; Fri, 22 Jan 2016 05:57:20 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 5AFF363737A0;
	Fri, 22 Jan 2016 07:57:20 -0600 (CST)
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
To: Jan Kara , Eric Sandeen 
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
 <1453435644-32261-3-git-send-email-sandeen@redhat.com>
 <20160122085549.GB16898@quack.suse.cz>
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com
From: Eric Sandeen 
Message-ID: <56A2353F.4000400@sandeen.net>
Date: Fri, 22 Jan 2016 07:57:19 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160122085549.GB16898@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453471040
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26353
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header


On 1/22/16 2:55 AM, Jan Kara wrote:
> On Thu 21-01-16 22:07:19, Eric Sandeen wrote:
>> Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
>> will return quota information for the id equal to or greater
>> than the id requested.  In other words, if the requested id has
>> no quota, the command will return quota information for the
>> next higher id which does have a quota set.  If no higher id
>> has an active quota, -ESRCH is returned.
>>
>> This allows filesystems to do efficient iteration in kernelspace,
>> much like extN filesystems do in userspace when asked to report
>> all active quotas.
>>
>> The patch adds a d_id field to struct qc_dqblk so that we can
>> pass back the id of the quota which was found, and return it
>> to userspace.
>>
>> Today, filesystems such as XFS require getpwent-style iterations,
>> and for systems which have i.e. LDAP backends, this can be very
>> slow, or even impossible if iteration is not allowed in the
>> configuration.
>>
>> Signed-off-by: Eric Sandeen 
> ...
>> diff --git a/fs/quota/quota.c b/fs/quota/quota.c
>> index ea66670..4bf8d40 100644
>> --- a/fs/quota/quota.c
>> +++ b/fs/quota/quota.c
>> @@ -33,6 +33,7 @@ static int check_quotactl_permission(struct super_block *sb, int type, int cmd,
>>  	/* allow to query information for dquots we "own" */
>>  	case Q_GETQUOTA:
>>  	case Q_XGETQUOTA:
>> +	case Q_XGETNEXTQUOTA:
> 
> IMO you should require CAP_SYS_ADMIN for the quotactl. Definitely doing the
> UID and GID checks for GETNEXTQUOTA looks strange to me since the returned
> structure may be for a different ID. Or did you assume that existing user
> will have quota structure allocated so we always return quotas for that ID
> in that case? I'm not sure this is good to rely on...

Oh whoops.  OK that was dumb of me, thanks for catching it.  No, I
didn't intend to rely on the asking user having a quota, it was
just a dumb mistake.  :)

>> +	ret = sb->s_qcop->get_nextdqblk(sb, qid, &qdq);
>> +	if (ret)
>> +	        return ret;
>> +	copy_to_xfs_dqblk(&fdq, &qdq, type, qdq.d_id);
>> +	if (copy_to_user(addr, &fdq, sizeof(fdq)))
>> +	        return -EFAULT;
>> +	return ret;
>> +}
> 
> So how about passing pointer to 'qid' to ->get_nextdqblk() and return the ID
> that way? That will also force you to fix the issue that you currently
> completely miss user-namespace conversions for the ID ;).

Ok.

> I definitely dislike mixing d_id in the qdq structure with arguments of fs
> callbacks and that d_id doesn't get filled for most callbacks. That is going
> to cause confusion.

Yeah, fair enough.  I'll change it.

Thanks,
-Eric

> 								Honza
> 

From sandeen@sandeen.net  Fri Jan 22 07:59:03 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id AA4D37F37
	for ; Fri, 22 Jan 2016 07:59:03 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 99CDB8F8040
	for ; Fri, 22 Jan 2016 05:59:00 -0800 (PST)
X-ASG-Debug-ID: 1453471136-04cbb026fc62720001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id YrOdz2VTrePY6BlR for ; Fri, 22 Jan 2016 05:58:57 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id E34B963737A0;
	Fri, 22 Jan 2016 07:58:56 -0600 (CST)
Subject: Re: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
To: Jan Kara , Eric Sandeen 
X-ASG-Orig-Subj: Re: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
References: <1453435644-32261-1-git-send-email-sandeen@redhat.com>
 <1453435644-32261-4-git-send-email-sandeen@redhat.com>
 <20160122092822.GC16898@quack.suse.cz>
Cc: linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, xfs@oss.sgi.com
From: Eric Sandeen 
Message-ID: <56A235A0.7060100@sandeen.net>
Date: Fri, 22 Jan 2016 07:58:56 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160122092822.GC16898@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453471136
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26353
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header


On 1/22/16 3:28 AM, Jan Kara wrote:
> On Thu 21-01-16 22:07:20, Eric Sandeen wrote:

...

> Same comments as for XFS version apply here. 

*nod*

> Furthermore:
> 
>> diff --git a/fs/quota/compat.c b/fs/quota/compat.c
>> index fb1892f..80773a4 100644
>> --- a/fs/quota/compat.c
>> +++ b/fs/quota/compat.c
>> @@ -19,6 +19,19 @@ struct compat_if_dqblk {
>>  	compat_uint_t dqb_valid;
>>  };
>>  
>> +struct compat_if_nextdqblk {
>> +	compat_u64 dqb_bhardlimit;
>> +	compat_u64 dqb_bsoftlimit;
>> +	compat_u64 dqb_curspace;
>> +	compat_u64 dqb_ihardlimit;
>> +	compat_u64 dqb_isoftlimit;
>> +	compat_u64 dqb_curinodes;
>> +	compat_u64 dqb_btime;
>> +	compat_u64 dqb_itime;
>> +	compat_uint_t dqb_valid;
>> +	compat_uint_t dqb_id;
>> +};
>> +
> 
> Is there a need for compat version of this structure? Everything is
> naturally aligned and the size is a multiple of 8 bytes. But these things
> keep surprising me... Added CC to linux-api in a hope that there's someone
> who definitely knows.

Ok, yeah, I wasn't sure.  I'll take all the compat stuff out of this one,
and can do a separate patch if needed, but I bet you're right, it's probably
not.  I did set up to test 32-compat calls, so I'll just test w/o this.

I'd be perfectly happy to drop it.  :)

Thanks for the review, Jan.

-Eric

From sandeen@sandeen.net  Fri Jan 22 12:25:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id EA8EA7F37
	for ; Fri, 22 Jan 2016 12:25:41 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id C27C3304053
	for ; Fri, 22 Jan 2016 10:25:41 -0800 (PST)
X-ASG-Debug-ID: 1453487139-04bdf04d6b64ff0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id auB3IRImohm6tiZM for ; Fri, 22 Jan 2016 10:25:39 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id D2B7C65B26CA; Fri, 22 Jan 2016 12:25:38 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 0/7 V3] quota: add new quotactl Q_GETNEXTQUOTA
Date: Fri, 22 Jan 2016 12:25:29 -0600
X-ASG-Orig-Subj: [PATCH 0/7 V3] quota: add new quotactl Q_GETNEXTQUOTA
Message-Id: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453487139
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26360
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

This adds a new quotactl, Q_GETNEXTQUOTA.

Q_GETNEXTQUOTA is exactly like Q_GETQUOTA, except that it will
return quota information for the id equal to or greater than
the id requested.  In other words, if the specified id has
no quota, the command will return quota information for the
next higher id which does have a quota set.  If no higher id
has an active quota, -ESRCH is returned.

So if you ask for id X, you can get back quota for id X,
id X+N, or -ESRCH if no higher id has a quota.

This allows filesystems to do efficient iteration in kernelspace,
much like extN filesystems do in userspace when asked to report
all active quotas.

Today, filesystems such as XFS require getpwent()-style iterations,
and for systems which have i.e. LDAP backends, this can be very
slow, or even impossible if iteration is not allowed in the
configuration.

Patches 1 and 4 are just small fixups that turned up along the way;
2 and 3 add the actual quota plumbing, and the rest are xfs-specific
to allow xfs to support this new interface.

For non-xfs quota, this does require a new structure which is
able to pass back the discovered ID along with the quota info.
For xfs-quota, the id is already present in the structure.

V3:
* Remove 32-bit compat stuff (i686/x86_64 at least works w/o it...)
* Require CAP_SYS_ADMIN for these calls
* Pass back found ID in &qid passed to ->get_nextdqblk, rather
  than modifying struct qc_dqblk
* Munge that found ID back through user-namespace conversions
  before returning it in the user structure.

Thanks,
-Eric

From sandeen@sandeen.net  Fri Jan 22 12:25:42 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 130947F47
	for ; Fri, 22 Jan 2016 12:25:42 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id DEAD9304062
	for ; Fri, 22 Jan 2016 10:25:41 -0800 (PST)
X-ASG-Debug-ID: 1453487139-04bdf04d6a64ff0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id MOMozuSj1G8o4vGy for ; Fri, 22 Jan 2016 10:25:39 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 25AA2658A362; Fri, 22 Jan 2016 12:25:39 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 4/7] xfs: don't overflow quota ID when initializing dqblk
Date: Fri, 22 Jan 2016 12:25:33 -0600
X-ASG-Orig-Subj: [PATCH 4/7] xfs: don't overflow quota ID when initializing dqblk
Message-Id: <1453487136-12681-5-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453487139
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26360
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Quota IDs are unsigned, and so we can pass in values up
to 2^32-1.  But if we try to initialize a block containing
values over MAX_INT, curid will overflow and assert.

curid holds a quota ID, so give it the proper
xfs_dqid_t type (and remove the now-impossible ASSERT).

Signed-off-by: Eric Sandeen 
---
 fs/xfs/xfs_dquot.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 9c44d38..c9c7c2d 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -232,7 +232,8 @@ xfs_qm_init_dquot_blk(
 {
 	struct xfs_quotainfo	*q = mp->m_quotainfo;
 	xfs_dqblk_t	*d;
-	int		curid, i;
+	xfs_dqid_t	curid;
+	int		i;
 
 	ASSERT(tp);
 	ASSERT(xfs_buf_islocked(bp));
@@ -243,7 +244,6 @@ xfs_qm_init_dquot_blk(
 	 * ID of the first dquot in the block - id's are zero based.
 	 */
 	curid = id - (id % q->qi_dqperchunk);
-	ASSERT(curid >= 0);
 	memset(d, 0, BBTOB(q->qi_dqchunklen));
 	for (i = 0; i < q->qi_dqperchunk; i++, d++, curid++) {
 		d->dd_diskdq.d_magic = cpu_to_be16(XFS_DQUOT_MAGIC);
-- 
1.7.1

From sandeen@sandeen.net  Fri Jan 22 12:25:42 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 27C327F4E
	for ; Fri, 22 Jan 2016 12:25:42 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 1E97C8F8033
	for ; Fri, 22 Jan 2016 10:25:42 -0800 (PST)
X-ASG-Debug-ID: 1453487139-04cbb026fc69f20001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id YMSCKhN7CHQr91HE for ; Fri, 22 Jan 2016 10:25:39 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id E6BAB678F64C; Fri, 22 Jan 2016 12:25:38 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 1/7] quota: remove unused cmd argument from quota_quotaon()
Date: Fri, 22 Jan 2016 12:25:30 -0600
X-ASG-Orig-Subj: [PATCH 1/7] quota: remove unused cmd argument from quota_quotaon()
Message-Id: <1453487136-12681-2-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453487139
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26360
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

The cmd argument to quota_quotaon() via Q_QUOTAON quotactl
is not used, so remove it.

Signed-off-by: Eric Sandeen 
---
 fs/quota/quota.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/quota/quota.c b/fs/quota/quota.c
index 3746367..ea66670 100644
--- a/fs/quota/quota.c
+++ b/fs/quota/quota.c
@@ -79,7 +79,7 @@ unsigned int qtype_enforce_flag(int type)
 	return 0;
 }
 
-static int quota_quotaon(struct super_block *sb, int type, int cmd, qid_t id,
+static int quota_quotaon(struct super_block *sb, int type, qid_t id,
 		         struct path *path)
 {
 	if (!sb->s_qcop->quota_on && !sb->s_qcop->quota_enable)
@@ -659,7 +659,7 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
 
 	switch (cmd) {
 	case Q_QUOTAON:
-		return quota_quotaon(sb, type, cmd, id, path);
+		return quota_quotaon(sb, type, id, path);
 	case Q_QUOTAOFF:
 		return quota_quotaoff(sb, type);
 	case Q_GETFMT:
-- 
1.7.1

From sandeen@sandeen.net  Fri Jan 22 12:25:42 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 464F57F50
	for ; Fri, 22 Jan 2016 12:25:42 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 2DC83304053
	for ; Fri, 22 Jan 2016 10:25:42 -0800 (PST)
X-ASG-Debug-ID: 1453487140-04cb6c0d6074db0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id SV0EBiiSA2A6wN8u for ; Fri, 22 Jan 2016 10:25:40 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 486F8658A364; Fri, 22 Jan 2016 12:25:39 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 6/7] xfs: Factor xfs_seek_hole_data into helper
Date: Fri, 22 Jan 2016 12:25:35 -0600
X-ASG-Orig-Subj: [PATCH 6/7] xfs: Factor xfs_seek_hole_data into helper
Message-Id: <1453487136-12681-7-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453487140
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26360
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Factor xfs_seek_hole_data into an unlocked helper which takes
an xfs inode rather than a file for internal use.

Also allow specification of "end" - the vfs lseek interface is
defined such that any offset past eof/i_size shall return -ENXIO,
but we will use this for quota code which does not maintain i_size,
and we want to be able to SEEK_DATA past i_size as well.  So the
lseek path can send in i_size, and the quota code can determine
its own ending offset.

Signed-off-by: Eric Sandeen 
---
 fs/xfs/xfs_file.c  |   82 ++++++++++++++++++++++++++++++++++++----------------
 fs/xfs/xfs_inode.h |    2 +
 2 files changed, 59 insertions(+), 25 deletions(-)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index ebe9b82..5dc7113 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -1337,31 +1337,31 @@ out:
 	return found;
 }
 
-STATIC loff_t
-xfs_seek_hole_data(
-	struct file		*file,
+/*
+ * caller must lock inode with xfs_ilock_data_map_shared,
+ * can we craft an appropriate ASSERT?
+ *
+ * end is because the VFS-level lseek interface is defined such that any
+ * offset past i_size shall return -ENXIO, but we use this for quota code
+ * which does not maintain i_size, and we want to SEEK_DATA past i_size.
+ */
+loff_t
+__xfs_seek_hole_data(
+	struct inode		*inode,
 	loff_t			start,
+	loff_t			end,
 	int			whence)
 {
-	struct inode		*inode = file->f_mapping->host;
 	struct xfs_inode	*ip = XFS_I(inode);
 	struct xfs_mount	*mp = ip->i_mount;
 	loff_t			uninitialized_var(offset);
-	xfs_fsize_t		isize;
 	xfs_fileoff_t		fsbno;
-	xfs_filblks_t		end;
-	uint			lock;
+	xfs_filblks_t		lastbno;
 	int			error;
 
-	if (XFS_FORCED_SHUTDOWN(mp))
-		return -EIO;
-
-	lock = xfs_ilock_data_map_shared(ip);
-
-	isize = i_size_read(inode);
-	if (start >= isize) {
+	if (start >= end) {
 		error = -ENXIO;
-		goto out_unlock;
+		goto out_error;
 	}
 
 	/*
@@ -1369,22 +1369,22 @@ xfs_seek_hole_data(
 	 * by fsbno to the end block of the file.
 	 */
 	fsbno = XFS_B_TO_FSBT(mp, start);
-	end = XFS_B_TO_FSB(mp, isize);
+	lastbno = XFS_B_TO_FSB(mp, end);
 
 	for (;;) {
 		struct xfs_bmbt_irec	map[2];
 		int			nmap = 2;
 		unsigned int		i;
 
-		error = xfs_bmapi_read(ip, fsbno, end - fsbno, map, &nmap,
+		error = xfs_bmapi_read(ip, fsbno, lastbno - fsbno, map, &nmap,
 				       XFS_BMAPI_ENTIRE);
 		if (error)
-			goto out_unlock;
+			goto out_error;
 
 		/* No extents at given offset, must be beyond EOF */
 		if (nmap == 0) {
 			error = -ENXIO;
-			goto out_unlock;
+			goto out_error;
 		}
 
 		for (i = 0; i < nmap; i++) {
@@ -1426,7 +1426,7 @@ xfs_seek_hole_data(
 			 * hole at the end of any file).
 		 	 */
 			if (whence == SEEK_HOLE) {
-				offset = isize;
+				offset = end;
 				break;
 			}
 			/*
@@ -1434,7 +1434,7 @@ xfs_seek_hole_data(
 			 */
 			ASSERT(whence == SEEK_DATA);
 			error = -ENXIO;
-			goto out_unlock;
+			goto out_error;
 		}
 
 		ASSERT(i > 1);
@@ -1445,14 +1445,14 @@ xfs_seek_hole_data(
 		 */
 		fsbno = map[i - 1].br_startoff + map[i - 1].br_blockcount;
 		start = XFS_FSB_TO_B(mp, fsbno);
-		if (start >= isize) {
+		if (start >= end) {
 			if (whence == SEEK_HOLE) {
-				offset = isize;
+				offset = end;
 				break;
 			}
 			ASSERT(whence == SEEK_DATA);
 			error = -ENXIO;
-			goto out_unlock;
+			goto out_error;
 		}
 	}
 
@@ -1464,7 +1464,39 @@ out:
 	 * situation in particular.
 	 */
 	if (whence == SEEK_HOLE)
-		offset = min_t(loff_t, offset, isize);
+		offset = min_t(loff_t, offset, end);
+
+	return offset;
+
+out_error:
+	return error;
+}
+
+STATIC loff_t
+xfs_seek_hole_data(
+	struct file		*file,
+	loff_t			start,
+	int			whence)
+{
+	struct inode		*inode = file->f_mapping->host;
+	struct xfs_inode	*ip = XFS_I(inode);
+	struct xfs_mount	*mp = ip->i_mount;
+	uint			lock;
+	loff_t			offset, end;
+	int			error = 0;
+
+	if (XFS_FORCED_SHUTDOWN(mp))
+		return -EIO;
+
+	lock = xfs_ilock_data_map_shared(ip);
+
+	end = i_size_read(inode);
+	offset = __xfs_seek_hole_data(inode, start, end, whence);
+	if (offset < 0) {
+		error = offset;
+		goto out_unlock;
+	}
+
 	offset = vfs_setpos(file, offset, inode->i_sb->s_maxbytes);
 
 out_unlock:
diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
index ca9e119..ed7e933 100644
--- a/fs/xfs/xfs_inode.h
+++ b/fs/xfs/xfs_inode.h
@@ -437,6 +437,8 @@ int	xfs_update_prealloc_flags(struct xfs_inode *ip,
 int	xfs_zero_eof(struct xfs_inode *ip, xfs_off_t offset,
 		     xfs_fsize_t isize, bool *did_zeroing);
 int	xfs_iozero(struct xfs_inode *ip, loff_t pos, size_t count);
+loff_t	__xfs_seek_hole_data(struct inode *inode, loff_t start,
+			     loff_t eof, int whence);
 
 
 /* from xfs_iops.c */
-- 
1.7.1

From sandeen@sandeen.net  Fri Jan 22 12:25:43 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 7B6977F51
	for ; Fri, 22 Jan 2016 12:25:42 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 725EF8F8033
	for ; Fri, 22 Jan 2016 10:25:42 -0800 (PST)
X-ASG-Debug-ID: 1453487140-04cbb026fc69f30001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id VEn3kYElu39fcKPy for ; Fri, 22 Jan 2016 10:25:40 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 583F7658A365; Fri, 22 Jan 2016 12:25:39 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 7/7] xfs: wire up Q_XGETNEXTQUOTA / get_nextdqblk
Date: Fri, 22 Jan 2016 12:25:36 -0600
X-ASG-Orig-Subj: [PATCH 7/7] xfs: wire up Q_XGETNEXTQUOTA / get_nextdqblk
Message-Id: <1453487136-12681-8-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453487140
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26360
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Add code to allow the Q_XGETNEXTQUOTA quotactl to quickly find
all active quotas by examining the quota inode, and skipping
over unallocated or uninitialized regions.

Userspace can then use this interface rather than i.e. a
getpwent() loop when asked to report all active quotas.

Signed-off-by: Eric Sandeen 
---
 fs/xfs/libxfs/xfs_quota_defs.h |    3 +-
 fs/xfs/xfs_dquot.c             |   96 ++++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_qm.h                |    4 +-
 fs/xfs/xfs_qm_syscalls.c       |   12 +++--
 fs/xfs/xfs_quotaops.c          |   36 ++++++++++++++-
 5 files changed, 142 insertions(+), 9 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h
index f51078f..8eed512 100644
--- a/fs/xfs/libxfs/xfs_quota_defs.h
+++ b/fs/xfs/libxfs/xfs_quota_defs.h
@@ -37,7 +37,7 @@ typedef __uint16_t	xfs_qwarncnt_t;
 #define XFS_DQ_PROJ		0x0002		/* project quota */
 #define XFS_DQ_GROUP		0x0004		/* a group quota */
 #define XFS_DQ_DIRTY		0x0008		/* dquot is dirty */
-#define XFS_DQ_FREEING		0x0010		/* dquot is beeing torn down */
+#define XFS_DQ_FREEING		0x0010		/* dquot is being torn down */
 
 #define XFS_DQ_ALLTYPES		(XFS_DQ_USER|XFS_DQ_PROJ|XFS_DQ_GROUP)
 
@@ -116,6 +116,7 @@ typedef __uint16_t	xfs_qwarncnt_t;
 #define XFS_QMOPT_DQREPAIR	0x0001000 /* repair dquot if damaged */
 #define XFS_QMOPT_GQUOTA	0x0002000 /* group dquot requested */
 #define XFS_QMOPT_ENOSPC	0x0004000 /* enospc instead of edquot (prj) */
+#define XFS_QMOPT_DQNEXT	0x0008000 /* return next dquot >= this ID */
 
 /*
  * flags to xfs_trans_mod_dquot to indicate which field needs to be
diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 5dbde0d..3b93e94 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -686,6 +686,56 @@ error0:
 }
 
 /*
+ * Advance to the next id in the current chunk, or if at the
+ * end of the chunk, skip ahead to first id in next allocated chunk
+ * using the SEEK_DATA interface.
+ */
+int
+xfs_dq_get_next_id(
+	xfs_mount_t		*mp,
+	uint			type,
+	xfs_dqid_t		*id,
+	loff_t			eof)
+{
+	struct xfs_inode	*quotip;
+	xfs_fsblock_t		start;
+	loff_t			offset;
+	uint			lock;
+	xfs_dqid_t		next_id;
+	int			error = 0;
+
+	/* Simple advance */
+	next_id = *id + 1;
+
+	/* If new ID is within the current chunk, advancing it sufficed */
+	if (next_id % mp->m_quotainfo->qi_dqperchunk) {
+		*id = next_id;
+		return 0;
+	}
+
+	/* Nope, next_id is now past the current chunk, so find the next one */
+	start = (xfs_fsblock_t)next_id / mp->m_quotainfo->qi_dqperchunk;
+
+	quotip = xfs_quota_inode(mp, type);
+	lock = xfs_ilock_data_map_shared(quotip);
+
+	offset = __xfs_seek_hole_data(VFS_I(quotip), XFS_FSB_TO_B(mp, start),
+				      eof, SEEK_DATA);
+	if (offset < 0)
+		error = offset;
+
+	xfs_iunlock(quotip, lock);
+
+	/* -ENXIO is essentially "no more data" */
+	if (error)
+		return (error == -ENXIO ? -ESRCH : error);
+
+	/* Convert next data offset back to a quota id */
+	*id = XFS_B_TO_FSB(mp, offset) * mp->m_quotainfo->qi_dqperchunk;
+	return 0;
+}
+
+/*
  * Given the file system, inode OR id, and type (UDQUOT/GDQUOT), return a
  * a locked dquot, doing an allocation (if requested) as needed.
  * When both an inode and an id are given, the inode's id takes precedence.
@@ -705,6 +755,7 @@ xfs_qm_dqget(
 	struct xfs_quotainfo	*qi = mp->m_quotainfo;
 	struct radix_tree_root *tree = xfs_dquot_tree(qi, type);
 	struct xfs_dquot	*dqp;
+	loff_t			eof = 0;
 	int			error;
 
 	ASSERT(XFS_IS_QUOTA_RUNNING(mp));
@@ -732,6 +783,21 @@ xfs_qm_dqget(
 	}
 #endif
 
+	/* Get the end of the quota file if we need it */
+	if (flags & XFS_QMOPT_DQNEXT) {
+		struct xfs_inode	*quotip;
+		xfs_fileoff_t		last;
+		uint			lock_mode;
+
+		quotip = xfs_quota_inode(mp, type);
+		lock_mode = xfs_ilock_data_map_shared(quotip);
+		error = xfs_bmap_last_offset(quotip, &last, XFS_DATA_FORK);
+		xfs_iunlock(quotip, lock_mode);
+		if (error)
+			return error;
+		eof = XFS_FSB_TO_B(mp, last);
+	}
+
 restart:
 	mutex_lock(&qi->qi_tree_lock);
 	dqp = radix_tree_lookup(tree, id);
@@ -745,6 +811,18 @@ restart:
 			goto restart;
 		}
 
+		/* uninit / unused quota found in radix tree, keep looking  */
+		if (flags & XFS_QMOPT_DQNEXT) {
+			if (XFS_IS_DQUOT_UNINITIALIZED(dqp)) {
+				xfs_dqunlock(dqp);
+				mutex_unlock(&qi->qi_tree_lock);
+				error = xfs_dq_get_next_id(mp, type, &id, eof);
+				if (error)
+					return error;
+				goto restart;
+			}
+		}
+
 		dqp->q_nrefs++;
 		mutex_unlock(&qi->qi_tree_lock);
 
@@ -771,6 +849,13 @@ restart:
 	if (ip)
 		xfs_ilock(ip, XFS_ILOCK_EXCL);
 
+	/* If we are asked to find next active id, keep looking */
+	if (error == -ENOENT && (flags & XFS_QMOPT_DQNEXT)) {
+		error = xfs_dq_get_next_id(mp, type, &id, eof);
+		if (!error)
+			goto restart;
+	}
+
 	if (error)
 		return error;
 
@@ -821,6 +906,17 @@ restart:
 	qi->qi_dquots++;
 	mutex_unlock(&qi->qi_tree_lock);
 
+	/* If we are asked to find next active id, keep looking */
+	if (flags & XFS_QMOPT_DQNEXT) {
+		if (XFS_IS_DQUOT_UNINITIALIZED(dqp)) {
+			xfs_qm_dqput(dqp);
+			error = xfs_dq_get_next_id(mp, type, &id, eof);
+			if (error)
+				return error;
+			goto restart;
+		}
+	}
+
  dqret:
 	ASSERT((ip == NULL) || xfs_isilocked(ip, XFS_ILOCK_EXCL));
 	trace_xfs_dqget_miss(dqp);
diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
index 8901a01..c68a38f 100644
--- a/fs/xfs/xfs_qm.h
+++ b/fs/xfs/xfs_qm.h
@@ -164,8 +164,8 @@ extern void		xfs_qm_dqrele_all_inodes(struct xfs_mount *, uint);
 
 /* quota ops */
 extern int		xfs_qm_scall_trunc_qfiles(struct xfs_mount *, uint);
-extern int		xfs_qm_scall_getquota(struct xfs_mount *, xfs_dqid_t,
-					uint, struct qc_dqblk *);
+extern int		xfs_qm_scall_getquota(struct xfs_mount *, xfs_dqid_t *,
+					uint, struct qc_dqblk *, uint);
 extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
 					struct qc_dqblk *);
 extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
index 3640c6e..0a25286 100644
--- a/fs/xfs/xfs_qm_syscalls.c
+++ b/fs/xfs/xfs_qm_syscalls.c
@@ -635,9 +635,10 @@ out:
 int
 xfs_qm_scall_getquota(
 	struct xfs_mount	*mp,
-	xfs_dqid_t		id,
+	xfs_dqid_t		*id,
 	uint			type,
-	struct qc_dqblk		*dst)
+	struct qc_dqblk		*dst,
+	uint			dqget_flags)
 {
 	struct xfs_dquot	*dqp;
 	int			error;
@@ -647,7 +648,7 @@ xfs_qm_scall_getquota(
 	 * we aren't passing the XFS_QMOPT_DOALLOC flag. If it doesn't
 	 * exist, we'll get ENOENT back.
 	 */
-	error = xfs_qm_dqget(mp, NULL, id, type, 0, &dqp);
+	error = xfs_qm_dqget(mp, NULL, *id, type, dqget_flags, &dqp);
 	if (error)
 		return error;
 
@@ -660,6 +661,9 @@ xfs_qm_scall_getquota(
 		goto out_put;
 	}
 
+	/* Fill in the ID we actually read from disk */
+	*id = be32_to_cpu(dqp->q_core.d_id);
+
 	memset(dst, 0, sizeof(*dst));
 	dst->d_spc_hardlimit =
 		XFS_FSB_TO_B(mp, be64_to_cpu(dqp->q_core.d_blk_hardlimit));
@@ -701,7 +705,7 @@ xfs_qm_scall_getquota(
 	if (((XFS_IS_UQUOTA_ENFORCED(mp) && type == XFS_DQ_USER) ||
 	     (XFS_IS_GQUOTA_ENFORCED(mp) && type == XFS_DQ_GROUP) ||
 	     (XFS_IS_PQUOTA_ENFORCED(mp) && type == XFS_DQ_PROJ)) &&
-	    id != 0) {
+	    *id != 0) {
 		if ((dst->d_space > dst->d_spc_softlimit) &&
 		    (dst->d_spc_softlimit > 0)) {
 			ASSERT(dst->d_spc_timer != 0);
diff --git a/fs/xfs/xfs_quotaops.c b/fs/xfs/xfs_quotaops.c
index 7795e0d..f82d79a 100644
--- a/fs/xfs/xfs_quotaops.c
+++ b/fs/xfs/xfs_quotaops.c
@@ -231,14 +231,45 @@ xfs_fs_get_dqblk(
 	struct qc_dqblk		*qdq)
 {
 	struct xfs_mount	*mp = XFS_M(sb);
+	xfs_dqid_t		id;
 
 	if (!XFS_IS_QUOTA_RUNNING(mp))
 		return -ENOSYS;
 	if (!XFS_IS_QUOTA_ON(mp))
 		return -ESRCH;
 
-	return xfs_qm_scall_getquota(mp, from_kqid(&init_user_ns, qid),
-				      xfs_quota_type(qid.type), qdq);
+	id = from_kqid(&init_user_ns, qid);
+	return xfs_qm_scall_getquota(mp, &id,
+				      xfs_quota_type(qid.type), qdq, 0);
+}
+
+/* Return quota info for active quota >= this qid */
+STATIC int
+xfs_fs_get_nextdqblk(
+	struct super_block	*sb,
+	struct kqid		*qid,
+	struct qc_dqblk		*qdq)
+{
+	int			ret;
+	struct xfs_mount	*mp = XFS_M(sb);
+	xfs_dqid_t		id;
+
+	if (!XFS_IS_QUOTA_RUNNING(mp))
+		return -ENOSYS;
+	if (!XFS_IS_QUOTA_ON(mp))
+		return -ESRCH;
+
+	id = from_kqid(&init_user_ns, *qid);
+	ret = xfs_qm_scall_getquota(mp, &id,
+				    xfs_quota_type(qid->type), qdq,
+				    XFS_QMOPT_DQNEXT);
+	if (ret)
+		return ret;
+
+	/* ID may be different, so convert back what we got */
+	*qid = make_kqid(current_user_ns(), qid->type, id);
+	return 0;
+	
 }
 
 STATIC int
@@ -267,5 +298,6 @@ const struct quotactl_ops xfs_quotactl_operations = {
 	.quota_disable		= xfs_quota_disable,
 	.rm_xquota		= xfs_fs_rm_xquota,
 	.get_dqblk		= xfs_fs_get_dqblk,
+	.get_nextdqblk		= xfs_fs_get_nextdqblk,
 	.set_dqblk		= xfs_fs_set_dqblk,
 };
-- 
1.7.1

From sandeen@sandeen.net  Fri Jan 22 12:25:44 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 65AA87F37
	for ; Fri, 22 Jan 2016 12:25:42 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id D3FA1AC004
	for ; Fri, 22 Jan 2016 10:25:41 -0800 (PST)
X-ASG-Debug-ID: 1453487139-04cb6c0d5d74da0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 0KWHorNNMWsSUnko for ; Fri, 22 Jan 2016 10:25:39 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 15BE4658A361; Fri, 22 Jan 2016 12:25:39 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
Date: Fri, 22 Jan 2016 12:25:32 -0600
X-ASG-Orig-Subj: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
Message-Id: <1453487136-12681-4-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453487139
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26360
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Q_GETNEXTQUOTA is exactly like Q_GETQUOTA, except that it
will return quota information for the id equal to or greater
than the id requested.  In other words, if the requested id has
no quota, the command will return quota information for the
next higher id which does have a quota set.  If no higher id
has an active quota, -ESRCH is returned.

This allows filesystems to do efficient iteration in kernelspace,
much like extN filesystems do in userspace when asked to report
all active quotas.

This does require a new data structure for userspace, as the
current structure does not include an ID for the returned quota
information.

Today, Ext4 with a hidden quota inode requires getpwent-style
iterations, and for systems which have i.e. LDAP backends,
this can be very slow, or even impossible if iteration is not
allowed in the configuration.

Signed-off-by: Eric Sandeen 
---
 fs/quota/quota.c           |   30 ++++++++++++++++++++++++++++++
 include/uapi/linux/quota.h |   14 ++++++++++++++
 2 files changed, 44 insertions(+), 0 deletions(-)

diff --git a/fs/quota/quota.c b/fs/quota/quota.c
index 0a6dd71..ffa4e0b 100644
--- a/fs/quota/quota.c
+++ b/fs/quota/quota.c
@@ -222,6 +222,34 @@ static int quota_getquota(struct super_block *sb, int type, qid_t id,
 	return 0;
 }
 
+/*
+ * Return quota for next active quota >= this id, if any exists,
+ * otherwise return -ESRCH via ->get_nextdqblk
+ */
+static int quota_getnextquota(struct super_block *sb, int type, qid_t id,
+			  void __user *addr)
+{
+	struct kqid qid;
+	struct qc_dqblk fdq;
+	struct if_nextdqblk idq;
+	int ret;
+
+	if (!sb->s_qcop->get_nextdqblk)
+		return -ENOSYS;
+	qid = make_kqid(current_user_ns(), type, id);
+	if (!qid_valid(qid))
+		return -EINVAL;
+	ret = sb->s_qcop->get_nextdqblk(sb, &qid, &fdq);
+	if (ret)
+		return ret;
+	/* struct if_nextdqblk is a superset of struct if_dqblk */
+	copy_to_if_dqblk((struct if_dqblk *)&idq, &fdq);
+	idq.dqb_id = from_kqid(current_user_ns(), qid);
+	if (copy_to_user(addr, &idq, sizeof(idq)))
+		return -EFAULT;
+	return 0;
+}
+
 static void copy_from_if_dqblk(struct qc_dqblk *dst, struct if_dqblk *src)
 {
 	dst->d_spc_hardlimit = qbtos(src->dqb_bhardlimit);
@@ -698,6 +726,8 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
 		return quota_setinfo(sb, type, addr);
 	case Q_GETQUOTA:
 		return quota_getquota(sb, type, id, addr);
+	case Q_GETNEXTQUOTA:
+		return quota_getnextquota(sb, type, id, addr);
 	case Q_SETQUOTA:
 		return quota_setquota(sb, type, id, addr);
 	case Q_SYNC:
diff --git a/include/uapi/linux/quota.h b/include/uapi/linux/quota.h
index 9c95b2c..38baddb 100644
--- a/include/uapi/linux/quota.h
+++ b/include/uapi/linux/quota.h
@@ -71,6 +71,7 @@
 #define Q_SETINFO  0x800006	/* set information about quota files */
 #define Q_GETQUOTA 0x800007	/* get user quota structure */
 #define Q_SETQUOTA 0x800008	/* set user quota structure */
+#define Q_GETNEXTQUOTA 0x800009	/* get disk limits and usage >= ID */
 
 /* Quota format type IDs */
 #define	QFMT_VFS_OLD 1
@@ -119,6 +120,19 @@ struct if_dqblk {
 	__u32 dqb_valid;
 };
 
+struct if_nextdqblk {
+	__u64 dqb_bhardlimit;
+	__u64 dqb_bsoftlimit;
+	__u64 dqb_curspace;
+	__u64 dqb_ihardlimit;
+	__u64 dqb_isoftlimit;
+	__u64 dqb_curinodes;
+	__u64 dqb_btime;
+	__u64 dqb_itime;
+	__u32 dqb_valid;
+	__u32 dqb_id;
+};
+
 /*
  * Structure used for setting quota information about file via quotactl
  * Following flags are used to specify which fields are valid
-- 
1.7.1

From sandeen@sandeen.net  Fri Jan 22 12:25:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id ECAF67F3F
	for ; Fri, 22 Jan 2016 12:25:41 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id D560F30405F
	for ; Fri, 22 Jan 2016 10:25:41 -0800 (PST)
X-ASG-Debug-ID: 1453487140-04cbb026fe69f30001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id i1YrIYG3se9GtKAb for ; Fri, 22 Jan 2016 10:25:40 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 378C2658A363; Fri, 22 Jan 2016 12:25:39 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 5/7] xfs: get quota inode from mp & flags rather than dqp
Date: Fri, 22 Jan 2016 12:25:34 -0600
X-ASG-Orig-Subj: [PATCH 5/7] xfs: get quota inode from mp & flags rather than dqp
Message-Id: <1453487136-12681-6-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453487140
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26360
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Allow us to get the appropriate quota inode from any
mp & quota flags, not necessarily associated with a
particular dqp.  Needed for when we are searching for
the next active ID with quotas and we want to examine
the quota inode.

Signed-off-by: Eric Sandeen 
---
 fs/xfs/xfs_dquot.c |    3 ++-
 fs/xfs/xfs_qm.h    |   10 +++++-----
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index c9c7c2d..5dbde0d 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -464,12 +464,13 @@ xfs_qm_dqtobp(
 	struct xfs_bmbt_irec	map;
 	int			nmaps = 1, error;
 	struct xfs_buf		*bp;
-	struct xfs_inode	*quotip = xfs_dq_to_quota_inode(dqp);
+	struct xfs_inode	*quotip;
 	struct xfs_mount	*mp = dqp->q_mount;
 	xfs_dqid_t		id = be32_to_cpu(dqp->q_core.d_id);
 	struct xfs_trans	*tp = (tpp ? *tpp : NULL);
 	uint			lock_mode;
 
+	quotip = xfs_quota_inode(dqp->q_mount, dqp->dq_flags);
 	dqp->q_fileoffset = (xfs_fileoff_t)id / mp->m_quotainfo->qi_dqperchunk;
 
 	lock_mode = xfs_ilock_data_map_shared(quotip);
diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
index 996a040..8901a01 100644
--- a/fs/xfs/xfs_qm.h
+++ b/fs/xfs/xfs_qm.h
@@ -104,15 +104,15 @@ xfs_dquot_tree(
 }
 
 static inline struct xfs_inode *
-xfs_dq_to_quota_inode(struct xfs_dquot *dqp)
+xfs_quota_inode(xfs_mount_t *mp, uint dq_flags)
 {
-	switch (dqp->dq_flags & XFS_DQ_ALLTYPES) {
+	switch (dq_flags & XFS_DQ_ALLTYPES) {
 	case XFS_DQ_USER:
-		return dqp->q_mount->m_quotainfo->qi_uquotaip;
+		return mp->m_quotainfo->qi_uquotaip;
 	case XFS_DQ_GROUP:
-		return dqp->q_mount->m_quotainfo->qi_gquotaip;
+		return mp->m_quotainfo->qi_gquotaip;
 	case XFS_DQ_PROJ:
-		return dqp->q_mount->m_quotainfo->qi_pquotaip;
+		return mp->m_quotainfo->qi_pquotaip;
 	default:
 		ASSERT(0);
 	}
-- 
1.7.1

From sandeen@sandeen.net  Fri Jan 22 12:25:44 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id EFB977F59
	for ; Fri, 22 Jan 2016 12:25:44 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D45F98F8033
	for ; Fri, 22 Jan 2016 10:25:41 -0800 (PST)
X-ASG-Debug-ID: 1453487139-04cbb026ff69f30001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id PC6XDxNGkWV3urc1 for ; Fri, 22 Jan 2016 10:25:39 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: by sandeen.net (Postfix, from userid 500)
	id 06854658A360; Fri, 22 Jan 2016 12:25:39 -0600 (CST)
From: Eric Sandeen 
To: linux-fsdevel@vger.kernel.org,
	xfs@oss.sgi.com
Cc: jack@suse.cz
Subject: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Date: Fri, 22 Jan 2016 12:25:31 -0600
X-ASG-Orig-Subj: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Message-Id: <1453487136-12681-3-git-send-email-sandeen@redhat.com>
X-Mailer: git-send-email 1.7.1
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453487139
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26360
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
will return quota information for the id equal to or greater
than the id requested.  In other words, if the requested id has
no quota, the command will return quota information for the
next higher id which does have a quota set.  If no higher id
has an active quota, -ESRCH is returned.

This allows filesystems to do efficient iteration in kernelspace,
much like extN filesystems do in userspace when asked to report
all active quotas.

The patch adds a d_id field to struct qc_dqblk so that we can
pass back the id of the quota which was found, and return it
to userspace.

Today, filesystems such as XFS require getpwent-style iterations,
and for systems which have i.e. LDAP backends, this can be very
slow, or even impossible if iteration is not allowed in the
configuration.

Signed-off-by: Eric Sandeen 
---
 fs/quota/quota.c               |   31 +++++++++++++++++++++++++++++++
 include/linux/quota.h          |    2 ++
 include/uapi/linux/dqblk_xfs.h |    1 +
 3 files changed, 34 insertions(+), 0 deletions(-)

diff --git a/fs/quota/quota.c b/fs/quota/quota.c
index ea66670..0a6dd71 100644
--- a/fs/quota/quota.c
+++ b/fs/quota/quota.c
@@ -625,6 +625,34 @@ static int quota_getxquota(struct super_block *sb, int type, qid_t id,
 	return ret;
 }
 
+/*
+ * Return quota for next active quota >= this id, if any exists,
+ * otherwise return -ESRCH via ->get_nextdqblk.
+ */
+static int quota_getnextxquota(struct super_block *sb, int type, qid_t id,
+			    void __user *addr)
+{
+	struct fs_disk_quota fdq;
+	struct qc_dqblk qdq;
+	struct kqid qid;
+	qid_t id_out;
+	int ret;
+
+	if (!sb->s_qcop->get_nextdqblk)
+		return -ENOSYS;
+	qid = make_kqid(current_user_ns(), type, id);
+	if (!qid_valid(qid))
+		return -EINVAL;
+	ret = sb->s_qcop->get_nextdqblk(sb, &qid, &qdq);
+	if (ret)
+		return ret;
+	id_out = from_kqid(current_user_ns(), qid);
+	copy_to_xfs_dqblk(&fdq, &qdq, type, id_out);
+	if (copy_to_user(addr, &fdq, sizeof(fdq)))
+		return -EFAULT;
+	return ret;
+}
+
 static int quota_rmxquota(struct super_block *sb, void __user *addr)
 {
 	__u32 flags;
@@ -690,6 +718,8 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
 		return quota_setxquota(sb, type, id, addr);
 	case Q_XGETQUOTA:
 		return quota_getxquota(sb, type, id, addr);
+	case Q_XGETNEXTQUOTA:
+		return quota_getnextxquota(sb, type, id, addr);
 	case Q_XQUOTASYNC:
 		if (sb->s_flags & MS_RDONLY)
 			return -EROFS;
@@ -712,6 +742,7 @@ static int quotactl_cmd_write(int cmd)
 	case Q_XGETQSTAT:
 	case Q_XGETQSTATV:
 	case Q_XGETQUOTA:
+	case Q_XGETNEXTQUOTA:
 	case Q_XQUOTASYNC:
 		return 0;
 	}
diff --git a/include/linux/quota.h b/include/linux/quota.h
index b2505ac..fba92f5 100644
--- a/include/linux/quota.h
+++ b/include/linux/quota.h
@@ -425,6 +425,8 @@ struct quotactl_ops {
 	int (*quota_sync)(struct super_block *, int);
 	int (*set_info)(struct super_block *, int, struct qc_info *);
 	int (*get_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *);
+	int (*get_nextdqblk)(struct super_block *, struct kqid *,
+			     struct qc_dqblk *);
 	int (*set_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *);
 	int (*get_state)(struct super_block *, struct qc_state *);
 	int (*rm_xquota)(struct super_block *, unsigned int);
diff --git a/include/uapi/linux/dqblk_xfs.h b/include/uapi/linux/dqblk_xfs.h
index dcd75cc..11b3b31 100644
--- a/include/uapi/linux/dqblk_xfs.h
+++ b/include/uapi/linux/dqblk_xfs.h
@@ -39,6 +39,7 @@
 #define Q_XQUOTARM	XQM_CMD(6)	/* free disk space used by dquots */
 #define Q_XQUOTASYNC	XQM_CMD(7)	/* delalloc flush, updates dquots */
 #define Q_XGETQSTATV	XQM_CMD(8)	/* newer version of get quota */
+#define Q_XGETNEXTQUOTA	XQM_CMD(9)	/* get disk limits and usage >= ID */
 
 /*
  * fs_disk_quota structure:
-- 
1.7.1

From darrick.wong@oracle.com  Fri Jan 22 16:01:59 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id D5F097F37
	for ; Fri, 22 Jan 2016 16:01:59 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id C5B26304032
	for ; Fri, 22 Jan 2016 14:01:56 -0800 (PST)
X-ASG-Debug-ID: 1453500114-04cb6c0d5f79830001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id s4nBGNasIrzK2bdn (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 14:01:54 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0MM1lpg008716
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 22 Jan 2016 22:01:48 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0MM1lvQ003574
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 22 Jan 2016 22:01:47 GMT
Received: from abhmp0018.oracle.com (abhmp0018.oracle.com [141.146.116.24])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0MM1kJN019255;
	Fri, 22 Jan 2016 22:01:47 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 14:01:46 -0800
Date: Fri, 22 Jan 2016 14:01:45 -0800
From: "Darrick J. Wong" 
To: "Josef 'Jeff' Sipek" 
Cc: Dave Chinner , xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
Message-ID: <20160122220145.GA26408@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
References: <20160111232657.GA7831@birch.djwong.org>
 <20160115205801.GB5757@birch.djwong.org>
 <20160115225219.GP6033@dastard>
 <20160120154035.GH1366@meili.valhalla.31bits.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160120154035.GH1366@meili.valhalla.31bits.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453500114
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26365
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Wed, Jan 20, 2016 at 10:40:36AM -0500, Josef 'Jeff' Sipek wrote:
> On Sat, Jan 16, 2016 at 09:52:19AM +1100, Dave Chinner wrote:
> > On Fri, Jan 15, 2016 at 12:58:01PM -0800, Darrick J. Wong wrote:
> > > On Mon, Jan 11, 2016 at 03:26:57PM -0800, Darrick J. Wong wrote:
> > > > Check the sizes of XFS on-disk structures when compiling the kernel.
> > > > Use this to catch inadvertent changes in structure size due to padding
> > > > and alignment issues, etc.
> > > > 
> > > > Signed-off-by: Darrick J. Wong 
> > ....
> > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
> > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
> > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
> > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
> > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
> > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
> > > > +}
> > > 
> > > Perhaps this huge function ought to hide away in its own file?
> > 
> > Yes, I think that's a good idea, and I think it should only be built
> > on debug builds, too.
> 
> Doesn't this turn into nothingness as far as the generated binary is
> concerned?  Because if it does, I don't see a reason to keep it debug-only.
> Afterall, non-debug builds need to be correct and these are pretty important
> checks.  We certainly don't want another ARM structure padding & alignment
> fiasco.

I moved the function to a separate header file and changed the declaration to:

static inline void __init xfs_check_ondisk_structs(void)

With any luck that should compile down to an empty inline function which should
be optimized out of the resulting code.  Worst case it's a separate function
that gets discarded after the module loads.  If Dave doesn't object, I'll leave
it enabled for all builds.

--D

> 
> Just my 2 cents,
> 
> Jeff.
> 
> -- 
> Only two things are infinite, the universe and human stupidity, and I'm not
> sure about the former.
> 		- Albert Einstein

From ross.zwisler@linux.intel.com  Fri Jan 22 17:06:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 486897F37
	for ; Fri, 22 Jan 2016 17:06:29 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 27CAD304039
	for ; Fri, 22 Jan 2016 15:06:26 -0800 (PST)
X-ASG-Debug-ID: 1453503982-04cb6c0d607a8c0001-NocioJ
Received: from mga14.intel.com ([192.55.52.115]) by cuda.sgi.com with ESMTP id qQzB17RSy5GoRTfz for ; Fri, 22 Jan 2016 15:06:22 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.115
Received: from orsmga001.jf.intel.com ([10.7.209.18])
  by fmsmga103.fm.intel.com with ESMTP; 22 Jan 2016 15:06:22 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,333,1449561600"; 
   d="scan'208";a="866516182"
Received: from rzwisler-desk1.amr.corp.intel.com (HELO tarkir.lm.intel.com) ([10.232.112.142])
  by orsmga001.jf.intel.com with ESMTP; 22 Jan 2016 15:06:22 -0800
From: Ross Zwisler 
To: linux-kernel@vger.kernel.org
Cc: Ross Zwisler ,
	"Theodore Ts'o" ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner ,
	Jan Kara ,
	Matthew Wilcox ,
	linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org,
	linux-nvdimm@lists.01.org,
	xfs@oss.sgi.com
Subject: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Date: Fri, 22 Jan 2016 16:06:11 -0700
X-ASG-Orig-Subj: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-Id: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
X-Mailer: git-send-email 2.5.0
X-Barracuda-Connect: UNKNOWN[192.55.52.115]
X-Barracuda-Start-Time: 1453503982
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_MJ1963, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26366
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

With the current DAX code the following race exists:

Process 1                	Process 2
---------			---------

__dax_fault() - read file f, index 0
  get_block() -> returns hole
                             	__dax_fault() - write file f, index 0
                                  get_block() -> allocates blocks
                                  dax_insert_mapping()
  dax_load_hole()
  *data corruption*

An analogous race exists between __dax_fault() loading a hole and
__dax_pmd_fault() allocating a PMD DAX page and trying to insert it, and
that race also ends in data corruption.

One solution to this race was proposed by Jan Kara:

  So we need some exclusion that makes sure pgoff->block mapping
  information is uptodate at the moment we insert it into page tables. The
  simplest reasonably fast thing I can see is:

  When handling a read fault, things stay as is and filesystem protects the
  fault with an equivalent of EXT4_I(inode)->i_mmap_sem held for reading.
  When handling a write fault we first grab EXT4_I(inode)->i_mmap_sem for
  reading and try a read fault. If __dax_fault() sees a hole returned from
  get_blocks() during a write fault, it bails out. Filesystem grabs
  EXT4_I(inode)->i_mmap_sem for writing and retries with different
  get_blocks() callback which will allocate blocks. That way we get proper
  exclusion for faults needing to allocate blocks.

This patch adds this logic to DAX, ext2, ext4 and XFS.  The changes for
these four components all appear in the same patch as opposed to being
spread out among multiple patches in a series because we are changing the
argument list to __dax_fault(), __dax_pmd_fault() and __dax_mkwrite().
This means we can't easily change things one component at a time and still
keep the series bisectable.

For ext4 this patch assumes that the journal entry is only needed when we
are actually allocating blocks with get_block().  An in-depth review of
this logic would be welcome.

I also fixed a bug in the ext4 implementation of ext4_dax_mkwrite().
Previously it assumed that the block allocation was already complete, so it
didn't create a journal entry.  For a read that creates a zero page to
cover a hole followed by a write that actually allocates storage, this is
incorrect.  The ext4_dax_mkwrite() -> __dax_mkwrite() -> __dax_fault() path
would call get_blocks() to allocate storage, so I'm pretty sure we need a
journal entry here.

With that fixed, I noticed that for both ext2 and ext4 the paths through
the .fault and .page_mkwrite vmops paths were exactly the same, so I
removed ext4_dax_mkwrite() and ext2_dax_mkwrite() and just use
ext4_dax_fault() and ext2_dax_fault() directly instead.

I'm still in the process of testing this patch, which is part of the reason
why it is marked as RFC.  I know of at least one deadlock that is easily
hit by doing a read of a hole followed by a write which allocates storage.
If you're using xfstests you can hit this easily with generic/075 with any
of the three filesytems.  I'll continue to track this down, but I wanted to
send out this RFC to sanity check the general approach.

Signed-off-by: Ross Zwisler 
Suggested-by: Jan Kara 
---
 fs/block_dev.c      | 19 ++++++++++--
 fs/dax.c            | 20 ++++++++++---
 fs/ext2/file.c      | 41 ++++++++++++-------------
 fs/ext4/file.c      | 86 +++++++++++++++++++++++++----------------------------
 fs/xfs/xfs_file.c   | 28 +++++++++++++----
 include/linux/dax.h |  8 +++--
 6 files changed, 121 insertions(+), 81 deletions(-)

diff --git a/fs/block_dev.c b/fs/block_dev.c
index 303b7cd..775f1b0 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -1733,13 +1733,28 @@ static const struct address_space_operations def_blk_aops = {
  */
 static int blkdev_dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 {
-	return __dax_fault(vma, vmf, blkdev_get_block, NULL);
+	int ret;
+
+	ret = __dax_fault(vma, vmf, blkdev_get_block, NULL, false);
+
+	if (WARN_ON_ONCE(ret == -EAGAIN))
+		ret = VM_FAULT_SIGBUS;
+
+	return ret;
 }
 
 static int blkdev_dax_pmd_fault(struct vm_area_struct *vma, unsigned long addr,
 		pmd_t *pmd, unsigned int flags)
 {
-	return __dax_pmd_fault(vma, addr, pmd, flags, blkdev_get_block, NULL);
+	int ret;
+
+	ret = __dax_pmd_fault(vma, addr, pmd, flags, blkdev_get_block, NULL,
+			false);
+
+	if (WARN_ON_ONCE(ret == -EAGAIN))
+		ret = VM_FAULT_SIGBUS;
+
+	return ret;
 }
 
 static void blkdev_vm_open(struct vm_area_struct *vma)
diff --git a/fs/dax.c b/fs/dax.c
index 206650f..7a927eb 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -582,13 +582,19 @@ static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh,
  *	extent mappings from @get_block, but it is optional for reads as
  *	dax_insert_mapping() will always zero unwritten blocks. If the fs does
  *	not support unwritten extents, the it should pass NULL.
+ * @alloc_ok: True if our caller is holding a lock that isolates us from other
+ *	DAX faults on the same inode.  This allows us to allocate new storage
+ *	with get_block() and not have to worry about races with other fault
+ *	handlers.  If this is unset and we need to allocate storage we will
+ *	return -EAGAIN to ask our caller to retry with the proper locking.
  *
  * When a page fault occurs, filesystems may call this helper in their
  * fault handler for DAX files. __dax_fault() assumes the caller has done all
  * the necessary locking for the page fault to proceed successfully.
  */
 int __dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
-			get_block_t get_block, dax_iodone_t complete_unwritten)
+			get_block_t get_block, dax_iodone_t complete_unwritten,
+			bool alloc_ok)
 {
 	struct file *file = vma->vm_file;
 	struct address_space *mapping = file->f_mapping;
@@ -642,6 +648,9 @@ int __dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
 
 	if (!buffer_mapped(&bh) && !buffer_unwritten(&bh) && !vmf->cow_page) {
 		if (vmf->flags & FAULT_FLAG_WRITE) {
+			if (!alloc_ok)
+				return -EAGAIN;
+
 			error = get_block(inode, block, &bh, 1);
 			count_vm_event(PGMAJFAULT);
 			mem_cgroup_count_vm_event(vma->vm_mm, PGMAJFAULT);
@@ -745,7 +754,7 @@ int dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
 		sb_start_pagefault(sb);
 		file_update_time(vma->vm_file);
 	}
-	result = __dax_fault(vma, vmf, get_block, complete_unwritten);
+	result = __dax_fault(vma, vmf, get_block, complete_unwritten, false);
 	if (vmf->flags & FAULT_FLAG_WRITE)
 		sb_end_pagefault(sb);
 
@@ -780,7 +789,7 @@ static void __dax_dbg(struct buffer_head *bh, unsigned long address,
 
 int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
 		pmd_t *pmd, unsigned int flags, get_block_t get_block,
-		dax_iodone_t complete_unwritten)
+		dax_iodone_t complete_unwritten, bool alloc_ok)
 {
 	struct file *file = vma->vm_file;
 	struct address_space *mapping = file->f_mapping;
@@ -836,6 +845,9 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
 		return VM_FAULT_SIGBUS;
 
 	if (!buffer_mapped(&bh) && write) {
+		if (!alloc_ok)
+			return -EAGAIN;
+
 		if (get_block(inode, block, &bh, 1) != 0)
 			return VM_FAULT_SIGBUS;
 		alloc = true;
@@ -1017,7 +1029,7 @@ int dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
 		file_update_time(vma->vm_file);
 	}
 	result = __dax_pmd_fault(vma, address, pmd, flags, get_block,
-				complete_unwritten);
+				complete_unwritten, false);
 	if (flags & FAULT_FLAG_WRITE)
 		sb_end_pagefault(sb);
 
diff --git a/fs/ext2/file.c b/fs/ext2/file.c
index 2c88d68..1106a9e 100644
--- a/fs/ext2/file.c
+++ b/fs/ext2/file.c
@@ -49,11 +49,17 @@ static int ext2_dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 		sb_start_pagefault(inode->i_sb);
 		file_update_time(vma->vm_file);
 	}
+
 	down_read(&ei->dax_sem);
+	ret = __dax_fault(vma, vmf, ext2_get_block, NULL, false);
+	up_read(&ei->dax_sem);
 
-	ret = __dax_fault(vma, vmf, ext2_get_block, NULL);
+	if (ret == -EAGAIN) {
+		down_write(&ei->dax_sem);
+		ret = __dax_fault(vma, vmf, ext2_get_block, NULL, true);
+		up_write(&ei->dax_sem);
+	}
 
-	up_read(&ei->dax_sem);
 	if (vmf->flags & FAULT_FLAG_WRITE)
 		sb_end_pagefault(inode->i_sb);
 	return ret;
@@ -70,33 +76,24 @@ static int ext2_dax_pmd_fault(struct vm_area_struct *vma, unsigned long addr,
 		sb_start_pagefault(inode->i_sb);
 		file_update_time(vma->vm_file);
 	}
+
 	down_read(&ei->dax_sem);
+	ret = __dax_pmd_fault(vma, addr, pmd, flags, ext2_get_block, NULL,
+			false);
+	up_read(&ei->dax_sem);
 
-	ret = __dax_pmd_fault(vma, addr, pmd, flags, ext2_get_block, NULL);
+	if (ret == -EAGAIN) {
+		down_write(&ei->dax_sem);
+		ret = __dax_pmd_fault(vma, addr, pmd, flags, ext2_get_block,
+				NULL, true);
+		up_write(&ei->dax_sem);
+	}
 
-	up_read(&ei->dax_sem);
 	if (flags & FAULT_FLAG_WRITE)
 		sb_end_pagefault(inode->i_sb);
 	return ret;
 }
 
-static int ext2_dax_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
-{
-	struct inode *inode = file_inode(vma->vm_file);
-	struct ext2_inode_info *ei = EXT2_I(inode);
-	int ret;
-
-	sb_start_pagefault(inode->i_sb);
-	file_update_time(vma->vm_file);
-	down_read(&ei->dax_sem);
-
-	ret = __dax_mkwrite(vma, vmf, ext2_get_block, NULL);
-
-	up_read(&ei->dax_sem);
-	sb_end_pagefault(inode->i_sb);
-	return ret;
-}
-
 static int ext2_dax_pfn_mkwrite(struct vm_area_struct *vma,
 		struct vm_fault *vmf)
 {
@@ -124,7 +121,7 @@ static int ext2_dax_pfn_mkwrite(struct vm_area_struct *vma,
 static const struct vm_operations_struct ext2_dax_vm_ops = {
 	.fault		= ext2_dax_fault,
 	.pmd_fault	= ext2_dax_pmd_fault,
-	.page_mkwrite	= ext2_dax_mkwrite,
+	.page_mkwrite	= ext2_dax_fault,
 	.pfn_mkwrite	= ext2_dax_pfn_mkwrite,
 };
 
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index fa899c9..abddc8a 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -204,24 +204,30 @@ static int ext4_dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 	if (write) {
 		sb_start_pagefault(sb);
 		file_update_time(vma->vm_file);
-		down_read(&EXT4_I(inode)->i_mmap_sem);
-		handle = ext4_journal_start_sb(sb, EXT4_HT_WRITE_PAGE,
-						EXT4_DATA_TRANS_BLOCKS(sb));
-	} else
-		down_read(&EXT4_I(inode)->i_mmap_sem);
+	}
 
-	if (IS_ERR(handle))
-		result = VM_FAULT_SIGBUS;
-	else
-		result = __dax_fault(vma, vmf, ext4_dax_mmap_get_block, NULL);
+	down_read(&EXT4_I(inode)->i_mmap_sem);
+	result = __dax_fault(vma, vmf, ext4_dax_mmap_get_block, NULL,
+			false);
+	up_read(&EXT4_I(inode)->i_mmap_sem);
 
-	if (write) {
-		if (!IS_ERR(handle))
+	if (result == -EAGAIN) {
+		down_write(&EXT4_I(inode)->i_mmap_sem);
+		handle = ext4_journal_start_sb(sb, EXT4_HT_WRITE_PAGE,
+				EXT4_DATA_TRANS_BLOCKS(sb));
+
+		if (IS_ERR(handle))
+			result = VM_FAULT_SIGBUS;
+		else {
+			result = __dax_fault(vma, vmf,
+					ext4_dax_mmap_get_block, NULL, true);
 			ext4_journal_stop(handle);
-		up_read(&EXT4_I(inode)->i_mmap_sem);
+		}
+		up_write(&EXT4_I(inode)->i_mmap_sem);
+	}
+
+	if (write)
 		sb_end_pagefault(sb);
-	} else
-		up_read(&EXT4_I(inode)->i_mmap_sem);
 
 	return result;
 }
@@ -238,47 +244,37 @@ static int ext4_dax_pmd_fault(struct vm_area_struct *vma, unsigned long addr,
 	if (write) {
 		sb_start_pagefault(sb);
 		file_update_time(vma->vm_file);
-		down_read(&EXT4_I(inode)->i_mmap_sem);
+	}
+
+	down_read(&EXT4_I(inode)->i_mmap_sem);
+	result = __dax_pmd_fault(vma, addr, pmd, flags,
+			ext4_dax_mmap_get_block, NULL, false);
+	up_read(&EXT4_I(inode)->i_mmap_sem);
+
+	if (result == -EAGAIN) {
+		down_write(&EXT4_I(inode)->i_mmap_sem);
 		handle = ext4_journal_start_sb(sb, EXT4_HT_WRITE_PAGE,
 				ext4_chunk_trans_blocks(inode,
 							PMD_SIZE / PAGE_SIZE));
-	} else
-		down_read(&EXT4_I(inode)->i_mmap_sem);
 
-	if (IS_ERR(handle))
-		result = VM_FAULT_SIGBUS;
-	else
-		result = __dax_pmd_fault(vma, addr, pmd, flags,
-				ext4_dax_mmap_get_block, NULL);
-
-	if (write) {
-		if (!IS_ERR(handle))
+		if (IS_ERR(handle))
+			result = VM_FAULT_SIGBUS;
+		else {
+			result = __dax_pmd_fault(vma, addr, pmd, flags,
+					ext4_dax_mmap_get_block, NULL, true);
 			ext4_journal_stop(handle);
-		up_read(&EXT4_I(inode)->i_mmap_sem);
+		}
+		up_write(&EXT4_I(inode)->i_mmap_sem);
+	}
+
+	if (write)
 		sb_end_pagefault(sb);
-	} else
-		up_read(&EXT4_I(inode)->i_mmap_sem);
 
 	return result;
 }
 
-static int ext4_dax_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
-{
-	int err;
-	struct inode *inode = file_inode(vma->vm_file);
-
-	sb_start_pagefault(inode->i_sb);
-	file_update_time(vma->vm_file);
-	down_read(&EXT4_I(inode)->i_mmap_sem);
-	err = __dax_mkwrite(vma, vmf, ext4_dax_mmap_get_block, NULL);
-	up_read(&EXT4_I(inode)->i_mmap_sem);
-	sb_end_pagefault(inode->i_sb);
-
-	return err;
-}
-
 /*
- * Handle write fault for VM_MIXEDMAP mappings. Similarly to ext4_dax_mkwrite()
+ * Handle write fault for VM_MIXEDMAP mappings. Similarly to ext4_dax_fault()
  * handler we check for races agaist truncate. Note that since we cycle through
  * i_mmap_sem, we are sure that also any hole punching that began before we
  * were called is finished by now and so if it included part of the file we
@@ -311,7 +307,7 @@ static int ext4_dax_pfn_mkwrite(struct vm_area_struct *vma,
 static const struct vm_operations_struct ext4_dax_vm_ops = {
 	.fault		= ext4_dax_fault,
 	.pmd_fault	= ext4_dax_pmd_fault,
-	.page_mkwrite	= ext4_dax_mkwrite,
+	.page_mkwrite	= ext4_dax_fault,
 	.pfn_mkwrite	= ext4_dax_pfn_mkwrite,
 };
 #else
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 55e16e2..81edbd4 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -1523,16 +1523,26 @@ xfs_filemap_page_mkwrite(
 
 	sb_start_pagefault(inode->i_sb);
 	file_update_time(vma->vm_file);
-	xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
 
 	if (IS_DAX(inode)) {
-		ret = __dax_mkwrite(vma, vmf, xfs_get_blocks_dax_fault, NULL);
+		xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
+		ret = __dax_mkwrite(vma, vmf, xfs_get_blocks_dax_fault, NULL,
+				false);
+		xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
+
+		if (ret == -EAGAIN) {
+			xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_EXCL);
+			ret = __dax_mkwrite(vma, vmf,
+					xfs_get_blocks_dax_fault, NULL, true);
+			xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_EXCL);
+		}
 	} else {
+		xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
 		ret = block_page_mkwrite(vma, vmf, xfs_get_blocks);
 		ret = block_page_mkwrite_return(ret);
+		xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
 	}
 
-	xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
 	sb_end_pagefault(inode->i_sb);
 
 	return ret;
@@ -1560,7 +1570,8 @@ xfs_filemap_fault(
 		 * changes to xfs_get_blocks_direct() to map unwritten extent
 		 * ioend for conversion on read-only mappings.
 		 */
-		ret = __dax_fault(vma, vmf, xfs_get_blocks_dax_fault, NULL);
+		ret = __dax_fault(vma, vmf, xfs_get_blocks_dax_fault, NULL,
+				false);
 	} else
 		ret = filemap_fault(vma, vmf);
 	xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
@@ -1598,9 +1609,16 @@ xfs_filemap_pmd_fault(
 
 	xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
 	ret = __dax_pmd_fault(vma, addr, pmd, flags, xfs_get_blocks_dax_fault,
-			      NULL);
+			      NULL, false);
 	xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
 
+	if (ret == -EAGAIN) {
+		xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_EXCL);
+		ret = __dax_pmd_fault(vma, addr, pmd, flags,
+				xfs_get_blocks_dax_fault, NULL, true);
+		xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_EXCL);
+	}
+
 	if (flags & FAULT_FLAG_WRITE)
 		sb_end_pagefault(inode->i_sb);
 
diff --git a/include/linux/dax.h b/include/linux/dax.h
index 8204c3d..783a2b6 100644
--- a/include/linux/dax.h
+++ b/include/linux/dax.h
@@ -13,12 +13,13 @@ int dax_truncate_page(struct inode *, loff_t from, get_block_t);
 int dax_fault(struct vm_area_struct *, struct vm_fault *, get_block_t,
 		dax_iodone_t);
 int __dax_fault(struct vm_area_struct *, struct vm_fault *, get_block_t,
-		dax_iodone_t);
+		dax_iodone_t, bool alloc_ok);
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 int dax_pmd_fault(struct vm_area_struct *, unsigned long addr, pmd_t *,
 				unsigned int flags, get_block_t, dax_iodone_t);
 int __dax_pmd_fault(struct vm_area_struct *, unsigned long addr, pmd_t *,
-				unsigned int flags, get_block_t, dax_iodone_t);
+				unsigned int flags, get_block_t, dax_iodone_t,
+				bool alloc_ok);
 #else
 static inline int dax_pmd_fault(struct vm_area_struct *vma, unsigned long addr,
 				pmd_t *pmd, unsigned int flags, get_block_t gb,
@@ -30,7 +31,8 @@ static inline int dax_pmd_fault(struct vm_area_struct *vma, unsigned long addr,
 #endif
 int dax_pfn_mkwrite(struct vm_area_struct *, struct vm_fault *);
 #define dax_mkwrite(vma, vmf, gb, iod)		dax_fault(vma, vmf, gb, iod)
-#define __dax_mkwrite(vma, vmf, gb, iod)	__dax_fault(vma, vmf, gb, iod)
+#define __dax_mkwrite(vma, vmf, gb, iod, alloc)  \
+	__dax_fault(vma, vmf, gb, iod, alloc)
 
 static inline bool vma_is_dax(struct vm_area_struct *vma)
 {
-- 
2.5.0

From darrick.wong@oracle.com  Fri Jan 22 18:34:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 1105F7F37
	for ; Fri, 22 Jan 2016 18:34:31 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E3BEB8F8040
	for ; Fri, 22 Jan 2016 16:34:29 -0800 (PST)
X-ASG-Debug-ID: 1453509267-04cb6c0d607c0b0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id 8tl7vGZN1L9NL99p (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:34:28 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0YQuV027521
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:34:27 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0YQcO031405
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:34:26 GMT
Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0YQjI012132;
	Sat, 23 Jan 2016 00:34:26 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:34:26 -0800
Subject: [PATCH 0/3] xfs: miscellaneous bugfixes
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 0/3] xfs: miscellaneous bugfixes
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:34:25 -0800
Message-ID: <20160123003425.2338.18133.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509268
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Hi all,

I'm pushing a pile of miscellaneous bugfixes for the kernel, xfsprogs,
xfstests, and xfsdocs.  None of these are rmap or reflink patches.
They've been compile-tested and run through xfstests on x86.

The biggest changes in these patches add structure size checking to
XFS and reorganize libxfs a bit so that xfs/122 (the userspace part
of structure size checking) will now pass with modern xfsprogs.

--D

From darrick.wong@oracle.com  Fri Jan 22 18:34:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY,
	UPPERCASE_50_75 autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id BDDF27F37
	for ; Fri, 22 Jan 2016 18:34:40 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 97392304039
	for ; Fri, 22 Jan 2016 16:34:37 -0800 (PST)
X-ASG-Debug-ID: 1453509275-04bdf04d6c6c5b0001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id j3Fs78bBf0lh3zA0 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:34:35 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0YYHd019119
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:34:34 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0YXip006335
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:34:34 GMT
Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0YXSK012146;
	Sat, 23 Jan 2016 00:34:33 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:34:33 -0800
Subject: [PATCH 1/3] xfs: use named array initializers for log item dumping
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 1/3] xfs: use named array initializers for log item dumping
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:34:31 -0800
Message-ID: <20160123003431.2338.77639.stgit@birch.djwong.org>
In-Reply-To: <20160123003425.2338.18133.stgit@birch.djwong.org>
References: <20160123003425.2338.18133.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453509275
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.49
X-Barracuda-Spam-Status: No, SCORE=0.49 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY, UPPERCASE_50_75, UPPERCASE_50_75_2
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines
	0.00 UPPERCASE_50_75        message body is 50-75% uppercase
	0.49 UPPERCASE_50_75_2      message body is 50-75% uppercase

Use named array initializers for the string arrays used to dump log
items, rather than depending on the order being maintained correctly.

Signed-off-by: Darrick J. Wong 
---
 fs/xfs/xfs_log.c |  132 ++++++++++++++++++++++++++++--------------------------
 1 file changed, 68 insertions(+), 64 deletions(-)


diff --git a/fs/xfs/xfs_log.c b/fs/xfs/xfs_log.c
index 2aa187e..4758f06 100644
--- a/fs/xfs/xfs_log.c
+++ b/fs/xfs/xfs_log.c
@@ -1989,77 +1989,81 @@ xlog_print_tic_res(
 	uint ophdr_spc = ticket->t_res_num_ophdrs * (uint)sizeof(xlog_op_header_t);
 
 	/* match with XLOG_REG_TYPE_* in xfs_log.h */
-	static char *res_type_str[XLOG_REG_TYPE_MAX] = {
-	    "bformat",
-	    "bchunk",
-	    "efi_format",
-	    "efd_format",
-	    "iformat",
-	    "icore",
-	    "iext",
-	    "ibroot",
-	    "ilocal",
-	    "iattr_ext",
-	    "iattr_broot",
-	    "iattr_local",
-	    "qformat",
-	    "dquot",
-	    "quotaoff",
-	    "LR header",
-	    "unmount",
-	    "commit",
-	    "trans header"
+#define REG_TYPE_STR(type, str)	[XLOG_REG_TYPE_##type] = str
+	static char *res_type_str[XLOG_REG_TYPE_MAX + 1] = {
+	    REG_TYPE_STR(BFORMAT, "bformat"),
+	    REG_TYPE_STR(BCHUNK, "bchunk"),
+	    REG_TYPE_STR(EFI_FORMAT, "efi_format"),
+	    REG_TYPE_STR(EFD_FORMAT, "efd_format"),
+	    REG_TYPE_STR(IFORMAT, "iformat"),
+	    REG_TYPE_STR(ICORE, "icore"),
+	    REG_TYPE_STR(IEXT, "iext"),
+	    REG_TYPE_STR(IBROOT, "ibroot"),
+	    REG_TYPE_STR(ILOCAL, "ilocal"),
+	    REG_TYPE_STR(IATTR_EXT, "iattr_ext"),
+	    REG_TYPE_STR(IATTR_BROOT, "iattr_broot"),
+	    REG_TYPE_STR(IATTR_LOCAL, "iattr_local"),
+	    REG_TYPE_STR(QFORMAT, "qformat"),
+	    REG_TYPE_STR(DQUOT, "dquot"),
+	    REG_TYPE_STR(QUOTAOFF, "quotaoff"),
+	    REG_TYPE_STR(LRHEADER, "LR header"),
+	    REG_TYPE_STR(UNMOUNT, "unmount"),
+	    REG_TYPE_STR(COMMIT, "commit"),
+	    REG_TYPE_STR(TRANSHDR, "trans header"),
+	    REG_TYPE_STR(ICREATE, "inode create")
 	};
+#undef REG_TYPE_STR
+#define TRANS_TYPE_STR(type)	[XFS_TRANS_##type] = #type
 	static char *trans_type_str[XFS_TRANS_TYPE_MAX] = {
-	    "SETATTR_NOT_SIZE",
-	    "SETATTR_SIZE",
-	    "INACTIVE",
-	    "CREATE",
-	    "CREATE_TRUNC",
-	    "TRUNCATE_FILE",
-	    "REMOVE",
-	    "LINK",
-	    "RENAME",
-	    "MKDIR",
-	    "RMDIR",
-	    "SYMLINK",
-	    "SET_DMATTRS",
-	    "GROWFS",
-	    "STRAT_WRITE",
-	    "DIOSTRAT",
-	    "WRITE_SYNC",
-	    "WRITEID",
-	    "ADDAFORK",
-	    "ATTRINVAL",
-	    "ATRUNCATE",
-	    "ATTR_SET",
-	    "ATTR_RM",
-	    "ATTR_FLAG",
-	    "CLEAR_AGI_BUCKET",
-	    "QM_SBCHANGE",
-	    "DUMMY1",
-	    "DUMMY2",
-	    "QM_QUOTAOFF",
-	    "QM_DQALLOC",
-	    "QM_SETQLIM",
-	    "QM_DQCLUSTER",
-	    "QM_QINOCREATE",
-	    "QM_QUOTAOFF_END",
-	    "FSYNC_TS",
-	    "GROWFSRT_ALLOC",
-	    "GROWFSRT_ZERO",
-	    "GROWFSRT_FREE",
-	    "SWAPEXT",
-	    "CHECKPOINT",
-	    "ICREATE",
-	    "CREATE_TMPFILE"
+	    TRANS_TYPE_STR(SETATTR_NOT_SIZE),
+	    TRANS_TYPE_STR(SETATTR_SIZE),
+	    TRANS_TYPE_STR(INACTIVE),
+	    TRANS_TYPE_STR(CREATE),
+	    TRANS_TYPE_STR(CREATE_TRUNC),
+	    TRANS_TYPE_STR(TRUNCATE_FILE),
+	    TRANS_TYPE_STR(REMOVE),
+	    TRANS_TYPE_STR(LINK),
+	    TRANS_TYPE_STR(RENAME),
+	    TRANS_TYPE_STR(MKDIR),
+	    TRANS_TYPE_STR(RMDIR),
+	    TRANS_TYPE_STR(SYMLINK),
+	    TRANS_TYPE_STR(SET_DMATTRS),
+	    TRANS_TYPE_STR(GROWFS),
+	    TRANS_TYPE_STR(STRAT_WRITE),
+	    TRANS_TYPE_STR(DIOSTRAT),
+	    TRANS_TYPE_STR(WRITEID),
+	    TRANS_TYPE_STR(ADDAFORK),
+	    TRANS_TYPE_STR(ATTRINVAL),
+	    TRANS_TYPE_STR(ATRUNCATE),
+	    TRANS_TYPE_STR(ATTR_SET),
+	    TRANS_TYPE_STR(ATTR_RM),
+	    TRANS_TYPE_STR(ATTR_FLAG),
+	    TRANS_TYPE_STR(CLEAR_AGI_BUCKET),
+	    TRANS_TYPE_STR(SB_CHANGE),
+	    TRANS_TYPE_STR(DUMMY1),
+	    TRANS_TYPE_STR(DUMMY2),
+	    TRANS_TYPE_STR(QM_QUOTAOFF),
+	    TRANS_TYPE_STR(QM_DQALLOC),
+	    TRANS_TYPE_STR(QM_SETQLIM),
+	    TRANS_TYPE_STR(QM_DQCLUSTER),
+	    TRANS_TYPE_STR(QM_QINOCREATE),
+	    TRANS_TYPE_STR(QM_QUOTAOFF_END),
+	    TRANS_TYPE_STR(FSYNC_TS),
+	    TRANS_TYPE_STR(GROWFSRT_ALLOC),
+	    TRANS_TYPE_STR(GROWFSRT_ZERO),
+	    TRANS_TYPE_STR(GROWFSRT_FREE),
+	    TRANS_TYPE_STR(SWAPEXT),
+	    TRANS_TYPE_STR(CHECKPOINT),
+	    TRANS_TYPE_STR(ICREATE),
+	    TRANS_TYPE_STR(CREATE_TMPFILE)
 	};
+#undef TRANS_TYPE_STR
 
 	xfs_warn(mp, "xlog_write: reservation summary:");
 	xfs_warn(mp, "  trans type  = %s (%u)",
 		 ((ticket->t_trans_type <= 0 ||
 		   ticket->t_trans_type > XFS_TRANS_TYPE_MAX) ?
-		  "bad-trans-type" : trans_type_str[ticket->t_trans_type-1]),
+		  "bad-trans-type" : trans_type_str[ticket->t_trans_type]),
 		 ticket->t_trans_type);
 	xfs_warn(mp, "  unit res    = %d bytes",
 		 ticket->t_unit_res);
@@ -2078,7 +2082,7 @@ xlog_print_tic_res(
 		uint r_type = ticket->t_res_arr[i].r_type;
 		xfs_warn(mp, "region[%u]: %s - %u bytes", i,
 			    ((r_type <= 0 || r_type > XLOG_REG_TYPE_MAX) ?
-			    "bad-rtype" : res_type_str[r_type-1]),
+			    "bad-rtype" : res_type_str[r_type]),
 			    ticket->t_res_arr[i].r_len);
 	}
 

From darrick.wong@oracle.com  Fri Jan 22 18:34:43 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 99F4D7F53
	for ; Fri, 22 Jan 2016 18:34:43 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 27FC7AC002
	for ; Fri, 22 Jan 2016 16:34:43 -0800 (PST)
X-ASG-Debug-ID: 1453509281-04cbb026ff71bd0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id rhDwTq1vFNnG24eH (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:34:41 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0YeB5027656
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:34:40 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0YddD006475
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:34:40 GMT
Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0YdRe005025;
	Sat, 23 Jan 2016 00:34:39 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:34:39 -0800
Subject: [PATCH 2/3] xfs: move struct xfs_attr_shortform to xfs_da_format.h
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 2/3] xfs: move struct xfs_attr_shortform to xfs_da_format.h
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:34:38 -0800
Message-ID: <20160123003438.2338.51748.stgit@birch.djwong.org>
In-Reply-To: <20160123003425.2338.18133.stgit@birch.djwong.org>
References: <20160123003425.2338.18133.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509281
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Move the shortform attr structure definition to the same place as the
other attribute structure definitions for consistency and also so that
xfs/122 verifies the structure size.

Signed-off-by: Darrick J. Wong 
---
 fs/xfs/libxfs/xfs_attr_sf.h    |   16 ----------------
 fs/xfs/libxfs/xfs_da_format.h  |   16 ++++++++++++++++
 fs/xfs/libxfs/xfs_inode_fork.c |    1 +
 3 files changed, 17 insertions(+), 16 deletions(-)


diff --git a/fs/xfs/libxfs/xfs_attr_sf.h b/fs/xfs/libxfs/xfs_attr_sf.h
index 919756e..90928bb 100644
--- a/fs/xfs/libxfs/xfs_attr_sf.h
+++ b/fs/xfs/libxfs/xfs_attr_sf.h
@@ -24,22 +24,6 @@
  * Small attribute lists are packed as tightly as possible so as
  * to fit into the literal area of the inode.
  */
-
-/*
- * Entries are packed toward the top as tight as possible.
- */
-typedef struct xfs_attr_shortform {
-	struct xfs_attr_sf_hdr {	/* constant-structure header block */
-		__be16	totsize;	/* total bytes in shortform list */
-		__u8	count;	/* count of active entries */
-	} hdr;
-	struct xfs_attr_sf_entry {
-		__uint8_t namelen;	/* actual length of name (no NULL) */
-		__uint8_t valuelen;	/* actual length of value (no NULL) */
-		__uint8_t flags;	/* flags bits (see xfs_attr_leaf.h) */
-		__uint8_t nameval[1];	/* name & value bytes concatenated */
-	} list[1];			/* variable sized array */
-} xfs_attr_shortform_t;
 typedef struct xfs_attr_sf_hdr xfs_attr_sf_hdr_t;
 typedef struct xfs_attr_sf_entry xfs_attr_sf_entry_t;
 
diff --git a/fs/xfs/libxfs/xfs_da_format.h b/fs/xfs/libxfs/xfs_da_format.h
index b14bbd6..8d4d8bc 100644
--- a/fs/xfs/libxfs/xfs_da_format.h
+++ b/fs/xfs/libxfs/xfs_da_format.h
@@ -641,6 +641,22 @@ xfs_dir2_block_leaf_p(struct xfs_dir2_block_tail *btp)
  */
 #define XFS_ATTR_LEAF_MAPSIZE	3	/* how many freespace slots */
 
+/*
+ * Entries are packed toward the top as tight as possible.
+ */
+typedef struct xfs_attr_shortform {
+	struct xfs_attr_sf_hdr {	/* constant-structure header block */
+		__be16	totsize;	/* total bytes in shortform list */
+		__u8	count;	/* count of active entries */
+	} hdr;
+	struct xfs_attr_sf_entry {
+		__uint8_t namelen;	/* actual length of name (no NULL) */
+		__uint8_t valuelen;	/* actual length of value (no NULL) */
+		__uint8_t flags;	/* flags bits (see xfs_attr_leaf.h) */
+		__uint8_t nameval[1];	/* name & value bytes concatenated */
+	} list[1];			/* variable sized array */
+} xfs_attr_shortform_t;
+
 typedef struct xfs_attr_leaf_map {	/* RLE map of free bytes */
 	__be16	base;			  /* base of free region */
 	__be16	size;			  /* length of free region */
diff --git a/fs/xfs/libxfs/xfs_inode_fork.c b/fs/xfs/libxfs/xfs_inode_fork.c
index 0defbd0..ef22a78 100644
--- a/fs/xfs/libxfs/xfs_inode_fork.c
+++ b/fs/xfs/libxfs/xfs_inode_fork.c
@@ -31,6 +31,7 @@
 #include "xfs_error.h"
 #include "xfs_trace.h"
 #include "xfs_attr_sf.h"
+#include "xfs_da_format.h"
 
 kmem_zone_t *xfs_ifork_zone;
 

From darrick.wong@oracle.com  Fri Jan 22 18:34:50 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 434597F37
	for ; Fri, 22 Jan 2016 18:34:50 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id D42A4AC001
	for ; Fri, 22 Jan 2016 16:34:49 -0800 (PST)
X-ASG-Debug-ID: 1453509287-04bdf04d6c6c5c0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id Faf8Sp1q3Yr9ddge (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:34:47 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0Ykkp027684
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:34:47 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0Yk53031835
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:34:46 GMT
Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0Yjla012253;
	Sat, 23 Jan 2016 00:34:46 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:34:45 -0800
Subject: [PATCH 3/3] xfs: check sizes of XFS on-disk structures at compile
 time
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 3/3] xfs: check sizes of XFS on-disk structures at compile
 time
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:34:44 -0800
Message-ID: <20160123003444.2338.37236.stgit@birch.djwong.org>
In-Reply-To: <20160123003425.2338.18133.stgit@birch.djwong.org>
References: <20160123003425.2338.18133.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509287
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Check the sizes of XFS on-disk structures when compiling the kernel.
Use this to catch inadvertent changes in structure size due to padding
and alignment issues, etc.

Signed-off-by: Darrick J. Wong 
---
 fs/xfs/xfs_ondisk.h |  108 +++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_super.c  |    3 +
 2 files changed, 111 insertions(+)
 create mode 100644 fs/xfs/xfs_ondisk.h


diff --git a/fs/xfs/xfs_ondisk.h b/fs/xfs/xfs_ondisk.h
new file mode 100644
index 0000000..56fab46
--- /dev/null
+++ b/fs/xfs/xfs_ondisk.h
@@ -0,0 +1,108 @@
+/*
+ * Copyright (c) 2016 Oracle.
+ * All Rights Reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write the Free Software Foundation,
+ * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+#ifndef __XFS_ONDISK_H
+#define __XFS_ONDISK_H
+
+#define XFS_CHECK_STRUCT_SIZE(structname, size) \
+	BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(struct " \
+		#structname ") is wrong, expected " #size)
+
+static inline void __init
+xfs_check_ondisk_structs(void)
+{
+	/* ag/file structures */
+	XFS_CHECK_STRUCT_SIZE(struct xfs_acl,			4);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_acl_entry,		12);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_agf,			224);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_agfl,			36);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_agi,			336);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_key,		8);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_rec,		16);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_bmdr_block,		4);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_btree_block,		72);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dinode,		176);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_disk_dquot,		104);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dqblk,			136);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dsb,			264);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dsymlink_hdr,		56);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_key,		4);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_rec,		16);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp,		8);
+	XFS_CHECK_STRUCT_SIZE(xfs_alloc_key_t,			8);
+	XFS_CHECK_STRUCT_SIZE(xfs_alloc_ptr_t,			4);
+	XFS_CHECK_STRUCT_SIZE(xfs_alloc_rec_t,			8);
+	XFS_CHECK_STRUCT_SIZE(xfs_inobt_ptr_t,			4);
+
+	/* dir/attr trees */
+	XFS_CHECK_STRUCT_SIZE(struct xfs_attr3_leaf_hdr,	80);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_attr3_leafblock,	88);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_attr3_rmt_hdr,		56);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_da3_blkinfo,		56);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_da3_intnode,		64);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_da3_node_hdr,		64);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_blk_hdr,		48);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_data_hdr,		64);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_free,		64);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_free_hdr,		64);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_leaf,		64);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_leaf_hdr,		64);
+	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_entry_t,		8);
+	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_hdr_t,		32);
+	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_map_t,		4);
+	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_name_local_t,	4);
+	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_name_remote_t,	12);
+	XFS_CHECK_STRUCT_SIZE(xfs_attr_leafblock_t,		40);
+	XFS_CHECK_STRUCT_SIZE(xfs_attr_shortform_t,		8);
+	XFS_CHECK_STRUCT_SIZE(xfs_da_blkinfo_t,			12);
+	XFS_CHECK_STRUCT_SIZE(xfs_da_intnode_t,			16);
+	XFS_CHECK_STRUCT_SIZE(xfs_da_node_entry_t,		8);
+	XFS_CHECK_STRUCT_SIZE(xfs_da_node_hdr_t,		16);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_data_free_t,		4);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_data_hdr_t,		16);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_data_unused_t,		6);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_free_hdr_t,		16);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_free_t,			16);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_ino4_t,			4);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_ino8_t,			8);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_inou_t,			8);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_leaf_entry_t,		8);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_leaf_hdr_t,		16);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_leaf_t,			16);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_leaf_tail_t,		4);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_sf_entry_t,		3);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_sf_hdr_t,		10);
+	XFS_CHECK_STRUCT_SIZE(xfs_dir2_sf_off_t,		2);
+
+	/* log structures */
+	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_32,	28);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_extent_32,		12);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_extent_64,		16);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_icdinode,		176);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_ictimestamp,		8);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_32,	52);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_64,	56);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
+	XFS_CHECK_STRUCT_SIZE(struct xfs_trans_header,		16);
+}
+
+#endif /* __XFS_ONDISK_H */
diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c
index 36bd882..f63d212 100644
--- a/fs/xfs/xfs_super.c
+++ b/fs/xfs/xfs_super.c
@@ -45,6 +45,7 @@
 #include "xfs_filestream.h"
 #include "xfs_quota.h"
 #include "xfs_sysfs.h"
+#include "xfs_ondisk.h"
 
 #include 
 #include 
@@ -1817,6 +1818,8 @@ init_xfs_fs(void)
 {
 	int			error;
 
+	xfs_check_ondisk_structs();
+
 	printk(KERN_INFO XFS_VERSION_STRING " with "
 			 XFS_BUILD_OPTIONS " enabled\n");
 

From darrick.wong@oracle.com  Fri Jan 22 18:35:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id A96BA7F37
	for ; Fri, 22 Jan 2016 18:35:13 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 9C9308F8040
	for ; Fri, 22 Jan 2016 16:35:13 -0800 (PST)
X-ASG-Debug-ID: 1453509305-04cb6c0d5f7c100001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id qr1jEVgVQJnJEney (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:35:05 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0Z4IJ019632
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:04 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0Z4fO032382
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:04 GMT
Received: from abhmp0017.oracle.com (abhmp0017.oracle.com [141.146.116.23])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0Z3UC012335;
	Sat, 23 Jan 2016 00:35:04 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:35:03 -0800
Subject: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:35:02 -0800
Message-ID: <20160123003502.2475.99558.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453509305
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Configure getopt to accept the -R argument to pwrite.

Signed-off-by: Darrick J. Wong 
---
 io/pwrite.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/io/pwrite.c b/io/pwrite.c
index fd9114d..c6efee9 100644
--- a/io/pwrite.c
+++ b/io/pwrite.c
@@ -259,7 +259,7 @@ pwrite_f(
 	init_cvtnum(&fsblocksize, &fssectsize);
 	bsize = fsblocksize;
 
-	while ((c = getopt(argc, argv, "b:Cdf:i:qs:S:uV:wWZ:")) != EOF) {
+	while ((c = getopt(argc, argv, "b:Cdf:i:qRs:S:uV:wWZ:")) != EOF) {
 		switch (c) {
 		case 'b':
 			tmp = cvtnum(fsblocksize, fssectsize, optarg);

From darrick.wong@oracle.com  Fri Jan 22 18:35:15 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 451387F5F
	for ; Fri, 22 Jan 2016 18:35:15 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id CABC3AC007
	for ; Fri, 22 Jan 2016 16:35:14 -0800 (PST)
X-ASG-Debug-ID: 1453509312-04cb6c0d607c110001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id lgtMWDSzixiB7eUr (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:35:13 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0ZC65028220
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:12 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0ZAJk018784
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:10 GMT
Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0Z95V005125;
	Sat, 23 Jan 2016 00:35:09 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:35:09 -0800
Subject: [PATCH 2/5] libxfs: refactor the btree size calculator code
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 2/5] libxfs: refactor the btree size calculator code
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:35:08 -0800
Message-ID: <20160123003508.2475.53723.stgit@birch.djwong.org>
In-Reply-To: <20160123003502.2475.99558.stgit@birch.djwong.org>
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: aserv0021.oracle.com [141.146.126.233]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509313
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Create a macro to generate btree height calculator functions.
This will be used (much) later when we get to the refcount
btree.

Signed-off-by: Darrick J. Wong 
---
 libxfs/xfs_bmap.c       |   18 +-----------------
 libxfs/xfs_bmap_btree.c |    9 +++++++++
 libxfs/xfs_bmap_btree.h |    3 +++
 libxfs/xfs_btree.c      |   28 ++++++++++++++++++++++++++++
 libxfs/xfs_btree.h      |    3 +++
 5 files changed, 44 insertions(+), 17 deletions(-)


diff --git a/libxfs/xfs_bmap.c b/libxfs/xfs_bmap.c
index aef7cf3..c134765 100644
--- a/libxfs/xfs_bmap.c
+++ b/libxfs/xfs_bmap.c
@@ -172,25 +172,9 @@ xfs_bmap_worst_indlen(
 	xfs_inode_t	*ip,		/* incore inode pointer */
 	xfs_filblks_t	len)		/* delayed extent length */
 {
-	int		level;		/* btree level number */
-	int		maxrecs;	/* maximum record count at this level */
-	xfs_mount_t	*mp;		/* mount structure */
 	xfs_filblks_t	rval;		/* return value */
 
-	mp = ip->i_mount;
-	maxrecs = mp->m_bmap_dmxr[0];
-	for (level = 0, rval = 0;
-	     level < XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK);
-	     level++) {
-		len += maxrecs - 1;
-		do_div(len, maxrecs);
-		rval += len;
-		if (len == 1)
-			return rval + XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
-				level - 1;
-		if (level == 0)
-			maxrecs = mp->m_bmap_dmxr[1];
-	}
+	rval = xfs_bmbt_calc_size(ip->i_mount, len);
 	return rval;
 }
 
diff --git a/libxfs/xfs_bmap_btree.c b/libxfs/xfs_bmap_btree.c
index 2ef1836..3c595e2 100644
--- a/libxfs/xfs_bmap_btree.c
+++ b/libxfs/xfs_bmap_btree.c
@@ -880,3 +880,12 @@ xfs_bmbt_change_owner(
 	xfs_btree_del_cursor(cur, error ? XFS_BTREE_ERROR : XFS_BTREE_NOERROR);
 	return error;
 }
+
+xfs_filblks_t
+xfs_bmbt_calc_size(
+	struct xfs_mount	*mp,
+	unsigned long		len)
+{
+	return xfs_btree_calc_size(mp, mp->m_bmap_dmxr,
+			XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK), len);
+}
diff --git a/libxfs/xfs_bmap_btree.h b/libxfs/xfs_bmap_btree.h
index 819a8a4..04bc6e2 100644
--- a/libxfs/xfs_bmap_btree.h
+++ b/libxfs/xfs_bmap_btree.h
@@ -140,4 +140,7 @@ extern int xfs_bmbt_change_owner(struct xfs_trans *tp, struct xfs_inode *ip,
 extern struct xfs_btree_cur *xfs_bmbt_init_cursor(struct xfs_mount *,
 		struct xfs_trans *, struct xfs_inode *, int);
 
+extern xfs_filblks_t xfs_bmbt_calc_size(struct xfs_mount *mp,
+		unsigned long len);
+
 #endif	/* __XFS_BMAP_BTREE_H__ */
diff --git a/libxfs/xfs_btree.c b/libxfs/xfs_btree.c
index 658f27e..d5f16c5 100644
--- a/libxfs/xfs_btree.c
+++ b/libxfs/xfs_btree.c
@@ -4081,6 +4081,34 @@ xfs_btree_change_owner(
 	return 0;
 }
 
+/*
+ * xfs_btree_calc_size() -- Calculate the number of blocks needed to store
+ *			    a given number of records.
+ */
+xfs_filblks_t
+xfs_btree_calc_size(
+	struct xfs_mount	*mp,
+	uint			*limits,
+	int			maxlevels,
+	unsigned long		len)
+{
+	int			level;
+	int			maxrecs;
+	xfs_filblks_t		rval;
+
+	maxrecs = limits[0];
+	for (level = 0, rval = 0; level < maxlevels; level++) {
+		len += maxrecs - 1;
+		do_div(len, maxrecs);
+		rval += len;
+		if (len == 1)
+			return rval + maxlevels - level - 1;
+		if (level == 0)
+			maxrecs = limits[1];
+	}
+	return rval;
+}
+
 /**
  * xfs_btree_sblock_v5hdr_verify() -- verify the v5 fields of a short-format
  *				      btree block
diff --git a/libxfs/xfs_btree.h b/libxfs/xfs_btree.h
index 48cb251..b25ffd3 100644
--- a/libxfs/xfs_btree.h
+++ b/libxfs/xfs_btree.h
@@ -465,6 +465,9 @@ static inline int xfs_btree_get_level(struct xfs_btree_block *block)
 #define XFS_BTREE_TRACE_ARGR(c, r)
 #define	XFS_BTREE_TRACE_CURSOR(c, t)
 
+xfs_filblks_t xfs_btree_calc_size(struct xfs_mount *mp, uint *limits,
+		int maxlevels, unsigned long len);
+
 bool xfs_btree_sblock_v5hdr_verify(struct xfs_buf *bp);
 bool xfs_btree_sblock_verify(struct xfs_buf *bp, unsigned int max_recs);
 

From darrick.wong@oracle.com  Fri Jan 22 18:35:24 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id D5AA77F4E
	for ; Fri, 22 Jan 2016 18:35:24 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id C9BF1304039
	for ; Fri, 22 Jan 2016 16:35:24 -0800 (PST)
X-ASG-Debug-ID: 1453509322-04cbb026ff71c20001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id 93Tqf5WjEEjnw9vo (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:35:23 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0ZLA7019751
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:22 GMT
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75])
	by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0ZLPa019328
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:21 GMT
Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22])
	by userv0122.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0ZKgZ021555;
	Sat, 23 Jan 2016 00:35:20 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:35:20 -0800
Subject: [PATCH 3/5] libxfs: move struct xfs_attr_shortform to
 xfs_da_format.h
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 3/5] libxfs: move struct xfs_attr_shortform to
 xfs_da_format.h
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:35:14 -0800
Message-ID: <20160123003514.2475.80318.stgit@birch.djwong.org>
In-Reply-To: <20160123003502.2475.99558.stgit@birch.djwong.org>
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: aserv0021.oracle.com [141.146.126.233]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453509323
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Move the shortform attr structure definition to the same place as the
other attribute structure definitions for consistency and also so that
xfs/122 verifies the structure size.

Signed-off-by: Darrick J. Wong 
---
 libxfs/xfs_attr_sf.h    |   16 ----------------
 libxfs/xfs_da_format.h  |   16 ++++++++++++++++
 libxfs/xfs_inode_fork.c |    1 +
 3 files changed, 17 insertions(+), 16 deletions(-)


diff --git a/libxfs/xfs_attr_sf.h b/libxfs/xfs_attr_sf.h
index 919756e..90928bb 100644
--- a/libxfs/xfs_attr_sf.h
+++ b/libxfs/xfs_attr_sf.h
@@ -24,22 +24,6 @@
  * Small attribute lists are packed as tightly as possible so as
  * to fit into the literal area of the inode.
  */
-
-/*
- * Entries are packed toward the top as tight as possible.
- */
-typedef struct xfs_attr_shortform {
-	struct xfs_attr_sf_hdr {	/* constant-structure header block */
-		__be16	totsize;	/* total bytes in shortform list */
-		__u8	count;	/* count of active entries */
-	} hdr;
-	struct xfs_attr_sf_entry {
-		__uint8_t namelen;	/* actual length of name (no NULL) */
-		__uint8_t valuelen;	/* actual length of value (no NULL) */
-		__uint8_t flags;	/* flags bits (see xfs_attr_leaf.h) */
-		__uint8_t nameval[1];	/* name & value bytes concatenated */
-	} list[1];			/* variable sized array */
-} xfs_attr_shortform_t;
 typedef struct xfs_attr_sf_hdr xfs_attr_sf_hdr_t;
 typedef struct xfs_attr_sf_entry xfs_attr_sf_entry_t;
 
diff --git a/libxfs/xfs_da_format.h b/libxfs/xfs_da_format.h
index b14bbd6..8d4d8bc 100644
--- a/libxfs/xfs_da_format.h
+++ b/libxfs/xfs_da_format.h
@@ -641,6 +641,22 @@ xfs_dir2_block_leaf_p(struct xfs_dir2_block_tail *btp)
  */
 #define XFS_ATTR_LEAF_MAPSIZE	3	/* how many freespace slots */
 
+/*
+ * Entries are packed toward the top as tight as possible.
+ */
+typedef struct xfs_attr_shortform {
+	struct xfs_attr_sf_hdr {	/* constant-structure header block */
+		__be16	totsize;	/* total bytes in shortform list */
+		__u8	count;	/* count of active entries */
+	} hdr;
+	struct xfs_attr_sf_entry {
+		__uint8_t namelen;	/* actual length of name (no NULL) */
+		__uint8_t valuelen;	/* actual length of value (no NULL) */
+		__uint8_t flags;	/* flags bits (see xfs_attr_leaf.h) */
+		__uint8_t nameval[1];	/* name & value bytes concatenated */
+	} list[1];			/* variable sized array */
+} xfs_attr_shortform_t;
+
 typedef struct xfs_attr_leaf_map {	/* RLE map of free bytes */
 	__be16	base;			  /* base of free region */
 	__be16	size;			  /* length of free region */
diff --git a/libxfs/xfs_inode_fork.c b/libxfs/xfs_inode_fork.c
index e1968b4..f40649e 100644
--- a/libxfs/xfs_inode_fork.c
+++ b/libxfs/xfs_inode_fork.c
@@ -27,6 +27,7 @@
 #include "xfs_bmap.h"
 #include "xfs_trace.h"
 #include "xfs_attr_sf.h"
+#include "xfs_da_format.h"
 
 kmem_zone_t *xfs_ifork_zone;
 

From darrick.wong@oracle.com  Fri Jan 22 18:35:30 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 0079D7F61
	for ; Fri, 22 Jan 2016 18:35:30 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 81845AC004
	for ; Fri, 22 Jan 2016 16:35:29 -0800 (PST)
X-ASG-Debug-ID: 1453509327-04cbb026ff71c40001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id SpeHJsnRWVs0pViN (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:35:27 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0ZRR2028326
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:27 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0ZQCh019479
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:26 GMT
Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0ZQwm012552;
	Sat, 23 Jan 2016 00:35:26 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:35:26 -0800
Subject: [PATCH 4/5] xfs_db: don't error out when blocksize > 64 * inodesize
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 4/5] xfs_db: don't error out when blocksize > 64 * inodesize
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:35:25 -0800
Message-ID: <20160123003525.2475.44264.stgit@birch.djwong.org>
In-Reply-To: <20160123003502.2475.99558.stgit@birch.djwong.org>
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: aserv0021.oracle.com [141.146.126.233]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509327
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

When the block size is large enough that multiple inode chunks can fit
in a single block (e.g. 64k blocks, 512 byte inodes) we must calculate
the per-chunk block size and the buffer offset correctly so that check
actually examines the correct metadata.

Signed-off-by: Darrick J. Wong 
---
 db/check.c |   13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)


diff --git a/db/check.c b/db/check.c
index 838db53..4e2768e 100644
--- a/db/check.c
+++ b/db/check.c
@@ -4386,6 +4386,7 @@ scanfunc_ino(
 	__u16			holemask;
 	xfs_agino_t		rino;
 	xfs_extlen_t		cblocks;
+	int			bufoff;
 
 	if (be32_to_cpu(block->bb_magic) != XFS_IBT_MAGIC &&
 	    be32_to_cpu(block->bb_magic) != XFS_IBT_CRC_MAGIC) {
@@ -4455,6 +4456,8 @@ scanfunc_ino(
 				rino = agino + startidx;
 				cblocks = (endidx - startidx) >>
 						mp->m_sb.sb_inopblog;
+				if (cblocks == 0)
+					cblocks = 1;
 
 				/* Check the sparse chunk alignment */
 				if (sparse &&
@@ -4468,8 +4471,9 @@ scanfunc_ino(
 				}
 
 				/* Check the block map */
-				set_dbmap(seqno, XFS_AGINO_TO_AGBNO(mp, rino),
-					cblocks, DBM_INODE, seqno, bno);
+				if (XFS_AGINO_TO_OFFSET(mp, rino) == 0)
+					set_dbmap(seqno, XFS_AGINO_TO_AGBNO(mp, rino),
+						cblocks, DBM_INODE, seqno, bno);
 
 				push_cur();
 				set_cur(&typtab[TYP_INODE],
@@ -4489,14 +4493,15 @@ scanfunc_ino(
 				}
 
 				/* Examine each inode in this chunk */
-				for (j = startidx; j < endidx; j++) {
+				bufoff = ((XFS_AGINO_TO_OFFSET(mp, rino) - startidx) << mp->m_sb.sb_inodelog);
+				for (j = startidx; j < endidx; j++, bufoff += (1 << mp->m_sb.sb_inodelog)) {
 					if (ino_issparse(&rp[i], j))
 						continue;
 					isfree = XFS_INOBT_IS_FREE_DISK(&rp[i], j);
 					if (isfree)
 						nfree++;
 					process_inode(agf, agino + j,
-						(xfs_dinode_t *)((char *)iocur_top->data + ((j - startidx) << mp->m_sb.sb_inodelog)),
+						(xfs_dinode_t *)((char *)iocur_top->data + bufoff),
 							isfree);
 				}
 				pop_cur();

From darrick.wong@oracle.com  Fri Jan 22 18:35:38 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 952C67F62
	for ; Fri, 22 Jan 2016 18:35:38 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 44114AC004
	for ; Fri, 22 Jan 2016 16:35:38 -0800 (PST)
X-ASG-Debug-ID: 1453509335-04bdf04d6a6c620001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id GZmwrTRI7sv1zJY1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:35:35 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0ZYb6028417
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:35 GMT
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0ZY98000763
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:35:34 GMT
Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8])
	by userv0122.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0ZYRf021612;
	Sat, 23 Jan 2016 00:35:34 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:35:33 -0800
Subject: [PATCH 5/5] xfs_io: print dedupe errors to stderr, not stdout
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 5/5] xfs_io: print dedupe errors to stderr, not stdout
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:35:31 -0800
Message-ID: <20160123003531.2475.59752.stgit@birch.djwong.org>
In-Reply-To: <20160123003502.2475.99558.stgit@birch.djwong.org>
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509335
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Signed-off-by: Darrick J. Wong 
---
 io/reflink.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)


diff --git a/io/reflink.c b/io/reflink.c
index def01be..a09e82d 100644
--- a/io/reflink.c
+++ b/io/reflink.c
@@ -78,11 +78,13 @@ dedupe_ioctl(
 			goto done;
 		}
 		if (info->status < 0) {
-			printf("dedupe: %s\n", _(strerror(-info->status)));
+			fprintf(stderr, "dedupe: %s\n",
+					_(strerror(-info->status)));
 			goto done;
 		}
 		if (info->status == XFS_EXTENT_DATA_DIFFERS) {
-			printf(_("Extents did not match.\n"));
+			fprintf(stderr, "dedupe: %s\n",
+					_("Extents did not match."));
 			goto done;
 		}
 		if (args->length != 0 &&

From darrick.wong@oracle.com  Fri Jan 22 18:36:18 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 6A1B67F51
	for ; Fri, 22 Jan 2016 18:36:18 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 408D38F804B
	for ; Fri, 22 Jan 2016 16:36:18 -0800 (PST)
X-ASG-Debug-ID: 1453509376-04bdf04d6c6c660001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id SAJPfFV2MSVkM7xn (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:36:16 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0aFZQ020413
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:36:15 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0aFOl009229
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:36:15 GMT
Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0aEdI005537;
	Sat, 23 Jan 2016 00:36:14 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:36:14 -0800
Subject: [PATCH 1/2] xfsdocs: document known testing procedures
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 1/2] xfsdocs: document known testing procedures
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:36:13 -0800
Message-ID: <20160123003613.2605.14510.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453509376
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Signed-off-by: Darrick J. Wong 
---
 design/XFS_Filesystem_Structure/docinfo.xml        |   14 ++++++++++++
 design/XFS_Filesystem_Structure/testing.asciidoc   |   23 ++++++++++++++++++++
 .../xfs_filesystem_structure.asciidoc              |    2 ++
 3 files changed, 39 insertions(+)
 create mode 100644 design/XFS_Filesystem_Structure/testing.asciidoc


diff --git a/design/XFS_Filesystem_Structure/docinfo.xml b/design/XFS_Filesystem_Structure/docinfo.xml
index ba97809..cc5596d 100644
--- a/design/XFS_Filesystem_Structure/docinfo.xml
+++ b/design/XFS_Filesystem_Structure/docinfo.xml
@@ -108,4 +108,18 @@
 			
 		
 	
+	
+		3.14
+		January 2016
+		
+			Darrick
+			Wong
+			
+		
+		
+			
+				Document disk format change testing.
+			
+		
+	
 
diff --git a/design/XFS_Filesystem_Structure/testing.asciidoc b/design/XFS_Filesystem_Structure/testing.asciidoc
new file mode 100644
index 0000000..f1c90bc
--- /dev/null
+++ b/design/XFS_Filesystem_Structure/testing.asciidoc
@@ -0,0 +1,23 @@
+[[Testing]]
+= Testing Filesystem Changes
+
+People put a lot of trust in filesystems to preserve their data in a reliable
+fashion.  To that end, it is very important that users and developers have
+access to a suite of regression tests that can be used to prove correct
+operation of any given filesystem code, or to analyze failures to fix problems
+found in the code.  The XFS regression test suite, +xfstests+, is hosted at
++git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git+.  Most tests apply to
+filesystems in general, but the suite also contains tests for features specific
+to each filesystem.
+
+When fixing bugs, it is important to provide a testcase exposing the bug so
+that the developers can avoid a future re-occurrence of the regression.
+Furthermore, if you're developing a new user-visible feature for XFS, please
+help the rest of the development community to sustain and maintain the whole
+codebase by providing generous test coverage to check its behavior.
+
+When altering, adding, or removing an on-disk data structure, please remember
+to update both the in-kernel structure size checks in +xfs_ondisk.h+ and to
+ensure that your changes are reflected in xfstest xfs/122.  These regression
+tests enable us to detect compiler bugs, alignment problems, and anything
+else that might result in the creation of incompatible filesystem images.
diff --git a/design/XFS_Filesystem_Structure/xfs_filesystem_structure.asciidoc b/design/XFS_Filesystem_Structure/xfs_filesystem_structure.asciidoc
index 53262bf..f580aab 100644
--- a/design/XFS_Filesystem_Structure/xfs_filesystem_structure.asciidoc
+++ b/design/XFS_Filesystem_Structure/xfs_filesystem_structure.asciidoc
@@ -52,6 +52,8 @@ include::common_types.asciidoc[]
 
 include::magic.asciidoc[]
 
+include::testing.asciidoc[]
+
 // return titles to normal
 :leveloffset: 0
 

From darrick.wong@oracle.com  Fri Jan 22 18:36:24 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 290517F51
	for ; Fri, 22 Jan 2016 18:36:24 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 1C3348F8040
	for ; Fri, 22 Jan 2016 16:36:24 -0800 (PST)
X-ASG-Debug-ID: 1453509382-04cb6c0d5e7c160001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id DSi00FYHPvT9w9Wj (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:36:22 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0aLrK020435
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:36:21 GMT
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0aLiW002274
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:36:21 GMT
Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18])
	by userv0122.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0aLSp021900;
	Sat, 23 Jan 2016 00:36:21 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:36:20 -0800
Subject: [PATCH 2/2] xfsdocs: update the on-disk format with changes for
 Linux 4.5
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 2/2] xfsdocs: update the on-disk format with changes for
 Linux 4.5
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:36:19 -0800
Message-ID: <20160123003619.2605.76125.stgit@birch.djwong.org>
In-Reply-To: <20160123003613.2605.14510.stgit@birch.djwong.org>
References: <20160123003613.2605.14510.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453509382
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Signed-off-by: Darrick J. Wong 
---
 .../XFS_Filesystem_Structure/ondisk_inode.asciidoc |   14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)


diff --git a/design/XFS_Filesystem_Structure/ondisk_inode.asciidoc b/design/XFS_Filesystem_Structure/ondisk_inode.asciidoc
index 4aabc55..f1b0421 100644
--- a/design/XFS_Filesystem_Structure/ondisk_inode.asciidoc
+++ b/design/XFS_Filesystem_Structure/ondisk_inode.asciidoc
@@ -66,7 +66,7 @@ of the literal area and +di_forkoff+. The attribute fork is located between
 [[Inode_Core]]
 == Inode Core
 
-The inode's core is 96 bytes in size and contains information about the file
+The inode's core is 176 bytes in size and contains information about the file
 itself including most stat data information about data and attribute forks after
 the core within the inode. It uses the following structure:
 
@@ -313,8 +313,16 @@ Counts the number of changes made to the attributes in this inode.
 Log sequence number of the last inode write.
 
 *di_flags2*::
-Specifies extended flags associated with a v3 inode.  There are no flags defined
-currently.
+Specifies extended flags associated with a v3 inode.
+
+.Version 3 Inode flags
+[options="header"]
+|=====
+| Flag				| Description
+| +XFS_DIFLAG2_DAX+		|
+For a file, enable DAX to increase performance on persistent-memory storage.
+If set on a directory, files created in the directory will inherit this flag.
+|=====
 
 *di_pad2*::
 Padding for future expansion of the inode.

From darrick.wong@oracle.com  Fri Jan 22 18:36:53 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 564CE7F51
	for ; Fri, 22 Jan 2016 18:36:53 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 3CF6B304039
	for ; Fri, 22 Jan 2016 16:36:53 -0800 (PST)
X-ASG-Debug-ID: 1453509411-04cb6c0d607c180001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id a7L9aYkrwZ3F8XUF (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:36:51 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0aoYo029189
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:36:50 GMT
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0aoE1009966
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:36:50 GMT
Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19])
	by userv0122.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0anQp022110;
	Sat, 23 Jan 2016 00:36:49 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:36:49 -0800
Subject: [PATCH 1/5] tools/mvtest: call out to the correct group sort program
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 1/5] tools/mvtest: call out to the correct group sort program
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: fstests@vger.kernel.org, xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:36:48 -0800
Message-ID: <20160123003648.2666.37062.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509411
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

mvtest should call the actual name of the sort-group program.
Fix a cut-paste error in finding the destination group file.

Signed-off-by: Darrick J. Wong 
---
 tools/mvtest |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


diff --git a/tools/mvtest b/tools/mvtest
index af601d6..20bc23e 100755
--- a/tools/mvtest
+++ b/tools/mvtest
@@ -33,7 +33,7 @@ sgroup="$(basename "$(dirname "tests/${src}")")"
 dgroup="$(basename "$(dirname "tests/${dest}")")"
 
 sgroupfile="tests/${sgroup}/group"
-dgroupfile="tests/${sgroup}/group"
+dgroupfile="tests/${dgroup}/group"
 
 git mv "tests/${src}" "tests/${dest}"
 git mv "tests/${src}.out" "tests/${dest}.out"
@@ -47,7 +47,7 @@ newgrpline="$(echo "${grpline}" | sed -e "s/^${sid} /${did} /g")"
 sed -e "/^${sid}.*$/d" -i "${sgroupfile}"
 cp "${dgroupfile}" "${dgroupfile}.new"
 append "${dgroupfile}.new" "${newgrpline}"
-"${dir}/sort-group.py" "${dgroupfile}.new"
+"${dir}/sort-group" "${dgroupfile}.new"
 mv "${dgroupfile}.new" "${dgroupfile}"
 
 echo "Moved \"${src}\" to \"${dest}\"."

From darrick.wong@oracle.com  Fri Jan 22 18:36:58 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 96B2D7F76
	for ; Fri, 22 Jan 2016 18:36:58 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 7B4FE8F8040
	for ; Fri, 22 Jan 2016 16:36:58 -0800 (PST)
X-ASG-Debug-ID: 1453509416-04cbb026fc71cd0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id mqS5pfhWivGCBd3x (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:36:57 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0auw0029216
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:36:56 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0audR022938
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:36:56 GMT
Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0atsA005732;
	Sat, 23 Jan 2016 00:36:56 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:36:55 -0800
Subject: [PATCH 2/5] reflink: fix off-by-one errors when iterating file
 blocks in a loop
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 2/5] reflink: fix off-by-one errors when iterating file
 blocks in a loop
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: fstests@vger.kernel.org, xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:36:54 -0800
Message-ID: <20160123003654.2666.17112.stgit@birch.djwong.org>
In-Reply-To: <20160123003648.2666.37062.stgit@birch.djwong.org>
References: <20160123003648.2666.37062.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: aserv0021.oracle.com [141.146.126.233]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509416
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

When we're iterating file blocks in a loop (via seq), we have to
end at $nr-1, not $nr.

Signed-off-by: Darrick J. Wong 
---
 tests/generic/183 |    2 +-
 tests/generic/185 |    2 +-
 tests/generic/186 |    2 +-
 tests/generic/187 |    2 +-
 tests/generic/194 |    2 +-
 tests/generic/195 |    2 +-
 tests/xfs/129     |    6 +++---
 tests/xfs/140     |    2 +-
 8 files changed, 10 insertions(+), 10 deletions(-)


diff --git a/tests/generic/183 b/tests/generic/183
index 5c65e9a..2cf9ce4 100755
--- a/tests/generic/183
+++ b/tests/generic/183
@@ -69,7 +69,7 @@ seq 0 2 $((NR-1)) | while read f; do
 	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
 	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
-seq 1 2 $NR | while read f; do
+seq 1 2 $((NR-1)) | while read f; do
 	_reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
 	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
diff --git a/tests/generic/185 b/tests/generic/185
index 384c9aa..55c54ac 100755
--- a/tests/generic/185
+++ b/tests/generic/185
@@ -69,7 +69,7 @@ seq 0 2 $((NR-1)) | while read f; do
 	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
 	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
-seq 1 2 $NR | while read f; do
+seq 1 2 $((NR-1)) | while read f; do
 	_reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
 	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
diff --git a/tests/generic/186 b/tests/generic/186
index d84166d..f1c0e6b 100755
--- a/tests/generic/186
+++ b/tests/generic/186
@@ -112,7 +112,7 @@ seq 0 2 $((NR-1)) | while read f; do
 	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
 	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
-seq 1 2 $NR | while read f; do
+seq 1 2 $((NR-1)) | while read f; do
 	_reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
 	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
diff --git a/tests/generic/187 b/tests/generic/187
index c85a0a9..ce35ec0 100755
--- a/tests/generic/187
+++ b/tests/generic/187
@@ -112,7 +112,7 @@ seq 0 2 $((NR-1)) | while read f; do
 	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
 	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
-seq 1 2 $NR | while read f; do
+seq 1 2 $((NR-1)) | while read f; do
 	_reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
 	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
diff --git a/tests/generic/194 b/tests/generic/194
index a441aee..b88297e 100755
--- a/tests/generic/194
+++ b/tests/generic/194
@@ -79,7 +79,7 @@ md5sum "$TESTDIR/file3" | _filter_scratch
 md5sum "$TESTDIR/file3.chk" | _filter_scratch
 
 echo "directio CoW across the transition"
-seq 1 2 $NR | while read f; do
+seq 1 2 $((NR-1)) | while read f; do
 	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
 	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
diff --git a/tests/generic/195 b/tests/generic/195
index 83e7b23..464a4d4 100755
--- a/tests/generic/195
+++ b/tests/generic/195
@@ -79,7 +79,7 @@ md5sum "$TESTDIR/file3" | _filter_scratch
 md5sum "$TESTDIR/file3.chk" | _filter_scratch
 
 echo "CoW across the transition"
-seq 1 2 $NR | while read f; do
+seq 1 2 $((NR-1)) | while read f; do
 	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
 	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
 done
diff --git a/tests/xfs/129 b/tests/xfs/129
index c8c47a2..3b7c59c 100755
--- a/tests/xfs/129
+++ b/tests/xfs/129
@@ -65,9 +65,9 @@ NR_BLKS=$((4 * BLKSZ / 12))
 _pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/file1" >> "$seqres.full"
 
 echo "Reflink every other block"
-seq 1 $((NR_BLKS / 2)) | while read nr; do
-	_reflink_range  "$TESTDIR/file1" $((nr * 2 * BLKSZ)) \
-			"$TESTDIR/file2" $((nr * 2 * BLKSZ)) $BLKSZ >> "$seqres.full"
+seq 1 2 $((NR_BLKS - 1)) | while read nr; do
+	_reflink_range  "$TESTDIR/file1" $((nr * BLKSZ)) \
+			"$TESTDIR/file2" $((nr * BLKSZ)) $BLKSZ >> "$seqres.full"
 done
 
 echo "Create metadump file"
diff --git a/tests/xfs/140 b/tests/xfs/140
index e20cd3f..45be9f4 100644
--- a/tests/xfs/140
+++ b/tests/xfs/140
@@ -70,7 +70,7 @@ cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "file1 and file2 do not match"
 cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
 
 echo "CoW every other block"
-seq 1 2 $NR | while read f; do
+seq 1 2 $((NR - 1)) | while read f; do
 	_pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
 	_pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
 done

From darrick.wong@oracle.com  Fri Jan 22 18:37:06 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 363717F5E
	for ; Fri, 22 Jan 2016 18:37:06 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 0E95E8F8040
	for ; Fri, 22 Jan 2016 16:37:06 -0800 (PST)
X-ASG-Debug-ID: 1453509423-04bdf04d6a6c670001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id 5qgO2E2tO2ZajEZQ (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:37:03 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0b22K029490
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:37:02 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0b2F4023143
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:37:02 GMT
Received: from abhmp0007.oracle.com (abhmp0007.oracle.com [141.146.116.13])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0b2VF005774;
	Sat, 23 Jan 2016 00:37:02 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:37:01 -0800
Subject: [PATCH 3/5] xfs/122: update against xfsprogs 4.3
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 3/5] xfs/122: update against xfsprogs 4.3
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: fstests@vger.kernel.org, xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:37:00 -0800
Message-ID: <20160123003700.2666.8133.stgit@birch.djwong.org>
In-Reply-To: <20160123003648.2666.37062.stgit@birch.djwong.org>
References: <20160123003648.2666.37062.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: aserv0021.oracle.com [141.146.126.233]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509423
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Make the xfsprogs sb layout and structure size checker work with
modern xfsprogs (i.e. 4.3+ and all the new v5 stuff).  The old test
wasn't picking up structures because they're no longer defined in the
"typedef struct xfs_foo { } xfs_foo_t;" style.

Signed-off-by: Darrick J. Wong 
---
 tests/xfs/122     |   29 +++++-
 tests/xfs/122.out |  265 +++++++++++++++++++++++++++++------------------------
 2 files changed, 166 insertions(+), 128 deletions(-)


diff --git a/tests/xfs/122 b/tests/xfs/122
index 682ffb4..e6697a2 100755
--- a/tests/xfs/122
+++ b/tests/xfs/122
@@ -74,10 +74,22 @@ progout=$tmp.output
 
 cat >$cprog <
-#include 
 EOF
+for hdr in /usr/include/xfs/xfs*.h; do
+	echo "#include <$(echo "$hdr" | sed -e 's|/usr/include/||g')>" >> $cprog
+done
+
+# missing:
+# xfs_attr_shortform_t
+# xfs_trans_header_t
 
 cat >$tmp.ignore <>$cprog
 #
 cat /usr/include/xfs/xfs*.h | indent |\
 _attribute_filter |\
-tee $seqres.full |\
-egrep '} *xfs_.*_t' |\
+egrep '(} *xfs_.*_t|^struct xfs_[a-z0-9_]*$)' > /tmp/barf
+
+cat /usr/include/xfs/xfs*.h | indent |\
+_attribute_filter |\
+egrep '(} *xfs_.*_t|^struct xfs_[a-z0-9_]*$)' |\
 egrep -v -f $tmp.ignore |\
-awk '{sub(/[;,]/,"",$2); print "printf(\"sizeof(", $2, ") = %d\\n\", sizeof(", $2, "));"}' \
->>$cprog
+sed -e 's/^.*}[[:space:]]*//g' -e 's/;.*$//g' -e 's/_t, /_t\n/g' |\
+sort | uniq |\
+awk '{printf("printf(\"sizeof(%s) = \\%zu\\n\", sizeof(%s));\n", $0, $0);}' |\
+cat >> $cprog
 
 #
 # Look at offsets of key ones which differ in lengths
@@ -192,7 +209,7 @@ cat /usr/include/xfs/xfs*.h | indent |\
 awk '
    /typedef struct xfs_sb/ { structon = 1; next }
    structon && $2 ~ /^sb_/ { sub(/[;,]/,"",$2)
-                             print "printf(\"offsetof(xfs_sb_t,", $2, ") = %d\\n\", offsetof(xfs_sb_t,", $2, "));"; next}
+                             printf("printf(\"offsetof(xfs_sb_t, %s) = \\%zu\\n\", offsetof(xfs_sb_t, %s));", $2, $2); next}
    structon && /}/ { structon = 0; next}
 '>>$cprog
 
diff --git a/tests/xfs/122.out b/tests/xfs/122.out
index c30e988..8ba121e 100644
--- a/tests/xfs/122.out
+++ b/tests/xfs/122.out
@@ -1,123 +1,144 @@
 QA output created by 122
-offsetof(xfs_sb_t, sb_agblklog ) = 124
-offsetof(xfs_sb_t, sb_agblocks ) = 84
-offsetof(xfs_sb_t, sb_agcount ) = 88
-offsetof(xfs_sb_t, sb_bad_features2 ) = 204
-offsetof(xfs_sb_t, sb_blocklog ) = 120
-offsetof(xfs_sb_t, sb_blocksize ) = 4
-offsetof(xfs_sb_t, sb_dblocks ) = 8
-offsetof(xfs_sb_t, sb_dirblklog ) = 192
-offsetof(xfs_sb_t, sb_fdblocks ) = 144
-offsetof(xfs_sb_t, sb_features2 ) = 200
-offsetof(xfs_sb_t, sb_flags ) = 178
-offsetof(xfs_sb_t, sb_fname[12] ) = 120
-offsetof(xfs_sb_t, sb_frextents ) = 152
-offsetof(xfs_sb_t, sb_gquotino ) = 168
-offsetof(xfs_sb_t, sb_icount ) = 128
-offsetof(xfs_sb_t, sb_ifree ) = 136
-offsetof(xfs_sb_t, sb_imax_pct ) = 127
-offsetof(xfs_sb_t, sb_inoalignmt ) = 180
-offsetof(xfs_sb_t, sb_inodelog ) = 122
-offsetof(xfs_sb_t, sb_inodesize ) = 104
-offsetof(xfs_sb_t, sb_inopblock ) = 106
-offsetof(xfs_sb_t, sb_inopblog ) = 123
-offsetof(xfs_sb_t, sb_inprogress ) = 126
-offsetof(xfs_sb_t, sb_logblocks ) = 96
-offsetof(xfs_sb_t, sb_logsectlog ) = 193
-offsetof(xfs_sb_t, sb_logsectsize ) = 194
-offsetof(xfs_sb_t, sb_logstart ) = 48
-offsetof(xfs_sb_t, sb_logsunit ) = 196
-offsetof(xfs_sb_t, sb_magicnum ) = 0
-offsetof(xfs_sb_t, sb_qflags ) = 176
-offsetof(xfs_sb_t, sb_rblocks ) = 16
-offsetof(xfs_sb_t, sb_rbmblocks ) = 92
-offsetof(xfs_sb_t, sb_rbmino ) = 64
-offsetof(xfs_sb_t, sb_rextents ) = 24
-offsetof(xfs_sb_t, sb_rextsize ) = 80
-offsetof(xfs_sb_t, sb_rextslog ) = 125
-offsetof(xfs_sb_t, sb_rootino ) = 56
-offsetof(xfs_sb_t, sb_rsumino ) = 72
-offsetof(xfs_sb_t, sb_sectlog ) = 121
-offsetof(xfs_sb_t, sb_sectsize ) = 102
-offsetof(xfs_sb_t, sb_shared_vn ) = 179
-offsetof(xfs_sb_t, sb_unit ) = 184
-offsetof(xfs_sb_t, sb_uquotino ) = 160
-offsetof(xfs_sb_t, sb_uuid ) = 32
-offsetof(xfs_sb_t, sb_versionnum ) = 100
-offsetof(xfs_sb_t, sb_width ) = 188
-sizeof( xfs_agf_t ) = 
-sizeof( xfs_agfl_t ) = 4
-sizeof( xfs_agi_t ) = 296
-sizeof( xfs_alloc_rec_incore_t ) = 8
-sizeof( xfs_alloc_rec_t ) = 8
-sizeof( xfs_attr_inactive_list_t ) = 8
-sizeof( xfs_attr_leaf_entry_t ) = 8
-sizeof( xfs_attr_leaf_hdr_t ) = 32
-sizeof( xfs_attr_leaf_map_t ) = 4
-sizeof( xfs_attr_leaf_name_local_t ) = 4
-sizeof( xfs_attr_leaf_name_remote_t ) = 12
-sizeof( xfs_attr_leafblock_t ) = 56
-sizeof( xfs_attr_shortform_t ) = 8
-sizeof( xfs_attrlist_cursor_t ) = 16
-sizeof( xfs_bmbt_key_t ) = 8
-sizeof( xfs_bmbt_rec_t ) = 16
-sizeof( xfs_bmdr_block_t ) = 4
-sizeof( xfs_btnum_t ) = 4
-sizeof( xfs_buf_log_format_t ) = 24
-sizeof( xfs_da_blkinfo_t ) = 12
-sizeof( xfs_da_intnode_t ) = 24
-sizeof( xfs_dinode_fmt_t ) = 4
-sizeof( xfs_dir2_data_free_t ) = 4
-sizeof( xfs_dir2_data_hdr_t ) = 16
-sizeof( xfs_dir2_data_unused_t ) = 6
-sizeof( xfs_dir2_free_hdr_t ) = 16
-sizeof( xfs_dir2_free_t ) = 20
-sizeof( xfs_dir2_ino4_t ) = 4
-sizeof( xfs_dir2_ino8_t ) = 8
-sizeof( xfs_dir2_inou_t ) = 8
-sizeof( xfs_dir2_leaf_entry_t ) = 8
-sizeof( xfs_dir2_leaf_hdr_t ) = 16
-sizeof( xfs_dir2_leaf_t ) = 32
-sizeof( xfs_dir2_leaf_tail_t ) = 4
-sizeof( xfs_dir2_sf_entry_t ) = 12
-sizeof( xfs_dir2_sf_hdr_t ) = 10
-sizeof( xfs_dir2_sf_off_t ) = 2
-sizeof( xfs_dir2_sf_t ) = 22
-sizeof( xfs_dir_leaf_entry_t ) = 8
-sizeof( xfs_dir_leaf_hdr_t ) = 32
-sizeof( xfs_dir_leaf_map_t ) = 4
-sizeof( xfs_dir_leaf_name_t ) = 9
-sizeof( xfs_dir_leafblock_t ) = 52
-sizeof( xfs_dir_sf_entry_t ) = 10
-sizeof( xfs_dir_sf_hdr_t ) = 9
-sizeof( xfs_dir_shortform_t ) = 19
-sizeof( xfs_disk_dquot_t ) = 104
-sizeof( xfs_dq_logformat_t ) = 24
-sizeof( xfs_dqblk_t ) = 136
-sizeof( xfs_dsb_t ) = 208
-sizeof( xfs_efd_log_format_32_t ) = 28
-sizeof( xfs_efd_log_format_64_t ) = 32
-sizeof( xfs_efi_log_format_32_t ) = 28
-sizeof( xfs_efi_log_format_64_t ) = 32
-sizeof( xfs_error_injection_t ) = 8
-sizeof( xfs_exntfmt_t ) = 4
-sizeof( xfs_exntst_t ) = 4
-sizeof( xfs_extent_32_t ) = 12
-sizeof( xfs_extent_64_t ) = 16
-sizeof( xfs_fid_t ) = 16
-sizeof( xfs_fsid_t ) = 8
-sizeof( xfs_fsop_counts_t ) = 32
-sizeof( xfs_fsop_geom_t ) = 112
-sizeof( xfs_fsop_resblks_t ) = 16
-sizeof( xfs_growfs_log_t ) = 8
-sizeof( xfs_handle_t ) = 24
-sizeof( xfs_inobt_key_t ) = 4
-sizeof( xfs_inobt_rec_t ) = 16
-sizeof( xfs_inode_log_format_32_t ) = 52
-sizeof( xfs_inode_log_format_64_t ) = 56
-sizeof( xfs_lookup_t ) = 4
-sizeof( xfs_qoff_logformat_t ) = 20
-sizeof( xfs_sb_field_t ) = 4
-sizeof( xfs_timestamp_t ) = 8
-sizeof( xfs_trans_header_t ) = 16
-sizeof( xfs_trans_reservations_t ) = 84
+offsetof(xfs_sb_t, sb_agblklog) = 124
+offsetof(xfs_sb_t, sb_agblocks) = 84
+offsetof(xfs_sb_t, sb_agcount) = 88
+offsetof(xfs_sb_t, sb_bad_features2) = 204
+offsetof(xfs_sb_t, sb_blocklog) = 120
+offsetof(xfs_sb_t, sb_blocksize) = 4
+offsetof(xfs_sb_t, sb_crc) = 224
+offsetof(xfs_sb_t, sb_dblocks) = 8
+offsetof(xfs_sb_t, sb_dirblklog) = 192
+offsetof(xfs_sb_t, sb_fdblocks) = 144
+offsetof(xfs_sb_t, sb_features2) = 200
+offsetof(xfs_sb_t, sb_features_compat) = 208
+offsetof(xfs_sb_t, sb_features_incompat) = 216
+offsetof(xfs_sb_t, sb_features_log_incompat) = 220
+offsetof(xfs_sb_t, sb_features_ro_compat) = 212
+offsetof(xfs_sb_t, sb_flags) = 178
+offsetof(xfs_sb_t, sb_fname[12]) = 120
+offsetof(xfs_sb_t, sb_frextents) = 152
+offsetof(xfs_sb_t, sb_gquotino) = 168
+offsetof(xfs_sb_t, sb_icount) = 128
+offsetof(xfs_sb_t, sb_ifree) = 136
+offsetof(xfs_sb_t, sb_imax_pct) = 127
+offsetof(xfs_sb_t, sb_inoalignmt) = 180
+offsetof(xfs_sb_t, sb_inodelog) = 122
+offsetof(xfs_sb_t, sb_inodesize) = 104
+offsetof(xfs_sb_t, sb_inopblock) = 106
+offsetof(xfs_sb_t, sb_inopblog) = 123
+offsetof(xfs_sb_t, sb_inprogress) = 126
+offsetof(xfs_sb_t, sb_logblocks) = 96
+offsetof(xfs_sb_t, sb_logsectlog) = 193
+offsetof(xfs_sb_t, sb_logsectsize) = 194
+offsetof(xfs_sb_t, sb_logstart) = 48
+offsetof(xfs_sb_t, sb_logsunit) = 196
+offsetof(xfs_sb_t, sb_lsn) = 240
+offsetof(xfs_sb_t, sb_magicnum) = 0
+offsetof(xfs_sb_t, sb_meta_uuid) = 248
+offsetof(xfs_sb_t, sb_pquotino) = 232
+offsetof(xfs_sb_t, sb_qflags) = 176
+offsetof(xfs_sb_t, sb_rblocks) = 16
+offsetof(xfs_sb_t, sb_rbmblocks) = 92
+offsetof(xfs_sb_t, sb_rbmino) = 64
+offsetof(xfs_sb_t, sb_rextents) = 24
+offsetof(xfs_sb_t, sb_rextsize) = 80
+offsetof(xfs_sb_t, sb_rextslog) = 125
+offsetof(xfs_sb_t, sb_rootino) = 56
+offsetof(xfs_sb_t, sb_rsumino) = 72
+offsetof(xfs_sb_t, sb_sectlog) = 121
+offsetof(xfs_sb_t, sb_sectsize) = 102
+offsetof(xfs_sb_t, sb_shared_vn) = 179
+offsetof(xfs_sb_t, sb_spino_align) = 228
+offsetof(xfs_sb_t, sb_unit) = 184
+offsetof(xfs_sb_t, sb_uquotino) = 160
+offsetof(xfs_sb_t, sb_uuid) = 32
+offsetof(xfs_sb_t, sb_versionnum) = 100
+offsetof(xfs_sb_t, sb_width) = 188
+sizeof(struct xfs_acl) = 4
+sizeof(struct xfs_acl_entry) = 12
+sizeof(struct xfs_attr3_leaf_hdr) = 80
+sizeof(struct xfs_attr3_leafblock) = 88
+sizeof(struct xfs_attr3_rmt_hdr) = 56
+sizeof(struct xfs_btree_block) = 72
+sizeof(struct xfs_clone_args) = 32
+sizeof(struct xfs_da3_blkinfo) = 56
+sizeof(struct xfs_da3_intnode) = 64
+sizeof(struct xfs_da3_node_hdr) = 64
+sizeof(struct xfs_dir3_blk_hdr) = 48
+sizeof(struct xfs_dir3_data_hdr) = 64
+sizeof(struct xfs_dir3_free) = 64
+sizeof(struct xfs_dir3_free_hdr) = 64
+sizeof(struct xfs_dir3_leaf) = 64
+sizeof(struct xfs_dir3_leaf_hdr) = 64
+sizeof(struct xfs_dsymlink_hdr) = 56
+sizeof(struct xfs_extent_data) = 24
+sizeof(struct xfs_extent_data_info) = 32
+sizeof(struct xfs_fs_eofblocks) = 128
+sizeof(struct xfs_icreate_log) = 28
+sizeof(xfs_agf_t) = 224
+sizeof(xfs_agfl_t) = 36
+sizeof(xfs_agi_t) = 336
+sizeof(xfs_alloc_key_t) = 8
+sizeof(xfs_alloc_rec_incore_t) = 8
+sizeof(xfs_alloc_rec_t) = 8
+sizeof(xfs_attr_leaf_entry_t) = 8
+sizeof(xfs_attr_leaf_hdr_t) = 32
+sizeof(xfs_attr_leaf_map_t) = 4
+sizeof(xfs_attr_leaf_name_local_t) = 4
+sizeof(xfs_attr_leaf_name_remote_t) = 12
+sizeof(xfs_attr_leafblock_t) = 40
+sizeof(xfs_attr_shortform_t) = 8
+sizeof(xfs_attrlist_cursor_t) = 16
+sizeof(xfs_bmbt_key_t) = 8
+sizeof(xfs_bmbt_rec_t) = 16
+sizeof(xfs_bmdr_block_t) = 4
+sizeof(xfs_bmdr_key_t) = 8
+sizeof(xfs_btnum_t) = 4
+sizeof(xfs_da_blkinfo_t) = 12
+sizeof(xfs_da_intnode_t) = 16
+sizeof(xfs_da_node_entry_t) = 8
+sizeof(xfs_da_node_hdr_t) = 16
+sizeof(xfs_dinode_fmt_t) = 4
+sizeof(xfs_dir2_data_free_t) = 4
+sizeof(xfs_dir2_data_hdr_t) = 16
+sizeof(xfs_dir2_data_unused_t) = 6
+sizeof(xfs_dir2_free_hdr_t) = 16
+sizeof(xfs_dir2_free_t) = 16
+sizeof(xfs_dir2_ino4_t) = 4
+sizeof(xfs_dir2_ino8_t) = 8
+sizeof(xfs_dir2_inou_t) = 8
+sizeof(xfs_dir2_leaf_entry_t) = 8
+sizeof(xfs_dir2_leaf_hdr_t) = 16
+sizeof(xfs_dir2_leaf_t) = 16
+sizeof(xfs_dir2_leaf_tail_t) = 4
+sizeof(xfs_dir2_sf_entry_t) = 3
+sizeof(xfs_dir2_sf_hdr_t) = 10
+sizeof(xfs_dir2_sf_off_t) = 2
+sizeof(xfs_disk_dquot_t) = 104
+sizeof(xfs_dq_logformat_t) = 24
+sizeof(xfs_dqblk_t) = 136
+sizeof(xfs_dsb_t) = 264
+sizeof(xfs_efd_log_format_32_t) = 28
+sizeof(xfs_efd_log_format_64_t) = 32
+sizeof(xfs_efi_log_format_32_t) = 28
+sizeof(xfs_efi_log_format_64_t) = 32
+sizeof(xfs_error_injection_t) = 8
+sizeof(xfs_exntfmt_t) = 4
+sizeof(xfs_exntst_t) = 4
+sizeof(xfs_extent_32_t) = 12
+sizeof(xfs_extent_64_t) = 16
+sizeof(xfs_fid_t) = 16
+sizeof(xfs_fsid_t) = 8
+sizeof(xfs_fsop_counts_t) = 32
+sizeof(xfs_fsop_geom_t) = 112
+sizeof(xfs_fsop_resblks_t) = 16
+sizeof(xfs_growfs_log_t) = 8
+sizeof(xfs_handle_t) = 24
+sizeof(xfs_inobt_key_t) = 4
+sizeof(xfs_inobt_rec_t) = 16
+sizeof(xfs_inode_log_format_32_t) = 52
+sizeof(xfs_inode_log_format_64_t) = 56
+sizeof(xfs_lookup_t) = 4
+sizeof(xfs_qoff_logformat_t) = 20
+sizeof(xfs_timestamp_t) = 8
+sizeof(xfs_trans_header_t) = 16

From darrick.wong@oracle.com  Fri Jan 22 18:37:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id F28217F5E
	for ; Fri, 22 Jan 2016 18:37:12 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 814EBAC001
	for ; Fri, 22 Jan 2016 16:37:12 -0800 (PST)
X-ASG-Debug-ID: 1453509430-04cb6c0d5f7c190001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id 926NCSikCpzNObIe (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:37:10 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0b9d7029584
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:37:09 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0b9l2003392
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:37:09 GMT
Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0b8LR027893;
	Sat, 23 Jan 2016 00:37:08 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:37:08 -0800
Subject: [PATCH 4/5] dedupe: record dedupe errors in golden output
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 4/5] dedupe: record dedupe errors in golden output
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: fstests@vger.kernel.org, xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:37:07 -0800
Message-ID: <20160123003706.2666.81569.stgit@birch.djwong.org>
In-Reply-To: <20160123003648.2666.37062.stgit@birch.djwong.org>
References: <20160123003648.2666.37062.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509430
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

The xfs_io dedupe command originally didn't print dedupe errors
to stderr like you'd expect.  Since that was fixed, the golden
output should be changed to reflect that.

Signed-off-by: Darrick J. Wong 
---
 tests/generic/122.out |    1 +
 tests/generic/136     |    2 ++
 tests/generic/136.out |    3 +++
 tests/generic/137     |    2 +-
 tests/generic/182.out |    2 ++
 5 files changed, 9 insertions(+), 1 deletion(-)


diff --git a/tests/generic/122.out b/tests/generic/122.out
index c7cfec2..7925e90 100644
--- a/tests/generic/122.out
+++ b/tests/generic/122.out
@@ -4,6 +4,7 @@ Create the original files
 5e3501f97fd2669babfcbd3e1972e833  TEST_DIR/test-122/file2
 Files 1-2 do not match (intentional)
 (Fail to) dedupe the middle blocks together
+dedupe: Extents did not match.
 Compare sections
 35ac8d7917305c385c30f3d82c30a8f6  TEST_DIR/test-122/file1
 5e3501f97fd2669babfcbd3e1972e833  TEST_DIR/test-122/file2
diff --git a/tests/generic/136 b/tests/generic/136
index 89ee751..5d021f8 100755
--- a/tests/generic/136
+++ b/tests/generic/136
@@ -82,7 +82,9 @@ test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
 test "${C3}" = "${C4}" || echo "file3 and file4 should match"
 
 echo "Dedupe the last blocks together"
+echo "1->2"
 _dedupe_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $BLKSZ 37 >> "$seqres.full"
+echo "1->3"
 _dedupe_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file3" $BLKSZ 37 >> "$seqres.full"
 _test_remount
 
diff --git a/tests/generic/136.out b/tests/generic/136.out
index 99a5465..f76f40a 100644
--- a/tests/generic/136.out
+++ b/tests/generic/136.out
@@ -5,6 +5,9 @@ c4fd505be25a0c91bcca9f502b9a8156  TEST_DIR/test-136/file2
 07ac67bf7f271195442509e79cde4cee  TEST_DIR/test-136/file3
 07ac67bf7f271195442509e79cde4cee  TEST_DIR/test-136/file4
 Dedupe the last blocks together
+1->2
+1->3
+dedupe: Extents did not match.
 c4fd505be25a0c91bcca9f502b9a8156  TEST_DIR/test-136/file1
 c4fd505be25a0c91bcca9f502b9a8156  TEST_DIR/test-136/file2
 07ac67bf7f271195442509e79cde4cee  TEST_DIR/test-136/file3
diff --git a/tests/generic/137 b/tests/generic/137
index a640f4f..9cc7948 100755
--- a/tests/generic/137
+++ b/tests/generic/137
@@ -83,7 +83,7 @@ done
 echo "Dedupe block two to the sevens"
 seq 1 $((NR_BLKS / 7)) | while read nr; do
 	_dedupe_range "$TESTDIR/file1" $((BLKSZ * 2)) "$TESTDIR/file1" \
-			$((nr * 7 * BLKSZ)) $BLKSZ >> "$seqres.full"
+			$((nr * 7 * BLKSZ)) $BLKSZ >> "$seqres.full" 2>&1
 done
 
 _test_remount
diff --git a/tests/generic/182.out b/tests/generic/182.out
index 9841b76..049a3ee 100644
--- a/tests/generic/182.out
+++ b/tests/generic/182.out
@@ -1,10 +1,12 @@
 QA output created by 182
 Create the original files
+dedupe: Extents did not match.
 f4820540fc0ac02750739896fe028d56  TEST_DIR/test-182/file1
 69ad53078a16243d98e21d9f8704a071  TEST_DIR/test-182/file2
 69ad53078a16243d98e21d9f8704a071  TEST_DIR/test-182/file2.chk
 Compare against check files
 Make the original file almost dedup-able
+dedupe: Extents did not match.
 f4820540fc0ac02750739896fe028d56  TEST_DIR/test-182/file1
 158d4e3578b94b89cbb44493a2110fb9  TEST_DIR/test-182/file2
 158d4e3578b94b89cbb44493a2110fb9  TEST_DIR/test-182/file2.chk

From darrick.wong@oracle.com  Fri Jan 22 18:37:50 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 374DE7F58
	for ; Fri, 22 Jan 2016 18:37:50 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 5BCBFAC002
	for ; Fri, 22 Jan 2016 16:37:49 -0800 (PST)
X-ASG-Debug-ID: 1453509438-04cbb026fe71cf0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id FQVAyRIBgfSFRteE (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:37:18 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0bGBU029676
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:37:17 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0N0bGxS010642
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:37:16 GMT
Received: from abhmp0018.oracle.com (abhmp0018.oracle.com [141.146.116.24])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0bFf2027935;
	Sat, 23 Jan 2016 00:37:15 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:37:14 -0800
Subject: [PATCH 5/5] tests: use lowercase variables for all the new reflink
 tests
From: "Darrick J. Wong" 
X-ASG-Orig-Subj: [PATCH 5/5] tests: use lowercase variables for all the new reflink
 tests
To: david@fromorbit.com, darrick.wong@oracle.com
Cc: fstests@vger.kernel.org, xfs@oss.sgi.com
Date: Fri, 22 Jan 2016 16:37:13 -0800
Message-ID: <20160123003713.2666.80882.stgit@birch.djwong.org>
In-Reply-To: <20160123003648.2666.37062.stgit@birch.djwong.org>
References: <20160123003648.2666.37062.stgit@birch.djwong.org>
User-Agent: StGit/0.17.1-dirty
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453509438
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Barracuda-BRTS-Status: 1
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26368
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Since $TESTDIR is a local variable, make it lowercase to avoid
confusion with $TEST_DIR.  While we're at it, make all the local
variables lowercase.

Signed-off-by: Darrick J. Wong 
---
 tests/generic/110 |   24 ++++-----
 tests/generic/111 |   42 ++++++++-------
 tests/generic/115 |   34 ++++++------
 tests/generic/116 |   42 ++++++++-------
 tests/generic/118 |   42 ++++++++-------
 tests/generic/119 |  112 ++++++++++++++++++++---------------------
 tests/generic/121 |   42 ++++++++-------
 tests/generic/122 |   42 ++++++++-------
 tests/generic/134 |   90 ++++++++++++++++-----------------
 tests/generic/136 |   90 ++++++++++++++++-----------------
 tests/generic/137 |   54 ++++++++++----------
 tests/generic/138 |  100 ++++++++++++++++++------------------
 tests/generic/139 |  100 ++++++++++++++++++------------------
 tests/generic/140 |  100 ++++++++++++++++++------------------
 tests/generic/142 |   34 ++++++------
 tests/generic/143 |   34 ++++++------
 tests/generic/144 |  124 +++++++++++++++++++++++----------------------
 tests/generic/145 |  146 +++++++++++++++++++++++++++--------------------------
 tests/generic/146 |  136 +++++++++++++++++++++++++------------------------
 tests/generic/147 |  142 ++++++++++++++++++++++++++--------------------------
 tests/generic/148 |   80 +++++++++++++++--------------
 tests/generic/149 |  136 +++++++++++++++++++++++++------------------------
 tests/generic/150 |   30 +++++------
 tests/generic/151 |   46 ++++++++---------
 tests/generic/152 |   52 +++++++++----------
 tests/generic/153 |   52 +++++++++----------
 tests/generic/154 |   60 +++++++++++-----------
 tests/generic/155 |   60 +++++++++++-----------
 tests/generic/156 |   58 +++++++++++----------
 tests/generic/157 |   68 ++++++++++++-------------
 tests/generic/158 |   70 +++++++++++++------------
 tests/generic/159 |   30 +++++------
 tests/generic/160 |   30 +++++------
 tests/generic/161 |   16 +++---
 tests/generic/162 |   22 ++++----
 tests/generic/163 |   22 ++++----
 tests/generic/164 |   28 +++++-----
 tests/generic/165 |   28 +++++-----
 tests/generic/166 |   18 +++----
 tests/generic/167 |   18 +++----
 tests/generic/168 |   22 ++++----
 tests/generic/170 |   22 ++++----
 tests/generic/171 |   42 ++++++++-------
 tests/generic/172 |   42 ++++++++-------
 tests/generic/173 |   42 ++++++++-------
 tests/generic/174 |   42 ++++++++-------
 tests/generic/175 |   24 ++++-----
 tests/generic/176 |   24 ++++-----
 tests/generic/178 |   28 +++++-----
 tests/generic/179 |   46 ++++++++---------
 tests/generic/180 |   46 ++++++++---------
 tests/generic/181 |   34 ++++++------
 tests/generic/182 |   76 ++++++++++++++--------------
 tests/generic/183 |   48 +++++++++--------
 tests/generic/185 |   48 +++++++++--------
 tests/generic/186 |   54 ++++++++++----------
 tests/generic/187 |   54 ++++++++++----------
 tests/generic/188 |   40 +++++++--------
 tests/generic/189 |   40 +++++++--------
 tests/generic/190 |   40 +++++++--------
 tests/generic/191 |   40 +++++++--------
 tests/generic/194 |   46 ++++++++---------
 tests/generic/195 |   46 ++++++++---------
 tests/generic/196 |   40 +++++++--------
 tests/generic/197 |   40 +++++++--------
 tests/generic/199 |   60 +++++++++++-----------
 tests/generic/200 |   60 +++++++++++-----------
 tests/generic/201 |   34 ++++++------
 tests/generic/202 |   32 ++++++------
 tests/generic/203 |   32 ++++++------
 tests/generic/205 |   48 +++++++++--------
 tests/generic/206 |   48 +++++++++--------
 tests/generic/216 |   48 +++++++++--------
 tests/generic/217 |   48 +++++++++--------
 tests/generic/218 |   40 +++++++--------
 tests/generic/220 |   40 +++++++--------
 tests/generic/222 |   48 +++++++++--------
 tests/generic/227 |   48 +++++++++--------
 tests/generic/229 |   94 +++++++++++++++++-----------------
 tests/generic/238 |   94 +++++++++++++++++-----------------
 tests/generic/242 |   46 ++++++++---------
 tests/generic/243 |   46 ++++++++---------
 tests/xfs/127     |   16 +++---
 tests/xfs/128     |  140 +++++++++++++++++++++++++--------------------------
 tests/xfs/129     |   28 +++++-----
 tests/xfs/131     |   16 +++---
 tests/xfs/132     |   68 ++++++++++++-------------
 tests/xfs/139     |   26 +++++----
 tests/xfs/140     |   34 ++++++------
 89 files changed, 2337 insertions(+), 2337 deletions(-)


diff --git a/tests/generic/110 b/tests/generic/110
index fffadbc..a43f02d 100755
--- a/tests/generic/110
+++ b/tests/generic/110
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -f $tmp.* "$TESTDIR"
+    rm -f $tmp.* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -53,37 +53,37 @@ _require_xfs_io_command "fiemap"
 _require_cp_reflink
 _require_test
 
-TESTDIR1=$TEST_DIR/test-$seq
-rm -rf $TESTDIR1
-mkdir $TESTDIR1
+testdir1=$TEST_DIR/test-$seq
+rm -rf $testdir1
+mkdir $testdir1
 
 _checksum_files() {
     for F in original copy1 copy2
     do
-        md5sum $TESTDIR1/$F | _filter_test_dir
+        md5sum $testdir1/$F | _filter_test_dir
     done
 }
 
 rm -f $seqres.full
 
 echo "Create the original file and reflink to copy1, copy2"
-$XFS_IO_PROG -f -c 'pwrite -S 0x61 0 9000' $TESTDIR1/original \
+$XFS_IO_PROG -f -c 'pwrite -S 0x61 0 9000' $testdir1/original \
     >> $seqres.full 2>&1
-cp --reflink $TESTDIR1/original $TESTDIR1/copy1
-cp --reflink $TESTDIR1/copy1 $TESTDIR1/copy2
-_verify_reflink $TESTDIR1/original $TESTDIR1/copy1
-_verify_reflink $TESTDIR1/original $TESTDIR1/copy2
+cp --reflink $testdir1/original $testdir1/copy1
+cp --reflink $testdir1/copy1 $testdir1/copy2
+_verify_reflink $testdir1/original $testdir1/copy1
+_verify_reflink $testdir1/original $testdir1/copy2
 echo "Original md5sums:"
 _checksum_files
 
 echo "Overwrite original file with new data"
-$XFS_IO_PROG -c 'pwrite -S 0x62 0 9000' $TESTDIR1/original \
+$XFS_IO_PROG -c 'pwrite -S 0x62 0 9000' $testdir1/original \
     >> $seqres.full 2>&1
 echo "md5sums after overwriting original:"
 _checksum_files
 
 echo "Overwrite copy1 with different new data"
-$XFS_IO_PROG -c 'pwrite -S 0x63 0 9000' $TESTDIR1/copy1 \
+$XFS_IO_PROG -c 'pwrite -S 0x63 0 9000' $testdir1/copy1 \
     >> $seqres.full 2>&1
 echo "md5sums after overwriting copy1:"
 _checksum_files
diff --git a/tests/generic/111 b/tests/generic/111
index 98077d8..ef9c424 100755
--- a/tests/generic/111
+++ b/tests/generic/111
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -f $tmp.* "$TESTDIR"
+    rm -f $tmp.* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -53,55 +53,55 @@ _require_xfs_io_command "fiemap"
 _require_cp_reflink
 _require_test
 
-TESTDIR1=$TEST_DIR/test-$seq
-rm -rf $TESTDIR1
-mkdir $TESTDIR1
+testdir1=$TEST_DIR/test-$seq
+rm -rf $testdir1
+mkdir $testdir1
 
 _checksum_files() {
     for F in original/file1 original/subdir/file2 \
                  copy1/file1 copy1/subdir/file2 \
                  copy2/file1 copy2/subdir/file2
         do
-            md5sum $TESTDIR1/$F | _filter_test_dir
+            md5sum $testdir1/$F | _filter_test_dir
         done
 }
 
 rm -f $seqres.full
 
-mkdir $TESTDIR1/original
-mkdir $TESTDIR1/original/subdir
+mkdir $testdir1/original
+mkdir $testdir1/original/subdir
 
 echo "Create the original files and reflink dirs"
-$XFS_IO_PROG -f -c 'pwrite -S 0x61 0 9000' $TESTDIR1/original/file1 \
+$XFS_IO_PROG -f -c 'pwrite -S 0x61 0 9000' $testdir1/original/file1 \
     >> $seqres.full 2>&1
 $XFS_IO_PROG -f -c 'pwrite -S 0x62 0 11000' \
-    $TESTDIR1/original/subdir/file2 >> $seqres.full 2>&1
-cp --recursive --reflink $TESTDIR1/original $TESTDIR1/copy1
-cp --recursive --reflink $TESTDIR1/copy1 $TESTDIR1/copy2
+    $testdir1/original/subdir/file2 >> $seqres.full 2>&1
+cp --recursive --reflink $testdir1/original $testdir1/copy1
+cp --recursive --reflink $testdir1/copy1 $testdir1/copy2
 
-_verify_reflink $TESTDIR1/original/file1 $TESTDIR1/copy1/file1
-_verify_reflink $TESTDIR1/original/subdir/file2 \
-    $TESTDIR1/copy1/subdir/file2
-_verify_reflink $TESTDIR1/original/file1 $TESTDIR1/copy2/file1
-_verify_reflink $TESTDIR1/original/subdir/file2 \
-    $TESTDIR1/copy2/subdir/file2
+_verify_reflink $testdir1/original/file1 $testdir1/copy1/file1
+_verify_reflink $testdir1/original/subdir/file2 \
+    $testdir1/copy1/subdir/file2
+_verify_reflink $testdir1/original/file1 $testdir1/copy2/file1
+_verify_reflink $testdir1/original/subdir/file2 \
+    $testdir1/copy2/subdir/file2
 
 echo "Original md5sums:"
 _checksum_files
 
 echo "Overwrite original/file1 and original/subdir/file2 with new data"
-$XFS_IO_PROG -c 'pwrite -S 0x63 0 13000' $TESTDIR1/original/file1 \
+$XFS_IO_PROG -c 'pwrite -S 0x63 0 13000' $testdir1/original/file1 \
     >> $seqres.full 2>&1
 $XFS_IO_PROG -c 'pwrite -S 0x64 5000 1000' \
-    $TESTDIR1/original/subdir/file2 >> $seqres.full 2>&1
+    $testdir1/original/subdir/file2 >> $seqres.full 2>&1
 echo "md5sums now:"
 _checksum_files
 
 echo "Overwrite copy1/file1 and copy1/subdir/file2 with new data"
-$XFS_IO_PROG -c 'pwrite -S 0x65 0 9000' $TESTDIR1/copy1/file1 \
+$XFS_IO_PROG -c 'pwrite -S 0x65 0 9000' $testdir1/copy1/file1 \
     >> $seqres.full 2>&1
 $XFS_IO_PROG -c 'pwrite -S 0x66 5000 25000' \
-    $TESTDIR1/copy1/subdir/file2 >> $seqres.full 2>&1
+    $testdir1/copy1/subdir/file2 >> $seqres.full 2>&1
 echo "md5sums now:"
 _checksum_files
 
diff --git a/tests/generic/115 b/tests/generic/115
index 6c8d92d..578d5bd 100755
--- a/tests/generic/115
+++ b/tests/generic/115
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -f $tmp.* "$TESTDIR"
+    rm -f $tmp.* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -53,31 +53,31 @@ _require_test
 
 rm -f $seqres.full
 
-TESTDIR1=$TEST_DIR/test-$seq
-rm -rf $TESTDIR1
-mkdir $TESTDIR1
+testdir1=$TEST_DIR/test-$seq
+rm -rf $testdir1
+mkdir $testdir1
 
 echo "Create the original files and reflink dirs"
-$XFS_IO_PROG -f -c 'pwrite -S 0x61 0 9000' $TESTDIR1/original \
+$XFS_IO_PROG -f -c 'pwrite -S 0x61 0 9000' $testdir1/original \
     >> $seqres.full
-cp --reflink $TESTDIR1/original $TESTDIR1/copy
+cp --reflink $testdir1/original $testdir1/copy
 
-_verify_reflink $TESTDIR1/original $TESTDIR1/copy
+_verify_reflink $testdir1/original $testdir1/copy
 
 echo "Move orig & reflink copy to subdir and md5sum:"
-mkdir $TESTDIR1/subdir
-mv $TESTDIR1/original $TESTDIR1/subdir/original_moved
-mv $TESTDIR1/copy $TESTDIR1/subdir/copy_moved
-_verify_reflink $TESTDIR1/subdir/original_moved \
-    $TESTDIR1/subdir/copy_moved
+mkdir $testdir1/subdir
+mv $testdir1/original $testdir1/subdir/original_moved
+mv $testdir1/copy $testdir1/subdir/copy_moved
+_verify_reflink $testdir1/subdir/original_moved \
+    $testdir1/subdir/copy_moved
 
-md5sum $TESTDIR1/subdir/original_moved | _filter_test_dir
-md5sum $TESTDIR1/subdir/copy_moved | _filter_test_dir
+md5sum $testdir1/subdir/original_moved | _filter_test_dir
+md5sum $testdir1/subdir/copy_moved | _filter_test_dir
 
 echo "remove orig from subdir and md5sum reflink copy:"
-rm $TESTDIR1/subdir/original_moved
-md5sum $TESTDIR1/subdir/copy_moved | _filter_test_dir
-rm -rf $TESTDIR1/subdir
+rm $testdir1/subdir/original_moved
+md5sum $testdir1/subdir/copy_moved | _filter_test_dir
+rm -rf $testdir1/subdir
 
 # success, all done
 status=0
diff --git a/tests/generic/116 b/tests/generic/116
index 2b4b505..5c361e2 100755
--- a/tests/generic/116
+++ b/tests/generic/116
@@ -34,7 +34,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -48,43 +48,43 @@ _require_test_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 $((BLKSZ * 2)) $((BLKSZ * 6)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $((BLKSZ * 6)) "$TESTDIR/file2" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 $((blksz * 2)) $((blksz * 6)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $((blksz * 6)) "$testdir/file2" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 8)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 8)) \
        || echo "Files do not match"
 
 echo "Reflink the middle blocks together"
-free_before="$(stat -f -c '%a' "$TESTDIR")"
-_reflink_range "$TESTDIR/file1" $((BLKSZ * 4)) "$TESTDIR/file2" \
-		$((BLKSZ * 4)) $((BLKSZ * 2)) >> "$seqres.full"
+free_before="$(stat -f -c '%a' "$testdir")"
+_reflink_range "$testdir/file1" $((blksz * 4)) "$testdir/file2" \
+		$((blksz * 4)) $((blksz * 2)) >> "$seqres.full"
 _test_remount
-free_after="$(stat -f -c '%a' "$TESTDIR")"
+free_after="$(stat -f -c '%a' "$testdir")"
 echo "freesp changed by $free_before -> $free_after" >> "$seqres.full"
 
 echo "Compare sections"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 4)) \
        || echo "Start sections do not match"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 4)) "$TESTDIR/file2" \
-		$((BLKSZ * 4)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 4)) "$testdir/file2" \
+		$((blksz * 4)) $((blksz * 2)) \
        || echo "Middle sections do not match"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 6)) "$TESTDIR/file2" \
-		$((BLKSZ * 6)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 6)) "$testdir/file2" \
+		$((blksz * 6)) $((blksz * 2)) \
        || echo "End sections do not match"
 
 # success, all done
diff --git a/tests/generic/118 b/tests/generic/118
index 651b72c..41e525c 100755
--- a/tests/generic/118
+++ b/tests/generic/118
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -49,43 +49,43 @@ _require_test_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 $((BLKSZ * 2)) $((BLKSZ * 6)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 $((BLKSZ * 2)) $((BLKSZ * 6)) "$TESTDIR/file2" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 $((blksz * 2)) $((blksz * 6)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 2)) $((blksz * 6)) "$testdir/file2" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 8)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 8)) \
        || echo "Files do not match (intentional)"
 
 echo "Reflink the middle blocks together"
-free_before="$(stat -f -c '%a' "$TESTDIR")"
-_reflink_range "$TESTDIR/file1" $((BLKSZ * 4)) "$TESTDIR/file2" \
-		$((BLKSZ * 4)) $((BLKSZ * 2)) >> "$seqres.full"
+free_before="$(stat -f -c '%a' "$testdir")"
+_reflink_range "$testdir/file1" $((blksz * 4)) "$testdir/file2" \
+		$((blksz * 4)) $((blksz * 2)) >> "$seqres.full"
 _test_remount
-free_after="$(stat -f -c '%a' "$TESTDIR")"
+free_after="$(stat -f -c '%a' "$testdir")"
 echo "freesp changed by $free_before -> $free_after" >> "$seqres.full"
 
 echo "Compare sections"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 4)) \
        || echo "Start sections do not match (intentional)"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 4)) "$TESTDIR/file2" \
-		$((BLKSZ * 4)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 4)) "$testdir/file2" \
+		$((blksz * 4)) $((blksz * 2)) \
        || echo "Middle sections do not match"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 6)) "$TESTDIR/file2" \
-		$((BLKSZ * 6)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 6)) "$testdir/file2" \
+		$((blksz * 6)) $((blksz * 2)) \
        || echo "End sections do not match (intentional)"
 
 # success, all done
diff --git a/tests/generic/119 b/tests/generic/119
index 9d57379..27e7604 100755
--- a/tests/generic/119
+++ b/tests/generic/119
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,119 +50,119 @@ _require_test_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR=$TEST_DIR/test-$seq
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir=$TEST_DIR/test-$seq
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 8)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * 8)) "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x63 0 $((BLKSZ * 8)) "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x64 0 $((BLKSZ * 8)) "$TESTDIR/file4" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 8)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * 8)) "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x63 0 $((blksz * 8)) "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x64 0 $((blksz * 8)) "$testdir/file4" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 8)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 8)) \
        || echo "Files 1-2 do not match (intentional)"
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file3" 0 $((BLKSZ * 8)) \
+_compare_range "$testdir/file1" 0 "$testdir/file3" 0 $((blksz * 8)) \
        || echo "Files 1-3 do not match (intentional)"
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file4" 0 $((BLKSZ * 8)) \
+_compare_range "$testdir/file1" 0 "$testdir/file4" 0 $((blksz * 8)) \
        || echo "Files 1-4 do not match (intentional)"
 
 echo "Reflink the first four blocks together, 1-2 3-4"
-free_before="$(stat -f -c '%a' "$TESTDIR")"
-_reflink_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 4)) >> "$seqres.full"
-_reflink_range "$TESTDIR/file3" 0 "$TESTDIR/file4" 0 $((BLKSZ * 4)) >> "$seqres.full"
+free_before="$(stat -f -c '%a' "$testdir")"
+_reflink_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 4)) >> "$seqres.full"
+_reflink_range "$testdir/file3" 0 "$testdir/file4" 0 $((blksz * 4)) >> "$seqres.full"
 _test_remount
-free_after="$(stat -f -c '%a' "$TESTDIR")"
+free_after="$(stat -f -c '%a' "$testdir")"
 echo "freesp changed by $free_before -> $free_after" >> "$seqres.full"
 
 echo "Compare sections"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 4)) \
        || echo "Sections of file 1-2 do not match"
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file3" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file1" 0 "$testdir/file3" 0 $((blksz * 4)) \
        || echo "Sections of file 1-3 do not match (intentional)"
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file4" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file1" 0 "$testdir/file4" 0 $((blksz * 4)) \
        || echo "Sections of file 1-4 do not match (intentional)"
 
-_compare_range "$TESTDIR/file2" 0 "$TESTDIR/file3" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file2" 0 "$testdir/file3" 0 $((blksz * 4)) \
        || echo "Sections of file 2-3 do not match (intentional)"
 
-_compare_range "$TESTDIR/file2" 0 "$TESTDIR/file4" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file2" 0 "$testdir/file4" 0 $((blksz * 4)) \
        || echo "Sections of file 2-4 do not match (intentional)"
 
-_compare_range "$TESTDIR/file3" 0 "$TESTDIR/file4" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file3" 0 "$testdir/file4" 0 $((blksz * 4)) \
        || echo "Sections of file 3-4 do not match"
 
 echo "Reflink the first two blocks together, 1-3 1-4"
-free_before="$(stat -f -c '%a' $TESTDIR)"
-_reflink_range "$TESTDIR/file1" 0 "$TESTDIR/file3" 0 $((BLKSZ * 2)) >> "$seqres.full"
-_reflink_range "$TESTDIR/file1" 0 "$TESTDIR/file4" 0 $((BLKSZ * 2)) >> "$seqres.full"
+free_before="$(stat -f -c '%a' $testdir)"
+_reflink_range "$testdir/file1" 0 "$testdir/file3" 0 $((blksz * 2)) >> "$seqres.full"
+_reflink_range "$testdir/file1" 0 "$testdir/file4" 0 $((blksz * 2)) >> "$seqres.full"
 _test_remount
-free_after="$(stat -f -c '%a' $TESTDIR)"
+free_after="$(stat -f -c '%a' $testdir)"
 echo "freesp changed by $free_before -> $free_after" >> "$seqres.full"
 
 echo "Compare sections"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 2)) \
        || echo "Sections of files 1-2 do not match"
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file3" 0 $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" 0 "$testdir/file3" 0 $((blksz * 2)) \
        || echo "Sections of files 1-3 do not match"
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file4" 0 $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" 0 "$testdir/file4" 0 $((blksz * 2)) \
        || echo "Sections of files 1-4 do not match"
 
-_compare_range "$TESTDIR/file2" 0 "$TESTDIR/file3" 0 $((BLKSZ * 2)) \
+_compare_range "$testdir/file2" 0 "$testdir/file3" 0 $((blksz * 2)) \
        || echo "Sections of files 2-3 do not match"
 
-_compare_range "$TESTDIR/file2" 0 "$TESTDIR/file4" 0 $((BLKSZ * 2)) \
+_compare_range "$testdir/file2" 0 "$testdir/file4" 0 $((blksz * 2)) \
        || echo "Sections of files 2-4 do not match"
 
-_compare_range "$TESTDIR/file3" 0 "$TESTDIR/file4" 0 $((BLKSZ * 2)) \
+_compare_range "$testdir/file3" 0 "$testdir/file4" 0 $((blksz * 2)) \
        || echo "Sections of files 3-4 do not match"
 
 echo "Compare previously reflinked sections"
-_compare_range "$TESTDIR/file1" $((BLKSZ * 2)) "$TESTDIR/file2" \
-		$((BLKSZ * 2)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 2)) "$testdir/file2" \
+		$((blksz * 2)) $((blksz * 2)) \
        || echo "Sections of file 1-2 do not match"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 2)) "$TESTDIR/file3" \
-		$((BLKSZ * 2)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 2)) "$testdir/file3" \
+		$((blksz * 2)) $((blksz * 2)) \
        || echo "Sections of file 1-3 do not match (intentional)"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 2)) "$TESTDIR/file4" \
-		$((BLKSZ * 2)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 2)) "$testdir/file4" \
+		$((blksz * 2)) $((blksz * 2)) \
        || echo "Sections of file 1-4 do not match (intentional)"
 
-_compare_range "$TESTDIR/file2" $((BLKSZ * 2)) "$TESTDIR/file3" \
-		$((BLKSZ * 2)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file2" $((blksz * 2)) "$testdir/file3" \
+		$((blksz * 2)) $((blksz * 2)) \
        || echo "Sections of file 2-3 do not match (intentional)"
 
-_compare_range "$TESTDIR/file2" $((BLKSZ * 2)) "$TESTDIR/file4" \
-		$((BLKSZ * 2)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file2" $((blksz * 2)) "$testdir/file4" \
+		$((blksz * 2)) $((blksz * 2)) \
        || echo "Sections of file 2-4 do not match (intentional)"
 
-_compare_range "$TESTDIR/file3" $((BLKSZ * 2)) "$TESTDIR/file4" \
-		$((BLKSZ * 2)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file3" $((blksz * 2)) "$testdir/file4" \
+		$((blksz * 2)) $((blksz * 2)) \
        || echo "Sections of file 3-4 do not match"
 
 # success, all done
diff --git a/tests/generic/121 b/tests/generic/121
index 9a5b383..0e15e8d 100755
--- a/tests/generic/121
+++ b/tests/generic/121
@@ -34,7 +34,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -48,43 +48,43 @@ _require_test_dedupe
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 $((BLKSZ * 2)) $((BLKSZ * 6)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $((BLKSZ * 6)) "$TESTDIR/file2" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 $((blksz * 2)) $((blksz * 6)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $((blksz * 6)) "$testdir/file2" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 8)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 8)) \
        || echo "Files 1-2 do not match (intentional)"
 
 echo "Dedupe the middle blocks together"
-free_before="$(stat -f -c '%a' "$TESTDIR")"
-_dedupe_range "$TESTDIR/file1" $((BLKSZ * 4)) "$TESTDIR/file2" \
-		$((BLKSZ * 4)) $((BLKSZ * 2)) >> "$seqres.full"
+free_before="$(stat -f -c '%a' "$testdir")"
+_dedupe_range "$testdir/file1" $((blksz * 4)) "$testdir/file2" \
+		$((blksz * 4)) $((blksz * 2)) >> "$seqres.full"
 _test_remount
-free_after="$(stat -f -c '%a' "$TESTDIR")"
+free_after="$(stat -f -c '%a' "$testdir")"
 echo "freesp changed by $free_before -> $free_after" >> "$seqres.full"
 
 echo "Compare sections"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 4)) \
        || echo "Start sections do not match"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 4)) "$TESTDIR/file2" \
-		$((BLKSZ * 4)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 4)) "$testdir/file2" \
+		$((blksz * 4)) $((blksz * 2)) \
        || echo "Middle sections do not match"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 6)) "$TESTDIR/file2" \
-		$((BLKSZ * 6)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 6)) "$testdir/file2" \
+		$((blksz * 6)) $((blksz * 2)) \
        || echo "End sections do not match"
 
 # success, all done
diff --git a/tests/generic/122 b/tests/generic/122
index 0edb48d..e7585f7 100755
--- a/tests/generic/122
+++ b/tests/generic/122
@@ -34,7 +34,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -48,43 +48,43 @@ _require_test_dedupe
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 $((BLKSZ * 2)) $((BLKSZ * 6)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 $((BLKSZ * 2)) $((BLKSZ * 6)) "$TESTDIR/file2" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 $((blksz * 2)) $((blksz * 6)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 2)) $((blksz * 6)) "$testdir/file2" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 "$((BLKSZ * 8))" \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 "$((blksz * 8))" \
        || echo "Files 1-2 do not match (intentional)"
 
 echo "(Fail to) dedupe the middle blocks together"
-free_before="$(stat -f -c '%a' "$TESTDIR")"
-_dedupe_range "$TESTDIR/file1" $((BLKSZ * 4)) "$TESTDIR/file2" \
-		$((BLKSZ * 4)) $((BLKSZ * 2)) >> "$seqres.full"
+free_before="$(stat -f -c '%a' "$testdir")"
+_dedupe_range "$testdir/file1" $((blksz * 4)) "$testdir/file2" \
+		$((blksz * 4)) $((blksz * 2)) >> "$seqres.full"
 _test_remount
-free_after="$(stat -f -c '%a' "$TESTDIR")"
+free_after="$(stat -f -c '%a' "$testdir")"
 echo "freesp changed by $free_before -> $free_after" >> "$seqres.full"
 
 echo "Compare sections"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $((BLKSZ * 4)) \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $((blksz * 4)) \
        || echo "Start sections do not match (intentional)"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 4)) "$TESTDIR/file2" \
-		$((BLKSZ * 4)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 4)) "$testdir/file2" \
+		$((blksz * 4)) $((blksz * 2)) \
        || echo "Middle sections do not match (intentional)"
 
-_compare_range "$TESTDIR/file1" $((BLKSZ * 6)) "$TESTDIR/file2" \
-		$((BLKSZ * 6)) $((BLKSZ * 2)) \
+_compare_range "$testdir/file1" $((blksz * 6)) "$testdir/file2" \
+		$((blksz * 6)) $((blksz * 2)) \
        || echo "End sections do not match (intentional)"
 
 # success, all done
diff --git a/tests/generic/134 b/tests/generic/134
index 4dc710c..1dda839 100755
--- a/tests/generic/134
+++ b/tests/generic/134
@@ -38,7 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,75 +52,75 @@ _require_test_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ + 37)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ + 37)) "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ + 37)) "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ + 37)) "$TESTDIR/file4" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz + 37)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz + 37)) "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz + 37)) "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz + 37)) "$testdir/file4" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
 
-C1="$(_md5_checksum $TESTDIR/file1)"
-C2="$(_md5_checksum $TESTDIR/file2)"
-C3="$(_md5_checksum $TESTDIR/file3)"
-C4="$(_md5_checksum $TESTDIR/file4)"
+c1="$(_md5_checksum $testdir/file1)"
+c2="$(_md5_checksum $testdir/file2)"
+c3="$(_md5_checksum $testdir/file3)"
+c4="$(_md5_checksum $testdir/file4)"
 
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C3}" = "${C4}" || echo "file3 and file4 should match"
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c3}" = "${c4}" || echo "file3 and file4 should match"
 
 echo "Reflink the last blocks together, 1-2 1-3"
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $BLKSZ 37 >> "$seqres.full"
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file3" $BLKSZ 37 >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $blksz 37 >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file3" $blksz 37 >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
 
-C1="$(_md5_checksum $TESTDIR/file1)"
-C2="$(_md5_checksum $TESTDIR/file2)"
-C3="$(_md5_checksum $TESTDIR/file3)"
-C4="$(_md5_checksum $TESTDIR/file4)"
+c1="$(_md5_checksum $testdir/file1)"
+c2="$(_md5_checksum $testdir/file2)"
+c3="$(_md5_checksum $testdir/file3)"
+c4="$(_md5_checksum $testdir/file4)"
 
 echo "Compare files"
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C3}" != "${C4}" || echo "file3 and file4 should match"
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c3}" != "${c4}" || echo "file3 and file4 should match"
 
 echo "Compare sections"
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $BLKSZ 37 \
+_compare_range "$testdir/file1" $blksz "$testdir/file2" $blksz 37 \
        || echo "End sections of files 1-2 do not match"
 
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file3" $BLKSZ 37 \
+_compare_range "$testdir/file1" $blksz "$testdir/file3" $blksz 37 \
        || echo "End sections of files 1-3 do not match"
 
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file4" $BLKSZ 37 \
+_compare_range "$testdir/file1" $blksz "$testdir/file4" $blksz 37 \
        || echo "End sections of files 1-4 do not match (intentional)"
 
-_compare_range "$TESTDIR/file2" $BLKSZ "$TESTDIR/file3" $BLKSZ 37 \
+_compare_range "$testdir/file2" $blksz "$testdir/file3" $blksz 37 \
        || echo "End sections of files 2-3 do not match"
 
-_compare_range "$TESTDIR/file2" $BLKSZ "$TESTDIR/file4" $BLKSZ 37 \
+_compare_range "$testdir/file2" $blksz "$testdir/file4" $blksz 37 \
        || echo "End sections of files 2-4 do not match (intentional)"
 
-_compare_range "$TESTDIR/file3" $BLKSZ "$TESTDIR/file4" $BLKSZ 37 \
+_compare_range "$testdir/file3" $blksz "$testdir/file4" $blksz 37 \
        || echo "End sections of files 3-4 do not match (intentional)"
 
 # success, all done
diff --git a/tests/generic/136 b/tests/generic/136
index 5d021f8..6bd6b14 100755
--- a/tests/generic/136
+++ b/tests/generic/136
@@ -38,7 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,77 +52,77 @@ _require_test_dedupe
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ + 37)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ + 37)) "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ + 37)) "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ + 37)) "$TESTDIR/file4" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz + 37)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz + 37)) "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz + 37)) "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz + 37)) "$testdir/file4" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
 
-C1="$(_md5_checksum $TESTDIR/file1)"
-C2="$(_md5_checksum $TESTDIR/file2)"
-C3="$(_md5_checksum $TESTDIR/file3)"
-C4="$(_md5_checksum $TESTDIR/file4)"
+c1="$(_md5_checksum $testdir/file1)"
+c2="$(_md5_checksum $testdir/file2)"
+c3="$(_md5_checksum $testdir/file3)"
+c4="$(_md5_checksum $testdir/file4)"
 
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C3}" = "${C4}" || echo "file3 and file4 should match"
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c3}" = "${c4}" || echo "file3 and file4 should match"
 
 echo "Dedupe the last blocks together"
 echo "1->2"
-_dedupe_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $BLKSZ 37 >> "$seqres.full"
+_dedupe_range "$testdir/file1" $blksz "$testdir/file2" $blksz 37 >> "$seqres.full"
 echo "1->3"
-_dedupe_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file3" $BLKSZ 37 >> "$seqres.full"
+_dedupe_range "$testdir/file1" $blksz "$testdir/file3" $blksz 37 >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
 
-C1="$(_md5_checksum $TESTDIR/file1)"
-C2="$(_md5_checksum $TESTDIR/file2)"
-C3="$(_md5_checksum $TESTDIR/file3)"
-C4="$(_md5_checksum $TESTDIR/file4)"
+c1="$(_md5_checksum $testdir/file1)"
+c2="$(_md5_checksum $testdir/file2)"
+c3="$(_md5_checksum $testdir/file3)"
+c4="$(_md5_checksum $testdir/file4)"
 
 echo "Compare files"
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C3}" = "${C4}" || echo "file3 and file4 should match"
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c3}" = "${c4}" || echo "file3 and file4 should match"
 
 echo "Compare sections"
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $BLKSZ 37 \
+_compare_range "$testdir/file1" $blksz "$testdir/file2" $blksz 37 \
        || echo "End sections of files 1-2 do not match"
 
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file3" $BLKSZ 37 \
+_compare_range "$testdir/file1" $blksz "$testdir/file3" $blksz 37 \
        || echo "End sections of files 1-3 do not match (intentional)"
 
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file4" $BLKSZ 37 \
+_compare_range "$testdir/file1" $blksz "$testdir/file4" $blksz 37 \
        || echo "End sections of files 1-4 do not match (intentional)"
 
-_compare_range "$TESTDIR/file2" $BLKSZ "$TESTDIR/file3" $BLKSZ 37 \
+_compare_range "$testdir/file2" $blksz "$testdir/file3" $blksz 37 \
        || echo "End sections of files 2-3 do not match (intentional)"
 
-_compare_range "$TESTDIR/file2" $BLKSZ "$TESTDIR/file4" $BLKSZ 37 \
+_compare_range "$testdir/file2" $blksz "$testdir/file4" $blksz 37 \
        || echo "End sections of files 2-4 do not match (intentional)"
 
-_compare_range "$TESTDIR/file3" $BLKSZ "$TESTDIR/file4" $BLKSZ 37 \
+_compare_range "$testdir/file3" $blksz "$testdir/file4" $blksz 37 \
        || echo "End sections of files 3-4 do not match"
 
 # success, all done
diff --git a/tests/generic/137 b/tests/generic/137
index 9cc7948..dcfd90d 100755
--- a/tests/generic/137
+++ b/tests/generic/137
@@ -38,7 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -54,47 +54,47 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $((BLKSZ * 2)) "$TESTDIR/file1" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $((blksz * 2)) "$testdir/file1" >> "$seqres.full"
 
-NR_BLKS=1024
+nr_blks=1024
 
 echo "fallocate half the file"
-"$XFS_IO_PROG" -f -c "falloc $((NR_BLKS * BLKSZ / 2)) $((NR_BLKS * BLKSZ / 2))" "$TESTDIR/file1" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "falloc $((nr_blks * blksz / 2)) $((nr_blks * blksz / 2))" "$testdir/file1" >> "$seqres.full"
 
 echo "Reflink block zero to the threes"
-seq 1 $((NR_BLKS / 3)) | while read nr; do
-	_reflink_range "$TESTDIR/file1" 0 "$TESTDIR/file1" $((nr * 3 * BLKSZ)) \
-			$BLKSZ >> "$seqres.full"
+seq 1 $((nr_blks / 3)) | while read nr; do
+	_reflink_range "$testdir/file1" 0 "$testdir/file1" $((nr * 3 * blksz)) \
+			$blksz >> "$seqres.full"
 done
 
 echo "Reflink block one to the fives"
-seq 1 $((NR_BLKS / 5)) | while read nr; do
-	_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file1" \
-			$((nr * 5 * BLKSZ)) $BLKSZ >> "$seqres.full"
+seq 1 $((nr_blks / 5)) | while read nr; do
+	_reflink_range "$testdir/file1" $blksz "$testdir/file1" \
+			$((nr * 5 * blksz)) $blksz >> "$seqres.full"
 done
 
 echo "Dedupe block two to the sevens"
-seq 1 $((NR_BLKS / 7)) | while read nr; do
-	_dedupe_range "$TESTDIR/file1" $((BLKSZ * 2)) "$TESTDIR/file1" \
-			$((nr * 7 * BLKSZ)) $BLKSZ >> "$seqres.full" 2>&1
+seq 1 $((nr_blks / 7)) | while read nr; do
+	_dedupe_range "$testdir/file1" $((blksz * 2)) "$testdir/file1" \
+			$((nr * 7 * blksz)) $blksz >> "$seqres.full" 2>&1
 done
 
 _test_remount
 
 echo "Check block mappings"
-md5sum "$TESTDIR/file1" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
 
-crcZ=$(_md5_range_checksum /dev/zero 0 $BLKSZ)
-crc0=$(_md5_range_checksum "$TESTDIR/file1" 0 $BLKSZ)
-crc1=$(_md5_range_checksum "$TESTDIR/file1" $BLKSZ $BLKSZ)
-crc2=$(_md5_range_checksum "$TESTDIR/file1" $((BLKSZ * 2)) $BLKSZ)
+crcZ=$(_md5_range_checksum /dev/zero 0 $blksz)
+crc0=$(_md5_range_checksum "$testdir/file1" 0 $blksz)
+crc1=$(_md5_range_checksum "$testdir/file1" $blksz $blksz)
+crc2=$(_md5_range_checksum "$testdir/file1" $((blksz * 2)) $blksz)
 
 check_block() {
 	lblk="$1"
@@ -102,26 +102,26 @@ check_block() {
 	rem5=$((lblk % 5))
 	rem3=$((lblk % 3))
 
-	crc=$(_md5_range_checksum "$TESTDIR/file1" $((lblk * BLKSZ)) $BLKSZ)
+	crc=$(_md5_range_checksum "$testdir/file1" $((lblk * blksz)) $blksz)
 
 	if [ $rem7 -eq 0 ]; then
 		if [ $rem5 -eq 0 ]; then
 			test $crc2 = $crc || echo "lblk $lblk doesn't match block 2"
 		elif [ $rem3 -eq 0 ]; then
 			test $crc0 = $crc || echo "lblk $lblk doesn't match block 0"
-		elif [ $lblk -lt $((NR_BLKS / 2)) ]; then
+		elif [ $lblk -lt $((nr_blks / 2)) ]; then
 			test $crcZ = $crc || echo "lblk $lblk isn't zeroed"
 		fi
 	elif [ $rem5 -eq 0 ]; then
 		test $crc1 = $crc || echo "lblk $lblk doesn't match block 1"
 	elif [ $rem3 -eq 0 ]; then
 		test $crc0 = $crc || echo "lblk $lblk doesn't match block 0"
-	elif [ $lblk -lt $((NR_BLKS / 2)) ]; then
+	elif [ $lblk -lt $((nr_blks / 2)) ]; then
 		test $crcZ = $crc || echo "lblk $lblk isn't zeroed"
 	fi
 }
 
-seq 3 $((NR_BLKS - 1)) | while read lblk; do
+seq 3 $((nr_blks - 1)) | while read lblk; do
 	err="$(check_block $lblk)"
 	test -n "$err" && echo "$lblk: $err"
 done
diff --git a/tests/generic/138 b/tests/generic/138
index c084679..59d23e9 100755
--- a/tests/generic/138
+++ b/tests/generic/138
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,101 +50,101 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 48 - 3)) "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ * 48 - 3)) "$TESTDIR/file3" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 48 - 3)) "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 48 - 3)) "$testdir/file3" >> "$seqres.full"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
 
-cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "Files 1-2 do not match"
-cmp -s "$TESTDIR/file1" "$TESTDIR/file3" || echo "Files 1-3 do not match"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file3" || echo "Files 2-3 do not match"
+cmp -s "$testdir/file1" "$testdir/file2" || echo "Files 1-2 do not match"
+cmp -s "$testdir/file1" "$testdir/file3" || echo "Files 1-3 do not match"
+cmp -s "$testdir/file2" "$testdir/file3" || echo "Files 2-3 do not match"
 
 echo "pagecache CoW the second file"
-_pwrite_byte 0x62 0 17 "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x62 0 17 "$TESTDIR/file3" >> "$seqres.full"
+_pwrite_byte 0x62 0 17 "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x62 0 17 "$testdir/file3" >> "$seqres.full"
 
-_pwrite_byte 0x62 $((BLKSZ * 16 - 34)) 17 "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x62 $((BLKSZ * 16 - 34)) 17 "$TESTDIR/file3" >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 16 - 34)) 17 "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 16 - 34)) 17 "$testdir/file3" >> "$seqres.full"
 
-_pwrite_byte 0x62 $((BLKSZ * 48 - 8)) 17 "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x62 $((BLKSZ * 48 - 8)) 17 "$TESTDIR/file3" >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 48 - 8)) 17 "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 48 - 8)) 17 "$testdir/file3" >> "$seqres.full"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
 
-cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "Files 1-2 do not match (intentional)"
-cmp -s "$TESTDIR/file1" "$TESTDIR/file3" || echo "Files 1-3 do not match (intentional)"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file3" || echo "Files 2-3 do not match"
+cmp -s "$testdir/file1" "$testdir/file2" || echo "Files 1-2 do not match (intentional)"
+cmp -s "$testdir/file1" "$testdir/file3" || echo "Files 1-3 do not match (intentional)"
+cmp -s "$testdir/file2" "$testdir/file3" || echo "Files 2-3 do not match"
 
 echo "Compare the CoW'd section to the before file"
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 17 \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 17 \
        || echo "Start sections do not match (intentional)"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 16 - 34)) \
-		"$TESTDIR/file2" $((BLKSZ * 16 - 34)) 17 \
+_compare_range  "$testdir/file1" $((blksz * 16 - 34)) \
+		"$testdir/file2" $((blksz * 16 - 34)) 17 \
        || echo "Middle sections do not match (intentional)"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 48 - 8)) \
-		"$TESTDIR/file2" $((BLKSZ * 48 - 8)) 17 \
+_compare_range  "$testdir/file1" $((blksz * 48 - 8)) \
+		"$testdir/file2" $((blksz * 48 - 8)) 17 \
        || echo "End sections do not match (intentional)"
 
 echo "Compare the CoW'd section to the after file"
-_compare_range "$TESTDIR/file2" 0 "$TESTDIR/file3" 0 17 \
+_compare_range "$testdir/file2" 0 "$testdir/file3" 0 17 \
        || echo "Start sections do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 16 - 34)) \
-		"$TESTDIR/file3" $((BLKSZ * 16 - 34)) 17 \
+_compare_range  "$testdir/file2" $((blksz * 16 - 34)) \
+		"$testdir/file3" $((blksz * 16 - 34)) 17 \
        || echo "Middle sections do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 48 - 8)) \
-		"$TESTDIR/file3" $((BLKSZ * 48 - 8)) 17 \
+_compare_range  "$testdir/file2" $((blksz * 48 - 8)) \
+		"$testdir/file3" $((blksz * 48 - 8)) 17 \
        || echo "End sections do not match"
 
 echo "Compare the not CoW'd sections"
-_compare_range "$TESTDIR/file1" 18 "$TESTDIR/file2" 18 17 \
+_compare_range "$testdir/file1" 18 "$testdir/file2" 18 17 \
        || echo "Start sections of 1-2 do not match"
 
-_compare_range "$TESTDIR/file2" 18 "$TESTDIR/file3" 18 17 \
+_compare_range "$testdir/file2" 18 "$testdir/file3" 18 17 \
        || echo "Start sections of 2-3 do not match"
 
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 16 - 17)) \
-		"$TESTDIR/file2" $((BLKSZ * 16 - 17)) 82 \
+_compare_range  "$testdir/file1" $((blksz * 16 - 17)) \
+		"$testdir/file2" $((blksz * 16 - 17)) 82 \
        || echo "Middle sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 16 - 17)) \
-		"$TESTDIR/file3" $((BLKSZ * 16 - 17)) 82 \
+_compare_range  "$testdir/file2" $((blksz * 16 - 17)) \
+		"$testdir/file3" $((blksz * 16 - 17)) 82 \
        || echo "Middle sections of 2-3 do not match"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 48 - 108)) \
-		"$TESTDIR/file2" $((BLKSZ * 48 - 108)) 100 \
+_compare_range  "$testdir/file1" $((blksz * 48 - 108)) \
+		"$testdir/file2" $((blksz * 48 - 108)) 100 \
        || echo "End sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 48 - 108)) \
-		"$TESTDIR/file3" $((BLKSZ * 48 - 108)) 100 \
+_compare_range  "$testdir/file2" $((blksz * 48 - 108)) \
+		"$testdir/file3" $((blksz * 48 - 108)) 100 \
        || echo "End sections of 2-3 do not match"
 
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 14)) \
-		"$TESTDIR/file2" $((BLKSZ * 14)) $BLKSZ \
+_compare_range  "$testdir/file1" $((blksz * 14)) \
+		"$testdir/file2" $((blksz * 14)) $blksz \
        || echo "Untouched sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 14)) \
-		"$TESTDIR/file3" $((BLKSZ * 14)) $BLKSZ \
+_compare_range  "$testdir/file2" $((blksz * 14)) \
+		"$testdir/file3" $((blksz * 14)) $blksz \
        || echo "Untouched sections of 2-3 do not match"
 
 # success, all done
diff --git a/tests/generic/139 b/tests/generic/139
index 54c68fb..59e5bff 100755
--- a/tests/generic/139
+++ b/tests/generic/139
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,100 +50,100 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR=$TEST_DIR/test-$seq
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir=$TEST_DIR/test-$seq
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 48 - 3)) "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ * 48 - 3)) "$TESTDIR/file3" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 48 - 3)) "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 48 - 3)) "$testdir/file3" >> "$seqres.full"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
 
-cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "Files 1-2 should match"
-cmp -s "$TESTDIR/file1" "$TESTDIR/file3" || echo "Files 1-3 should match"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file3" || echo "Files 2-3 should match"
+cmp -s "$testdir/file1" "$testdir/file2" || echo "Files 1-2 should match"
+cmp -s "$testdir/file1" "$testdir/file3" || echo "Files 1-3 should match"
+cmp -s "$testdir/file2" "$testdir/file3" || echo "Files 2-3 should match"
 
 echo "directio CoW the second file"
-_pwrite_byte 0x62 0 $BLKSZ "$TESTDIR/file2" -d >> "$seqres.full"
-_pwrite_byte 0x62 0 $BLKSZ "$TESTDIR/file3" -d >> "$seqres.full"
+_pwrite_byte 0x62 0 $blksz "$testdir/file2" -d >> "$seqres.full"
+_pwrite_byte 0x62 0 $blksz "$testdir/file3" -d >> "$seqres.full"
 
-_pwrite_byte 0x62 $((BLKSZ * 16 - 512)) 512 "$TESTDIR/file2" -d >> "$seqres.full"
-_pwrite_byte 0x62 $((BLKSZ * 16 - 512)) 512 "$TESTDIR/file3" -d >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 16 - 512)) 512 "$testdir/file2" -d >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 16 - 512)) 512 "$testdir/file3" -d >> "$seqres.full"
 
-_pwrite_byte 0x62 $((BLKSZ * 48)) $BLKSZ "$TESTDIR/file2" -d >> "$seqres.full"
-_pwrite_byte 0x62 $((BLKSZ * 48)) $BLKSZ "$TESTDIR/file3" -d >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 48)) $blksz "$testdir/file2" -d >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 48)) $blksz "$testdir/file3" -d >> "$seqres.full"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
 
-cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "Files 1-2 should not match (intentional)"
-cmp -s "$TESTDIR/file1" "$TESTDIR/file3" || echo "Files 1-3 should not match (intentional)"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file3" || echo "Files 2-3 should match"
+cmp -s "$testdir/file1" "$testdir/file2" || echo "Files 1-2 should not match (intentional)"
+cmp -s "$testdir/file1" "$testdir/file3" || echo "Files 1-3 should not match (intentional)"
+cmp -s "$testdir/file2" "$testdir/file3" || echo "Files 2-3 should match"
 
 echo "Compare the CoW'd section to the before file"
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 $BLKSZ \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 $blksz \
        || echo "Start sections do not match (intentional)"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 16 - 512)) \
-		"$TESTDIR/file2" $((BLKSZ * 16 - 512)) 512 \
+_compare_range  "$testdir/file1" $((blksz * 16 - 512)) \
+		"$testdir/file2" $((blksz * 16 - 512)) 512 \
        || echo "Middle sections do not match (intentional)"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 48 - 512)) \
-		"$TESTDIR/file2" $((BLKSZ * 48 - 512)) $BLKSZ \
+_compare_range  "$testdir/file1" $((blksz * 48 - 512)) \
+		"$testdir/file2" $((blksz * 48 - 512)) $blksz \
        || echo "End sections do not match (intentional)"
 
 echo "Compare the CoW'd section to the after file"
-_compare_range "$TESTDIR/file2" 0 "$TESTDIR/file3" 0 $BLKSZ \
+_compare_range "$testdir/file2" 0 "$testdir/file3" 0 $blksz \
        || echo "Start sections do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 16 - 512)) \
-		"$TESTDIR/file3" $((BLKSZ * 16 - 512)) 512 \
+_compare_range  "$testdir/file2" $((blksz * 16 - 512)) \
+		"$testdir/file3" $((blksz * 16 - 512)) 512 \
        || echo "Middle sections do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 48 - 512)) \
-		"$TESTDIR/file3" $((BLKSZ * 48 - 512)) $BLKSZ \
+_compare_range  "$testdir/file2" $((blksz * 48 - 512)) \
+		"$testdir/file3" $((blksz * 48 - 512)) $blksz \
        || echo "End sections do not match"
 
 echo "Compare the not CoW'd sections"
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $BLKSZ 512 \
+_compare_range "$testdir/file1" $blksz "$testdir/file2" $blksz 512 \
        || echo "Start sections of 1-2 do not match"
-_compare_range "$TESTDIR/file2" $BLKSZ "$TESTDIR/file3" $BLKSZ 512 \
+_compare_range "$testdir/file2" $blksz "$testdir/file3" $blksz 512 \
        || echo "Start sections of 2-3 do not match"
 
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 16 - 1024)) \
-		"$TESTDIR/file2" $((BLKSZ * 16 - 1024)) 512 \
+_compare_range  "$testdir/file1" $((blksz * 16 - 1024)) \
+		"$testdir/file2" $((blksz * 16 - 1024)) 512 \
        || echo "Middle sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 16 - 1024)) \
-		"$TESTDIR/file3" $((BLKSZ * 16 - 1024)) 512 \
+_compare_range  "$testdir/file2" $((blksz * 16 - 1024)) \
+		"$testdir/file3" $((blksz * 16 - 1024)) 512 \
        || echo "Middle sections of 2-3 do not match"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 48 - 1024)) \
-		"$TESTDIR/file2" $((BLKSZ * 48 - 1024)) 512 \
+_compare_range  "$testdir/file1" $((blksz * 48 - 1024)) \
+		"$testdir/file2" $((blksz * 48 - 1024)) 512 \
        || echo "End sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 48 - 1024)) \
-		"$TESTDIR/file3" $((BLKSZ * 48 - 1024)) 512 \
+_compare_range  "$testdir/file2" $((blksz * 48 - 1024)) \
+		"$testdir/file3" $((blksz * 48 - 1024)) 512 \
        || echo "End sections of 2-3 do not match"
 
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 16)) \
-		"$TESTDIR/file2" $((BLKSZ * 16)) 512 \
+_compare_range  "$testdir/file1" $((blksz * 16)) \
+		"$testdir/file2" $((blksz * 16)) 512 \
        || echo "Untouched sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 16)) \
-		"$TESTDIR/file3" $((BLKSZ * 16)) 512 \
+_compare_range  "$testdir/file2" $((blksz * 16)) \
+		"$testdir/file3" $((blksz * 16)) 512 \
        || echo "Untouched sections of 2-3 do not match"
 
 # success, all done
diff --git a/tests/generic/140 b/tests/generic/140
index 6299d8e..e8e7655 100755
--- a/tests/generic/140
+++ b/tests/generic/140
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,101 +50,101 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 48 - 3)) "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ * 48 - 3)) "$TESTDIR/file3" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 48 - 3)) "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 48 - 3)) "$testdir/file3" >> "$seqres.full"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
 
-cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "Files 1-2 do not match"
-cmp -s "$TESTDIR/file1" "$TESTDIR/file3" || echo "Files 1-3 do not match"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file3" || echo "Files 2-3 do not match"
+cmp -s "$testdir/file1" "$testdir/file2" || echo "Files 1-2 do not match"
+cmp -s "$testdir/file1" "$testdir/file3" || echo "Files 1-3 do not match"
+cmp -s "$testdir/file2" "$testdir/file3" || echo "Files 2-3 do not match"
 
 echo "mmap CoW the second file"
-_mwrite_byte 0x62 0 17 $((BLKSZ * 48 - 3)) "$TESTDIR/file2" >> "$seqres.full"
-_mwrite_byte 0x62 0 17 $((BLKSZ * 48 - 3)) "$TESTDIR/file3" >> "$seqres.full"
+_mwrite_byte 0x62 0 17 $((blksz * 48 - 3)) "$testdir/file2" >> "$seqres.full"
+_mwrite_byte 0x62 0 17 $((blksz * 48 - 3)) "$testdir/file3" >> "$seqres.full"
 
-_mwrite_byte 0x62 $((BLKSZ * 16 - 34)) 17 $((BLKSZ * 48 - 3)) "$TESTDIR/file2" >> "$seqres.full"
-_mwrite_byte 0x62 $((BLKSZ * 16 - 34)) 17 $((BLKSZ * 48 - 3)) "$TESTDIR/file3" >> "$seqres.full"
+_mwrite_byte 0x62 $((blksz * 16 - 34)) 17 $((blksz * 48 - 3)) "$testdir/file2" >> "$seqres.full"
+_mwrite_byte 0x62 $((blksz * 16 - 34)) 17 $((blksz * 48 - 3)) "$testdir/file3" >> "$seqres.full"
 
-_mwrite_byte 0x62 $((BLKSZ * 48 - 20)) 17 $((BLKSZ * 48 - 3)) "$TESTDIR/file2" >> "$seqres.full"
-_mwrite_byte 0x62 $((BLKSZ * 48 - 20)) 17 $((BLKSZ * 48 - 3)) "$TESTDIR/file3" >> "$seqres.full"
+_mwrite_byte 0x62 $((blksz * 48 - 20)) 17 $((blksz * 48 - 3)) "$testdir/file2" >> "$seqres.full"
+_mwrite_byte 0x62 $((blksz * 48 - 20)) 17 $((blksz * 48 - 3)) "$testdir/file3" >> "$seqres.full"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
 
-cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "Files 1-2 do not match (intentional)"
-cmp -s "$TESTDIR/file1" "$TESTDIR/file3" || echo "Files 1-3 do not match (intentional)"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file3" || echo "Files 2-3 do not match"
+cmp -s "$testdir/file1" "$testdir/file2" || echo "Files 1-2 do not match (intentional)"
+cmp -s "$testdir/file1" "$testdir/file3" || echo "Files 1-3 do not match (intentional)"
+cmp -s "$testdir/file2" "$testdir/file3" || echo "Files 2-3 do not match"
 
 echo "Compare the CoW'd section to the before file"
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file2" 0 17 \
+_compare_range "$testdir/file1" 0 "$testdir/file2" 0 17 \
        || echo "Start sections do not match (intentional)"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 16 - 34)) \
-		"$TESTDIR/file2" $((BLKSZ * 16 - 34)) 17 \
+_compare_range  "$testdir/file1" $((blksz * 16 - 34)) \
+		"$testdir/file2" $((blksz * 16 - 34)) 17 \
        || echo "Middle sections do not match (intentional)"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 48 - 20)) \
-		"$TESTDIR/file2" $((BLKSZ * 48 - 20)) 17 \
+_compare_range  "$testdir/file1" $((blksz * 48 - 20)) \
+		"$testdir/file2" $((blksz * 48 - 20)) 17 \
        || echo "End sections do not match (intentional)"
 
 echo "Compare the CoW'd section to the after file"
-_compare_range "$TESTDIR/file2" 0 "$TESTDIR/file3" 0 17 \
+_compare_range "$testdir/file2" 0 "$testdir/file3" 0 17 \
        || echo "Start sections do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 16 - 34)) \
-		"$TESTDIR/file3" $((BLKSZ * 16 - 34)) 17 \
+_compare_range  "$testdir/file2" $((blksz * 16 - 34)) \
+		"$testdir/file3" $((blksz * 16 - 34)) 17 \
        || echo "Middle sections do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 48 - 20)) \
-		"$TESTDIR/file3" $((BLKSZ * 48 - 20)) 17 \
+_compare_range  "$testdir/file2" $((blksz * 48 - 20)) \
+		"$testdir/file3" $((blksz * 48 - 20)) 17 \
        || echo "End sections do not match"
 
 echo "Compare the not CoW'd sections"
-_compare_range "$TESTDIR/file1" 18 "$TESTDIR/file2" 18 17 \
+_compare_range "$testdir/file1" 18 "$testdir/file2" 18 17 \
        || echo "Start sections of 1-2 do not match"
 
-_compare_range "$TESTDIR/file2" 18 "$TESTDIR/file3" 18 17 \
+_compare_range "$testdir/file2" 18 "$testdir/file3" 18 17 \
        || echo "Start sections of 2-3 do not match"
 
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 16 - 17)) \
-		"$TESTDIR/file2" $((BLKSZ * 16 - 17)) 82 \
+_compare_range  "$testdir/file1" $((blksz * 16 - 17)) \
+		"$testdir/file2" $((blksz * 16 - 17)) 82 \
        || echo "Middle sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 16 - 17)) \
-		"$TESTDIR/file3" $((BLKSZ * 16 - 17)) 82 \
+_compare_range  "$testdir/file2" $((blksz * 16 - 17)) \
+		"$testdir/file3" $((blksz * 16 - 17)) 82 \
        || echo "Middle sections of 2-3 do not match"
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 48 - 120)) \
-		"$TESTDIR/file2" $((BLKSZ * 48 - 120)) 100 \
+_compare_range  "$testdir/file1" $((blksz * 48 - 120)) \
+		"$testdir/file2" $((blksz * 48 - 120)) 100 \
        || echo "End sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 48 - 120)) \
-		"$TESTDIR/file3" $((BLKSZ * 48 - 120)) 100 \
+_compare_range  "$testdir/file2" $((blksz * 48 - 120)) \
+		"$testdir/file3" $((blksz * 48 - 120)) 100 \
        || echo "End sections of 2-3 do not match"
 
 
-_compare_range  "$TESTDIR/file1" $((BLKSZ * 14)) \
-		"$TESTDIR/file2" $((BLKSZ * 14)) $BLKSZ \
+_compare_range  "$testdir/file1" $((blksz * 14)) \
+		"$testdir/file2" $((blksz * 14)) $blksz \
        || echo "Untouched sections of 1-2 do not match"
 
-_compare_range  "$TESTDIR/file2" $((BLKSZ * 14)) \
-		"$TESTDIR/file3" $((BLKSZ * 14)) $BLKSZ \
+_compare_range  "$testdir/file2" $((blksz * 14)) \
+		"$testdir/file3" $((blksz * 14)) $blksz \
        || echo "Untouched sections of 2-3 do not match"
 
 # success, all done
diff --git a/tests/generic/142 b/tests/generic/142
index cf6d634..2ac4d10 100755
--- a/tests/generic/142
+++ b/tests/generic/142
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,37 +52,37 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ=65536
-NR=9
-_pwrite_byte 0x61 0 $((BLKSZ * 256)) "$TESTDIR/file1" >> "$seqres.full"
+blksz=65536
+nr=9
+_pwrite_byte 0x61 0 $((blksz * 256)) "$testdir/file1" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-csum="$(_md5_checksum "$TESTDIR/file1")"
+md5sum "$testdir/file1" | _filter_test_dir
+csum="$(_md5_checksum "$testdir/file1")"
 
 echo "Create the reflink copies"
-seq 2 $NR | while read i; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file$i"
+seq 2 $nr | while read i; do
+	_cp_reflink "$testdir/file1" "$testdir/file$i"
 done
 _test_remount
 
 echo "Rewrite the copies"
-seq 2 $NR | while read i; do
-	_pwrite_byte 0x62 0 $((BLKSZ * 256)) "$TESTDIR/file$i" >> "$seqres.full"
+seq 2 $nr | while read i; do
+	_pwrite_byte 0x62 0 $((blksz * 256)) "$testdir/file$i" >> "$seqres.full"
 done
 _test_remount
 
 echo "Examine original file"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-mod_csum="$(_md5_checksum "$TESTDIR/file2")"
-new_csum="$(_md5_checksum "$TESTDIR/file1")"
+mod_csum="$(_md5_checksum "$testdir/file2")"
+new_csum="$(_md5_checksum "$testdir/file1")"
 test "${csum}" != "${mod_csum}" || echo "checksums do not match"
 test "${csum}" = "${new_csum}" || echo "checksums do not match"
 
diff --git a/tests/generic/143 b/tests/generic/143
index d2ccb94..c0bc672 100755
--- a/tests/generic/143
+++ b/tests/generic/143
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,37 +52,37 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ=65536
-NR=9
-_pwrite_byte 0x61 0 $((BLKSZ * 256)) "$TESTDIR/file1" >> "$seqres.full"
+blksz=65536
+nr=9
+_pwrite_byte 0x61 0 $((blksz * 256)) "$testdir/file1" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-csum="$(_md5_checksum "$TESTDIR/file1")"
+md5sum "$testdir/file1" | _filter_test_dir
+csum="$(_md5_checksum "$testdir/file1")"
 
 echo "Create the reflink copies"
-seq 2 $NR | while read i; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file$i"
+seq 2 $nr | while read i; do
+	_cp_reflink "$testdir/file1" "$testdir/file$i"
 done
 _test_remount
 
 echo "Rewrite the copies"
-seq 2 $NR | while read i; do
-	_pwrite_byte 0x62 0 $((BLKSZ * 256)) "$TESTDIR/file$i" -d >> "$seqres.full"
+seq 2 $nr | while read i; do
+	_pwrite_byte 0x62 0 $((blksz * 256)) "$testdir/file$i" -d >> "$seqres.full"
 done
 _test_remount
 
 echo "Examine original file"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
-mod_csum="$(_md5_checksum "$TESTDIR/file2")"
-new_csum="$(_md5_checksum "$TESTDIR/file1")"
+mod_csum="$(_md5_checksum "$testdir/file2")"
+new_csum="$(_md5_checksum "$testdir/file1")"
 test "${csum}" != "${mod_csum}" || echo "checksums do not match"
 test "${csum}" = "${new_csum}" || echo "checksums do not match"
 
diff --git a/tests/generic/144 b/tests/generic/144
index bee0893..631fa33 100755
--- a/tests/generic/144
+++ b/tests/generic/144
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,90 +52,90 @@ _require_xfs_io_command "truncate"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 5 + 37)) "$TESTDIR/file1" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 5 + 37)) "$testdir/file1" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $BLKSZ \
-		$((BLKSZ * 4 + 37)) >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $blksz \
+		$((blksz * 4 + 37)) >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $((BLKSZ * 5 + 37))" "$TESTDIR/file3" >> "$seqres.full"
-_reflink_range "$TESTDIR/file1" 0 "$TESTDIR/file3" 0 $BLKSZ >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $((blksz * 5 + 37))" "$testdir/file3" >> "$seqres.full"
+_reflink_range "$testdir/file1" 0 "$testdir/file3" 0 $blksz >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $((BLKSZ * 5 + 37))" "$TESTDIR/file4" >> "$seqres.full"
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file4" $BLKSZ $BLKSZ >> "$seqres.full"
-_reflink_range "$TESTDIR/file1" $((BLKSZ * 3)) "$TESTDIR/file4" $((BLKSZ * 3)) \
-		$BLKSZ >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $((blksz * 5 + 37))" "$testdir/file4" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file4" $blksz $blksz >> "$seqres.full"
+_reflink_range "$testdir/file1" $((blksz * 3)) "$testdir/file4" $((blksz * 3)) \
+		$blksz >> "$seqres.full"
 
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file5"
+_cp_reflink "$testdir/file1" "$testdir/file5"
 _test_remount
 
 echo "Compare sections"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file5" | _filter_test_dir
-
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $BLKSZ \
-		$((BLKSZ * 4 + 37)) \
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file5" | _filter_test_dir
+
+_compare_range "$testdir/file1" $blksz "$testdir/file2" $blksz \
+		$((blksz * 4 + 37)) \
 	|| echo "shared parts of files 1-2 changed"
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file3" 0 $BLKSZ \
+_compare_range "$testdir/file1" 0 "$testdir/file3" 0 $blksz \
 	|| echo "shared parts of files 1-3 changed"
 
-_compare_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file4" $BLKSZ $BLKSZ \
+_compare_range "$testdir/file1" $blksz "$testdir/file4" $blksz $blksz \
 	|| echo "shared parts of files 1-4 changed"
 
-_compare_range "$TESTDIR/file1" 0 "$TESTDIR/file5" 0 $((BLKSZ * 5 + 37)) \
+_compare_range "$testdir/file1" 0 "$testdir/file5" 0 $((blksz * 5 + 37)) \
 	|| echo "shared parts of files 1-5 changed"
 
 echo "Compare files"
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-C5="$(_md5_checksum "$TESTDIR/file5")"
-
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C1}"  = "${C5}" || echo "file1 and file5 should match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C2}" != "${C5}" || echo "file2 and file5 should not match"
-test "${C3}" != "${C4}" || echo "file3 and file4 should not match"
-test "${C3}" != "${C5}" || echo "file3 and file5 should not match"
-test "${C4}" != "${C5}" || echo "file4 and file5 should not match"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+c5="$(_md5_checksum "$testdir/file5")"
+
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c1}"  = "${c5}" || echo "file1 and file5 should match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c2}" != "${c5}" || echo "file2 and file5 should not match"
+test "${c3}" != "${c4}" || echo "file3 and file4 should not match"
+test "${c3}" != "${c5}" || echo "file3 and file5 should not match"
+test "${c4}" != "${c5}" || echo "file4 and file5 should not match"
 
 echo "falloc everything"
-"$XFS_IO_PROG" -f -c "falloc 0 $((BLKSZ * 5))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "falloc 0 $((BLKSZ * 5))" "$TESTDIR/file3" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "falloc 0 $((BLKSZ * 5))" "$TESTDIR/file4" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "falloc 0 $((blksz * 5))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "falloc 0 $((blksz * 5))" "$testdir/file3" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "falloc 0 $((blksz * 5))" "$testdir/file4" >> "$seqres.full"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file5" | _filter_test_dir
-
-D1="$(_md5_checksum "$TESTDIR/file1")"
-D2="$(_md5_checksum "$TESTDIR/file2")"
-D3="$(_md5_checksum "$TESTDIR/file3")"
-D4="$(_md5_checksum "$TESTDIR/file4")"
-D5="$(_md5_checksum "$TESTDIR/file5")"
-
-test "${C1}" = "${D1}" || echo "file1 should not change"
-test "${C2}" = "${D2}" || echo "file2 should not change"
-test "${C3}" = "${D3}" || echo "file3 should not change"
-test "${C4}" = "${D4}" || echo "file4 should not change"
-test "${C5}" = "${D5}" || echo "file2 should not change"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file5" | _filter_test_dir
+
+d1="$(_md5_checksum "$testdir/file1")"
+d2="$(_md5_checksum "$testdir/file2")"
+d3="$(_md5_checksum "$testdir/file3")"
+d4="$(_md5_checksum "$testdir/file4")"
+d5="$(_md5_checksum "$testdir/file5")"
+
+test "${c1}" = "${d1}" || echo "file1 should not change"
+test "${c2}" = "${d2}" || echo "file2 should not change"
+test "${c3}" = "${d3}" || echo "file3 should not change"
+test "${c4}" = "${d4}" || echo "file4 should not change"
+test "${c5}" = "${d5}" || echo "file2 should not change"
 
 # success, all done
 status=0
diff --git a/tests/generic/145 b/tests/generic/145
index 8a14f82..2b8d74f 100755
--- a/tests/generic/145
+++ b/tests/generic/145
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -53,89 +53,89 @@ _require_xfs_io_command "fcollapse"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file3"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file4"
-
-"$XFS_IO_PROG" -f -c "falloc 0 $((BLKSZ * 4))" "$TESTDIR/file1"
-"$XFS_IO_PROG" -f -c "falloc 0 $((BLKSZ * 4))" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "falloc 0 $((BLKSZ * 4))" "$TESTDIR/file3"
-"$XFS_IO_PROG" -f -c "falloc 0 $((BLKSZ * 4))" "$TESTDIR/file4"
-
-_pwrite_byte 0x62 0 $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $BLKSZ $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 2)) $blksz "$testdir/file1" >> "$seqres.full"
+
+_cp_reflink "$testdir/file1" "$testdir/file2"
+_cp_reflink "$testdir/file1" "$testdir/file3"
+_cp_reflink "$testdir/file1" "$testdir/file4"
+
+"$XFS_IO_PROG" -f -c "falloc 0 $((blksz * 4))" "$testdir/file1"
+"$XFS_IO_PROG" -f -c "falloc 0 $((blksz * 4))" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "falloc 0 $((blksz * 4))" "$testdir/file3"
+"$XFS_IO_PROG" -f -c "falloc 0 $((blksz * 4))" "$testdir/file4"
+
+_pwrite_byte 0x62 0 $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
+
+_pwrite_byte 0x61 0 $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $blksz $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz * 2)) $blksz "$testdir/file3.chk" >> "$seqres.full"
+
+_pwrite_byte 0x61 0 $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz * 2)) $blksz "$testdir/file4.chk" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
-md5sum "$TESTDIR/file4.chk" | _filter_test_dir
-
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" = "${C3}" || echo "file1 and file3 should match"
-test "${C1}" = "${C4}" || echo "file1 and file4 should match"
-test "${C2}" = "${C3}" || echo "file2 and file3 should match"
-test "${C2}" = "${C4}" || echo "file2 and file4 should match"
-test "${C3}" = "${C4}" || echo "file3 and file4 should match"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
+md5sum "$testdir/file4.chk" | _filter_test_dir
+
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" = "${c3}" || echo "file1 and file3 should match"
+test "${c1}" = "${c4}" || echo "file1 and file4 should match"
+test "${c2}" = "${c3}" || echo "file2 and file3 should match"
+test "${c2}" = "${c4}" || echo "file2 and file4 should match"
+test "${c3}" = "${c4}" || echo "file3 and file4 should match"
 
 echo "fcollapse files"
-"$XFS_IO_PROG" -f -c "fcollapse 0 $BLKSZ" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "fcollapse $BLKSZ $BLKSZ" "$TESTDIR/file3"
-"$XFS_IO_PROG" -f -c "fcollapse $((BLKSZ * 2)) $BLKSZ" "$TESTDIR/file4"
+"$XFS_IO_PROG" -f -c "fcollapse 0 $blksz" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "fcollapse $blksz $blksz" "$testdir/file3"
+"$XFS_IO_PROG" -f -c "fcollapse $((blksz * 2)) $blksz" "$testdir/file4"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
-md5sum "$TESTDIR/file4.chk" | _filter_test_dir
-
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C3}" != "${C4}" || echo "file3 and file4 should not match"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
+md5sum "$testdir/file4.chk" | _filter_test_dir
+
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c3}" != "${c4}" || echo "file3 and file4 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
-cmp -s "$TESTDIR/file3" "$TESTDIR/file3.chk" || echo "file3 and file3.chk do not match"
-cmp -s "$TESTDIR/file4" "$TESTDIR/file4.chk" || echo "file4 and file4.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file3" "$testdir/file3.chk" || echo "file3 and file3.chk do not match"
+cmp -s "$testdir/file4" "$testdir/file4.chk" || echo "file4 and file4.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/146 b/tests/generic/146
index 9e3eff6..17dc306 100755
--- a/tests/generic/146
+++ b/tests/generic/146
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,84 +52,84 @@ _require_xfs_io_command "fpunch"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file3"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file4"
-
-_pwrite_byte 0x00 0 $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $BLKSZ $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 2)) $blksz "$testdir/file1" >> "$seqres.full"
+
+_cp_reflink "$testdir/file1" "$testdir/file2"
+_cp_reflink "$testdir/file1" "$testdir/file3"
+_cp_reflink "$testdir/file1" "$testdir/file4"
+
+_pwrite_byte 0x00 0 $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
+
+_pwrite_byte 0x61 0 $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $blksz $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 2)) $blksz "$testdir/file3.chk" >> "$seqres.full"
+
+_pwrite_byte 0x61 0 $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz * 2)) $blksz "$testdir/file4.chk" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
-md5sum "$TESTDIR/file4.chk" | _filter_test_dir
-
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" = "${C3}" || echo "file1 and file3 should match"
-test "${C1}" = "${C4}" || echo "file1 and file4 should match"
-test "${C2}" = "${C3}" || echo "file2 and file3 should match"
-test "${C2}" = "${C4}" || echo "file2 and file4 should match"
-test "${C3}" = "${C4}" || echo "file3 and file4 should match"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
+md5sum "$testdir/file4.chk" | _filter_test_dir
+
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" = "${c3}" || echo "file1 and file3 should match"
+test "${c1}" = "${c4}" || echo "file1 and file4 should match"
+test "${c2}" = "${c3}" || echo "file2 and file3 should match"
+test "${c2}" = "${c4}" || echo "file2 and file4 should match"
+test "${c3}" = "${c4}" || echo "file3 and file4 should match"
 
 echo "fpunch files"
-"$XFS_IO_PROG" -f -c "fpunch 0 $BLKSZ" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "fpunch $BLKSZ $BLKSZ" "$TESTDIR/file3"
-"$XFS_IO_PROG" -f -c "fpunch $((BLKSZ * 2)) $BLKSZ" "$TESTDIR/file4"
+"$XFS_IO_PROG" -f -c "fpunch 0 $blksz" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "fpunch $blksz $blksz" "$testdir/file3"
+"$XFS_IO_PROG" -f -c "fpunch $((blksz * 2)) $blksz" "$testdir/file4"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
-md5sum "$TESTDIR/file4.chk" | _filter_test_dir
-
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C3}" != "${C4}" || echo "file3 and file4 should not match"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
+md5sum "$testdir/file4.chk" | _filter_test_dir
+
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c3}" != "${c4}" || echo "file3 and file4 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
-cmp -s "$TESTDIR/file3" "$TESTDIR/file3.chk" || echo "file3 and file3.chk do not match"
-cmp -s "$TESTDIR/file4" "$TESTDIR/file4.chk" || echo "file4 and file4.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file3" "$testdir/file3.chk" || echo "file3 and file3.chk do not match"
+cmp -s "$testdir/file4" "$testdir/file4.chk" || echo "file4 and file4.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/147 b/tests/generic/147
index c35bdd9..b160131 100755
--- a/tests/generic/147
+++ b/tests/generic/147
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,87 +52,87 @@ _require_xfs_io_command "finsert"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file3"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file4"
-
-_pwrite_byte 0x00 0 $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $BLKSZ $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 2)) $blksz "$testdir/file1" >> "$seqres.full"
+
+_cp_reflink "$testdir/file1" "$testdir/file2"
+_cp_reflink "$testdir/file1" "$testdir/file3"
+_cp_reflink "$testdir/file1" "$testdir/file4"
+
+_pwrite_byte 0x00 0 $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 3)) $blksz "$testdir/file2.chk" >> "$seqres.full"
+
+_pwrite_byte 0x61 0 $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $blksz $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $((blksz * 2)) $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 3)) $blksz "$testdir/file3.chk" >> "$seqres.full"
+
+_pwrite_byte 0x61 0 $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz * 2)) $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 3)) $blksz "$testdir/file4.chk" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
-md5sum "$TESTDIR/file4.chk" | _filter_test_dir
-
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" = "${C3}" || echo "file1 and file3 should match"
-test "${C1}" = "${C4}" || echo "file1 and file4 should match"
-test "${C2}" = "${C3}" || echo "file2 and file3 should match"
-test "${C2}" = "${C4}" || echo "file2 and file4 should match"
-test "${C3}" = "${C4}" || echo "file3 and file4 should match"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
+md5sum "$testdir/file4.chk" | _filter_test_dir
+
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" = "${c3}" || echo "file1 and file3 should match"
+test "${c1}" = "${c4}" || echo "file1 and file4 should match"
+test "${c2}" = "${c3}" || echo "file2 and file3 should match"
+test "${c2}" = "${c4}" || echo "file2 and file4 should match"
+test "${c3}" = "${c4}" || echo "file3 and file4 should match"
 
 echo "finsert files"
-"$XFS_IO_PROG" -f -c "finsert 0 $BLKSZ" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "finsert $BLKSZ $BLKSZ" "$TESTDIR/file3"
-"$XFS_IO_PROG" -f -c "finsert $((BLKSZ * 2)) $BLKSZ" "$TESTDIR/file4"
+"$XFS_IO_PROG" -f -c "finsert 0 $blksz" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "finsert $blksz $blksz" "$testdir/file3"
+"$XFS_IO_PROG" -f -c "finsert $((blksz * 2)) $blksz" "$testdir/file4"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
-md5sum "$TESTDIR/file4.chk" | _filter_test_dir
-
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C3}" != "${C4}" || echo "file3 and file4 should not match"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
+md5sum "$testdir/file4.chk" | _filter_test_dir
+
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c3}" != "${c4}" || echo "file3 and file4 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
-cmp -s "$TESTDIR/file3" "$TESTDIR/file3.chk" || echo "file3 and file3.chk do not match"
-cmp -s "$TESTDIR/file4" "$TESTDIR/file4.chk" || echo "file4 and file4.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file3" "$testdir/file3.chk" || echo "file3 and file3.chk do not match"
+cmp -s "$testdir/file4" "$testdir/file4.chk" || echo "file4 and file4.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/148 b/tests/generic/148
index 3d7a762..cf5567b 100755
--- a/tests/generic/148
+++ b/tests/generic/148
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -53,63 +53,63 @@ _require_xfs_io_command "truncate"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ 37 "$TESTDIR/file1" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz 37 "$testdir/file1" >> "$seqres.full"
 
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file3"
+_cp_reflink "$testdir/file1" "$testdir/file2"
+_cp_reflink "$testdir/file1" "$testdir/file3"
 
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ 34 "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 0 $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz 34 "$testdir/file2.chk" >> "$seqres.full"
 
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ 37 "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ + 37)) 3 "$TESTDIR/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x61 0 $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz 37 "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz + 37)) 3 "$testdir/file3.chk" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
 
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" = "${C3}" || echo "file1 and file3 should match"
-test "${C2}" = "${C3}" || echo "file2 and file3 should match"
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" = "${c3}" || echo "file1 and file3 should match"
+test "${c2}" = "${c3}" || echo "file2 and file3 should match"
 
 echo "truncate files"
-"$XFS_IO_PROG" -f -c "truncate $((BLKSZ + 34))" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "truncate $((BLKSZ + 40))" "$TESTDIR/file3"
+"$XFS_IO_PROG" -f -c "truncate $((blksz + 34))" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "truncate $((blksz + 40))" "$testdir/file3"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
 
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
-cmp -s "$TESTDIR/file3" "$TESTDIR/file3.chk" || echo "file3 and file3.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file3" "$testdir/file3.chk" || echo "file3 and file3.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/149 b/tests/generic/149
index 44b5ae4..7d3189e 100755
--- a/tests/generic/149
+++ b/tests/generic/149
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,84 +52,84 @@ _require_xfs_io_command "fzero"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file3"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file4"
-
-_pwrite_byte 0x00 0 $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
-
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $BLKSZ $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
-
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file4.chk" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 2)) $blksz "$testdir/file1" >> "$seqres.full"
+
+_cp_reflink "$testdir/file1" "$testdir/file2"
+_cp_reflink "$testdir/file1" "$testdir/file3"
+_cp_reflink "$testdir/file1" "$testdir/file4"
+
+_pwrite_byte 0x00 0 $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
+
+_pwrite_byte 0x61 0 $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $blksz $blksz "$testdir/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * 2)) $blksz "$testdir/file3.chk" >> "$seqres.full"
+
+_pwrite_byte 0x61 0 $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file4.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz * 2)) $blksz "$testdir/file4.chk" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
-md5sum "$TESTDIR/file4.chk" | _filter_test_dir
-
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
-test "${C1}" = "${C3}" || echo "file1 and file3 should match"
-test "${C1}" = "${C4}" || echo "file1 and file4 should match"
-test "${C2}" = "${C3}" || echo "file2 and file3 should match"
-test "${C2}" = "${C4}" || echo "file2 and file4 should match"
-test "${C3}" = "${C4}" || echo "file3 and file4 should match"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
+md5sum "$testdir/file4.chk" | _filter_test_dir
+
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
+test "${c1}" = "${c3}" || echo "file1 and file3 should match"
+test "${c1}" = "${c4}" || echo "file1 and file4 should match"
+test "${c2}" = "${c3}" || echo "file2 and file3 should match"
+test "${c2}" = "${c4}" || echo "file2 and file4 should match"
+test "${c3}" = "${c4}" || echo "file3 and file4 should match"
 
 echo "fzero files"
-"$XFS_IO_PROG" -f -c "fzero 0 $BLKSZ" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "fzero $BLKSZ $BLKSZ" "$TESTDIR/file3"
-"$XFS_IO_PROG" -f -c "fzero $((BLKSZ * 2)) $BLKSZ" "$TESTDIR/file4"
+"$XFS_IO_PROG" -f -c "fzero 0 $blksz" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "fzero $blksz $blksz" "$testdir/file3"
+"$XFS_IO_PROG" -f -c "fzero $((blksz * 2)) $blksz" "$testdir/file4"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file3" | _filter_test_dir
-md5sum "$TESTDIR/file4" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
-md5sum "$TESTDIR/file3.chk" | _filter_test_dir
-md5sum "$TESTDIR/file4.chk" | _filter_test_dir
-
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
-C3="$(_md5_checksum "$TESTDIR/file3")"
-C4="$(_md5_checksum "$TESTDIR/file4")"
-
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
-test "${C1}" != "${C3}" || echo "file1 and file3 should not match"
-test "${C1}" != "${C4}" || echo "file1 and file4 should not match"
-test "${C2}" != "${C3}" || echo "file2 and file3 should not match"
-test "${C2}" != "${C4}" || echo "file2 and file4 should not match"
-test "${C3}" != "${C4}" || echo "file3 and file4 should not match"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file3" | _filter_test_dir
+md5sum "$testdir/file4" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
+md5sum "$testdir/file3.chk" | _filter_test_dir
+md5sum "$testdir/file4.chk" | _filter_test_dir
+
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
+c3="$(_md5_checksum "$testdir/file3")"
+c4="$(_md5_checksum "$testdir/file4")"
+
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c3}" || echo "file1 and file3 should not match"
+test "${c1}" != "${c4}" || echo "file1 and file4 should not match"
+test "${c2}" != "${c3}" || echo "file2 and file3 should not match"
+test "${c2}" != "${c4}" || echo "file2 and file4 should not match"
+test "${c3}" != "${c4}" || echo "file3 and file4 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
-cmp -s "$TESTDIR/file3" "$TESTDIR/file3.chk" || echo "file3 and file3.chk do not match"
-cmp -s "$TESTDIR/file4" "$TESTDIR/file4.chk" || echo "file4 and file4.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file3" "$testdir/file3.chk" || echo "file3 and file3.chk do not match"
+cmp -s "$testdir/file4" "$testdir/file4.chk" || echo "file4 and file4.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/150 b/tests/generic/150
index 3046e92..f3a6311 100755
--- a/tests/generic/150
+++ b/tests/generic/150
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,28 +50,28 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f $TESTDIR -c '%S')"
-BLKS=2000
-MARGIN=100
-SZ=$((BLKSZ * BLKS))
-NR=7
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f $testdir -c '%S')"
+blks=2000
+margin=100
+sz=$((blksz * blks))
+nr=7
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
 sync
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
+free_blocks0=$(stat -f "$testdir" -c '%f')
 
 echo "Create the reflink copies"
-for i in `seq 2 $NR`; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file.$i"
+for i in `seq 2 $nr`; do
+	_cp_reflink "$testdir/file1" "$testdir/file.$i"
 done
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
+free_blocks1=$(stat -f "$testdir" -c '%f')
 
-_within_tolerance "free blocks after reflink" $FREE_BLOCKS1 $FREE_BLOCKS0 $MARGIN -v
+_within_tolerance "free blocks after reflink" $free_blocks1 $free_blocks0 $margin -v
 
 # success, all done
 status=0
diff --git a/tests/generic/151 b/tests/generic/151
index c42ca1d..286b9ec 100755
--- a/tests/generic/151
+++ b/tests/generic/151
@@ -39,7 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -54,44 +54,44 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-BLKS=2000
-MARGIN=100
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
-NR=7
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+blks=2000
+margin=100
+sz=$((blksz * blks))
+free_blocks0=$(stat -f "$testdir" -c '%f')
+nr=7
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
 sync
 
 echo "Create the reflink copies"
-for i in `seq 2 $NR`; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file.$i"
+for i in `seq 2 $nr`; do
+	_cp_reflink "$testdir/file1" "$testdir/file.$i"
 done
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/survivor"
+_cp_reflink "$testdir/file1" "$testdir/survivor"
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
+free_blocks1=$(stat -f "$testdir" -c '%f')
 
 echo "Delete most of the files"
-rm -rf "$TESTDIR"/file*
+rm -rf "$testdir"/file*
 _test_remount
-FREE_BLOCKS2=$(stat -f "$TESTDIR" -c '%f')
+free_blocks2=$(stat -f "$testdir" -c '%f')
 
 echo "Delete all the files"
-rm -rf "$TESTDIR"/*
+rm -rf "$testdir"/*
 _test_remount
-FREE_BLOCKS3=$(stat -f "$TESTDIR" -c '%f')
-#echo $FREE_BLOCKS0 $FREE_BLOCKS1 $FREE_BLOCKS2 $FREE_BLOCKS3
+free_blocks3=$(stat -f "$testdir" -c '%f')
+#echo $free_blocks0 $free_blocks1 $free_blocks2 $free_blocks3
 
-_within_tolerance "free blocks after reflink" $FREE_BLOCKS1 $((FREE_BLOCKS0 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after reflink" $free_blocks1 $((free_blocks0 - blks)) $margin -v
 
-_within_tolerance "free blocks after deleting some reflink copies" $FREE_BLOCKS2 $FREE_BLOCKS1 $MARGIN -v
+_within_tolerance "free blocks after deleting some reflink copies" $free_blocks2 $free_blocks1 $margin -v
 
-_within_tolerance "free blocks after deleting all copies" $FREE_BLOCKS3 $FREE_BLOCKS0 $MARGIN -v
+_within_tolerance "free blocks after deleting all copies" $free_blocks3 $free_blocks0 $margin -v
 
 # success, all done
 status=0
diff --git a/tests/generic/152 b/tests/generic/152
index 95ffe40..506a754 100755
--- a/tests/generic/152
+++ b/tests/generic/152
@@ -39,7 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -55,48 +55,48 @@ _require_xfs_io_command "fpunch"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-BLKS=2000
-MARGIN=100
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+blks=2000
+margin=100
+sz=$((blksz * blks))
+free_blocks0=$(stat -f "$testdir" -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
 sync
 
 echo "Create the reflink copies"
-for i in `seq 2 $NR`; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file$i"
+for i in `seq 2 $nr`; do
+	_cp_reflink "$testdir/file1" "$testdir/file$i"
 done
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
+free_blocks1=$(stat -f "$testdir" -c '%f')
 
 echo "Punch most of the blocks"
-"$XFS_IO_PROG" -f -c "fpunch 0 $SZ" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "fpunch 0 $((SZ / 2))" "$TESTDIR/file3"
-"$XFS_IO_PROG" -f -c "fpunch $((SZ / 2)) $((SZ / 2))" "$TESTDIR/file4"
+"$XFS_IO_PROG" -f -c "fpunch 0 $sz" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "fpunch 0 $((sz / 2))" "$testdir/file3"
+"$XFS_IO_PROG" -f -c "fpunch $((sz / 2)) $((sz / 2))" "$testdir/file4"
 _test_remount
-FREE_BLOCKS2=$(stat -f "$TESTDIR" -c '%f')
+free_blocks2=$(stat -f "$testdir" -c '%f')
 
 echo "Punch all the files"
-for i in `seq 2 $NR`; do
-	"$XFS_IO_PROG" -f -c "fpunch 0 $SZ" "$TESTDIR/file$i"
+for i in `seq 2 $nr`; do
+	"$XFS_IO_PROG" -f -c "fpunch 0 $sz" "$testdir/file$i"
 done
-"$XFS_IO_PROG" -f -c "fpunch 0 $SZ" "$TESTDIR/file1"
+"$XFS_IO_PROG" -f -c "fpunch 0 $sz" "$testdir/file1"
 _test_remount
-FREE_BLOCKS3=$(stat -f "$TESTDIR" -c '%f')
-#echo $FREE_BLOCKS0 $FREE_BLOCKS1 $FREE_BLOCKS2 $FREE_BLOCKS3
+free_blocks3=$(stat -f "$testdir" -c '%f')
+#echo $free_blocks0 $free_blocks1 $free_blocks2 $free_blocks3
 
-_within_tolerance "free blocks after reflink" $FREE_BLOCKS1 $((FREE_BLOCKS0 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after reflink" $free_blocks1 $((free_blocks0 - blks)) $margin -v
 
-_within_tolerance "free blocks after punching some reflink copies" $FREE_BLOCKS2 $FREE_BLOCKS1 $MARGIN -v
+_within_tolerance "free blocks after punching some reflink copies" $free_blocks2 $free_blocks1 $margin -v
 
-_within_tolerance "free blocks after punching all copies" $FREE_BLOCKS3 $FREE_BLOCKS0 $MARGIN -v
+_within_tolerance "free blocks after punching all copies" $free_blocks3 $free_blocks0 $margin -v
 
 # success, all done
 status=0
diff --git a/tests/generic/153 b/tests/generic/153
index 2deee20..011879e 100755
--- a/tests/generic/153
+++ b/tests/generic/153
@@ -39,7 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -55,47 +55,47 @@ _require_xfs_io_command "fcollapse"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-BLKS=2000
-MARGIN=100
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+blks=2000
+margin=100
+sz=$((blksz * blks))
+free_blocks0=$(stat -f "$testdir" -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
 
 echo "Create the reflink copies"
-for i in `seq 2 $NR`; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file$i"
+for i in `seq 2 $nr`; do
+	_cp_reflink "$testdir/file1" "$testdir/file$i"
 done
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
+free_blocks1=$(stat -f "$testdir" -c '%f')
 
 echo "Collapse most of the blocks"
-"$XFS_IO_PROG" -f -c "fcollapse 0 $(((BLKS - 1) * BLKSZ))" $TESTDIR/file2
-"$XFS_IO_PROG" -f -c "fcollapse 0 $((SZ / 2))" $TESTDIR/file3
-"$XFS_IO_PROG" -f -c "fcollapse $((SZ / 2)) $(( ((BLKS / 2) - 1) * BLKSZ))" $TESTDIR/file4
+"$XFS_IO_PROG" -f -c "fcollapse 0 $(((blks - 1) * blksz))" $testdir/file2
+"$XFS_IO_PROG" -f -c "fcollapse 0 $((sz / 2))" $testdir/file3
+"$XFS_IO_PROG" -f -c "fcollapse $((sz / 2)) $(( ((blks / 2) - 1) * blksz))" $testdir/file4
 _test_remount
-FREE_BLOCKS2=$(stat -f "$TESTDIR" -c '%f')
+free_blocks2=$(stat -f "$testdir" -c '%f')
 
 echo "Collpase nearly all the files"
-"$XFS_IO_PROG" -f -c "fcollapse 0 $(( ((BLKS / 2) - 1) * BLKSZ))" $TESTDIR/file3
-"$XFS_IO_PROG" -f -c "fcollapse 0 $((SZ / 2))" $TESTDIR/file4
-"$XFS_IO_PROG" -f -c "fcollapse 0 $(( (BLKS - 1) * BLKSZ))" $TESTDIR/file1
+"$XFS_IO_PROG" -f -c "fcollapse 0 $(( ((blks / 2) - 1) * blksz))" $testdir/file3
+"$XFS_IO_PROG" -f -c "fcollapse 0 $((sz / 2))" $testdir/file4
+"$XFS_IO_PROG" -f -c "fcollapse 0 $(( (blks - 1) * blksz))" $testdir/file1
 _test_remount
-FREE_BLOCKS3=$(stat -f "$TESTDIR" -c '%f')
-#echo $FREE_BLOCKS0 $FREE_BLOCKS1 $FREE_BLOCKS2 $FREE_BLOCKS3
+free_blocks3=$(stat -f "$testdir" -c '%f')
+#echo $free_blocks0 $free_blocks1 $free_blocks2 $free_blocks3
 
-_within_tolerance "free blocks after reflink" $FREE_BLOCKS1 $((FREE_BLOCKS0 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after reflink" $free_blocks1 $((free_blocks0 - blks)) $margin -v
 
-_within_tolerance "free blocks after fcollapsing some reflink copies" $FREE_BLOCKS2 $FREE_BLOCKS1 $MARGIN -v
+_within_tolerance "free blocks after fcollapsing some reflink copies" $free_blocks2 $free_blocks1 $margin -v
 
-_within_tolerance "free blocks after fcollapsing all copies" $FREE_BLOCKS3 $FREE_BLOCKS0 $MARGIN -v
+_within_tolerance "free blocks after fcollapsing all copies" $free_blocks3 $free_blocks0 $margin -v
 
 # success, all done
 status=0
diff --git a/tests/generic/154 b/tests/generic/154
index 173b981..474c578 100755
--- a/tests/generic/154
+++ b/tests/generic/154
@@ -39,7 +39,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -54,56 +54,56 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-BLKS=2000
-MARGIN=100
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
-NR=4
-SZ=$((BLKS * BLKSZ))
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+blks=2000
+margin=100
+free_blocks0=$(stat -f "$testdir" -c '%f')
+nr=4
+sz=$((blks * blksz))
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
 
 echo "Create the reflink copies"
-for i in `seq 2 $NR`; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file$i"
+for i in `seq 2 $nr`; do
+	_cp_reflink "$testdir/file1" "$testdir/file$i"
 done
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
+free_blocks1=$(stat -f "$testdir" -c '%f')
 
 echo "Rewrite some of the blocks"
-_pwrite_byte 0x62 0 $SZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x63 0 $((SZ / 2)) "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x64 $((SZ / 2)) $((SZ / 2)) "$TESTDIR/file4" >> "$seqres.full"
+_pwrite_byte 0x62 0 $sz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x63 0 $((sz / 2)) "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x64 $((sz / 2)) $((sz / 2)) "$testdir/file4" >> "$seqres.full"
 _test_remount
-FREE_BLOCKS2=$(stat -f "$TESTDIR" -c '%f')
+free_blocks2=$(stat -f "$testdir" -c '%f')
 
 echo "Rewrite all the files"
-_pwrite_byte 0x62 0 $SZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x63 0 $SZ "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x64 0 $SZ "$TESTDIR/file4" >> "$seqres.full"
+_pwrite_byte 0x62 0 $sz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x63 0 $sz "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x64 0 $sz "$testdir/file4" >> "$seqres.full"
 _test_remount
-FREE_BLOCKS3=$(stat -f "$TESTDIR" -c '%f')
+free_blocks3=$(stat -f "$testdir" -c '%f')
 
 echo "Rewrite the original file"
-_pwrite_byte 0x65 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x65 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
-FREE_BLOCKS4=$(stat -f "$TESTDIR" -c '%f')
-#echo $FREE_BLOCKS0 $FREE_BLOCKS1 $FREE_BLOCKS2 $FREE_BLOCKS3 $FREE_BLOCKS4
+free_blocks4=$(stat -f "$testdir" -c '%f')
+#echo $free_blocks0 $free_blocks1 $free_blocks2 $free_blocks3 $free_blocks4
 
-_within_tolerance "free blocks after reflinking" $FREE_BLOCKS1 $((FREE_BLOCKS0 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after reflinking" $free_blocks1 $((free_blocks0 - blks)) $margin -v
 
-_within_tolerance "free blocks after partially CoWing some copies" $FREE_BLOCKS2 $((FREE_BLOCKS1 - (2 * BLKS))) $MARGIN -v
+_within_tolerance "free blocks after partially CoWing some copies" $free_blocks2 $((free_blocks1 - (2 * blks))) $margin -v
 
-_within_tolerance "free blocks after CoWing all copies" $FREE_BLOCKS3 $((FREE_BLOCKS2 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after CoWing all copies" $free_blocks3 $((free_blocks2 - blks)) $margin -v
 
-_within_tolerance "free blocks after overwriting original" $FREE_BLOCKS4 $FREE_BLOCKS3 $MARGIN -v
+_within_tolerance "free blocks after overwriting original" $free_blocks4 $free_blocks3 $margin -v
 
-_within_tolerance "free blocks after all tests" $FREE_BLOCKS4 $((FREE_BLOCKS0 - (4 * BLKS))) $MARGIN -v
+_within_tolerance "free blocks after all tests" $free_blocks4 $((free_blocks0 - (4 * blks))) $margin -v
 
 # success, all done
 status=0
diff --git a/tests/generic/155 b/tests/generic/155
index 2551295..611cf1a 100755
--- a/tests/generic/155
+++ b/tests/generic/155
@@ -42,7 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -58,56 +58,56 @@ _require_xfs_io_command "fzero"
 
 rm -f "$seqres.full"
 
-TESTDIR=$TEST_DIR/test-$seq
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir=$TEST_DIR/test-$seq
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-BLKS=2000
-MARGIN=100
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+blks=2000
+margin=100
+sz=$((blksz * blks))
+free_blocks0=$(stat -f "$testdir" -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
 
 echo "Create the reflink copies"
-for i in `seq 2 $NR`; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file$i"
+for i in `seq 2 $nr`; do
+	_cp_reflink "$testdir/file1" "$testdir/file$i"
 done
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
+free_blocks1=$(stat -f "$testdir" -c '%f')
 
 echo "Rewrite some of the blocks"
-"$XFS_IO_PROG" -f -c "fzero 0 $SZ" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x63 0 $((SZ / 2)) "$TESTDIR/file3" -d >> "$seqres.full"
-_mwrite_byte 0x64 $((SZ / 2)) $((SZ / 2)) $SZ "$TESTDIR/file4" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "fzero 0 $sz" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x63 0 $((sz / 2)) "$testdir/file3" -d >> "$seqres.full"
+_mwrite_byte 0x64 $((sz / 2)) $((sz / 2)) $sz "$testdir/file4" >> "$seqres.full"
 _test_remount
-FREE_BLOCKS2=$(stat -f "$TESTDIR" -c '%f')
+free_blocks2=$(stat -f "$testdir" -c '%f')
 
 echo "Rewrite all the files"
-_pwrite_byte 0x62 0 $SZ "$TESTDIR/file2" -d >> "$seqres.full"
-_mwrite_byte 0x63 0 $SZ $SZ "$TESTDIR/file3" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "fzero 0 $SZ" $TESTDIR/file4 >> "$seqres.full"
+_pwrite_byte 0x62 0 $sz "$testdir/file2" -d >> "$seqres.full"
+_mwrite_byte 0x63 0 $sz $sz "$testdir/file3" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "fzero 0 $sz" $testdir/file4 >> "$seqres.full"
 _test_remount
-FREE_BLOCKS3=$(stat -f "$TESTDIR" -c '%f')
+free_blocks3=$(stat -f "$testdir" -c '%f')
 
 echo "Rewrite the original file"
-_pwrite_byte 0x65 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x65 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
-FREE_BLOCKS4=$(stat -f "$TESTDIR" -c '%f')
-#echo $FREE_BLOCKS0 $FREE_BLOCKS1 $FREE_BLOCKS2 $FREE_BLOCKS3 $FREE_BLOCKS4
+free_blocks4=$(stat -f "$testdir" -c '%f')
+#echo $free_blocks0 $free_blocks1 $free_blocks2 $free_blocks3 $free_blocks4
 
-_within_tolerance "free blocks after reflinking" $FREE_BLOCKS1 $((FREE_BLOCKS0 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after reflinking" $free_blocks1 $((free_blocks0 - blks)) $margin -v
 
-_within_tolerance "free blocks after partially CoWing some copies" $FREE_BLOCKS2 $((FREE_BLOCKS1 - (2 * BLKS))) $MARGIN -v
+_within_tolerance "free blocks after partially CoWing some copies" $free_blocks2 $((free_blocks1 - (2 * blks))) $margin -v
 
-_within_tolerance "free blocks after CoWing all copies" $FREE_BLOCKS3 $((FREE_BLOCKS2 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after CoWing all copies" $free_blocks3 $((free_blocks2 - blks)) $margin -v
 
-_within_tolerance "free blocks after overwriting original" $FREE_BLOCKS4 $FREE_BLOCKS3 $MARGIN -v
+_within_tolerance "free blocks after overwriting original" $free_blocks4 $free_blocks3 $margin -v
 
-_within_tolerance "free blocks after all tests" $FREE_BLOCKS4 $((FREE_BLOCKS0 - (4 * BLKS))) $MARGIN -v
+_within_tolerance "free blocks after all tests" $free_blocks4 $((free_blocks0 - (4 * blks))) $margin -v
 
 # success, all done
 status=0
diff --git a/tests/generic/156 b/tests/generic/156
index 78b7e8c..7194e87 100755
--- a/tests/generic/156
+++ b/tests/generic/156
@@ -45,7 +45,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -66,58 +66,58 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-BLKS=2000
-MARGIN=100
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+blks=2000
+margin=100
+sz=$((blksz * blks))
+free_blocks0=$(stat -f "$testdir" -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
 
 echo "Create the reflink copies"
-for i in `seq 2 $NR`; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file$i"
+for i in `seq 2 $nr`; do
+	_cp_reflink "$testdir/file1" "$testdir/file$i"
 done
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
+free_blocks1=$(stat -f "$testdir" -c '%f')
 
 echo "funshare part of a file"
-"$XFS_IO_PROG" -f -c "falloc 0 $((SZ / 2))" "$TESTDIR/file2"
+"$XFS_IO_PROG" -f -c "falloc 0 $((sz / 2))" "$testdir/file2"
 _test_remount
 
 echo "funshare some of the copies"
-"$XFS_IO_PROG" -f -c "falloc 0 $SZ" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "falloc 0 $SZ" "$TESTDIR/file3"
+"$XFS_IO_PROG" -f -c "falloc 0 $sz" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "falloc 0 $sz" "$testdir/file3"
 _test_remount
-FREE_BLOCKS2=$(stat -f "$TESTDIR" -c '%f')
+free_blocks2=$(stat -f "$testdir" -c '%f')
 
 echo "funshare the rest of the files"
-"$XFS_IO_PROG" -f -c "falloc 0 $SZ" "$TESTDIR/file4"
-"$XFS_IO_PROG" -f -c "falloc 0 $SZ" "$TESTDIR/file1"
+"$XFS_IO_PROG" -f -c "falloc 0 $sz" "$testdir/file4"
+"$XFS_IO_PROG" -f -c "falloc 0 $sz" "$testdir/file1"
 _test_remount
-FREE_BLOCKS3=$(stat -f "$TESTDIR" -c '%f')
+free_blocks3=$(stat -f "$testdir" -c '%f')
 
 echo "Rewrite the original file"
-_pwrite_byte 0x65 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x65 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
-FREE_BLOCKS4=$(stat -f "$TESTDIR" -c '%f')
-#echo $FREE_BLOCKS0 $FREE_BLOCKS1 $FREE_BLOCKS2 $FREE_BLOCKS3 $FREE_BLOCKS4
+free_blocks4=$(stat -f "$testdir" -c '%f')
+#echo $free_blocks0 $free_blocks1 $free_blocks2 $free_blocks3 $free_blocks4
 
-_within_tolerance "free blocks after reflinking" $FREE_BLOCKS1 $((FREE_BLOCKS0 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after reflinking" $free_blocks1 $((free_blocks0 - blks)) $margin -v
 
-_within_tolerance "free blocks after nocow'ing some copies" $FREE_BLOCKS2 $((FREE_BLOCKS1 - (2 * BLKS))) $MARGIN -v
+_within_tolerance "free blocks after nocow'ing some copies" $free_blocks2 $((free_blocks1 - (2 * blks))) $margin -v
 
-_within_tolerance "free blocks after nocow'ing all copies" $FREE_BLOCKS3 $((FREE_BLOCKS2 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after nocow'ing all copies" $free_blocks3 $((free_blocks2 - blks)) $margin -v
 
-_within_tolerance "free blocks after overwriting original" $FREE_BLOCKS4 $FREE_BLOCKS3 $MARGIN -v
+_within_tolerance "free blocks after overwriting original" $free_blocks4 $free_blocks3 $margin -v
 
-_within_tolerance "free blocks after all tests" $FREE_BLOCKS4 $((FREE_BLOCKS0 - (4 * BLKS))) $MARGIN -v
+_within_tolerance "free blocks after all tests" $free_blocks4 $((free_blocks0 - (4 * blks))) $margin -v
 
 # success, all done
 status=0
diff --git a/tests/generic/157 b/tests/generic/157
index 67f3ea5..a5b84fc 100755
--- a/tests/generic/157
+++ b/tests/generic/157
@@ -32,7 +32,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -52,31 +52,31 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR1="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR1"
-mkdir "$TESTDIR1"
+testdir1="$TEST_DIR/test-$seq"
+rm -rf "$testdir1"
+mkdir "$testdir1"
 
-TESTDIR2=$SCRATCH_MNT/test-$seq
-rm -rf "$TESTDIR2"
-mkdir "$TESTDIR2"
+testdir2=$SCRATCH_MNT/test-$seq
+rm -rf "$testdir2"
+mkdir "$testdir2"
 
 echo "Create the original files"
-BLKSZ="$(stat -f $TESTDIR1 -c '%S')"
-BLKS=1000
-MARGIN=50
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f $TESTDIR1 -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file2" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR2/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR2/file2" >> "$seqres.full"
-mkdir "$TESTDIR1/dir1"
-seq 1 $((2 * BLKSZ / 250)) | while read f; do
-	touch "$TESTDIR1/dir1/$f"
+blksz="$(stat -f $testdir1 -c '%S')"
+blks=1000
+margin=50
+sz=$((blksz * blks))
+free_blocks0=$(stat -f $testdir1 -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir1/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir1/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir2/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir2/file2" >> "$seqres.full"
+mkdir "$testdir1/dir1"
+seq 1 $((2 * blksz / 250)) | while read f; do
+	touch "$testdir1/dir1/$f"
 done
-mknod "$TESTDIR1/dev1" c 1 3
-mkfifo "$TESTDIR1/fifo1"
+mknod "$testdir1/dev1" c 1 3
+mkfifo "$testdir1/fifo1"
 sync
 
 _filter_enotty() {
@@ -88,38 +88,38 @@ _filter_einval() {
 }
 
 echo "Try cross-device reflink"
-_reflink_range "$TESTDIR1/file1" 0 "$TESTDIR2/file1" 0 $BLKSZ
+_reflink_range "$testdir1/file1" 0 "$testdir2/file1" 0 $blksz
 
 echo "Try unaligned reflink"
-_reflink_range "$TESTDIR1/file1" 37 "$TESTDIR1/file1" 59 23
+_reflink_range "$testdir1/file1" 37 "$testdir1/file1" 59 23
 
 echo "Try overlapping reflink"
-_reflink_range "$TESTDIR1/file1" 0 "$TESTDIR1/file1" 1 $((BLKSZ * 2))
+_reflink_range "$testdir1/file1" 0 "$testdir1/file1" 1 $((blksz * 2))
 
 echo "Try reflink past EOF"
-_reflink_range "$TESTDIR1/file1" $(( (BLKS + 10) * BLKSZ)) "$TESTDIR1/file1" 0 $BLKSZ
+_reflink_range "$testdir1/file1" $(( (blks + 10) * blksz)) "$testdir1/file1" 0 $blksz
 
 echo "Try to reflink a dir"
-_reflink_range "$TESTDIR1/dir1" 0 "$TESTDIR1/file2" 0 $BLKSZ
+_reflink_range "$testdir1/dir1" 0 "$testdir1/file2" 0 $blksz
 
 echo "Try to reflink a device"
-_reflink_range "$TESTDIR1/dev1" 0 "$TESTDIR1/file2" 0 $BLKSZ
+_reflink_range "$testdir1/dev1" 0 "$testdir1/file2" 0 $blksz
 
 echo "Try to reflink to a dir"
-_reflink_range "$TESTDIR1/file1" 0 "$TESTDIR1/dir1" 0 $BLKSZ 2>&1 | _filter_test_dir
+_reflink_range "$testdir1/file1" 0 "$testdir1/dir1" 0 $blksz 2>&1 | _filter_test_dir
 
 echo "Try to reflink to a device"
-_reflink_range "$TESTDIR1/file1" 0 "$TESTDIR1/dev1" 0 $BLKSZ 2>&1 | _filter_enotty
+_reflink_range "$testdir1/file1" 0 "$testdir1/dev1" 0 $blksz 2>&1 | _filter_enotty
 
 echo "Try to reflink to a fifo"
-_reflink_range "$TESTDIR1/file1" 0 "$TESTDIR1/fifo1" 0 $BLKSZ -n 2>&1 | _filter_enotty
+_reflink_range "$testdir1/file1" 0 "$testdir1/fifo1" 0 $blksz -n 2>&1 | _filter_enotty
 
 echo "Try to reflink an append-only file"
-_reflink_range "$TESTDIR1/file1" 0 "$TESTDIR1/file3" 0 $BLKSZ -a 2>&1 | _filter_einval
+_reflink_range "$testdir1/file1" 0 "$testdir1/file3" 0 $blksz -a 2>&1 | _filter_einval
 
 echo "Reflink two files"
-_reflink_range "$TESTDIR1/file1" 0 "$TESTDIR1/file2" 0 $BLKSZ >> "$seqres.full"
-_reflink_range "$TESTDIR2/file1" 0 "$TESTDIR2/file2" 0 $BLKSZ >> "$seqres.full"
+_reflink_range "$testdir1/file1" 0 "$testdir1/file2" 0 $blksz >> "$seqres.full"
+_reflink_range "$testdir2/file1" 0 "$testdir2/file2" 0 $blksz >> "$seqres.full"
 
 echo "Check scratch fs"
 _scratch_unmount
diff --git a/tests/generic/158 b/tests/generic/158
index 18bda38..58b7090 100755
--- a/tests/generic/158
+++ b/tests/generic/158
@@ -32,7 +32,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -52,32 +52,32 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR1="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR1"
-mkdir "$TESTDIR1"
+testdir1="$TEST_DIR/test-$seq"
+rm -rf "$testdir1"
+mkdir "$testdir1"
 
-TESTDIR2=$SCRATCH_MNT/test-$seq
-rm -rf "$TESTDIR2"
-mkdir "$TESTDIR2"
+testdir2=$SCRATCH_MNT/test-$seq
+rm -rf "$testdir2"
+mkdir "$testdir2"
 
 echo "Create the original files"
-BLKSZ="$(stat -f $TESTDIR1 -c '%S')"
-BLKS=1000
-MARGIN=50
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f $TESTDIR1 -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file2" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file3" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR2/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR2/file2" >> "$seqres.full"
-mkdir "$TESTDIR1/dir1"
-seq 1 $((2 * BLKSZ / 250)) | while read f; do
-	touch "$TESTDIR1/dir1/$f"
+blksz="$(stat -f $testdir1 -c '%S')"
+blks=1000
+margin=50
+sz=$((blksz * blks))
+free_blocks0=$(stat -f $testdir1 -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir1/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir1/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir1/file3" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir2/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir2/file2" >> "$seqres.full"
+mkdir "$testdir1/dir1"
+seq 1 $((2 * blksz / 250)) | while read f; do
+	touch "$testdir1/dir1/$f"
 done
-mknod "$TESTDIR1/dev1" c 1 3
-mkfifo "$TESTDIR1/fifo1"
+mknod "$testdir1/dev1" c 1 3
+mkfifo "$testdir1/fifo1"
 sync
 
 _filter_enotty() {
@@ -89,38 +89,38 @@ _filter_eperm() {
 }
 
 echo "Try cross-device dedupe"
-_dedupe_range "$TESTDIR1/file1" 0 "$TESTDIR2/file1" 0 $BLKSZ
+_dedupe_range "$testdir1/file1" 0 "$testdir2/file1" 0 $blksz
 
 echo "Try unaligned dedupe"
-_dedupe_range "$TESTDIR1/file1" 37 "$TESTDIR1/file1" 59 23
+_dedupe_range "$testdir1/file1" 37 "$testdir1/file1" 59 23
 
 echo "Try overlapping dedupe"
-_dedupe_range "$TESTDIR1/file1" 0 "$TESTDIR1/file1" 1 $((BLKSZ * 2))
+_dedupe_range "$testdir1/file1" 0 "$testdir1/file1" 1 $((blksz * 2))
 
 echo "Try dedupe past EOF"
-_dedupe_range "$TESTDIR1/file1" $(( (BLKS + 10) * BLKSZ)) "$TESTDIR1/file1" 0 $BLKSZ
+_dedupe_range "$testdir1/file1" $(( (blks + 10) * blksz)) "$testdir1/file1" 0 $blksz
 
 echo "Try to dedupe a dir"
-_dedupe_range "$TESTDIR1/dir1" 0 "$TESTDIR1/file2" 0 $BLKSZ
+_dedupe_range "$testdir1/dir1" 0 "$testdir1/file2" 0 $blksz
 
 echo "Try to dedupe a device"
-_dedupe_range "$TESTDIR1/dev1" 0 "$TESTDIR1/file2" 0 $BLKSZ 2>&1 | _filter_enotty
+_dedupe_range "$testdir1/dev1" 0 "$testdir1/file2" 0 $blksz 2>&1 | _filter_enotty
 
 echo "Try to dedupe to a dir"
-_dedupe_range "$TESTDIR1/file1" 0 "$TESTDIR1/dir1" 0 $BLKSZ 2>&1 | _filter_test_dir
+_dedupe_range "$testdir1/file1" 0 "$testdir1/dir1" 0 $blksz 2>&1 | _filter_test_dir
 
 echo "Try to dedupe to a device"
-_dedupe_range "$TESTDIR1/file1" 0 "$TESTDIR1/dev1" 0 $BLKSZ 2>&1 | _filter_eperm
+_dedupe_range "$testdir1/file1" 0 "$testdir1/dev1" 0 $blksz 2>&1 | _filter_eperm
 
 echo "Try to dedupe to a fifo"
-_dedupe_range "$TESTDIR1/file1" 0 "$TESTDIR1/fifo1" 0 $BLKSZ -n 2>&1 | _filter_eperm
+_dedupe_range "$testdir1/file1" 0 "$testdir1/fifo1" 0 $blksz -n 2>&1 | _filter_eperm
 
 echo "Try to dedupe an append-only file"
-_dedupe_range "$TESTDIR1/file1" 0 "$TESTDIR1/file3" 0 $BLKSZ -a >> "$seqres.full"
+_dedupe_range "$testdir1/file1" 0 "$testdir1/file3" 0 $blksz -a >> "$seqres.full"
 
 echo "Dedupe two files"
-_dedupe_range "$TESTDIR1/file1" 0 "$TESTDIR1/file2" 0 $BLKSZ >> "$seqres.full"
-_dedupe_range "$TESTDIR2/file1" 0 "$TESTDIR2/file2" 0 $BLKSZ >> "$seqres.full"
+_dedupe_range "$testdir1/file1" 0 "$testdir1/file2" 0 $blksz >> "$seqres.full"
+_dedupe_range "$testdir2/file1" 0 "$testdir2/file2" 0 $blksz >> "$seqres.full"
 
 echo "Check scratch fs"
 _scratch_unmount
diff --git a/tests/generic/159 b/tests/generic/159
index 369663d..14baf64 100755
--- a/tests/generic/159
+++ b/tests/generic/159
@@ -32,7 +32,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -49,25 +49,25 @@ _require_test_reflink
 rm -f "$seqres.full"
 
 echo "Format and mount"
-TESTDIR1="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR1"
-mkdir "$TESTDIR1"
+testdir1="$TEST_DIR/test-$seq"
+rm -rf "$testdir1"
+mkdir "$testdir1"
 
 echo "Create the original files"
-BLKSZ="$(stat -f $TESTDIR1 -c '%S')"
-BLKS=1000
-MARGIN=50
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f $TESTDIR1 -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file2" >> "$seqres.full"
+blksz="$(stat -f $testdir1 -c '%S')"
+blks=1000
+margin=50
+sz=$((blksz * blks))
+free_blocks0=$(stat -f $testdir1 -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir1/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir1/file2" >> "$seqres.full"
 sync
 
 echo "Try reflink on immutable files"
-$CHATTR_PROG +i $TESTDIR1/file1 $TESTDIR1/file2
-_reflink_range "$TESTDIR1/file1" 0 "$TESTDIR1/file2" 0 $BLKSZ 2>&1 | _filter_test_dir
-$CHATTR_PROG -i $TESTDIR1/file1 $TESTDIR1/file2
+$CHATTR_PROG +i $testdir1/file1 $testdir1/file2
+_reflink_range "$testdir1/file1" 0 "$testdir1/file2" 0 $blksz 2>&1 | _filter_test_dir
+$CHATTR_PROG -i $testdir1/file1 $testdir1/file2
 
 # success, all done
 status=0
diff --git a/tests/generic/160 b/tests/generic/160
index 4dfee12..acd2c4e 100755
--- a/tests/generic/160
+++ b/tests/generic/160
@@ -32,7 +32,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -49,25 +49,25 @@ _require_test_dedupe
 rm -f "$seqres.full"
 
 echo "Format and mount"
-TESTDIR1="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR1"
-mkdir "$TESTDIR1"
+testdir1="$TEST_DIR/test-$seq"
+rm -rf "$testdir1"
+mkdir "$testdir1"
 
 echo "Create the original files"
-BLKSZ="$(stat -f $TESTDIR1 -c '%S')"
-BLKS=1000
-MARGIN=50
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f $TESTDIR1 -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR1/file2" >> "$seqres.full"
+blksz="$(stat -f $testdir1 -c '%S')"
+blks=1000
+margin=50
+sz=$((blksz * blks))
+free_blocks0=$(stat -f $testdir1 -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir1/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir1/file2" >> "$seqres.full"
 sync
 
 echo "Try dedupe on immutable files"
-$CHATTR_PROG +i $TESTDIR1/file1 $TESTDIR1/file2
-_dedupe_range "$TESTDIR1/file1" 0 "$TESTDIR1/file2" 0 $BLKSZ 2>&1 | _filter_test_dir
-$CHATTR_PROG -i $TESTDIR1/file1 $TESTDIR1/file2
+$CHATTR_PROG +i $testdir1/file1 $testdir1/file2
+_dedupe_range "$testdir1/file1" 0 "$testdir1/file2" 0 $blksz 2>&1 | _filter_test_dir
+$CHATTR_PROG -i $testdir1/file1 $testdir1/file2
 
 # success, all done
 status=0
diff --git a/tests/generic/161 b/tests/generic/161
index efb137c..7fb8963 100755
--- a/tests/generic/161
+++ b/tests/generic/161
@@ -51,22 +51,22 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=4096
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize files"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2"
 _scratch_remount
 
 echo "Delete while rewriting"
-rm -rf "$TESTDIR/file1" &
-_pwrite_byte 0x62 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
+rm -rf "$testdir/file1" &
+_pwrite_byte 0x62 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
 wait
 
 echo "Check fs"
diff --git a/tests/generic/162 b/tests/generic/162
index 6bd3a33..2fb947a 100755
--- a/tests/generic/162
+++ b/tests/generic/162
@@ -50,24 +50,24 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=512
 nr_loops=$((loops - 1))
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize files"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file2" >> "$seqres.full"
 _scratch_remount
 
 overwrite() {
-	while [ ! -e "$TESTDIR/finished" ]; do
+	while [ ! -e "$testdir/finished" ]; do
 		seq $nr_loops -1 0 | while read i; do
-			_pwrite_byte 0x61 $((i * BLKSZ)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
+			_pwrite_byte 0x61 $((i * blksz)) $blksz "$testdir/file2" >> "$seqres.full"
 		done
 	done
 }
@@ -76,13 +76,13 @@ echo "Dedupe and rewrite the file!"
 overwrite &
 for i in `seq 1 2`; do
 	seq $nr_loops -1 0 | while read i; do
-		_dedupe_range   "$TESTDIR/file1" $((i * BLKSZ)) \
-				"$TESTDIR/file2" $((i * BLKSZ)) $BLKSZ >> "$seqres.full"
+		_dedupe_range   "$testdir/file1" $((i * blksz)) \
+				"$testdir/file2" $((i * blksz)) $blksz >> "$seqres.full"
 		[ $? -ne 0 ] && break
 	done
 done
 echo "Finished dedupeing"
-touch "$TESTDIR/finished"
+touch "$testdir/finished"
 wait
 
 echo "Check fs"
diff --git a/tests/generic/163 b/tests/generic/163
index 1f5042f..0186443 100755
--- a/tests/generic/163
+++ b/tests/generic/163
@@ -50,24 +50,24 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=512
 nr_loops=$((loops - 1))
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize files"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file2" >> "$seqres.full"
 _scratch_remount
 
 overwrite() {
-	while [ ! -e "$TESTDIR/finished" ]; do
+	while [ ! -e "$testdir/finished" ]; do
 		seq $nr_loops -1 0 | while read i; do
-			_pwrite_byte 0x61 $((i * BLKSZ)) $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
+			_pwrite_byte 0x61 $((i * blksz)) $blksz "$testdir/file1" >> "$seqres.full"
 		done
 	done
 }
@@ -76,13 +76,13 @@ echo "Dedupe and rewrite the file!"
 overwrite &
 for i in `seq 1 2`; do
 	seq $nr_loops -1 0 | while read i; do
-		_dedupe_range   "$TESTDIR/file1" $((i * BLKSZ)) \
-				"$TESTDIR/file2" $((i * BLKSZ)) $BLKSZ >> "$seqres.full"
+		_dedupe_range   "$testdir/file1" $((i * blksz)) \
+				"$testdir/file2" $((i * blksz)) $blksz >> "$seqres.full"
 		[ $? -ne 0 ] && break
 	done
 done
 echo "Finished dedupeing"
-touch "$TESTDIR/finished"
+touch "$testdir/finished"
 wait
 
 echo "Check fs"
diff --git a/tests/generic/164 b/tests/generic/164
index 4df953d..087c6ba 100755
--- a/tests/generic/164
+++ b/tests/generic/164
@@ -52,19 +52,19 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=512
 nr_loops=$((loops - 1))
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize files"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((loops * BLKSZ)) "$TESTDIR/file2" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file3"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((loops * blksz)) "$testdir/file2" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file3"
 _scratch_remount
 
 fbytes() {
@@ -72,8 +72,8 @@ fbytes() {
 }
 
 reader() {
-	while [ ! -e "$TESTDIR/finished" ]; do
-		_read_range "$TESTDIR/file3" 0 $((loops * BLKSZ)) | fbytes
+	while [ ! -e "$testdir/finished" ]; do
+		_read_range "$testdir/file3" 0 $((loops * blksz)) | fbytes
 	done
 }
 
@@ -81,18 +81,18 @@ echo "Reflink and reread the files!"
 reader &
 for i in `seq 1 2`; do
 	seq $nr_loops -1 0 | while read i; do
-		_reflink_range  "$TESTDIR/file1" $((i * BLKSZ)) \
-				"$TESTDIR/file3" $((i * BLKSZ)) $BLKSZ >> "$seqres.full"
+		_reflink_range  "$testdir/file1" $((i * blksz)) \
+				"$testdir/file3" $((i * blksz)) $blksz >> "$seqres.full"
 		[ $? -ne 0 ] && break
 	done
 	seq $nr_loops -1 0 | while read i; do
-		_reflink_range  "$TESTDIR/file2" $((i * BLKSZ)) \
-				"$TESTDIR/file3" $((i * BLKSZ)) $BLKSZ >> "$seqres.full"
+		_reflink_range  "$testdir/file2" $((i * blksz)) \
+				"$testdir/file3" $((i * blksz)) $blksz >> "$seqres.full"
 		[ $? -ne 0 ] && break
 	done
 done
 echo "Finished reflinking"
-touch "$TESTDIR/finished"
+touch "$testdir/finished"
 wait
 
 echo "Check fs"
diff --git a/tests/generic/165 b/tests/generic/165
index 49187ec..6bd15e1 100755
--- a/tests/generic/165
+++ b/tests/generic/165
@@ -52,19 +52,19 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=512
 nr_loops=$((loops - 1))
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize files"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((loops * BLKSZ)) "$TESTDIR/file2" >> "$seqres.full"
-_cp_reflink $TESTDIR/file1 $TESTDIR/file3
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((loops * blksz)) "$testdir/file2" >> "$seqres.full"
+_cp_reflink $testdir/file1 $testdir/file3
 _scratch_remount
 
 fbytes() {
@@ -72,8 +72,8 @@ fbytes() {
 }
 
 reader() {
-	while [ ! -e "$TESTDIR/finished" ]; do
-		_read_range "$TESTDIR/file3" 0 $((loops * BLKSZ)) -d | fbytes
+	while [ ! -e "$testdir/finished" ]; do
+		_read_range "$testdir/file3" 0 $((loops * blksz)) -d | fbytes
 	done
 }
 
@@ -81,18 +81,18 @@ echo "Reflink and dio reread the files!"
 reader &
 for i in `seq 1 2`; do
 	seq $nr_loops -1 0 | while read i; do
-		_reflink_range  "$TESTDIR/file1" $((i * BLKSZ)) \
-				"$TESTDIR/file3" $((i * BLKSZ)) $BLKSZ >> "$seqres.full"
+		_reflink_range  "$testdir/file1" $((i * blksz)) \
+				"$testdir/file3" $((i * blksz)) $blksz >> "$seqres.full"
 		[ $? -ne 0 ] && break
 	done
 	seq $nr_loops -1 0 | while read i; do
-		_reflink_range  "$TESTDIR/file2" $((i * BLKSZ)) \
-				"$TESTDIR/file3" $((i * BLKSZ)) $BLKSZ >> "$seqres.full"
+		_reflink_range  "$testdir/file2" $((i * blksz)) \
+				"$testdir/file3" $((i * blksz)) $blksz >> "$seqres.full"
 		[ $? -ne 0 ] && break
 	done
 done
 echo "Finished reflinking"
-touch "$TESTDIR/finished"
+touch "$testdir/finished"
 wait
 
 echo "Check fs"
diff --git a/tests/generic/166 b/tests/generic/166
index 93c4bc9..6cfb821 100755
--- a/tests/generic/166
+++ b/tests/generic/166
@@ -52,24 +52,24 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=1024
 nr_loops=$((loops - 1))
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize file"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
 _scratch_remount
 
 # Snapshot creator...
 snappy() {
 	n=0
-	while [ ! -e "$TESTDIR/finished" ]; do
-		_cp_reflink "$TESTDIR/file1" "$TESTDIR/snap_$n" || break
+	while [ ! -e "$testdir/finished" ]; do
+		_cp_reflink "$testdir/file1" "$testdir/snap_$n" || break
 		n=$((n + 1))
 	done
 }
@@ -77,9 +77,9 @@ snappy() {
 echo "Snapshot a file undergoing directio rewrite"
 snappy &
 seq $nr_loops -1 0 | while read i; do
-	_pwrite_byte 0x63 $((i * BLKSZ)) $BLKSZ -d "$TESTDIR/file1" >> "$seqres.full"
+	_pwrite_byte 0x63 $((i * blksz)) $blksz -d "$testdir/file1" >> "$seqres.full"
 done
-touch $TESTDIR/finished
+touch $testdir/finished
 wait
 
 echo "Check for damage"
diff --git a/tests/generic/167 b/tests/generic/167
index 6d70737..fc5a86c 100755
--- a/tests/generic/167
+++ b/tests/generic/167
@@ -52,24 +52,24 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=1024
 nr_loops=$((loops - 1))
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize file"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
 _scratch_remount
 
 # Snapshot creator...
 snappy() {
 	n=0
-	while [ ! -e "$TESTDIR/finished" ]; do
-		_cp_reflink "$TESTDIR/file1" "$TESTDIR/snap_$n" || break
+	while [ ! -e "$testdir/finished" ]; do
+		_cp_reflink "$testdir/file1" "$testdir/snap_$n" || break
 		n=$((n + 1))
 	done
 }
@@ -77,9 +77,9 @@ snappy() {
 echo "Snapshot a file undergoing buffered rewrite"
 snappy &
 seq $nr_loops -1 0 | while read i; do
-	_pwrite_byte 0x63 $((i * BLKSZ)) $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
+	_pwrite_byte 0x63 $((i * blksz)) $blksz "$testdir/file1" >> "$seqres.full"
 done
-touch $TESTDIR/finished
+touch $testdir/finished
 wait
 
 echo "Check for damage"
diff --git a/tests/generic/168 b/tests/generic/168
index 414dcb0..ee3848d 100755
--- a/tests/generic/168
+++ b/tests/generic/168
@@ -51,25 +51,25 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=1024
 nr_loops=$((loops - 1))
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize files"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((loops * BLKSZ)) "$TESTDIR/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((loops * blksz)) "$testdir/file2" >> "$seqres.full"
 _scratch_remount
 
 # Direct I/O overwriter...
 overwrite() {
-	while [ ! -e "$TESTDIR/finished" ]; do
+	while [ ! -e "$testdir/finished" ]; do
 		seq $nr_loops -1 0 | while read i; do
-			_pwrite_byte 0x63 $((i * BLKSZ)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
+			_pwrite_byte 0x63 $((i * blksz)) $blksz "$testdir/file2" >> "$seqres.full"
 		done
 	done
 }
@@ -78,12 +78,12 @@ echo "Reflink and write the target"
 overwrite &
 seq 1 10 | while read j; do
 	seq 0 $nr_loops | while read i; do
-		_reflink_range  "$TESTDIR/file1" $((i * BLKSZ)) \
-				"$TESTDIR/file2" $((i * BLKSZ)) $BLKSZ >> "$seqres.full"
+		_reflink_range  "$testdir/file1" $((i * blksz)) \
+				"$testdir/file2" $((i * blksz)) $blksz >> "$seqres.full"
 		[ $? -ne 0 ] && exit
 	done
 done
-touch "$TESTDIR/finished"
+touch "$testdir/finished"
 wait
 
 echo "Check for damage"
diff --git a/tests/generic/170 b/tests/generic/170
index 704b646..6d27810 100755
--- a/tests/generic/170
+++ b/tests/generic/170
@@ -51,25 +51,25 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 loops=1024
 nr_loops=$((loops - 1))
-BLKSZ=65536
+blksz=65536
 
 echo "Initialize files"
 echo > "$seqres.full"
-_pwrite_byte 0x61 0 $((loops * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((loops * BLKSZ)) "$TESTDIR/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((loops * blksz)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((loops * blksz)) "$testdir/file2" >> "$seqres.full"
 _scratch_remount
 
 # Direct I/O overwriter...
 overwrite() {
-	while [ ! -e "$TESTDIR/finished" ]; do
+	while [ ! -e "$testdir/finished" ]; do
 		seq $nr_loops -1 0 | while read i; do
-			_pwrite_byte 0x63 $((i * BLKSZ)) $BLKSZ -d "$TESTDIR/file2" >> "$seqres.full"
+			_pwrite_byte 0x63 $((i * blksz)) $blksz -d "$testdir/file2" >> "$seqres.full"
 		done
 	done
 }
@@ -78,12 +78,12 @@ echo "Reflink and dio write the target"
 overwrite &
 seq 1 10 | while read j; do
 	seq 0 $nr_loops | while read i; do
-		_reflink_range  "$TESTDIR/file1" $((i * BLKSZ)) \
-				"$TESTDIR/file2" $((i * BLKSZ)) $BLKSZ >> "$seqres.full"
+		_reflink_range  "$testdir/file1" $((i * blksz)) \
+				"$testdir/file2" $((i * blksz)) $blksz >> "$seqres.full"
 		[ $? -ne 0 ] && exit
 	done
 done
-touch "$TESTDIR/finished"
+touch "$testdir/finished"
 wait
 
 echo "Check for damage"
diff --git a/tests/generic/171 b/tests/generic/171
index 99e4d5e..ec3729d 100755
--- a/tests/generic/171
+++ b/tests/generic/171
@@ -33,7 +33,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -53,36 +53,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Reformat with appropriate size"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-NR_BLKS=10240
+blksz="$(stat -f "$testdir" -c '%S')"
+nr_blks=10240
 umount "$SCRATCH_MNT"
-SZ_BYTES=$((NR_BLKS * 8 * BLKSZ))
-if [ $SZ_BYTES -lt $((32 * 1048576)) ]; then
-	SZ_BYTES=$((32 * 1048576))
+sz_bytes=$((nr_blks * 8 * blksz))
+if [ $sz_bytes -lt $((32 * 1048576)) ]; then
+	sz_bytes=$((32 * 1048576))
 fi
-_scratch_mkfs_sized $SZ_BYTES >> "$seqres.full" 2>&1
+_scratch_mkfs_sized $sz_bytes >> "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create a big file and reflink it"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" >> "$seqres.full" 2>&1
-_cp_reflink "$TESTDIR/bigfile" "$TESTDIR/clonefile"
+_pwrite_byte 0x61 0 $((blksz * nr_blks)) "$testdir/bigfile" >> "$seqres.full" 2>&1
+_cp_reflink "$testdir/bigfile" "$testdir/clonefile"
 sync
 
 echo "Allocate the rest of the space"
-NR_FREE="$(stat -f -c '%f' "$TESTDIR")"
-touch "$TESTDIR/file0" "$TESTDIR/file1"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_FREE)) "$TESTDIR/eat_my_space" >> "$seqres.full" 2>&1
+nr_free="$(stat -f -c '%f' "$testdir")"
+touch "$testdir/file0" "$testdir/file1"
+_pwrite_byte 0x61 0 $((blksz * nr_free)) "$testdir/eat_my_space" >> "$seqres.full" 2>&1
 sync
 
 echo "CoW the big file"
-out="$(_pwrite_byte 0x62 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" 2>&1)"
+out="$(_pwrite_byte 0x62 0 $((blksz * nr_blks)) "$testdir/bigfile" 2>&1)"
 echo "${out}" | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC"
 echo "${out}" >> "$seqres.full" 2>&1
 echo "${out}"
@@ -90,13 +90,13 @@ echo "${out}"
 echo "Remount and try CoW again"
 _scratch_remount
 
-out="$(_pwrite_byte 0x62 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" 2>&1)"
+out="$(_pwrite_byte 0x62 0 $((blksz * nr_blks)) "$testdir/bigfile" 2>&1)"
 echo "${out}" | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC"
 echo "${out}" >> "$seqres.full" 2>&1
 echo "${out}"
 
-#filefrag -v $TESTDIR/bigfile
-#filefrag -v $TESTDIR/clonefile
+#filefrag -v $testdir/bigfile
+#filefrag -v $testdir/clonefile
 
 echo "Check scratch fs"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/172 b/tests/generic/172
index eb00f0a..1988c8d 100755
--- a/tests/generic/172
+++ b/tests/generic/172
@@ -33,7 +33,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -53,36 +53,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Reformat with appropriate size"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-NR_BLKS=10240
+blksz="$(stat -f "$testdir" -c '%S')"
+nr_blks=10240
 umount "$SCRATCH_MNT"
-SZ_BYTES=$((NR_BLKS * 3 / 2 * BLKSZ))
-if [ $SZ_BYTES -lt $((32 * 1048576)) ]; then
-	SZ_BYTES=$((32 * 1048576))
+sz_bytes=$((nr_blks * 3 / 2 * blksz))
+if [ $sz_bytes -lt $((32 * 1048576)) ]; then
+	sz_bytes=$((32 * 1048576))
 fi
-_scratch_mkfs_sized $SZ_BYTES >> "$seqres.full" 2>&1
+_scratch_mkfs_sized $sz_bytes >> "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create a big file and reflink it"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" >> "$seqres.full" 2>&1
-_cp_reflink "$TESTDIR/bigfile" "$TESTDIR/clonefile"
+_pwrite_byte 0x61 0 $((blksz * nr_blks)) "$testdir/bigfile" >> "$seqres.full" 2>&1
+_cp_reflink "$testdir/bigfile" "$testdir/clonefile"
 sync
 
 echo "Allocate the rest of the space"
-NR_FREE="$(stat -f -c '%f' "$TESTDIR")"
-touch "$TESTDIR/file0" "$TESTDIR/file1"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_FREE)) "$TESTDIR/eat_my_space" >> "$seqres.full" 2>&1
+nr_free="$(stat -f -c '%f' "$testdir")"
+touch "$testdir/file0" "$testdir/file1"
+_pwrite_byte 0x61 0 $((blksz * nr_free)) "$testdir/eat_my_space" >> "$seqres.full" 2>&1
 sync
 
 echo "CoW the big file"
-out="$(_pwrite_byte 0x62 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" 2>&1)"
+out="$(_pwrite_byte 0x62 0 $((blksz * nr_blks)) "$testdir/bigfile" 2>&1)"
 echo "${out}" | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC"
 echo "${out}" >> "$seqres.full" 2>&1
 echo "${out}"
@@ -90,13 +90,13 @@ echo "${out}"
 echo "Remount and try CoW again"
 _scratch_remount
 
-out="$(_pwrite_byte 0x62 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" 2>&1)"
+out="$(_pwrite_byte 0x62 0 $((blksz * nr_blks)) "$testdir/bigfile" 2>&1)"
 echo "${out}" | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC"
 echo "${out}" >> "$seqres.full" 2>&1
 echo "${out}"
 
-#filefrag -v $TESTDIR/bigfile
-#filefrag -v $TESTDIR/clonefile
+#filefrag -v $testdir/bigfile
+#filefrag -v $testdir/clonefile
 
 echo "Check scratch fs"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/173 b/tests/generic/173
index 76463d2..5bb9ce6 100755
--- a/tests/generic/173
+++ b/tests/generic/173
@@ -33,7 +33,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -53,36 +53,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Reformat with appropriate size"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-NR_BLKS=10240
+blksz="$(stat -f "$testdir" -c '%S')"
+nr_blks=10240
 umount "$SCRATCH_MNT"
-SZ_BYTES=$((NR_BLKS * 8 * BLKSZ))
-if [ $SZ_BYTES -lt $((32 * 1048576)) ]; then
-	SZ_BYTES=$((32 * 1048576))
+sz_bytes=$((nr_blks * 8 * blksz))
+if [ $sz_bytes -lt $((32 * 1048576)) ]; then
+	sz_bytes=$((32 * 1048576))
 fi
-_scratch_mkfs_sized $SZ_BYTES >> "$seqres.full" 2>&1
+_scratch_mkfs_sized $sz_bytes >> "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create a big file and reflink it"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" >> "$seqres.full" 2>&1
-_cp_reflink "$TESTDIR/bigfile" "$TESTDIR/clonefile"
+_pwrite_byte 0x61 0 $((blksz * nr_blks)) "$testdir/bigfile" >> "$seqres.full" 2>&1
+_cp_reflink "$testdir/bigfile" "$testdir/clonefile"
 sync
 
 echo "Allocate the rest of the space"
-NR_FREE="$(stat -f -c '%f' "$TESTDIR")"
-touch "$TESTDIR/file0" "$TESTDIR/file1"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_FREE)) "$TESTDIR/eat_my_space" >> "$seqres.full" 2>&1
+nr_free="$(stat -f -c '%f' "$testdir")"
+touch "$testdir/file0" "$testdir/file1"
+_pwrite_byte 0x61 0 $((blksz * nr_free)) "$testdir/eat_my_space" >> "$seqres.full" 2>&1
 sync
 
 echo "mmap CoW the big file"
-out="$(_mwrite_byte 0x62 0 $((BLKSZ * NR_BLKS)) $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" 2>&1)"
+out="$(_mwrite_byte 0x62 0 $((blksz * nr_blks)) $((blksz * nr_blks)) "$testdir/bigfile" 2>&1)"
 err="$?"
 if [ "$err" -lt 128 ]; then
 	echo "mmap CoW should have failed with SIGBUS, got SIG$(kill -l $err)"
@@ -91,14 +91,14 @@ fi
 echo "Remount and try CoW again"
 _scratch_remount
 
-out="$(_mwrite_byte 0x62 0 $((BLKSZ * NR_BLKS)) $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" 2>&1)"
+out="$(_mwrite_byte 0x62 0 $((blksz * nr_blks)) $((blksz * nr_blks)) "$testdir/bigfile" 2>&1)"
 err="$?"
 if [ "$err" -lt 128 ]; then
 	echo "mmap CoW should have failed with SIGBUS, got SIG$(kill -l $err)"
 fi
 
-#filefrag -v $TESTDIR/bigfile
-#filefrag -v $TESTDIR/clonefile
+#filefrag -v $testdir/bigfile
+#filefrag -v $testdir/clonefile
 
 echo "Check scratch fs"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/174 b/tests/generic/174
index 8df292a..a8e0bb5 100755
--- a/tests/generic/174
+++ b/tests/generic/174
@@ -33,7 +33,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -53,36 +53,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Reformat with appropriate size"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-NR_BLKS=10240
+blksz="$(stat -f "$testdir" -c '%S')"
+nr_blks=10240
 umount "$SCRATCH_MNT"
-SZ_BYTES=$((NR_BLKS * 8 * BLKSZ))
-if [ $SZ_BYTES -lt $((32 * 1048576)) ]; then
-	SZ_BYTES=$((32 * 1048576))
+sz_bytes=$((nr_blks * 8 * blksz))
+if [ $sz_bytes -lt $((32 * 1048576)) ]; then
+	sz_bytes=$((32 * 1048576))
 fi
-_scratch_mkfs_sized $SZ_BYTES >> "$seqres.full" 2>&1
+_scratch_mkfs_sized $sz_bytes >> "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create a big file and reflink it"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" >> "$seqres.full" 2>&1
-_cp_reflink "$TESTDIR/bigfile" "$TESTDIR/clonefile"
+_pwrite_byte 0x61 0 $((blksz * nr_blks)) "$testdir/bigfile" >> "$seqres.full" 2>&1
+_cp_reflink "$testdir/bigfile" "$testdir/clonefile"
 sync
 
 echo "Allocate the rest of the space"
-NR_FREE="$(stat -f -c '%f' "$TESTDIR")"
-touch "$TESTDIR/file0" "$TESTDIR/file1"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_FREE)) "$TESTDIR/eat_my_space" >> "$seqres.full" 2>&1
+nr_free="$(stat -f -c '%f' "$testdir")"
+touch "$testdir/file0" "$testdir/file1"
+_pwrite_byte 0x61 0 $((blksz * nr_free)) "$testdir/eat_my_space" >> "$seqres.full" 2>&1
 sync
 
 echo "CoW the big file"
-out="$(_pwrite_byte 0x62 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" -d 2>&1)"
+out="$(_pwrite_byte 0x62 0 $((blksz * nr_blks)) "$testdir/bigfile" -d 2>&1)"
 echo "${out}" | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC"
 echo "${out}" >> "$seqres.full" 2>&1
 echo "${out}"
@@ -90,13 +90,13 @@ echo "${out}"
 echo "Remount and try CoW again"
 _scratch_remount
 
-out="$(_pwrite_byte 0x62 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/bigfile" -d 2>&1)"
+out="$(_pwrite_byte 0x62 0 $((blksz * nr_blks)) "$testdir/bigfile" -d 2>&1)"
 echo "${out}" | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC"
 echo "${out}" >> "$seqres.full" 2>&1
 echo "${out}"
 
-#filefrag -v $TESTDIR/bigfile
-#filefrag -v $TESTDIR/clonefile
+#filefrag -v $testdir/bigfile
+#filefrag -v $testdir/clonefile
 
 echo "Check scratch fs"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/175 b/tests/generic/175
index a7da533..b37ff66 100755
--- a/tests/generic/175
+++ b/tests/generic/175
@@ -34,7 +34,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -54,30 +54,30 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 # Well let's hope the maximum reflink count is (less than (ha!)) 2^32...
 
 echo "Create a one block file"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-_pwrite_byte 0x61 0 $BLKSZ "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+_pwrite_byte 0x61 0 $blksz "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $blksz "$testdir/file2" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2" >> "$seqres.full"
 
 nr=32
 fnr=32
 for i in $(seq 0 $fnr); do
-	echo " ++ Reflink size $i, $(( (2 ** i) * BLKSZ)) bytes" | tee -a "$seqres.full"
-	n=$(( (2 ** i) * BLKSZ))
-	_reflink_range "$TESTDIR/file1" 0 "$TESTDIR/file1" $n $n >> "$seqres.full" || break
+	echo " ++ Reflink size $i, $(( (2 ** i) * blksz)) bytes" | tee -a "$seqres.full"
+	n=$(( (2 ** i) * blksz))
+	_reflink_range "$testdir/file1" 0 "$testdir/file1" $n $n >> "$seqres.full" || break
 done
 
 nrf=$((nr - fnr))
 echo "Clone $((2 ** nrf)) files"
 seq 0 $((2 ** nrf)) | while read i; do
-	_cp-reflink "$TESTDIR/file1" "$TESTDIR/file1-$i"
+	_cp-reflink "$testdir/file1" "$testdir/file1-$i"
 done
 
 echo "Check scratch fs"
diff --git a/tests/generic/176 b/tests/generic/176
index bf930b7..0ca0b17 100755
--- a/tests/generic/176
+++ b/tests/generic/176
@@ -32,7 +32,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR1"
+    rm -rf "$tmp".* "$testdir1"
 }
 
 # get standard environment, filters and checks
@@ -51,23 +51,23 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-NR_FREE="$(stat -f -c '%f' "$TESTDIR")"
+blksz="$(stat -f "$testdir" -c '%S')"
+nr_free="$(stat -f -c '%f' "$testdir")"
 echo "Create a big file"
-touch "$TESTDIR/file0" "$TESTDIR/file1"
-_pwrite_byte 0x61 0 $((BLKSZ * NR_FREE)) "$TESTDIR/bigfile" >> "$seqres.full" 2>&1
+touch "$testdir/file0" "$testdir/file1"
+_pwrite_byte 0x61 0 $((blksz * nr_free)) "$testdir/bigfile" >> "$seqres.full" 2>&1
 _scratch_remount
-sz="$(stat -c '%s' "$TESTDIR/bigfile")"
+sz="$(stat -c '%s' "$testdir/bigfile")"
 
-blks="$((sz / BLKSZ))"
+blks="$((sz / blksz))"
 echo "Try to reflink"
 seq 0 $blks | while read lblk; do
-	fname="$TESTDIR/file$((lblk % 2))"
-	out="$(_reflink_range "$TESTDIR/bigfile" $((lblk * BLKSZ)) "$fname" $((lblk * BLKSZ)) $BLKSZ 2>&1)"
+	fname="$testdir/file$((lblk % 2))"
+	out="$(_reflink_range "$testdir/bigfile" $((lblk * blksz)) "$fname" $((lblk * blksz)) $blksz 2>&1)"
 	echo "$fname: $out" >> "$seqres.full"
 	echo "$out" | grep -q "No space left on device" && break
 done
diff --git a/tests/generic/178 b/tests/generic/178
index 9828dac..f1bc963 100755
--- a/tests/generic/178
+++ b/tests/generic/178
@@ -32,7 +32,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -48,29 +48,29 @@ _require_xfs_io_command "fpunch"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-NR=512
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
+blksz=65536
+nr=512
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
 
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
+_cp_reflink "$testdir/file1" "$testdir/file2"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
 echo "Write and punch"
-_pwrite_byte 0x62 0 $((BLKSZ * 256)) "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "fpunch $BLKSZ $((BLKSZ * 254))" "$TESTDIR/file2"
+_pwrite_byte 0x62 0 $((blksz * 256)) "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "fpunch $blksz $((blksz * 254))" "$testdir/file2"
 _test_remount
 
 echo "Compare results"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
 
 # success, all done
 status=0
diff --git a/tests/generic/179 b/tests/generic/179
index 9312d4d..8e1e059 100755
--- a/tests/generic/179
+++ b/tests/generic/179
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,45 +51,45 @@ _require_xfs_io_command "fpunch"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 3)) "$TESTDIR/file1" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 3)) "$testdir/file1" >> "$seqres.full"
 
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
+_cp_reflink "$testdir/file1" "$testdir/file2"
 
-_pwrite_byte 0x61 0 $((BLKSZ * 3)) "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ - 17)) $((BLKSZ + 17)) "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 3)) "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz - 17)) $((blksz + 17)) "$testdir/file2.chk" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
 
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
 
 echo "fpunch files"
-"$XFS_IO_PROG" -f -c "fpunch $((BLKSZ - 17)) $((BLKSZ + 17))" "$TESTDIR/file2"
+"$XFS_IO_PROG" -f -c "fpunch $((blksz - 17)) $((blksz + 17))" "$testdir/file2"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
 
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/180 b/tests/generic/180
index af6f2ae..dc6cbbb 100755
--- a/tests/generic/180
+++ b/tests/generic/180
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,45 +51,45 @@ _require_xfs_io_command "fzero"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 3)) "$TESTDIR/file1" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 3)) "$testdir/file1" >> "$seqres.full"
 
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
+_cp_reflink "$testdir/file1" "$testdir/file2"
 
-_pwrite_byte 0x61 0 $((BLKSZ * 3)) "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ - 17)) $((BLKSZ + 17)) "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 3)) "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz - 17)) $((blksz + 17)) "$testdir/file2.chk" >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
 
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
 
 echo "fzero files"
-"$XFS_IO_PROG" -f -c "fzero $((BLKSZ - 17)) $((BLKSZ + 17))" "$TESTDIR/file2"
+"$XFS_IO_PROG" -f -c "fzero $((blksz - 17)) $((blksz + 17))" "$testdir/file2"
 _test_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
 
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/181 b/tests/generic/181
index 03c9ccc..cb6639f 100755
--- a/tests/generic/181
+++ b/tests/generic/181
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,30 +50,30 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 256)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * 256)) "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * 2)) "$TESTDIR/file2.chk" >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $((BLKSZ * 255)) "$TESTDIR/file2.chk" >> "$seqres.full"
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) 0 >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 256)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * 256)) "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * 2)) "$testdir/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $((blksz * 255)) "$testdir/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) 0 >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
 
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/182 b/tests/generic/182
index 885159e..bf5cd38 100755
--- a/tests/generic/182
+++ b/tests/generic/182
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,67 +50,67 @@ _require_cp_reflink
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ * 256)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * 257)) "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * 257)) "$TESTDIR/file2.chk" >> "$seqres.full"
-_dedupe_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) 0 >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz * 256)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * 257)) "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * 257)) "$testdir/file2.chk" >> "$seqres.full"
+_dedupe_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) 0 >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
 
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
 
 echo "Make the original file almost dedup-able"
-_pwrite_byte 0x61 0 $((BLKSZ * 256)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ * 256)) "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ * 256)) "$TESTDIR/file2.chk" >> "$seqres.full"
-_dedupe_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) 0 >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 256)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 256)) "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 256)) "$testdir/file2.chk" >> "$seqres.full"
+_dedupe_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) 0 >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
 
-test "${C1}" != "${C2}" || echo "file1 and file2 should not match"
+test "${c1}" != "${c2}" || echo "file1 and file2 should not match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
 
 echo "Make the original file dedup-able"
-_pwrite_byte 0x61 0 $((BLKSZ * 256)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ * 257)) "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 0 $((BLKSZ * 257)) "$TESTDIR/file2.chk" >> "$seqres.full"
-_dedupe_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) 0 >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 256)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 257)) "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $((blksz * 257)) "$testdir/file2.chk" >> "$seqres.full"
+_dedupe_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) 0 >> "$seqres.full"
 _test_remount
 
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-md5sum "$TESTDIR/file2.chk" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file2.chk" | _filter_test_dir
 
-C1="$(_md5_checksum "$TESTDIR/file1")"
-C2="$(_md5_checksum "$TESTDIR/file2")"
+c1="$(_md5_checksum "$testdir/file1")"
+c2="$(_md5_checksum "$testdir/file2")"
 
-test "${C1}" = "${C2}" || echo "file1 and file2 should match"
+test "${c1}" = "${c2}" || echo "file1 and file2 should match"
 
 echo "Compare against check files"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
 
 # success, all done
 status=0
diff --git a/tests/generic/183 b/tests/generic/183
index 2cf9ce4..8141f38 100755
--- a/tests/generic/183
+++ b/tests/generic/183
@@ -38,7 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -56,41 +56,41 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file2" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file2" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
-seq 1 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 1 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file2" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "directio CoW across the transition"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/185 b/tests/generic/185
index 55c54ac..1d6dff2 100755
--- a/tests/generic/185
+++ b/tests/generic/185
@@ -38,7 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -56,41 +56,41 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file2" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file2" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
-seq 1 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 1 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file2" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "CoW across the transition"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/186 b/tests/generic/186
index f1c0e6b..9aac966 100755
--- a/tests/generic/186
+++ b/tests/generic/186
@@ -40,7 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-#    rm -rf "$tmp".* "$TESTDIR"
+#    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -90,8 +90,8 @@ _fragment_freesp()
 	$XFS_IO_PROG -fc "pwrite -S 0x65 0 $avail" "${file}.${i}"
 
 	# Punch out whatever we need
-	seq 1 $((NR * 4)) | while read f; do
-		$XFS_IO_PROG -f -c "fpunch $((f * 2 * BLKSZ)) $BLKSZ" "$file"
+	seq 1 $((nr * 4)) | while read f; do
+		$XFS_IO_PROG -f -c "fpunch $((f * 2 * blksz)) $blksz" "$file"
 	done
 }
 
@@ -99,43 +99,43 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=1024
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file2" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=1024
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file2" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
-seq 1 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 1 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file2" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
-_fragment_freesp "$TESTDIR/bigfile" >> "$seqres.full" 2>&1
+_fragment_freesp "$testdir/bigfile" >> "$seqres.full" 2>&1
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "CoW with multiple extents?"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/187 b/tests/generic/187
index ce35ec0..b3f1542 100755
--- a/tests/generic/187
+++ b/tests/generic/187
@@ -40,7 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-#    rm -rf "$tmp".* "$TESTDIR"
+#    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -90,8 +90,8 @@ _fragment_freesp()
 	$XFS_IO_PROG -fc "pwrite -S 0x65 0 $avail" "${file}.${i}"
 
 	# Punch out whatever we need
-	seq 1 $((NR * 4)) | while read f; do
-		$XFS_IO_PROG -f -c "fpunch $((f * 2 * BLKSZ)) $BLKSZ" "$file"
+	seq 1 $((nr * 4)) | while read f; do
+		$XFS_IO_PROG -f -c "fpunch $((f * 2 * blksz)) $blksz" "$file"
 	done
 }
 
@@ -99,43 +99,43 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=1024
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file2" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=1024
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file2" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
-seq 1 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 1 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file2" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
-_fragment_freesp "$TESTDIR/bigfile" >> "$seqres.full" 2>&1
+_fragment_freesp "$testdir/bigfile" >> "$seqres.full" 2>&1
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "CoW with multiple extents?"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/188 b/tests/generic/188
index 8e715ec..c67b847 100755
--- a/tests/generic/188
+++ b/tests/generic/188
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -56,36 +56,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-$XFS_IO_PROG -f -c "falloc 0 $((BLKSZ * NR))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x00 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+$XFS_IO_PROG -f -c "falloc 0 $((blksz * nr))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x00 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "directio CoW across the transition"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/189 b/tests/generic/189
index a4b984f..1954235 100755
--- a/tests/generic/189
+++ b/tests/generic/189
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -56,36 +56,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-$XFS_IO_PROG -f -c "falloc 0 $((BLKSZ * NR))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x00 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+$XFS_IO_PROG -f -c "falloc 0 $((blksz * nr))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x00 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "CoW across the transition"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/190 b/tests/generic/190
index ee53bfd..d3d573b 100755
--- a/tests/generic/190
+++ b/tests/generic/190
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -56,36 +56,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-$XFS_IO_PROG -f -c "truncate $((BLKSZ * NR))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x00 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+$XFS_IO_PROG -f -c "truncate $((blksz * nr))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x00 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "directio CoW across the transition"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/191 b/tests/generic/191
index 6174a79..8b19084 100755
--- a/tests/generic/191
+++ b/tests/generic/191
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -56,36 +56,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-$XFS_IO_PROG -f -c "truncate $((BLKSZ * NR))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x00 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+$XFS_IO_PROG -f -c "truncate $((blksz * nr))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x00 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "CoW across the transition"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/194 b/tests/generic/194
index b88297e..1fc2cd7 100755
--- a/tests/generic/194
+++ b/tests/generic/194
@@ -38,7 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -57,40 +57,40 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-$XFS_IO_PROG -f -c "truncate $((BLKSZ * NR))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x00 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+$XFS_IO_PROG -f -c "truncate $((blksz * nr))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x00 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "directio CoW across the transition"
-seq 1 2 $((NR-1)) | while read f; do
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 1 2 $((nr-1)) | while read f; do
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3" >> "$seqres.full"
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/195 b/tests/generic/195
index 464a4d4..087275c 100755
--- a/tests/generic/195
+++ b/tests/generic/195
@@ -38,7 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -57,40 +57,40 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-$XFS_IO_PROG -f -c "truncate $((BLKSZ * NR))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x00 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+$XFS_IO_PROG -f -c "truncate $((blksz * nr))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x00 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "CoW across the transition"
-seq 1 2 $((NR-1)) | while read f; do
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 1 2 $((nr-1)) | while read f; do
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3" >> "$seqres.full"
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/196 b/tests/generic/196
index 48be50c..d815232 100755
--- a/tests/generic/196
+++ b/tests/generic/196
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -56,36 +56,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "directio CoW across the transition"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/197 b/tests/generic/197
index 9bffffe..54005ae 100755
--- a/tests/generic/197
+++ b/tests/generic/197
@@ -37,7 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -56,36 +56,36 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "CoW across the transition"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/199 b/tests/generic/199
index a67c04a..eb567e4 100755
--- a/tests/generic/199
+++ b/tests/generic/199
@@ -41,7 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -62,59 +62,59 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-$XFS_IO_PROG -f -c "truncate $((BLKSZ * NR))" "$TESTDIR/file3" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+$XFS_IO_PROG -f -c "truncate $((blksz * nr))" "$testdir/file3" >> "$seqres.full"
 # 0 blocks are reflinked
-seq 0 5 $NR | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 0 5 $nr | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 sync
 # 1 blocks are unwritten
-seq 1 5 $NR | while read f; do
-	$XFS_IO_PROG -f -c "falloc $((BLKSZ * f)) $BLKSZ" "$TESTDIR/file3" >> "$seqres.full"
-	_pwrite_byte 0x00 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 1 5 $nr | while read f; do
+	$XFS_IO_PROG -f -c "falloc $((blksz * f)) $blksz" "$testdir/file3" >> "$seqres.full"
+	_pwrite_byte 0x00 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 sync
 # 2 blocks are holes
-seq 2 5 $NR | while read f; do
-	_pwrite_byte 0x00 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 2 5 $nr | while read f; do
+	_pwrite_byte 0x00 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 # 3 blocks are regular
-seq 3 5 $NR | while read f; do
-	_pwrite_byte 0x71 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
-	_pwrite_byte 0x71 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 3 5 $nr | while read f; do
+	_pwrite_byte 0x71 $((blksz * f)) $blksz "$testdir/file3" >> "$seqres.full"
+	_pwrite_byte 0x71 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 sync
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "directio CoW across the transition"
 # 4 blocks are delalloc (do later)
-seq 4 5 $NR | while read f; do
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 4 5 $nr | while read f; do
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3" >> "$seqres.full"
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 # now cow
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/200 b/tests/generic/200
index 9ef533e..991fed0 100755
--- a/tests/generic/200
+++ b/tests/generic/200
@@ -41,7 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -62,59 +62,59 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-$XFS_IO_PROG -f -c "truncate $((BLKSZ * NR))" "$TESTDIR/file3" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+$XFS_IO_PROG -f -c "truncate $((blksz * nr))" "$testdir/file3" >> "$seqres.full"
 # 0 blocks are reflinked
-seq 0 5 $NR | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 0 5 $nr | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 sync
 # 1 blocks are unwritten
-seq 1 5 $NR | while read f; do
-	$XFS_IO_PROG -f -c "falloc $((BLKSZ * f)) $BLKSZ" "$TESTDIR/file3" >> "$seqres.full"
-	_pwrite_byte 0x00 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 1 5 $nr | while read f; do
+	$XFS_IO_PROG -f -c "falloc $((blksz * f)) $blksz" "$testdir/file3" >> "$seqres.full"
+	_pwrite_byte 0x00 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 sync
 # 2 blocks are holes
-seq 2 5 $NR | while read f; do
-	_pwrite_byte 0x00 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 2 5 $nr | while read f; do
+	_pwrite_byte 0x00 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 # 3 blocks are regular
-seq 3 5 $NR | while read f; do
-	_pwrite_byte 0x71 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
-	_pwrite_byte 0x71 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 3 5 $nr | while read f; do
+	_pwrite_byte 0x71 $((blksz * f)) $blksz "$testdir/file3" >> "$seqres.full"
+	_pwrite_byte 0x71 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 sync
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "directio CoW across the transition"
 # 4 blocks are delalloc (do later)
-seq 4 5 $NR | while read f; do
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
-	_pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+seq 4 5 $nr | while read f; do
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3" >> "$seqres.full"
+	_pwrite_byte 0x62 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 # now cow
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x63 $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2)) "$TESTDIR/file3.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x63 $((blksz * nr / 4)) $((blksz * nr / 2)) "$testdir/file3.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/201 b/tests/generic/201
index fa81b5a..ffc1ce4 100755
--- a/tests/generic/201
+++ b/tests/generic/201
@@ -33,7 +33,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,30 +52,30 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-NR=64
-_pwrite_byte 0x61 0 $((BLKSZ * NR)) "$TESTDIR/file1" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file3" >> "$seqres.full"
-_pwrite_byte 0x62 0 $((BLKSZ * NR)) "$TESTDIR/file3.chk" >> "$seqres.full"
-seq 0 2 $((NR-1)) | while read f; do
-	_reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
+blksz=65536
+nr=64
+_pwrite_byte 0x61 0 $((blksz * nr)) "$testdir/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file3" >> "$seqres.full"
+_pwrite_byte 0x62 0 $((blksz * nr)) "$testdir/file3.chk" >> "$seqres.full"
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range "$testdir/file1" $((blksz * f)) "$testdir/file3" $((blksz * f)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * f)) $blksz "$testdir/file3.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file3.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file3.chk" | _filter_scratch
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * NR / 2)) $((BLKSZ * NR / 4)) $((BLKSZ * NR / 2))" "$TESTDIR/file3" >> "$seqres.full"
-rm "$TESTDIR/file3"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * nr / 2)) $((blksz * nr / 4)) $((blksz * nr / 2))" "$testdir/file3" >> "$seqres.full"
+rm "$testdir/file3"
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/202 b/tests/generic/202
index 2d03b55..375b65f 100755
--- a/tests/generic/202
+++ b/tests/generic/202
@@ -32,7 +32,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,31 +51,31 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ + 17)) "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
-_pwrite_byte 0x61 0 $((BLKSZ + 17)) "$TESTDIR/file2.chk" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz + 17)) "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2"
+_pwrite_byte 0x61 0 $((blksz + 17)) "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file2.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file2.chk" | _filter_scratch
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) 17" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) 17" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) 17" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) 17" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file2.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file2.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/203 b/tests/generic/203
index 8ed765e..549f2ba 100755
--- a/tests/generic/203
+++ b/tests/generic/203
@@ -32,7 +32,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,31 +51,31 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
 echo "Create the original files"
-BLKSZ=65536
-_pwrite_byte 0x61 0 $((BLKSZ + 17)) "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
-_pwrite_byte 0x61 0 $((BLKSZ + 17)) "$TESTDIR/file2.chk" >> "$seqres.full"
+blksz=65536
+_pwrite_byte 0x61 0 $((blksz + 17)) "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2"
+_pwrite_byte 0x61 0 $((blksz + 17)) "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file2.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file2.chk" | _filter_scratch
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $BLKSZ $BLKSZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $BLKSZ $BLKSZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $blksz $blksz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $blksz $blksz" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file2.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file2.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/205 b/tests/generic/205
index 525138a..8f0ec45 100755
--- a/tests/generic/205
+++ b/tests/generic/205
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,48 +50,48 @@ _require_scratch_reflink
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 echo "Create the original files"
-_pwrite_byte 0x61 0 $PAGESZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $pagesz "$testdir/file1" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2.chk" >> "$seqres.full"
 
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 3)) $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 3)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/206 b/tests/generic/206
index 0a734a5..f2c4ecc 100755
--- a/tests/generic/206
+++ b/tests/generic/206
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -50,48 +50,48 @@ _require_scratch_reflink
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 echo "Create the original files"
-_pwrite_byte 0x61 0 $PAGESZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $pagesz "$testdir/file1" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2.chk" >> "$seqres.full"
 
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 3)) $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 3)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/216 b/tests/generic/216
index d4ab7aa..3789cb0 100755
--- a/tests/generic/216
+++ b/tests/generic/216
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,48 +51,48 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 echo "Create the original files"
-_pwrite_byte 0x61 0 $PAGESZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $pagesz "$testdir/file1" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2.chk" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "falloc -k $BLKSZ $BLKSZ" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x00 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "falloc -k $blksz $blksz" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x00 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "falloc -k $((BLKSZ * 3)) $BLKSZ" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "falloc -k $((blksz * 3)) $blksz" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz * 3)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/217 b/tests/generic/217
index 5e5c831..4ae603f 100755
--- a/tests/generic/217
+++ b/tests/generic/217
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,48 +51,48 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 echo "Create the original files"
-_pwrite_byte 0x61 0 $PAGESZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $pagesz "$testdir/file1" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2.chk" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "falloc -k $BLKSZ $BLKSZ" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x00 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "falloc -k $blksz $blksz" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x00 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "falloc -k $((BLKSZ * 3)) $BLKSZ" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x00 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "falloc -k $((blksz * 3)) $blksz" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x00 $((blksz * 3)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/218 b/tests/generic/218
index 7c31092..d63c06f 100755
--- a/tests/generic/218
+++ b/tests/generic/218
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,42 +51,42 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 echo "Create the original files"
-_pwrite_byte 0x61 0 $PAGESZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $pagesz "$testdir/file1" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2.chk" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/220 b/tests/generic/220
index 3ab9881..ccdb32e 100755
--- a/tests/generic/220
+++ b/tests/generic/220
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,42 +51,42 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 echo "Create the original files"
-_pwrite_byte 0x61 0 $PAGESZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $pagesz "$testdir/file1" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2.chk" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/222 b/tests/generic/222
index 3e344dc..cb5a788 100755
--- a/tests/generic/222
+++ b/tests/generic/222
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,48 +51,48 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 echo "Create the original files"
-_pwrite_byte 0x61 0 $PAGESZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $pagesz "$testdir/file1" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2.chk" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "CoW and unmount"
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 3)) $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 3)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/227 b/tests/generic/227
index ad52997..d200424 100755
--- a/tests/generic/227
+++ b/tests/generic/227
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,48 +51,48 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 echo "Create the original files"
-_pwrite_byte 0x61 0 $PAGESZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x61 0 $pagesz "$testdir/file1" >> "$seqres.full"
 
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "truncate $pagesz" "$testdir/file2.chk" >> "$seqres.full"
 
-_reflink_range "$TESTDIR/file1" $BLKSZ "$TESTDIR/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_reflink_range "$testdir/file1" $blksz "$testdir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 2)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "CoW and unmount"
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 $BLKSZ $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 $blksz $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 3)) $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 $((blksz * 3)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || _fail "file1 and file2 don't match."
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || _fail "file2 and file2.chk don't match."
+! cmp -s "$testdir/file1" "$testdir/file2" || _fail "file1 and file2 don't match."
+cmp -s "$testdir/file2" "$testdir/file2.chk" || _fail "file2 and file2.chk don't match."
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/229 b/tests/generic/229
index 4a65135..cdaddaf 100755
--- a/tests/generic/229
+++ b/tests/generic/229
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,19 +51,19 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 runtest() {
 	echo "runtest $1 $2"
@@ -73,19 +73,19 @@ runtest() {
 
 	echo "Create the original files"
 	mkdir -p "$dir"
-	_pwrite_byte 0x61 0 $PAGESZ "$dir/file1" >> "$seqres.full"
+	_pwrite_byte 0x61 0 $pagesz "$dir/file1" >> "$seqres.full"
 
-	"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$dir/file2" >> "$seqres.full"
-	"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$dir/file2.chk" >> "$seqres.full"
+	"$XFS_IO_PROG" -f -c "truncate $pagesz" "$dir/file2" >> "$seqres.full"
+	"$XFS_IO_PROG" -f -c "truncate $pagesz" "$dir/file2.chk" >> "$seqres.full"
 
 	case "$b2" in
 	"regular")
-		_pwrite_byte 0x61 $BLKSZ $BLKSZ "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x61 $BLKSZ $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		_pwrite_byte 0x61 $blksz $blksz "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x61 $blksz $blksz "$dir/file2.chk" >> "$seqres.full"
 		;;
 	"unwritten")
-		"$XFS_IO_PROG" -f -c "falloc -k $BLKSZ $BLKSZ" "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x00 $BLKSZ $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		"$XFS_IO_PROG" -f -c "falloc -k $blksz $blksz" "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x00 $blksz $blksz "$dir/file2.chk" >> "$seqres.full"
 		;;
 	"hole")
 		;;
@@ -95,19 +95,19 @@ runtest() {
 
 	case "$b4" in
 	"regular")
-		_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		_pwrite_byte 0x61 $((blksz * 3)) $blksz "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x61 $((blksz * 3)) $blksz "$dir/file2.chk" >> "$seqres.full"
 		;;
 	"unwritten")
-		"$XFS_IO_PROG" -f -c "falloc -k $((BLKSZ * 3)) $BLKSZ" "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x00 $((BLKSZ * 3)) $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		"$XFS_IO_PROG" -f -c "falloc -k $((blksz * 3)) $blksz" "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x00 $((blksz * 3)) $blksz "$dir/file2.chk" >> "$seqres.full"
 		;;
 	"hole")
 		;;
 	esac
 
-	_reflink_range "$dir/file1" $BLKSZ "$dir/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+	_reflink_range "$dir/file1" $blksz "$dir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * 2)) $blksz "$dir/file2.chk" >> "$seqres.full"
 	_scratch_remount
 
 	echo "Compare files"
@@ -116,17 +116,17 @@ runtest() {
 
 	echo "CoW and unmount"
 	if [ "$b2" = "delalloc" ]; then
-		_pwrite_byte 0x61 $BLKSZ $BLKSZ "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x61 $BLKSZ $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		_pwrite_byte 0x61 $blksz $blksz "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x61 $blksz $blksz "$dir/file2.chk" >> "$seqres.full"
 	fi
 
 	if [ "$b4" = "delalloc" ]; then
-		_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		_pwrite_byte 0x61 $((blksz * 3)) $blksz "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x61 $((blksz * 3)) $blksz "$dir/file2.chk" >> "$seqres.full"
 	fi
 
-	"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$dir/file2" >> "$seqres.full"
-	"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((BLKSZ + 17)) $((BLKSZ * 3 - 34))" "$dir/file2.chk" >> "$seqres.full"
+	"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$dir/file2" >> "$seqres.full"
+	"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $((blksz + 17)) $((blksz * 3 - 34))" "$dir/file2.chk" >> "$seqres.full"
 	_scratch_remount
 
 	echo "Compare files"
@@ -134,25 +134,25 @@ runtest() {
 	cmp -s "$dir/file2" "$dir/file2.chk" || _fail "file2 and file2.chk don't match."
 }
 
-runtest regular delalloc "$TESTDIR/r-d"
-runtest regular unwritten "$TESTDIR/r-u"
-runtest regular hole "$TESTDIR/r-h"
-runtest regular regular "$TESTDIR/r-r"
-
-runtest hole delalloc "$TESTDIR/h-d"
-runtest hole unwritten "$TESTDIR/h-u"
-runtest hole hole "$TESTDIR/h-h"
-runtest hole regular "$TESTDIR/h-r"
-
-runtest unwritten delalloc "$TESTDIR/u-d"
-runtest unwritten unwritten "$TESTDIR/u-u"
-runtest unwritten hole "$TESTDIR/u-h"
-runtest unwritten regular "$TESTDIR/u-r"
-
-runtest delalloc delalloc "$TESTDIR/d-d"
-runtest delalloc unwritten "$TESTDIR/d-u"
-runtest delalloc hole "$TESTDIR/d-h"
-runtest delalloc regular "$TESTDIR/d-r"
+runtest regular delalloc "$testdir/r-d"
+runtest regular unwritten "$testdir/r-u"
+runtest regular hole "$testdir/r-h"
+runtest regular regular "$testdir/r-r"
+
+runtest hole delalloc "$testdir/h-d"
+runtest hole unwritten "$testdir/h-u"
+runtest hole hole "$testdir/h-h"
+runtest hole regular "$testdir/h-r"
+
+runtest unwritten delalloc "$testdir/u-d"
+runtest unwritten unwritten "$testdir/u-u"
+runtest unwritten hole "$testdir/u-h"
+runtest unwritten regular "$testdir/u-r"
+
+runtest delalloc delalloc "$testdir/d-d"
+runtest delalloc unwritten "$testdir/d-u"
+runtest delalloc hole "$testdir/d-h"
+runtest delalloc regular "$testdir/d-r"
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/238 b/tests/generic/238
index e02396b..98530e0 100755
--- a/tests/generic/238
+++ b/tests/generic/238
@@ -36,7 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -51,19 +51,19 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-PAGESZ=$(getconf PAGE_SIZE)
-BLKSZ=$((PAGESZ / 4))
+pagesz=$(getconf PAGE_SIZE)
+blksz=$((pagesz / 4))
 
 echo "Format and mount"
-_scratch_mkfs_blocksized $BLKSZ > "$seqres.full" 2>&1
+_scratch_mkfs_blocksized $blksz > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-REAL_BLKSZ=$(stat -f -c '%S' $TESTDIR)
-test "$REAL_BLKSZ" != "$BLKSZ" && _notrun "Failed to format with small blocksize."
+real_blksz=$(stat -f -c '%S' $testdir)
+test "$real_blksz" != "$blksz" && _notrun "Failed to format with small blocksize."
 
 runtest() {
 	echo "runtest $1 $2"
@@ -73,19 +73,19 @@ runtest() {
 
 	echo "Create the original files"
 	mkdir -p "$dir"
-	_pwrite_byte 0x61 0 $PAGESZ "$dir/file1" >> "$seqres.full"
+	_pwrite_byte 0x61 0 $pagesz "$dir/file1" >> "$seqres.full"
 
-	"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$dir/file2" >> "$seqres.full"
-	"$XFS_IO_PROG" -f -c "truncate $PAGESZ" "$dir/file2.chk" >> "$seqres.full"
+	"$XFS_IO_PROG" -f -c "truncate $pagesz" "$dir/file2" >> "$seqres.full"
+	"$XFS_IO_PROG" -f -c "truncate $pagesz" "$dir/file2.chk" >> "$seqres.full"
 
 	case "$b2" in
 	"regular")
-		_pwrite_byte 0x61 $BLKSZ $BLKSZ "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x61 $BLKSZ $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		_pwrite_byte 0x61 $blksz $blksz "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x61 $blksz $blksz "$dir/file2.chk" >> "$seqres.full"
 		;;
 	"unwritten")
-		"$XFS_IO_PROG" -f -c "falloc -k $BLKSZ $BLKSZ" "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x00 $BLKSZ $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		"$XFS_IO_PROG" -f -c "falloc -k $blksz $blksz" "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x00 $blksz $blksz "$dir/file2.chk" >> "$seqres.full"
 		;;
 	"hole")
 		;;
@@ -95,19 +95,19 @@ runtest() {
 
 	case "$b4" in
 	"regular")
-		_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		_pwrite_byte 0x61 $((blksz * 3)) $blksz "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x61 $((blksz * 3)) $blksz "$dir/file2.chk" >> "$seqres.full"
 		;;
 	"unwritten")
-		"$XFS_IO_PROG" -f -c "falloc -k $((BLKSZ * 3)) $BLKSZ" "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x00 $((BLKSZ * 3)) $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		"$XFS_IO_PROG" -f -c "falloc -k $((blksz * 3)) $blksz" "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x00 $((blksz * 3)) $blksz "$dir/file2.chk" >> "$seqres.full"
 		;;
 	"hole")
 		;;
 	esac
 
-	_reflink_range "$dir/file1" $BLKSZ "$dir/file2" $((BLKSZ * 2)) $BLKSZ >> "$seqres.full"
-	_pwrite_byte 0x61 $((BLKSZ * 2)) $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+	_reflink_range "$dir/file1" $blksz "$dir/file2" $((blksz * 2)) $blksz >> "$seqres.full"
+	_pwrite_byte 0x61 $((blksz * 2)) $blksz "$dir/file2.chk" >> "$seqres.full"
 	_scratch_remount
 
 	echo "Compare files"
@@ -116,17 +116,17 @@ runtest() {
 
 	echo "CoW and unmount"
 	if [ "$b2" = "delalloc" ]; then
-		_pwrite_byte 0x61 $BLKSZ $BLKSZ "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x61 $BLKSZ $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		_pwrite_byte 0x61 $blksz $blksz "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x61 $blksz $blksz "$dir/file2.chk" >> "$seqres.full"
 	fi
 
 	if [ "$b4" = "delalloc" ]; then
-		_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$dir/file2" >> "$seqres.full"
-		_pwrite_byte 0x61 $((BLKSZ * 3)) $BLKSZ "$dir/file2.chk" >> "$seqres.full"
+		_pwrite_byte 0x61 $((blksz * 3)) $blksz "$dir/file2" >> "$seqres.full"
+		_pwrite_byte 0x61 $((blksz * 3)) $blksz "$dir/file2.chk" >> "$seqres.full"
 	fi
 
-	"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$dir/file2" >> "$seqres.full"
-	"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $BLKSZ $((BLKSZ * 3))" "$dir/file2.chk" >> "$seqres.full"
+	"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$dir/file2" >> "$seqres.full"
+	"$XFS_IO_PROG" -f -c "pwrite -S 0x63 $blksz $((blksz * 3))" "$dir/file2.chk" >> "$seqres.full"
 	_scratch_remount
 
 	echo "Compare files"
@@ -134,25 +134,25 @@ runtest() {
 	cmp -s "$dir/file2" "$dir/file2.chk" || _fail "file2 and file2.chk don't match."
 }
 
-runtest regular delalloc "$TESTDIR/r-d"
-runtest regular unwritten "$TESTDIR/r-u"
-runtest regular hole "$TESTDIR/r-h"
-runtest regular regular "$TESTDIR/r-r"
-
-runtest hole delalloc "$TESTDIR/h-d"
-runtest hole unwritten "$TESTDIR/h-u"
-runtest hole hole "$TESTDIR/h-h"
-runtest hole regular "$TESTDIR/h-r"
-
-runtest unwritten delalloc "$TESTDIR/u-d"
-runtest unwritten unwritten "$TESTDIR/u-u"
-runtest unwritten hole "$TESTDIR/u-h"
-runtest unwritten regular "$TESTDIR/u-r"
-
-runtest delalloc delalloc "$TESTDIR/d-d"
-runtest delalloc unwritten "$TESTDIR/d-u"
-runtest delalloc hole "$TESTDIR/d-h"
-runtest delalloc regular "$TESTDIR/d-r"
+runtest regular delalloc "$testdir/r-d"
+runtest regular unwritten "$testdir/r-u"
+runtest regular hole "$testdir/r-h"
+runtest regular regular "$testdir/r-r"
+
+runtest hole delalloc "$testdir/h-d"
+runtest hole unwritten "$testdir/h-u"
+runtest hole hole "$testdir/h-h"
+runtest hole regular "$testdir/h-r"
+
+runtest unwritten delalloc "$testdir/u-d"
+runtest unwritten unwritten "$testdir/u-u"
+runtest unwritten hole "$testdir/u-h"
+runtest unwritten regular "$testdir/u-r"
+
+runtest delalloc delalloc "$testdir/d-d"
+runtest delalloc unwritten "$testdir/d-u"
+runtest delalloc hole "$testdir/d-h"
+runtest delalloc regular "$testdir/d-r"
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/242 b/tests/generic/242
index 5f1f502..a5a17c3 100755
--- a/tests/generic/242
+++ b/tests/generic/242
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -55,40 +55,40 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-BLKSZ=65536
-NR=6400
-BSZ=1280
+blksz=65536
+nr=6400
+bsz=1280
 
-FREE_BLOCKS=$(stat -f -c '%a' "$TESTDIR")
-REAL_BLKSZ=$(stat -f -c '%S' "$TESTDIR")
-SPACE_NEEDED=$(((BLKSZ * NR * 3) * 5 / 4))
-SPACE_AVAIL=$((FREE_BLOCKS * REAL_BLKSZ))
-test $SPACE_NEEDED -gt $SPACE_AVAIL && _notrun "Not enough space. $SPACE_AVAIL < $SPACE_NEEDED"
+free_blocks=$(stat -f -c '%a' "$testdir")
+real_blksz=$(stat -f -c '%S' "$testdir")
+space_needed=$(((blksz * nr * 3) * 5 / 4))
+space_avail=$((free_blocks * real_blksz))
+test $space_needed -gt $space_avail && _notrun "Not enough space. $space_avail < $space_needed"
 
 echo "Create the original files"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x61 -b $((BLKSZ * BSZ)) 0 $((BLKSZ * NR))" "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x61 -b $((BLKSZ * BSZ)) 0 $((BLKSZ * NR))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x61 -b $((blksz * bsz)) 0 $((blksz * nr))" "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x61 -b $((blksz * bsz)) 0 $((blksz * nr))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file2.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file2.chk" | _filter_scratch
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * BSZ)) 0 $((BLKSZ * NR))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * BSZ)) 0 $((BLKSZ * NR))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * bsz)) 0 $((blksz * nr))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * bsz)) 0 $((blksz * nr))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file2.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file2.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/generic/243 b/tests/generic/243
index 2162eb7..b7ca466 100755
--- a/tests/generic/243
+++ b/tests/generic/243
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -55,40 +55,40 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf $TESTDIR
-mkdir $TESTDIR
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf $testdir
+mkdir $testdir
 
-BLKSZ=65536
-NR=6400
-BSZ=1280
+blksz=65536
+nr=6400
+bsz=1280
 
-FREE_BLOCKS=$(stat -f -c '%a' "$TESTDIR")
-REAL_BLKSZ=$(stat -f -c '%S' "$TESTDIR")
-SPACE_NEEDED=$(((BLKSZ * NR * 3) * 5 / 4))
-SPACE_AVAIL=$((FREE_BLOCKS * REAL_BLKSZ))
-test $SPACE_NEEDED -gt $SPACE_AVAIL && _notrun "Not enough space. $SPACE_AVAIL < $SPACE_NEEDED"
+free_blocks=$(stat -f -c '%a' "$testdir")
+real_blksz=$(stat -f -c '%S' "$testdir")
+space_needed=$(((blksz * nr * 3) * 5 / 4))
+space_avail=$((free_blocks * real_blksz))
+test $space_needed -gt $space_avail && _notrun "Not enough space. $space_avail < $space_needed"
 
 echo "Create the original files"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x61 -b $((BLKSZ * BSZ)) 0 $((BLKSZ * NR))" "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x61 -b $((BLKSZ * BSZ)) 0 $((BLKSZ * NR))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x61 -b $((blksz * bsz)) 0 $((blksz * nr))" "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x61 -b $((blksz * bsz)) 0 $((blksz * nr))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file2.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file2.chk" | _filter_scratch
 
 echo "CoW and unmount"
-"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((BLKSZ * BSZ)) 0 $((BLKSZ * NR))" "$TESTDIR/file2" >> "$seqres.full"
-"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((BLKSZ * BSZ)) 0 $((BLKSZ * NR))" "$TESTDIR/file2.chk" >> "$seqres.full"
+"$XFS_IO_PROG" -d -f -c "pwrite -S 0x63 -b $((blksz * bsz)) 0 $((blksz * nr))" "$testdir/file2" >> "$seqres.full"
+"$XFS_IO_PROG" -f -c "pwrite -S 0x63 -b $((blksz * bsz)) 0 $((blksz * nr))" "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file2.chk" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file2.chk" | _filter_scratch
 
 echo "Check for damage"
 umount "$SCRATCH_MNT"
diff --git a/tests/xfs/127 b/tests/xfs/127
index 61301de..c31cd17 100755
--- a/tests/xfs/127
+++ b/tests/xfs/127
@@ -51,22 +51,22 @@ echo "Format and mount"
 _scratch_mkfs -d size=$((2 * 4096 * 4096)) -l size=4194304 > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file and reflink to copy1, copy2"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-_pwrite_byte 0x61 0 $((BLKSZ * 14 + 71)) "$TESTDIR/original" >> "$seqres.full"
-_cp_reflink "$TESTDIR/original" "$TESTDIR/copy1"
-_cp_reflink "$TESTDIR/copy1" "$TESTDIR/copy2"
+blksz="$(stat -f "$testdir" -c '%S')"
+_pwrite_byte 0x61 0 $((blksz * 14 + 71)) "$testdir/original" >> "$seqres.full"
+_cp_reflink "$testdir/original" "$testdir/copy1"
+_cp_reflink "$testdir/copy1" "$testdir/copy2"
 
 echo "Grow fs"
 "$XFS_GROWFS_PROG" "$SCRATCH_MNT" 2>&1 |  _filter_growfs >> "$seqres.full"
 _scratch_remount
 
 echo "Create more reflink copies"
-_cp_reflink "$TESTDIR/original" "$TESTDIR/copy3"
+_cp_reflink "$testdir/original" "$testdir/copy3"
 
 xfs_info "$SCRATCH_MNT" >> "$seqres.full"
 
diff --git a/tests/xfs/128 b/tests/xfs/128
index f461c1a..a96291a 100755
--- a/tests/xfs/128
+++ b/tests/xfs/128
@@ -52,93 +52,93 @@ echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
+free_blocks0=$(stat -f "$testdir" -c '%f')
 
 echo "Create the original file and reflink to file2, file3"
-BLKS=2000
-MARGIN=160
-BLKSZ=65536
-REAL_BLKSZ="$(stat -f $TESTDIR -c '%S')"
-BLKSZ_FACTOR=$((BLKSZ / REAL_BLKSZ))
-_pwrite_byte 0x61 0 $((BLKS * BLKSZ)) "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2"
-_cp_reflink "$TESTDIR/file2" "$TESTDIR/file3"
-_cp_reflink "$TESTDIR/file3" "$TESTDIR/file4"
+blks=2000
+margin=160
+blksz=65536
+real_blksz="$(stat -f $testdir -c '%S')"
+blksz_factor=$((blksz / real_blksz))
+_pwrite_byte 0x61 0 $((blks * blksz)) "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2"
+_cp_reflink "$testdir/file2" "$testdir/file3"
+_cp_reflink "$testdir/file3" "$testdir/file4"
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
+free_blocks1=$(stat -f "$testdir" -c '%f')
 
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file4" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file4" | _filter_scratch
 
-C01=$(_md5_checksum "$TESTDIR/file1")
-C02=$(_md5_checksum "$TESTDIR/file2")
-C03=$(_md5_checksum "$TESTDIR/file3")
-C04=$(_md5_checksum "$TESTDIR/file4")
+c01=$(_md5_checksum "$testdir/file1")
+c02=$(_md5_checksum "$testdir/file2")
+c03=$(_md5_checksum "$testdir/file3")
+c04=$(_md5_checksum "$testdir/file4")
 
 echo "CoW the reflink copies"
-_pwrite_byte 0x62 $BLKSZ $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x63 $(( BLKSZ * (BLKS - 1) )) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
+_pwrite_byte 0x62 $blksz $blksz "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x63 $(( blksz * (blks - 1) )) $blksz "$testdir/file3" >> "$seqres.full"
 _test_remount
-FREE_BLOCKS2=$(stat -f "$TESTDIR" -c '%f')
+free_blocks2=$(stat -f "$testdir" -c '%f')
 
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file4" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file4" | _filter_scratch
 
-C11=$(_md5_checksum "$TESTDIR/file1")
-C12=$(_md5_checksum "$TESTDIR/file2")
-C13=$(_md5_checksum "$TESTDIR/file3")
-C14=$(_md5_checksum "$TESTDIR/file4")
+c11=$(_md5_checksum "$testdir/file1")
+c12=$(_md5_checksum "$testdir/file2")
+c13=$(_md5_checksum "$testdir/file3")
+c14=$(_md5_checksum "$testdir/file4")
 
 echo "Defragment"
-lsattr -l "$TESTDIR/" | _filter_scratch
-xfs_fsr -v -d "$TESTDIR/file1" >> "$seqres.full"
-xfs_fsr -v -d "$TESTDIR/file2" >> "$seqres.full" # fsr probably breaks the link
-xfs_fsr -v -d "$TESTDIR/file3" >> "$seqres.full" # fsr probably breaks the link
-xfs_fsr -v -d "$TESTDIR/file4" >> "$seqres.full" # fsr probably ignores this file
+lsattr -l "$testdir/" | _filter_scratch
+xfs_fsr -v -d "$testdir/file1" >> "$seqres.full"
+xfs_fsr -v -d "$testdir/file2" >> "$seqres.full" # fsr probably breaks the link
+xfs_fsr -v -d "$testdir/file3" >> "$seqres.full" # fsr probably breaks the link
+xfs_fsr -v -d "$testdir/file4" >> "$seqres.full" # fsr probably ignores this file
 _test_remount
-FREE_BLOCKS3=$(stat -f "$TESTDIR" -c '%f')
+free_blocks3=$(stat -f "$testdir" -c '%f')
 
-md5sum "$TESTDIR/file1" | _filter_scratch
-md5sum "$TESTDIR/file2" | _filter_scratch
-md5sum "$TESTDIR/file3" | _filter_scratch
-md5sum "$TESTDIR/file4" | _filter_scratch
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
+md5sum "$testdir/file3" | _filter_scratch
+md5sum "$testdir/file4" | _filter_scratch
 
-C21=$(_md5_checksum "$TESTDIR/file1")
-C22=$(_md5_checksum "$TESTDIR/file2")
-C23=$(_md5_checksum "$TESTDIR/file3")
-C24=$(_md5_checksum "$TESTDIR/file4")
+c21=$(_md5_checksum "$testdir/file1")
+c22=$(_md5_checksum "$testdir/file2")
+c23=$(_md5_checksum "$testdir/file3")
+c24=$(_md5_checksum "$testdir/file4")
 
 echo "Check files"
-test $C01 = $C02 || echo "Files 1-2 do not match"
-test $C01 = $C03 || echo "Files 1-3 do not match"
-test $C01 = $C04 || echo "Files 1-4 do not match"
-test $C02 = $C03 || echo "Files 2-3 do not match"
-test $C02 = $C04 || echo "Files 2-4 do not match"
-test $C03 = $C04 || echo "Files 3-4 do not match"
-
-test $C01 = $C11 || echo "File1 should not be different after CoW"
-test $C02 != $C12 || echo "File2 should be different after CoW"
-test $C03 != $C13 || echo "File3 should be different after CoW"
-test $C04 = $C14 || echo "File4 should not be different after CoW"
-
-test $C11 = $C21 || echo "File1 changed by defrag"
-test $C12 = $C22 || echo "File2 changed by defrag"
-test $C13 = $C23 || echo "File3 changed by defrag"
-test $C14 = $C24 || echo "File4 changed by defrag"
-
-#echo $FREE_BLOCKS0 $FREE_BLOCKS1 $FREE_BLOCKS2 $FREE_BLOCKS3
-
-_within_tolerance "free blocks after creating some reflink copies" $FREE_BLOCKS1 $((FREE_BLOCKS0 - (BLKS * BLKSZ_FACTOR) )) $MARGIN -v
-_within_tolerance "free blocks after CoW some reflink copies" $FREE_BLOCKS2 $((FREE_BLOCKS1 - 2)) $MARGIN -v
-_within_tolerance "free blocks after defragging all reflink copies" $FREE_BLOCKS3 $((FREE_BLOCKS2 - (BLKS * 2 * BLKSZ_FACTOR))) $MARGIN -v
-_within_tolerance "free blocks after all tests" $FREE_BLOCKS3 $((FREE_BLOCKS0 - (BLKS * 3 * BLKSZ_FACTOR))) $MARGIN -v
+test $c01 = $c02 || echo "Files 1-2 do not match"
+test $c01 = $c03 || echo "Files 1-3 do not match"
+test $c01 = $c04 || echo "Files 1-4 do not match"
+test $c02 = $c03 || echo "Files 2-3 do not match"
+test $c02 = $c04 || echo "Files 2-4 do not match"
+test $c03 = $c04 || echo "Files 3-4 do not match"
+
+test $c01 = $c11 || echo "File1 should not be different after CoW"
+test $c02 != $c12 || echo "File2 should be different after CoW"
+test $c03 != $c13 || echo "File3 should be different after CoW"
+test $c04 = $c14 || echo "File4 should not be different after CoW"
+
+test $c11 = $c21 || echo "File1 changed by defrag"
+test $c12 = $c22 || echo "File2 changed by defrag"
+test $c13 = $c23 || echo "File3 changed by defrag"
+test $c14 = $c24 || echo "File4 changed by defrag"
+
+#echo $free_blocks0 $free_blocks1 $free_blocks2 $free_blocks3
+
+_within_tolerance "free blocks after creating some reflink copies" $free_blocks1 $((free_blocks0 - (blks * blksz_factor) )) $margin -v
+_within_tolerance "free blocks after CoW some reflink copies" $free_blocks2 $((free_blocks1 - 2)) $margin -v
+_within_tolerance "free blocks after defragging all reflink copies" $free_blocks3 $((free_blocks2 - (blks * 2 * blksz_factor))) $margin -v
+_within_tolerance "free blocks after all tests" $free_blocks3 $((free_blocks0 - (blks * 3 * blksz_factor))) $margin -v
 
 echo "Check scratch fs"
 umount "$SCRATCH_MNT"
diff --git a/tests/xfs/129 b/tests/xfs/129
index 3b7c59c..8d608f2 100755
--- a/tests/xfs/129
+++ b/tests/xfs/129
@@ -35,7 +35,7 @@ _cleanup()
 {
     cd /
     umount "$SCRATCH_MNT" > /dev/null 2>&1
-    rm -rf "$tmp".* "$TESTDIR" "$METADUMP_FILE" "$TEST_DIR/image"
+    rm -rf "$tmp".* "$testdir" "$metadump_file" "$TEST_DIR/image"
 }
 
 # get standard environment, filters and checks
@@ -54,34 +54,34 @@ rm -f "$seqres.full"
 _scratch_mkfs >/dev/null 2>&1
 _scratch_mount
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
-METADUMP_FILE="$TEST_DIR/${seq}_metadump"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
+metadump_file="$TEST_DIR/${seq}_metadump"
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-NR_BLKS=$((4 * BLKSZ / 12))
-_pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+nr_blks=$((4 * blksz / 12))
+_pwrite_byte 0x61 0 $((blksz * nr_blks)) "$testdir/file1" >> "$seqres.full"
 
 echo "Reflink every other block"
-seq 1 2 $((NR_BLKS - 1)) | while read nr; do
-	_reflink_range  "$TESTDIR/file1" $((nr * BLKSZ)) \
-			"$TESTDIR/file2" $((nr * BLKSZ)) $BLKSZ >> "$seqres.full"
+seq 1 2 $((nr_blks - 1)) | while read nr; do
+	_reflink_range  "$testdir/file1" $((nr * blksz)) \
+			"$testdir/file2" $((nr * blksz)) $blksz >> "$seqres.full"
 done
 
 echo "Create metadump file"
 _scratch_unmount
-_scratch_metadump "$METADUMP_FILE"
+_scratch_metadump "$metadump_file"
 
 # Now restore the obfuscated one back and take a look around
 echo "Restore metadump"
-xfs_mdrestore "$METADUMP_FILE" "$TEST_DIR/image"
+xfs_mdrestore "$metadump_file" "$TEST_DIR/image"
 _mount -t $FSTYP "$TEST_DIR/image" "$SCRATCH_MNT"
 umount "$SCRATCH_MNT"
 
 echo "Check restored fs"
-_check_generic_filesystem "$METADUMP_FILE"
+_check_generic_filesystem "$metadump_file"
 
 # success, all done
 status=0
diff --git a/tests/xfs/131 b/tests/xfs/131
index 5d092f1..8442821 100755
--- a/tests/xfs/131
+++ b/tests/xfs/131
@@ -33,7 +33,7 @@ _cleanup()
 {
     cd /
     umount "$SCRATCH_MNT" > /dev/null 2>&1
-    rm -rf "$tmp".* "$TESTDIR" "$METADUMP_FILE"
+    rm -rf "$tmp".* "$testdir" "$metadump_file"
 }
 
 # get standard environment, filters and checks
@@ -54,18 +54,18 @@ echo "Format and mount scratch device"
 _scratch_mkfs >> "$seqres.full"
 _scratch_mount
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ=65536
-$XFS_IO_PROG -R -f -c "truncate $BLKSZ" "$TESTDIR/file1"
+blksz=65536
+$XFS_IO_PROG -R -f -c "truncate $blksz" "$testdir/file1"
 
 echo "Reflink every block"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" 2>&1 | _filter_scratch
+_cp_reflink "$testdir/file1" "$testdir/file2" 2>&1 | _filter_scratch
 
-test -s "$TESTDIR/file2" && _fail "Should not be able to reflink a realtime file."
+test -s "$testdir/file2" && _fail "Should not be able to reflink a realtime file."
 
 echo "Check restored fs"
 umount "$SCRATCH_MNT"
diff --git a/tests/xfs/132 b/tests/xfs/132
index ec7b178..79a6d57 100755
--- a/tests/xfs/132
+++ b/tests/xfs/132
@@ -49,7 +49,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -67,63 +67,63 @@ _require_xfs_io_command "falloc"
 
 rm -f "$seqres.full"
 
-TESTDIR="$TEST_DIR/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
+testdir="$TEST_DIR/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
 
 echo "Create the original file blocks"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
-BLKS=2000
-MARGIN=100
-SZ=$((BLKSZ * BLKS))
-FREE_BLOCKS0=$(stat -f "$TESTDIR" -c '%f')
-NR=4
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+blksz="$(stat -f "$testdir" -c '%S')"
+blks=2000
+margin=100
+sz=$((blksz * blks))
+free_blocks0=$(stat -f "$testdir" -c '%f')
+nr=4
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
 
 echo "Create the reflink copies"
-for i in `seq 2 $NR`; do
-	_cp_reflink "$TESTDIR/file1" "$TESTDIR/file$i"
+for i in `seq 2 $nr`; do
+	_cp_reflink "$testdir/file1" "$testdir/file$i"
 done
 _test_remount
-FREE_BLOCKS1=$(stat -f "$TESTDIR" -c '%f')
-lsattr -l $TESTDIR/ | _filter_test_dir
+free_blocks1=$(stat -f "$testdir" -c '%f')
+lsattr -l $testdir/ | _filter_test_dir
 
 echo "funshare part of a file"
-"$XFS_IO_PROG" -f -c "falloc 0 $((SZ / 2))" "$TESTDIR/file2"
+"$XFS_IO_PROG" -f -c "falloc 0 $((sz / 2))" "$testdir/file2"
 _test_remount
-lsattr -l $TESTDIR/ | _filter_test_dir
+lsattr -l $testdir/ | _filter_test_dir
 
 echo "funshare some of the copies"
-"$XFS_IO_PROG" -f -c "falloc 0 $SZ" "$TESTDIR/file2"
-"$XFS_IO_PROG" -f -c "falloc 0 $SZ" "$TESTDIR/file3"
+"$XFS_IO_PROG" -f -c "falloc 0 $sz" "$testdir/file2"
+"$XFS_IO_PROG" -f -c "falloc 0 $sz" "$testdir/file3"
 _test_remount
-FREE_BLOCKS2=$(stat -f "$TESTDIR" -c '%f')
-lsattr -l $TESTDIR/ | _filter_test_dir
+free_blocks2=$(stat -f "$testdir" -c '%f')
+lsattr -l $testdir/ | _filter_test_dir
 
 echo "funshare the rest of the files"
-"$XFS_IO_PROG" -f -c "falloc 0 $SZ" "$TESTDIR/file4"
-"$XFS_IO_PROG" -f -c "falloc 0 $SZ" "$TESTDIR/file1"
+"$XFS_IO_PROG" -f -c "falloc 0 $sz" "$testdir/file4"
+"$XFS_IO_PROG" -f -c "falloc 0 $sz" "$testdir/file1"
 _test_remount
-FREE_BLOCKS3=$(stat -f "$TESTDIR" -c '%f')
-lsattr -l $TESTDIR/ | _filter_test_dir
+free_blocks3=$(stat -f "$testdir" -c '%f')
+lsattr -l $testdir/ | _filter_test_dir
 
 echo "Rewrite the original file"
-_pwrite_byte 0x65 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x65 0 $sz "$testdir/file1" >> "$seqres.full"
 _test_remount
-FREE_BLOCKS4=$(stat -f "$TESTDIR" -c '%f')
-lsattr -l $TESTDIR/ | _filter_test_dir
-#echo $FREE_BLOCKS0 $FREE_BLOCKS1 $FREE_BLOCKS2 $FREE_BLOCKS3 $FREE_BLOCKS4
+free_blocks4=$(stat -f "$testdir" -c '%f')
+lsattr -l $testdir/ | _filter_test_dir
+#echo $free_blocks0 $free_blocks1 $free_blocks2 $free_blocks3 $free_blocks4
 
-_within_tolerance "free blocks after reflinking" $FREE_BLOCKS1 $((FREE_BLOCKS0 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after reflinking" $free_blocks1 $((free_blocks0 - blks)) $margin -v
 
-_within_tolerance "free blocks after nocow'ing some copies" $FREE_BLOCKS2 $((FREE_BLOCKS1 - (2 * BLKS))) $MARGIN -v
+_within_tolerance "free blocks after nocow'ing some copies" $free_blocks2 $((free_blocks1 - (2 * blks))) $margin -v
 
-_within_tolerance "free blocks after nocow'ing all copies" $FREE_BLOCKS3 $((FREE_BLOCKS2 - BLKS)) $MARGIN -v
+_within_tolerance "free blocks after nocow'ing all copies" $free_blocks3 $((free_blocks2 - blks)) $margin -v
 
-_within_tolerance "free blocks after overwriting original" $FREE_BLOCKS4 $FREE_BLOCKS3 $MARGIN -v
+_within_tolerance "free blocks after overwriting original" $free_blocks4 $free_blocks3 $margin -v
 
-_within_tolerance "free blocks after all tests" $FREE_BLOCKS4 $((FREE_BLOCKS0 - (4 * BLKS))) $MARGIN -v
+_within_tolerance "free blocks after all tests" $free_blocks4 $((free_blocks0 - (4 * blks))) $margin -v
 
 # success, all done
 status=0
diff --git a/tests/xfs/139 b/tests/xfs/139
index 152c6bd..7400cf8 100644
--- a/tests/xfs/139
+++ b/tests/xfs/139
@@ -33,7 +33,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,26 +52,26 @@ echo "Format and mount"
 _scratch_mkfs -d agsize=$((32 * 1048576)) > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
-BLKSZ="$(stat -f "$TESTDIR" -c '%S')"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
+blksz="$(stat -f "$testdir" -c '%S')"
 
 echo "Create the original files"
-SZ=$((48 * 1048576))
-NR=$((SZ / BLKSZ))
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" >> "$seqres.full"
+sz=$((48 * 1048576))
+nr=$((sz / blksz))
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2" >> "$seqres.full"
 _scratch_remount
 
 echo "CoW every other block"
-_pwrite_byte 0x62 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
+_pwrite_byte 0x62 0 $sz "$testdir/file1" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$TESTDIR/file1" | _filter_test_dir
-md5sum "$TESTDIR/file2" | _filter_test_dir
-#filefrag -v "$TESTDIR/file1" "$TESTDIR/file2"
+md5sum "$testdir/file1" | _filter_test_dir
+md5sum "$testdir/file2" | _filter_test_dir
+#filefrag -v "$testdir/file1" "$testdir/file2"
 
 echo "Check scratch fs"
 umount "$SCRATCH_MNT"
diff --git a/tests/xfs/140 b/tests/xfs/140
index 45be9f4..133bb72 100644
--- a/tests/xfs/140
+++ b/tests/xfs/140
@@ -33,7 +33,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
     cd /
-    rm -rf "$tmp".* "$TESTDIR"
+    rm -rf "$tmp".* "$testdir"
 }
 
 # get standard environment, filters and checks
@@ -52,33 +52,33 @@ echo "Format and mount"
 _scratch_mkfs -d agsize=$((32 * 1048576)) > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
-TESTDIR="$SCRATCH_MNT/test-$seq"
-rm -rf "$TESTDIR"
-mkdir "$TESTDIR"
-BLKSZ="$(stat -f -c '%S' "$TESTDIR")"
+testdir="$SCRATCH_MNT/test-$seq"
+rm -rf "$testdir"
+mkdir "$testdir"
+blksz="$(stat -f -c '%S' "$testdir")"
 
 echo "Create the original files"
-SZ=$((48 * 1048576))
-NR=$((SZ / BLKSZ))
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file1" >> "$seqres.full"
-_cp_reflink "$TESTDIR/file1" "$TESTDIR/file2" >> "$seqres.full"
-_pwrite_byte 0x61 0 $SZ "$TESTDIR/file2.chk" >> "$seqres.full"
+sz=$((48 * 1048576))
+nr=$((sz / blksz))
+_pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
+_cp_reflink "$testdir/file1" "$testdir/file2" >> "$seqres.full"
+_pwrite_byte 0x61 0 $sz "$testdir/file2.chk" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "file1 and file2 do not match"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
+cmp -s "$testdir/file1" "$testdir/file2" || echo "file1 and file2 do not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
 
 echo "CoW every other block"
-seq 1 2 $((NR - 1)) | while read f; do
-	_pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
-	_pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
+seq 1 2 $((nr - 1)) | while read f; do
+	_pwrite_byte 0x62 $((f * blksz)) $blksz "$testdir/file2" >> "$seqres.full"
+	_pwrite_byte 0x62 $((f * blksz)) $blksz "$testdir/file2.chk" >> "$seqres.full"
 done
 _scratch_remount
 
 echo "Compare files"
-! cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "file1 and file2 must not match"
-cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
+! cmp -s "$testdir/file1" "$testdir/file2" || echo "file1 and file2 must not match"
+cmp -s "$testdir/file2" "$testdir/file2.chk" || echo "file2 and file2.chk do not match"
 
 echo "Check scratch fs"
 umount "$SCRATCH_MNT"

From darrick.wong@oracle.com  Fri Jan 22 18:55:23 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id C6FE77F5D
	for ; Fri, 22 Jan 2016 18:55:23 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 5618AAC005
	for ; Fri, 22 Jan 2016 16:55:23 -0800 (PST)
X-ASG-Debug-ID: 1453510520-04cbb026fc72100001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id LLsOykhsWEj0MSso (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:55:21 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0sgdE032553
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:54:43 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0sgbc031181
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:54:42 GMT
Received: from abhmp0008.oracle.com (abhmp0008.oracle.com [141.146.116.14])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0sf2E000981;
	Sat, 23 Jan 2016 00:54:41 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:54:41 -0800
Date: Fri, 22 Jan 2016 16:54:39 -0800
From: "Darrick J. Wong" 
To: Eric Biggers 
Cc: david@fromorbit.com, linux-fsdevel@vger.kernel.org,
        linux-api@vger.kernel.org, xfs@oss.sgi.com,
        linux-btrfs@vger.kernel.org, hch@infradead.org
Subject: Re: [PATCH 8/9] vfs: hoist the btrfs deduplication ioctl to the vfs
Message-ID: <20160123005439.GA5496@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 8/9] vfs: hoist the btrfs deduplication ioctl to the vfs
References: <20151219085505.12478.71157.stgit@birch.djwong.org>
 <20151219085559.12478.33700.stgit@birch.djwong.org>
 <20160112060714.GA4980@zzz>
 <20160112091432.GB7832@birch.djwong.org>
 <20160113023658.GA1551@zzz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160113023658.GA1551@zzz>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453510521
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26369
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Tue, Jan 12, 2016 at 08:36:58PM -0600, Eric Biggers wrote:
> On Tue, Jan 12, 2016 at 01:14:32AM -0800, Darrick J. Wong wrote:
> > 
> > Frankly, I also wouldn't mind changing the VFS dedupe ioctl that to something
> > that resembles the clone_range interface:
> > 
> > int ioctl(int dest_fd, FIDEDUPERANGE, struct file_dedupe_range * arg);
> 
> Getting rid of the vectorization certainly avoids a lot of API complexity.  I
> don't know precisely why it was made vectorized in the first place, but to me it
> seems it shouldn't be vectorized unless there's a strong reason for it to be.

I suppose the idea was that you could ask the kernel to dedupe a bunch of files
"atomically" and that locks would be held (at least on the source file) during
the entire operation...?

Ick.  Given that you can have 65536 vectors of 16MB apiece, you could be
reading up to 1TB of data into the pagecache.  That's a lot for a single
ioctl, though I guess we unlock all the inodes between each vector.

(Welll... we don't check for pending signals.  Urk.)

> > Not sure what you mean by 'symmetric', but in any case the convention seems
> > to be that src_fd's storage is shared with dest_fd if there's a match.
> > ...
> > I think the result of an earlier discussion was that src_fd must be readable,
> > and dest_fd must be writable or appendable.
> 
> Deduplication is a strange operation since you're not "reading" or "writing" in
> the traditional sense.  All file data stays exactly the same from a user
> perspective.
> 
> So rather than arbitrarily say that you're "reading" from one file and "writing"
> to the other, one possibility is to have an API where you submit two files and
> the implementation decides how to best deduplicate them.  That could involve
> sharing existing extents in file 1 with file 2; or sharing existing extents in
> file 2 with file 1; or creating a new extent and referencing it from each file.
> It would be up to the implementation to choose the most efficient approach.

That's a very good point!  We needn't limit the API to whatever the first
implementer does; all of the above are valid interpretations of what dedupe
could involve.  I might even argue that this is the use-case for the vectorized
dedupe call -- analyze all these proposed deduplications and figure out the
best plan to make it happen.

(Ofc the current implementation doesn't allow this, but we can change the
code when the singularity happens, etc.)

> But I take it that the existing btrfs ioctl doesn't work that way, and it always
> will share the existing extents in file 1 with file 2.
> 
> That's probably a perfectly fine way to do it, but I was wondering whether it
> really makes sense to hard-code that detail in the API, since an API could be
> defined more generally.
> 
> And from a practical perpective, people using the ioctl to deduplicate two files
> need to know which one to submit to the API as the "source" and which as the
> "destination", if it matters.
> 
> Related to this are the permissions you need on each file descriptor.  Since
> deduplication isn't "reading" or "writing" in the traditional sense it could,
> theoretically, be argued that no permissions at all should be required: neither
> FMODE_READ nor FMODE_WRITE.

I don't like the idea of being able to make major changes to a file that
I've opened with O_RDONLY. :)

I suppose if you're going to allow the FS to figure out its own strategy
and all that, then by the above reasoning one ought to have all the files
opened as writable.

> Most likely are concerns that would make that a bad idea, though.  Perhaps
> someone could create mischief by causing fragmentation in system files.
> 
> If this was previously discussed, can you point me to that discussion?

I don't know if there was a previous discussion, alas.

> The proposed man page is also very brief on permissions, only mentioning them
> indirectly in the error codes section.

There's a lot of unfortunate hand-waving in that manpage -- I wasn't involved
in designing the initial ioctl, so for the most part I'm simply sniffing my
way through based on observed behaviors (of duperemove) and guessing based on
the source code as I write the XFS version. :)

> > > The man page also says you get EPERM if "dest_fd is immutable" and ETXTBSY if
> > > "one of the files is a swap file", which I don't see actually happening in
> > > the implementation; it seems those error codes perhaps exist at all for this
> > > ioctl but rather be left to open(..., O_WRONLY).
> > 
> > Those could be hoisted to the VFS (from the XFS implementation), I think.
> 
> If the destination file has to already be open for writing then it can't be
> immutable.  So I don't think that error code is needed.  Checking for an active
> swapfile, on the other hand, may be valid, although I don't see such a check
> anywhere in the version of the code I'm looking at.

It's at the top of xfs_file_share_range() in the more recent RFCs of XFS reflink.

> > I'll later take a look at how many of these issues apply to clone/clone_range.
> 
> clone and clone_range seem to avoid the two biggest issues I saw: they aren't
> vectorized, and there's a natural distinction between the "source" and the
> "destination" of the operation.  They definitely need careful consideration as
> well, though.

Indeed.  I appreciate your review of the manpages/patches!

--D

> --
> To unsubscribe from this list: send the line "unsubscribe linux-api" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

From darrick.wong@oracle.com  Fri Jan 22 18:58:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 339557F56
	for ; Fri, 22 Jan 2016 18:58:40 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id C09A2AC002
	for ; Fri, 22 Jan 2016 16:58:39 -0800 (PST)
X-ASG-Debug-ID: 1453510717-04bdf04d6b6cb40001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id kYuVT8uhHkwHPDt4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 16:58:38 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N0wT7E003004
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:58:30 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0wT8D004475
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 00:58:29 GMT
Received: from abhmp0017.oracle.com (abhmp0017.oracle.com [141.146.116.23])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N0wTkk011643;
	Sat, 23 Jan 2016 00:58:29 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 16:58:29 -0800
Date: Fri, 22 Jan 2016 16:58:28 -0800
From: "Darrick J. Wong" 
To: viro@ZenIV.linux.org.uk
Cc: linux-fsdevel@vger.kernel.org, david@fromorbit.com,
        linux-api@vger.kernel.org, xfs@oss.sgi.com, ebiggers3@gmail.com
Subject: [PATCH] vfs: abort dedupe loop if fatal signals are pending
Message-ID: <20160123005828.GB5496@birch.djwong.org>
X-ASG-Orig-Subj: [PATCH] vfs: abort dedupe loop if fatal signals are pending
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453510718
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26370
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

If the program running dedupe receives a fatal signal during the
dedupe loop, we should bail out to avoid tying up the system.

Signed-off-by: Darrick J. Wong 
---
 fs/read_write.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/read_write.c b/fs/read_write.c
index aaaad52..aab334e 100644
--- a/fs/read_write.c
+++ b/fs/read_write.c
@@ -1617,6 +1617,9 @@ next_file:
 		mnt_drop_write_file(dst_file);
 next_loop:
 		fdput(dst_fd);
+
+		if (fatal_signal_pending(current))
+			goto out;
 	}
 
 out:

From viro@ftp.linux.org.uk  Fri Jan 22 19:30:30 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id BE7617F51
	for ; Fri, 22 Jan 2016 19:30:30 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id AC9768F8059
	for ; Fri, 22 Jan 2016 17:30:30 -0800 (PST)
X-ASG-Debug-ID: 1453512624-04cbb026fd72a50001-NocioJ
Received: from ZenIV.linux.org.uk (zeniv.linux.org.uk [195.92.253.2]) by cuda.sgi.com with ESMTP id 12bGfyvsNHTqBELw (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 22 Jan 2016 17:30:25 -0800 (PST)
X-Barracuda-Envelope-From: viro@ftp.linux.org.uk
X-Barracuda-Apparent-Source-IP: 195.92.253.2
Received: from viro by ZenIV.linux.org.uk with local (Exim 4.76 #1 (Red Hat Linux))
	id 1aMn1x-0001ao-FK; Sat, 23 Jan 2016 01:30:21 +0000
Date: Sat, 23 Jan 2016 01:30:21 +0000
From: Al Viro 
To: "Darrick J. Wong" 
Cc: linux-fsdevel@vger.kernel.org, david@fromorbit.com,
	linux-api@vger.kernel.org, xfs@oss.sgi.com, ebiggers3@gmail.com
Subject: Re: [PATCH] vfs: abort dedupe loop if fatal signals are pending
Message-ID: <20160123013021.GL17997@ZenIV.linux.org.uk>
X-ASG-Orig-Subj: Re: [PATCH] vfs: abort dedupe loop if fatal signals are pending
References: <20160123005828.GB5496@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160123005828.GB5496@birch.djwong.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: Al Viro 
X-Barracuda-Connect: zeniv.linux.org.uk[195.92.253.2]
X-Barracuda-Start-Time: 1453512625
X-Barracuda-Encrypted: DHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26371
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri, Jan 22, 2016 at 04:58:28PM -0800, Darrick J. Wong wrote:
> If the program running dedupe receives a fatal signal during the
> dedupe loop, we should bail out to avoid tying up the system.

Applied.

From darrick.wong@oracle.com  Fri Jan 22 19:32:02 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 42ACB7F4E
	for ; Fri, 22 Jan 2016 19:32:02 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id BB1F4AC001
	for ; Fri, 22 Jan 2016 17:31:58 -0800 (PST)
X-ASG-Debug-ID: 1453512716-04cb6c0d5f7d1b0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id Zllq6RsXCarn4kL1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 17:31:57 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N1VWuI004800
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 01:31:33 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0N1VWbJ030520
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 01:31:32 GMT
Received: from abhmp0018.oracle.com (abhmp0018.oracle.com [141.146.116.24])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0N1VWlj028413;
	Sat, 23 Jan 2016 01:31:32 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 17:31:31 -0800
Date: Fri, 22 Jan 2016 17:31:30 -0800
From: "Darrick J. Wong" 
To: david@fromorbit.com
Cc: Christoph Hellwig , xfs@oss.sgi.com
Subject: RFCv4+ WIP rollup of XFS reverse-mapping, reflink, and dedupe support
Message-ID: <20160123013130.GC5496@birch.djwong.org>
X-ASG-Orig-Subj: RFCv4+ WIP rollup of XFS reverse-mapping, reflink, and dedupe support
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453512717
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26371
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Hi all,

I pushed all the current reflink/rmap code to github.  This time there are some
incompat disk format changes, since I added a separate CoW extent size hint per
discussion with Dave; and moved the reflink inode flag out of the way of the
DAX inode flag.

The last patch in the kernel tree tries to record extents being used to
satisfy CoW requests in the refcount btree so that if we crash we can later
find those extents (which are no longer pending) and free them.  I'm abandoning
the use of EFIs to auto-zap the pending extents because I don't want the log
size to be an upper bound on the number of CoW IOs we can have in flight at a
given time.

Furthermore, I think EFIs will interact badly with cowextsize -- since we must
maintain an EFI for every extent that is alloc'd but not yet mapped, a writeout
in the middle of the extent when the rest of the extent isn't yet dirty
requires us to log an EFD to kill the first EFI and then log two more EFIs.  If
the log happens to be /full/ (of pending EFIs) when we do this, we won't have
space to log the extra EFIs and boom.  All this assumes that we can relog EFIs
to avoid crashing the log head into the tail, which we can't at the moment.

Avoid running reflink and rmap at the same time -- there are some xfs_buf
deadlocks that I haven't sorted out yet.  xfs/821 fails until I figure out
a permanent solution to per-AG pools.

On the plus side, there's now an inheritable CoW extent size hint so that we
can fight fragmentation, and a lot more testing of IO errors and fragmentation
problems that crop up with CoW.

I'll rebase on upstream after the merge window closes on Sunday.

https://github.com/djwong/linux/commits/for-dave
https://github.com/djwong/xfsprogs/tree/for-dave
https://github.com/djwong/xfstests/tree/for-dave
https://github.com/djwong/xfs-documentation/tree/for-dave

--D

From willy@linux.intel.com  Fri Jan 22 20:01:10 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 2933F7F4E
	for ; Fri, 22 Jan 2016 20:01:10 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 0A8CE8F804B
	for ; Fri, 22 Jan 2016 18:01:06 -0800 (PST)
X-ASG-Debug-ID: 1453514465-04cb6c0d5f7d8e0001-NocioJ
Received: from mga01.intel.com ([192.55.52.88]) by cuda.sgi.com with ESMTP id 9VbkrqHnaYS0b6sl for ; Fri, 22 Jan 2016 18:01:05 -0800 (PST)
X-Barracuda-Envelope-From: willy@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.88
Received: from orsmga003.jf.intel.com ([10.7.209.27])
  by fmsmga101.fm.intel.com with ESMTP; 22 Jan 2016 18:01:05 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,334,1449561600"; 
   d="scan'208";a="732672491"
Received: from agu1-mobl2.amr.corp.intel.com (HELO thog.int.wil.cx) ([10.254.78.135])
  by orsmga003.jf.intel.com with SMTP; 22 Jan 2016 18:01:05 -0800
Received: by thog.int.wil.cx (Postfix, from userid 1000)
	id AD5FA5FAD7; Fri, 22 Jan 2016 21:01:02 -0500 (EST)
Date: Fri, 22 Jan 2016 21:01:02 -0500
From: Matthew Wilcox 
To: Ross Zwisler 
Cc: linux-kernel@vger.kernel.org, Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams ,
	Dave Chinner , Jan Kara ,
	linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-nvdimm@lists.01.org, xfs@oss.sgi.com
Subject: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-ID: <20160123020102.GG2948@linux.intel.com>
X-ASG-Orig-Subj: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
References: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.88]
X-Barracuda-Start-Time: 1453514465
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 1.51
X-Barracuda-Spam-Status: No, SCORE=1.51 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA_TO_FROM_DOMAIN_MATCH, BSF_SC0_SA_TO_FROM_DOMAIN_MATCH_HL, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26371
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.01 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH Sender Domain Matches Recipient
	                           Domain
	1.40 BSF_SC0_SA_TO_FROM_DOMAIN_MATCH_HL Custom Rule
	                           SA_TO_FROM_DOMAIN_MATCH_HL

On Fri, Jan 22, 2016 at 04:06:11PM -0700, Ross Zwisler wrote:
> +++ b/fs/block_dev.c
> @@ -1733,13 +1733,28 @@ static const struct address_space_operations def_blk_aops = {
>   */
>  static int blkdev_dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
>  {
> -	return __dax_fault(vma, vmf, blkdev_get_block, NULL);
> +	int ret;
> +
> +	ret = __dax_fault(vma, vmf, blkdev_get_block, NULL, false);
> +
> +	if (WARN_ON_ONCE(ret == -EAGAIN))
> +		ret = VM_FAULT_SIGBUS;
> +
> +	return ret;
>  }

Let's not mix up -E returns and VM_FAULT returns.  We already have a
perfectly good VM_FAULT return value -- VM_FAULT_RETRY.

From sandeen@sandeen.net  Fri Jan 22 22:56:02 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id DA4737F3F
	for ; Fri, 22 Jan 2016 22:56:02 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 6ACD3AC001
	for ; Fri, 22 Jan 2016 20:55:59 -0800 (PST)
X-ASG-Debug-ID: 1453524953-04cb6c0d5d805b0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id C6nb1Au1sh11PXhG for ; Fri, 22 Jan 2016 20:55:53 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 3FD3F63D5B55
	for ; Fri, 22 Jan 2016 22:55:53 -0600 (CST)
Subject: Re: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
From: Eric Sandeen 
Message-ID: <56A307D8.60202@sandeen.net>
Date: Fri, 22 Jan 2016 22:55:52 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160123003502.2475.99558.stgit@birch.djwong.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453524953
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26374
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/22/16 6:35 PM, Darrick J. Wong wrote:
> Configure getopt to accept the -R argument to pwrite.

Did you have anything against "F" and "B"? :)

(They're missing too but in the case statements & help)

-Eric

> Signed-off-by: Darrick J. Wong 
> ---
>  io/pwrite.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> 
> diff --git a/io/pwrite.c b/io/pwrite.c
> index fd9114d..c6efee9 100644
> --- a/io/pwrite.c
> +++ b/io/pwrite.c
> @@ -259,7 +259,7 @@ pwrite_f(
>  	init_cvtnum(&fsblocksize, &fssectsize);
>  	bsize = fsblocksize;
>  
> -	while ((c = getopt(argc, argv, "b:Cdf:i:qs:S:uV:wWZ:")) != EOF) {
> +	while ((c = getopt(argc, argv, "b:Cdf:i:qRs:S:uV:wWZ:")) != EOF) {
>  		switch (c) {
>  		case 'b':
>  			tmp = cvtnum(fsblocksize, fssectsize, optarg);
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

From sandeen@sandeen.net  Fri Jan 22 23:08:18 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 009C07F51
	for ; Fri, 22 Jan 2016 23:08:18 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id E517A304032
	for ; Fri, 22 Jan 2016 21:08:14 -0800 (PST)
X-ASG-Debug-ID: 1453525693-04cb6c0d5e80920001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id pTr9PImUrlBRvapA for ; Fri, 22 Jan 2016 21:08:13 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 18A5B63D5B55
	for ; Fri, 22 Jan 2016 23:08:13 -0600 (CST)
Subject: Re: [PATCH 3/5] libxfs: move struct xfs_attr_shortform to
 xfs_da_format.h
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH 3/5] libxfs: move struct xfs_attr_shortform to
 xfs_da_format.h
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
 <20160123003514.2475.80318.stgit@birch.djwong.org>
From: Eric Sandeen 
Message-ID: <56A30ABC.9010508@sandeen.net>
Date: Fri, 22 Jan 2016 23:08:12 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160123003514.2475.80318.stgit@birch.djwong.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453525693
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26374
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/22/16 6:35 PM, Darrick J. Wong wrote:
> Move the shortform attr structure definition to the same place as the
> other attribute structure definitions for consistency and also so that
> xfs/122 verifies the structure size.
> 
> Signed-off-by: Darrick J. Wong 

Reviewed-by: Eric Sandeen 

> ---
>  libxfs/xfs_attr_sf.h    |   16 ----------------
>  libxfs/xfs_da_format.h  |   16 ++++++++++++++++
>  libxfs/xfs_inode_fork.c |    1 +
>  3 files changed, 17 insertions(+), 16 deletions(-)
> 
> 
> diff --git a/libxfs/xfs_attr_sf.h b/libxfs/xfs_attr_sf.h
> index 919756e..90928bb 100644
> --- a/libxfs/xfs_attr_sf.h
> +++ b/libxfs/xfs_attr_sf.h
> @@ -24,22 +24,6 @@
>   * Small attribute lists are packed as tightly as possible so as
>   * to fit into the literal area of the inode.
>   */
> -
> -/*
> - * Entries are packed toward the top as tight as possible.
> - */
> -typedef struct xfs_attr_shortform {
> -	struct xfs_attr_sf_hdr {	/* constant-structure header block */
> -		__be16	totsize;	/* total bytes in shortform list */
> -		__u8	count;	/* count of active entries */
> -	} hdr;
> -	struct xfs_attr_sf_entry {
> -		__uint8_t namelen;	/* actual length of name (no NULL) */
> -		__uint8_t valuelen;	/* actual length of value (no NULL) */
> -		__uint8_t flags;	/* flags bits (see xfs_attr_leaf.h) */
> -		__uint8_t nameval[1];	/* name & value bytes concatenated */
> -	} list[1];			/* variable sized array */
> -} xfs_attr_shortform_t;
>  typedef struct xfs_attr_sf_hdr xfs_attr_sf_hdr_t;
>  typedef struct xfs_attr_sf_entry xfs_attr_sf_entry_t;
>  
> diff --git a/libxfs/xfs_da_format.h b/libxfs/xfs_da_format.h
> index b14bbd6..8d4d8bc 100644
> --- a/libxfs/xfs_da_format.h
> +++ b/libxfs/xfs_da_format.h
> @@ -641,6 +641,22 @@ xfs_dir2_block_leaf_p(struct xfs_dir2_block_tail *btp)
>   */
>  #define XFS_ATTR_LEAF_MAPSIZE	3	/* how many freespace slots */
>  
> +/*
> + * Entries are packed toward the top as tight as possible.
> + */
> +typedef struct xfs_attr_shortform {
> +	struct xfs_attr_sf_hdr {	/* constant-structure header block */
> +		__be16	totsize;	/* total bytes in shortform list */
> +		__u8	count;	/* count of active entries */
> +	} hdr;
> +	struct xfs_attr_sf_entry {
> +		__uint8_t namelen;	/* actual length of name (no NULL) */
> +		__uint8_t valuelen;	/* actual length of value (no NULL) */
> +		__uint8_t flags;	/* flags bits (see xfs_attr_leaf.h) */
> +		__uint8_t nameval[1];	/* name & value bytes concatenated */
> +	} list[1];			/* variable sized array */
> +} xfs_attr_shortform_t;
> +
>  typedef struct xfs_attr_leaf_map {	/* RLE map of free bytes */
>  	__be16	base;			  /* base of free region */
>  	__be16	size;			  /* length of free region */
> diff --git a/libxfs/xfs_inode_fork.c b/libxfs/xfs_inode_fork.c
> index e1968b4..f40649e 100644
> --- a/libxfs/xfs_inode_fork.c
> +++ b/libxfs/xfs_inode_fork.c
> @@ -27,6 +27,7 @@
>  #include "xfs_bmap.h"
>  #include "xfs_trace.h"
>  #include "xfs_attr_sf.h"
> +#include "xfs_da_format.h"
>  
>  kmem_zone_t *xfs_ifork_zone;
>  
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

From darrick.wong@oracle.com  Sat Jan 23 00:18:23 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id BEEA07F47
	for ; Sat, 23 Jan 2016 00:18:23 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id A0C628F8049
	for ; Fri, 22 Jan 2016 22:18:20 -0800 (PST)
X-ASG-Debug-ID: 1453529894-04bdf04d6a72610001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id 373acw7kLON6Uzsd (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 22 Jan 2016 22:18:15 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0N6IEsx011313
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL);
	Sat, 23 Jan 2016 06:18:14 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0N6ID7E028060
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 06:18:13 GMT
Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0N6IDLZ027089;
	Sat, 23 Jan 2016 06:18:13 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 22 Jan 2016 22:18:12 -0800
Date: Fri, 22 Jan 2016 22:18:11 -0800
From: "Darrick J. Wong" 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
Message-ID: <20160123061811.GC5500@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
 <56A307D8.60202@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A307D8.60202@sandeen.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453529895
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26375
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Fri, Jan 22, 2016 at 10:55:52PM -0600, Eric Sandeen wrote:
> On 1/22/16 6:35 PM, Darrick J. Wong wrote:
> > Configure getopt to accept the -R argument to pwrite.
> 
> Did you have anything against "F" and "B"? :)
> 
> (They're missing too but in the case statements & help)

No, but I /was/ wondering if the omission was deliberate?

--D

> 
> -Eric
> 
> > Signed-off-by: Darrick J. Wong 
> > ---
> >  io/pwrite.c |    2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > 
> > diff --git a/io/pwrite.c b/io/pwrite.c
> > index fd9114d..c6efee9 100644
> > --- a/io/pwrite.c
> > +++ b/io/pwrite.c
> > @@ -259,7 +259,7 @@ pwrite_f(
> >  	init_cvtnum(&fsblocksize, &fssectsize);
> >  	bsize = fsblocksize;
> >  
> > -	while ((c = getopt(argc, argv, "b:Cdf:i:qs:S:uV:wWZ:")) != EOF) {
> > +	while ((c = getopt(argc, argv, "b:Cdf:i:qRs:S:uV:wWZ:")) != EOF) {
> >  		switch (c) {
> >  		case 'b':
> >  			tmp = cvtnum(fsblocksize, fssectsize, optarg);
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> > 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From fdmanana@gmail.com  Sat Jan 23 05:32:33 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,T_DKIM_INVALID
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id D0BE37F37
	for ; Sat, 23 Jan 2016 05:32:33 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id C0434304053
	for ; Sat, 23 Jan 2016 03:32:30 -0800 (PST)
X-ASG-Debug-ID: 1453548747-04cb6c0d5e89730001-NocioJ
Received: from mail-ig0-f193.google.com (mail-ig0-f193.google.com [209.85.213.193]) by cuda.sgi.com with ESMTP id VjraFakPozn8eR05 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 23 Jan 2016 03:32:28 -0800 (PST)
X-Barracuda-Envelope-From: fdmanana@gmail.com
X-Barracuda-Apparent-Source-IP: 209.85.213.193
Received: by mail-ig0-f193.google.com with SMTP id h5so1049697igh.0
        for ; Sat, 23 Jan 2016 03:32:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:reply-to:in-reply-to:references:date:message-id
         :subject:from:to:cc:content-type;
        bh=QX50PCrt49rBhjJVomUtd9XbLNNyRu4PO0ziRLM0K6A=;
        b=VnpZZI/RE4N4XuYGbHveguVTKypOcE4DUam152K/NV/YaLIcCDiERTHNTDHMtr8/yc
         9ETb2OQ045ucUPDoMQ24im3OEcvmh0XxQ1Zw/tOSMxmLlpIYFL6V9NCq+/xNoJ8270qI
         4DB8ZPoRloco/QyxT2x9+KMd/Z23vY6uta34kLd8QeLpuRbkdZGWHTQbQvibCgi6AuNa
         k0rjpRZ33NunX/xLPNP0Euco+bsf7HZ8lMUQOgA6cRtXIKvZ7Cxbflo9YhDpdE2US2yW
         7ZFYZKXjoML9rnUITtYuMB5VlIxTcp9uHmkReJ10DJw0n3GUZ9pO4dzlcRbJ9d9Rrtd5
         dr5g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:reply-to:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=QX50PCrt49rBhjJVomUtd9XbLNNyRu4PO0ziRLM0K6A=;
        b=Vs9xOOldw/Nmtj7vD+IZrUBVCexiFEv8mMxBCAEEIuFrdh4P4sl4BtynAn2CFnOtnb
         k4vY40Ig7zjFh5auhVccrt+wNjHJ/Q6iKcjReOQClpoq3xhZn7B+i0mmx3YkvJiTgYb+
         U5zqJhiaa/dVzk7GbbJkSy1x6TBaY/Ked9hIaaokLR09RjvlBJOjt2TvNKhj6SVfE3kl
         xRBp4dSX5aDlcJYEgIA0AhyDUUXskGGJWWGaAreWvBn1x190tjJ8ITCsWbm8NKBHv//T
         0ZEPork/TIvsI0tEFfnTHqAdqkZ0M5Uo3C0ZtblzXRBLbduXRMdOwXk57i0ZoftKMUNe
         I1eg==
X-Gm-Message-State: AG10YOTp9ZhK8/DgkBmiWP/eH/91KdwHtqW6k2zPFEURa1MtZ75q6CaQELqIl/zm3/zxaK0/KukwBTDzaH2+Ow==
MIME-Version: 1.0
X-Received: by 10.50.150.100 with SMTP id uh4mr7947589igb.45.1453548747714;
 Sat, 23 Jan 2016 03:32:27 -0800 (PST)
Received: by 10.36.192.134 with HTTP; Sat, 23 Jan 2016 03:32:27 -0800 (PST)
Reply-To: fdmanana@gmail.com
In-Reply-To: <20160123003654.2666.17112.stgit@birch.djwong.org>
References: <20160123003648.2666.37062.stgit@birch.djwong.org>
	<20160123003654.2666.17112.stgit@birch.djwong.org>
Date: Sat, 23 Jan 2016 11:32:27 +0000
Message-ID: 
Subject: Re: [PATCH 2/5] reflink: fix off-by-one errors when iterating file
 blocks in a loop
From: Filipe Manana 
X-ASG-Orig-Subj: Re: [PATCH 2/5] reflink: fix off-by-one errors when iterating file
 blocks in a loop
To: "Darrick J. Wong" 
Cc: Dave Chinner , fstests@vger.kernel.org, 
	"xfs@oss.sgi.com" 
Content-Type: text/plain; charset=UTF-8
X-Barracuda-Connect: mail-ig0-f193.google.com[209.85.213.193]
X-Barracuda-Start-Time: 1453548748
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26381
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

On Sat, Jan 23, 2016 at 12:36 AM, Darrick J. Wong
 wrote:
> When we're iterating file blocks in a loop (via seq), we have to
> end at $nr-1, not $nr.
>
> Signed-off-by: Darrick J. Wong 
> ---
>  tests/generic/183 |    2 +-
>  tests/generic/185 |    2 +-
>  tests/generic/186 |    2 +-
>  tests/generic/187 |    2 +-
>  tests/generic/194 |    2 +-
>  tests/generic/195 |    2 +-
>  tests/xfs/129     |    6 +++---
>  tests/xfs/140     |    2 +-
>  8 files changed, 10 insertions(+), 10 deletions(-)
>
>
> diff --git a/tests/generic/183 b/tests/generic/183
> index 5c65e9a..2cf9ce4 100755
> --- a/tests/generic/183
> +++ b/tests/generic/183
> @@ -69,7 +69,7 @@ seq 0 2 $((NR-1)) | while read f; do
>         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> -seq 1 2 $NR | while read f; do
> +seq 1 2 $((NR-1)) | while read f; do

Just curious (and no bike shedding intended), why not make the loop in
a much more simple and common way such as:

for ((i = 1; i < $NR; i += 2)); do
...
done

or  for i in `seq 1 2 $((NR - 1))`; do .... done



>         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> diff --git a/tests/generic/185 b/tests/generic/185
> index 384c9aa..55c54ac 100755
> --- a/tests/generic/185
> +++ b/tests/generic/185
> @@ -69,7 +69,7 @@ seq 0 2 $((NR-1)) | while read f; do
>         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> -seq 1 2 $NR | while read f; do
> +seq 1 2 $((NR-1)) | while read f; do
>         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> diff --git a/tests/generic/186 b/tests/generic/186
> index d84166d..f1c0e6b 100755
> --- a/tests/generic/186
> +++ b/tests/generic/186
> @@ -112,7 +112,7 @@ seq 0 2 $((NR-1)) | while read f; do
>         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> -seq 1 2 $NR | while read f; do
> +seq 1 2 $((NR-1)) | while read f; do
>         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> diff --git a/tests/generic/187 b/tests/generic/187
> index c85a0a9..ce35ec0 100755
> --- a/tests/generic/187
> +++ b/tests/generic/187
> @@ -112,7 +112,7 @@ seq 0 2 $((NR-1)) | while read f; do
>         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> -seq 1 2 $NR | while read f; do
> +seq 1 2 $((NR-1)) | while read f; do
>         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> diff --git a/tests/generic/194 b/tests/generic/194
> index a441aee..b88297e 100755
> --- a/tests/generic/194
> +++ b/tests/generic/194
> @@ -79,7 +79,7 @@ md5sum "$TESTDIR/file3" | _filter_scratch
>  md5sum "$TESTDIR/file3.chk" | _filter_scratch
>
>  echo "directio CoW across the transition"
> -seq 1 2 $NR | while read f; do
> +seq 1 2 $((NR-1)) | while read f; do
>         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
>         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> diff --git a/tests/generic/195 b/tests/generic/195
> index 83e7b23..464a4d4 100755
> --- a/tests/generic/195
> +++ b/tests/generic/195
> @@ -79,7 +79,7 @@ md5sum "$TESTDIR/file3" | _filter_scratch
>  md5sum "$TESTDIR/file3.chk" | _filter_scratch
>
>  echo "CoW across the transition"
> -seq 1 2 $NR | while read f; do
> +seq 1 2 $((NR-1)) | while read f; do
>         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
>         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>  done
> diff --git a/tests/xfs/129 b/tests/xfs/129
> index c8c47a2..3b7c59c 100755
> --- a/tests/xfs/129
> +++ b/tests/xfs/129
> @@ -65,9 +65,9 @@ NR_BLKS=$((4 * BLKSZ / 12))
>  _pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/file1" >> "$seqres.full"
>
>  echo "Reflink every other block"
> -seq 1 $((NR_BLKS / 2)) | while read nr; do
> -       _reflink_range  "$TESTDIR/file1" $((nr * 2 * BLKSZ)) \
> -                       "$TESTDIR/file2" $((nr * 2 * BLKSZ)) $BLKSZ >> "$seqres.full"
> +seq 1 2 $((NR_BLKS - 1)) | while read nr; do
> +       _reflink_range  "$TESTDIR/file1" $((nr * BLKSZ)) \
> +                       "$TESTDIR/file2" $((nr * BLKSZ)) $BLKSZ >> "$seqres.full"
>  done
>
>  echo "Create metadump file"
> diff --git a/tests/xfs/140 b/tests/xfs/140
> index e20cd3f..45be9f4 100644
> --- a/tests/xfs/140
> +++ b/tests/xfs/140
> @@ -70,7 +70,7 @@ cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "file1 and file2 do not match"
>  cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
>
>  echo "CoW every other block"
> -seq 1 2 $NR | while read f; do
> +seq 1 2 $((NR - 1)) | while read f; do
>         _pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
>         _pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
>  done
>
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

From bfoster@redhat.com  Sat Jan 23 07:13:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 1EFDF7F37
	for ; Sat, 23 Jan 2016 07:13:29 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 018CC304039
	for ; Sat, 23 Jan 2016 05:13:29 -0800 (PST)
X-ASG-Debug-ID: 1453554807-04bdf04d6a79a80001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 4c8YEloZFqmn4H2H (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 23 Jan 2016 05:13:27 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	by mx1.redhat.com (Postfix) with ESMTPS id AC7D18CB55
	for ; Sat, 23 Jan 2016 13:13:27 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-215.bos.redhat.com [10.18.41.215])
	by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0NDDRj8006838;
	Sat, 23 Jan 2016 08:13:27 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 88F5F1237FD; Sat, 23 Jan 2016 08:13:26 -0500 (EST)
Date: Sat, 23 Jan 2016 08:13:26 -0500
From: Brian Foster 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com, dwysocha@redhat.com
Subject: Re: [PATCH] mkfs.xfs.8: Clarify mkfs vs. mount block size limits.
Message-ID: <20160123131326.GB2795@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH] mkfs.xfs.8: Clarify mkfs vs. mount block size limits.
References: <56A02144.9030206@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A02144.9030206@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453554807
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 20, 2016 at 06:07:32PM -0600, Eric Sandeen wrote:
> The current explanation is a bit vague to the uninitiated;
> clarify the text to make it obvious that while the mkfs.xfs
> utility is able to create valid filesystems with block sizes
> up to 65536, the Linux kernel can only mount filesystems with
> page sized or smaller blocks.
> 
> Signed-off-by: Eric Sandeen 
> ---

Reviewed-by: Brian Foster 

> 
> diff --git a/man/man8/mkfs.xfs.8 b/man/man8/mkfs.xfs.8
> index 1fe510b..980c7bc 100644
> --- a/man/man8/mkfs.xfs.8
> +++ b/man/man8/mkfs.xfs.8
> @@ -126,7 +126,10 @@ or in bytes with
>  .BR size= .
>  The default value is 4096 bytes (4 KiB), the minimum is 512, and the
>  maximum is 65536 (64 KiB).
> -XFS on Linux currently only supports pagesize or smaller blocks.
> +Although
> +.B mkfs.xfs
> +will accept any of these values and create a valid filesystem,
> +XFS on Linux can only mount filesystems with pagesize or smaller blocks.
>  .TP
>  .BI \-m " global_metadata_options"
>  These options specify metadata format options that either apply to the entire
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Sat Jan 23 07:13:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id E14027F37
	for ; Sat, 23 Jan 2016 07:13:29 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id C49A28F804B
	for ; Sat, 23 Jan 2016 05:13:26 -0800 (PST)
X-ASG-Debug-ID: 1453554802-04bdf04d6d79a50001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id MoHDDSvEJW6v5Q0B (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 23 Jan 2016 05:13:22 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	by mx1.redhat.com (Postfix) with ESMTPS id 2AAC33B712
	for ; Sat, 23 Jan 2016 13:13:22 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-215.bos.redhat.com [10.18.41.215])
	by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0NDDLqS006821;
	Sat, 23 Jan 2016 08:13:21 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 003C31237FD; Sat, 23 Jan 2016 08:13:20 -0500 (EST)
Date: Sat, 23 Jan 2016 08:13:20 -0500
From: Brian Foster 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: don't overflow quota ID when initializing dqblk
Message-ID: <20160123131318.GA2795@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH] xfs: don't overflow quota ID when initializing dqblk
References: <56A01BB4.1050409@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A01BB4.1050409@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453554802
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 20, 2016 at 05:43:48PM -0600, Eric Sandeen wrote:
> Quota IDs are unsigned, and so we can pass in values up
> to 2^32-1.  But if we try to initialize a block containing
> values over MAX_INT, curid will overflow and assert.
> 
> curid holds a quota ID, so give it the proper
> xfs_dqid_t type (and remove the now-impossible ASSERT).
> 
> Signed-off-by: Eric Sandeen 
> ---

Reviewed-by: Brian Foster 

> 
> diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
> index 83ce4aa..5d92bb5 100644
> --- a/fs/xfs/xfs_dquot.c
> +++ b/fs/xfs/xfs_dquot.c
> @@ -232,7 +232,8 @@ xfs_qm_init_dquot_blk(
>  {
>  	struct xfs_quotainfo	*q = mp->m_quotainfo;
>  	xfs_dqblk_t	*d;
> -	int		curid, i;
> +	xfs_dqid_t	curid;
> +	int		i;
>  
>  	ASSERT(tp);
>  	ASSERT(xfs_buf_islocked(bp));
> @@ -243,7 +244,6 @@ xfs_qm_init_dquot_blk(
>  	 * ID of the first dquot in the block - id's are zero based.
>  	 */
>  	curid = id - (id % q->qi_dqperchunk);
> -	ASSERT(curid >= 0);
>  	memset(d, 0, BBTOB(q->qi_dqchunklen));
>  	for (i = 0; i < q->qi_dqperchunk; i++, d++, curid++) {
>  		d->dd_diskdq.d_magic = cpu_to_be16(XFS_DQUOT_MAGIC);
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From sandeen@sandeen.net  Sat Jan 23 12:03:08 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 41FE17F37
	for ; Sat, 23 Jan 2016 12:03:08 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 14911304039
	for ; Sat, 23 Jan 2016 10:03:07 -0800 (PST)
X-ASG-Debug-ID: 1453572181-04bdf04d6d81c80001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id vzfbSaG8DIE8osek for ; Sat, 23 Jan 2016 10:03:02 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 7E5D963D5841
	for ; Sat, 23 Jan 2016 12:03:01 -0600 (CST)
Subject: Re: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
 <56A307D8.60202@sandeen.net> <20160123061811.GC5500@birch.djwong.org>
From: Eric Sandeen 
Message-ID: <56A3C054.4000804@sandeen.net>
Date: Sat, 23 Jan 2016 12:03:00 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160123061811.GC5500@birch.djwong.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453572181
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26387
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------



On 1/23/16 12:18 AM, Darrick J. Wong wrote:
> On Fri, Jan 22, 2016 at 10:55:52PM -0600, Eric Sandeen wrote:
>> On 1/22/16 6:35 PM, Darrick J. Wong wrote:
>>> Configure getopt to accept the -R argument to pwrite.
>>
>> Did you have anything against "F" and "B"? :)
>>
>> (They're missing too but in the case statements & help)
> 
> No, but I /was/ wondering if the omission was deliberate?

I don't think so; the commit that added it (8fb2237)
seemed to do everything but add the getopt chars.

Oh, and however much gets fixed up, probably needs a man page
update too.

Thanks,
-Eric

> --D
> 
>>
>> -Eric
>>
>>> Signed-off-by: Darrick J. Wong 
>>> ---
>>>  io/pwrite.c |    2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>>
>>> diff --git a/io/pwrite.c b/io/pwrite.c
>>> index fd9114d..c6efee9 100644
>>> --- a/io/pwrite.c
>>> +++ b/io/pwrite.c
>>> @@ -259,7 +259,7 @@ pwrite_f(
>>>  	init_cvtnum(&fsblocksize, &fssectsize);
>>>  	bsize = fsblocksize;
>>>  
>>> -	while ((c = getopt(argc, argv, "b:Cdf:i:qs:S:uV:wWZ:")) != EOF) {
>>> +	while ((c = getopt(argc, argv, "b:Cdf:i:qRs:S:uV:wWZ:")) != EOF) {
>>>  		switch (c) {
>>>  		case 'b':
>>>  			tmp = cvtnum(fsblocksize, fssectsize, optarg);
>>>
>>> _______________________________________________
>>> xfs mailing list
>>> xfs@oss.sgi.com
>>> http://oss.sgi.com/mailman/listinfo/xfs
>>>
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

From darrick.wong@oracle.com  Sat Jan 23 14:52:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 840F77F37
	for ; Sat, 23 Jan 2016 14:52:29 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 11B6FAC002
	for ; Sat, 23 Jan 2016 12:52:25 -0800 (PST)
X-ASG-Debug-ID: 1453582342-04cb6c0d6092e80001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id ZSBiqgC3J6mBu1RA (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 23 Jan 2016 12:52:23 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0NKqDMC007928
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 20:52:13 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0NKqDYK013586
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Sat, 23 Jan 2016 20:52:13 GMT
Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0NKqCXP028771;
	Sat, 23 Jan 2016 20:52:12 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Sat, 23 Jan 2016 12:52:12 -0800
Date: Sat, 23 Jan 2016 12:52:11 -0800
From: "Darrick J. Wong" 
To: Filipe Manana 
Cc: Dave Chinner , fstests@vger.kernel.org,
        "xfs@oss.sgi.com" 
Subject: Re: [PATCH 2/5] reflink: fix off-by-one errors when iterating file
 blocks in a loop
Message-ID: <20160123205211.GB26408@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 2/5] reflink: fix off-by-one errors when iterating file
 blocks in a loop
References: <20160123003648.2666.37062.stgit@birch.djwong.org>
 <20160123003654.2666.17112.stgit@birch.djwong.org>
 
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453582343
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26390
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Sat, Jan 23, 2016 at 11:32:27AM +0000, Filipe Manana wrote:
> On Sat, Jan 23, 2016 at 12:36 AM, Darrick J. Wong
>  wrote:
> > When we're iterating file blocks in a loop (via seq), we have to
> > end at $nr-1, not $nr.
> >
> > Signed-off-by: Darrick J. Wong 
> > ---
> >  tests/generic/183 |    2 +-
> >  tests/generic/185 |    2 +-
> >  tests/generic/186 |    2 +-
> >  tests/generic/187 |    2 +-
> >  tests/generic/194 |    2 +-
> >  tests/generic/195 |    2 +-
> >  tests/xfs/129     |    6 +++---
> >  tests/xfs/140     |    2 +-
> >  8 files changed, 10 insertions(+), 10 deletions(-)
> >
> >
> > diff --git a/tests/generic/183 b/tests/generic/183
> > index 5c65e9a..2cf9ce4 100755
> > --- a/tests/generic/183
> > +++ b/tests/generic/183
> > @@ -69,7 +69,7 @@ seq 0 2 $((NR-1)) | while read f; do
> >         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
> >         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > -seq 1 2 $NR | while read f; do
> > +seq 1 2 $((NR-1)) | while read f; do
> 
> Just curious (and no bike shedding intended), why not make the loop in
> a much more simple and common way such as:
> 
> for ((i = 1; i < $NR; i += 2)); do
> ...
> done

Not that it matters here, but that's a bashism, isn't it?

> or  for i in `seq 1 2 $((NR - 1))`; do .... done

 No particular reason. :)

I suppose if NR is large then "seq | while read" avoids spraying a large number
of arguments into the command interpreter, though that clearly isn't the case
here.

--D

> 
> 
> 
> >         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > diff --git a/tests/generic/185 b/tests/generic/185
> > index 384c9aa..55c54ac 100755
> > --- a/tests/generic/185
> > +++ b/tests/generic/185
> > @@ -69,7 +69,7 @@ seq 0 2 $((NR-1)) | while read f; do
> >         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
> >         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > -seq 1 2 $NR | while read f; do
> > +seq 1 2 $((NR-1)) | while read f; do
> >         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > diff --git a/tests/generic/186 b/tests/generic/186
> > index d84166d..f1c0e6b 100755
> > --- a/tests/generic/186
> > +++ b/tests/generic/186
> > @@ -112,7 +112,7 @@ seq 0 2 $((NR-1)) | while read f; do
> >         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
> >         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > -seq 1 2 $NR | while read f; do
> > +seq 1 2 $((NR-1)) | while read f; do
> >         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > diff --git a/tests/generic/187 b/tests/generic/187
> > index c85a0a9..ce35ec0 100755
> > --- a/tests/generic/187
> > +++ b/tests/generic/187
> > @@ -112,7 +112,7 @@ seq 0 2 $((NR-1)) | while read f; do
> >         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
> >         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > -seq 1 2 $NR | while read f; do
> > +seq 1 2 $((NR-1)) | while read f; do
> >         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > diff --git a/tests/generic/194 b/tests/generic/194
> > index a441aee..b88297e 100755
> > --- a/tests/generic/194
> > +++ b/tests/generic/194
> > @@ -79,7 +79,7 @@ md5sum "$TESTDIR/file3" | _filter_scratch
> >  md5sum "$TESTDIR/file3.chk" | _filter_scratch
> >
> >  echo "directio CoW across the transition"
> > -seq 1 2 $NR | while read f; do
> > +seq 1 2 $((NR-1)) | while read f; do
> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > diff --git a/tests/generic/195 b/tests/generic/195
> > index 83e7b23..464a4d4 100755
> > --- a/tests/generic/195
> > +++ b/tests/generic/195
> > @@ -79,7 +79,7 @@ md5sum "$TESTDIR/file3" | _filter_scratch
> >  md5sum "$TESTDIR/file3.chk" | _filter_scratch
> >
> >  echo "CoW across the transition"
> > -seq 1 2 $NR | while read f; do
> > +seq 1 2 $((NR-1)) | while read f; do
> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
> >  done
> > diff --git a/tests/xfs/129 b/tests/xfs/129
> > index c8c47a2..3b7c59c 100755
> > --- a/tests/xfs/129
> > +++ b/tests/xfs/129
> > @@ -65,9 +65,9 @@ NR_BLKS=$((4 * BLKSZ / 12))
> >  _pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/file1" >> "$seqres.full"
> >
> >  echo "Reflink every other block"
> > -seq 1 $((NR_BLKS / 2)) | while read nr; do
> > -       _reflink_range  "$TESTDIR/file1" $((nr * 2 * BLKSZ)) \
> > -                       "$TESTDIR/file2" $((nr * 2 * BLKSZ)) $BLKSZ >> "$seqres.full"
> > +seq 1 2 $((NR_BLKS - 1)) | while read nr; do
> > +       _reflink_range  "$TESTDIR/file1" $((nr * BLKSZ)) \
> > +                       "$TESTDIR/file2" $((nr * BLKSZ)) $BLKSZ >> "$seqres.full"
> >  done
> >
> >  echo "Create metadump file"
> > diff --git a/tests/xfs/140 b/tests/xfs/140
> > index e20cd3f..45be9f4 100644
> > --- a/tests/xfs/140
> > +++ b/tests/xfs/140
> > @@ -70,7 +70,7 @@ cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "file1 and file2 do not match"
> >  cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
> >
> >  echo "CoW every other block"
> > -seq 1 2 $NR | while read f; do
> > +seq 1 2 $((NR - 1)) | while read f; do
> >         _pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
> >         _pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
> >  done
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe fstests" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> -- 
> Filipe David Manana,
> 
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."

From fdmanana@gmail.com  Sat Jan 23 16:00:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,T_DKIM_INVALID
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 5D26A7F47
	for ; Sat, 23 Jan 2016 16:00:13 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 0C4FCAC001
	for ; Sat, 23 Jan 2016 14:00:09 -0800 (PST)
X-ASG-Debug-ID: 1453586405-04bdf04d6c85fb0001-NocioJ
Received: from mail-io0-f193.google.com (mail-io0-f193.google.com [209.85.223.193]) by cuda.sgi.com with ESMTP id 1mixoPCu8SenrEfB (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sat, 23 Jan 2016 14:00:05 -0800 (PST)
X-Barracuda-Envelope-From: fdmanana@gmail.com
X-Barracuda-RBL-Trusted-Forwarder: 209.85.223.193
Received: by mail-io0-f193.google.com with SMTP id f81so10018939iof.0
        for ; Sat, 23 Jan 2016 14:00:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:reply-to:in-reply-to:references:date:message-id
         :subject:from:to:cc:content-type;
        bh=z7qxNIV/585/FexUEvGL6xqWLd8+Wdk3Shs6YfiR0ZY=;
        b=tKeQ0z0TmPy9zDalHtxQUMTEifRGbuUIUBYazgqk1M35R3uUXabzZCaIh4nsmZvA/P
         LbHoRAvsSFcdXX6pAM+lG5lozETOEJAJAbo1pD268wFszvmJB+IsGs63tDXzn1zcfNiz
         UF4lBEZoq4TVlghEVrbGkAD6N2fIByrN6z9KfIRfgAWIniGVv1DYBdTNx1yuErJ4ZlCb
         nr8s80IMqw5oQPsax/KE/zgS4Rz7nriZDbKkd6HpSoogj4AOufN2LHI8VqDC/POgyIyK
         usEQWgLjVp7JjWQvgq7PtIXw6uEyh+Z0JjIV8KUxYd4KFABDGhFiDaLEMfENvW783dUS
         RdzA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:reply-to:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=z7qxNIV/585/FexUEvGL6xqWLd8+Wdk3Shs6YfiR0ZY=;
        b=BtCDOAEp/dxu7kW4sbNAKhYjLmW3p8fAzv9LQnUG4LPzM/vKH0N9t4+GG/zuz6zjMc
         QeAT4GpZSTn7ViEqLZs8QAudXH1+hpt5UMWKIpOTyO8Mmr+u9IcpQkAts5t44ybvHrdD
         Vu8lHMYWoKXnPGgycKhRxYI4fKeoCjsdBZGBy5Z4bGTsYRytdsFp+fim3iRFD0HMbzer
         l1h+66pXnsqAClXW1xGfaVY7gM1viOM7yeJFy0yKdjFhZq9z2I6y/In3ZLk+i8jstWNd
         U8lD6X7+vk+HtbsLyGcakMXfusbvqrZvk1e7tKq3CYX62ukmuCKsdUGfxskLPrTwMQeB
         4Z+g==
X-Gm-Message-State: AG10YOThPMSbh8XlvKSbke93CmIhsoF1a1+YYPvbk/N9bM7alIQrUbRMH8cskcC2jz1EZOBPB7z4KMaxLxp8cA==
MIME-Version: 1.0
X-Received: by 10.107.130.209 with SMTP id m78mr9542771ioi.69.1453586405234;
 Sat, 23 Jan 2016 14:00:05 -0800 (PST)
Received: by 10.36.192.134 with HTTP; Sat, 23 Jan 2016 14:00:05 -0800 (PST)
Reply-To: fdmanana@gmail.com
In-Reply-To: <20160123205211.GB26408@birch.djwong.org>
References: <20160123003648.2666.37062.stgit@birch.djwong.org>
	<20160123003654.2666.17112.stgit@birch.djwong.org>
	
	<20160123205211.GB26408@birch.djwong.org>
Date: Sat, 23 Jan 2016 22:00:05 +0000
Message-ID: 
Subject: Re: [PATCH 2/5] reflink: fix off-by-one errors when iterating file
 blocks in a loop
From: Filipe Manana 
X-ASG-Orig-Subj: Re: [PATCH 2/5] reflink: fix off-by-one errors when iterating file
 blocks in a loop
To: "Darrick J. Wong" 
Cc: Dave Chinner , fstests@vger.kernel.org, 
	"xfs@oss.sgi.com" 
Content-Type: text/plain; charset=UTF-8
X-Barracuda-Connect: mail-io0-f193.google.com[209.85.223.193]
X-Barracuda-Start-Time: 1453586405
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26391
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature

On Sat, Jan 23, 2016 at 8:52 PM, Darrick J. Wong
 wrote:
> On Sat, Jan 23, 2016 at 11:32:27AM +0000, Filipe Manana wrote:
>> On Sat, Jan 23, 2016 at 12:36 AM, Darrick J. Wong
>>  wrote:
>> > When we're iterating file blocks in a loop (via seq), we have to
>> > end at $nr-1, not $nr.
>> >
>> > Signed-off-by: Darrick J. Wong 
>> > ---
>> >  tests/generic/183 |    2 +-
>> >  tests/generic/185 |    2 +-
>> >  tests/generic/186 |    2 +-
>> >  tests/generic/187 |    2 +-
>> >  tests/generic/194 |    2 +-
>> >  tests/generic/195 |    2 +-
>> >  tests/xfs/129     |    6 +++---
>> >  tests/xfs/140     |    2 +-
>> >  8 files changed, 10 insertions(+), 10 deletions(-)
>> >
>> >
>> > diff --git a/tests/generic/183 b/tests/generic/183
>> > index 5c65e9a..2cf9ce4 100755
>> > --- a/tests/generic/183
>> > +++ b/tests/generic/183
>> > @@ -69,7 +69,7 @@ seq 0 2 $((NR-1)) | while read f; do
>> >         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>> >         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > -seq 1 2 $NR | while read f; do
>> > +seq 1 2 $((NR-1)) | while read f; do
>>
>> Just curious (and no bike shedding intended), why not make the loop in
>> a much more simple and common way such as:
>>
>> for ((i = 1; i < $NR; i += 2)); do
>> ...
>> done
>
> Not that it matters here, but that's a bashism, isn't it?

Yep, but all tests and scripts (./check, common/rc, etc) start with a
shebang referring to bash.

>
>> or  for i in `seq 1 2 $((NR - 1))`; do .... done
>
>  No particular reason. :)
>
> I suppose if NR is large then "seq | while read" avoids spraying a large number
> of arguments into the command interpreter, though that clearly isn't the case
> here.

Right.
Thanks.

>
> --D
>
>>
>>
>>
>> >         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > diff --git a/tests/generic/185 b/tests/generic/185
>> > index 384c9aa..55c54ac 100755
>> > --- a/tests/generic/185
>> > +++ b/tests/generic/185
>> > @@ -69,7 +69,7 @@ seq 0 2 $((NR-1)) | while read f; do
>> >         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>> >         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > -seq 1 2 $NR | while read f; do
>> > +seq 1 2 $((NR-1)) | while read f; do
>> >         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > diff --git a/tests/generic/186 b/tests/generic/186
>> > index d84166d..f1c0e6b 100755
>> > --- a/tests/generic/186
>> > +++ b/tests/generic/186
>> > @@ -112,7 +112,7 @@ seq 0 2 $((NR-1)) | while read f; do
>> >         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>> >         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > -seq 1 2 $NR | while read f; do
>> > +seq 1 2 $((NR-1)) | while read f; do
>> >         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > diff --git a/tests/generic/187 b/tests/generic/187
>> > index c85a0a9..ce35ec0 100755
>> > --- a/tests/generic/187
>> > +++ b/tests/generic/187
>> > @@ -112,7 +112,7 @@ seq 0 2 $((NR-1)) | while read f; do
>> >         _reflink_range "$TESTDIR/file1" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>> >         _pwrite_byte 0x61 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > -seq 1 2 $NR | while read f; do
>> > +seq 1 2 $((NR-1)) | while read f; do
>> >         _reflink_range "$TESTDIR/file2" $((BLKSZ * f)) "$TESTDIR/file3" $((BLKSZ * f)) $BLKSZ >> "$seqres.full"
>> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > diff --git a/tests/generic/194 b/tests/generic/194
>> > index a441aee..b88297e 100755
>> > --- a/tests/generic/194
>> > +++ b/tests/generic/194
>> > @@ -79,7 +79,7 @@ md5sum "$TESTDIR/file3" | _filter_scratch
>> >  md5sum "$TESTDIR/file3.chk" | _filter_scratch
>> >
>> >  echo "directio CoW across the transition"
>> > -seq 1 2 $NR | while read f; do
>> > +seq 1 2 $((NR-1)) | while read f; do
>> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
>> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > diff --git a/tests/generic/195 b/tests/generic/195
>> > index 83e7b23..464a4d4 100755
>> > --- a/tests/generic/195
>> > +++ b/tests/generic/195
>> > @@ -79,7 +79,7 @@ md5sum "$TESTDIR/file3" | _filter_scratch
>> >  md5sum "$TESTDIR/file3.chk" | _filter_scratch
>> >
>> >  echo "CoW across the transition"
>> > -seq 1 2 $NR | while read f; do
>> > +seq 1 2 $((NR-1)) | while read f; do
>> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3" >> "$seqres.full"
>> >         _pwrite_byte 0x62 $((BLKSZ * f)) $BLKSZ "$TESTDIR/file3.chk" >> "$seqres.full"
>> >  done
>> > diff --git a/tests/xfs/129 b/tests/xfs/129
>> > index c8c47a2..3b7c59c 100755
>> > --- a/tests/xfs/129
>> > +++ b/tests/xfs/129
>> > @@ -65,9 +65,9 @@ NR_BLKS=$((4 * BLKSZ / 12))
>> >  _pwrite_byte 0x61 0 $((BLKSZ * NR_BLKS)) "$TESTDIR/file1" >> "$seqres.full"
>> >
>> >  echo "Reflink every other block"
>> > -seq 1 $((NR_BLKS / 2)) | while read nr; do
>> > -       _reflink_range  "$TESTDIR/file1" $((nr * 2 * BLKSZ)) \
>> > -                       "$TESTDIR/file2" $((nr * 2 * BLKSZ)) $BLKSZ >> "$seqres.full"
>> > +seq 1 2 $((NR_BLKS - 1)) | while read nr; do
>> > +       _reflink_range  "$TESTDIR/file1" $((nr * BLKSZ)) \
>> > +                       "$TESTDIR/file2" $((nr * BLKSZ)) $BLKSZ >> "$seqres.full"
>> >  done
>> >
>> >  echo "Create metadump file"
>> > diff --git a/tests/xfs/140 b/tests/xfs/140
>> > index e20cd3f..45be9f4 100644
>> > --- a/tests/xfs/140
>> > +++ b/tests/xfs/140
>> > @@ -70,7 +70,7 @@ cmp -s "$TESTDIR/file1" "$TESTDIR/file2" || echo "file1 and file2 do not match"
>> >  cmp -s "$TESTDIR/file2" "$TESTDIR/file2.chk" || echo "file2 and file2.chk do not match"
>> >
>> >  echo "CoW every other block"
>> > -seq 1 2 $NR | while read f; do
>> > +seq 1 2 $((NR - 1)) | while read f; do
>> >         _pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2" >> "$seqres.full"
>> >         _pwrite_byte 0x62 $((f * BLKSZ)) $BLKSZ "$TESTDIR/file2.chk" >> "$seqres.full"
>> >  done
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe fstests" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>> --
>> Filipe David Manana,
>>
>> "Reasonable men adapt themselves to the world.
>>  Unreasonable men adapt the world to themselves.
>>  That's why all progress depends on unreasonable men."



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

From debbugs@buxtehude.debian.org  Sun Jan 24 09:15:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 7ED477F37
	for ; Sun, 24 Jan 2016 09:15:13 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 6DE6F8F8039
	for ; Sun, 24 Jan 2016 07:15:13 -0800 (PST)
X-ASG-Debug-ID: 1453648510-04cbb026fc9d4b0001-NocioJ
Received: from buxtehude.debian.org (buxtehude.debian.org [206.12.19.147]) by cuda.sgi.com with ESMTP id VeZOtySYiy8UDJ8g (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 24 Jan 2016 07:15:10 -0800 (PST)
X-Barracuda-Envelope-From: debbugs@buxtehude.debian.org
X-Barracuda-Apparent-Source-IP: 206.12.19.147
Received: from debbugs by buxtehude.debian.org with local (Exim 4.84)
	(envelope-from )
	id 1aNMNg-0002ae-G0; Sun, 24 Jan 2016 15:15:08 +0000
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
From: owner@bugs.debian.org (Debian Bug Tracking System)
To: Jakub Wilk 
CC: xfs@oss.sgi.com
Subject: Processed: Re: Bug#766811: spurious library links
Message-ID: 
X-ASG-Orig-Subj: Processed: Re: Bug#766811: spurious library links
References: <20160124151308.GA7620@jwilk.net>
 <20141026004825.GA27863@bongo.bofh.it>
X-Debian-PR-Package: xfslibs-dev
X-Debian-PR-Source: xfsprogs
X-Debian-PR-Message: transcript
X-Loop: owner@bugs.debian.org
Date: Sun, 24 Jan 2016 15:15:08 +0000
X-Barracuda-Connect: buxtehude.debian.org[206.12.19.147]
X-Barracuda-Start-Time: 1453648510
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26408
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Processing control commands:

> found -1 4.3.0
Bug #766811 {Done: Marco d'Itri } [xfslibs-dev] spurious libra=
ry links
Marked as found in versions xfsprogs/4.3.0 and reopened.

--=20
766811: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D766811
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

From debbugs@buxtehude.debian.org  Sun Jan 24 09:15:16 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 055707F37
	for ; Sun, 24 Jan 2016 09:15:16 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E875A8F8039
	for ; Sun, 24 Jan 2016 07:15:12 -0800 (PST)
X-ASG-Debug-ID: 1453648510-04cb6c0d5da5480001-NocioJ
Received: from buxtehude.debian.org (buxtehude.debian.org [206.12.19.147]) by cuda.sgi.com with ESMTP id W1tTqNAhXhG2nbHz (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Sun, 24 Jan 2016 07:15:10 -0800 (PST)
X-Barracuda-Envelope-From: debbugs@buxtehude.debian.org
X-Barracuda-Apparent-Source-IP: 206.12.19.147
Received: from debbugs by buxtehude.debian.org with local (Exim 4.84)
	(envelope-from )
	id 1aNMNf-0002aV-Li; Sun, 24 Jan 2016 15:15:07 +0000
X-Loop: owner@bugs.debian.org
Subject: Bug#766811: spurious library links
Reply-To: Jakub Wilk , 766811@bugs.debian.org
X-ASG-Orig-Subj: Bug#766811: spurious library links
Resent-From: Jakub Wilk 
Resent-To: debian-bugs-dist@lists.debian.org
Resent-CC: XFS Development Team 
X-Loop: owner@bugs.debian.org
Resent-Date: Sun, 24 Jan 2016 15:15:05 +0000
Resent-Message-ID: 
X-Debian-PR-Message: followup 766811
X-Debian-PR-Package: xfslibs-dev
X-Debian-PR-Keywords: 
X-Debian-PR-Source: xfsprogs
Received: via spool by 766811-submit@bugs.debian.org id=B766811.14536483939559
          (code B ref 766811); Sun, 24 Jan 2016 15:15:05 +0000
Received: (at 766811) by bugs.debian.org; 24 Jan 2016 15:13:13 +0000
Received: from muffat.debian.org ([2607:f8f0:610:4000:6564:a62:ce0c:1392])
	from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=muffat.debian.org,EMAIL=hostmaster@muffat.debian.org (verified)
	by buxtehude.debian.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
	(Exim 4.84)
	(envelope-from )
	id 1aNMLp-0002U1-7m
	for 766811@bugs.debian.org; Sun, 24 Jan 2016 15:13:13 +0000
Received: from paradis2.debian.org ([2001:41c8:1000:21::21:31] helo=paradis.debian.org)
	from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=paradis.debian.org,EMAIL=hostmaster@paradis.debian.org (verified)
	by muffat.debian.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
	(Exim 4.84)
	(envelope-from )
	id 1aNMLp-0008Pf-0z; Sun, 24 Jan 2016 15:13:13 +0000
Received: from jwilk by paradis.debian.org with local (Exim 4.84)
	(envelope-from )
	id 1aNMLn-0008Fl-1K; Sun, 24 Jan 2016 15:13:11 +0000
Date: Sun, 24 Jan 2016 16:13:08 +0100
From: Jakub Wilk 
To: 766811@bugs.debian.org
Cc: Marco d'Itri 
Message-ID: <20160124151308.GA7620@jwilk.net>
References: <20141026004825.GA27863@bongo.bofh.it>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <20141026004825.GA27863@bongo.bofh.it>
User-Agent: Mutt/1.5.23.1 (2014-03-12)
X-Barracuda-Connect: buxtehude.debian.org[206.12.19.147]
X-Barracuda-Start-Time: 1453648510
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26408
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Control: found -1 4.3.0

* Marco d'Itri , 2014-10-26, 02:48:
>These links do not appear to have any purpose and should be removed:
>
>/lib/libhandle.a -> /usr/lib/libhandle.a
>/lib/libhandle.la -> /usr/lib/libhandle.la
>/usr/lib/libhandle.so -> /lib/libhandle.so

I can still see them in the current version:

$ dpkg -c xfslibs-dev_4.3.0_amd64.deb | grep ^l.*/usr/lib/libhandle
lrwxrwxrwx root/root         0 2015-11-27 01:33 ./usr/lib/libhandle.so -> /lib/libhandle.so
lrwxrwxrwx root/root         0 2015-11-27 01:33 ./lib/libhandle.a -> /usr/lib/libhandle.a
lrwxrwxrwx root/root         0 2015-11-27 01:33 ./lib/libhandle.la -> /usr/lib/libhandle.la

-- 
Jakub Wilk

From qboosh@pld-linux.org  Sun Jan 24 09:56:27 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 5E9407F37
	for ; Sun, 24 Jan 2016 09:56:27 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 4E833304032
	for ; Sun, 24 Jan 2016 07:56:23 -0800 (PST)
X-ASG-Debug-ID: 1453650974-04cbb026fe9e0d0001-NocioJ
Received: from mail.qboosh.pl (mail.qboosh.pl [217.73.31.61]) by cuda.sgi.com with ESMTP id xlI6nyz2P4GKYJvj (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 24 Jan 2016 07:56:15 -0800 (PST)
X-Barracuda-Envelope-From: qboosh@pld-linux.org
X-Barracuda-Apparent-Source-IP: 217.73.31.61
Received: from stranger.qboosh.pl (83-238-154-144.static.ip.netia.com.pl [83.238.154.144])
	by mail.qboosh.pl (Postfix) with ESMTPSA id 52E0B1A26DA2;
	Sun, 24 Jan 2016 16:56:13 +0100 (CET)
Received: from stranger.qboosh.pl (localhost [127.0.0.1])
	by stranger.qboosh.pl (8.14.8/8.14.8) with ESMTP id u0OFwsMN027970;
	Sun, 24 Jan 2016 16:58:54 +0100
Received: (from qboosh@localhost)
	by stranger.qboosh.pl (8.14.8/8.14.8/Submit) id u0OFwr44027963;
	Sun, 24 Jan 2016 16:58:53 +0100
Date: Sun, 24 Jan 2016 16:58:53 +0100
From: Jakub Bogusz 
To: xfs@oss.sgi.com
Cc: Christoph Hellwig 
Subject: Polish translation update + localization remark for xfsprogs 4.3.0
Message-ID: <20160124155853.GA10063@stranger.qboosh.pl>
X-ASG-Orig-Subj: Polish translation update + localization remark for xfsprogs 4.3.0
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mail.qboosh.pl[217.73.31.61]
X-Barracuda-Start-Time: 1453650975
X-Barracuda-Encrypted: DHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26409
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Hello,

I've just updated Polish translation for xfsprogs 4.3.0, it's available
at:
http://qboosh.pl/pl.po/xfsprogs-4.3.0.pl.po

I have noticed untranslatable message part in repair/versions.c:228:

_("WARNING: you have a V1 inode filesystem. It %s be converted to a\n"
  "\tversion 2 inode filesystem. If you do not want this, run an older\n"
  "\tversion of xfs_repair.\n"), no_modify ? "would" : "will");

There are untranslatable "would"/"will" strings.
This code could look for example like this:

_("WARNING: you have a V1 inode filesystem. %s to a\n"
  "\tversion 2 inode filesystem. If you do not want this, run an older\n"
  "\tversion of xfs_repair.\n"),
  no_modify ? _("It would be converted") : _("It will be converted"));

(note: using _("would") : _("will") is not enough - these words are
untranslatable to many languages without wider context)


-- 
Jakub Bogusz    http://qboosh.pl/

From sandeen@sandeen.net  Sun Jan 24 11:08:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 27F3B7F37
	for ; Sun, 24 Jan 2016 11:08:31 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 18A57304039
	for ; Sun, 24 Jan 2016 09:08:27 -0800 (PST)
X-ASG-Debug-ID: 1453655305-04cbb026fe9f7e0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id eWCR4HuHWCjTN2n8 for ; Sun, 24 Jan 2016 09:08:25 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 694E061A5FC7
	for ; Sun, 24 Jan 2016 11:08:25 -0600 (CST)
Subject: [PATCH] xfs_repair: Fix untranslatable strings
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH] xfs_repair: Fix untranslatable strings
References: <20160124155853.GA10063@stranger.qboosh.pl>
From: Eric Sandeen 
Message-ID: <56A50508.4060207@sandeen.net>
Date: Sun, 24 Jan 2016 11:08:24 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160124155853.GA10063@stranger.qboosh.pl>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453655305
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26410
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Don't substitute in "would" or "will" for %s here, it makes
it untranslatable.  Just spell it out twice like every other
message in the function.

Reported-by: Jakub Bogusz 
Signed-off-by: Eric Sandeen 
---

diff --git a/repair/versions.c b/repair/versions.c
index 1585385..978f48b 100644
--- a/repair/versions.c
+++ b/repair/versions.c
@@ -224,10 +224,17 @@ _("WARNING:  you have disallowed attr2 attributes but this filesystem\n"
 	}
 
 	if (!(sb->sb_versionnum & XFS_SB_VERSION_NLINKBIT)) {
-		do_warn(
-_("WARNING: you have a V1 inode filesystem. It %s be converted to a\n"
+		if (!no_modify) {
+			do_warn(
+_("WARNING: you have a V1 inode filesystem. It will be converted to a\n"
   "\tversion 2 inode filesystem. If you do not want this, run an older\n"
-  "\tversion of xfs_repair.\n"), no_modify ? "would" : "will");
+  "\tversion of xfs_repair.\n"));
+		} else  {
+			do_warn(
+_("WARNING: you have a V1 inode filesystem. It would be converted to a\n"
+  "\tversion 2 inode filesystem. If you do not want this, run an older\n"
+  "\tversion of xfs_repair.\n"));
+		}
 	}
 
 	if (xfs_sb_version_hasquota(sb))  {

From david@fromorbit.com  Sun Jan 24 16:01:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id CDA987F37
	for ; Sun, 24 Jan 2016 16:01:40 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 6F08FAC002
	for ; Sun, 24 Jan 2016 14:01:37 -0800 (PST)
X-ASG-Debug-ID: 1453672894-04bdf04d6a9e7b0001-NocioJ
Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by cuda.sgi.com with ESMTP id 17N3fMWex1iEUfj1 for ; Sun, 24 Jan 2016 14:01:34 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.129
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2BODAAxSaVWPDYJLHleKAECgw+BP4ZfgXieSgEBAQEBAQaLXIVChAOGCQQCAoEeTQEBAQEBAQcBAQEBQAE/hEEBAQEDAScTHCMFCwgDGAklDwUlAwcaExuHeAe8UAEBCAIeGIVShTWJBgWWdo1NgWeHa4Uwjj+CYxiBZCguhzwBAQE
Received: from ppp121-44-9-54.lns20.syd4.internode.on.net (HELO dastard) ([121.44.9.54])
  by ipmail06.adl2.internode.on.net with ESMTP; 25 Jan 2016 08:31:08 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aNSiZ-0002Rc-UD; Mon, 25 Jan 2016 09:01:07 +1100
Date: Mon, 25 Jan 2016 09:01:07 +1100
From: Dave Chinner 
To: Ross Zwisler 
Cc: linux-kernel@vger.kernel.org, Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams , Jan Kara ,
	Matthew Wilcox , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-nvdimm@lists.01.org,
	xfs@oss.sgi.com
Subject: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-ID: <20160124220107.GI20456@dastard>
X-ASG-Orig-Subj: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
References: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail06.adl2.internode.on.net[150.101.137.129]
X-Barracuda-Start-Time: 1453672894
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26416
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri, Jan 22, 2016 at 04:06:11PM -0700, Ross Zwisler wrote:
> With the current DAX code the following race exists:
> 
> Process 1                	Process 2
> ---------			---------
> 
> __dax_fault() - read file f, index 0
>   get_block() -> returns hole
>                              	__dax_fault() - write file f, index 0
>                                   get_block() -> allocates blocks
>                                   dax_insert_mapping()
>   dax_load_hole()
>   *data corruption*
> 
> An analogous race exists between __dax_fault() loading a hole and
> __dax_pmd_fault() allocating a PMD DAX page and trying to insert it, and
> that race also ends in data corruption.

Ok, so why doesn't this problem exist for the normal page cache
insertion case with concurrent read vs write faults?  It's because
the write fault first does a read fault and so always the write
fault always has a page in the radix tree for the get_block call
that allocates the extents, right?

And DAX has an optimisation in the page fault part where it skips
the read fault part of the write fault?  And so essentially the DAX
write fault is missing the object (page lock of page in the radix
tree) that the non-DAX write fault uses to avoid this problem?

What happens if we get rid of that DAX write fault optimisation that
skips the initial read fault? The write fault will always run on a
mapping that has a hole loaded, right?, so the race between
dax_load_hole() and dax_insert_mapping() goes away, because nothing
will be calling dax_load_hole() once the write fault is allocating
blocks....

> One solution to this race was proposed by Jan Kara:
> 
>   So we need some exclusion that makes sure pgoff->block mapping
>   information is uptodate at the moment we insert it into page tables. The
>   simplest reasonably fast thing I can see is:
> 
>   When handling a read fault, things stay as is and filesystem protects the
>   fault with an equivalent of EXT4_I(inode)->i_mmap_sem held for reading.
>   When handling a write fault we first grab EXT4_I(inode)->i_mmap_sem for
>   reading and try a read fault. If __dax_fault() sees a hole returned from
>   get_blocks() during a write fault, it bails out. Filesystem grabs
>   EXT4_I(inode)->i_mmap_sem for writing and retries with different
>   get_blocks() callback which will allocate blocks. That way we get proper
>   exclusion for faults needing to allocate blocks.
> 
> This patch adds this logic to DAX, ext2, ext4 and XFS.

It's too ugly to live. It hacks around a special DAX optimisation in
the fault code by adding special case locking to the filesystems,
and adds a siginificant new locking constraint to the page fault
path.

If the write fault first goes through the read fault path and loads
the hole, this race condition simply does not exist. I'd suggest
that we get rid of the DAX optimisation that skips read fault
processing on write fault so that this problem simply goes away.
Yes, it means write faults on holes will be a little slower (which,
quite frankly, I don't care at all about), but it means we don't
need to hack special cases into code that should not have to care
about various different types of page fault races. Correctness
first, speed later.

FWIW, this also means we can get rid of the hacks in the filesystem
code where we have to handle write faults through the ->fault
handler rather than the ->page_mkwrite handler.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From tdm@sgi.com  Sun Jan 24 21:11:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,RP_MATCHES_RCVD
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 53F6D7F37
	for ; Sun, 24 Jan 2016 21:11:41 -0600 (CST)
Received: from xmail.sgi.com (pv-excas1-dc21.corp.sgi.com [137.38.106.7])
	by relay3.corp.sgi.com (Postfix) with ESMTP id D8E9CAC001
	for ; Sun, 24 Jan 2016 19:11:37 -0800 (PST)
Received: from P-EXMB4-DC21.corp.sgi.com ([169.254.6.28]) by
 PV-EXCAS1-DC21.corp.sgi.com ([137.38.106.7]) with mapi id 14.03.0210.002;
 Sun, 24 Jan 2016 21:11:37 -0600
From: Troy McCorkell 
To: "xfs@oss.sgi.com" 
Subject: oss.sgi.com - maintenance Jan 25 08:00 CT USA
Thread-Topic: oss.sgi.com - maintenance Jan 25 08:00 CT USA
Thread-Index: AdFXHhnqa0ZqzAQZTXmzURG/JETbDw==
Date: Mon, 25 Jan 2016 03:11:36 +0000
Message-ID: 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [134.15.0.138]
Content-Type: multipart/alternative;
	boundary="_000_DF733A1EC32F124D86BC1188068FECB101A72585FEPEXMB4DC21cor_"
MIME-Version: 1.0

--_000_DF733A1EC32F124D86BC1188068FECB101A72585FEPEXMB4DC21cor_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

oss.sgi.com will be down approximately 30 minutes starting at 08:00 CT USA
on Monday January 25th for maintenance.

--_000_DF733A1EC32F124D86BC1188068FECB101A72585FEPEXMB4DC21cor_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable








oss.sgi.com will be down approximately 30 minutes starting at 08:00 =
CT USA

on Monday January 25th for maintenance.






--_000_DF733A1EC32F124D86BC1188068FECB101A72585FEPEXMB4DC21cor_--

From jack@suse.cz  Mon Jan 25 07:59:17 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 26C747F37
	for ; Mon, 25 Jan 2016 07:59:17 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 16E24304032
	for ; Mon, 25 Jan 2016 05:59:13 -0800 (PST)
X-ASG-Debug-ID: 1453730350-04cbb026feb8780001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id OeYZrNotFn9TYBFG (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 25 Jan 2016 05:59:11 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 249F5ABB0;
	Mon, 25 Jan 2016 13:59:08 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id D4BA4823DA; Mon, 25 Jan 2016 14:59:21 +0100 (CET)
Date: Mon, 25 Jan 2016 14:59:21 +0100
From: Jan Kara 
To: Dave Chinner 
Cc: Ross Zwisler ,
	linux-kernel@vger.kernel.org, Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams , Jan Kara ,
	Matthew Wilcox , linux-ext4@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, linux-nvdimm@lists.01.org,
	xfs@oss.sgi.com
Subject: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-ID: <20160125135921.GE24938@quack.suse.cz>
X-ASG-Orig-Subj: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
References: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
 <20160124220107.GI20456@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160124220107.GI20456@dastard>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453730351
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26432
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Mon 25-01-16 09:01:07, Dave Chinner wrote:
> On Fri, Jan 22, 2016 at 04:06:11PM -0700, Ross Zwisler wrote:
> > With the current DAX code the following race exists:
> > 
> > Process 1                	Process 2
> > ---------			---------
> > 
> > __dax_fault() - read file f, index 0
> >   get_block() -> returns hole
> >                              	__dax_fault() - write file f, index 0
> >                                   get_block() -> allocates blocks
> >                                   dax_insert_mapping()
> >   dax_load_hole()
> >   *data corruption*
> > 
> > An analogous race exists between __dax_fault() loading a hole and
> > __dax_pmd_fault() allocating a PMD DAX page and trying to insert it, and
> > that race also ends in data corruption.
> 
> Ok, so why doesn't this problem exist for the normal page cache
> insertion case with concurrent read vs write faults?  It's because
> the write fault first does a read fault and so always the write
> fault always has a page in the radix tree for the get_block call
> that allocates the extents, right?

Yes, any fault (read or write) has a page to lock which avoids races for
normal fault path.

> And DAX has an optimisation in the page fault part where it skips
> the read fault part of the write fault?  And so essentially the DAX
> write fault is missing the object (page lock of page in the radix
> tree) that the non-DAX write fault uses to avoid this problem?
> 
> What happens if we get rid of that DAX write fault optimisation that
> skips the initial read fault? The write fault will always run on a
> mapping that has a hole loaded, right?, so the race between
> dax_load_hole() and dax_insert_mapping() goes away, because nothing
> will be calling dax_load_hole() once the write fault is allocating
> blocks....

So frankly I don't like mixing of page locks into the DAX fault locking.
Also your scheme would require more tricks to deal with races between PMD
write faults racing with PTE read faults since you don't want to require
2MB worth of hole-pages to be able to do a PMD write fault. Transparent
huge pages deal with this issue using compound pages but I'd like to avoid
that horror in the DAX path...

> > One solution to this race was proposed by Jan Kara:
> > 
> >   So we need some exclusion that makes sure pgoff->block mapping
> >   information is uptodate at the moment we insert it into page tables. The
> >   simplest reasonably fast thing I can see is:
> > 
> >   When handling a read fault, things stay as is and filesystem protects the
> >   fault with an equivalent of EXT4_I(inode)->i_mmap_sem held for reading.
> >   When handling a write fault we first grab EXT4_I(inode)->i_mmap_sem for
> >   reading and try a read fault. If __dax_fault() sees a hole returned from
> >   get_blocks() during a write fault, it bails out. Filesystem grabs
> >   EXT4_I(inode)->i_mmap_sem for writing and retries with different
> >   get_blocks() callback which will allocate blocks. That way we get proper
> >   exclusion for faults needing to allocate blocks.
> > 
> > This patch adds this logic to DAX, ext2, ext4 and XFS.
> 
> It's too ugly to live. It hacks around a special DAX optimisation in
> the fault code by adding special case locking to the filesystems,
> and adds a siginificant new locking constraint to the page fault
> path.
> 
> If the write fault first goes through the read fault path and loads
> the hole, this race condition simply does not exist. I'd suggest
> that we get rid of the DAX optimisation that skips read fault
> processing on write fault so that this problem simply goes away.
> Yes, it means write faults on holes will be a little slower (which,
> quite frankly, I don't care at all about), but it means we don't
> need to hack special cases into code that should not have to care
> about various different types of page fault races. Correctness
> first, speed later.
> 
> FWIW, this also means we can get rid of the hacks in the filesystem
> code where we have to handle write faults through the ->fault
> handler rather than the ->page_mkwrite handler.

So I don't mind doing read-fault first. But as I wrote above I don't think
it solves all the issues. The rule I wanted to introduce is:

1) Fault requiring block allocation require exclusive lock held over the
whole fault.

2) Fault not requiring allocation is enough with shared lock held over the
whole fault.

So we can certainly grab i_mmap_sem exclusively whenever we hit a write
fault and it would be correct. This looks like a quite clean design to me.

As a performance optimization (which is upto each filesystem) we can try to
satisfy write fault without allocation while holding i_mmap_sem in a shared
mode and if that fails, grab the lock exclusively and retry. Still don't like
it?

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Mon Jan 25 08:46:56 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 2D3C27F37
	for ; Mon, 25 Jan 2016 08:46:56 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id E5ED7304032
	for ; Mon, 25 Jan 2016 06:46:52 -0800 (PST)
X-ASG-Debug-ID: 1453733209-04bdf04d6ab3b50001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id y3GpoCsVrYbklP7N (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 25 Jan 2016 06:46:50 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 03714ABB0;
	Mon, 25 Jan 2016 14:46:48 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 9056F823DA; Mon, 25 Jan 2016 15:47:02 +0100 (CET)
Date: Mon, 25 Jan 2016 15:47:02 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz
Subject: Re: [PATCH 1/7] quota: remove unused cmd argument from
 quota_quotaon()
Message-ID: <20160125144702.GJ24938@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 1/7] quota: remove unused cmd argument from
 quota_quotaon()
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-2-git-send-email-sandeen@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453487136-12681-2-git-send-email-sandeen@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453733210
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26433
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri 22-01-16 12:25:30, Eric Sandeen wrote:
> The cmd argument to quota_quotaon() via Q_QUOTAON quotactl
> is not used, so remove it.
> 
> Signed-off-by: Eric Sandeen 

Looks good. You can add:

Reviewed-by: Jan Kara 

								Honza

> ---
>  fs/quota/quota.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/quota/quota.c b/fs/quota/quota.c
> index 3746367..ea66670 100644
> --- a/fs/quota/quota.c
> +++ b/fs/quota/quota.c
> @@ -79,7 +79,7 @@ unsigned int qtype_enforce_flag(int type)
>  	return 0;
>  }
>  
> -static int quota_quotaon(struct super_block *sb, int type, int cmd, qid_t id,
> +static int quota_quotaon(struct super_block *sb, int type, qid_t id,
>  		         struct path *path)
>  {
>  	if (!sb->s_qcop->quota_on && !sb->s_qcop->quota_enable)
> @@ -659,7 +659,7 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
>  
>  	switch (cmd) {
>  	case Q_QUOTAON:
> -		return quota_quotaon(sb, type, cmd, id, path);
> +		return quota_quotaon(sb, type, id, path);
>  	case Q_QUOTAOFF:
>  		return quota_quotaoff(sb, type);
>  	case Q_GETFMT:
> -- 
> 1.7.1
> 
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Mon Jan 25 08:49:18 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 67E457CBD
	for ; Mon, 25 Jan 2016 08:49:18 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 2C8C78F8049
	for ; Mon, 25 Jan 2016 06:49:18 -0800 (PST)
X-ASG-Debug-ID: 1453733355-04bdf04d6ab3c30001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id xATlRAmT0yeLaYKH (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 25 Jan 2016 06:49:16 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id AD042ABB0;
	Mon, 25 Jan 2016 14:49:13 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 4F51E823DA; Mon, 25 Jan 2016 15:49:28 +0100 (CET)
Date: Mon, 25 Jan 2016 15:49:28 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz
Subject: Re: [PATCH 1/7] quota: remove unused cmd argument from
 quota_quotaon()
Message-ID: <20160125144928.GK24938@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 1/7] quota: remove unused cmd argument from
 quota_quotaon()
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-2-git-send-email-sandeen@redhat.com>
 <20160125144702.GJ24938@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160125144702.GJ24938@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453733355
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26433
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Mon 25-01-16 15:47:02, Jan Kara wrote:
> On Fri 22-01-16 12:25:30, Eric Sandeen wrote:
> > The cmd argument to quota_quotaon() via Q_QUOTAON quotactl
> > is not used, so remove it.
> > 
> > Signed-off-by: Eric Sandeen 
> 
> Looks good. You can add:
> 
> Reviewed-by: Jan Kara 

Ah, I'm in a reviewe mode... ;) I'll just take this patch to my tree.

								Honza

> > ---
> >  fs/quota/quota.c |    4 ++--
> >  1 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/fs/quota/quota.c b/fs/quota/quota.c
> > index 3746367..ea66670 100644
> > --- a/fs/quota/quota.c
> > +++ b/fs/quota/quota.c
> > @@ -79,7 +79,7 @@ unsigned int qtype_enforce_flag(int type)
> >  	return 0;
> >  }
> >  
> > -static int quota_quotaon(struct super_block *sb, int type, int cmd, qid_t id,
> > +static int quota_quotaon(struct super_block *sb, int type, qid_t id,
> >  		         struct path *path)
> >  {
> >  	if (!sb->s_qcop->quota_on && !sb->s_qcop->quota_enable)
> > @@ -659,7 +659,7 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
> >  
> >  	switch (cmd) {
> >  	case Q_QUOTAON:
> > -		return quota_quotaon(sb, type, cmd, id, path);
> > +		return quota_quotaon(sb, type, id, path);
> >  	case Q_QUOTAOFF:
> >  		return quota_quotaoff(sb, type);
> >  	case Q_GETFMT:
> > -- 
> > 1.7.1
> > 
> -- 
> Jan Kara 
> SUSE Labs, CR
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Mon Jan 25 08:51:10 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 3E29B7F37
	for ; Mon, 25 Jan 2016 08:51:10 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id C3C57AC002
	for ; Mon, 25 Jan 2016 06:51:06 -0800 (PST)
X-ASG-Debug-ID: 1453733464-04bdf04d6db3ce0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id bfZikI0brjcu4hgU (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 25 Jan 2016 06:51:05 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id B38C9ABB0;
	Mon, 25 Jan 2016 14:51:02 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 45B14823DA; Mon, 25 Jan 2016 15:51:17 +0100 (CET)
Date: Mon, 25 Jan 2016 15:51:17 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz
Subject: Re: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
Message-ID: <20160125145117.GL24938@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 3/7] quota: add new quotactl Q_GETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-4-git-send-email-sandeen@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453487136-12681-4-git-send-email-sandeen@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453733464
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26433
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri 22-01-16 12:25:32, Eric Sandeen wrote:
> Q_GETNEXTQUOTA is exactly like Q_GETQUOTA, except that it
> will return quota information for the id equal to or greater
> than the id requested.  In other words, if the requested id has
> no quota, the command will return quota information for the
> next higher id which does have a quota set.  If no higher id
> has an active quota, -ESRCH is returned.
> 
> This allows filesystems to do efficient iteration in kernelspace,
> much like extN filesystems do in userspace when asked to report
> all active quotas.
> 
> This does require a new data structure for userspace, as the
> current structure does not include an ID for the returned quota
> information.
> 
> Today, Ext4 with a hidden quota inode requires getpwent-style
> iterations, and for systems which have i.e. LDAP backends,
> this can be very slow, or even impossible if iteration is not
> allowed in the configuration.
> 
> Signed-off-by: Eric Sandeen 

The patch seems to be missing addition of Q_GETNEXTQUOTA into
quotactl_cmd_write(). Otherwise it looks good to me.

							Honza

> ---
>  fs/quota/quota.c           |   30 ++++++++++++++++++++++++++++++
>  include/uapi/linux/quota.h |   14 ++++++++++++++
>  2 files changed, 44 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/quota/quota.c b/fs/quota/quota.c
> index 0a6dd71..ffa4e0b 100644
> --- a/fs/quota/quota.c
> +++ b/fs/quota/quota.c
> @@ -222,6 +222,34 @@ static int quota_getquota(struct super_block *sb, int type, qid_t id,
>  	return 0;
>  }
>  
> +/*
> + * Return quota for next active quota >= this id, if any exists,
> + * otherwise return -ESRCH via ->get_nextdqblk
> + */
> +static int quota_getnextquota(struct super_block *sb, int type, qid_t id,
> +			  void __user *addr)
> +{
> +	struct kqid qid;
> +	struct qc_dqblk fdq;
> +	struct if_nextdqblk idq;
> +	int ret;
> +
> +	if (!sb->s_qcop->get_nextdqblk)
> +		return -ENOSYS;
> +	qid = make_kqid(current_user_ns(), type, id);
> +	if (!qid_valid(qid))
> +		return -EINVAL;
> +	ret = sb->s_qcop->get_nextdqblk(sb, &qid, &fdq);
> +	if (ret)
> +		return ret;
> +	/* struct if_nextdqblk is a superset of struct if_dqblk */
> +	copy_to_if_dqblk((struct if_dqblk *)&idq, &fdq);
> +	idq.dqb_id = from_kqid(current_user_ns(), qid);
> +	if (copy_to_user(addr, &idq, sizeof(idq)))
> +		return -EFAULT;
> +	return 0;
> +}
> +
>  static void copy_from_if_dqblk(struct qc_dqblk *dst, struct if_dqblk *src)
>  {
>  	dst->d_spc_hardlimit = qbtos(src->dqb_bhardlimit);
> @@ -698,6 +726,8 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
>  		return quota_setinfo(sb, type, addr);
>  	case Q_GETQUOTA:
>  		return quota_getquota(sb, type, id, addr);
> +	case Q_GETNEXTQUOTA:
> +		return quota_getnextquota(sb, type, id, addr);
>  	case Q_SETQUOTA:
>  		return quota_setquota(sb, type, id, addr);
>  	case Q_SYNC:
> diff --git a/include/uapi/linux/quota.h b/include/uapi/linux/quota.h
> index 9c95b2c..38baddb 100644
> --- a/include/uapi/linux/quota.h
> +++ b/include/uapi/linux/quota.h
> @@ -71,6 +71,7 @@
>  #define Q_SETINFO  0x800006	/* set information about quota files */
>  #define Q_GETQUOTA 0x800007	/* get user quota structure */
>  #define Q_SETQUOTA 0x800008	/* set user quota structure */
> +#define Q_GETNEXTQUOTA 0x800009	/* get disk limits and usage >= ID */
>  
>  /* Quota format type IDs */
>  #define	QFMT_VFS_OLD 1
> @@ -119,6 +120,19 @@ struct if_dqblk {
>  	__u32 dqb_valid;
>  };
>  
> +struct if_nextdqblk {
> +	__u64 dqb_bhardlimit;
> +	__u64 dqb_bsoftlimit;
> +	__u64 dqb_curspace;
> +	__u64 dqb_ihardlimit;
> +	__u64 dqb_isoftlimit;
> +	__u64 dqb_curinodes;
> +	__u64 dqb_btime;
> +	__u64 dqb_itime;
> +	__u32 dqb_valid;
> +	__u32 dqb_id;
> +};
> +
>  /*
>   * Structure used for setting quota information about file via quotactl
>   * Following flags are used to specify which fields are valid
> -- 
> 1.7.1
> 
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Mon Jan 25 08:51:30 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 8192F7F37
	for ; Mon, 25 Jan 2016 08:51:30 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 37CC48F8033
	for ; Mon, 25 Jan 2016 06:51:30 -0800 (PST)
X-ASG-Debug-ID: 1453733487-04bdf04d6db3d00001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id OAEwd6uvkOAOiCkA (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 25 Jan 2016 06:51:28 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id C4815ABB0;
	Mon, 25 Jan 2016 14:51:25 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 17A23823DA; Mon, 25 Jan 2016 15:51:40 +0100 (CET)
Date: Mon, 25 Jan 2016 15:51:40 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Message-ID: <20160125145140.GM24938@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-3-git-send-email-sandeen@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453487136-12681-3-git-send-email-sandeen@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453733487
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26433
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri 22-01-16 12:25:31, Eric Sandeen wrote:
> Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
> will return quota information for the id equal to or greater
> than the id requested.  In other words, if the requested id has
> no quota, the command will return quota information for the
> next higher id which does have a quota set.  If no higher id
> has an active quota, -ESRCH is returned.
> 
> This allows filesystems to do efficient iteration in kernelspace,
> much like extN filesystems do in userspace when asked to report
> all active quotas.
> 
> The patch adds a d_id field to struct qc_dqblk so that we can
> pass back the id of the quota which was found, and return it
> to userspace.

This paragraph is no longer true. Otherwise the patch looks good to me.

								Honza

> 
> Today, filesystems such as XFS require getpwent-style iterations,
> and for systems which have i.e. LDAP backends, this can be very
> slow, or even impossible if iteration is not allowed in the
> configuration.
> 
> Signed-off-by: Eric Sandeen 
> ---
>  fs/quota/quota.c               |   31 +++++++++++++++++++++++++++++++
>  include/linux/quota.h          |    2 ++
>  include/uapi/linux/dqblk_xfs.h |    1 +
>  3 files changed, 34 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/quota/quota.c b/fs/quota/quota.c
> index ea66670..0a6dd71 100644
> --- a/fs/quota/quota.c
> +++ b/fs/quota/quota.c
> @@ -625,6 +625,34 @@ static int quota_getxquota(struct super_block *sb, int type, qid_t id,
>  	return ret;
>  }
>  
> +/*
> + * Return quota for next active quota >= this id, if any exists,
> + * otherwise return -ESRCH via ->get_nextdqblk.
> + */
> +static int quota_getnextxquota(struct super_block *sb, int type, qid_t id,
> +			    void __user *addr)
> +{
> +	struct fs_disk_quota fdq;
> +	struct qc_dqblk qdq;
> +	struct kqid qid;
> +	qid_t id_out;
> +	int ret;
> +
> +	if (!sb->s_qcop->get_nextdqblk)
> +		return -ENOSYS;
> +	qid = make_kqid(current_user_ns(), type, id);
> +	if (!qid_valid(qid))
> +		return -EINVAL;
> +	ret = sb->s_qcop->get_nextdqblk(sb, &qid, &qdq);
> +	if (ret)
> +		return ret;
> +	id_out = from_kqid(current_user_ns(), qid);
> +	copy_to_xfs_dqblk(&fdq, &qdq, type, id_out);
> +	if (copy_to_user(addr, &fdq, sizeof(fdq)))
> +		return -EFAULT;
> +	return ret;
> +}
> +
>  static int quota_rmxquota(struct super_block *sb, void __user *addr)
>  {
>  	__u32 flags;
> @@ -690,6 +718,8 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
>  		return quota_setxquota(sb, type, id, addr);
>  	case Q_XGETQUOTA:
>  		return quota_getxquota(sb, type, id, addr);
> +	case Q_XGETNEXTQUOTA:
> +		return quota_getnextxquota(sb, type, id, addr);
>  	case Q_XQUOTASYNC:
>  		if (sb->s_flags & MS_RDONLY)
>  			return -EROFS;
> @@ -712,6 +742,7 @@ static int quotactl_cmd_write(int cmd)
>  	case Q_XGETQSTAT:
>  	case Q_XGETQSTATV:
>  	case Q_XGETQUOTA:
> +	case Q_XGETNEXTQUOTA:
>  	case Q_XQUOTASYNC:
>  		return 0;
>  	}
> diff --git a/include/linux/quota.h b/include/linux/quota.h
> index b2505ac..fba92f5 100644
> --- a/include/linux/quota.h
> +++ b/include/linux/quota.h
> @@ -425,6 +425,8 @@ struct quotactl_ops {
>  	int (*quota_sync)(struct super_block *, int);
>  	int (*set_info)(struct super_block *, int, struct qc_info *);
>  	int (*get_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *);
> +	int (*get_nextdqblk)(struct super_block *, struct kqid *,
> +			     struct qc_dqblk *);
>  	int (*set_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *);
>  	int (*get_state)(struct super_block *, struct qc_state *);
>  	int (*rm_xquota)(struct super_block *, unsigned int);
> diff --git a/include/uapi/linux/dqblk_xfs.h b/include/uapi/linux/dqblk_xfs.h
> index dcd75cc..11b3b31 100644
> --- a/include/uapi/linux/dqblk_xfs.h
> +++ b/include/uapi/linux/dqblk_xfs.h
> @@ -39,6 +39,7 @@
>  #define Q_XQUOTARM	XQM_CMD(6)	/* free disk space used by dquots */
>  #define Q_XQUOTASYNC	XQM_CMD(7)	/* delalloc flush, updates dquots */
>  #define Q_XGETQSTATV	XQM_CMD(8)	/* newer version of get quota */
> +#define Q_XGETNEXTQUOTA	XQM_CMD(9)	/* get disk limits and usage >= ID */
>  
>  /*
>   * fs_disk_quota structure:
> -- 
> 1.7.1
> 
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Mon Jan 25 09:07:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 2F3997CBD
	for ; Mon, 25 Jan 2016 09:07:05 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id D9385304059
	for ; Mon, 25 Jan 2016 07:07:01 -0800 (PST)
X-ASG-Debug-ID: 1453734418-04bdf04d6db44f0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id Swla8qqZM3264JHw (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 25 Jan 2016 07:06:59 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 029E4ABB4;
	Mon, 25 Jan 2016 15:06:56 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id C4496823DA; Mon, 25 Jan 2016 16:07:10 +0100 (CET)
Date: Mon, 25 Jan 2016 16:07:10 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz,
	Dave Chinner 
Subject: Re: [PATCH 0/7 V3] quota: add new quotactl Q_GETNEXTQUOTA
Message-ID: <20160125150710.GN24938@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 0/7 V3] quota: add new quotactl Q_GETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453734419
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26433
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri 22-01-16 12:25:29, Eric Sandeen wrote:
> This adds a new quotactl, Q_GETNEXTQUOTA.
> 
> Q_GETNEXTQUOTA is exactly like Q_GETQUOTA, except that it will
> return quota information for the id equal to or greater than
> the id requested.  In other words, if the specified id has
> no quota, the command will return quota information for the
> next higher id which does have a quota set.  If no higher id
> has an active quota, -ESRCH is returned.
> 
> So if you ask for id X, you can get back quota for id X,
> id X+N, or -ESRCH if no higher id has a quota.
> 
> This allows filesystems to do efficient iteration in kernelspace,
> much like extN filesystems do in userspace when asked to report
> all active quotas.
> 
> Today, filesystems such as XFS require getpwent()-style iterations,
> and for systems which have i.e. LDAP backends, this can be very
> slow, or even impossible if iteration is not allowed in the
> configuration.
> 
> Patches 1 and 4 are just small fixups that turned up along the way;
> 2 and 3 add the actual quota plumbing, and the rest are xfs-specific
> to allow xfs to support this new interface.
> 
> For non-xfs quota, this does require a new structure which is
> able to pass back the discovered ID along with the quota info.
> For xfs-quota, the id is already present in the structure.
> 
> V3:
> * Remove 32-bit compat stuff (i686/x86_64 at least works w/o it...)
> * Require CAP_SYS_ADMIN for these calls
> * Pass back found ID in &qid passed to ->get_nextdqblk, rather
>   than modifying struct qc_dqblk
> * Munge that found ID back through user-namespace conversions
>   before returning it in the user structure.

So I've taken patch 1/7 into my tree since that is completely independent.
Patches 2/7 and 3/7 look mostly good to me (except for that small omission
in 3/7) so I can take them to my tree as well. Once I'll finish testing
some UDF changes I have pending I'll push the tree to:

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git for_next

and Dave can pull from there to have a basis for XFS patches. I want to
implement Q_GETNEXTQUOTA for VFS quota formats as well and those will go
through my tree so that's why I want the interface changes go via my tree.

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From tdm@sgi.com  Mon Jan 25 09:16:33 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,RP_MATCHES_RCVD
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id CB3637CBD
	for ; Mon, 25 Jan 2016 09:16:33 -0600 (CST)
Received: from estes.americas.sgi.com (estes.americas.sgi.com [128.162.236.10])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 93257304032
	for ; Mon, 25 Jan 2016 07:16:33 -0800 (PST)
Received: from [128.162.232.11] (porter.americas.sgi.com [128.162.232.11])
	by estes.americas.sgi.com (Postfix) with ESMTP id 6815D70028CF
	for ; Mon, 25 Jan 2016 09:16:33 -0600 (CST)
Message-ID: <56A63C51.100@sgi.com>
Date: Mon, 25 Jan 2016 09:16:33 -0600
From: Troy McCorkell 
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: "xfs@oss.sgi.com" 
Subject: Re: oss.sgi.com - maintenance Jan 25 08:00 CT USA
References: 
In-Reply-To: 
Content-Type: multipart/alternative;
 boundary="------------060809000900010602010307"

This is a multi-part message in MIME format.
--------------060809000900010602010307
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 01/24/2016 09:11 PM, Troy McCorkell wrote:
> oss.sgi.com will be down approximately 30 minutes starting at 08:00 CT USA
> on Monday January 25th for maintenance.
>
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>    

Rebooting oss.sgi.com one more time.   Expect a 15 minute outage.

--------------060809000900010602010307
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit




  


On 01/24/2016 09:11 PM, Troy McCorkell wrote:


  
  
  
oss.sgi.com
will be down approximately 30 minutes starting at 08:00 CT USA

on Monday January 25th for maintenance.

  

  

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
  





Rebooting oss.sgi.com one more time.   Expect a 15 minute outage.




--------------060809000900010602010307--

From tdm@sgi.com  Mon Jan 25 09:38:20 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,RP_MATCHES_RCVD
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 97E627CA1
	for ; Mon, 25 Jan 2016 09:38:20 -0600 (CST)
Received: from estes.americas.sgi.com (estes.americas.sgi.com [128.162.236.10])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 7076B8F8033
	for ; Mon, 25 Jan 2016 07:38:17 -0800 (PST)
Received: from [128.162.232.11] (porter.americas.sgi.com [128.162.232.11])
	by estes.americas.sgi.com (Postfix) with ESMTP id 32A397002915
	for ; Mon, 25 Jan 2016 09:38:17 -0600 (CST)
Message-ID: <56A64169.5080507@sgi.com>
Date: Mon, 25 Jan 2016 09:38:17 -0600
From: Troy McCorkell 
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: "xfs@oss.sgi.com" 
Subject: Re: oss.sgi.com - maintenance Jan 25 08:00 CT USA
References:  <56A63C51.100@sgi.com>
In-Reply-To: <56A63C51.100@sgi.com>
Content-Type: multipart/alternative;
 boundary="------------010905080904090803060909"

This is a multi-part message in MIME format.
--------------010905080904090803060909
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 01/25/2016 09:16 AM, Troy McCorkell wrote:
> On 01/24/2016 09:11 PM, Troy McCorkell wrote:
>> oss.sgi.com will be down approximately 30 minutes starting at 08:00 
>> CT USA
>> on Monday January 25th for maintenance.
>>
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
>>    
>
> Rebooting oss.sgi.com one more time.   Expect a 15 minute outage.

Maintenance is complete on oss.sgi.com.   The system is online and 
available.
If you experience any issues with the system, drop me an email.

Thanks,
Troy McCorkell

--------------010905080904090803060909
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit




  


On 01/25/2016 09:16 AM, Troy McCorkell wrote:


  
On 01/24/2016 09:11 PM, Troy McCorkell wrote:
  

    
    
    
oss.sgi.com
will
be down approximately 30 minutes starting at 08:00 CT USA

on Monday January 25th for maintenance.

    

    

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
  

  

  

Rebooting oss.sgi.com one more time.   Expect a 15 minute outage.





Maintenance is complete on oss.sgi.com.   The system is online and
available.

If you experience any issues with the system, drop me an email.



Thanks,

Troy McCorkell




--------------010905080904090803060909--

From bfoster@redhat.com  Mon Jan 25 09:43:41 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 4812D7CA1
	for ; Mon, 25 Jan 2016 09:43:41 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 1EAB7304053
	for ; Mon, 25 Jan 2016 07:43:40 -0800 (PST)
X-ASG-Debug-ID: 1453736617-04cbb026ffbb4f0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id T4ECQTF0yTZTsxCK (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 07:43:38 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id AA9F07AE90;
	Mon, 25 Jan 2016 15:43:37 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PFha9Q030793;
	Mon, 25 Jan 2016 10:43:37 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 6725D120BCD; Mon, 25 Jan 2016 10:43:36 -0500 (EST)
Date: Mon, 25 Jan 2016 10:43:36 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/8] xfs: introduce inode log format object
Message-ID: <20160125154335.GA4108@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 1/8] xfs: introduce inode log format object
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
 <1452751765-4420-2-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452751765-4420-2-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453736618
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 14, 2016 at 05:09:18PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> We currently carry around and log an entire inode core in the
> struct xfs_inode. A lot of the information in the inode core is
> duplicated in the VFS inode, but we cannot remove this duplication
> of infomration because the inode core is logged directly in
> xfs_inode_item_format().
> 
> Add a new function xfs_inode_item_format_core() that copies the
> inode core data into a struct xfs_icdinode that is pulled directly
> from the log vector buffer. This means we no longer directly
> copy the inode core, but copy the structures one member at a time.
> This will be slightly less efficient than copying, but will allow us
> to remove duplicate and unnecessary items from the struct xfs_inode.
> 
> To enable us to do this, call the new structure a xfs_log_dinode,
> so that we know it's different to the physical xfs_dinode and the
> in-core xfs_icdinode.
> 
> Signed-off-by: Dave Chinner 
> ---

Reviewed-by: Brian Foster 

>  fs/xfs/libxfs/xfs_inode_buf.c  |   8 +--
>  fs/xfs/libxfs/xfs_inode_buf.h  |  53 ++++++++++++++++-
>  fs/xfs/libxfs/xfs_log_format.h |  15 ++---
>  fs/xfs/xfs_icache.c            |   2 +-
>  fs/xfs/xfs_inode.c             |   6 +-
>  fs/xfs/xfs_inode.h             |   2 +-
>  fs/xfs/xfs_inode_item.c        | 128 +++++++++++++++++++++++++++++++++++++++--
>  fs/xfs/xfs_inode_item.h        |   2 +
>  fs/xfs/xfs_log_recover.c       |  52 +++++++++--------
>  9 files changed, 221 insertions(+), 47 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
> index 1aabfda..63d46bf 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.c
> +++ b/fs/xfs/libxfs/xfs_inode_buf.c
> @@ -196,8 +196,8 @@ xfs_imap_to_bp(
>  
>  void
>  xfs_dinode_from_disk(
> -	xfs_icdinode_t		*to,
> -	xfs_dinode_t		*from)
> +	struct xfs_icdinode	*to,
> +	struct xfs_dinode	*from)
>  {
>  	to->di_magic = be16_to_cpu(from->di_magic);
>  	to->di_mode = be16_to_cpu(from->di_mode);
> @@ -243,8 +243,8 @@ xfs_dinode_from_disk(
>  
>  void
>  xfs_dinode_to_disk(
> -	xfs_dinode_t		*to,
> -	xfs_icdinode_t		*from)
> +	struct xfs_dinode	*to,
> +	struct xfs_icdinode	*from)
>  {
>  	to->di_magic = cpu_to_be16(from->di_magic);
>  	to->di_mode = cpu_to_be16(from->di_mode);
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
> index 9308c47..642f2a2 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.h
> +++ b/fs/xfs/libxfs/xfs_inode_buf.h
> @@ -20,7 +20,58 @@
>  
>  struct xfs_inode;
>  struct xfs_dinode;
> -struct xfs_icdinode;
> +
> +/*
> + * In memory representation of the XFS inode. This is held in the in-core
> + * struct xfs_inode to represent the on disk values, but no longer needs to be
> + * identical to the on-disk structure as it is always translated to on-disk
> + * format specific structures at the appropriate time.
> + */
> +struct xfs_icdinode {
> +	__uint16_t	di_magic;	/* inode magic # = XFS_DINODE_MAGIC */
> +	__uint16_t	di_mode;	/* mode and type of file */
> +	__int8_t	di_version;	/* inode version */
> +	__int8_t	di_format;	/* format of di_c data */
> +	__uint16_t	di_onlink;	/* old number of links to file */
> +	__uint32_t	di_uid;		/* owner's user id */
> +	__uint32_t	di_gid;		/* owner's group id */
> +	__uint32_t	di_nlink;	/* number of links to file */
> +	__uint16_t	di_projid_lo;	/* lower part of owner's project id */
> +	__uint16_t	di_projid_hi;	/* higher part of owner's project id */
> +	__uint8_t	di_pad[6];	/* unused, zeroed space */
> +	__uint16_t	di_flushiter;	/* incremented on flush */
> +	xfs_ictimestamp_t di_atime;	/* time last accessed */
> +	xfs_ictimestamp_t di_mtime;	/* time last modified */
> +	xfs_ictimestamp_t di_ctime;	/* time created/inode modified */
> +	xfs_fsize_t	di_size;	/* number of bytes in file */
> +	xfs_rfsblock_t	di_nblocks;	/* # of direct & btree blocks used */
> +	xfs_extlen_t	di_extsize;	/* basic/minimum extent size for file */
> +	xfs_extnum_t	di_nextents;	/* number of extents in data fork */
> +	xfs_aextnum_t	di_anextents;	/* number of extents in attribute fork*/
> +	__uint8_t	di_forkoff;	/* attr fork offs, <<3 for 64b align */
> +	__int8_t	di_aformat;	/* format of attr fork's data */
> +	__uint32_t	di_dmevmask;	/* DMIG event mask */
> +	__uint16_t	di_dmstate;	/* DMIG state info */
> +	__uint16_t	di_flags;	/* random flags, XFS_DIFLAG_... */
> +	__uint32_t	di_gen;		/* generation number */
> +
> +	/* di_next_unlinked is the only non-core field in the old dinode */
> +	xfs_agino_t	di_next_unlinked;/* agi unlinked list ptr */
> +
> +	/* start of the extended dinode, writable fields */
> +	__uint32_t	di_crc;		/* CRC of the inode */
> +	__uint64_t	di_changecount;	/* number of attribute changes */
> +	xfs_lsn_t	di_lsn;		/* flush sequence */
> +	__uint64_t	di_flags2;	/* more random flags */
> +	__uint8_t	di_pad2[16];	/* more padding for future expansion */
> +
> +	/* fields only written to during inode creation */
> +	xfs_ictimestamp_t di_crtime;	/* time created */
> +	xfs_ino_t	di_ino;		/* inode number */
> +	uuid_t		di_uuid;	/* UUID of the filesystem */
> +
> +	/* structure must be padded to 64 bit alignment */
> +};
>  
>  /*
>   * Inode location information.  Stored in the inode and passed to
> diff --git a/fs/xfs/libxfs/xfs_log_format.h b/fs/xfs/libxfs/xfs_log_format.h
> index 2653146..d00ed63 100644
> --- a/fs/xfs/libxfs/xfs_log_format.h
> +++ b/fs/xfs/libxfs/xfs_log_format.h
> @@ -290,6 +290,7 @@ typedef struct xfs_inode_log_format_64 {
>  	__int32_t		ilf_boffset;	/* off of inode in buffer */
>  } xfs_inode_log_format_64_t;
>  
> +
>  /*
>   * Flags for xfs_trans_log_inode flags field.
>   */
> @@ -360,10 +361,10 @@ typedef struct xfs_ictimestamp {
>  } xfs_ictimestamp_t;
>  
>  /*
> - * NOTE:  This structure must be kept identical to struct xfs_dinode
> - *	  except for the endianness annotations.
> + * Define the format of the inode core that is logged. This structure must be
> + * kept identical to struct xfs_dinode except for the endianness annotations.
>   */
> -typedef struct xfs_icdinode {
> +struct xfs_log_dinode {
>  	__uint16_t	di_magic;	/* inode magic # = XFS_DINODE_MAGIC */
>  	__uint16_t	di_mode;	/* mode and type of file */
>  	__int8_t	di_version;	/* inode version */
> @@ -407,13 +408,13 @@ typedef struct xfs_icdinode {
>  	uuid_t		di_uuid;	/* UUID of the filesystem */
>  
>  	/* structure must be padded to 64 bit alignment */
> -} xfs_icdinode_t;
> +};
>  
> -static inline uint xfs_icdinode_size(int version)
> +static inline uint xfs_log_dinode_size(int version)
>  {
>  	if (version == 3)
> -		return sizeof(struct xfs_icdinode);
> -	return offsetof(struct xfs_icdinode, di_next_unlinked);
> +		return sizeof(struct xfs_log_dinode);
> +	return offsetof(struct xfs_log_dinode, di_next_unlinked);
>  }
>  
>  /*
> diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
> index d7a490f..7c26f86 100644
> --- a/fs/xfs/xfs_icache.c
> +++ b/fs/xfs/xfs_icache.c
> @@ -79,7 +79,7 @@ xfs_inode_alloc(
>  	memset(&ip->i_df, 0, sizeof(xfs_ifork_t));
>  	ip->i_flags = 0;
>  	ip->i_delayed_blks = 0;
> -	memset(&ip->i_d, 0, sizeof(xfs_icdinode_t));
> +	memset(&ip->i_d, 0, sizeof(ip->i_d));
>  
>  	return ip;
>  }
> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> index ae3758a..7e24232 100644
> --- a/fs/xfs/xfs_inode.c
> +++ b/fs/xfs/xfs_inode.c
> @@ -650,9 +650,9 @@ _xfs_dic2xflags(
>  
>  uint
>  xfs_ip2xflags(
> -	xfs_inode_t		*ip)
> +	struct xfs_inode	*ip)
>  {
> -	xfs_icdinode_t		*dic = &ip->i_d;
> +	struct xfs_icdinode	*dic = &ip->i_d;
>  
>  	return _xfs_dic2xflags(dic->di_flags) |
>  				(XFS_IFORK_Q(ip) ? XFS_XFLAG_HASATTR : 0);
> @@ -660,7 +660,7 @@ xfs_ip2xflags(
>  
>  uint
>  xfs_dic2xflags(
> -	xfs_dinode_t		*dip)
> +	struct xfs_dinode	*dip)
>  {
>  	return _xfs_dic2xflags(be16_to_cpu(dip->di_flags)) |
>  				(XFS_DFORK_Q(dip) ? XFS_XFLAG_HASATTR : 0);
> diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
> index ca9e119..aef5452 100644
> --- a/fs/xfs/xfs_inode.h
> +++ b/fs/xfs/xfs_inode.h
> @@ -63,7 +63,7 @@ typedef struct xfs_inode {
>  	unsigned long		i_flags;	/* see defined flags below */
>  	unsigned int		i_delayed_blks;	/* count of delay alloc blks */
>  
> -	xfs_icdinode_t		i_d;		/* most of ondisk inode */
> +	struct xfs_icdinode	i_d;		/* most of ondisk inode */
>  
>  	/* VFS inode */
>  	struct inode		i_vnode;	/* embedded VFS inode */
> diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
> index d14b12b..3ad9972 100644
> --- a/fs/xfs/xfs_inode_item.c
> +++ b/fs/xfs/xfs_inode_item.c
> @@ -135,7 +135,7 @@ xfs_inode_item_size(
>  
>  	*nvecs += 2;
>  	*nbytes += sizeof(struct xfs_inode_log_format) +
> -		   xfs_icdinode_size(ip->i_d.di_version);
> +		   xfs_log_dinode_size(ip->i_d.di_version);
>  
>  	xfs_inode_item_data_fork_size(iip, nvecs, nbytes);
>  	if (XFS_IFORK_Q(ip))
> @@ -322,6 +322,127 @@ xfs_inode_item_format_attr_fork(
>  	}
>  }
>  
> +static void
> +xfs_icdinode_to_log_dinode(
> +	struct xfs_icdinode	*from,
> +	struct xfs_log_dinode	*to)
> +{
> +	to->di_magic = from->di_magic;
> +	to->di_mode = from->di_mode;
> +	to->di_version = from->di_version;
> +	to->di_format = from->di_format;
> +	to->di_onlink = from->di_onlink;
> +	to->di_uid = from->di_uid;
> +	to->di_gid = from->di_gid;
> +	to->di_nlink = from->di_nlink;
> +	to->di_projid_lo = from->di_projid_lo;
> +	to->di_projid_hi = from->di_projid_hi;
> +	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
> +	to->di_atime.t_sec = from->di_atime.t_sec;
> +	to->di_atime.t_nsec = from->di_atime.t_nsec;
> +	to->di_mtime.t_sec = from->di_mtime.t_sec;
> +	to->di_mtime.t_nsec = from->di_mtime.t_nsec;
> +	to->di_ctime.t_sec = from->di_ctime.t_sec;
> +	to->di_ctime.t_nsec = from->di_ctime.t_nsec;
> +	to->di_size = from->di_size;
> +	to->di_nblocks = from->di_nblocks;
> +	to->di_extsize = from->di_extsize;
> +	to->di_nextents = from->di_nextents;
> +	to->di_anextents = from->di_anextents;
> +	to->di_forkoff = from->di_forkoff;
> +	to->di_aformat = from->di_aformat;
> +	to->di_dmevmask = from->di_dmevmask;
> +	to->di_dmstate = from->di_dmstate;
> +	to->di_flags = from->di_flags;
> +	to->di_gen = from->di_gen;
> +
> +	if (from->di_version == 3) {
> +		to->di_changecount = from->di_changecount;
> +		to->di_crtime.t_sec = from->di_crtime.t_sec;
> +		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
> +		to->di_flags2 = from->di_flags2;
> +		to->di_ino = from->di_ino;
> +		to->di_lsn = from->di_lsn;
> +		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
> +		uuid_copy(&to->di_uuid, &from->di_uuid);
> +		to->di_flushiter = 0;
> +	} else {
> +		to->di_flushiter = from->di_flushiter;
> +	}
> +}
> +
> +/*
> + * Recovery needs to be able to convert a log dinode back to a real dinode
> + * for writeback we do that by converting a log dinode to a icdinode, and
> + * then passing that to the formatting function.
> + */
> +void
> +xfs_log_dinode_to_icdinode(
> +	struct xfs_log_dinode	*from,
> +	struct xfs_icdinode	*to)
> +{
> +	to->di_magic = from->di_magic;
> +	to->di_mode = from->di_mode;
> +	to->di_version = from->di_version;
> +	to->di_format = from->di_format;
> +	to->di_onlink = from->di_onlink;
> +	to->di_uid = from->di_uid;
> +	to->di_gid = from->di_gid;
> +	to->di_nlink = from->di_nlink;
> +	to->di_projid_lo = from->di_projid_lo;
> +	to->di_projid_hi = from->di_projid_hi;
> +	memset(to->di_pad, 0, sizeof(to->di_pad));
> +	to->di_atime.t_sec = from->di_atime.t_sec;
> +	to->di_atime.t_nsec = from->di_atime.t_nsec;
> +	to->di_mtime.t_sec = from->di_mtime.t_sec;
> +	to->di_mtime.t_nsec = from->di_mtime.t_nsec;
> +	to->di_ctime.t_sec = from->di_ctime.t_sec;
> +	to->di_ctime.t_nsec = from->di_ctime.t_nsec;
> +	to->di_size = from->di_size;
> +	to->di_nblocks = from->di_nblocks;
> +	to->di_extsize = from->di_extsize;
> +	to->di_nextents = from->di_nextents;
> +	to->di_anextents = from->di_anextents;
> +	to->di_forkoff = from->di_forkoff;
> +	to->di_aformat = from->di_aformat;
> +	to->di_dmevmask = from->di_dmevmask;
> +	to->di_dmstate = from->di_dmstate;
> +	to->di_flags = from->di_flags;
> +	to->di_gen = from->di_gen;
> +
> +	if (from->di_version == 3) {
> +		to->di_changecount = from->di_changecount;
> +		to->di_crtime.t_sec = from->di_crtime.t_sec;
> +		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
> +		to->di_flags2 = from->di_flags2;
> +		to->di_ino = from->di_ino;
> +		to->di_lsn = from->di_lsn;
> +		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
> +		uuid_copy(&to->di_uuid, &from->di_uuid);
> +		to->di_flushiter = 0;
> +	} else {
> +		to->di_flushiter = from->di_flushiter;
> +	}
> +}
> +
> +/*
> + * Format the inode core. Current timestamp data is only in the VFS inode
> + * fields, so we need to grab them from there. Hence rather than just copying
> + * the XFS inode core structure, format the fields directly into the iovec.
> + */
> +static void
> +xfs_inode_item_format_core(
> +	struct xfs_inode	*ip,
> +	struct xfs_log_vec	*lv,
> +	struct xfs_log_iovec	**vecp)
> +{
> +	struct xfs_log_dinode	*dic;
> +
> +	dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE);
> +	xfs_icdinode_to_log_dinode(&ip->i_d, dic);
> +	xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version));
> +}
> +
>  /*
>   * This is called to fill in the vector of log iovecs for the given inode
>   * log item.  It fills the first item with an inode log format structure,
> @@ -351,10 +472,7 @@ xfs_inode_item_format(
>  	ilf->ilf_size = 2; /* format + core */
>  	xlog_finish_iovec(lv, vecp, sizeof(struct xfs_inode_log_format));
>  
> -	xlog_copy_iovec(lv, &vecp, XLOG_REG_TYPE_ICORE,
> -			&ip->i_d,
> -			xfs_icdinode_size(ip->i_d.di_version));
> -
> +	xfs_inode_item_format_core(ip, lv, &vecp);
>  	xfs_inode_item_format_data_fork(iip, ilf, lv, &vecp);
>  	if (XFS_IFORK_Q(ip)) {
>  		xfs_inode_item_format_attr_fork(iip, ilf, lv, &vecp);
> diff --git a/fs/xfs/xfs_inode_item.h b/fs/xfs/xfs_inode_item.h
> index 4c7722e..2426118 100644
> --- a/fs/xfs/xfs_inode_item.h
> +++ b/fs/xfs/xfs_inode_item.h
> @@ -49,6 +49,8 @@ extern void xfs_istale_done(struct xfs_buf *, struct xfs_log_item *);
>  extern void xfs_iflush_abort(struct xfs_inode *, bool);
>  extern int xfs_inode_item_format_convert(xfs_log_iovec_t *,
>  					 xfs_inode_log_format_t *);
> +extern void xfs_log_dinode_to_icdinode(struct xfs_log_dinode *from,
> +				       struct xfs_icdinode *to);
>  
>  extern struct kmem_zone	*xfs_ili_zone;
>  
> diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
> index da37beb..3120f7b 100644
> --- a/fs/xfs/xfs_log_recover.c
> +++ b/fs/xfs/xfs_log_recover.c
> @@ -2839,7 +2839,8 @@ xlog_recover_inode_pass2(
>  	int			error;
>  	int			attr_index;
>  	uint			fields;
> -	xfs_icdinode_t		*dicp;
> +	struct xfs_log_dinode	*ldip;
> +	struct xfs_icdinode	icic;
>  	uint			isize;
>  	int			need_free = 0;
>  
> @@ -2892,8 +2893,8 @@ xlog_recover_inode_pass2(
>  		error = -EFSCORRUPTED;
>  		goto out_release;
>  	}
> -	dicp = item->ri_buf[1].i_addr;
> -	if (unlikely(dicp->di_magic != XFS_DINODE_MAGIC)) {
> +	ldip = item->ri_buf[1].i_addr;
> +	if (unlikely(ldip->di_magic != XFS_DINODE_MAGIC)) {
>  		xfs_alert(mp,
>  			"%s: Bad inode log record, rec ptr 0x%p, ino %Ld",
>  			__func__, item, in_f->ilf_ino);
> @@ -2929,13 +2930,13 @@ xlog_recover_inode_pass2(
>  	 * to skip replay when the on disk inode is newer than the log one
>  	 */
>  	if (!xfs_sb_version_hascrc(&mp->m_sb) &&
> -	    dicp->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
> +	    ldip->di_flushiter < be16_to_cpu(dip->di_flushiter)) {
>  		/*
>  		 * Deal with the wrap case, DI_MAX_FLUSH is less
>  		 * than smaller numbers
>  		 */
>  		if (be16_to_cpu(dip->di_flushiter) == DI_MAX_FLUSH &&
> -		    dicp->di_flushiter < (DI_MAX_FLUSH >> 1)) {
> +		    ldip->di_flushiter < (DI_MAX_FLUSH >> 1)) {
>  			/* do nothing */
>  		} else {
>  			trace_xfs_log_recover_inode_skip(log, in_f);
> @@ -2945,13 +2946,13 @@ xlog_recover_inode_pass2(
>  	}
>  
>  	/* Take the opportunity to reset the flush iteration count */
> -	dicp->di_flushiter = 0;
> +	ldip->di_flushiter = 0;
>  
> -	if (unlikely(S_ISREG(dicp->di_mode))) {
> -		if ((dicp->di_format != XFS_DINODE_FMT_EXTENTS) &&
> -		    (dicp->di_format != XFS_DINODE_FMT_BTREE)) {
> +	if (unlikely(S_ISREG(ldip->di_mode))) {
> +		if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) &&
> +		    (ldip->di_format != XFS_DINODE_FMT_BTREE)) {
>  			XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(3)",
> -					 XFS_ERRLEVEL_LOW, mp, dicp);
> +					 XFS_ERRLEVEL_LOW, mp, ldip);
>  			xfs_alert(mp,
>  		"%s: Bad regular inode log record, rec ptr 0x%p, "
>  		"ino ptr = 0x%p, ino bp = 0x%p, ino %Ld",
> @@ -2959,12 +2960,12 @@ xlog_recover_inode_pass2(
>  			error = -EFSCORRUPTED;
>  			goto out_release;
>  		}
> -	} else if (unlikely(S_ISDIR(dicp->di_mode))) {
> -		if ((dicp->di_format != XFS_DINODE_FMT_EXTENTS) &&
> -		    (dicp->di_format != XFS_DINODE_FMT_BTREE) &&
> -		    (dicp->di_format != XFS_DINODE_FMT_LOCAL)) {
> +	} else if (unlikely(S_ISDIR(ldip->di_mode))) {
> +		if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) &&
> +		    (ldip->di_format != XFS_DINODE_FMT_BTREE) &&
> +		    (ldip->di_format != XFS_DINODE_FMT_LOCAL)) {
>  			XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(4)",
> -					     XFS_ERRLEVEL_LOW, mp, dicp);
> +					     XFS_ERRLEVEL_LOW, mp, ldip);
>  			xfs_alert(mp,
>  		"%s: Bad dir inode log record, rec ptr 0x%p, "
>  		"ino ptr = 0x%p, ino bp = 0x%p, ino %Ld",
> @@ -2973,32 +2974,32 @@ xlog_recover_inode_pass2(
>  			goto out_release;
>  		}
>  	}
> -	if (unlikely(dicp->di_nextents + dicp->di_anextents > dicp->di_nblocks)){
> +	if (unlikely(ldip->di_nextents + ldip->di_anextents > ldip->di_nblocks)){
>  		XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(5)",
> -				     XFS_ERRLEVEL_LOW, mp, dicp);
> +				     XFS_ERRLEVEL_LOW, mp, ldip);
>  		xfs_alert(mp,
>  	"%s: Bad inode log record, rec ptr 0x%p, dino ptr 0x%p, "
>  	"dino bp 0x%p, ino %Ld, total extents = %d, nblocks = %Ld",
>  			__func__, item, dip, bp, in_f->ilf_ino,
> -			dicp->di_nextents + dicp->di_anextents,
> -			dicp->di_nblocks);
> +			ldip->di_nextents + ldip->di_anextents,
> +			ldip->di_nblocks);
>  		error = -EFSCORRUPTED;
>  		goto out_release;
>  	}
> -	if (unlikely(dicp->di_forkoff > mp->m_sb.sb_inodesize)) {
> +	if (unlikely(ldip->di_forkoff > mp->m_sb.sb_inodesize)) {
>  		XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(6)",
> -				     XFS_ERRLEVEL_LOW, mp, dicp);
> +				     XFS_ERRLEVEL_LOW, mp, ldip);
>  		xfs_alert(mp,
>  	"%s: Bad inode log record, rec ptr 0x%p, dino ptr 0x%p, "
>  	"dino bp 0x%p, ino %Ld, forkoff 0x%x", __func__,
> -			item, dip, bp, in_f->ilf_ino, dicp->di_forkoff);
> +			item, dip, bp, in_f->ilf_ino, ldip->di_forkoff);
>  		error = -EFSCORRUPTED;
>  		goto out_release;
>  	}
> -	isize = xfs_icdinode_size(dicp->di_version);
> +	isize = xfs_log_dinode_size(ldip->di_version);
>  	if (unlikely(item->ri_buf[1].i_len > isize)) {
>  		XFS_CORRUPTION_ERROR("xlog_recover_inode_pass2(7)",
> -				     XFS_ERRLEVEL_LOW, mp, dicp);
> +				     XFS_ERRLEVEL_LOW, mp, ldip);
>  		xfs_alert(mp,
>  			"%s: Bad inode log record length %d, rec ptr 0x%p",
>  			__func__, item->ri_buf[1].i_len, item);
> @@ -3007,7 +3008,8 @@ xlog_recover_inode_pass2(
>  	}
>  
>  	/* The core is in in-core format */
> -	xfs_dinode_to_disk(dip, dicp);
> +	xfs_log_dinode_to_icdinode(ldip, &icic);
> +	xfs_dinode_to_disk(dip, &icic);
>  
>  	/* the rest is in on-disk format */
>  	if (item->ri_buf[1].i_len > isize) {
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Mon Jan 25 09:43:44 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id CD02D7CA1
	for ; Mon, 25 Jan 2016 09:43:44 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id C51B2304053
	for ; Mon, 25 Jan 2016 07:43:44 -0800 (PST)
X-ASG-Debug-ID: 1453736621-04cbb026fdbb4f0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id RFKsj8EM7MqgJAqF (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 07:43:42 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id 9CA5FC027326;
	Mon, 25 Jan 2016 15:43:41 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PFhebt030843;
	Mon, 25 Jan 2016 10:43:41 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 49774120BCD; Mon, 25 Jan 2016 10:43:40 -0500 (EST)
Date: Mon, 25 Jan 2016 10:43:40 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/8] xfs: remove timestamps from incore inode
Message-ID: <20160125154339.GB4108@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 2/8] xfs: remove timestamps from incore inode
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
 <1452751765-4420-3-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452751765-4420-3-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453736622
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 14, 2016 at 05:09:19PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> The struct xfs_inode has two copies of the current timestamps in it,
> one in the vfs inode and one in the struct xfs_icdinode. Now that we
> no longer log the struct xfs_icdinode directly, we don't need to
> keep the timestamps in this structure. instead we can copy them
> straight out of the VFS inode when formatting the inode log item or
> the on-disk inode.
> 
> This reduces the struct xfs_inode in size by 24 bytes.
> 
> Signed-off-by: Dave Chinner 
> ---

Reviewed-by: Brian Foster 

>  fs/xfs/libxfs/xfs_inode_buf.c | 91 +++++++++++++++++++++++++++++++++++++------
>  fs/xfs/libxfs/xfs_inode_buf.h |  9 ++---
>  fs/xfs/libxfs/xfs_rtbitmap.c  |  2 +-
>  fs/xfs/xfs_inode.c            | 21 +++++-----
>  fs/xfs/xfs_inode_item.c       | 73 ++++++----------------------------
>  fs/xfs/xfs_inode_item.h       |  2 -
>  fs/xfs/xfs_iops.c             | 37 ++++--------------
>  fs/xfs/xfs_itable.c           | 16 +++++---
>  fs/xfs/xfs_log_recover.c      |  8 ++--
>  fs/xfs/xfs_rtalloc.c          |  2 +-
>  fs/xfs/xfs_trans_inode.c      | 12 +-----
>  11 files changed, 130 insertions(+), 143 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
> index 63d46bf..26e3c46 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.c
> +++ b/fs/xfs/libxfs/xfs_inode_buf.c
> @@ -195,10 +195,13 @@ xfs_imap_to_bp(
>  }
>  
>  void
> -xfs_dinode_from_disk(
> -	struct xfs_icdinode	*to,
> +xfs_inode_from_disk(
> +	struct xfs_inode	*ip,
>  	struct xfs_dinode	*from)
>  {
> +	struct xfs_icdinode	*to = &ip->i_d;
> +	struct inode		*inode = VFS_I(ip);
> +
>  	to->di_magic = be16_to_cpu(from->di_magic);
>  	to->di_mode = be16_to_cpu(from->di_mode);
>  	to->di_version = from ->di_version;
> @@ -211,12 +214,20 @@ xfs_dinode_from_disk(
>  	to->di_projid_hi = be16_to_cpu(from->di_projid_hi);
>  	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
>  	to->di_flushiter = be16_to_cpu(from->di_flushiter);
> -	to->di_atime.t_sec = be32_to_cpu(from->di_atime.t_sec);
> -	to->di_atime.t_nsec = be32_to_cpu(from->di_atime.t_nsec);
> -	to->di_mtime.t_sec = be32_to_cpu(from->di_mtime.t_sec);
> -	to->di_mtime.t_nsec = be32_to_cpu(from->di_mtime.t_nsec);
> -	to->di_ctime.t_sec = be32_to_cpu(from->di_ctime.t_sec);
> -	to->di_ctime.t_nsec = be32_to_cpu(from->di_ctime.t_nsec);
> +
> +	/*
> +	 * time is signed, so need to convert to signed 32 bit before
> +	 * storing in inode timestamp which may be 64 bit. Otherwise
> +	 * a time before epoch is converted to a time long after epoch
> +	 * on 64 bit systems.
> +	 */
> +	inode->i_atime.tv_sec = (int)be32_to_cpu(from->di_atime.t_sec);
> +	inode->i_atime.tv_nsec = (int)be32_to_cpu(from->di_atime.t_nsec);
> +	inode->i_mtime.tv_sec = (int)be32_to_cpu(from->di_mtime.t_sec);
> +	inode->i_mtime.tv_nsec = (int)be32_to_cpu(from->di_mtime.t_nsec);
> +	inode->i_ctime.tv_sec = (int)be32_to_cpu(from->di_ctime.t_sec);
> +	inode->i_ctime.tv_nsec = (int)be32_to_cpu(from->di_ctime.t_nsec);
> +
>  	to->di_size = be64_to_cpu(from->di_size);
>  	to->di_nblocks = be64_to_cpu(from->di_nblocks);
>  	to->di_extsize = be32_to_cpu(from->di_extsize);
> @@ -242,9 +253,63 @@ xfs_dinode_from_disk(
>  }
>  
>  void
> -xfs_dinode_to_disk(
> -	struct xfs_dinode	*to,
> -	struct xfs_icdinode	*from)
> +xfs_inode_to_disk(
> +	struct xfs_inode	*ip,
> +	struct xfs_dinode	*to)
> +{
> +	struct xfs_icdinode	*from = &ip->i_d;
> +	struct inode		*inode = VFS_I(ip);
> +
> +	to->di_magic = cpu_to_be16(from->di_magic);
> +	to->di_mode = cpu_to_be16(from->di_mode);
> +	to->di_version = from ->di_version;
> +	to->di_format = from->di_format;
> +	to->di_onlink = cpu_to_be16(from->di_onlink);
> +	to->di_uid = cpu_to_be32(from->di_uid);
> +	to->di_gid = cpu_to_be32(from->di_gid);
> +	to->di_nlink = cpu_to_be32(from->di_nlink);
> +	to->di_projid_lo = cpu_to_be16(from->di_projid_lo);
> +	to->di_projid_hi = cpu_to_be16(from->di_projid_hi);
> +	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
> +
> +	to->di_atime.t_sec = cpu_to_be32(inode->i_atime.tv_sec);
> +	to->di_atime.t_nsec = cpu_to_be32(inode->i_atime.tv_nsec);
> +	to->di_mtime.t_sec = cpu_to_be32(inode->i_mtime.tv_sec);
> +	to->di_mtime.t_nsec = cpu_to_be32(inode->i_mtime.tv_nsec);
> +	to->di_ctime.t_sec = cpu_to_be32(inode->i_ctime.tv_sec);
> +	to->di_ctime.t_nsec = cpu_to_be32(inode->i_ctime.tv_nsec);
> +
> +	to->di_size = cpu_to_be64(from->di_size);
> +	to->di_nblocks = cpu_to_be64(from->di_nblocks);
> +	to->di_extsize = cpu_to_be32(from->di_extsize);
> +	to->di_nextents = cpu_to_be32(from->di_nextents);
> +	to->di_anextents = cpu_to_be16(from->di_anextents);
> +	to->di_forkoff = from->di_forkoff;
> +	to->di_aformat = from->di_aformat;
> +	to->di_dmevmask = cpu_to_be32(from->di_dmevmask);
> +	to->di_dmstate = cpu_to_be16(from->di_dmstate);
> +	to->di_flags = cpu_to_be16(from->di_flags);
> +	to->di_gen = cpu_to_be32(from->di_gen);
> +
> +	if (from->di_version == 3) {
> +		to->di_changecount = cpu_to_be64(from->di_changecount);
> +		to->di_crtime.t_sec = cpu_to_be32(from->di_crtime.t_sec);
> +		to->di_crtime.t_nsec = cpu_to_be32(from->di_crtime.t_nsec);
> +		to->di_flags2 = cpu_to_be64(from->di_flags2);
> +		to->di_ino = cpu_to_be64(from->di_ino);
> +		to->di_lsn = cpu_to_be64(from->di_lsn);
> +		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
> +		uuid_copy(&to->di_uuid, &from->di_uuid);
> +		to->di_flushiter = 0;
> +	} else {
> +		to->di_flushiter = cpu_to_be16(from->di_flushiter);
> +	}
> +}
> +
> +void
> +xfs_log_dinode_to_disk(
> +	struct xfs_log_dinode	*from,
> +	struct xfs_dinode	*to)
>  {
>  	to->di_magic = cpu_to_be16(from->di_magic);
>  	to->di_mode = cpu_to_be16(from->di_mode);
> @@ -257,12 +322,14 @@ xfs_dinode_to_disk(
>  	to->di_projid_lo = cpu_to_be16(from->di_projid_lo);
>  	to->di_projid_hi = cpu_to_be16(from->di_projid_hi);
>  	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
> +
>  	to->di_atime.t_sec = cpu_to_be32(from->di_atime.t_sec);
>  	to->di_atime.t_nsec = cpu_to_be32(from->di_atime.t_nsec);
>  	to->di_mtime.t_sec = cpu_to_be32(from->di_mtime.t_sec);
>  	to->di_mtime.t_nsec = cpu_to_be32(from->di_mtime.t_nsec);
>  	to->di_ctime.t_sec = cpu_to_be32(from->di_ctime.t_sec);
>  	to->di_ctime.t_nsec = cpu_to_be32(from->di_ctime.t_nsec);
> +
>  	to->di_size = cpu_to_be64(from->di_size);
>  	to->di_nblocks = cpu_to_be64(from->di_nblocks);
>  	to->di_extsize = cpu_to_be32(from->di_extsize);
> @@ -403,7 +470,7 @@ xfs_iread(
>  	 * Otherwise, just get the truly permanent information.
>  	 */
>  	if (dip->di_mode) {
> -		xfs_dinode_from_disk(&ip->i_d, dip);
> +		xfs_inode_from_disk(ip, dip);
>  		error = xfs_iformat_fork(ip, dip);
>  		if (error)  {
>  #ifdef DEBUG
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
> index 642f2a2..adcc9bf 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.h
> +++ b/fs/xfs/libxfs/xfs_inode_buf.h
> @@ -40,9 +40,6 @@ struct xfs_icdinode {
>  	__uint16_t	di_projid_hi;	/* higher part of owner's project id */
>  	__uint8_t	di_pad[6];	/* unused, zeroed space */
>  	__uint16_t	di_flushiter;	/* incremented on flush */
> -	xfs_ictimestamp_t di_atime;	/* time last accessed */
> -	xfs_ictimestamp_t di_mtime;	/* time last modified */
> -	xfs_ictimestamp_t di_ctime;	/* time created/inode modified */
>  	xfs_fsize_t	di_size;	/* number of bytes in file */
>  	xfs_rfsblock_t	di_nblocks;	/* # of direct & btree blocks used */
>  	xfs_extlen_t	di_extsize;	/* basic/minimum extent size for file */
> @@ -89,8 +86,10 @@ int	xfs_imap_to_bp(struct xfs_mount *, struct xfs_trans *,
>  int	xfs_iread(struct xfs_mount *, struct xfs_trans *,
>  		  struct xfs_inode *, uint);
>  void	xfs_dinode_calc_crc(struct xfs_mount *, struct xfs_dinode *);
> -void	xfs_dinode_to_disk(struct xfs_dinode *to, struct xfs_icdinode *from);
> -void	xfs_dinode_from_disk(struct xfs_icdinode *to, struct xfs_dinode *from);
> +void	xfs_inode_to_disk(struct xfs_inode *ip, struct xfs_dinode *to);
> +void	xfs_inode_from_disk(struct xfs_inode *ip, struct xfs_dinode *from);
> +void	xfs_log_dinode_to_disk(struct xfs_log_dinode *from,
> +			       struct xfs_dinode *to);
>  
>  #if defined(DEBUG)
>  void	xfs_inobp_check(struct xfs_mount *, struct xfs_buf *);
> diff --git a/fs/xfs/libxfs/xfs_rtbitmap.c b/fs/xfs/libxfs/xfs_rtbitmap.c
> index 9b59ffa..acc71dd 100644
> --- a/fs/xfs/libxfs/xfs_rtbitmap.c
> +++ b/fs/xfs/libxfs/xfs_rtbitmap.c
> @@ -983,7 +983,7 @@ xfs_rtfree_extent(
>  	    mp->m_sb.sb_rextents) {
>  		if (!(mp->m_rbmip->i_d.di_flags & XFS_DIFLAG_NEWRTBM))
>  			mp->m_rbmip->i_d.di_flags |= XFS_DIFLAG_NEWRTBM;
> -		*(__uint64_t *)&mp->m_rbmip->i_d.di_atime = 0;
> +		*(__uint64_t *)&VFS_I(mp->m_rbmip)->i_atime = 0;
>  		xfs_trans_log_inode(tp, mp->m_rbmip, XFS_ILOG_CORE);
>  	}
>  	return 0;
> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> index 7e24232..3e9b8a0 100644
> --- a/fs/xfs/xfs_inode.c
> +++ b/fs/xfs/xfs_inode.c
> @@ -757,6 +757,7 @@ xfs_ialloc(
>  	uint		flags;
>  	int		error;
>  	struct timespec	tv;
> +	struct inode	*inode;
>  
>  	/*
>  	 * Call the space management code to pick
> @@ -782,6 +783,7 @@ xfs_ialloc(
>  	if (error)
>  		return error;
>  	ASSERT(ip != NULL);
> +	inode = VFS_I(ip);
>  
>  	/*
>  	 * We always convert v1 inodes to v2 now - we only support filesystems
> @@ -823,10 +825,9 @@ xfs_ialloc(
>  	ASSERT(ip->i_d.di_nblocks == 0);
>  
>  	tv = current_fs_time(mp->m_super);
> -	ip->i_d.di_mtime.t_sec = (__int32_t)tv.tv_sec;
> -	ip->i_d.di_mtime.t_nsec = (__int32_t)tv.tv_nsec;
> -	ip->i_d.di_atime = ip->i_d.di_mtime;
> -	ip->i_d.di_ctime = ip->i_d.di_mtime;
> +	inode->i_mtime = tv;
> +	inode->i_atime = tv;
> +	inode->i_ctime = tv;
>  
>  	/*
>  	 * di_gen will have been taken care of in xfs_iread.
> @@ -844,7 +845,8 @@ xfs_ialloc(
>  		ip->i_d.di_lsn = 0;
>  		ip->i_d.di_flags2 = 0;
>  		memset(&(ip->i_d.di_pad2[0]), 0, sizeof(ip->i_d.di_pad2));
> -		ip->i_d.di_crtime = ip->i_d.di_mtime;
> +		ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec;
> +		ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec;
>  	}
>  
>  
> @@ -3509,12 +3511,11 @@ xfs_iflush_int(
>  		ip->i_d.di_flushiter++;
>  
>  	/*
> -	 * Copy the dirty parts of the inode into the on-disk
> -	 * inode.  We always copy out the core of the inode,
> -	 * because if the inode is dirty at all the core must
> -	 * be.
> +	 * Copy the dirty parts of the inode into the on-disk inode.  We always
> +	 * copy out the core of the inode, because if the inode is dirty at all
> +	 * the core must be.
>  	 */
> -	xfs_dinode_to_disk(dip, &ip->i_d);
> +	xfs_inode_to_disk(ip, dip);
>  
>  	/* Wrap, we never let the log put out DI_MAX_FLUSH */
>  	if (ip->i_d.di_flushiter == DI_MAX_FLUSH)
> diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
> index 3ad9972..9dcbf58 100644
> --- a/fs/xfs/xfs_inode_item.c
> +++ b/fs/xfs/xfs_inode_item.c
> @@ -323,10 +323,13 @@ xfs_inode_item_format_attr_fork(
>  }
>  
>  static void
> -xfs_icdinode_to_log_dinode(
> -	struct xfs_icdinode	*from,
> +xfs_inode_to_log_dinode(
> +	struct xfs_inode	*ip,
>  	struct xfs_log_dinode	*to)
>  {
> +	struct xfs_icdinode	*from = &ip->i_d;
> +	struct inode		*inode = VFS_I(ip);
> +
>  	to->di_magic = from->di_magic;
>  	to->di_mode = from->di_mode;
>  	to->di_version = from->di_version;
> @@ -338,66 +341,14 @@ xfs_icdinode_to_log_dinode(
>  	to->di_projid_lo = from->di_projid_lo;
>  	to->di_projid_hi = from->di_projid_hi;
>  	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
> -	to->di_atime.t_sec = from->di_atime.t_sec;
> -	to->di_atime.t_nsec = from->di_atime.t_nsec;
> -	to->di_mtime.t_sec = from->di_mtime.t_sec;
> -	to->di_mtime.t_nsec = from->di_mtime.t_nsec;
> -	to->di_ctime.t_sec = from->di_ctime.t_sec;
> -	to->di_ctime.t_nsec = from->di_ctime.t_nsec;
> -	to->di_size = from->di_size;
> -	to->di_nblocks = from->di_nblocks;
> -	to->di_extsize = from->di_extsize;
> -	to->di_nextents = from->di_nextents;
> -	to->di_anextents = from->di_anextents;
> -	to->di_forkoff = from->di_forkoff;
> -	to->di_aformat = from->di_aformat;
> -	to->di_dmevmask = from->di_dmevmask;
> -	to->di_dmstate = from->di_dmstate;
> -	to->di_flags = from->di_flags;
> -	to->di_gen = from->di_gen;
>  
> -	if (from->di_version == 3) {
> -		to->di_changecount = from->di_changecount;
> -		to->di_crtime.t_sec = from->di_crtime.t_sec;
> -		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
> -		to->di_flags2 = from->di_flags2;
> -		to->di_ino = from->di_ino;
> -		to->di_lsn = from->di_lsn;
> -		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
> -		uuid_copy(&to->di_uuid, &from->di_uuid);
> -		to->di_flushiter = 0;
> -	} else {
> -		to->di_flushiter = from->di_flushiter;
> -	}
> -}
> +	to->di_atime.t_sec = inode->i_atime.tv_sec;
> +	to->di_atime.t_nsec = inode->i_atime.tv_nsec;
> +	to->di_mtime.t_sec = inode->i_mtime.tv_sec;
> +	to->di_mtime.t_nsec = inode->i_mtime.tv_nsec;
> +	to->di_ctime.t_sec = inode->i_ctime.tv_sec;
> +	to->di_ctime.t_nsec = inode->i_ctime.tv_nsec;
>  
> -/*
> - * Recovery needs to be able to convert a log dinode back to a real dinode
> - * for writeback we do that by converting a log dinode to a icdinode, and
> - * then passing that to the formatting function.
> - */
> -void
> -xfs_log_dinode_to_icdinode(
> -	struct xfs_log_dinode	*from,
> -	struct xfs_icdinode	*to)
> -{
> -	to->di_magic = from->di_magic;
> -	to->di_mode = from->di_mode;
> -	to->di_version = from->di_version;
> -	to->di_format = from->di_format;
> -	to->di_onlink = from->di_onlink;
> -	to->di_uid = from->di_uid;
> -	to->di_gid = from->di_gid;
> -	to->di_nlink = from->di_nlink;
> -	to->di_projid_lo = from->di_projid_lo;
> -	to->di_projid_hi = from->di_projid_hi;
> -	memset(to->di_pad, 0, sizeof(to->di_pad));
> -	to->di_atime.t_sec = from->di_atime.t_sec;
> -	to->di_atime.t_nsec = from->di_atime.t_nsec;
> -	to->di_mtime.t_sec = from->di_mtime.t_sec;
> -	to->di_mtime.t_nsec = from->di_mtime.t_nsec;
> -	to->di_ctime.t_sec = from->di_ctime.t_sec;
> -	to->di_ctime.t_nsec = from->di_ctime.t_nsec;
>  	to->di_size = from->di_size;
>  	to->di_nblocks = from->di_nblocks;
>  	to->di_extsize = from->di_extsize;
> @@ -439,7 +390,7 @@ xfs_inode_item_format_core(
>  	struct xfs_log_dinode	*dic;
>  
>  	dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE);
> -	xfs_icdinode_to_log_dinode(&ip->i_d, dic);
> +	xfs_inode_to_log_dinode(ip, dic);
>  	xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version));
>  }
>  
> diff --git a/fs/xfs/xfs_inode_item.h b/fs/xfs/xfs_inode_item.h
> index 2426118..4c7722e 100644
> --- a/fs/xfs/xfs_inode_item.h
> +++ b/fs/xfs/xfs_inode_item.h
> @@ -49,8 +49,6 @@ extern void xfs_istale_done(struct xfs_buf *, struct xfs_log_item *);
>  extern void xfs_iflush_abort(struct xfs_inode *, bool);
>  extern int xfs_inode_item_format_convert(xfs_log_iovec_t *,
>  					 xfs_inode_log_format_t *);
> -extern void xfs_log_dinode_to_icdinode(struct xfs_log_dinode *from,
> -				       struct xfs_icdinode *to);
>  
>  extern struct kmem_zone	*xfs_ili_zone;
>  
> diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
> index 245268a..b008677 100644
> --- a/fs/xfs/xfs_iops.c
> +++ b/fs/xfs/xfs_iops.c
> @@ -517,21 +517,12 @@ xfs_setattr_time(
>  
>  	ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL));
>  
> -	if (iattr->ia_valid & ATTR_ATIME) {
> +	if (iattr->ia_valid & ATTR_ATIME)
>  		inode->i_atime = iattr->ia_atime;
> -		ip->i_d.di_atime.t_sec = iattr->ia_atime.tv_sec;
> -		ip->i_d.di_atime.t_nsec = iattr->ia_atime.tv_nsec;
> -	}
> -	if (iattr->ia_valid & ATTR_CTIME) {
> +	if (iattr->ia_valid & ATTR_CTIME)
>  		inode->i_ctime = iattr->ia_ctime;
> -		ip->i_d.di_ctime.t_sec = iattr->ia_ctime.tv_sec;
> -		ip->i_d.di_ctime.t_nsec = iattr->ia_ctime.tv_nsec;
> -	}
> -	if (iattr->ia_valid & ATTR_MTIME) {
> +	if (iattr->ia_valid & ATTR_MTIME)
>  		inode->i_mtime = iattr->ia_mtime;
> -		ip->i_d.di_mtime.t_sec = iattr->ia_mtime.tv_sec;
> -		ip->i_d.di_mtime.t_nsec = iattr->ia_mtime.tv_nsec;
> -	}
>  }
>  
>  int
> @@ -986,21 +977,13 @@ xfs_vn_update_time(
>  	}
>  
>  	xfs_ilock(ip, XFS_ILOCK_EXCL);
> -	if (flags & S_CTIME) {
> +	if (flags & S_CTIME)
>  		inode->i_ctime = *now;
> -		ip->i_d.di_ctime.t_sec = (__int32_t)now->tv_sec;
> -		ip->i_d.di_ctime.t_nsec = (__int32_t)now->tv_nsec;
> -	}
> -	if (flags & S_MTIME) {
> +	if (flags & S_MTIME)
>  		inode->i_mtime = *now;
> -		ip->i_d.di_mtime.t_sec = (__int32_t)now->tv_sec;
> -		ip->i_d.di_mtime.t_nsec = (__int32_t)now->tv_nsec;
> -	}
> -	if (flags & S_ATIME) {
> +	if (flags & S_ATIME)
>  		inode->i_atime = *now;
> -		ip->i_d.di_atime.t_sec = (__int32_t)now->tv_sec;
> -		ip->i_d.di_atime.t_nsec = (__int32_t)now->tv_nsec;
> -	}
> +
>  	xfs_trans_ijoin(tp, ip, XFS_ILOCK_EXCL);
>  	xfs_trans_log_inode(tp, ip, XFS_ILOG_TIMESTAMP);
>  	return xfs_trans_commit(tp);
> @@ -1247,12 +1230,6 @@ xfs_setup_inode(
>  
>  	inode->i_generation = ip->i_d.di_gen;
>  	i_size_write(inode, ip->i_d.di_size);
> -	inode->i_atime.tv_sec	= ip->i_d.di_atime.t_sec;
> -	inode->i_atime.tv_nsec	= ip->i_d.di_atime.t_nsec;
> -	inode->i_mtime.tv_sec	= ip->i_d.di_mtime.t_sec;
> -	inode->i_mtime.tv_nsec	= ip->i_d.di_mtime.t_nsec;
> -	inode->i_ctime.tv_sec	= ip->i_d.di_ctime.t_sec;
> -	inode->i_ctime.tv_nsec	= ip->i_d.di_ctime.t_nsec;
>  	xfs_diflags_to_iflags(inode, ip);
>  
>  	ip->d_ops = ip->i_mount->m_nondir_inode_ops;
> diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
> index 930ebd8..2acda42 100644
> --- a/fs/xfs/xfs_itable.c
> +++ b/fs/xfs/xfs_itable.c
> @@ -57,6 +57,7 @@ xfs_bulkstat_one_int(
>  {
>  	struct xfs_icdinode	*dic;		/* dinode core info pointer */
>  	struct xfs_inode	*ip;		/* incore inode pointer */
> +	struct inode		*inode;
>  	struct xfs_bstat	*buf;		/* return buffer */
>  	int			error = 0;	/* error value */
>  
> @@ -77,6 +78,7 @@ xfs_bulkstat_one_int(
>  
>  	ASSERT(ip != NULL);
>  	ASSERT(ip->i_imap.im_blkno != 0);
> +	inode = VFS_I(ip);
>  
>  	dic = &ip->i_d;
>  
> @@ -91,12 +93,14 @@ xfs_bulkstat_one_int(
>  	buf->bs_uid = dic->di_uid;
>  	buf->bs_gid = dic->di_gid;
>  	buf->bs_size = dic->di_size;
> -	buf->bs_atime.tv_sec = dic->di_atime.t_sec;
> -	buf->bs_atime.tv_nsec = dic->di_atime.t_nsec;
> -	buf->bs_mtime.tv_sec = dic->di_mtime.t_sec;
> -	buf->bs_mtime.tv_nsec = dic->di_mtime.t_nsec;
> -	buf->bs_ctime.tv_sec = dic->di_ctime.t_sec;
> -	buf->bs_ctime.tv_nsec = dic->di_ctime.t_nsec;
> +
> +	buf->bs_atime.tv_sec = inode->i_atime.tv_sec;
> +	buf->bs_atime.tv_nsec = inode->i_atime.tv_nsec;
> +	buf->bs_mtime.tv_sec = inode->i_mtime.tv_sec;
> +	buf->bs_mtime.tv_nsec = inode->i_mtime.tv_nsec;
> +	buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec;
> +	buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec;
> +
>  	buf->bs_xflags = xfs_ip2xflags(ip);
>  	buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog;
>  	buf->bs_extents = dic->di_nextents;
> diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
> index 3120f7b..4b79cf0 100644
> --- a/fs/xfs/xfs_log_recover.c
> +++ b/fs/xfs/xfs_log_recover.c
> @@ -2793,7 +2793,7 @@ xfs_recover_inode_owner_change(
>  		return -ENOMEM;
>  
>  	/* instantiate the inode */
> -	xfs_dinode_from_disk(&ip->i_d, dip);
> +	xfs_inode_from_disk(ip, dip);
>  	ASSERT(ip->i_d.di_version >= 3);
>  
>  	error = xfs_iformat_fork(ip, dip);
> @@ -2840,7 +2840,6 @@ xlog_recover_inode_pass2(
>  	int			attr_index;
>  	uint			fields;
>  	struct xfs_log_dinode	*ldip;
> -	struct xfs_icdinode	icic;
>  	uint			isize;
>  	int			need_free = 0;
>  
> @@ -3007,9 +3006,8 @@ xlog_recover_inode_pass2(
>  		goto out_release;
>  	}
>  
> -	/* The core is in in-core format */
> -	xfs_log_dinode_to_icdinode(ldip, &icic);
> -	xfs_dinode_to_disk(dip, &icic);
> +	/* recover the log dinode inode into the on disk inode */
> +	xfs_log_dinode_to_disk(ldip, dip);
>  
>  	/* the rest is in on-disk format */
>  	if (item->ri_buf[1].i_len > isize) {
> diff --git a/fs/xfs/xfs_rtalloc.c b/fs/xfs/xfs_rtalloc.c
> index be02a68..abf4443 100644
> --- a/fs/xfs/xfs_rtalloc.c
> +++ b/fs/xfs/xfs_rtalloc.c
> @@ -1272,7 +1272,7 @@ xfs_rtpick_extent(
>  
>  	ASSERT(xfs_isilocked(mp->m_rbmip, XFS_ILOCK_EXCL));
>  
> -	seqp = (__uint64_t *)&mp->m_rbmip->i_d.di_atime;
> +	seqp = (__uint64_t *)&VFS_I(mp->m_rbmip)->i_atime;
>  	if (!(mp->m_rbmip->i_d.di_flags & XFS_DIFLAG_NEWRTBM)) {
>  		mp->m_rbmip->i_d.di_flags |= XFS_DIFLAG_NEWRTBM;
>  		*seqp = 0;
> diff --git a/fs/xfs/xfs_trans_inode.c b/fs/xfs/xfs_trans_inode.c
> index b97f1df..3f0d466 100644
> --- a/fs/xfs/xfs_trans_inode.c
> +++ b/fs/xfs/xfs_trans_inode.c
> @@ -75,18 +75,10 @@ xfs_trans_ichgtime(
>  
>  	tv = current_fs_time(inode->i_sb);
>  
> -	if ((flags & XFS_ICHGTIME_MOD) &&
> -	    !timespec_equal(&inode->i_mtime, &tv)) {
> +	if (flags & XFS_ICHGTIME_MOD)
>  		inode->i_mtime = tv;
> -		ip->i_d.di_mtime.t_sec = tv.tv_sec;
> -		ip->i_d.di_mtime.t_nsec = tv.tv_nsec;
> -	}
> -	if ((flags & XFS_ICHGTIME_CHG) &&
> -	    !timespec_equal(&inode->i_ctime, &tv)) {
> +	if (flags & XFS_ICHGTIME_CHG)
>  		inode->i_ctime = tv;
> -		ip->i_d.di_ctime.t_sec = tv.tv_sec;
> -		ip->i_d.di_ctime.t_nsec = tv.tv_nsec;
> -	}
>  }
>  
>  /*
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Mon Jan 25 09:43:47 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 78F687CAD
	for ; Mon, 25 Jan 2016 09:43:47 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 597C28F8033
	for ; Mon, 25 Jan 2016 07:43:47 -0800 (PST)
X-ASG-Debug-ID: 1453736625-04cbb026ffbb500001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id P4bitZiSraTTTmwJ (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 07:43:46 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26])
	by mx1.redhat.com (Postfix) with ESMTPS id A766C8F4E1;
	Mon, 25 Jan 2016 15:43:45 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PFhjhT026944;
	Mon, 25 Jan 2016 10:43:45 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id C5D47120BCD; Mon, 25 Jan 2016 10:43:44 -0500 (EST)
Date: Mon, 25 Jan 2016 10:43:44 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 3/8] xfs; cull unnecessary icdinode fields
Message-ID: <20160125154344.GC4108@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 3/8] xfs; cull unnecessary icdinode fields
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
 <1452751765-4420-4-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452751765-4420-4-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453736626
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 14, 2016 at 05:09:20PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> Now that the struct xfs_icdinode is not directly related to the
> on-disk format, we can cull things in it we really don't need to
> store:
> 
> 	- magic number never changes
> 	- padding is not necessary
> 	- next_unlinked is never used
> 	- inode number is redundant
> 	- uuid is redundant
> 	- lsn is accessed directly from dinode
> 	- inode CRC is only accessed directly from dinode
> 
> Hence we can remove these from the struct xfs_icdinode and redirect
> the code that uses them to the xfs_dinode appripriately.  This
> reduces the size of the struct icdinode from 152 bytes to 88 bytes,
> and removes a fair chunk of unnecessary code, too.
> 
> Signed-off-by: Dave Chinner 
> ---

A couple nits and a minor potential bug: you have a semicolon in the
subject header rather than a colon...

>  fs/xfs/libxfs/xfs_inode_buf.c | 39 +++++++++++++--------------------------
>  fs/xfs/libxfs/xfs_inode_buf.h | 27 +++++++--------------------
>  fs/xfs/xfs_inode.c            | 19 +------------------
>  fs/xfs/xfs_inode_item.c       | 19 +++++++++++--------
>  4 files changed, 32 insertions(+), 72 deletions(-)
> 
...
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
> index adcc9bf..69d626e 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.h
> +++ b/fs/xfs/libxfs/xfs_inode_buf.h
> @@ -22,24 +22,22 @@ struct xfs_inode;
>  struct xfs_dinode;
>  
>  /*
> - * In memory representation of the XFS inode. This is held in the in-core
> - * struct xfs_inode to represent the on disk values, but no longer needs to be
> - * identical to the on-disk structure as it is always translated to on-disk
> - * format specific structures at the appropriate time.
> + * In memory representation of the XFS inode. This is held in the in-core struct
> + * xfs_inode to represent the on disk values, but it's struct is in no way

Nit:						     its

> + * related to what is stored on disk. That is, this structure is always
> + * translated to on-disk format specific structures at the appropriate time.

FWIW, it might be more clear to say something like: "This is held in the
in-core struct xfs_inode and represents on-disk values, but the
structure is not in on-disk format. In other words, the structure is
always translated to on-disk format specific structures ... "

>   */
>  struct xfs_icdinode {
> -	__uint16_t	di_magic;	/* inode magic # = XFS_DINODE_MAGIC */
>  	__uint16_t	di_mode;	/* mode and type of file */
>  	__int8_t	di_version;	/* inode version */
>  	__int8_t	di_format;	/* format of di_c data */
>  	__uint16_t	di_onlink;	/* old number of links to file */
> +	__uint16_t	di_flushiter;	/* incremented on flush */
>  	__uint32_t	di_uid;		/* owner's user id */
>  	__uint32_t	di_gid;		/* owner's group id */
>  	__uint32_t	di_nlink;	/* number of links to file */
>  	__uint16_t	di_projid_lo;	/* lower part of owner's project id */
>  	__uint16_t	di_projid_hi;	/* higher part of owner's project id */
> -	__uint8_t	di_pad[6];	/* unused, zeroed space */
> -	__uint16_t	di_flushiter;	/* incremented on flush */
>  	xfs_fsize_t	di_size;	/* number of bytes in file */
>  	xfs_rfsblock_t	di_nblocks;	/* # of direct & btree blocks used */
>  	xfs_extlen_t	di_extsize;	/* basic/minimum extent size for file */
...
> diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
> index 9dcbf58..ae60087 100644
> --- a/fs/xfs/xfs_inode_item.c
> +++ b/fs/xfs/xfs_inode_item.c
> @@ -325,12 +325,14 @@ xfs_inode_item_format_attr_fork(
>  static void
>  xfs_inode_to_log_dinode(
>  	struct xfs_inode	*ip,
> -	struct xfs_log_dinode	*to)
> +	struct xfs_log_dinode	*to,
> +	xfs_lsn_t		lsn)
>  {
>  	struct xfs_icdinode	*from = &ip->i_d;
>  	struct inode		*inode = VFS_I(ip);
>  
> -	to->di_magic = from->di_magic;
> +	to->di_magic = XFS_DINODE_MAGIC;
> +
>  	to->di_mode = from->di_mode;
>  	to->di_version = from->di_version;
>  	to->di_format = from->di_format;
> @@ -340,8 +342,8 @@ xfs_inode_to_log_dinode(
>  	to->di_nlink = from->di_nlink;
>  	to->di_projid_lo = from->di_projid_lo;
>  	to->di_projid_hi = from->di_projid_hi;
> -	memcpy(to->di_pad, from->di_pad, sizeof(to->di_pad));
>  
> +	memset(to->di_pad, 0, sizeof(to->di_pad));
>  	to->di_atime.t_sec = inode->i_atime.tv_sec;
>  	to->di_atime.t_nsec = inode->i_atime.tv_nsec;
>  	to->di_mtime.t_sec = inode->i_mtime.tv_sec;
> @@ -366,10 +368,11 @@ xfs_inode_to_log_dinode(
>  		to->di_crtime.t_sec = from->di_crtime.t_sec;
>  		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
>  		to->di_flags2 = from->di_flags2;
> -		to->di_ino = from->di_ino;
> -		to->di_lsn = from->di_lsn;
> -		memcpy(to->di_pad2, from->di_pad2, sizeof(to->di_pad2));
> -		uuid_copy(&to->di_uuid, &from->di_uuid);
> +
> +		to->di_ino = ip->i_ino;
> +		to->di_lsn = lsn;
> +		memset(to->di_pad2, 0, sizeof(to->di_pad2));
> +		uuid_copy(&to->di_uuid, &ip->i_mount->m_sb.sb_uuid);

Shouldn't this be sb_meta_uuid?

Brian

>  		to->di_flushiter = 0;
>  	} else {
>  		to->di_flushiter = from->di_flushiter;
> @@ -390,7 +393,7 @@ xfs_inode_item_format_core(
>  	struct xfs_log_dinode	*dic;
>  
>  	dic = xlog_prepare_iovec(lv, vecp, XLOG_REG_TYPE_ICORE);
> -	xfs_inode_to_log_dinode(ip, dic);
> +	xfs_inode_to_log_dinode(ip, dic, ip->i_itemp->ili_item.li_lsn);
>  	xlog_finish_iovec(lv, *vecp, xfs_log_dinode_size(ip->i_d.di_version));
>  }
>  
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Mon Jan 25 09:43:51 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 9537B7CA2
	for ; Mon, 25 Jan 2016 09:43:51 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 8ECE58F8035
	for ; Mon, 25 Jan 2016 07:43:51 -0800 (PST)
X-ASG-Debug-ID: 1453736629-04cb6c0d60c1c30001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 6DfEHmZ4DnEY4GGx (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 07:43:50 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26])
	by mx1.redhat.com (Postfix) with ESMTPS id 74999B6A2E;
	Mon, 25 Jan 2016 15:43:49 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PFhnIp026972;
	Mon, 25 Jan 2016 10:43:49 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 97625120BCD; Mon, 25 Jan 2016 10:43:48 -0500 (EST)
Date: Mon, 25 Jan 2016 10:43:48 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 4/8] xfs: move v1 inode conversion to xfs_inode_from_disk
Message-ID: <20160125154348.GD4108@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 4/8] xfs: move v1 inode conversion to xfs_inode_from_disk
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
 <1452751765-4420-5-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452751765-4420-5-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453736629
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 14, 2016 at 05:09:21PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> So we don't have to carry an di_onlink variable around anymore, move
> the inode conversion from v1 inode format to v2 inode format into
> xfs_inode_from_disk(). This means we can remove the di_onlink fields
> from the struct xfs_icdinode.
> 
> Signed-off-by: Dave Chinner 
> ---

Reviewed-by: Brian Foster 

>  fs/xfs/libxfs/xfs_inode_buf.c  | 42 ++++++++++++++++++++----------------------
>  fs/xfs/libxfs/xfs_inode_buf.h  |  1 -
>  fs/xfs/libxfs/xfs_log_format.h |  2 +-
>  fs/xfs/xfs_inode.c             |  2 --
>  fs/xfs/xfs_inode_item.c        |  2 +-
>  5 files changed, 22 insertions(+), 27 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
> index 17ff4e8..8a1b460 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.c
> +++ b/fs/xfs/libxfs/xfs_inode_buf.c
> @@ -204,13 +204,25 @@ xfs_inode_from_disk(
>  
>  	to->di_mode = be16_to_cpu(from->di_mode);
>  	to->di_version = from ->di_version;
> +
> +	/*
> +	 * Convert v1 inodes immediately to v2 inode format as this is the
> +	 * minimum inode version format we support in the rest of the code.
> +	 */
> +	if (to->di_version == 1) {
> +		to->di_nlink = be16_to_cpu(from->di_onlink);
> +		to->di_projid_lo = 0;
> +		to->di_projid_hi = 0;
> +		to->di_version = 2;
> +	} else {
> +		to->di_nlink = be32_to_cpu(from->di_nlink);
> +		to->di_projid_lo = be16_to_cpu(from->di_projid_lo);
> +		to->di_projid_hi = be16_to_cpu(from->di_projid_hi);
> +	}
> +
>  	to->di_format = from->di_format;
> -	to->di_onlink = be16_to_cpu(from->di_onlink);
>  	to->di_uid = be32_to_cpu(from->di_uid);
>  	to->di_gid = be32_to_cpu(from->di_gid);
> -	to->di_nlink = be32_to_cpu(from->di_nlink);
> -	to->di_projid_lo = be16_to_cpu(from->di_projid_lo);
> -	to->di_projid_hi = be16_to_cpu(from->di_projid_hi);
>  	to->di_flushiter = be16_to_cpu(from->di_flushiter);
>  
>  	/*
> @@ -256,11 +268,11 @@ xfs_inode_to_disk(
>  	struct inode		*inode = VFS_I(ip);
>  
>  	to->di_magic = cpu_to_be16(XFS_DINODE_MAGIC);
> +	to->di_onlink = 0;
>  
>  	to->di_mode = cpu_to_be16(from->di_mode);
>  	to->di_version = from->di_version;
>  	to->di_format = from->di_format;
> -	to->di_onlink = cpu_to_be16(from->di_onlink);
>  	to->di_uid = cpu_to_be32(from->di_uid);
>  	to->di_gid = cpu_to_be32(from->di_gid);
>  	to->di_nlink = cpu_to_be32(from->di_nlink);
> @@ -310,9 +322,9 @@ xfs_log_dinode_to_disk(
>  {
>  	to->di_magic = cpu_to_be16(from->di_magic);
>  	to->di_mode = cpu_to_be16(from->di_mode);
> -	to->di_version = from ->di_version;
> +	to->di_version = from->di_version;
>  	to->di_format = from->di_format;
> -	to->di_onlink = cpu_to_be16(from->di_onlink);
> +	to->di_onlink = 0;
>  	to->di_uid = cpu_to_be32(from->di_uid);
>  	to->di_gid = cpu_to_be32(from->di_gid);
>  	to->di_nlink = cpu_to_be32(from->di_nlink);
> @@ -492,21 +504,7 @@ xfs_iread(
>  		ip->i_d.di_mode = 0;
>  	}
>  
> -	/*
> -	 * Automatically convert version 1 inode formats in memory to version 2
> -	 * inode format. If the inode is modified, it will get logged and
> -	 * rewritten as a version 2 inode. We can do this because we set the
> -	 * superblock feature bit for v2 inodes unconditionally during mount
> -	 * and it means the reast of the code can assume the inode version is 2
> -	 * or higher.
> -	 */
> -	if (ip->i_d.di_version == 1) {
> -		ip->i_d.di_version = 2;
> -		ip->i_d.di_nlink = ip->i_d.di_onlink;
> -		ip->i_d.di_onlink = 0;
> -		xfs_set_projid(ip, 0);
> -	}
> -
> +	ASSERT(ip->i_d.di_version >= 2);
>  	ip->i_delayed_blks = 0;
>  
>  	/*
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
> index 69d626e..feb04e6 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.h
> +++ b/fs/xfs/libxfs/xfs_inode_buf.h
> @@ -31,7 +31,6 @@ struct xfs_icdinode {
>  	__uint16_t	di_mode;	/* mode and type of file */
>  	__int8_t	di_version;	/* inode version */
>  	__int8_t	di_format;	/* format of di_c data */
> -	__uint16_t	di_onlink;	/* old number of links to file */
>  	__uint16_t	di_flushiter;	/* incremented on flush */
>  	__uint32_t	di_uid;		/* owner's user id */
>  	__uint32_t	di_gid;		/* owner's group id */
> diff --git a/fs/xfs/libxfs/xfs_log_format.h b/fs/xfs/libxfs/xfs_log_format.h
> index d00ed63..03f90b9 100644
> --- a/fs/xfs/libxfs/xfs_log_format.h
> +++ b/fs/xfs/libxfs/xfs_log_format.h
> @@ -369,7 +369,7 @@ struct xfs_log_dinode {
>  	__uint16_t	di_mode;	/* mode and type of file */
>  	__int8_t	di_version;	/* inode version */
>  	__int8_t	di_format;	/* format of di_c data */
> -	__uint16_t	di_onlink;	/* old number of links to file */
> +	__uint8_t	di_pad3[2];	/* unused in v2/3 inodes */
>  	__uint32_t	di_uid;		/* owner's user id */
>  	__uint32_t	di_gid;		/* owner's group id */
>  	__uint32_t	di_nlink;	/* number of links to file */
> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> index 3f020b5..914ec41 100644
> --- a/fs/xfs/xfs_inode.c
> +++ b/fs/xfs/xfs_inode.c
> @@ -794,9 +794,7 @@ xfs_ialloc(
>  		ip->i_d.di_version = 2;
>  
>  	ip->i_d.di_mode = mode;
> -	ip->i_d.di_onlink = 0;
>  	ip->i_d.di_nlink = nlink;
> -	ASSERT(ip->i_d.di_nlink == nlink);
>  	ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid());
>  	ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid());
>  	xfs_set_projid(ip, prid);
> diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
> index ae60087..d777b7a 100644
> --- a/fs/xfs/xfs_inode_item.c
> +++ b/fs/xfs/xfs_inode_item.c
> @@ -336,7 +336,6 @@ xfs_inode_to_log_dinode(
>  	to->di_mode = from->di_mode;
>  	to->di_version = from->di_version;
>  	to->di_format = from->di_format;
> -	to->di_onlink = from->di_onlink;
>  	to->di_uid = from->di_uid;
>  	to->di_gid = from->di_gid;
>  	to->di_nlink = from->di_nlink;
> @@ -344,6 +343,7 @@ xfs_inode_to_log_dinode(
>  	to->di_projid_hi = from->di_projid_hi;
>  
>  	memset(to->di_pad, 0, sizeof(to->di_pad));
> +	memset(to->di_pad3, 0, sizeof(to->di_pad3));
>  	to->di_atime.t_sec = inode->i_atime.tv_sec;
>  	to->di_atime.t_nsec = inode->i_atime.tv_nsec;
>  	to->di_mtime.t_sec = inode->i_mtime.tv_sec;
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From sandeen@redhat.com  Mon Jan 25 10:04:38 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id C29EC7CA1
	for ; Mon, 25 Jan 2016 10:04:38 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 4D753AC002
	for ; Mon, 25 Jan 2016 08:04:35 -0800 (PST)
X-ASG-Debug-ID: 1453737873-04cb6c0d5fc2710001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id xVhW1pLy8I0csfBF (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 08:04:34 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	by mx1.redhat.com (Postfix) with ESMTPS id 8738E8E69C;
	Mon, 25 Jan 2016 16:04:33 +0000 (UTC)
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PG4WeY022798
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Mon, 25 Jan 2016 11:04:33 -0500
Subject: Re: [PATCH 0/7 V3] quota: add new quotactl Q_GETNEXTQUOTA
To: Jan Kara 
X-ASG-Orig-Subj: Re: [PATCH 0/7 V3] quota: add new quotactl Q_GETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <20160125150710.GN24938@quack.suse.cz>
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com,
        Dave Chinner 
From: Eric Sandeen 
Message-ID: <56A64790.9090100@redhat.com>
Date: Mon, 25 Jan 2016 10:04:32 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160125150710.GN24938@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453737874
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On 1/25/16 9:07 AM, Jan Kara wrote:
> On Fri 22-01-16 12:25:29, Eric Sandeen wrote:

...

> So I've taken patch 1/7 into my tree since that is completely independent.
> Patches 2/7 and 3/7 look mostly good to me (except for that small omission
> in 3/7) so I can take them to my tree as well. Once I'll finish testing
> some UDF changes I have pending I'll push the tree to:

Jan, thanks for catching & fixing those issues.

> git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git for_next
> 
> and Dave can pull from there to have a basis for XFS patches. I want to
> implement Q_GETNEXTQUOTA for VFS quota formats as well and those will go
> through my tree so that's why I want the interface changes go via my tree.

Makes sense to me.

Thanks,
-Eric

From mjseger@gmail.com  Mon Jan 25 10:38:14 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,
	T_DKIM_INVALID autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id C90187CA1
	for ; Mon, 25 Jan 2016 10:38:13 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id F189BAC008
	for ; Mon, 25 Jan 2016 08:38:12 -0800 (PST)
X-ASG-Debug-ID: 1453739887-04cbb026ffbcbc0001-NocioJ
Received: from mail-wm0-f43.google.com (mail-wm0-f43.google.com [74.125.82.43]) by cuda.sgi.com with ESMTP id 9hEBcIdn16NE0KqO (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 25 Jan 2016 08:38:08 -0800 (PST)
X-Barracuda-Envelope-From: mjseger@gmail.com
X-Barracuda-Apparent-Source-IP: 74.125.82.43
Received: by mail-wm0-f43.google.com with SMTP id l65so71429710wmf.1
        for ; Mon, 25 Jan 2016 08:38:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        bh=QzxDNPJBnylT4cCi9l+HdXqaLLrFHbZzI6pswIiCMvw=;
        b=i4t2v5w/eCc8JxzMJZLwAwbiHvG7+tDiOg993NwifGVG8gzb2xbE5b/vuRAc5fzAGb
         8QWitaqzs097IdhdRkSoGPa23guaBokDzUjZcVpWb0FflO5yRtICKiUZNB0BaC5pYOAH
         jjpCEV/EVmyRn+prSjCeU/Hk1adxqrl18dokiUfKnlC5lqw7cd0rPHVjxCobue/U0WhQ
         NcrfPW6cq6sj+ftnzLzXE+QZcBgGpA/8fbnSzifx0tTns8RBz4P878dnMupTfSQBMqDx
         MxnPmAsfjZlaTVdl5VZI9XPjwhfCaf8getobQnt18FGmPYZLI6wY+Q5YuZd84l58zskn
         Z0xg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=QzxDNPJBnylT4cCi9l+HdXqaLLrFHbZzI6pswIiCMvw=;
        b=Zsx5T8CNBXbsGe42vQYfCc7eD9xZ1G/yeNZ/1ojkXa9rXCt1v23TeITEHXa3oN5zi6
         6rJe5rFTUAd9yjBQLti2p7ZgbaHfzCDpD9zK+fPZL/BH8iof05ya8nVy4X5hYM2uipEh
         lAe4zvJ0tqqBAwMAs6f5rvftMI60oaKV/pswzLXW4l5OhT28xBCmxzi6HNMRc1EFOW86
         QlvKKYNCPgAHDikB5noFotK6ezEd4xDKVw3ZvHwfADWkvXIt5xVf9VJickTuuwz/tzIc
         0KhdX9wWavSALnPGJZ3xh7rzUD6zBjWNzhdewVxBmWRiJfkQiGLsdTB3ZvtVrrmg996S
         Esmg==
X-Gm-Message-State: AG10YOQLpfA71BNJfvvCW9qTntkEBKkjh10vpjPQeUB6+rp9UFkynglYz562dFOBRv+jmrsMeB8NDS6Fax8eew==
MIME-Version: 1.0
X-Received: by 10.194.19.101 with SMTP id d5mr5016746wje.72.1453739887405;
 Mon, 25 Jan 2016 08:38:07 -0800 (PST)
Received: by 10.194.78.99 with HTTP; Mon, 25 Jan 2016 08:38:07 -0800 (PST)
In-Reply-To: <20160106234904.GL21461@dastard>
References: 
	<20160106220454.GI21461@dastard>
	<20160106221004.GJ21461@dastard>
	
	<20160106234904.GL21461@dastard>
Date: Mon, 25 Jan 2016 11:38:07 -0500
Message-ID: 
Subject: Re: xfs and swift
From: Mark Seger 
X-ASG-Orig-Subj: Re: xfs and swift
To: Dave Chinner 
Cc: Laurence Oberman , Linux fs XFS 
Content-Type: multipart/alternative; boundary=047d7b5d457e95b6bc052a2b31af
X-Barracuda-Connect: mail-wm0-f43.google.com[74.125.82.43]
X-Barracuda-Start-Time: 1453739888
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_SA210e, DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26435
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.00 BSF_SC5_SA210e         Custom Rule SA210e

--047d7b5d457e95b6bc052a2b31af
Content-Type: text/plain; charset=UTF-8

since getting your last reply I've been doing a lot more trying to
understand the behavior of what I'm seeing by writing some non-swift code
that sort of does what swift does with respect to a directory structure.
 in my case I have 1024 top level dirs, 4096 under each.  each 1k file I'm
creating gets it's only directory under these so there are clearly a lot of
directories.

xfs writes out about 25M objects and then the performance goes into the
toilet.  I'm sure what you said before about having to flush data and
causing big delays, but would it be continuous?  each entry in the
following table shows the time to write 10K files so the 2 blocks are 1M
each

Sat Jan 23 12:15:09 2016
 16.114386  14.656736  14.789760  17.418389  14.613157  15.938176
 14.865369  14.962058  17.297193  15.953590
 14.895471  15.560252  14.789937  14.308618  16.390057  16.561789
 15.713806  14.843791  15.940992  16.466924
 15.842781  15.611230  17.102329  15.006291  14.454088  17.923662
 13.378340  16.084664  15.996794  13.736398
 18.125125  14.462063  18.101833  15.355139  16.603660  14.205896
 16.474111  16.212237  15.072443  14.217581
 16.273899  14.905624  17.285019  14.955722  13.769731  18.308619
 15.601386  15.832661  14.342416  16.516657
 14.697575  15.719496  16.723135  16.808668  15.443325  14.608358
 17.031334  16.426377  13.900535  13.528603
 16.197697  16.839241  14.802707  15.507915  14.864337  15.836943
 15.660089  15.998911  13.956739  14.337318
 16.416974  17.729661  14.936045  13.450859  15.943900  15.106077
 15.541450  16.523752  16.555945  14.440305
 14.937772  16.486544  13.780310  16.944841  14.867400  18.214934
 14.142108  15.931952  14.424949  15.533156
 16.010153  16.323108  14.423508  15.970071  15.277186  15.561362
 14.978766  15.855935  16.953906  14.247016
Sat Jan 23 12:41:09 2016
 15.908483  15.638943  17.681281  15.188704  15.721495  13.359225
 15.999421  15.858876  16.402176  16.416312
 15.443946  14.675751  15.470643  15.573755  15.422241  16.336590
 17.220916  13.974890  15.877780  62.650921
 62.667990  46.334603  53.546195  69.465447  65.006016  68.761229
 70.754684  97.571669 104.811261 104.229302
105.605257 105.166030 105.058075 105.519703 106.573306 106.708545
106.114733 105.643131 106.049387 106.379378
104.239131 104.268931 103.852929 103.549319 103.516169 103.007015
103.724020 104.519983 105.839203 105.324985
104.328205 104.932713 103.051548 104.938652 102.769383 102.851609
101.432277 102.269842 100.937972 103.450103
103.477628 103.636130 103.444242 103.023145 102.565047 102.853115
101.402610  98.928230  99.310677  99.669667
101.140554  99.628664 102.093801 100.580659 101.762283 101.369349
102.637014 102.240950 101.778506 101.144526
100.899476 102.294952 102.029285 100.871166 102.763222 102.910690
104.892447 104.748194 105.403636 106.159345
106.413154 104.626632 105.775004 104.579775 104.778526 104.634778
106.233381 104.063642 106.635481 104.314503

if I look at the disk loads at the time, I see a dramatic increase in disk
reads that correspond to the slow writes so I'm guessing at least some
writes are waiting in the queue as you can see there - thanks to laurence
for the patch to show disk read wait times ;)

# DISK STATISTICS (/sec)
#
<---------reads---------------><---------writes--------------><--------averages-------->
Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
 Wait  RWSize  QLen  Wait SvcTim Util
12:45:30 sdb              0      0    0    0     0  270040    105 2276  119
    4     118    16     4      0   62
12:45:31 sdb              0      0    0    0     0  273776    120 2262  121
    4     121    18     4      0   57
12:45:32 sdb              4      0    1    4     0  100164     57  909  110
    4     110     6     4      0   84
12:45:33 sdb              0      0    0    0     0  229992     87 1924  120
    1     119     2     1      0   68
12:45:34 sdb              4      0    1    4     4  153528     59 1304  118
    0     117     1     0      0   78
12:45:35 sdb              0      0    0    0     0  220896     97 1895  117
    1     116     1     1      0   62
12:45:36 sdb              0      0    0    0     0  419084    197 3504  120
    0     119     1     0      0   32
12:45:37 sdb              0      0    0    0     0  428076    193 3662  117
    0     116     1     0      0   32
12:45:38 sdb              0      0    0    0     0  428492    181 3560  120
    0     120     1     0      0   30
12:45:39 sdb              0      0    0    0     0  426024    199 3641  117
    0     117     1     0      0   32
12:45:40 sdb              0      0    0    0     0  429764    200 3589  120
    0     119     1     0      0   28
12:45:41 sdb              0      0    0    0     0  410204    165 3430  120
    0     119     3     0      0   36
12:45:42 sdb              0      0    0    0     0  406192    196 3437  118
    0     118     5     0      0   39
12:45:43 sdb              0      0    0    0     0  420952    175 3552  119
    0     118     1     0      0   34
12:45:44 sdb              0      0    0    0     0  428424    197 3645  118
    0     117     1     0      0   31
12:45:45 sdb              0      0    0    0     0  192464     76 1599  120
    8     120    18     8      0   75
12:45:46 sdb              0      0    0    0     0  340522    205 2951  115
    2     115    16     2      0   41
12:45:47 sdb              0      0    0    0     0  429128    193 3664  117
    0     117     1     0      0   28
12:45:48 sdb              0      0    0    0     0  402600    164 3311  122
    0     121     3     0      0   39
12:45:49 sdb              0      0    0    0     0  435316    195 3701  118
    0     117     1     0      0   36
12:45:50 sdb              0      0    0    0     0  367976    162 3152  117
    1     116     7     1      0   46
12:45:51 sdb              0      0    0    0     0  255716    125 2153  119
    4     118    16     4      0   60

# DISK STATISTICS (/sec)
#
<---------reads---------------><---------writes--------------><--------averages-------->
Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
 Wait  RWSize  QLen  Wait SvcTim Util
12:45:52 sdb              0      0    0    0     0  360144    149 3006  120
    1     119     9     1      0   46
12:45:53 sdb              0      0    0    0     0  343500    162 2909  118
    1     118    11     1      0   43
12:45:54 sdb              0      0    0    0     0  256636    119 2188  117
    2     117    11     2      0   54
12:45:55 sdb              0      0    0    0     0  149000     47 1260  118
   14     118    22    14      0   79
12:45:56 sdb              0      0    0    0     0  198544     88 1654  120
    7     120    19     7      0   67
12:45:57 sdb              0      0    0    0     0  320688    151 2731  117
    1     117     8     1      0   53
12:45:58 sdb              0      0    0    0     0  422176    190 3532  120
    0     119     1     0      0   32
12:45:59 sdb              0      0    0    0     0  266540    115 2233  119
    5     119    13     5      0   93
12:46:00 sdb              8      0    2    4   690  291116    129 2463  118
    3     118     9     3      0   82
12:46:01 sdb              0      0    0    0     0  249964    118 2160  116
    4     115    15     4      0   60
12:46:02 sdb           4736      0   37  128     0  424680    167 3522  121
    0     120     1     0      0   28
12:46:03 sdb           5016      0   42  119     0  391364    196 3344  117
    0     117     6     0      0   34
12:46:04 sdb              0      0    0    0     0  415436    172 3501  119
    0     118     2     0      0   33
12:46:05 sdb              0      0    0    0     0  398736    192 3373  118
    0     118     3     0      0   39
12:46:06 sdb              0      0    0    0     0  367292    155 3015  122
    0     121     6     0      0   39
12:46:07 sdb              0      0    0    0     0  420392    201 3614  116
    0     116     1     0      0   30
12:46:08 sdb              0      0    0    0     0  424828    172 3547  120
    0     119     1     0      0   32
12:46:09 sdb              0      0    0    0     0  500380    234 4277  117
    0     116     2     0      0   34
12:46:10 sdb              0      0    0    0     0  104500      7  698  150
    0     149     1     0      1   87
12:46:11 sdb              8      0    1    8  1260   77252     45  647  119
    0     119     1     2      1   92
12:46:12 sdb              8      0    1    8  1244   73956     31  615  120
    0     120     1     2      1   94
12:46:13 sdb              8      0    1    8   228  149552     64 1256  119
    0     118     1     0      0   85

# DISK STATISTICS (/sec)
#
<---------reads---------------><---------writes--------------><--------averages-------->
Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
 Wait  RWSize  QLen  Wait SvcTim Util
12:46:14 sdb              8      0    1    8  1232   37124     28  319  116
    0     116     1     3      3   99
12:46:15 sdb             16      0    2    8   720    2776     23  120   23
    1      22     1    13      8   99
12:46:16 sdb              0      0    0    0     0  108180     16  823  131
    0     131     1     0      1   90
12:46:17 sdb              8      0    1    8  1260   37136     28  322  115
    0     114     1     3      2   94
12:46:18 sdb              8      0    1    8  1252  108680     57  875  124
    0     124     1     1      1   88
12:46:19 sdb              0      0    0    0     0       0      0    0    0
    0       0     1     0      0  100
12:46:20 sdb             16      0    2    8   618   81516     49  685  119
    0     118     1     1      1   94
12:46:21 sdb             16      0    2    8   640  225788    106 1907  118
    0     118     1     0      0   75
12:46:22 sdb             32      0    4    8    95   73892     17  627  118
    0     117     1     0      1   93
12:46:23 sdb             24      0    3    8   408  257012    119 2171  118
    0     118     1     0      0   65
12:46:24 sdb             12      0    3    4     5    3608      0   20  180
    0     157     1     0     43  100
12:46:25 sdb             44      0    7    6   210   74072     41  625  119
    0     117     1     2      1   97
12:46:26 sdb             48      0    6    8   216  202852    112 1819  112
    0     111     1     0      0   92
12:46:27 sdb             52      0    7    7   233  307156    137 2648  116
    0     115     1     0      0   95
12:46:28 sdb             16      0    2    8   100   93168      7  638  146
    0     145     1     0      1   97
12:46:29 sdb             16      0    2    8   642   37028     16  319  116
    0     115     1     4      3   99
12:46:30 sdb             16      0    2    8   624   39068     36  342  114
    0     113     1     3      2   99
12:46:31 sdb             80      0   10    8    94  253892    105 2169  117
    0     116     1     0      0   84
12:46:32 sdb              0      0    0    0     0    5676      0   33  172
    0     172     1     0     30  100
12:46:33 sdb             16      0    2    8   642   69236     28  583  119
    0     118     1     2      1   96
12:46:34 sdb              8      0    1    8  1032   37132     30  315  118
    0     117     1     3      3  100
12:46:35 sdb             16      0    2    8   822   56292     15  515  109
    0     108     1     3      1  100

# DISK STATISTICS (/sec)
#
<---------reads---------------><---------writes--------------><--------averages-------->
Pct
#Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size
 Wait  RWSize  QLen  Wait SvcTim Util
12:46:36 sdb              8      0    1    8    44   58768     15  452  130
    0     129     1     0      2   96
12:46:37 sdb             28      0    4    7   390  114944     89 1100  104
    0     104     1     1      0   88
12:46:38 sdb              0      0    0    0     0   29668      0  172  172
   12     172     1    12      5   98
12:46:39 sdb             80      0   10    8    90  100084     31  882  113
    0     112     1     1      1   91
12:46:40 sdb              0      0    0    0     0   24244      0  139  174
    0     174     1     0      7  100
12:46:41 sdb              8      0    1    8  1224       0      0    0    0
    0       8     1  1224   1000  100
12:46:42 sdb              8      0    1    8  1244   42368     29  354  120
    0     119     1     3      2   96
12:46:43 sdb             36      0    5    7   251   51428     32  507  101
    0     100     1     2      1   94
12:46:44 sdb             24      0    3    8    70    5732     31  147   39
   15      38     2    16      6   99
12:46:45 sdb             32      0    4    8     4  213056     53 1647  129
    0     129     1     0      0   74
12:46:46 sdb              8      0    1    8  1220   37416     28  328  114
    0     113     1     3      2   96
12:46:47 sdb              8      0    1    8  1248   58572     67  607   96
    0      96     1     2      1   93
12:46:48 sdb             40      0    5    8    84  274808     82 2173  126
    0     126     1     0      0   70
12:46:49 sdb              0      0    0    0     0       0      0    0    0
    0       0     1     0      0  100
12:46:50 sdb              8      0    1    8  1248       0      0    0    0
    0       8     1  1248   1000  100
12:46:51 sdb              8      0    1    8  1272       0      0    0    0
    0       8     1  1272   1000  100
12:46:52 sdb             24      0    3    8   414  205240    113 1798  114
    0     113     1     0      0   75
12:46:53 sdb              8      0    1    8   876   92476     48  839  110
    0     110     1     1      1   89
12:46:54 sdb              0      0    0    0     0   38700      0  225  172
    0     172     1     0      4   99
12:46:55 sdb             16      0    2    8   582  150680     73 1262  119
    0     119     1     1      0   87
12:46:56 sdb              8      0    1    8  1228       0      0    0    0
    0       8     1  1228   1000  100
12:46:57 sdb              8      0    1    8  1244       0      0    0    0
    0       8     1  1244   1000  100

next I played back the collectl process data and sorted by disk reads and
discovered the top process, corresponding to the long disk reads was
xfsaild.  btw - I also see the slab xfs_inode using about 60GB.

It's also worth noting that I'm only doing 1-2MB/sec of writes and the rest
of the data looks like it's coming from xfs journaling because when I look
at the xfs stats I'm seeing on the order of 200-400MB/sec xfs logging
writes - clearly they're not all going to disk.  Once the read waits
increase everything slows down including xfs logging (since it's doing
less).

I'm sure the simple answer may be that it is what it is, but I'm also
wondering without changes to swift itself, might there be some ways to
improve the situation by adding more memory or making any other tuning
changes?  The system I'm currently running my tests on has 128GB.

-mark

On Wed, Jan 6, 2016 at 6:49 PM, Dave Chinner  wrote:

> On Wed, Jan 06, 2016 at 05:46:33PM -0500, Mark Seger wrote:
> > dave, thanks for getting back to me and the pointer to the config doc.
> >  lots to absorb and play with.
> >
> > the real challenge for me is that I'm doing testing as different levels.
> > While i realize running 100 parallel swift PUT threads on a small system
> is
> > not the ideal way to do things, it's the only easy way to get massive
> > numbers of objects into the fillesystem and once there, the performance
> of
> > a single stream is pretty poor and by instrumenting the swift code I can
> > clearly see excess time being spent in creating/writing the objects and
> so
> > that's lead us to believe the problem lies in the way xfs is configured.
> >  creating a new directory structure on that same mount point immediately
> > results in high levels of performance.
> >
> > As an attempt to try to reproduce the problems w/o swift, I wrote a
> little
> > python script that simply creates files in a 2-tier structure, the first
> > tier consisting of 1024 directories and each directory contains 4096
> > subdirectories into which 1K files are created.
>
> So you created something with even greater fan-out than what your
> swift app is using?
>
> > I'm doing this for 10000
> > objects as a time and then timing them, reporting the times, 10 per line
> so
> > each line represents 100 thousand file creates.
> >
> > Here too I'm seeing degradation and if I look at what happens when there
> > are already 3M files and I write 1M more, I see these creation times/10
> > thousand:
> >
> >  1.004236  0.961419  0.996514  1.012150  1.101794  0.999422  0.994796
> >  1.214535  0.997276  1.306736
> >  2.793429  1.201471  1.133576  1.069682  1.030985  1.096341  1.052602
> >  1.391364  0.999480  1.914125
> >  1.193892  0.967206  1.263310  0.890472  1.051962  4.253694  1.145573
> >  1.528848 13.586892  4.925790
> >  3.975442  8.896552  1.197005  3.904226  7.503806  1.294842  1.816422
> >  9.329792  7.270323  5.936545
> >  7.058685  5.516841  4.527271  1.956592  1.382551  1.510339  1.318341
> > 13.255939  6.938845  4.106066
> >  2.612064  2.028795  4.647980  7.371628  5.473423  5.823201 14.229120
> >  0.899348  3.539658  8.501498
> >  4.662593  6.423530  7.980757  6.367012  3.414239  7.364857  4.143751
> >  6.317348 11.393067  1.273371
> > 146.067300  1.317814  1.176529  1.177830 52.206605  1.112854  2.087990
> > 42.328220  1.178436  1.335202
> > 49.118140  1.368696  1.515826 44.690431  0.927428  0.920801  0.985965
> >  1.000591  1.027458 60.650443
> >  1.771318  2.690499  2.262868  1.061343  0.932998 64.064210 37.726213
> >  1.245129  0.743771  0.996683
> >
> > nothing one set of 10K took almost 3 minutes!
>
> Which is no surprise because you have slow disks and a *lot* of
> memory. At some point the journal and/or memory is going to fill up
> with dirty objects and have to block waiting for writeback. At that
> point there's going to be several hundred thousand dirty inodes that
> need to be flushed to disk before progress can be made again.  That
> metadata writeback will be seek bound, and that's where all the
> delay comes from.
>
> We've been through this problem several times now with different
> swift users over the past couple of years. Please go and search the
> list archives, because every time the solution has been the same:
>
>         - reduce the directory heirarchy to a single level with, at
>           most, the number of directories matching the expected
>           *production* concurrency level
>         - reduce the XFS log size down to 32-128MB to limit dirty
>           metadata object buildup in memory
>         - reduce the number of AGs to as small as necessary to
>           maintain /allocation/ concurrency to limit the number of
>           different locations XFS writes to the disks (typically
>           10-20x less than the application level concurrency)
>         - use a 3.16+ kernel with the free inode btree on-disk
>           format feature to keep inode allocation CPU overhead low
>           and consistent regardless of the number of inodes already
>           allocated in the filesystem.
>
> > my main questions at this point are is this performance expected and/or
> > might a newer kernel help?  and might it be possible to significantly
> > improve things via tuning or is it what it is?  I do realize I'm starting
> > with an empty directory tree whose performance degrades as it fills, but
> if
> > I wanted to tune for say 10M or maybe 100M files might I be able to
> expect
>
> The mkfs defaults will work just fine with that many files in the
> filesystem. Your application configuration and data store layout is
> likely to be your biggest problem here.
>
> Cheers,
>
> Dave.
>
> --
> Dave Chinner
> david@fromorbit.com
>

--047d7b5d457e95b6bc052a2b31af
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: base64

PGRpdiBkaXI9Imx0ciI+c2luY2UgZ2V0dGluZyB5b3VyIGxhc3QgcmVwbHkgSSYjMzk7dmUgYmVl
biBkb2luZyBhIGxvdCBtb3JlIHRyeWluZyB0byB1bmRlcnN0YW5kIHRoZSBiZWhhdmlvciBvZiB3
aGF0IEkmIzM5O20gc2VlaW5nIGJ5IHdyaXRpbmcgc29tZSBub24tc3dpZnQgY29kZSB0aGF0IHNv
cnQgb2YgZG9lcyB3aGF0IHN3aWZ0IGRvZXMgd2l0aCByZXNwZWN0IHRvIGEgZGlyZWN0b3J5IHN0
cnVjdHVyZS4gwqBpbiBteSBjYXNlIEkgaGF2ZSAxMDI0IHRvcCBsZXZlbCBkaXJzLCA0MDk2IHVu
ZGVyIGVhY2guIMKgZWFjaCAxayBmaWxlIEkmIzM5O20gY3JlYXRpbmcgZ2V0cyBpdCYjMzk7cyBv
bmx5IGRpcmVjdG9yeSB1bmRlciB0aGVzZSBzbyB0aGVyZSBhcmUgY2xlYXJseSBhIGxvdCBvZiBk
aXJlY3Rvcmllcy48ZGl2Pjxicj48L2Rpdj48ZGl2PnhmcyB3cml0ZXMgb3V0IGFib3V0IDI1TSBv
YmplY3RzIGFuZCB0aGVuIHRoZSBwZXJmb3JtYW5jZSBnb2VzIGludG8gdGhlIHRvaWxldC7CoCBJ
JiMzOTttIHN1cmUgd2hhdCB5b3Ugc2FpZCBiZWZvcmUgYWJvdXQgaGF2aW5nIHRvIGZsdXNoIGRh
dGEgYW5kIGNhdXNpbmcgYmlnIGRlbGF5cywgYnV0IHdvdWxkIGl0IGJlIGNvbnRpbnVvdXM/IMKg
ZWFjaCBlbnRyeSBpbiB0aGUgZm9sbG93aW5nIHRhYmxlIHNob3dzIHRoZSB0aW1lIHRvIHdyaXRl
IDEwSyBmaWxlcyBzbyB0aGUgMiBibG9ja3MgYXJlIDFNIGVhY2g8L2Rpdj48ZGl2Pjxicj48L2Rp
dj48ZGl2PjxkaXY+U2F0IEphbiAyMyAxMjoxNTowOSAyMDE2PC9kaXY+PGRpdj7CoDE2LjExNDM4
NiDCoDE0LjY1NjczNiDCoDE0Ljc4OTc2MCDCoDE3LjQxODM4OSDCoDE0LjYxMzE1NyDCoDE1Ljkz
ODE3NiDCoDE0Ljg2NTM2OSDCoDE0Ljk2MjA1OCDCoDE3LjI5NzE5MyDCoDE1Ljk1MzU5MDwvZGl2
PjxkaXY+wqAxNC44OTU0NzEgwqAxNS41NjAyNTIgwqAxNC43ODk5MzcgwqAxNC4zMDg2MTggwqAx
Ni4zOTAwNTcgwqAxNi41NjE3ODkgwqAxNS43MTM4MDYgwqAxNC44NDM3OTEgwqAxNS45NDA5OTIg
wqAxNi40NjY5MjQ8L2Rpdj48ZGl2PsKgMTUuODQyNzgxIMKgMTUuNjExMjMwIMKgMTcuMTAyMzI5
IMKgMTUuMDA2MjkxIMKgMTQuNDU0MDg4IMKgMTcuOTIzNjYyIMKgMTMuMzc4MzQwIMKgMTYuMDg0
NjY0IMKgMTUuOTk2Nzk0IMKgMTMuNzM2Mzk4PC9kaXY+PGRpdj7CoDE4LjEyNTEyNSDCoDE0LjQ2
MjA2MyDCoDE4LjEwMTgzMyDCoDE1LjM1NTEzOSDCoDE2LjYwMzY2MCDCoDE0LjIwNTg5NiDCoDE2
LjQ3NDExMSDCoDE2LjIxMjIzNyDCoDE1LjA3MjQ0MyDCoDE0LjIxNzU4MTwvZGl2PjxkaXY+wqAx
Ni4yNzM4OTkgwqAxNC45MDU2MjQgwqAxNy4yODUwMTkgwqAxNC45NTU3MjIgwqAxMy43Njk3MzEg
wqAxOC4zMDg2MTkgwqAxNS42MDEzODYgwqAxNS44MzI2NjEgwqAxNC4zNDI0MTYgwqAxNi41MTY2
NTc8L2Rpdj48ZGl2PsKgMTQuNjk3NTc1IMKgMTUuNzE5NDk2IMKgMTYuNzIzMTM1IMKgMTYuODA4
NjY4IMKgMTUuNDQzMzI1IMKgMTQuNjA4MzU4IMKgMTcuMDMxMzM0IMKgMTYuNDI2Mzc3IMKgMTMu
OTAwNTM1IMKgMTMuNTI4NjAzPC9kaXY+PGRpdj7CoDE2LjE5NzY5NyDCoDE2LjgzOTI0MSDCoDE0
LjgwMjcwNyDCoDE1LjUwNzkxNSDCoDE0Ljg2NDMzNyDCoDE1LjgzNjk0MyDCoDE1LjY2MDA4OSDC
oDE1Ljk5ODkxMSDCoDEzLjk1NjczOSDCoDE0LjMzNzMxODwvZGl2PjxkaXY+wqAxNi40MTY5NzQg
wqAxNy43Mjk2NjEgwqAxNC45MzYwNDUgwqAxMy40NTA4NTkgwqAxNS45NDM5MDAgwqAxNS4xMDYw
NzcgwqAxNS41NDE0NTAgwqAxNi41MjM3NTIgwqAxNi41NTU5NDUgwqAxNC40NDAzMDU8L2Rpdj48
ZGl2PsKgMTQuOTM3NzcyIMKgMTYuNDg2NTQ0IMKgMTMuNzgwMzEwIMKgMTYuOTQ0ODQxIMKgMTQu
ODY3NDAwIMKgMTguMjE0OTM0IMKgMTQuMTQyMTA4IMKgMTUuOTMxOTUyIMKgMTQuNDI0OTQ5IMKg
MTUuNTMzMTU2PC9kaXY+PGRpdj7CoDE2LjAxMDE1MyDCoDE2LjMyMzEwOCDCoDE0LjQyMzUwOCDC
oDE1Ljk3MDA3MSDCoDE1LjI3NzE4NiDCoDE1LjU2MTM2MiDCoDE0Ljk3ODc2NiDCoDE1Ljg1NTkz
NSDCoDE2Ljk1MzkwNiDCoDE0LjI0NzAxNjwvZGl2PjxkaXY+U2F0IEphbiAyMyAxMjo0MTowOSAy
MDE2PC9kaXY+PGRpdj7CoDE1LjkwODQ4MyDCoDE1LjYzODk0MyDCoDE3LjY4MTI4MSDCoDE1LjE4
ODcwNCDCoDE1LjcyMTQ5NSDCoDEzLjM1OTIyNSDCoDE1Ljk5OTQyMSDCoDE1Ljg1ODg3NiDCoDE2
LjQwMjE3NiDCoDE2LjQxNjMxMjwvZGl2PjxkaXY+wqAxNS40NDM5NDYgwqAxNC42NzU3NTEgwqAx
NS40NzA2NDMgwqAxNS41NzM3NTUgwqAxNS40MjIyNDEgwqAxNi4zMzY1OTAgwqAxNy4yMjA5MTYg
wqAxMy45NzQ4OTAgwqAxNS44Nzc3ODAgwqA2Mi42NTA5MjE8L2Rpdj48ZGl2PsKgNjIuNjY3OTkw
IMKgNDYuMzM0NjAzIMKgNTMuNTQ2MTk1IMKgNjkuNDY1NDQ3IMKgNjUuMDA2MDE2IMKgNjguNzYx
MjI5IMKgNzAuNzU0Njg0IMKgOTcuNTcxNjY5IDEwNC44MTEyNjEgMTA0LjIyOTMwMjwvZGl2Pjxk
aXY+MTA1LjYwNTI1NyAxMDUuMTY2MDMwIDEwNS4wNTgwNzUgMTA1LjUxOTcwMyAxMDYuNTczMzA2
IDEwNi43MDg1NDUgMTA2LjExNDczMyAxMDUuNjQzMTMxIDEwNi4wNDkzODcgMTA2LjM3OTM3ODwv
ZGl2PjxkaXY+MTA0LjIzOTEzMSAxMDQuMjY4OTMxIDEwMy44NTI5MjkgMTAzLjU0OTMxOSAxMDMu
NTE2MTY5IDEwMy4wMDcwMTUgMTAzLjcyNDAyMCAxMDQuNTE5OTgzIDEwNS44MzkyMDMgMTA1LjMy
NDk4NTwvZGl2PjxkaXY+MTA0LjMyODIwNSAxMDQuOTMyNzEzIDEwMy4wNTE1NDggMTA0LjkzODY1
MiAxMDIuNzY5MzgzIDEwMi44NTE2MDkgMTAxLjQzMjI3NyAxMDIuMjY5ODQyIDEwMC45Mzc5NzIg
MTAzLjQ1MDEwMzwvZGl2PjxkaXY+MTAzLjQ3NzYyOCAxMDMuNjM2MTMwIDEwMy40NDQyNDIgMTAz
LjAyMzE0NSAxMDIuNTY1MDQ3IDEwMi44NTMxMTUgMTAxLjQwMjYxMCDCoDk4LjkyODIzMCDCoDk5
LjMxMDY3NyDCoDk5LjY2OTY2NzwvZGl2PjxkaXY+MTAxLjE0MDU1NCDCoDk5LjYyODY2NCAxMDIu
MDkzODAxIDEwMC41ODA2NTkgMTAxLjc2MjI4MyAxMDEuMzY5MzQ5IDEwMi42MzcwMTQgMTAyLjI0
MDk1MCAxMDEuNzc4NTA2IDEwMS4xNDQ1MjY8L2Rpdj48ZGl2PjEwMC44OTk0NzYgMTAyLjI5NDk1
MiAxMDIuMDI5Mjg1IDEwMC44NzExNjYgMTAyLjc2MzIyMiAxMDIuOTEwNjkwIDEwNC44OTI0NDcg
MTA0Ljc0ODE5NCAxMDUuNDAzNjM2IDEwNi4xNTkzNDU8L2Rpdj48L2Rpdj48ZGl2PjxkaXY+MTA2
LjQxMzE1NCAxMDQuNjI2NjMyIDEwNS43NzUwMDQgMTA0LjU3OTc3NSAxMDQuNzc4NTI2IDEwNC42
MzQ3NzggMTA2LjIzMzM4MSAxMDQuMDYzNjQyIDEwNi42MzU0ODEgMTA0LjMxNDUwMzwvZGl2Pjwv
ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+aWYgSSBsb29rIGF0IHRoZSBkaXNrIGxvYWRzIGF0IHRo
ZSB0aW1lLCBJIHNlZSBhIGRyYW1hdGljIGluY3JlYXNlIGluIGRpc2sgcmVhZHMgdGhhdCBjb3Jy
ZXNwb25kIHRvIHRoZSBzbG93IHdyaXRlcyBzbyBJJiMzOTttIGd1ZXNzaW5nIGF0IGxlYXN0IHNv
bWUgd3JpdGVzIGFyZSB3YWl0aW5nIGluIHRoZSBxdWV1ZSBhcyB5b3UgY2FuIHNlZSB0aGVyZSAt
IHRoYW5rcyB0byBsYXVyZW5jZSBmb3IgdGhlIHBhdGNoIHRvIHNob3cgZGlzayByZWFkIHdhaXQg
dGltZXMgOyk8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3Nw
YWNlLCBtb25vc3BhY2UiPiMgRElTSyBTVEFUSVNUSUNTICgvc2VjKTwvZm9udD48L2Rpdj48ZGl2
Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4jIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgICZsdDstLS0tLS0tLS1yZWFkcy0tLS0tLS0tLS0tLS0tLSZndDsmbHQ7LS0tLS0tLS0t
d3JpdGVzLS0tLS0tLS0tLS0tLS0mZ3Q7Jmx0Oy0tLS0tLS0tYXZlcmFnZXMtLS0tLS0tLSZndDsg
UGN0PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPiNU
aW1lIMKgIMKgIE5hbWUgwqAgwqAgwqAgS0J5dGVzIE1lcmdlZCDCoElPcyBTaXplIMKgV2FpdCDC
oEtCeXRlcyBNZXJnZWQgwqBJT3MgU2l6ZSDCoFdhaXQgwqBSV1NpemUgwqBRTGVuIMKgV2FpdCBT
dmNUaW0gVXRpbDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3Nw
YWNlIj4xMjo0NTozMCBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAg
wqAgwqAwIMKgIMKgIDAgwqAyNzAwNDAgwqAgwqAxMDUgMjI3NiDCoDExOSDCoCDCoCA0IMKgIMKg
IDExOCDCoCDCoDE2IMKgIMKgIDQgwqAgwqAgwqAwIMKgIDYyPC9mb250PjwvZGl2PjxkaXY+PGZv
bnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ1OjMxIHNkYiDCoCDCoCDCoCDCoCDC
oCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDI3Mzc3NiDCoCDCoDEy
MCAyMjYyIMKgMTIxIMKgIMKgIDQgwqAgwqAgMTIxIMKgIMKgMTggwqAgwqAgNCDCoCDCoCDCoDAg
wqAgNTc8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+
MTI6NDU6MzIgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgNCDCoCDCoCDCoDAgwqAgwqAxIMKgIMKg
NCDCoCDCoCAwIMKgMTAwMTY0IMKgIMKgIDU3IMKgOTA5IMKgMTEwIMKgIMKgIDQgwqAgwqAgMTEw
IMKgIMKgIDYgwqAgwqAgNCDCoCDCoCDCoDAgwqAgODQ8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBm
YWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDU6MzMgc2RiIMKgIMKgIMKgIMKgIMKgIMKg
IMKgMCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgMjI5OTkyIMKgIMKgIDg3IDE5
MjQgwqAxMjAgwqAgwqAgMSDCoCDCoCAxMTkgwqAgwqAgMiDCoCDCoCAxIMKgIMKgIMKgMCDCoCA2
ODwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0
NTozNCBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqA0IMKgIMKgIMKgMCDCoCDCoDEgwqAgwqA0IMKg
IMKgIDQgwqAxNTM1MjggwqAgwqAgNTkgMTMwNCDCoDExOCDCoCDCoCAwIMKgIMKgIDExNyDCoCDC
oCAxIMKgIMKgIDAgwqAgwqAgwqAwIMKgIDc4PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0i
bW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ1OjM1IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAg
wqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDIyMDg5NiDCoCDCoCA5NyAxODk1IMKg
MTE3IMKgIMKgIDEgwqAgwqAgMTE2IMKgIMKgIDEgwqAgwqAgMSDCoCDCoCDCoDAgwqAgNjI8L2Zv
bnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDU6MzYg
c2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAw
IMKgNDE5MDg0IMKgIMKgMTk3IDM1MDQgwqAxMjAgwqAgwqAgMCDCoCDCoCAxMTkgwqAgwqAgMSDC
oCDCoCAwIMKgIMKgIMKgMCDCoCAzMjwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9z
cGFjZSwgbW9ub3NwYWNlIj4xMjo0NTozNyBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKg
IMKgMCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqA0MjgwNzYgwqAgwqAxOTMgMzY2MiDCoDExNyDC
oCDCoCAwIMKgIMKgIDExNiDCoCDCoCAxIMKgIMKgIDAgwqAgwqAgwqAwIMKgIDMyPC9mb250Pjwv
ZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ1OjM4IHNkYiDC
oCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDQy
ODQ5MiDCoCDCoDE4MSAzNTYwIMKgMTIwIMKgIMKgIDAgwqAgwqAgMTIwIMKgIMKgIDEgwqAgwqAg
MCDCoCDCoCDCoDAgwqAgMzA8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2Us
IG1vbm9zcGFjZSI+MTI6NDU6Mzkgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCDCoCDCoCDCoDAg
wqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgNDI2MDI0IMKgIMKgMTk5IDM2NDEgwqAxMTcgwqAgwqAg
MCDCoCDCoCAxMTcgwqAgwqAgMSDCoCDCoCAwIMKgIMKgIMKgMCDCoCAzMjwvZm9udD48L2Rpdj48
ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NTo0MCBzZGIgwqAgwqAg
wqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqA0Mjk3NjQg
wqAgwqAyMDAgMzU4OSDCoDEyMCDCoCDCoCAwIMKgIMKgIDExOSDCoCDCoCAxIMKgIMKgIDAgwqAg
wqAgwqAwIMKgIDI4PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25v
c3BhY2UiPjEyOjQ1OjQxIHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKg
MCDCoCDCoDAgwqAgwqAgMCDCoDQxMDIwNCDCoCDCoDE2NSAzNDMwIMKgMTIwIMKgIMKgIDAgwqAg
wqAgMTE5IMKgIMKgIDMgwqAgwqAgMCDCoCDCoCDCoDAgwqAgMzY8L2ZvbnQ+PC9kaXY+PGRpdj48
Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDU6NDIgc2RiIMKgIMKgIMKgIMKg
IMKgIMKgIMKgMCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgNDA2MTkyIMKgIMKg
MTk2IDM0MzcgwqAxMTggwqAgwqAgMCDCoCDCoCAxMTggwqAgwqAgNSDCoCDCoCAwIMKgIMKgIMKg
MCDCoCAzOTwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNl
Ij4xMjo0NTo0MyBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAgwqAg
wqAwIMKgIMKgIDAgwqA0MjA5NTIgwqAgwqAxNzUgMzU1MiDCoDExOSDCoCDCoCAwIMKgIMKgIDEx
OCDCoCDCoCAxIMKgIMKgIDAgwqAgwqAgwqAwIMKgIDM0PC9mb250PjwvZGl2PjxkaXY+PGZvbnQg
ZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ1OjQ0IHNkYiDCoCDCoCDCoCDCoCDCoCDC
oCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDQyODQyNCDCoCDCoDE5NyAz
NjQ1IMKgMTE4IMKgIMKgIDAgwqAgwqAgMTE3IMKgIMKgIDEgwqAgwqAgMCDCoCDCoCDCoDAgwqAg
MzE8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6
NDU6NDUgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDC
oCDCoCAwIMKgMTkyNDY0IMKgIMKgIDc2IDE1OTkgwqAxMjAgwqAgwqAgOCDCoCDCoCAxMjAgwqAg
wqAxOCDCoCDCoCA4IMKgIMKgIMKgMCDCoCA3NTwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9
Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NTo0NiBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAw
IMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqAzNDA1MjIgwqAgwqAyMDUgMjk1MSDC
oDExNSDCoCDCoCAyIMKgIMKgIDExNSDCoCDCoDE2IMKgIMKgIDIgwqAgwqAgwqAwIMKgIDQxPC9m
b250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ1OjQ3
IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAg
MCDCoDQyOTEyOCDCoCDCoDE5MyAzNjY0IMKgMTE3IMKgIMKgIDAgwqAgwqAgMTE3IMKgIMKgIDEg
wqAgwqAgMCDCoCDCoCDCoDAgwqAgMjg8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25v
c3BhY2UsIG1vbm9zcGFjZSI+MTI6NDU6NDggc2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCDCoCDC
oCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgNDAyNjAwIMKgIMKgMTY0IDMzMTEgwqAxMjIg
wqAgwqAgMCDCoCDCoCAxMjEgwqAgwqAgMyDCoCDCoCAwIMKgIMKgIMKgMCDCoCAzOTwvZm9udD48
L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NTo0OSBzZGIg
wqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqA0
MzUzMTYgwqAgwqAxOTUgMzcwMSDCoDExOCDCoCDCoCAwIMKgIMKgIDExNyDCoCDCoCAxIMKgIMKg
IDAgwqAgwqAgwqAwIMKgIDM2PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNl
LCBtb25vc3BhY2UiPjEyOjQ1OjUwIHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAw
IMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDM2Nzk3NiDCoCDCoDE2MiAzMTUyIMKgMTE3IMKgIMKg
IDEgwqAgwqAgMTE2IMKgIMKgIDcgwqAgwqAgMSDCoCDCoCDCoDAgwqAgNDY8L2ZvbnQ+PC9kaXY+
PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDU6NTEgc2RiIMKgIMKg
IMKgIMKgIMKgIMKgIMKgMCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgMjU1NzE2
IMKgIMKgMTI1IDIxNTMgwqAxMTkgwqAgwqAgNCDCoCDCoCAxMTggwqAgwqAxNiDCoCDCoCA0IMKg
IMKgIMKgMCDCoCA2MDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9u
b3NwYWNlIj48YnI+PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25v
c3BhY2UiPiMgRElTSyBTVEFUSVNUSUNTICgvc2VjKTwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZh
Y2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4jIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgICZs
dDstLS0tLS0tLS1yZWFkcy0tLS0tLS0tLS0tLS0tLSZndDsmbHQ7LS0tLS0tLS0td3JpdGVzLS0t
LS0tLS0tLS0tLS0mZ3Q7Jmx0Oy0tLS0tLS0tYXZlcmFnZXMtLS0tLS0tLSZndDsgUGN0PC9mb250
PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPiNUaW1lIMKgIMKg
IE5hbWUgwqAgwqAgwqAgS0J5dGVzIE1lcmdlZCDCoElPcyBTaXplIMKgV2FpdCDCoEtCeXRlcyBN
ZXJnZWQgwqBJT3MgU2l6ZSDCoFdhaXQgwqBSV1NpemUgwqBRTGVuIMKgV2FpdCBTdmNUaW0gVXRp
bDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0
NTo1MiBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAwIMKg
IMKgIDAgwqAzNjAxNDQgwqAgwqAxNDkgMzAwNiDCoDEyMCDCoCDCoCAxIMKgIMKgIDExOSDCoCDC
oCA5IMKgIMKgIDEgwqAgwqAgwqAwIMKgIDQ2PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0i
bW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ1OjUzIHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAg
wqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDM0MzUwMCDCoCDCoDE2MiAyOTA5IMKg
MTE4IMKgIMKgIDEgwqAgwqAgMTE4IMKgIMKgMTEgwqAgwqAgMSDCoCDCoCDCoDAgwqAgNDM8L2Zv
bnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDU6NTQg
c2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAw
IMKgMjU2NjM2IMKgIMKgMTE5IDIxODggwqAxMTcgwqAgwqAgMiDCoCDCoCAxMTcgwqAgwqAxMSDC
oCDCoCAyIMKgIMKgIMKgMCDCoCA1NDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9z
cGFjZSwgbW9ub3NwYWNlIj4xMjo0NTo1NSBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKg
IMKgMCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqAxNDkwMDAgwqAgwqAgNDcgMTI2MCDCoDExOCDC
oCDCoDE0IMKgIMKgIDExOCDCoCDCoDIyIMKgIMKgMTQgwqAgwqAgwqAwIMKgIDc5PC9mb250Pjwv
ZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ1OjU2IHNkYiDC
oCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDE5
ODU0NCDCoCDCoCA4OCAxNjU0IMKgMTIwIMKgIMKgIDcgwqAgwqAgMTIwIMKgIMKgMTkgwqAgwqAg
NyDCoCDCoCDCoDAgwqAgNjc8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2Us
IG1vbm9zcGFjZSI+MTI6NDU6NTcgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCDCoCDCoCDCoDAg
wqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgMzIwNjg4IMKgIMKgMTUxIDI3MzEgwqAxMTcgwqAgwqAg
MSDCoCDCoCAxMTcgwqAgwqAgOCDCoCDCoCAxIMKgIMKgIMKgMCDCoCA1MzwvZm9udD48L2Rpdj48
ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NTo1OCBzZGIgwqAgwqAg
wqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqA0MjIxNzYg
wqAgwqAxOTAgMzUzMiDCoDEyMCDCoCDCoCAwIMKgIMKgIDExOSDCoCDCoCAxIMKgIMKgIDAgwqAg
wqAgwqAwIMKgIDMyPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25v
c3BhY2UiPjEyOjQ1OjU5IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKg
MCDCoCDCoDAgwqAgwqAgMCDCoDI2NjU0MCDCoCDCoDExNSAyMjMzIMKgMTE5IMKgIMKgIDUgwqAg
wqAgMTE5IMKgIMKgMTMgwqAgwqAgNSDCoCDCoCDCoDAgwqAgOTM8L2ZvbnQ+PC9kaXY+PGRpdj48
Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6MDAgc2RiIMKgIMKgIMKgIMKg
IMKgIMKgIMKgOCDCoCDCoCDCoDAgwqAgwqAyIMKgIMKgNCDCoCA2OTAgwqAyOTExMTYgwqAgwqAx
MjkgMjQ2MyDCoDExOCDCoCDCoCAzIMKgIMKgIDExOCDCoCDCoCA5IMKgIMKgIDMgwqAgwqAgwqAw
IMKgIDgyPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2Ui
PjEyOjQ2OjAxIHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDC
oDAgwqAgwqAgMCDCoDI0OTk2NCDCoCDCoDExOCAyMTYwIMKgMTE2IMKgIMKgIDQgwqAgwqAgMTE1
IMKgIMKgMTUgwqAgwqAgNCDCoCDCoCDCoDAgwqAgNjA8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBm
YWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6MDIgc2RiIMKgIMKgIMKgIMKgIMKgIDQ3
MzYgwqAgwqAgwqAwIMKgIDM3IMKgMTI4IMKgIMKgIDAgwqA0MjQ2ODAgwqAgwqAxNjcgMzUyMiDC
oDEyMSDCoCDCoCAwIMKgIMKgIDEyMCDCoCDCoCAxIMKgIMKgIDAgwqAgwqAgwqAwIMKgIDI4PC9m
b250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjAz
IHNkYiDCoCDCoCDCoCDCoCDCoCA1MDE2IMKgIMKgIMKgMCDCoCA0MiDCoDExOSDCoCDCoCAwIMKg
MzkxMzY0IMKgIMKgMTk2IDMzNDQgwqAxMTcgwqAgwqAgMCDCoCDCoCAxMTcgwqAgwqAgNiDCoCDC
oCAwIMKgIMKgIMKgMCDCoCAzNDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFj
ZSwgbW9ub3NwYWNlIj4xMjo0NjowNCBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKg
MCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqA0MTU0MzYgwqAgwqAxNzIgMzUwMSDCoDExOSDCoCDC
oCAwIMKgIMKgIDExOCDCoCDCoCAyIMKgIMKgIDAgwqAgwqAgwqAwIMKgIDMzPC9mb250PjwvZGl2
PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjA1IHNkYiDCoCDC
oCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDM5ODcz
NiDCoCDCoDE5MiAzMzczIMKgMTE4IMKgIMKgIDAgwqAgwqAgMTE4IMKgIMKgIDMgwqAgwqAgMCDC
oCDCoCDCoDAgwqAgMzk8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1v
bm9zcGFjZSI+MTI6NDY6MDYgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCDCoCDCoCDCoDAgwqAg
wqAwIMKgIMKgMCDCoCDCoCAwIMKgMzY3MjkyIMKgIMKgMTU1IDMwMTUgwqAxMjIgwqAgwqAgMCDC
oCDCoCAxMjEgwqAgwqAgNiDCoCDCoCAwIMKgIMKgIMKgMCDCoCAzOTwvZm9udD48L2Rpdj48ZGl2
Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NjowNyBzZGIgwqAgwqAgwqAg
wqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqA0MjAzOTIgwqAg
wqAyMDEgMzYxNCDCoDExNiDCoCDCoCAwIMKgIMKgIDExNiDCoCDCoCAxIMKgIMKgIDAgwqAgwqAg
wqAwIMKgIDMwPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3Bh
Y2UiPjEyOjQ2OjA4IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDC
oCDCoDAgwqAgwqAgMCDCoDQyNDgyOCDCoCDCoDE3MiAzNTQ3IMKgMTIwIMKgIMKgIDAgwqAgwqAg
MTE5IMKgIMKgIDEgwqAgwqAgMCDCoCDCoCDCoDAgwqAgMzI8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9u
dCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6MDkgc2RiIMKgIMKgIMKgIMKgIMKg
IMKgIMKgMCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgNTAwMzgwIMKgIMKgMjM0
IDQyNzcgwqAxMTcgwqAgwqAgMCDCoCDCoCAxMTYgwqAgwqAgMiDCoCDCoCAwIMKgIMKgIMKgMCDC
oCAzNDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4x
Mjo0NjoxMCBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAw
IMKgIMKgIDAgwqAxMDQ1MDAgwqAgwqAgwqA3IMKgNjk4IMKgMTUwIMKgIMKgIDAgwqAgwqAgMTQ5
IMKgIMKgIDEgwqAgwqAgMCDCoCDCoCDCoDEgwqAgODc8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBm
YWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6MTEgc2RiIMKgIMKgIMKgIMKgIMKgIMKg
IMKgOCDCoCDCoCDCoDAgwqAgwqAxIMKgIMKgOCDCoDEyNjAgwqAgNzcyNTIgwqAgwqAgNDUgwqA2
NDcgwqAxMTkgwqAgwqAgMCDCoCDCoCAxMTkgwqAgwqAgMSDCoCDCoCAyIMKgIMKgIMKgMSDCoCA5
MjwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0
NjoxMiBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqA4IMKgIMKgIMKgMCDCoCDCoDEgwqAgwqA4IMKg
MTI0NCDCoCA3Mzk1NiDCoCDCoCAzMSDCoDYxNSDCoDEyMCDCoCDCoCAwIMKgIMKgIDEyMCDCoCDC
oCAxIMKgIMKgIDIgwqAgwqAgwqAxIMKgIDk0PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0i
bW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjEzIHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDgg
wqAgwqAgwqAwIMKgIMKgMSDCoCDCoDggwqAgMjI4IMKgMTQ5NTUyIMKgIMKgIDY0IDEyNTYgwqAx
MTkgwqAgwqAgMCDCoCDCoCAxMTggwqAgwqAgMSDCoCDCoCAwIMKgIMKgIMKgMCDCoCA4NTwvZm9u
dD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj48YnI+PC9mb250
PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPiMgRElTSyBTVEFU
SVNUSUNTICgvc2VjKTwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9u
b3NwYWNlIj4jIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgICZsdDstLS0tLS0tLS1yZWFkcy0t
LS0tLS0tLS0tLS0tLSZndDsmbHQ7LS0tLS0tLS0td3JpdGVzLS0tLS0tLS0tLS0tLS0mZ3Q7Jmx0
Oy0tLS0tLS0tYXZlcmFnZXMtLS0tLS0tLSZndDsgUGN0PC9mb250PjwvZGl2PjxkaXY+PGZvbnQg
ZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPiNUaW1lIMKgIMKgIE5hbWUgwqAgwqAgwqAgS0J5
dGVzIE1lcmdlZCDCoElPcyBTaXplIMKgV2FpdCDCoEtCeXRlcyBNZXJnZWQgwqBJT3MgU2l6ZSDC
oFdhaXQgwqBSV1NpemUgwqBRTGVuIMKgV2FpdCBTdmNUaW0gVXRpbDwvZm9udD48L2Rpdj48ZGl2
Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NjoxNCBzZGIgwqAgwqAgwqAg
wqAgwqAgwqAgwqA4IMKgIMKgIMKgMCDCoCDCoDEgwqAgwqA4IMKgMTIzMiDCoCAzNzEyNCDCoCDC
oCAyOCDCoDMxOSDCoDExNiDCoCDCoCAwIMKgIMKgIDExNiDCoCDCoCAxIMKgIMKgIDMgwqAgwqAg
wqAzIMKgIDk5PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3Bh
Y2UiPjEyOjQ2OjE1IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCAxNiDCoCDCoCDCoDAgwqAgwqAyIMKg
IMKgOCDCoCA3MjAgwqAgwqAyNzc2IMKgIMKgIDIzIMKgMTIwIMKgIDIzIMKgIMKgIDEgwqAgwqAg
wqAyMiDCoCDCoCAxIMKgIMKgMTMgwqAgwqAgwqA4IMKgIDk5PC9mb250PjwvZGl2PjxkaXY+PGZv
bnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjE2IHNkYiDCoCDCoCDCoCDCoCDC
oCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoDEwODE4MCDCoCDCoCAx
NiDCoDgyMyDCoDEzMSDCoCDCoCAwIMKgIMKgIDEzMSDCoCDCoCAxIMKgIMKgIDAgwqAgwqAgwqAx
IMKgIDkwPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2Ui
PjEyOjQ2OjE3IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDggwqAgwqAgwqAwIMKgIMKgMSDCoCDC
oDggwqAxMjYwIMKgIDM3MTM2IMKgIMKgIDI4IMKgMzIyIMKgMTE1IMKgIMKgIDAgwqAgwqAgMTE0
IMKgIMKgIDEgwqAgwqAgMyDCoCDCoCDCoDIgwqAgOTQ8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBm
YWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6MTggc2RiIMKgIMKgIMKgIMKgIMKgIMKg
IMKgOCDCoCDCoCDCoDAgwqAgwqAxIMKgIMKgOCDCoDEyNTIgwqAxMDg2ODAgwqAgwqAgNTcgwqA4
NzUgwqAxMjQgwqAgwqAgMCDCoCDCoCAxMjQgwqAgwqAgMSDCoCDCoCAxIMKgIMKgIMKgMSDCoCA4
ODwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0
NjoxOSBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAwIMKg
IMKgIDAgwqAgwqAgwqAgMCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgIMKgIMKg
IDAgwqAgwqAgMSDCoCDCoCAwIMKgIMKgIMKgMCDCoDEwMDwvZm9udD48L2Rpdj48ZGl2Pjxmb250
IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NjoyMCBzZGIgwqAgwqAgwqAgwqAgwqAg
wqAgMTYgwqAgwqAgwqAwIMKgIMKgMiDCoCDCoDggwqAgNjE4IMKgIDgxNTE2IMKgIMKgIDQ5IMKg
Njg1IMKgMTE5IMKgIMKgIDAgwqAgwqAgMTE4IMKgIMKgIDEgwqAgwqAgMSDCoCDCoCDCoDEgwqAg
OTQ8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6
NDY6MjEgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIDE2IMKgIMKgIMKgMCDCoCDCoDIgwqAgwqA4IMKg
IDY0MCDCoDIyNTc4OCDCoCDCoDEwNiAxOTA3IMKgMTE4IMKgIMKgIDAgwqAgwqAgMTE4IMKgIMKg
IDEgwqAgwqAgMCDCoCDCoCDCoDAgwqAgNzU8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJt
b25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6MjIgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIDMyIMKg
IMKgIMKgMCDCoCDCoDQgwqAgwqA4IMKgIMKgOTUgwqAgNzM4OTIgwqAgwqAgMTcgwqA2MjcgwqAx
MTggwqAgwqAgMCDCoCDCoCAxMTcgwqAgwqAgMSDCoCDCoCAwIMKgIMKgIMKgMSDCoCA5MzwvZm9u
dD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NjoyMyBz
ZGIgwqAgwqAgwqAgwqAgwqAgwqAgMjQgwqAgwqAgwqAwIMKgIMKgMyDCoCDCoDggwqAgNDA4IMKg
MjU3MDEyIMKgIMKgMTE5IDIxNzEgwqAxMTggwqAgwqAgMCDCoCDCoCAxMTggwqAgwqAgMSDCoCDC
oCAwIMKgIMKgIMKgMCDCoCA2NTwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFj
ZSwgbW9ub3NwYWNlIj4xMjo0NjoyNCBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgMTIgwqAgwqAgwqAw
IMKgIMKgMyDCoCDCoDQgwqAgwqAgNSDCoCDCoDM2MDggwqAgwqAgwqAwIMKgIDIwIMKgMTgwIMKg
IMKgIDAgwqAgwqAgMTU3IMKgIMKgIDEgwqAgwqAgMCDCoCDCoCA0MyDCoDEwMDwvZm9udD48L2Rp
dj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NjoyNSBzZGIgwqAg
wqAgwqAgwqAgwqAgwqAgNDQgwqAgwqAgwqAwIMKgIMKgNyDCoCDCoDYgwqAgMjEwIMKgIDc0MDcy
IMKgIMKgIDQxIMKgNjI1IMKgMTE5IMKgIMKgIDAgwqAgwqAgMTE3IMKgIMKgIDEgwqAgwqAgMiDC
oCDCoCDCoDEgwqAgOTc8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1v
bm9zcGFjZSI+MTI6NDY6MjYgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIDQ4IMKgIMKgIMKgMCDCoCDC
oDYgwqAgwqA4IMKgIDIxNiDCoDIwMjg1MiDCoCDCoDExMiAxODE5IMKgMTEyIMKgIMKgIDAgwqAg
wqAgMTExIMKgIMKgIDEgwqAgwqAgMCDCoCDCoCDCoDAgwqAgOTI8L2ZvbnQ+PC9kaXY+PGRpdj48
Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6Mjcgc2RiIMKgIMKgIMKgIMKg
IMKgIMKgIDUyIMKgIMKgIMKgMCDCoCDCoDcgwqAgwqA3IMKgIDIzMyDCoDMwNzE1NiDCoCDCoDEz
NyAyNjQ4IMKgMTE2IMKgIMKgIDAgwqAgwqAgMTE1IMKgIMKgIDEgwqAgwqAgMCDCoCDCoCDCoDAg
wqAgOTU8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+
MTI6NDY6Mjggc2RiIMKgIMKgIMKgIMKgIMKgIMKgIDE2IMKgIMKgIMKgMCDCoCDCoDIgwqAgwqA4
IMKgIDEwMCDCoCA5MzE2OCDCoCDCoCDCoDcgwqA2MzggwqAxNDYgwqAgwqAgMCDCoCDCoCAxNDUg
wqAgwqAgMSDCoCDCoCAwIMKgIMKgIMKgMSDCoCA5NzwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZh
Y2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0NjoyOSBzZGIgwqAgwqAgwqAgwqAgwqAgwqAg
MTYgwqAgwqAgwqAwIMKgIMKgMiDCoCDCoDggwqAgNjQyIMKgIDM3MDI4IMKgIMKgIDE2IMKgMzE5
IMKgMTE2IMKgIMKgIDAgwqAgwqAgMTE1IMKgIMKgIDEgwqAgwqAgNCDCoCDCoCDCoDMgwqAgOTk8
L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6
MzAgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIDE2IMKgIMKgIMKgMCDCoCDCoDIgwqAgwqA4IMKgIDYy
NCDCoCAzOTA2OCDCoCDCoCAzNiDCoDM0MiDCoDExNCDCoCDCoCAwIMKgIMKgIDExMyDCoCDCoCAx
IMKgIMKgIDMgwqAgwqAgwqAyIMKgIDk5PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9u
b3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjMxIHNkYiDCoCDCoCDCoCDCoCDCoCDCoCA4MCDCoCDC
oCDCoDAgwqAgMTAgwqAgwqA4IMKgIMKgOTQgwqAyNTM4OTIgwqAgwqAxMDUgMjE2OSDCoDExNyDC
oCDCoCAwIMKgIMKgIDExNiDCoCDCoCAxIMKgIMKgIDAgwqAgwqAgwqAwIMKgIDg0PC9mb250Pjwv
ZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjMyIHNkYiDC
oCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoCDC
oDU2NzYgwqAgwqAgwqAwIMKgIDMzIMKgMTcyIMKgIMKgIDAgwqAgwqAgMTcyIMKgIMKgIDEgwqAg
wqAgMCDCoCDCoCAzMCDCoDEwMDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFj
ZSwgbW9ub3NwYWNlIj4xMjo0NjozMyBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgMTYgwqAgwqAgwqAw
IMKgIMKgMiDCoCDCoDggwqAgNjQyIMKgIDY5MjM2IMKgIMKgIDI4IMKgNTgzIMKgMTE5IMKgIMKg
IDAgwqAgwqAgMTE4IMKgIMKgIDEgwqAgwqAgMiDCoCDCoCDCoDEgwqAgOTY8L2ZvbnQ+PC9kaXY+
PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6MzQgc2RiIMKgIMKg
IMKgIMKgIMKgIMKgIMKgOCDCoCDCoCDCoDAgwqAgwqAxIMKgIMKgOCDCoDEwMzIgwqAgMzcxMzIg
wqAgwqAgMzAgwqAzMTUgwqAxMTggwqAgwqAgMCDCoCDCoCAxMTcgwqAgwqAgMSDCoCDCoCAzIMKg
IMKgIMKgMyDCoDEwMDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9u
b3NwYWNlIj4xMjo0NjozNSBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgMTYgwqAgwqAgwqAwIMKgIMKg
MiDCoCDCoDggwqAgODIyIMKgIDU2MjkyIMKgIMKgIDE1IMKgNTE1IMKgMTA5IMKgIMKgIDAgwqAg
wqAgMTA4IMKgIMKgIDEgwqAgwqAgMyDCoCDCoCDCoDEgwqAxMDA8L2ZvbnQ+PC9kaXY+PGRpdj48
Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+PGJyPjwvZm9udD48L2Rpdj48ZGl2Pjxm
b250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4jIERJU0sgU1RBVElTVElDUyAoL3NlYyk8
L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+IyDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCAmbHQ7LS0tLS0tLS0tcmVhZHMtLS0tLS0tLS0tLS0tLS0m
Z3Q7Jmx0Oy0tLS0tLS0tLXdyaXRlcy0tLS0tLS0tLS0tLS0tJmd0OyZsdDstLS0tLS0tLWF2ZXJh
Z2VzLS0tLS0tLS0mZ3Q7IFBjdDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFj
ZSwgbW9ub3NwYWNlIj4jVGltZSDCoCDCoCBOYW1lIMKgIMKgIMKgIEtCeXRlcyBNZXJnZWQgwqBJ
T3MgU2l6ZSDCoFdhaXQgwqBLQnl0ZXMgTWVyZ2VkIMKgSU9zIFNpemUgwqBXYWl0IMKgUldTaXpl
IMKgUUxlbiDCoFdhaXQgU3ZjVGltIFV0aWw8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJt
b25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6MzYgc2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgOCDC
oCDCoCDCoDAgwqAgwqAxIMKgIMKgOCDCoCDCoDQ0IMKgIDU4NzY4IMKgIMKgIDE1IMKgNDUyIMKg
MTMwIMKgIMKgIDAgwqAgwqAgMTI5IMKgIMKgIDEgwqAgwqAgMCDCoCDCoCDCoDIgwqAgOTY8L2Zv
bnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6Mzcg
c2RiIMKgIMKgIMKgIMKgIMKgIMKgIDI4IMKgIMKgIMKgMCDCoCDCoDQgwqAgwqA3IMKgIDM5MCDC
oDExNDk0NCDCoCDCoCA4OSAxMTAwIMKgMTA0IMKgIMKgIDAgwqAgwqAgMTA0IMKgIMKgIDEgwqAg
wqAgMSDCoCDCoCDCoDAgwqAgODg8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3Bh
Y2UsIG1vbm9zcGFjZSI+MTI6NDY6Mzggc2RiIMKgIMKgIMKgIMKgIMKgIMKgIMKgMCDCoCDCoCDC
oDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgIDI5NjY4IMKgIMKgIMKgMCDCoDE3MiDCoDE3MiDC
oCDCoDEyIMKgIMKgIDE3MiDCoCDCoCAxIMKgIMKgMTIgwqAgwqAgwqA1IMKgIDk4PC9mb250Pjwv
ZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjM5IHNkYiDC
oCDCoCDCoCDCoCDCoCDCoCA4MCDCoCDCoCDCoDAgwqAgMTAgwqAgwqA4IMKgIMKgOTAgwqAxMDAw
ODQgwqAgwqAgMzEgwqA4ODIgwqAxMTMgwqAgwqAgMCDCoCDCoCAxMTIgwqAgwqAgMSDCoCDCoCAx
IMKgIMKgIMKgMSDCoCA5MTwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwg
bW9ub3NwYWNlIj4xMjo0Njo0MCBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqAwIMKgIMKgIMKgMCDC
oCDCoDAgwqAgwqAwIMKgIMKgIDAgwqAgMjQyNDQgwqAgwqAgwqAwIMKgMTM5IMKgMTc0IMKgIMKg
IDAgwqAgwqAgMTc0IMKgIMKgIDEgwqAgwqAgMCDCoCDCoCDCoDcgwqAxMDA8L2ZvbnQ+PC9kaXY+
PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6NDEgc2RiIMKgIMKg
IMKgIMKgIMKgIMKgIMKgOCDCoCDCoCDCoDAgwqAgwqAxIMKgIMKgOCDCoDEyMjQgwqAgwqAgwqAg
MCDCoCDCoCDCoDAgwqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgIMKgIMKgIDggwqAgwqAgMSDCoDEy
MjQgwqAgMTAwMCDCoDEwMDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwg
bW9ub3NwYWNlIj4xMjo0Njo0MiBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqA4IMKgIMKgIMKgMCDC
oCDCoDEgwqAgwqA4IMKgMTI0NCDCoCA0MjM2OCDCoCDCoCAyOSDCoDM1NCDCoDEyMCDCoCDCoCAw
IMKgIMKgIDExOSDCoCDCoCAxIMKgIMKgIDMgwqAgwqAgwqAyIMKgIDk2PC9mb250PjwvZGl2Pjxk
aXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjQzIHNkYiDCoCDCoCDC
oCDCoCDCoCDCoCAzNiDCoCDCoCDCoDAgwqAgwqA1IMKgIMKgNyDCoCAyNTEgwqAgNTE0MjggwqAg
wqAgMzIgwqA1MDcgwqAxMDEgwqAgwqAgMCDCoCDCoCAxMDAgwqAgwqAgMSDCoCDCoCAyIMKgIMKg
IMKgMSDCoCA5NDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3Nw
YWNlIj4xMjo0Njo0NCBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgMjQgwqAgwqAgwqAwIMKgIMKgMyDC
oCDCoDggwqAgwqA3MCDCoCDCoDU3MzIgwqAgwqAgMzEgwqAxNDcgwqAgMzkgwqAgwqAxNSDCoCDC
oCDCoDM4IMKgIMKgIDIgwqAgwqAxNiDCoCDCoCDCoDYgwqAgOTk8L2ZvbnQ+PC9kaXY+PGRpdj48
Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6NDUgc2RiIMKgIMKgIMKgIMKg
IMKgIMKgIDMyIMKgIMKgIMKgMCDCoCDCoDQgwqAgwqA4IMKgIMKgIDQgwqAyMTMwNTYgwqAgwqAg
NTMgMTY0NyDCoDEyOSDCoCDCoCAwIMKgIMKgIDEyOSDCoCDCoCAxIMKgIMKgIDAgwqAgwqAgwqAw
IMKgIDc0PC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2Ui
PjEyOjQ2OjQ2IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDggwqAgwqAgwqAwIMKgIMKgMSDCoCDC
oDggwqAxMjIwIMKgIDM3NDE2IMKgIMKgIDI4IMKgMzI4IMKgMTE0IMKgIMKgIDAgwqAgwqAgMTEz
IMKgIMKgIDEgwqAgwqAgMyDCoCDCoCDCoDIgwqAgOTY8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBm
YWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6NDcgc2RiIMKgIMKgIMKgIMKgIMKgIMKg
IMKgOCDCoCDCoCDCoDAgwqAgwqAxIMKgIMKgOCDCoDEyNDggwqAgNTg1NzIgwqAgwqAgNjcgwqA2
MDcgwqAgOTYgwqAgwqAgMCDCoCDCoCDCoDk2IMKgIMKgIDEgwqAgwqAgMiDCoCDCoCDCoDEgwqAg
OTM8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBmYWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6
NDY6NDggc2RiIMKgIMKgIMKgIMKgIMKgIMKgIDQwIMKgIMKgIMKgMCDCoCDCoDUgwqAgwqA4IMKg
IMKgODQgwqAyNzQ4MDggwqAgwqAgODIgMjE3MyDCoDEyNiDCoCDCoCAwIMKgIMKgIDEyNiDCoCDC
oCAxIMKgIMKgIDAgwqAgwqAgwqAwIMKgIDcwPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0i
bW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjQ5IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAg
wqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoCDCoCDCoCAwIMKgIMKgIMKgMCDCoCDC
oDAgwqAgwqAwIMKgIMKgIDAgwqAgwqAgwqAgMCDCoCDCoCAxIMKgIMKgIDAgwqAgwqAgwqAwIMKg
MTAwPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEy
OjQ2OjUwIHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDggwqAgwqAgwqAwIMKgIMKgMSDCoCDCoDgg
wqAxMjQ4IMKgIMKgIMKgIDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAgMCDCoCDCoCDC
oCA4IMKgIMKgIDEgwqAxMjQ4IMKgIDEwMDAgwqAxMDA8L2ZvbnQ+PC9kaXY+PGRpdj48Zm9udCBm
YWNlPSJtb25vc3BhY2UsIG1vbm9zcGFjZSI+MTI6NDY6NTEgc2RiIMKgIMKgIMKgIMKgIMKgIMKg
IMKgOCDCoCDCoCDCoDAgwqAgwqAxIMKgIMKgOCDCoDEyNzIgwqAgwqAgwqAgMCDCoCDCoCDCoDAg
wqAgwqAwIMKgIMKgMCDCoCDCoCAwIMKgIMKgIMKgIDggwqAgwqAgMSDCoDEyNzIgwqAgMTAwMCDC
oDEwMDwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4x
Mjo0Njo1MiBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgMjQgwqAgwqAgwqAwIMKgIMKgMyDCoCDCoDgg
wqAgNDE0IMKgMjA1MjQwIMKgIMKgMTEzIDE3OTggwqAxMTQgwqAgwqAgMCDCoCDCoCAxMTMgwqAg
wqAgMSDCoCDCoCAwIMKgIMKgIMKgMCDCoCA3NTwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9
Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0Njo1MyBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgwqA4
IMKgIMKgIMKgMCDCoCDCoDEgwqAgwqA4IMKgIDg3NiDCoCA5MjQ3NiDCoCDCoCA0OCDCoDgzOSDC
oDExMCDCoCDCoCAwIMKgIMKgIDExMCDCoCDCoCAxIMKgIMKgIDEgwqAgwqAgwqAxIMKgIDg5PC9m
b250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPjEyOjQ2OjU0
IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDCoDAgwqAgwqAg
MCDCoCAzODcwMCDCoCDCoCDCoDAgwqAyMjUgwqAxNzIgwqAgwqAgMCDCoCDCoCAxNzIgwqAgwqAg
MSDCoCDCoCAwIMKgIMKgIMKgNCDCoCA5OTwvZm9udD48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1v
bm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0Njo1NSBzZGIgwqAgwqAgwqAgwqAgwqAgwqAgMTYgwqAg
wqAgwqAwIMKgIMKgMiDCoCDCoDggwqAgNTgyIMKgMTUwNjgwIMKgIMKgIDczIDEyNjIgwqAxMTkg
wqAgwqAgMCDCoCDCoCAxMTkgwqAgwqAgMSDCoCDCoCAxIMKgIMKgIMKgMCDCoCA4NzwvZm9udD48
L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwgbW9ub3NwYWNlIj4xMjo0Njo1NiBzZGIg
wqAgwqAgwqAgwqAgwqAgwqAgwqA4IMKgIMKgIMKgMCDCoCDCoDEgwqAgwqA4IMKgMTIyOCDCoCDC
oCDCoCAwIMKgIMKgIMKgMCDCoCDCoDAgwqAgwqAwIMKgIMKgIDAgwqAgwqAgwqAgOCDCoCDCoCAx
IMKgMTIyOCDCoCAxMDAwIMKgMTAwPC9mb250PjwvZGl2PjxkaXY+PGZvbnQgZmFjZT0ibW9ub3Nw
YWNlLCBtb25vc3BhY2UiPjEyOjQ2OjU3IHNkYiDCoCDCoCDCoCDCoCDCoCDCoCDCoDggwqAgwqAg
wqAwIMKgIMKgMSDCoCDCoDggwqAxMjQ0IMKgIMKgIMKgIDAgwqAgwqAgwqAwIMKgIMKgMCDCoCDC
oDAgwqAgwqAgMCDCoCDCoCDCoCA4IMKgIMKgIDEgwqAxMjQ0IMKgIDEwMDAgwqAxMDA8L2ZvbnQ+
PC9kaXY+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5uZXh0IEkgcGxheWVkIGJhY2sgdGhlIGNv
bGxlY3RsIHByb2Nlc3MgZGF0YSBhbmQgc29ydGVkIGJ5IGRpc2sgcmVhZHMgYW5kIGRpc2NvdmVy
ZWQgdGhlIHRvcCBwcm9jZXNzLCBjb3JyZXNwb25kaW5nIHRvIHRoZSBsb25nIGRpc2sgcmVhZHMg
d2FzIHhmc2FpbGQuIMKgYnR3IC0gSSBhbHNvIHNlZSB0aGUgc2xhYiB4ZnNfaW5vZGUgdXNpbmcg
YWJvdXQgNjBHQi48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pkl0JiMzOTtzIGFsc28gd29ydGgg
bm90aW5nIHRoYXQgSSYjMzk7bSBvbmx5IGRvaW5nIDEtMk1CL3NlYyBvZiB3cml0ZXMgYW5kIHRo
ZSByZXN0IG9mIHRoZSBkYXRhIGxvb2tzIGxpa2UgaXQmIzM5O3MgY29taW5nIGZyb20geGZzIGpv
dXJuYWxpbmcgYmVjYXVzZSB3aGVuIEkgbG9vayBhdCB0aGUgeGZzIHN0YXRzIEkmIzM5O20gc2Vl
aW5nIG9uIHRoZSBvcmRlciBvZiAyMDAtNDAwTUIvc2VjIHhmcyBsb2dnaW5nIHdyaXRlcyAtIGNs
ZWFybHkgdGhleSYjMzk7cmUgbm90IGFsbCBnb2luZyB0byBkaXNrLsKgIE9uY2UgdGhlIHJlYWQg
d2FpdHMgaW5jcmVhc2UgZXZlcnl0aGluZyBzbG93cyBkb3duIGluY2x1ZGluZyB4ZnMgbG9nZ2lu
ZyAoc2luY2UgaXQmIzM5O3MgZG9pbmcgbGVzcykuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5J
JiMzOTttIHN1cmUgdGhlIHNpbXBsZSBhbnN3ZXIgbWF5IGJlIHRoYXQgaXQgaXMgd2hhdCBpdCBp
cywgYnV0IEkmIzM5O20gYWxzbyB3b25kZXJpbmcgd2l0aG91dCBjaGFuZ2VzIHRvIHN3aWZ0IGl0
c2VsZiwgbWlnaHQgdGhlcmUgYmUgc29tZSB3YXlzIHRvIGltcHJvdmUgdGhlIHNpdHVhdGlvbiBi
eSBhZGRpbmcgbW9yZSBtZW1vcnkgb3IgbWFraW5nIGFueSBvdGhlciB0dW5pbmcgY2hhbmdlcz/C
oCBUaGUgc3lzdGVtIEkmIzM5O20gY3VycmVudGx5IHJ1bm5pbmcgbXkgdGVzdHMgb24gaGFzIDEy
OEdCLjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+LW1hcmvCoDwvZGl2PjxkaXY+PGJyPjwvZGl2
PjxkaXY+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5P
biBXZWQsIEphbiA2LCAyMDE2IGF0IDY6NDkgUE0sIERhdmUgQ2hpbm5lciA8c3BhbiBkaXI9Imx0
ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpkYXZpZEBmcm9tb3JiaXQuY29tIiB0YXJnZXQ9Il9ibGFu
ayI+ZGF2aWRAZnJvbW9yYml0LmNvbTwvYT4mZ3Q7PC9zcGFuPiB3cm90ZTo8YnI+PGJsb2NrcXVv
dGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2Jv
cmRlci1sZWZ0LXdpZHRoOjFweDtib3JkZXItbGVmdC1jb2xvcjpyZ2IoMjA0LDIwNCwyMDQpO2Jv
cmRlci1sZWZ0LXN0eWxlOnNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPjxzcGFuIGNsYXNzPSIiPk9u
IFdlZCwgSmFuIDA2LCAyMDE2IGF0IDA1OjQ2OjMzUE0gLTA1MDAsIE1hcmsgU2VnZXIgd3JvdGU6
PGJyPg0KJmd0OyBkYXZlLCB0aGFua3MgZm9yIGdldHRpbmcgYmFjayB0byBtZSBhbmQgdGhlIHBv
aW50ZXIgdG8gdGhlIGNvbmZpZyBkb2MuPGJyPg0KJmd0O8KgIGxvdHMgdG8gYWJzb3JiIGFuZCBw
bGF5IHdpdGguPGJyPg0KJmd0Ozxicj4NCiZndDsgdGhlIHJlYWwgY2hhbGxlbmdlIGZvciBtZSBp
cyB0aGF0IEkmIzM5O20gZG9pbmcgdGVzdGluZyBhcyBkaWZmZXJlbnQgbGV2ZWxzLjxicj4NCiZn
dDsgV2hpbGUgaSByZWFsaXplIHJ1bm5pbmcgMTAwIHBhcmFsbGVsIHN3aWZ0IFBVVCB0aHJlYWRz
IG9uIGEgc21hbGwgc3lzdGVtIGlzPGJyPg0KJmd0OyBub3QgdGhlIGlkZWFsIHdheSB0byBkbyB0
aGluZ3MsIGl0JiMzOTtzIHRoZSBvbmx5IGVhc3kgd2F5IHRvIGdldCBtYXNzaXZlPGJyPg0KJmd0
OyBudW1iZXJzIG9mIG9iamVjdHMgaW50byB0aGUgZmlsbGVzeXN0ZW0gYW5kIG9uY2UgdGhlcmUs
IHRoZSBwZXJmb3JtYW5jZSBvZjxicj4NCiZndDsgYSBzaW5nbGUgc3RyZWFtIGlzIHByZXR0eSBw
b29yIGFuZCBieSBpbnN0cnVtZW50aW5nIHRoZSBzd2lmdCBjb2RlIEkgY2FuPGJyPg0KJmd0OyBj
bGVhcmx5IHNlZSBleGNlc3MgdGltZSBiZWluZyBzcGVudCBpbiBjcmVhdGluZy93cml0aW5nIHRo
ZSBvYmplY3RzIGFuZCBzbzxicj4NCiZndDsgdGhhdCYjMzk7cyBsZWFkIHVzIHRvIGJlbGlldmUg
dGhlIHByb2JsZW0gbGllcyBpbiB0aGUgd2F5IHhmcyBpcyBjb25maWd1cmVkLjxicj4NCiZndDvC
oCBjcmVhdGluZyBhIG5ldyBkaXJlY3Rvcnkgc3RydWN0dXJlIG9uIHRoYXQgc2FtZSBtb3VudCBw
b2ludCBpbW1lZGlhdGVseTxicj4NCiZndDsgcmVzdWx0cyBpbiBoaWdoIGxldmVscyBvZiBwZXJm
b3JtYW5jZS48YnI+DQomZ3Q7PGJyPg0KJmd0OyBBcyBhbiBhdHRlbXB0IHRvIHRyeSB0byByZXBy
b2R1Y2UgdGhlIHByb2JsZW1zIHcvbyBzd2lmdCwgSSB3cm90ZSBhIGxpdHRsZTxicj4NCiZndDsg
cHl0aG9uIHNjcmlwdCB0aGF0IHNpbXBseSBjcmVhdGVzIGZpbGVzIGluIGEgMi10aWVyIHN0cnVj
dHVyZSwgdGhlIGZpcnN0PGJyPg0KJmd0OyB0aWVyIGNvbnNpc3Rpbmcgb2YgMTAyNCBkaXJlY3Rv
cmllcyBhbmQgZWFjaCBkaXJlY3RvcnkgY29udGFpbnMgNDA5Njxicj4NCiZndDsgc3ViZGlyZWN0
b3JpZXMgaW50byB3aGljaCAxSyBmaWxlcyBhcmUgY3JlYXRlZC48YnI+DQo8YnI+DQo8L3NwYW4+
U28geW91IGNyZWF0ZWQgc29tZXRoaW5nIHdpdGggZXZlbiBncmVhdGVyIGZhbi1vdXQgdGhhbiB3
aGF0IHlvdXI8YnI+DQpzd2lmdCBhcHAgaXMgdXNpbmc/PGJyPg0KPGRpdj48ZGl2IGNsYXNzPSJo
NSI+PGJyPg0KJmd0OyBJJiMzOTttIGRvaW5nIHRoaXMgZm9yIDEwMDAwPGJyPg0KJmd0OyBvYmpl
Y3RzIGFzIGEgdGltZSBhbmQgdGhlbiB0aW1pbmcgdGhlbSwgcmVwb3J0aW5nIHRoZSB0aW1lcywg
MTAgcGVyIGxpbmUgc288YnI+DQomZ3Q7IGVhY2ggbGluZSByZXByZXNlbnRzIDEwMCB0aG91c2Fu
ZCBmaWxlIGNyZWF0ZXMuPGJyPg0KJmd0Ozxicj4NCiZndDsgSGVyZSB0b28gSSYjMzk7bSBzZWVp
bmcgZGVncmFkYXRpb24gYW5kIGlmIEkgbG9vayBhdCB3aGF0IGhhcHBlbnMgd2hlbiB0aGVyZTxi
cj4NCiZndDsgYXJlIGFscmVhZHkgM00gZmlsZXMgYW5kIEkgd3JpdGUgMU0gbW9yZSwgSSBzZWUg
dGhlc2UgY3JlYXRpb24gdGltZXMvMTA8YnI+DQomZ3Q7IHRob3VzYW5kOjxicj4NCiZndDs8YnI+
DQomZ3Q7wqAgMS4wMDQyMzbCoCAwLjk2MTQxOcKgIDAuOTk2NTE0wqAgMS4wMTIxNTDCoCAxLjEw
MTc5NMKgIDAuOTk5NDIywqAgMC45OTQ3OTY8YnI+DQomZ3Q7wqAgMS4yMTQ1MzXCoCAwLjk5NzI3
NsKgIDEuMzA2NzM2PGJyPg0KJmd0O8KgIDIuNzkzNDI5wqAgMS4yMDE0NzHCoCAxLjEzMzU3NsKg
IDEuMDY5NjgywqAgMS4wMzA5ODXCoCAxLjA5NjM0McKgIDEuMDUyNjAyPGJyPg0KJmd0O8KgIDEu
MzkxMzY0wqAgMC45OTk0ODDCoCAxLjkxNDEyNTxicj4NCiZndDvCoCAxLjE5Mzg5MsKgIDAuOTY3
MjA2wqAgMS4yNjMzMTDCoCAwLjg5MDQ3MsKgIDEuMDUxOTYywqAgNC4yNTM2OTTCoCAxLjE0NTU3
Mzxicj4NCiZndDvCoCAxLjUyODg0OCAxMy41ODY4OTLCoCA0LjkyNTc5MDxicj4NCiZndDvCoCAz
Ljk3NTQ0MsKgIDguODk2NTUywqAgMS4xOTcwMDXCoCAzLjkwNDIyNsKgIDcuNTAzODA2wqAgMS4y
OTQ4NDLCoCAxLjgxNjQyMjxicj4NCiZndDvCoCA5LjMyOTc5MsKgIDcuMjcwMzIzwqAgNS45MzY1
NDU8YnI+DQomZ3Q7wqAgNy4wNTg2ODXCoCA1LjUxNjg0McKgIDQuNTI3MjcxwqAgMS45NTY1OTLC
oCAxLjM4MjU1McKgIDEuNTEwMzM5wqAgMS4zMTgzNDE8YnI+DQomZ3Q7IDEzLjI1NTkzOcKgIDYu
OTM4ODQ1wqAgNC4xMDYwNjY8YnI+DQomZ3Q7wqAgMi42MTIwNjTCoCAyLjAyODc5NcKgIDQuNjQ3
OTgwwqAgNy4zNzE2MjjCoCA1LjQ3MzQyM8KgIDUuODIzMjAxIDE0LjIyOTEyMDxicj4NCiZndDvC
oCAwLjg5OTM0OMKgIDMuNTM5NjU4wqAgOC41MDE0OTg8YnI+DQomZ3Q7wqAgNC42NjI1OTPCoCA2
LjQyMzUzMMKgIDcuOTgwNzU3wqAgNi4zNjcwMTLCoCAzLjQxNDIzOcKgIDcuMzY0ODU3wqAgNC4x
NDM3NTE8YnI+DQomZ3Q7wqAgNi4zMTczNDggMTEuMzkzMDY3wqAgMS4yNzMzNzE8YnI+DQomZ3Q7
IDE0Ni4wNjczMDDCoCAxLjMxNzgxNMKgIDEuMTc2NTI5wqAgMS4xNzc4MzAgNTIuMjA2NjA1wqAg
MS4xMTI4NTTCoCAyLjA4Nzk5MDxicj4NCiZndDsgNDIuMzI4MjIwwqAgMS4xNzg0MzbCoCAxLjMz
NTIwMjxicj4NCiZndDsgNDkuMTE4MTQwwqAgMS4zNjg2OTbCoCAxLjUxNTgyNiA0NC42OTA0MzHC
oCAwLjkyNzQyOMKgIDAuOTIwODAxwqAgMC45ODU5NjU8YnI+DQomZ3Q7wqAgMS4wMDA1OTHCoCAx
LjAyNzQ1OCA2MC42NTA0NDM8YnI+DQomZ3Q7wqAgMS43NzEzMTjCoCAyLjY5MDQ5OcKgIDIuMjYy
ODY4wqAgMS4wNjEzNDPCoCAwLjkzMjk5OCA2NC4wNjQyMTAgMzcuNzI2MjEzPGJyPg0KJmd0O8Kg
IDEuMjQ1MTI5wqAgMC43NDM3NzHCoCAwLjk5NjY4Mzxicj4NCiZndDs8YnI+DQomZ3Q7IG5vdGhp
bmcgb25lIHNldCBvZiAxMEsgdG9vayBhbG1vc3QgMyBtaW51dGVzITxicj4NCjxicj4NCjwvZGl2
PjwvZGl2PldoaWNoIGlzIG5vIHN1cnByaXNlIGJlY2F1c2UgeW91IGhhdmUgc2xvdyBkaXNrcyBh
bmQgYSAqbG90KiBvZjxicj4NCm1lbW9yeS4gQXQgc29tZSBwb2ludCB0aGUgam91cm5hbCBhbmQv
b3IgbWVtb3J5IGlzIGdvaW5nIHRvIGZpbGwgdXA8YnI+DQp3aXRoIGRpcnR5IG9iamVjdHMgYW5k
IGhhdmUgdG8gYmxvY2sgd2FpdGluZyBmb3Igd3JpdGViYWNrLiBBdCB0aGF0PGJyPg0KcG9pbnQg
dGhlcmUmIzM5O3MgZ29pbmcgdG8gYmUgc2V2ZXJhbCBodW5kcmVkIHRob3VzYW5kIGRpcnR5IGlu
b2RlcyB0aGF0PGJyPg0KbmVlZCB0byBiZSBmbHVzaGVkIHRvIGRpc2sgYmVmb3JlIHByb2dyZXNz
IGNhbiBiZSBtYWRlIGFnYWluLsKgIFRoYXQ8YnI+DQptZXRhZGF0YSB3cml0ZWJhY2sgd2lsbCBi
ZSBzZWVrIGJvdW5kLCBhbmQgdGhhdCYjMzk7cyB3aGVyZSBhbGwgdGhlPGJyPg0KZGVsYXkgY29t
ZXMgZnJvbS48YnI+DQo8YnI+DQpXZSYjMzk7dmUgYmVlbiB0aHJvdWdoIHRoaXMgcHJvYmxlbSBz
ZXZlcmFsIHRpbWVzIG5vdyB3aXRoIGRpZmZlcmVudDxicj4NCnN3aWZ0IHVzZXJzIG92ZXIgdGhl
IHBhc3QgY291cGxlIG9mIHllYXJzLiBQbGVhc2UgZ28gYW5kIHNlYXJjaCB0aGU8YnI+DQpsaXN0
IGFyY2hpdmVzLCBiZWNhdXNlIGV2ZXJ5IHRpbWUgdGhlIHNvbHV0aW9uIGhhcyBiZWVuIHRoZSBz
YW1lOjxicj4NCjxicj4NCsKgIMKgIMKgIMKgIC0gcmVkdWNlIHRoZSBkaXJlY3RvcnkgaGVpcmFy
Y2h5IHRvIGEgc2luZ2xlIGxldmVsIHdpdGgsIGF0PGJyPg0KwqAgwqAgwqAgwqAgwqAgbW9zdCwg
dGhlIG51bWJlciBvZiBkaXJlY3RvcmllcyBtYXRjaGluZyB0aGUgZXhwZWN0ZWQ8YnI+DQrCoCDC
oCDCoCDCoCDCoCAqcHJvZHVjdGlvbiogY29uY3VycmVuY3kgbGV2ZWw8YnI+DQrCoCDCoCDCoCDC
oCAtIHJlZHVjZSB0aGUgWEZTIGxvZyBzaXplIGRvd24gdG8gMzItMTI4TUIgdG8gbGltaXQgZGly
dHk8YnI+DQrCoCDCoCDCoCDCoCDCoCBtZXRhZGF0YSBvYmplY3QgYnVpbGR1cCBpbiBtZW1vcnk8
YnI+DQrCoCDCoCDCoCDCoCAtIHJlZHVjZSB0aGUgbnVtYmVyIG9mIEFHcyB0byBhcyBzbWFsbCBh
cyBuZWNlc3NhcnkgdG88YnI+DQrCoCDCoCDCoCDCoCDCoCBtYWludGFpbiAvYWxsb2NhdGlvbi8g
Y29uY3VycmVuY3kgdG8gbGltaXQgdGhlIG51bWJlciBvZjxicj4NCsKgIMKgIMKgIMKgIMKgIGRp
ZmZlcmVudCBsb2NhdGlvbnMgWEZTIHdyaXRlcyB0byB0aGUgZGlza3MgKHR5cGljYWxseTxicj4N
CsKgIMKgIMKgIMKgIMKgIDEwLTIweCBsZXNzIHRoYW4gdGhlIGFwcGxpY2F0aW9uIGxldmVsIGNv
bmN1cnJlbmN5KTxicj4NCsKgIMKgIMKgIMKgIC0gdXNlIGEgMy4xNisga2VybmVsIHdpdGggdGhl
IGZyZWUgaW5vZGUgYnRyZWUgb24tZGlzazxicj4NCsKgIMKgIMKgIMKgIMKgIGZvcm1hdCBmZWF0
dXJlIHRvIGtlZXAgaW5vZGUgYWxsb2NhdGlvbiBDUFUgb3ZlcmhlYWQgbG93PGJyPg0KwqAgwqAg
wqAgwqAgwqAgYW5kIGNvbnNpc3RlbnQgcmVnYXJkbGVzcyBvZiB0aGUgbnVtYmVyIG9mIGlub2Rl
cyBhbHJlYWR5PGJyPg0KwqAgwqAgwqAgwqAgwqAgYWxsb2NhdGVkIGluIHRoZSBmaWxlc3lzdGVt
Ljxicj4NCjxzcGFuIGNsYXNzPSIiPjxicj4NCiZndDsgbXkgbWFpbiBxdWVzdGlvbnMgYXQgdGhp
cyBwb2ludCBhcmUgaXMgdGhpcyBwZXJmb3JtYW5jZSBleHBlY3RlZCBhbmQvb3I8YnI+DQomZ3Q7
IG1pZ2h0IGEgbmV3ZXIga2VybmVsIGhlbHA/wqAgYW5kIG1pZ2h0IGl0IGJlIHBvc3NpYmxlIHRv
IHNpZ25pZmljYW50bHk8YnI+DQomZ3Q7IGltcHJvdmUgdGhpbmdzIHZpYSB0dW5pbmcgb3IgaXMg
aXQgd2hhdCBpdCBpcz/CoCBJIGRvIHJlYWxpemUgSSYjMzk7bSBzdGFydGluZzxicj4NCiZndDsg
d2l0aCBhbiBlbXB0eSBkaXJlY3RvcnkgdHJlZSB3aG9zZSBwZXJmb3JtYW5jZSBkZWdyYWRlcyBh
cyBpdCBmaWxscywgYnV0IGlmPGJyPg0KJmd0OyBJIHdhbnRlZCB0byB0dW5lIGZvciBzYXkgMTBN
IG9yIG1heWJlIDEwME0gZmlsZXMgbWlnaHQgSSBiZSBhYmxlIHRvIGV4cGVjdDxicj4NCjxicj4N
Cjwvc3Bhbj5UaGUgbWtmcyBkZWZhdWx0cyB3aWxsIHdvcmsganVzdCBmaW5lIHdpdGggdGhhdCBt
YW55IGZpbGVzIGluIHRoZTxicj4NCmZpbGVzeXN0ZW0uIFlvdXIgYXBwbGljYXRpb24gY29uZmln
dXJhdGlvbiBhbmQgZGF0YSBzdG9yZSBsYXlvdXQgaXM8YnI+DQpsaWtlbHkgdG8gYmUgeW91ciBi
aWdnZXN0IHByb2JsZW0gaGVyZS48YnI+DQo8ZGl2IGNsYXNzPSIiPjxkaXYgY2xhc3M9Img1Ij48
YnI+DQpDaGVlcnMsPGJyPg0KPGJyPg0KRGF2ZS48YnI+DQo8YnI+DQotLTxicj4NCkRhdmUgQ2hp
bm5lcjxicj4NCjxhIGhyZWY9Im1haWx0bzpkYXZpZEBmcm9tb3JiaXQuY29tIj5kYXZpZEBmcm9t
b3JiaXQuY29tPC9hPjxicj4NCjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9k
aXY+PC9kaXY+PC9kaXY+DQo=
--047d7b5d457e95b6bc052a2b31af--

From bschubert@ddn.com  Mon Jan 25 12:24:49 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 609BB7CA1
	for ; Mon, 25 Jan 2016 12:24:49 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id ED7D0AC002
	for ; Mon, 25 Jan 2016 10:24:45 -0800 (PST)
X-ASG-Debug-ID: 1453746284-04bdf04d6cbb8f0001-NocioJ
Received: from legacy.ddn.com (legacy.ddn.com [64.47.133.206]) by cuda.sgi.com with ESMTP id rSCYAu6iWwgh8oE5 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 25 Jan 2016 10:24:44 -0800 (PST)
X-Barracuda-Envelope-From: bschubert@ddn.com
X-Barracuda-Apparent-Source-IP: 64.47.133.206
Received: from LAX-EX-MB2.datadirect.datadirectnet.com
 ([fe80::96:2379:f1b2:ef2d]) by LAX-EX-CAHT2.datadirect.datadirectnet.com
 ([fe80::a499:249:3d8b:f1a4%12]) with mapi id 14.03.0224.002; Mon, 25 Jan 2016
 10:24:43 -0800
From: Bernd Schubert 
To: Mark Seger , Linux fs XFS 
CC: Laurence Oberman 
Subject: Re: xfs and swift
Thread-Topic: xfs and swift
X-ASG-Orig-Subj: Re: xfs and swift
Thread-Index: AQHRV52oAMBQxHEMt0OfT2+g6ZBqCQ==
Date: Mon, 25 Jan 2016 18:24:42 +0000
Message-ID: <56A66869.3080506@ddn.com>
References: 
In-Reply-To: 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.4.0
x-originating-ip: [10.64.108.27]
Content-Type: text/plain; charset="utf-8"
Content-ID: <66D27453C634284794EC93306E65EF43@ddn.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Barracuda-Connect: legacy.ddn.com[64.47.133.206]
X-Barracuda-Start-Time: 1453746284
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.02
X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, THREAD_INDEX, THREAD_TOPIC
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26439
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.01 THREAD_INDEX           thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig==
	0.01 THREAD_TOPIC           Thread-Topic: ...(Japanese Subject)...
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

SGkgTWFyayENCg0KT24gMDEvMDYvMjAxNiAwNDoxNSBQTSwgTWFyayBTZWdlciB3cm90ZToNCj4g
SSd2ZSByZWNlbnRseSBmb3VuZCB0aGUgcGVyZm9ybWFuY2Ugb3VyIGRldmVsb3BtZW50IHN3aWZ0
IHN5c3RlbSBpcw0KPiBkZWdyYWRpbmcgb3ZlciB0aW1lIGFzIHRoZSBudW1iZXIgb2Ygb2JqZWN0
cy9maWxlcyBpbmNyZWFzZXMuICBUaGlzIGlzIGENCj4gcmVsYXRpdmVseSBzbWFsbCBzeXN0ZW0s
IGVhY2ggc2VydmVyIGhhcyAzIDQwMEdCIGRpc2tzLiAgVGhlIHN5c3RlbSBJJ20NCj4gY3VycmVu
dGx5IGxvb2tpbmcgYXQgaGFzIGFib3V0IDcwR0IgdGllZCB1cCBpbiBzbGFicyBhbG9uZSwgY2xv
c2UgdG8gNTVHQg0KPiBpbiB4ZnMgaW5vZGVzIGFuZCBpbGksIGFuZCBhYm91dCAyR0IgZnJlZS4g
IFRoZSBrZXJuZWwNCj4gaXMgMy4xNC41Ny0xLWFtZDY0LWhsaW51eC4NCj4gDQo+IEhlcmUncyB0
aGUgd2F5IHRoZSBmaWxlc3lzdGVtcyBhcmUgbW91bnRlZDoNCj4gDQo+IC9kZXYvc2RiMSBvbiAv
c3J2L25vZGUvZGlzazAgdHlwZSB4ZnMNCj4gKHJ3LG5vYXRpbWUsbm9kaXJhdGltZSxhdHRyMixu
b2JhcnJpZXIsaW5vZGU2NCxsb2didWZzPTgsbG9nYnNpemU9MjU2ayxzdW5pdD01MTIsc3dpZHRo
PTE1MzYsbm9xdW90YSkNCj4gDQo+IEkgY2FuIGRvIGFib3V0IDIwMDAgMUsgZmlsZSBjcmVhdGVz
L3NlYyB3aGVuIHJ1bm5pbmcgMiBtaW51dGUgUFVUIHRlc3RzIGF0DQo+IDEwMCB0aHJlYWRzLiAg
SWYgSSByZXBlYXQgdGhhdCB0ZXN0cyBmb3IgbXVsdGlwbGUgaG91cnMsIEkgc2VlIHRoZSBudW1i
ZXINCj4gb2YgSU9QUyBzdGVhZGlseSBkZWNyZWFzaW5nIHRvIGFib3V0IDc3MCBhbmQgdGhlIHZl
cnkgbmV4dCBydW4gaXQgZHJvcHMgdG8NCj4gMjYwIGFuZCBjb250aW51ZXMgdG8gZmFsbCBmcm9t
IHRoZXJlLiAgVGhpcyBoYXBwZW5zIGF0IGFib3V0IDEyTSBmaWxlcy4NCj4gDQo+IFRoZSBkaXJl
Y3Rvcnkgc3RydWN0dXJlIGlzIDIgdGllcmVkLCB3aXRoIDEwMDAgZGlyZWN0b3JpZXMgcGVyIHRp
ZXIgc28gd2UNCj4gY2FuIGhhdmUgYWJvdXQgMU0gb2YgdGhlbSwgdGhvdWdoIHRoZXkgZG9uJ3Qg
Y3VycmVudGx5IGFsbCBleGlzdC4NCg0KVGhpcyBzb3VuZHMgcHJldHR5IG11Y2ggbGlrZSBoYXNo
IGRpcmVjdG9yaWVzIGFzIHVzZWQgYnkgc29tZSBwYXJhbGxlbA0KZmlsZSBzeXN0ZW1zIChMdXN0
cmUgYW5kIGluIHRoZSBwYXN0IEJlZUdGUykuIEZvciB1cyB0aGUgZmlsZSBjcmVhdGUNCnNsb3cg
ZG93biB3YXMgZHVlIHRvIGxvb2t1cCBpbiBkaXJlY3RvcmllcyBpZiBhIGZpbGUgd2l0aCB0aGUg
c2FtZSBuYW1lDQphbHJlYWR5IGV4aXN0cy4gQXQgbGVhc3QgZm9yIGV4dDQgaXQgd2FzIHJhdGhl
ciBlYXN5IHRvIGRlbW9uc3RyYXRlIHRoYXQNCnNpbXBseSBjYWNoaW5nIGRpcmVjdG9yeSBibG9j
a3Mgd291bGQgZWxpbWluYXRlIHRoYXQgaXNzdWUuDQpXZSB0aGVuIGNvbnNpZGVyZWQgd29ya2lu
ZyBvbiBhIGJldHRlciBrZXJuZWwgY2FjaGUsIGJ1dCBpbiB0aGUgZW5kDQpzaW1wbHkgZm91bmQg
YSB3YXkgdG8gZ2V0IHJpZCBvZiBzdWNoIGEgc2ltcGxlIGRpcmVjdG9yeSBzdHJ1Y3R1cmUgaW4N
CkJlZUdGUyBhbmQgY2hhbmdlZCBpdCB0byBhIG1vcmUgY29tcGxleCBsYXlvdXQsIGJ1dCB3aXRo
IGxlc3MgcmFuZG9tDQphY2Nlc3MgYW5kIHNvIHdlIGNvdWxkIGVsaW1pbmF0ZSB0aGUgbWFpbiBy
ZWFzb24gZm9yIHRoZSBzbG93IGRvd24uDQoNCk5vdyBJIGhhdmUgbm8gaWRlYSB3aGF0IGEgInN3
aWZ0IHN5c3RlbSIgaXMgYW5kIGluIHdoaWNoIG9yZGVyIGl0DQpjcmVhdGVzIGFuZCBhY2Nlc3Nl
cyB0aG9zZSBmaWxlcyBhbmQgaWYgaXQgd291bGQgYmUgcG9zc2libGUgdG8gY2hhbmdlDQp0aGUg
YWNjZXNzIHBhdHRlcm4uIE9uZSB0aGluZyB5b3UgbWlnaHQgdHJ5IGFuZCB3aGljaCBzaG91bGQg
d29yayBtdWNoDQpiZXR0ZXIgc2luY2UgMy4xMSBpcyB0aGUgdmZzX2NhY2hlX3ByZXNzdXJlIHNl
dHRpbmcuIFRoZSBsb3dlciBpdCBpcyB0aGUNCmxlc3MgZGVudHJpZXMvaW5vZGVzIGFyZSBkcm9w
cGVkIGZyb20gY2FjaGUgd2hlbiBwYWdlcyBhcmUgbmVlZGVkIGZvcg0KZmlsZSBkYXRhLg0KDQoN
Cg0KQ2hlZXJzLA0KQmVybmQ=

From bfoster@redhat.com  Mon Jan 25 12:25:15 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 0906D7CA5
	for ; Mon, 25 Jan 2016 12:25:15 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id E7F75304039
	for ; Mon, 25 Jan 2016 10:25:11 -0800 (PST)
X-ASG-Debug-ID: 1453746310-04cbb026ffc1ab0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id uWAjIzqdNvNheuye (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 10:25:10 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	by mx1.redhat.com (Postfix) with ESMTPS id 2B74B96DC;
	Mon, 25 Jan 2016 18:25:10 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PIP9fZ027898;
	Mon, 25 Jan 2016 13:25:09 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id DCBEC120BCD; Mon, 25 Jan 2016 13:25:08 -0500 (EST)
Date: Mon, 25 Jan 2016 13:25:08 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 5/8] xfs: use vfs inode nlink field everywhere
Message-ID: <20160125182508.GA27301@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 5/8] xfs: use vfs inode nlink field everywhere
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
 <1452751765-4420-6-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452751765-4420-6-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453746310
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 14, 2016 at 05:09:22PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> The Vfs tracks the inodenlink just like the xfs_icdinode. We can
> remove the variable from the icdinode and use the vfs inode variable
> everywhere, reducing the size of the xfs_icdinode by a further 4
> bytes.
> 
> Signed-off-by: Dave Chinner 
> ---
>  fs/xfs/libxfs/xfs_inode_buf.c |  6 ++--
>  fs/xfs/libxfs/xfs_inode_buf.h |  1 -
>  fs/xfs/xfs_inode.c            | 73 ++++++++++++++++++++-----------------------
>  fs/xfs/xfs_inode.h            |  2 --
>  fs/xfs/xfs_inode_item.c       |  2 +-
>  fs/xfs/xfs_iops.c             |  3 +-
>  fs/xfs/xfs_itable.c           |  2 +-
>  fs/xfs/xfs_log_recover.c      |  2 +-
>  8 files changed, 41 insertions(+), 50 deletions(-)
> 
...
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
> index feb04e6..774d71f 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.h
> +++ b/fs/xfs/libxfs/xfs_inode_buf.h
> @@ -34,7 +34,6 @@ struct xfs_icdinode {
>  	__uint16_t	di_flushiter;	/* incremented on flush */
>  	__uint32_t	di_uid;		/* owner's user id */
>  	__uint32_t	di_gid;		/* owner's group id */
> -	__uint32_t	di_nlink;	/* number of links to file */
>  	__uint16_t	di_projid_lo;	/* lower part of owner's project id */
>  	__uint16_t	di_projid_hi;	/* higher part of owner's project id */
>  	xfs_fsize_t	di_size;	/* number of bytes in file */
> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> index 914ec41..9ee766b 100644
> --- a/fs/xfs/xfs_inode.c
> +++ b/fs/xfs/xfs_inode.c
> @@ -58,7 +58,7 @@ kmem_zone_t *xfs_inode_zone;
>  #define	XFS_ITRUNC_MAX_EXTENTS	2
>  
>  STATIC int xfs_iflush_int(xfs_inode_t *, xfs_buf_t *);
> -
> +STATIC int xfs_iunlink(xfs_trans_t *, xfs_inode_t *, bool);

Nit:			 (struct xfs_trans *, struct xfs_inode *, ...)

Might as well fix the neighbors as well I suppose.

>  STATIC int xfs_iunlink_remove(xfs_trans_t *, xfs_inode_t *);
>  
>  /*
...
> diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
> index b008677..c424d4b 100644
> --- a/fs/xfs/xfs_iops.c
> +++ b/fs/xfs/xfs_iops.c
> @@ -455,7 +455,7 @@ xfs_vn_getattr(
>  	stat->size = XFS_ISIZE(ip);
>  	stat->dev = inode->i_sb->s_dev;
>  	stat->mode = ip->i_d.di_mode;
> -	stat->nlink = ip->i_d.di_nlink;
> +	stat->nlink = inode->i_nlink;
>  	stat->uid = inode->i_uid;
>  	stat->gid = inode->i_gid;
>  	stat->ino = ip->i_ino;
> @@ -1212,7 +1212,6 @@ xfs_setup_inode(
>  	hlist_add_fake(&inode->i_hash);
>  
>  	inode->i_mode	= ip->i_d.di_mode;
> -	set_nlink(inode, ip->i_d.di_nlink);
>  	inode->i_uid    = xfs_uid_to_kuid(ip->i_d.di_uid);
>  	inode->i_gid    = xfs_gid_to_kgid(ip->i_d.di_gid);
>  

I'm wondering if we have an issue here if we happen to get an inode
that's reclaimable. E.g., we get an xfs_fs_destroy_inode() call on an
inode and set the reclaimable tag. IIUC, a lookup that occurs after that
point but before we actually reclaim the inode can go through the
associated XFS_IRECLAIMABLE section of xfs_iget_cache_hit(). In there,
we call inode_init_always() which fixes inode->i_nlink = 1.

It looks like we'd call into here (xfs_setup_inode()) since we set
XFS_INEW, and thus previously this would update inode->i_nlink based on
the ip->i_d.di_nlink value, but I don't see where that would happen
now..  hmm?

Brian

> diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
> index 2acda42..cfb6527 100644
> --- a/fs/xfs/xfs_itable.c
> +++ b/fs/xfs/xfs_itable.c
> @@ -85,7 +85,6 @@ xfs_bulkstat_one_int(
>  	/* xfs_iget returns the following without needing
>  	 * further change.
>  	 */
> -	buf->bs_nlink = dic->di_nlink;
>  	buf->bs_projid_lo = dic->di_projid_lo;
>  	buf->bs_projid_hi = dic->di_projid_hi;
>  	buf->bs_ino = ino;
> @@ -94,6 +93,7 @@ xfs_bulkstat_one_int(
>  	buf->bs_gid = dic->di_gid;
>  	buf->bs_size = dic->di_size;
>  
> +	buf->bs_nlink = inode->i_nlink;
>  	buf->bs_atime.tv_sec = inode->i_atime.tv_sec;
>  	buf->bs_atime.tv_nsec = inode->i_atime.tv_nsec;
>  	buf->bs_mtime.tv_sec = inode->i_mtime.tv_sec;
> diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
> index 4b79cf0..611c25c 100644
> --- a/fs/xfs/xfs_log_recover.c
> +++ b/fs/xfs/xfs_log_recover.c
> @@ -4337,7 +4337,7 @@ xlog_recover_process_one_iunlink(
>  	if (error)
>  		goto fail_iput;
>  
> -	ASSERT(ip->i_d.di_nlink == 0);
> +	ASSERT(VFS_I(ip)->i_nlink == 0);
>  	ASSERT(ip->i_d.di_mode != 0);
>  
>  	/* setup for the next pass */
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Mon Jan 25 12:25:20 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id B0AC27CA9
	for ; Mon, 25 Jan 2016 12:25:20 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 8D4598F8035
	for ; Mon, 25 Jan 2016 10:25:16 -0800 (PST)
X-ASG-Debug-ID: 1453746314-04cb6c0d60c59f0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id BE6fW0mEEgPJBCwh (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 10:25:15 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	by mx1.redhat.com (Postfix) with ESMTPS id D248FC00354F;
	Mon, 25 Jan 2016 18:25:14 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PIPEK7027917;
	Mon, 25 Jan 2016 13:25:14 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id CF238120BCD; Mon, 25 Jan 2016 13:25:13 -0500 (EST)
Date: Mon, 25 Jan 2016 13:25:13 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 6/8] xfs: move inode generation count to VFS inode
Message-ID: <20160125182513.GB27301@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 6/8] xfs: move inode generation count to VFS inode
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
 <1452751765-4420-7-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452751765-4420-7-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453746315
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 14, 2016 at 05:09:23PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> Pull another 4 bytes out of the xfs_icdinode.
> 
> Signed-off-by: Dave Chinner 
> ---
>  fs/xfs/libxfs/xfs_inode_buf.c | 8 ++++----
>  fs/xfs/libxfs/xfs_inode_buf.h | 1 -
>  fs/xfs/xfs_export.c           | 2 +-
>  fs/xfs/xfs_inode.c            | 5 +----
>  fs/xfs/xfs_inode_item.c       | 2 +-
>  fs/xfs/xfs_ioctl.c            | 2 +-
>  fs/xfs/xfs_iops.c             | 1 -
>  fs/xfs/xfs_itable.c           | 2 +-
>  8 files changed, 9 insertions(+), 14 deletions(-)
> 
...
> diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
> index c424d4b..397ce85 100644
> --- a/fs/xfs/xfs_iops.c
> +++ b/fs/xfs/xfs_iops.c
> @@ -1227,7 +1227,6 @@ xfs_setup_inode(
>  		break;
>  	}
>  
> -	inode->i_generation = ip->i_d.di_gen;
>  	i_size_write(inode, ip->i_d.di_size);
>  	xfs_diflags_to_iflags(inode, ip);
>  

Same question here wrt to reclaimable inodes as the previous patch and
nlink..? Seems Ok otherwise.

Brian

> diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
> index cfb6527..6162e65 100644
> --- a/fs/xfs/xfs_itable.c
> +++ b/fs/xfs/xfs_itable.c
> @@ -100,11 +100,11 @@ xfs_bulkstat_one_int(
>  	buf->bs_mtime.tv_nsec = inode->i_mtime.tv_nsec;
>  	buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec;
>  	buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec;
> +	buf->bs_gen = inode->i_generation;
>  
>  	buf->bs_xflags = xfs_ip2xflags(ip);
>  	buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog;
>  	buf->bs_extents = dic->di_nextents;
> -	buf->bs_gen = dic->di_gen;
>  	memset(buf->bs_pad, 0, sizeof(buf->bs_pad));
>  	buf->bs_dmevmask = dic->di_dmevmask;
>  	buf->bs_dmstate = dic->di_dmstate;
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Mon Jan 25 12:25:21 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 8739B7CA9
	for ; Mon, 25 Jan 2016 12:25:21 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 00080AC004
	for ; Mon, 25 Jan 2016 10:25:20 -0800 (PST)
X-ASG-Debug-ID: 1453746319-04cb6c0d5ec59f0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 3QaphaJl3fEyyzqR (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 10:25:19 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26])
	by mx1.redhat.com (Postfix) with ESMTPS id 2B1A4C0AC93F;
	Mon, 25 Jan 2016 18:25:19 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PIPItn014729;
	Mon, 25 Jan 2016 13:25:18 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 0E246120BCD; Mon, 25 Jan 2016 13:25:18 -0500 (EST)
Date: Mon, 25 Jan 2016 13:25:18 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 7/8] xfs: move di_changecount to VFS inode
Message-ID: <20160125182517.GC27301@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 7/8] xfs: move di_changecount to VFS inode
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
 <1452751765-4420-8-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452751765-4420-8-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453746319
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 14, 2016 at 05:09:24PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> We can store the di_changecount in the i_version field of the VFS
> inode and remove another 8 bytes from the xfs_icdinode.
> 
> Signed-off-by: Dave Chinner 
> ---

Reviewed-by: Brian Foster 

>  fs/xfs/libxfs/xfs_inode_buf.c | 4 ++--
>  fs/xfs/libxfs/xfs_inode_buf.h | 1 -
>  fs/xfs/xfs_inode.c            | 2 +-
>  fs/xfs/xfs_inode_item.c       | 2 +-
>  fs/xfs/xfs_trans_inode.c      | 2 +-
>  5 files changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.c b/fs/xfs/libxfs/xfs_inode_buf.c
> index 0a3f10b..9807c21 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.c
> +++ b/fs/xfs/libxfs/xfs_inode_buf.c
> @@ -251,7 +251,7 @@ xfs_inode_from_disk(
>  	to->di_flags	= be16_to_cpu(from->di_flags);
>  
>  	if (to->di_version == 3) {
> -		to->di_changecount = be64_to_cpu(from->di_changecount);
> +		inode->i_version = be64_to_cpu(from->di_changecount);
>  		to->di_crtime.t_sec = be32_to_cpu(from->di_crtime.t_sec);
>  		to->di_crtime.t_nsec = be32_to_cpu(from->di_crtime.t_nsec);
>  		to->di_flags2 = be64_to_cpu(from->di_flags2);
> @@ -300,7 +300,7 @@ xfs_inode_to_disk(
>  	to->di_flags = cpu_to_be16(from->di_flags);
>  
>  	if (from->di_version == 3) {
> -		to->di_changecount = cpu_to_be64(from->di_changecount);
> +		to->di_changecount = cpu_to_be64(inode->i_version);
>  		to->di_crtime.t_sec = cpu_to_be32(from->di_crtime.t_sec);
>  		to->di_crtime.t_nsec = cpu_to_be32(from->di_crtime.t_nsec);
>  		to->di_flags2 = cpu_to_be64(from->di_flags2);
> diff --git a/fs/xfs/libxfs/xfs_inode_buf.h b/fs/xfs/libxfs/xfs_inode_buf.h
> index 68da576..e49c229 100644
> --- a/fs/xfs/libxfs/xfs_inode_buf.h
> +++ b/fs/xfs/libxfs/xfs_inode_buf.h
> @@ -47,7 +47,6 @@ struct xfs_icdinode {
>  	__uint16_t	di_dmstate;	/* DMIG state info */
>  	__uint16_t	di_flags;	/* random flags, XFS_DIFLAG_... */
>  
> -	__uint64_t	di_changecount;	/* number of attribute changes */
>  	__uint64_t	di_flags2;	/* more random flags */
>  
>  	xfs_ictimestamp_t di_crtime;	/* time created */
> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> index 391d676..96aee23 100644
> --- a/fs/xfs/xfs_inode.c
> +++ b/fs/xfs/xfs_inode.c
> @@ -832,7 +832,7 @@ xfs_ialloc(
>  	ip->i_d.di_flags = 0;
>  
>  	if (ip->i_d.di_version == 3) {
> -		ip->i_d.di_changecount = 1;
> +		inode->i_version = 1;
>  		ip->i_d.di_flags2 = 0;
>  		ip->i_d.di_crtime.t_sec = (__int32_t)tv.tv_sec;
>  		ip->i_d.di_crtime.t_nsec = (__int32_t)tv.tv_nsec;
> diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
> index 5cdf5ef..334d657 100644
> --- a/fs/xfs/xfs_inode_item.c
> +++ b/fs/xfs/xfs_inode_item.c
> @@ -364,7 +364,7 @@ xfs_inode_to_log_dinode(
>  	to->di_flags = from->di_flags;
>  
>  	if (from->di_version == 3) {
> -		to->di_changecount = from->di_changecount;
> +		to->di_changecount = inode->i_version;
>  		to->di_crtime.t_sec = from->di_crtime.t_sec;
>  		to->di_crtime.t_nsec = from->di_crtime.t_nsec;
>  		to->di_flags2 = from->di_flags2;
> diff --git a/fs/xfs/xfs_trans_inode.c b/fs/xfs/xfs_trans_inode.c
> index 3f0d466..11a3af0 100644
> --- a/fs/xfs/xfs_trans_inode.c
> +++ b/fs/xfs/xfs_trans_inode.c
> @@ -117,7 +117,7 @@ xfs_trans_log_inode(
>  	 */
>  	if (!(ip->i_itemp->ili_item.li_desc->lid_flags & XFS_LID_DIRTY) &&
>  	    IS_I_VERSION(VFS_I(ip))) {
> -		ip->i_d.di_changecount = ++VFS_I(ip)->i_version;
> +		VFS_I(ip)->i_version++;
>  		flags |= XFS_ILOG_CORE;
>  	}
>  
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Mon Jan 25 12:25:28 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 9DF707CA9
	for ; Mon, 25 Jan 2016 12:25:28 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 31019AC004
	for ; Mon, 25 Jan 2016 10:25:28 -0800 (PST)
X-ASG-Debug-ID: 1453746323-04bdf04d6bbb960001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id D5zTCpcJjoapOTGF (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2016 10:25:23 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id 11781C076624;
	Mon, 25 Jan 2016 18:25:23 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0PIPMS2016607;
	Mon, 25 Jan 2016 13:25:22 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id D3C7B120BCD; Mon, 25 Jan 2016 13:25:21 -0500 (EST)
Date: Mon, 25 Jan 2016 13:25:21 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 8/8] xfs: mode di_mode to vfs inode
Message-ID: <20160125182521.GD27301@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 8/8] xfs: mode di_mode to vfs inode
References: <1452751765-4420-1-git-send-email-david@fromorbit.com>
 <1452751765-4420-9-git-send-email-david@fromorbit.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452751765-4420-9-git-send-email-david@fromorbit.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453746323
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 14, 2016 at 05:09:25PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> Move the di_mode value from the xfs_icdinode to the VFS inode, reducing
> the xfs_icdinode byte another 2 bytes and collapsing another 2 byte hole
> in the structure.
> 
> Signed-off-by: Dave Chinner 
> ---
>  fs/fs-writeback.c              |  1 +
>  fs/xfs/libxfs/xfs_bmap.c       |  6 +++---
>  fs/xfs/libxfs/xfs_dir2.c       | 12 +++++------
>  fs/xfs/libxfs/xfs_inode_buf.c  |  8 +++----
>  fs/xfs/libxfs/xfs_inode_buf.h  |  1 -
>  fs/xfs/libxfs/xfs_inode_fork.c |  2 +-
>  fs/xfs/xfs_bmap_util.c         |  4 ++--
>  fs/xfs/xfs_dir2_readdir.c      |  2 +-
>  fs/xfs/xfs_file.c              |  6 +++---
>  fs/xfs/xfs_filestream.c        |  4 ++--
>  fs/xfs/xfs_icache.c            | 42 +++++++++++++++++++++++++++++++-----
>  fs/xfs/xfs_inode.c             | 48 ++++++++++++++++++++----------------------
>  fs/xfs/xfs_inode.h             |  4 ++--
>  fs/xfs/xfs_inode_item.c        |  2 +-
>  fs/xfs/xfs_ioctl.c             | 14 ++++++------
>  fs/xfs/xfs_iops.c              | 12 ++++-------
>  fs/xfs/xfs_itable.c            |  2 +-
>  fs/xfs/xfs_log_recover.c       |  2 +-
>  fs/xfs/xfs_mount.c             |  2 +-
>  19 files changed, 100 insertions(+), 74 deletions(-)
> 
> diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
> index 023f6a1..0858457 100644
> --- a/fs/fs-writeback.c
> +++ b/fs/fs-writeback.c
> @@ -2106,6 +2106,7 @@ EXPORT_SYMBOL(__mark_inode_dirty);
>   * completed by the time we have gained the lock and waited for all IO that is
>   * in progress regardless of the order callers are granted the lock.
>   */
> +#include "linux/delay.h"
>  static void wait_sb_inodes(struct super_block *sb)
>  {
>  	struct inode *inode, *old_inode = NULL;

What's this hunk about?

...
> diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
> index 7c26f86..c1edb0f 100644
> --- a/fs/xfs/xfs_icache.c
> +++ b/fs/xfs/xfs_icache.c
...
> @@ -135,6 +138,35 @@ xfs_inode_free(
>  }
>  
>  /*
> + * When we recycle a reclaimable inode, we need to re-initialise the VFS inode
> + * part of the structure. This is made more complex by the fact we store
> + * information about the on-disk values in the VFS inode and so we can't just
> + * overwrite it's values unconditionally. Hence we save the parameters we

Nit:		its

> + * need to retain across reinitialisation, and rewrite them into the VFS inode
> + * after resetting it's state even if resetting fails.

... and above as well.

That aside, it looks like xfs_reinit_inode() addresses my previous
questions around reclaimable inodes. Could we introduce this function
properly when it is first needed (the nlink patch) and update it
appropriately?

Brian

> + */
> +static int
> +xfs_reinit_inode(
> +	struct xfs_mount	*mp,
> +	struct inode		*inode)
> +{
> +	uint32_t	nlink = inode->i_nlink;
> +	umode_t		mode = inode->i_mode;
> +	uint32_t	generation = inode->i_generation;
> +	uint64_t	version = inode->i_version;
> +	int		error;
> +
> +	error = inode_init_always(mp->m_super, inode);
> +
> +	set_nlink(inode, nlink);
> +	inode->i_mode = mode;
> +	inode->i_generation = generation;
> +	inode->i_version = version;
> +
> +	return error;
> +}
> +
> +/*
>   * Check the validity of the inode we just found it the cache
>   */
>  static int
> @@ -185,7 +217,7 @@ xfs_iget_cache_hit(
>  	/*
>  	 * If lookup is racing with unlink return an error immediately.
>  	 */
> -	if (ip->i_d.di_mode == 0 && !(flags & XFS_IGET_CREATE)) {
> +	if (VFS_I(ip)->i_mode == 0 && !(flags & XFS_IGET_CREATE)) {
>  		error = -ENOENT;
>  		goto out_error;
>  	}
> @@ -208,7 +240,7 @@ xfs_iget_cache_hit(
>  		spin_unlock(&ip->i_flags_lock);
>  		rcu_read_unlock();
>  
> -		error = inode_init_always(mp->m_super, inode);
> +		error = xfs_reinit_inode(mp, inode);
>  		if (error) {
>  			/*
>  			 * Re-initializing the inode failed, and we are in deep
> @@ -295,7 +327,7 @@ xfs_iget_cache_miss(
>  
>  	trace_xfs_iget_miss(ip);
>  
> -	if ((ip->i_d.di_mode == 0) && !(flags & XFS_IGET_CREATE)) {
> +	if ((VFS_I(ip)->i_mode == 0) && !(flags & XFS_IGET_CREATE)) {
>  		error = -ENOENT;
>  		goto out_destroy;
>  	}
> @@ -444,7 +476,7 @@ again:
>  	 * If we have a real type for an on-disk inode, we can setup the inode
>  	 * now.	 If it's a new inode being created, xfs_ialloc will handle it.
>  	 */
> -	if (xfs_iflags_test(ip, XFS_INEW) && ip->i_d.di_mode != 0)
> +	if (xfs_iflags_test(ip, XFS_INEW) && VFS_I(ip)->i_mode != 0)
>  		xfs_setup_existing_inode(ip);
>  	return 0;
>  
> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> index 96aee23..a0e7cdf 100644
> --- a/fs/xfs/xfs_inode.c
> +++ b/fs/xfs/xfs_inode.c
> @@ -793,7 +793,7 @@ xfs_ialloc(
>  	if (ip->i_d.di_version == 1)
>  		ip->i_d.di_version = 2;
>  
> -	ip->i_d.di_mode = mode;
> +	inode->i_mode = mode;
>  	set_nlink(inode, nlink);
>  	ip->i_d.di_uid = xfs_kuid_to_uid(current_fsuid());
>  	ip->i_d.di_gid = xfs_kgid_to_gid(current_fsgid());
> @@ -801,9 +801,8 @@ xfs_ialloc(
>  
>  	if (pip && XFS_INHERIT_GID(pip)) {
>  		ip->i_d.di_gid = pip->i_d.di_gid;
> -		if ((pip->i_d.di_mode & S_ISGID) && S_ISDIR(mode)) {
> -			ip->i_d.di_mode |= S_ISGID;
> -		}
> +		if ((VFS_I(pip)->i_mode & S_ISGID) && S_ISDIR(mode))
> +			inode->i_mode |= S_ISGID;
>  	}
>  
>  	/*
> @@ -812,10 +811,9 @@ xfs_ialloc(
>  	 * (and only if the irix_sgid_inherit compatibility variable is set).
>  	 */
>  	if ((irix_sgid_inherit) &&
> -	    (ip->i_d.di_mode & S_ISGID) &&
> -	    (!in_group_p(xfs_gid_to_kgid(ip->i_d.di_gid)))) {
> -		ip->i_d.di_mode &= ~S_ISGID;
> -	}
> +	    (inode->i_mode & S_ISGID) &&
> +	    (!in_group_p(xfs_gid_to_kgid(ip->i_d.di_gid))))
> +		inode->i_mode &= ~S_ISGID;
>  
>  	ip->i_d.di_size = 0;
>  	ip->i_d.di_nextents = 0;
> @@ -1407,7 +1405,7 @@ xfs_link(
>  
>  	trace_xfs_link(tdp, target_name);
>  
> -	ASSERT(!S_ISDIR(sip->i_d.di_mode));
> +	ASSERT(!S_ISDIR(VFS_I(sip)->i_mode));
>  
>  	if (XFS_FORCED_SHUTDOWN(mp))
>  		return -EIO;
> @@ -1614,7 +1612,7 @@ xfs_release(
>  	xfs_mount_t	*mp = ip->i_mount;
>  	int		error;
>  
> -	if (!S_ISREG(ip->i_d.di_mode) || (ip->i_d.di_mode == 0))
> +	if (!S_ISREG(VFS_I(ip)->i_mode) || (VFS_I(ip)->i_mode == 0))
>  		return 0;
>  
>  	/* If this is a read-only mount, don't do this (would generate I/O) */
> @@ -1849,7 +1847,7 @@ xfs_inactive(
>  	 * If the inode is already free, then there can be nothing
>  	 * to clean up here.
>  	 */
> -	if (ip->i_d.di_mode == 0) {
> +	if (VFS_I(ip)->i_mode == 0) {
>  		ASSERT(ip->i_df.if_real_bytes == 0);
>  		ASSERT(ip->i_df.if_broot_bytes == 0);
>  		return;
> @@ -1873,7 +1871,7 @@ xfs_inactive(
>  		return;
>  	}
>  
> -	if (S_ISREG(ip->i_d.di_mode) &&
> +	if (S_ISREG(VFS_I(ip)->i_mode) &&
>  	    (ip->i_d.di_size != 0 || XFS_ISIZE(ip) != 0 ||
>  	     ip->i_d.di_nextents > 0 || ip->i_delayed_blks > 0))
>  		truncate = 1;
> @@ -1882,7 +1880,7 @@ xfs_inactive(
>  	if (error)
>  		return;
>  
> -	if (S_ISLNK(ip->i_d.di_mode))
> +	if (S_ISLNK(VFS_I(ip)->i_mode))
>  		error = xfs_inactive_symlink(ip);
>  	else if (truncate)
>  		error = xfs_inactive_truncate(ip);
> @@ -1944,7 +1942,7 @@ xfs_iunlink(
>  	int		error;
>  
>  	ASSERT(VFS_I(ip)->i_nlink == 0 || ignore_linkcount);
> -	ASSERT(ip->i_d.di_mode != 0);
> +	ASSERT(VFS_I(ip)->i_mode != 0);
>  
>  	mp = tp->t_mountp;
>  
> @@ -2387,7 +2385,7 @@ xfs_ifree(
>  	ASSERT(VFS_I(ip)->i_nlink == 0);
>  	ASSERT(ip->i_d.di_nextents == 0);
>  	ASSERT(ip->i_d.di_anextents == 0);
> -	ASSERT(ip->i_d.di_size == 0 || !S_ISREG(ip->i_d.di_mode));
> +	ASSERT(ip->i_d.di_size == 0 || !S_ISREG(VFS_I(ip)->i_mode));
>  	ASSERT(ip->i_d.di_nblocks == 0);
>  
>  	/*
> @@ -2401,7 +2399,7 @@ xfs_ifree(
>  	if (error)
>  		return error;
>  
> -	ip->i_d.di_mode = 0;		/* mark incore inode as free */
> +	VFS_I(ip)->i_mode = 0;		/* mark incore inode as free */
>  	ip->i_d.di_flags = 0;
>  	ip->i_d.di_dmevmask = 0;
>  	ip->i_d.di_forkoff = 0;		/* mark the attr fork not in use */
> @@ -2498,7 +2496,7 @@ xfs_remove(
>  {
>  	xfs_mount_t		*mp = dp->i_mount;
>  	xfs_trans_t             *tp = NULL;
> -	int			is_dir = S_ISDIR(ip->i_d.di_mode);
> +	int			is_dir = S_ISDIR(VFS_I(ip)->i_mode);
>  	int                     error = 0;
>  	xfs_bmap_free_t         free_list;
>  	xfs_fsblock_t           first_block;
> @@ -2743,7 +2741,7 @@ xfs_cross_rename(
>  	if (dp1 != dp2) {
>  		dp2_flags = XFS_ICHGTIME_MOD | XFS_ICHGTIME_CHG;
>  
> -		if (S_ISDIR(ip2->i_d.di_mode)) {
> +		if (S_ISDIR(VFS_I(ip2)->i_mode)) {
>  			error = xfs_dir_replace(tp, ip2, &xfs_name_dotdot,
>  						dp1->i_ino, first_block,
>  						free_list, spaceres);
> @@ -2751,7 +2749,7 @@ xfs_cross_rename(
>  				goto out_trans_abort;
>  
>  			/* transfer ip2 ".." reference to dp1 */
> -			if (!S_ISDIR(ip1->i_d.di_mode)) {
> +			if (!S_ISDIR(VFS_I(ip1)->i_mode)) {
>  				error = xfs_droplink(tp, dp2);
>  				if (error)
>  					goto out_trans_abort;
> @@ -2770,7 +2768,7 @@ xfs_cross_rename(
>  			ip2_flags |= XFS_ICHGTIME_MOD | XFS_ICHGTIME_CHG;
>  		}
>  
> -		if (S_ISDIR(ip1->i_d.di_mode)) {
> +		if (S_ISDIR(VFS_I(ip1)->i_mode)) {
>  			error = xfs_dir_replace(tp, ip1, &xfs_name_dotdot,
>  						dp2->i_ino, first_block,
>  						free_list, spaceres);
> @@ -2778,7 +2776,7 @@ xfs_cross_rename(
>  				goto out_trans_abort;
>  
>  			/* transfer ip1 ".." reference to dp2 */
> -			if (!S_ISDIR(ip2->i_d.di_mode)) {
> +			if (!S_ISDIR(VFS_I(ip2)->i_mode)) {
>  				error = xfs_droplink(tp, dp1);
>  				if (error)
>  					goto out_trans_abort;
> @@ -2875,7 +2873,7 @@ xfs_rename(
>  	struct xfs_inode	*inodes[__XFS_SORT_INODES];
>  	int			num_inodes = __XFS_SORT_INODES;
>  	bool			new_parent = (src_dp != target_dp);
> -	bool			src_is_directory = S_ISDIR(src_ip->i_d.di_mode);
> +	bool			src_is_directory = S_ISDIR(VFS_I(src_ip)->i_mode);
>  	int			spaceres;
>  	int			error;
>  
> @@ -3004,7 +3002,7 @@ xfs_rename(
>  		 * target and source are directories and that target can be
>  		 * destroyed, or that neither is a directory.
>  		 */
> -		if (S_ISDIR(target_ip->i_d.di_mode)) {
> +		if (S_ISDIR(VFS_I(target_ip)->i_mode)) {
>  			/*
>  			 * Make sure target dir is empty.
>  			 */
> @@ -3434,7 +3432,7 @@ xfs_iflush_int(
>  			__func__, ip->i_ino, be16_to_cpu(dip->di_magic), dip);
>  		goto corrupt_out;
>  	}
> -	if (S_ISREG(ip->i_d.di_mode)) {
> +	if (S_ISREG(VFS_I(ip)->i_mode)) {
>  		if (XFS_TEST_ERROR(
>  		    (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS) &&
>  		    (ip->i_d.di_format != XFS_DINODE_FMT_BTREE),
> @@ -3444,7 +3442,7 @@ xfs_iflush_int(
>  				__func__, ip->i_ino, ip);
>  			goto corrupt_out;
>  		}
> -	} else if (S_ISDIR(ip->i_d.di_mode)) {
> +	} else if (S_ISDIR(VFS_I(ip)->i_mode)) {
>  		if (XFS_TEST_ERROR(
>  		    (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS) &&
>  		    (ip->i_d.di_format != XFS_DINODE_FMT_BTREE) &&
> diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
> index e74d13d..d627749 100644
> --- a/fs/xfs/xfs_inode.h
> +++ b/fs/xfs/xfs_inode.h
> @@ -88,7 +88,7 @@ static inline struct inode *VFS_I(struct xfs_inode *ip)
>   */
>  static inline xfs_fsize_t XFS_ISIZE(struct xfs_inode *ip)
>  {
> -	if (S_ISREG(ip->i_d.di_mode))
> +	if (S_ISREG(VFS_I(ip)->i_mode))
>  		return i_size_read(VFS_I(ip));
>  	return ip->i_d.di_size;
>  }
> @@ -369,7 +369,7 @@ static inline int xfs_isiflocked(struct xfs_inode *ip)
>   */
>  #define XFS_INHERIT_GID(pip)	\
>  	(((pip)->i_mount->m_flags & XFS_MOUNT_GRPID) || \
> -	 ((pip)->i_d.di_mode & S_ISGID))
> +	 (VFS_I(pip)->i_mode & S_ISGID))
>  
>  int		xfs_release(struct xfs_inode *ip);
>  void		xfs_inactive(struct xfs_inode *ip);
> diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
> index 334d657..bd9808f 100644
> --- a/fs/xfs/xfs_inode_item.c
> +++ b/fs/xfs/xfs_inode_item.c
> @@ -333,7 +333,6 @@ xfs_inode_to_log_dinode(
>  
>  	to->di_magic = XFS_DINODE_MAGIC;
>  
> -	to->di_mode = from->di_mode;
>  	to->di_version = from->di_version;
>  	to->di_format = from->di_format;
>  	to->di_uid = from->di_uid;
> @@ -351,6 +350,7 @@ xfs_inode_to_log_dinode(
>  	to->di_ctime.t_nsec = inode->i_ctime.tv_nsec;
>  	to->di_nlink = inode->i_nlink;
>  	to->di_gen = inode->i_generation;
> +	to->di_mode = inode->i_mode;
>  
>  	to->di_size = from->di_size;
>  	to->di_nblocks = from->di_nblocks;
> diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
> index 2c40c5f..5069bd8 100644
> --- a/fs/xfs/xfs_ioctl.c
> +++ b/fs/xfs/xfs_ioctl.c
> @@ -962,7 +962,7 @@ xfs_set_diflags(
>  		di_flags |= XFS_DIFLAG_NODEFRAG;
>  	if (xflags & XFS_XFLAG_FILESTREAM)
>  		di_flags |= XFS_DIFLAG_FILESTREAM;
> -	if (S_ISDIR(ip->i_d.di_mode)) {
> +	if (S_ISDIR(VFS_I(ip)->i_mode)) {
>  		if (xflags & XFS_XFLAG_RTINHERIT)
>  			di_flags |= XFS_DIFLAG_RTINHERIT;
>  		if (xflags & XFS_XFLAG_NOSYMLINKS)
> @@ -971,7 +971,7 @@ xfs_set_diflags(
>  			di_flags |= XFS_DIFLAG_EXTSZINHERIT;
>  		if (xflags & XFS_XFLAG_PROJINHERIT)
>  			di_flags |= XFS_DIFLAG_PROJINHERIT;
> -	} else if (S_ISREG(ip->i_d.di_mode)) {
> +	} else if (S_ISREG(VFS_I(ip)->i_mode)) {
>  		if (xflags & XFS_XFLAG_REALTIME)
>  			di_flags |= XFS_DIFLAG_REALTIME;
>  		if (xflags & XFS_XFLAG_EXTSIZE)
> @@ -1112,14 +1112,14 @@ xfs_ioctl_setattr_check_extsize(
>  {
>  	struct xfs_mount	*mp = ip->i_mount;
>  
> -	if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) && !S_ISREG(ip->i_d.di_mode))
> +	if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) && !S_ISREG(VFS_I(ip)->i_mode))
>  		return -EINVAL;
>  
>  	if ((fa->fsx_xflags & XFS_XFLAG_EXTSZINHERIT) &&
> -	    !S_ISDIR(ip->i_d.di_mode))
> +	    !S_ISDIR(VFS_I(ip)->i_mode))
>  		return -EINVAL;
>  
> -	if (S_ISREG(ip->i_d.di_mode) && ip->i_d.di_nextents &&
> +	if (S_ISREG(VFS_I(ip)->i_mode) && ip->i_d.di_nextents &&
>  	    ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) != fa->fsx_extsize))
>  		return -EINVAL;
>  
> @@ -1240,9 +1240,9 @@ xfs_ioctl_setattr(
>  	 * successful return from chown()
>  	 */
>  
> -	if ((ip->i_d.di_mode & (S_ISUID|S_ISGID)) &&
> +	if ((VFS_I(ip)->i_mode & (S_ISUID|S_ISGID)) &&
>  	    !capable_wrt_inode_uidgid(VFS_I(ip), CAP_FSETID))
> -		ip->i_d.di_mode &= ~(S_ISUID|S_ISGID);
> +		VFS_I(ip)->i_mode &= ~(S_ISUID|S_ISGID);
>  
>  	/* Change the ownerships and register project quota modifications */
>  	if (xfs_get_projid(ip) != fa->fsx_projid) {
> diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
> index 397ce85..34bff1e 100644
> --- a/fs/xfs/xfs_iops.c
> +++ b/fs/xfs/xfs_iops.c
> @@ -454,7 +454,7 @@ xfs_vn_getattr(
>  
>  	stat->size = XFS_ISIZE(ip);
>  	stat->dev = inode->i_sb->s_dev;
> -	stat->mode = ip->i_d.di_mode;
> +	stat->mode = inode->i_mode;
>  	stat->nlink = inode->i_nlink;
>  	stat->uid = inode->i_uid;
>  	stat->gid = inode->i_gid;
> @@ -501,9 +501,6 @@ xfs_setattr_mode(
>  
>  	ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL));
>  
> -	ip->i_d.di_mode &= S_IFMT;
> -	ip->i_d.di_mode |= mode & ~S_IFMT;
> -
>  	inode->i_mode &= S_IFMT;
>  	inode->i_mode |= mode & ~S_IFMT;
>  }
> @@ -647,9 +644,9 @@ xfs_setattr_nonsize(
>  		 * The set-user-ID and set-group-ID bits of a file will be
>  		 * cleared upon successful return from chown()
>  		 */
> -		if ((ip->i_d.di_mode & (S_ISUID|S_ISGID)) &&
> +		if ((inode->i_mode & (S_ISUID|S_ISGID)) &&
>  		    !capable(CAP_FSETID))
> -			ip->i_d.di_mode &= ~(S_ISUID|S_ISGID);
> +			inode->i_mode &= ~(S_ISUID|S_ISGID);
>  
>  		/*
>  		 * Change the ownerships and register quota modifications
> @@ -759,7 +756,7 @@ xfs_setattr_size(
>  
>  	ASSERT(xfs_isilocked(ip, XFS_IOLOCK_EXCL));
>  	ASSERT(xfs_isilocked(ip, XFS_MMAPLOCK_EXCL));
> -	ASSERT(S_ISREG(ip->i_d.di_mode));
> +	ASSERT(S_ISREG(inode->i_mode));
>  	ASSERT((iattr->ia_valid & (ATTR_UID|ATTR_GID|ATTR_ATIME|ATTR_ATIME_SET|
>  		ATTR_MTIME_SET|ATTR_KILL_PRIV|ATTR_TIMES_SET)) == 0);
>  
> @@ -1211,7 +1208,6 @@ xfs_setup_inode(
>  	/* make the inode look hashed for the writeback code */
>  	hlist_add_fake(&inode->i_hash);
>  
> -	inode->i_mode	= ip->i_d.di_mode;
>  	inode->i_uid    = xfs_uid_to_kuid(ip->i_d.di_uid);
>  	inode->i_gid    = xfs_gid_to_kgid(ip->i_d.di_gid);
>  
> diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
> index 6162e65..ce73eb3 100644
> --- a/fs/xfs/xfs_itable.c
> +++ b/fs/xfs/xfs_itable.c
> @@ -88,7 +88,6 @@ xfs_bulkstat_one_int(
>  	buf->bs_projid_lo = dic->di_projid_lo;
>  	buf->bs_projid_hi = dic->di_projid_hi;
>  	buf->bs_ino = ino;
> -	buf->bs_mode = dic->di_mode;
>  	buf->bs_uid = dic->di_uid;
>  	buf->bs_gid = dic->di_gid;
>  	buf->bs_size = dic->di_size;
> @@ -101,6 +100,7 @@ xfs_bulkstat_one_int(
>  	buf->bs_ctime.tv_sec = inode->i_ctime.tv_sec;
>  	buf->bs_ctime.tv_nsec = inode->i_ctime.tv_nsec;
>  	buf->bs_gen = inode->i_generation;
> +	buf->bs_mode = inode->i_mode;
>  
>  	buf->bs_xflags = xfs_ip2xflags(ip);
>  	buf->bs_extsize = dic->di_extsize << mp->m_sb.sb_blocklog;
> diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
> index 611c25c..bd6f23b 100644
> --- a/fs/xfs/xfs_log_recover.c
> +++ b/fs/xfs/xfs_log_recover.c
> @@ -4338,7 +4338,7 @@ xlog_recover_process_one_iunlink(
>  		goto fail_iput;
>  
>  	ASSERT(VFS_I(ip)->i_nlink == 0);
> -	ASSERT(ip->i_d.di_mode != 0);
> +	ASSERT(VFS_I(ip)->i_mode != 0);
>  
>  	/* setup for the next pass */
>  	agino = be32_to_cpu(dip->di_next_unlinked);
> diff --git a/fs/xfs/xfs_mount.c b/fs/xfs/xfs_mount.c
> index bb753b3..d306105 100644
> --- a/fs/xfs/xfs_mount.c
> +++ b/fs/xfs/xfs_mount.c
> @@ -865,7 +865,7 @@ xfs_mountfs(
>  
>  	ASSERT(rip != NULL);
>  
> -	if (unlikely(!S_ISDIR(rip->i_d.di_mode))) {
> +	if (unlikely(!S_ISDIR(VFS_I(rip)->i_mode))) {
>  		xfs_warn(mp, "corrupted root inode %llu: not a directory",
>  			(unsigned long long)rip->i_ino);
>  		xfs_iunlock(rip, XFS_ILOCK_EXCL);
> -- 
> 2.5.0
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From mjseger@gmail.com  Mon Jan 25 13:00:57 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,
	T_DKIM_INVALID autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 552DD7CA1
	for ; Mon, 25 Jan 2016 13:00:57 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 21AE18F8033
	for ; Mon, 25 Jan 2016 11:00:57 -0800 (PST)
X-ASG-Debug-ID: 1453748453-04bdf04d6bbc600001-NocioJ
Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by cuda.sgi.com with ESMTP id ilTl78pjlf86qIsi (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 25 Jan 2016 11:00:53 -0800 (PST)
X-Barracuda-Envelope-From: mjseger@gmail.com
X-Barracuda-Apparent-Source-IP: 74.125.82.41
Received: by mail-wm0-f41.google.com with SMTP id 123so78174347wmz.0
        for ; Mon, 25 Jan 2016 11:00:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        bh=6UlFYtSARIaU9QDhxE3FWL0M1qVyEYgGPUVY6rPD2ro=;
        b=N9fTwEZLxxhpOKQnrouYfW0dJk7k7aN6o4lFEC8jFn2UK4KKGuthQTiMN4wBMlPf1o
         BqaBoP1Xk//4MnBwQmRtpL4T316+dfasnx6XJQHZnAQ2i/gO/U28yGTRsk/OzMj1esCR
         mnSdSjDnz06FzFRoYyg2dA0GZ/W5wFFr3ipw3zhPkKTdGYTrrLZXog/hQi3bjtTC30ep
         5oIX1vtD4Lev4CKAogZXbbOTqmpOAk71tr84o+LhX2Pk4rIEx8OEg+j1w+WWT7r3Xv8T
         HJN+00v5rXywxppggI1kPMXgC23w6Pzj1SZptVt8Tb1V9uRD8YPR3Mi57p8stuDkj4BZ
         ZEPg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=6UlFYtSARIaU9QDhxE3FWL0M1qVyEYgGPUVY6rPD2ro=;
        b=CsTkeldts1SGHk3vF0Xlsr8Eo3947UJUxQVFl4rBanMi/bA0G6xOVWn4NpS2QYR+X6
         /GekIs6ypgpjGs0abCduJYPWBhhOx+OSSe49NqIcgK6wc7E28DF+LGqLtYQmNvTE+ecy
         Mq1ycPzPj4MWLfrDfsHpg6bReqaulGNS+10wrLkCE8Hm+AlnRedDGMbBbK9XbwgBJp0f
         ICGvBPCbI4CnavWZ6OUzNxN9y51HcZxQFpVo32V5tp/++xrfhXaJ9rW9r6cp5VVKTw5r
         ypxCN4KMIRJjU6+VTP+xW1uQPIocR7db6yTPtdHb5+xLIKtP3BGhSD03Q/8UGjnuqwf/
         M0pw==
X-Gm-Message-State: AG10YORGxA4D9Na8E0I+ZoyQuJulnIjWQQi9HoXCLPbmGbqcNO6cnQ8t3XmshJd4elIVqFz1O735qrph4tk5mw==
MIME-Version: 1.0
X-Received: by 10.28.0.132 with SMTP id 126mr21072095wma.67.1453748452702;
 Mon, 25 Jan 2016 11:00:52 -0800 (PST)
Received: by 10.194.78.99 with HTTP; Mon, 25 Jan 2016 11:00:52 -0800 (PST)
In-Reply-To: <56A66869.3080506@ddn.com>
References: 
	<56A66869.3080506@ddn.com>
Date: Mon, 25 Jan 2016 14:00:52 -0500
Message-ID: 
Subject: Re: xfs and swift
From: Mark Seger 
X-ASG-Orig-Subj: Re: xfs and swift
To: Bernd Schubert 
Cc: Linux fs XFS , Laurence Oberman 
Content-Type: multipart/alternative; boundary=001a113c808c1dc745052a2d30e1
X-Barracuda-Connect: mail-wm0-f41.google.com[74.125.82.41]
X-Barracuda-Start-Time: 1453748453
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26440
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	0.00 HTML_MESSAGE           BODY: HTML included in message

--001a113c808c1dc745052a2d30e1
Content-Type: text/plain; charset=UTF-8

hey bernd, long time no chat.  it turns out you don't have to know what
swift is because I've been able to demonstrate this behavior with a very
simple python script that simply creates files in a 3-tier hierarchy.  the
third level directories each contain a single file which for my testing are
all 1K.

I have played wiht cache_pressure and it doesn't seem to make a difference,
though that was awhlle ago and perhaps it is worth revisiting. one thing
you may get a hoot out of, being a collectl user, is I have an xfs plugin
that lets you look at a ton of xfs stats either in realtime or after the
fact just like any other collectl stat.  I just havent' added it to the kit
yet.

-mark

On Mon, Jan 25, 2016 at 1:24 PM, Bernd Schubert  wrote:

> Hi Mark!
>
> On 01/06/2016 04:15 PM, Mark Seger wrote:
> > I've recently found the performance our development swift system is
> > degrading over time as the number of objects/files increases.  This is a
> > relatively small system, each server has 3 400GB disks.  The system I'm
> > currently looking at has about 70GB tied up in slabs alone, close to 55GB
> > in xfs inodes and ili, and about 2GB free.  The kernel
> > is 3.14.57-1-amd64-hlinux.
> >
> > Here's the way the filesystems are mounted:
> >
> > /dev/sdb1 on /srv/node/disk0 type xfs
> >
> (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=1536,noquota)
> >
> > I can do about 2000 1K file creates/sec when running 2 minute PUT tests
> at
> > 100 threads.  If I repeat that tests for multiple hours, I see the number
> > of IOPS steadily decreasing to about 770 and the very next run it drops
> to
> > 260 and continues to fall from there.  This happens at about 12M files.
> >
> > The directory structure is 2 tiered, with 1000 directories per tier so we
> > can have about 1M of them, though they don't currently all exist.
>
> This sounds pretty much like hash directories as used by some parallel
> file systems (Lustre and in the past BeeGFS). For us the file create
> slow down was due to lookup in directories if a file with the same name
> already exists. At least for ext4 it was rather easy to demonstrate that
> simply caching directory blocks would eliminate that issue.
> We then considered working on a better kernel cache, but in the end
> simply found a way to get rid of such a simple directory structure in
> BeeGFS and changed it to a more complex layout, but with less random
> access and so we could eliminate the main reason for the slow down.
>
> Now I have no idea what a "swift system" is and in which order it
> creates and accesses those files and if it would be possible to change
> the access pattern. One thing you might try and which should work much
> better since 3.11 is the vfs_cache_pressure setting. The lower it is the
> less dentries/inodes are dropped from cache when pages are needed for
> file data.
>
>
>
> Cheers,
> Bernd

--001a113c808c1dc745052a2d30e1
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable


hey bernd, long time no chat. =C2=A0it turns out you don&#=
39;t have to know what swift is because I've been able to demonstrate t=
his behavior with a very simple python script that simply creates files in =
a 3-tier hierarchy. =C2=A0the third level directories each contain a single=
 file which for my testing are all 1K.

I have played wih=
t cache_pressure and it doesn't seem to make a difference, though that =
was awhlle ago and perhaps it is worth revisiting. one thing you may get a =
hoot out of, being a collectl user, is I have an xfs plugin that lets you l=
ook at a ton of xfs stats either in realtime or after the fact just like an=
y other collectl stat.=C2=A0 I just havent' added it to the kit yet.

-mark

On Mon, Jan 25, 2016 at 1:24 PM, Bernd Schubert <bs=
chubert@ddn.com> wrote:
Hi =
Mark!



On 01/06/2016 04:15 PM, Mark Seger wrote:

> I've recently found the performance our development swift system i=
s

> degrading over time as the number of objects/files increases.=C2=A0 Th=
is is a

> relatively small system, each server has 3 400GB disks.=C2=A0 The syst=
em I'm

> currently looking at has about 70GB tied up in slabs alone, close to 5=
5GB

> in xfs inodes and ili, and about 2GB free.=C2=A0 The kernel

> is 3.14.57-1-amd64-hlinux.

>

> Here's the way the filesystems are mounted:

>

> /dev/sdb1 on /srv/node/disk0 type xfs

> (rw,noatime,nodiratime,attr2,nobarrier,inode64,logbufs=3D8,logbsize=3D=
256k,sunit=3D512,swidth=3D1536,noquota)

>

> I can do about 2000 1K file creates/sec when running 2 minute PUT test=
s at

> 100 threads.=C2=A0 If I repeat that tests for multiple hours, I see th=
e number

> of IOPS steadily decreasing to about 770 and the very next run it drop=
s to

> 260 and continues to fall from there.=C2=A0 This happens at about 12M =
files.

>

> The directory structure is 2 tiered, with 1000 directories per tier so=
 we

> can have about 1M of them, though they don't currently all exist.<=
br>


This sounds pretty much like hash directories as used by some parall=
el

file systems (Lustre and in the past BeeGFS). For us the file create

slow down was due to lookup in directories if a file with the same name

already exists. At least for ext4 it was rather easy to demonstrate that
simply caching directory blocks would eliminate that issue.

We then considered working on a better kernel cache, but in the end

simply found a way to get rid of such a simple directory structure in

BeeGFS and changed it to a more complex layout, but with less random

access and so we could eliminate the main reason for the slow down.



Now I have no idea what a "swift system" is and in which order it=


creates and accesses those files and if it would be possible to change

the access pattern. One thing you might try and which should work much

better since 3.11 is the vfs_cache_pressure setting. The lower it is the
less dentries/inodes are dropped from cache when pages are needed for

file data.







Cheers,

Bernd



--001a113c808c1dc745052a2d30e1--

From bschubert@ddn.com  Mon Jan 25 13:33:24 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 9CB2E7CA1
	for ; Mon, 25 Jan 2016 13:33:24 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 5B3D28F8039
	for ; Mon, 25 Jan 2016 11:33:21 -0800 (PST)
X-ASG-Debug-ID: 1453750395-04bdf04d6bbd2b0001-NocioJ
Received: from legacy.ddn.com (legacy.ddn.com [64.47.133.206]) by cuda.sgi.com with ESMTP id B0o0yMoj2cxzkElK (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 25 Jan 2016 11:33:15 -0800 (PST)
X-Barracuda-Envelope-From: bschubert@ddn.com
X-Barracuda-Apparent-Source-IP: 64.47.133.206
Received: from LAX-EX-MB2.datadirect.datadirectnet.com
 ([fe80::96:2379:f1b2:ef2d]) by LAX-EX-CAHT2.datadirect.datadirectnet.com
 ([fe80::a499:249:3d8b:f1a4%12]) with mapi id 14.03.0224.002; Mon, 25 Jan 2016
 11:33:14 -0800
From: Bernd Schubert 
To: Mark Seger 
CC: Laurence Oberman , Linux fs XFS 
Subject: Re: xfs and swift
Thread-Topic: xfs and swift
X-ASG-Orig-Subj: Re: xfs and swift
Thread-Index: AQHRV52oAMBQxHEMt0OfT2+g6ZBqCZ8NHEwAgAAJCQA=
Date: Mon, 25 Jan 2016 19:33:13 +0000
Message-ID: <56A67878.3080803@ddn.com>
References: 
 <56A66869.3080506@ddn.com>
 
In-Reply-To: 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.4.0
x-originating-ip: [10.64.108.27]
Content-Type: text/plain; charset="utf-8"
Content-ID: <2BC931E86097324EAB665FE219A1FE1C@ddn.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Barracuda-Connect: legacy.ddn.com[64.47.133.206]
X-Barracuda-Start-Time: 1453750395
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.02
X-Barracuda-Spam-Status: No, SCORE=0.02 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, THREAD_INDEX, THREAD_TOPIC
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26441
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.01 THREAD_INDEX           thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig==
	0.01 THREAD_TOPIC           Thread-Topic: ...(Japanese Subject)...
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

SGkgTWFyayENCg0KT24gMDEvMjUvMjAxNiAwODowMCBQTSwgTWFyayBTZWdlciB3cm90ZToNCj4g
aGV5IGJlcm5kLCBsb25nIHRpbWUgbm8gY2hhdC4gIGl0IHR1cm5zIG91dCB5b3UgZG9uJ3QgaGF2
ZSB0byBrbm93IHdoYXQNCj4gc3dpZnQgaXMgYmVjYXVzZSBJJ3ZlIGJlZW4gYWJsZSB0byBkZW1v
bnN0cmF0ZSB0aGlzIGJlaGF2aW9yIHdpdGggYSB2ZXJ5DQo+IHNpbXBsZSBweXRob24gc2NyaXB0
IHRoYXQgc2ltcGx5IGNyZWF0ZXMgZmlsZXMgaW4gYSAzLXRpZXIgaGllcmFyY2h5LiAgdGhlDQo+
IHRoaXJkIGxldmVsIGRpcmVjdG9yaWVzIGVhY2ggY29udGFpbiBhIHNpbmdsZSBmaWxlIHdoaWNo
IGZvciBteSB0ZXN0aW5nIGFyZQ0KPiBhbGwgMUsuDQoNClNvIHdoYXQgaXMgdGhlIHNjcmlwdCBl
eGFjdGx5IGRvaW5nPyBEb2VzIGl0IGNyZWF0ZSB0aG9zZSBmaWxlcw0Kc2VxdWVudGlhbGx5IHBl
ciBkaXIgb3IgcmFuZG9tbHkgYmV0d2VlbiB0aG9zZSBkaXJzPw0KDQpCdHcsIEkgaGFkIGJlZW4g
dGFsa2luZyBhYm91dCB0aGF0IGlzc3VlIGF0IGxpbnV4IHBsdW1iZXJzIGluIDIwMTMNCg0KaHR0
cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1OX2JaT0daQWItWQ0KDQo+IA0KPiBJIGhhdmUg
cGxheWVkIHdpaHQgY2FjaGVfcHJlc3N1cmUgYW5kIGl0IGRvZXNuJ3Qgc2VlbSB0byBtYWtlIGEg
ZGlmZmVyZW5jZSwNCj4gdGhvdWdoIHRoYXQgd2FzIGF3aGxsZSBhZ28gYW5kIHBlcmhhcHMgaXQg
aXMgd29ydGggcmV2aXNpdGluZy4gb25lIHRoaW5nDQoNClRoZXJlIGFyZSBzZXZlcmFsIHBhdGNo
ZXMgZnJvbSBNZWwgR29ybWFuIGluIDMuMTEsIHdoaWNoIHJlYWxseSBtYWRlIGENCmRpZmZlcmVu
Y2UgZm9yIG1lLiBTbyB1bmxlc3MgeW91IHRlc3RlZCB3aXRoID49IDMuMTEgeW91IHNob3VsZCBw
cm9iYWJseQ0KcmUtdGVzdC4NCg0KPiB5b3UgbWF5IGdldCBhIGhvb3Qgb3V0IG9mLCBiZWluZyBh
IGNvbGxlY3RsIHVzZXIsIGlzIEkgaGF2ZSBhbiB4ZnMgcGx1Z2luDQo+IHRoYXQgbGV0cyB5b3Ug
bG9vayBhdCBhIHRvbiBvZiB4ZnMgc3RhdHMgZWl0aGVyIGluIHJlYWx0aW1lIG9yIGFmdGVyIHRo
ZQ0KPiBmYWN0IGp1c3QgbGlrZSBhbnkgb3RoZXIgY29sbGVjdGwgc3RhdC4gIEkganVzdCBoYXZl
bnQnIGFkZGVkIGl0IHRvIHRoZSBraXQNCj4geWV0Lg0KDQoNCkhtbSwgSSBjdXJyZW50bHkgZG9u
J3QgaGF2ZSBnb29kIGEgdGVzdCBzeXN0ZW0gZm9yIHRoYXQuIEknbSB3b3JraW5nIG9uDQphbiBl
bnRpcmVseSBkaWZmZXJlbnQgcHJvamVjdCBub3cgYW5kIHdoaWxlIHRoaXMgaXMgYWxzbyBhIHBh
cmFsbGVsIGZpbGUNCnN5c3RlbSwgaXQgZG9lcyBub3QgaGF2ZSBhIGxpbnV4IGZpbGUgc3lzdGVt
IGluIGJldHdlZW4sIGJ1dCBoYXMgaXRzIG93bg0KKGxvZyByb3RhdGVkKSBsYXlvdXQuDQoNCg0K
Q2hlZXJzLA0KQmVybmQ=

From willy@linux.intel.com  Mon Jan 25 14:46:53 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 737C77CA1
	for ; Mon, 25 Jan 2016 14:46:53 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id DA2A9AC002
	for ; Mon, 25 Jan 2016 12:46:52 -0800 (PST)
X-ASG-Debug-ID: 1453754810-04cb6c0d5dc8c50001-NocioJ
Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by cuda.sgi.com with ESMTP id 3Gvq8uzKiV7ErzRb for ; Mon, 25 Jan 2016 12:46:50 -0800 (PST)
X-Barracuda-Envelope-From: willy@linux.intel.com
X-Barracuda-Apparent-Source-IP: 134.134.136.20
Received: from fmsmga004.fm.intel.com ([10.253.24.48])
  by orsmga101.jf.intel.com with ESMTP; 25 Jan 2016 12:46:39 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,346,1449561600"; 
   d="scan'208";a="35994909"
Received: from agu1-mobl2.amr.corp.intel.com (HELO thog.int.wil.cx) ([10.254.78.135])
  by fmsmga004.fm.intel.com with SMTP; 25 Jan 2016 12:46:37 -0800
Received: by thog.int.wil.cx (Postfix, from userid 1000)
	id 419105F93D; Mon, 25 Jan 2016 15:46:36 -0500 (EST)
Date: Mon, 25 Jan 2016 15:46:36 -0500
From: Matthew Wilcox 
To: Dave Chinner 
Cc: Ross Zwisler ,
	linux-kernel@vger.kernel.org, Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams , Jan Kara ,
	linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-nvdimm@lists.01.org, xfs@oss.sgi.com
Subject: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-ID: <20160125204636.GI2948@linux.intel.com>
X-ASG-Orig-Subj: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
References: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
 <20160124220107.GI20456@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160124220107.GI20456@dastard>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mga02.intel.com[134.134.136.20]
X-Barracuda-Start-Time: 1453754810
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26443
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Mon, Jan 25, 2016 at 09:01:07AM +1100, Dave Chinner wrote:
> On Fri, Jan 22, 2016 at 04:06:11PM -0700, Ross Zwisler wrote:
> > With the current DAX code the following race exists:
> > 
> > Process 1                	Process 2
> > ---------			---------
> > 
> > __dax_fault() - read file f, index 0
> >   get_block() -> returns hole
> >                              	__dax_fault() - write file f, index 0
> >                                   get_block() -> allocates blocks
> >                                   dax_insert_mapping()
> >   dax_load_hole()
> >   *data corruption*
> > 
> > An analogous race exists between __dax_fault() loading a hole and
> > __dax_pmd_fault() allocating a PMD DAX page and trying to insert it, and
> > that race also ends in data corruption.
> 
> Ok, so why doesn't this problem exist for the normal page cache
> insertion case with concurrent read vs write faults?  It's because
> the write fault first does a read fault and so always the write
> fault always has a page in the radix tree for the get_block call
> that allocates the extents, right?

No, it's because allocation of blocks is separated from allocation of
struct page.

> And DAX has an optimisation in the page fault part where it skips
> the read fault part of the write fault?  And so essentially the DAX
> write fault is missing the object (page lock of page in the radix
> tree) that the non-DAX write fault uses to avoid this problem?
>
> What happens if we get rid of that DAX write fault optimisation that
> skips the initial read fault? The write fault will always run on a
> mapping that has a hole loaded, right?, so the race between
> dax_load_hole() and dax_insert_mapping() goes away, because nothing
> will be calling dax_load_hole() once the write fault is allocating
> blocks....

So in your proposal, we'd look in the radix tree, find nothing,
call get_block(..., 0).  If we get something back, we can insert it.
If we hit a hole, we allocate a struct page, put it in the radix tree
and return to user space.  If that was a write fault after all, it'll
come back to us through the ->page_mkwrite handler where we can take the
page lock on the allocated struct page, then call down to DAX which calls
back through get_block to allocate?  Then DAX kicks the struct page out
of the page cache and frees it.

That seems to work to me.  And we can get rid of pfn_mkwrite at the same
time which seems like a win to me.

From jack@suse.cz  Tue Jan 26 02:46:50 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 92D1E7CA2
	for ; Tue, 26 Jan 2016 02:46:50 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 7231A8F8033
	for ; Tue, 26 Jan 2016 00:46:47 -0800 (PST)
X-ASG-Debug-ID: 1453798004-04cb6c0d60d7800001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id DTaozrfhVl6GrNnN (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 26 Jan 2016 00:46:45 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 9756AAAC1;
	Tue, 26 Jan 2016 08:46:41 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 145A9823DA; Tue, 26 Jan 2016 09:46:54 +0100 (CET)
Date: Tue, 26 Jan 2016 09:46:54 +0100
From: Jan Kara 
To: Matthew Wilcox 
Cc: Dave Chinner ,
	Ross Zwisler ,
	linux-kernel@vger.kernel.org, Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams , Jan Kara ,
	linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-nvdimm@lists.01.org, xfs@oss.sgi.com
Subject: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-ID: <20160126084654.GT24938@quack.suse.cz>
X-ASG-Orig-Subj: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
References: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
 <20160124220107.GI20456@dastard>
 <20160125204636.GI2948@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160125204636.GI2948@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453798004
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26459
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Mon 25-01-16 15:46:36, Matthew Wilcox wrote:
> On Mon, Jan 25, 2016 at 09:01:07AM +1100, Dave Chinner wrote:
> > On Fri, Jan 22, 2016 at 04:06:11PM -0700, Ross Zwisler wrote:
> > > With the current DAX code the following race exists:
> > > 
> > > Process 1                	Process 2
> > > ---------			---------
> > > 
> > > __dax_fault() - read file f, index 0
> > >   get_block() -> returns hole
> > >                              	__dax_fault() - write file f, index 0
> > >                                   get_block() -> allocates blocks
> > >                                   dax_insert_mapping()
> > >   dax_load_hole()
> > >   *data corruption*
> > > 
> > > An analogous race exists between __dax_fault() loading a hole and
> > > __dax_pmd_fault() allocating a PMD DAX page and trying to insert it, and
> > > that race also ends in data corruption.
> > 
> > Ok, so why doesn't this problem exist for the normal page cache
> > insertion case with concurrent read vs write faults?  It's because
> > the write fault first does a read fault and so always the write
> > fault always has a page in the radix tree for the get_block call
> > that allocates the extents, right?
> 
> No, it's because allocation of blocks is separated from allocation of
> struct page.
> 
> > And DAX has an optimisation in the page fault part where it skips
> > the read fault part of the write fault?  And so essentially the DAX
> > write fault is missing the object (page lock of page in the radix
> > tree) that the non-DAX write fault uses to avoid this problem?
> >
> > What happens if we get rid of that DAX write fault optimisation that
> > skips the initial read fault? The write fault will always run on a
> > mapping that has a hole loaded, right?, so the race between
> > dax_load_hole() and dax_insert_mapping() goes away, because nothing
> > will be calling dax_load_hole() once the write fault is allocating
> > blocks....
> 
> So in your proposal, we'd look in the radix tree, find nothing,
> call get_block(..., 0).  If we get something back, we can insert it.
> If we hit a hole, we allocate a struct page, put it in the radix tree
> and return to user space.  If that was a write fault after all, it'll
> come back to us through the ->page_mkwrite handler where we can take the
> page lock on the allocated struct page, then call down to DAX which calls
> back through get_block to allocate?  Then DAX kicks the struct page out
> of the page cache and frees it.
> 
> That seems to work to me.  And we can get rid of pfn_mkwrite at the same
> time which seems like a win to me.

Getting rid of pfn_mkwrite() would be nice, I agree. But the above scheme
still has issues when PMD pages come into play. Or maybe to start from the
beginning: How would you like PMD faults to work? Because there is no
obvious 'struct page' to protect the allocation of 2MB worth of blocks. You
could resort to similar tricks like transparent huge pages do (compound
pages) but then the cure is IMHO worse than the disease.

There is one option: No allocation of blocks for PMD faults. That would
make code much simpler and solve the races but I'm not sure whether that is
really an acceptable loss of functionality...

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From willy@linux.intel.com  Tue Jan 26 06:48:17 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 9B8C27CA2
	for ; Tue, 26 Jan 2016 06:48:17 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 7A5048F8033
	for ; Tue, 26 Jan 2016 04:48:17 -0800 (PST)
X-ASG-Debug-ID: 1453812495-04cb6c0d5fdc120001-NocioJ
Received: from mga14.intel.com ([192.55.52.115]) by cuda.sgi.com with ESMTP id FIwWWnbpXHvyQeIK for ; Tue, 26 Jan 2016 04:48:15 -0800 (PST)
X-Barracuda-Envelope-From: willy@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.115
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
  by fmsmga103.fm.intel.com with ESMTP; 26 Jan 2016 04:48:16 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,350,1449561600"; 
   d="scan'208";a="901357524"
Received: from agu1-mobl2.amr.corp.intel.com (HELO thog.int.wil.cx) ([10.254.78.135])
  by fmsmga002.fm.intel.com with SMTP; 26 Jan 2016 04:48:13 -0800
Received: by thog.int.wil.cx (Postfix, from userid 1000)
	id 309265F96A; Tue, 26 Jan 2016 07:48:12 -0500 (EST)
Date: Tue, 26 Jan 2016 07:48:12 -0500
From: Matthew Wilcox 
To: Jan Kara 
Cc: Dave Chinner ,
	Ross Zwisler ,
	linux-kernel@vger.kernel.org, Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams , Jan Kara ,
	linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-nvdimm@lists.01.org, xfs@oss.sgi.com
Subject: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-ID: <20160126124812.GJ2948@linux.intel.com>
X-ASG-Orig-Subj: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
References: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
 <20160124220107.GI20456@dastard>
 <20160125135921.GE24938@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160125135921.GE24938@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.115]
X-Barracuda-Start-Time: 1453812495
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26463
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Mon, Jan 25, 2016 at 02:59:21PM +0100, Jan Kara wrote:
> On Mon 25-01-16 09:01:07, Dave Chinner wrote:
> > What happens if we get rid of that DAX write fault optimisation that
> > skips the initial read fault? The write fault will always run on a
> > mapping that has a hole loaded, right?, so the race between
> > dax_load_hole() and dax_insert_mapping() goes away, because nothing
> > will be calling dax_load_hole() once the write fault is allocating
> > blocks....
> 
> So frankly I don't like mixing of page locks into the DAX fault locking.
> Also your scheme would require more tricks to deal with races between PMD
> write faults racing with PTE read faults since you don't want to require
> 2MB worth of hole-pages to be able to do a PMD write fault. Transparent
> huge pages deal with this issue using compound pages but I'd like to avoid
> that horror in the DAX path...

I *think* that what Dave's proposing (and if he isn't, I'm proposing it
for him) is that the filesystem takes its allocation lock shared during
the ->fault handler, then in the ->page_mkwrite handler, it knows that an
allocation is coming, so it takes its allocation lock in exclusive mode.

So read vs write faults won't be able to race because the allocation lock
will prevent it.

From jack@suse.cz  Tue Jan 26 06:57:03 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 547327CA2
	for ; Tue, 26 Jan 2016 06:57:03 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 322AB304059
	for ; Tue, 26 Jan 2016 04:57:03 -0800 (PST)
X-ASG-Debug-ID: 1453813020-04cbb026fcd84f0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 3HQiuBO9z51e2SJG (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 26 Jan 2016 04:57:01 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id A972FABF8;
	Tue, 26 Jan 2016 12:56:58 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 92EC1823DA; Tue, 26 Jan 2016 13:57:10 +0100 (CET)
Date: Tue, 26 Jan 2016 13:57:10 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Message-ID: <20160126125710.GA23820@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-3-git-send-email-sandeen@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1453487136-12681-3-git-send-email-sandeen@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453813020
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26463
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri 22-01-16 12:25:31, Eric Sandeen wrote:
> Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
> will return quota information for the id equal to or greater
> than the id requested.  In other words, if the requested id has
> no quota, the command will return quota information for the
> next higher id which does have a quota set.  If no higher id
> has an active quota, -ESRCH is returned.

Actually, is -ESRCH the right return value? It seems XFS traditionally
returns -ENOENT when id doesn't exist. So that would look more logical to
me.

								Honza

> This allows filesystems to do efficient iteration in kernelspace,
> much like extN filesystems do in userspace when asked to report
> all active quotas.
> 
> The patch adds a d_id field to struct qc_dqblk so that we can
> pass back the id of the quota which was found, and return it
> to userspace.
> 
> Today, filesystems such as XFS require getpwent-style iterations,
> and for systems which have i.e. LDAP backends, this can be very
> slow, or even impossible if iteration is not allowed in the
> configuration.
> 
> Signed-off-by: Eric Sandeen 
> ---
>  fs/quota/quota.c               |   31 +++++++++++++++++++++++++++++++
>  include/linux/quota.h          |    2 ++
>  include/uapi/linux/dqblk_xfs.h |    1 +
>  3 files changed, 34 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/quota/quota.c b/fs/quota/quota.c
> index ea66670..0a6dd71 100644
> --- a/fs/quota/quota.c
> +++ b/fs/quota/quota.c
> @@ -625,6 +625,34 @@ static int quota_getxquota(struct super_block *sb, int type, qid_t id,
>  	return ret;
>  }
>  
> +/*
> + * Return quota for next active quota >= this id, if any exists,
> + * otherwise return -ESRCH via ->get_nextdqblk.
> + */
> +static int quota_getnextxquota(struct super_block *sb, int type, qid_t id,
> +			    void __user *addr)
> +{
> +	struct fs_disk_quota fdq;
> +	struct qc_dqblk qdq;
> +	struct kqid qid;
> +	qid_t id_out;
> +	int ret;
> +
> +	if (!sb->s_qcop->get_nextdqblk)
> +		return -ENOSYS;
> +	qid = make_kqid(current_user_ns(), type, id);
> +	if (!qid_valid(qid))
> +		return -EINVAL;
> +	ret = sb->s_qcop->get_nextdqblk(sb, &qid, &qdq);
> +	if (ret)
> +		return ret;
> +	id_out = from_kqid(current_user_ns(), qid);
> +	copy_to_xfs_dqblk(&fdq, &qdq, type, id_out);
> +	if (copy_to_user(addr, &fdq, sizeof(fdq)))
> +		return -EFAULT;
> +	return ret;
> +}
> +
>  static int quota_rmxquota(struct super_block *sb, void __user *addr)
>  {
>  	__u32 flags;
> @@ -690,6 +718,8 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
>  		return quota_setxquota(sb, type, id, addr);
>  	case Q_XGETQUOTA:
>  		return quota_getxquota(sb, type, id, addr);
> +	case Q_XGETNEXTQUOTA:
> +		return quota_getnextxquota(sb, type, id, addr);
>  	case Q_XQUOTASYNC:
>  		if (sb->s_flags & MS_RDONLY)
>  			return -EROFS;
> @@ -712,6 +742,7 @@ static int quotactl_cmd_write(int cmd)
>  	case Q_XGETQSTAT:
>  	case Q_XGETQSTATV:
>  	case Q_XGETQUOTA:
> +	case Q_XGETNEXTQUOTA:
>  	case Q_XQUOTASYNC:
>  		return 0;
>  	}
> diff --git a/include/linux/quota.h b/include/linux/quota.h
> index b2505ac..fba92f5 100644
> --- a/include/linux/quota.h
> +++ b/include/linux/quota.h
> @@ -425,6 +425,8 @@ struct quotactl_ops {
>  	int (*quota_sync)(struct super_block *, int);
>  	int (*set_info)(struct super_block *, int, struct qc_info *);
>  	int (*get_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *);
> +	int (*get_nextdqblk)(struct super_block *, struct kqid *,
> +			     struct qc_dqblk *);
>  	int (*set_dqblk)(struct super_block *, struct kqid, struct qc_dqblk *);
>  	int (*get_state)(struct super_block *, struct qc_state *);
>  	int (*rm_xquota)(struct super_block *, unsigned int);
> diff --git a/include/uapi/linux/dqblk_xfs.h b/include/uapi/linux/dqblk_xfs.h
> index dcd75cc..11b3b31 100644
> --- a/include/uapi/linux/dqblk_xfs.h
> +++ b/include/uapi/linux/dqblk_xfs.h
> @@ -39,6 +39,7 @@
>  #define Q_XQUOTARM	XQM_CMD(6)	/* free disk space used by dquots */
>  #define Q_XQUOTASYNC	XQM_CMD(7)	/* delalloc flush, updates dquots */
>  #define Q_XGETQSTATV	XQM_CMD(8)	/* newer version of get quota */
> +#define Q_XGETNEXTQUOTA	XQM_CMD(9)	/* get disk limits and usage >= ID */
>  
>  /*
>   * fs_disk_quota structure:
> -- 
> 1.7.1
> 
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Tue Jan 26 07:05:21 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 68F8E7CA2
	for ; Tue, 26 Jan 2016 07:05:21 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 585BD304032
	for ; Tue, 26 Jan 2016 05:05:18 -0800 (PST)
X-ASG-Debug-ID: 1453813511-04cb6c0d60dc790001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id j04Km81LKq7Dz0uE (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 26 Jan 2016 05:05:12 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id B0FB7AC2E;
	Tue, 26 Jan 2016 13:05:08 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id E33A2823DA; Tue, 26 Jan 2016 14:05:21 +0100 (CET)
Date: Tue, 26 Jan 2016 14:05:21 +0100
From: Jan Kara 
To: Matthew Wilcox 
Cc: Jan Kara , Dave Chinner ,
	Ross Zwisler ,
	linux-kernel@vger.kernel.org, Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams , Jan Kara ,
	linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-nvdimm@lists.01.org, xfs@oss.sgi.com
Subject: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-ID: <20160126130521.GB23820@quack.suse.cz>
X-ASG-Orig-Subj: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
References: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
 <20160124220107.GI20456@dastard>
 <20160125135921.GE24938@quack.suse.cz>
 <20160126124812.GJ2948@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160126124812.GJ2948@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453813511
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26463
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue 26-01-16 07:48:12, Matthew Wilcox wrote:
> On Mon, Jan 25, 2016 at 02:59:21PM +0100, Jan Kara wrote:
> > On Mon 25-01-16 09:01:07, Dave Chinner wrote:
> > > What happens if we get rid of that DAX write fault optimisation that
> > > skips the initial read fault? The write fault will always run on a
> > > mapping that has a hole loaded, right?, so the race between
> > > dax_load_hole() and dax_insert_mapping() goes away, because nothing
> > > will be calling dax_load_hole() once the write fault is allocating
> > > blocks....
> > 
> > So frankly I don't like mixing of page locks into the DAX fault locking.
> > Also your scheme would require more tricks to deal with races between PMD
> > write faults racing with PTE read faults since you don't want to require
> > 2MB worth of hole-pages to be able to do a PMD write fault. Transparent
> > huge pages deal with this issue using compound pages but I'd like to avoid
> > that horror in the DAX path...
> 
> I *think* that what Dave's proposing (and if he isn't, I'm proposing it
> for him) is that the filesystem takes its allocation lock shared during
> the ->fault handler, then in the ->page_mkwrite handler, it knows that an
> allocation is coming, so it takes its allocation lock in exclusive mode.
> 
> So read vs write faults won't be able to race because the allocation lock
> will prevent it.

So this is correct and clean design but we will take the lock in exclusive
mode (and thus hurt scalability) for every write fault, not just for the
ones allocating blocks. And at the moment we take exclusive lock for write
faults, there's no more need for having the hole page instantiated - we can
still do it for simplicity but it's no longer necessary to avoid data
corruption.

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From jack@suse.cz  Tue Jan 26 07:10:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 8802B7CA2
	for ; Tue, 26 Jan 2016 07:10:05 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 75F588F8039
	for ; Tue, 26 Jan 2016 05:10:02 -0800 (PST)
X-ASG-Debug-ID: 1453813799-04cb6c0d5edc9b0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id 3y3AAE5JV9aGnHSV (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 26 Jan 2016 05:10:00 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 2FB21ACCC;
	Tue, 26 Jan 2016 13:09:58 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 83377823DA; Tue, 26 Jan 2016 14:10:11 +0100 (CET)
Date: Tue, 26 Jan 2016 14:10:11 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com, jack@suse.cz
Subject: Re: [PATCH 0/7 V3] quota: add new quotactl Q_GETNEXTQUOTA
Message-ID: <20160126131011.GC23820@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 0/7 V3] quota: add new quotactl Q_GETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="J2SCkAp4GZ/dPZZf"
Content-Disposition: inline
In-Reply-To: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453813800
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26463
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header


--J2SCkAp4GZ/dPZZf
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Just for reference attached are two patches that implement support for
Q_GETNEXTQUOTA and Q_XGETNEXTQUOTA in quota-tools. I have used it for
testing XFS and VFS infrastructure for these new quotactls.

								Honza
On Fri 22-01-16 12:25:29, Eric Sandeen wrote:
> This adds a new quotactl, Q_GETNEXTQUOTA.
> 
> Q_GETNEXTQUOTA is exactly like Q_GETQUOTA, except that it will
> return quota information for the id equal to or greater than
> the id requested.  In other words, if the specified id has
> no quota, the command will return quota information for the
> next higher id which does have a quota set.  If no higher id
> has an active quota, -ESRCH is returned.
> 
> So if you ask for id X, you can get back quota for id X,
> id X+N, or -ESRCH if no higher id has a quota.
> 
> This allows filesystems to do efficient iteration in kernelspace,
> much like extN filesystems do in userspace when asked to report
> all active quotas.
> 
> Today, filesystems such as XFS require getpwent()-style iterations,
> and for systems which have i.e. LDAP backends, this can be very
> slow, or even impossible if iteration is not allowed in the
> configuration.
> 
> Patches 1 and 4 are just small fixups that turned up along the way;
> 2 and 3 add the actual quota plumbing, and the rest are xfs-specific
> to allow xfs to support this new interface.
> 
> For non-xfs quota, this does require a new structure which is
> able to pass back the discovered ID along with the quota info.
> For xfs-quota, the id is already present in the structure.
> 
> V3:
> * Remove 32-bit compat stuff (i686/x86_64 at least works w/o it...)
> * Require CAP_SYS_ADMIN for these calls
> * Pass back found ID in &qid passed to ->get_nextdqblk, rather
>   than modifying struct qc_dqblk
> * Munge that found ID back through user-namespace conversions
>   before returning it in the user structure.
> 
> Thanks,
> -Eric
-- 
Jan Kara 
SUSE Labs, CR

--J2SCkAp4GZ/dPZZf
Content-Type: text/x-patch; charset=us-ascii
Content-Disposition: attachment; filename="0001-Scan-dquots-using-Q_GETNEXTQUOTA.patch"

>From 9ae91376373b1b0a8d4573834d0d99722a5909f6 Mon Sep 17 00:00:00 2001
From: Jan Kara 
Date: Tue, 26 Jan 2016 13:10:59 +0100
Subject: [PATCH 1/2] Scan dquots using Q_GETNEXTQUOTA

Check for new kernel quotactl Q_GETNEXTQUOTA and if available use it for
scanning all dquot structures.

Signed-off-by: Jan Kara 
---
 quota.h           | 14 ++++++++++++++
 quotaio_generic.c | 34 ++++++++++++++++++++++++++++++++++
 quotaio_generic.h |  4 ++++
 quotaio_meta.c    | 14 +++++++++++++-
 4 files changed, 65 insertions(+), 1 deletion(-)

diff --git a/quota.h b/quota.h
index 0c3842774e6f..0607e04b1f02 100644
--- a/quota.h
+++ b/quota.h
@@ -63,6 +63,7 @@ typedef int64_t qsize_t;	/* Type in which we store size limitations */
 #define Q_SETINFO  0x800006	/* set information about quota files */
 #define Q_GETQUOTA 0x800007	/* get user quota structure */
 #define Q_SETQUOTA 0x800008	/* set user quota structure */
+#define Q_GETNEXTQUOTA 0x800009	/* get disk limits and usage >= ID */
 
 /*
  * Quota structure used for communication with userspace via quotactl
@@ -91,6 +92,19 @@ struct if_dqblk {
 	u_int32_t dqb_valid;
 };
 
+struct if_nextdqblk {
+	u_int64_t dqb_bhardlimit;
+	u_int64_t dqb_bsoftlimit;
+	u_int64_t dqb_curspace;
+	u_int64_t dqb_ihardlimit;
+	u_int64_t dqb_isoftlimit;
+	u_int64_t dqb_curinodes;
+	u_int64_t dqb_btime;
+	u_int64_t dqb_itime;
+	u_int32_t dqb_valid;
+	u_int32_t dqb_id;
+};
+
 /*
  * Structure used for setting quota information about file via quotactl
  * Following flags are used to specify which fields are valid
diff --git a/quotaio_generic.c b/quotaio_generic.c
index 5001a56f8a6b..06b16e791624 100644
--- a/quotaio_generic.c
+++ b/quotaio_generic.c
@@ -161,3 +161,37 @@ int generic_scan_dquots(struct quota_handle *h,
 	free(dquot);
 	return ret;
 }
+
+int vfs_scan_dquots(struct quota_handle *h,
+		    int (*process_dquot)(struct dquot *dquot, char *dqname))
+{
+	struct dquot *dquot = get_empty_dquot();
+	qid_t id = 0;
+	struct if_nextdqblk kdqblk;
+	int ret;
+
+	dquot->dq_h = h;
+	while (1) {
+		ret = quotactl(QCMD(Q_GETNEXTQUOTA, h->qh_type),
+			       h->qh_quotadev, id, (void *)&kdqblk);
+		if (ret < 0)
+			break;
+
+		/*
+		 * This is a slight hack but we know struct if_dqblk is a
+		 * subset of struct if_nextdqblk
+		 */
+		generic_kern2utildqblk(&dquot->dq_dqb,
+				       (struct if_dqblk *)&kdqblk);
+		dquot->dq_id = kdqblk.dqb_id;
+		ret = process_dquot(dquot, NULL);
+		if (ret < 0)
+			break;
+		id = kdqblk.dqb_id + 1;
+	}
+	free(dquot);
+
+	if (errno == ESRCH)
+		return 0;
+	return ret;
+}
diff --git a/quotaio_generic.h b/quotaio_generic.h
index 5edc11cd947e..a7930f0214bd 100644
--- a/quotaio_generic.h
+++ b/quotaio_generic.h
@@ -27,4 +27,8 @@ int generic_scan_dquots(struct quota_handle *h,
 			int (*process_dquot)(struct dquot *dquot, char *dqname),
 			int (*get_dquot)(struct dquot *dquot));
 
+/* Scan all dquots using kernel quotactl to get existing ids */
+int vfs_scan_dquots(struct quota_handle *h,
+		    int (*process_dquot)(struct dquot *dquot, char *dqname));
+
 #endif
diff --git a/quotaio_meta.c b/quotaio_meta.c
index e52b4f4322f1..ad6ff7ab2299 100644
--- a/quotaio_meta.c
+++ b/quotaio_meta.c
@@ -8,6 +8,7 @@
 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -55,7 +56,18 @@ static int meta_commit_dquot(struct dquot *dquot, int flags)
 
 static int meta_scan_dquots(struct quota_handle *h, int (*process_dquot)(struct dquot *dquot, char *dqname))
 {
-	return generic_scan_dquots(h, process_dquot, vfs_get_dquot);
+	struct if_nextdqblk kdqblk;
+	int ret;
+
+	ret = quotactl(QCMD(Q_GETNEXTQUOTA, h->qh_type), h->qh_quotadev, 0,
+		       (void *)&kdqblk);
+	/*
+	 * Fall back to scanning using passwd if Q_GETNEXTQUOTA is not
+	 * supported
+	 */
+	if (ret < 0 && (errno == ENOSYS || errno == EINVAL))
+		return generic_scan_dquots(h, process_dquot, vfs_get_dquot);
+	return vfs_scan_dquots(h, process_dquot);
 }
 
 struct quotafile_ops quotafile_ops_meta = {
-- 
2.6.2


--J2SCkAp4GZ/dPZZf
Content-Type: text/x-patch; charset=us-ascii
Content-Disposition: attachment; filename="0002-Add-support-for-scanning-using-Q_XGETNEXTQUOTA.patch"

>From 304bed24716fdbb7c636ea8df0462d4f979f23a0 Mon Sep 17 00:00:00 2001
From: Jan Kara 
Date: Tue, 26 Jan 2016 14:06:59 +0100
Subject: [PATCH 2/2] Add support for scanning using Q_XGETNEXTQUOTA

Add support for scanning of all available quota structures using
Q_XGETNEXTQUOTA quotactl.

Signed-off-by: Jan Kara 
---
 quotaio_xfs.c | 42 +++++++++++++++++++++++++++++++++++++++---
 quotaio_xfs.h |  1 +
 2 files changed, 40 insertions(+), 3 deletions(-)

diff --git a/quotaio_xfs.c b/quotaio_xfs.c
index 903c03e6d3f8..14646411dbab 100644
--- a/quotaio_xfs.c
+++ b/quotaio_xfs.c
@@ -191,15 +191,51 @@ static int xfs_get_dquot(struct dquot *dq)
 	return 0;
 }
 
+static int xfs_kernel_scan_dquots(struct quota_handle *h,
+		int (*process_dquot)(struct dquot *dquot, char *dqname))
+{
+	struct dquot *dquot = get_empty_dquot();
+	qid_t id = 0;
+	struct xfs_kern_dqblk xdqblk;
+	int ret;
+
+	dquot->dq_h = h;
+	while (1) {
+		ret = quotactl(QCMD(Q_XGETNEXTQUOTA, h->qh_type),
+			       h->qh_quotadev, id, (void *)&xdqblk);
+		if (ret < 0)
+			break;
+
+		xfs_kern2utildqblk(&dquot->dq_dqb, &xdqblk);
+		dquot->dq_id = xdqblk.d_id;
+		ret = process_dquot(dquot, NULL);
+		if (ret < 0)
+			break;
+		id = xdqblk.d_id + 1;
+	}
+	free(dquot);
+
+	if (errno == ESRCH)
+		return 0;
+	return ret;
+}
+
 /*
  *	Scan all known dquots and call callback on each
  */
 static int xfs_scan_dquots(struct quota_handle *h, int (*process_dquot) (struct dquot *dquot, char *dqname))
 {
-	if (!XFS_USRQUOTA(h) && !XFS_GRPQUOTA(h))
-		return 0;
+	int ret;
+	struct xfs_kern_dqblk xdqblk;
 
-	return generic_scan_dquots(h, process_dquot, xfs_get_dquot);
+	ret = quotactl(QCMD(Q_XGETNEXTQUOTA, h->qh_type), h->qh_quotadev, 0,
+		       (void *)&xdqblk);
+	if (ret < 0 && (errno == ENOSYS || errno == EINVAL)) {
+		if (!XFS_USRQUOTA(h) && !XFS_GRPQUOTA(h))
+			return 0;
+		return generic_scan_dquots(h, process_dquot, xfs_get_dquot);
+	}
+	return xfs_kernel_scan_dquots(h, process_dquot);
 }
 
 /*
diff --git a/quotaio_xfs.h b/quotaio_xfs.h
index 54725b044d63..2236da48f832 100644
--- a/quotaio_xfs.h
+++ b/quotaio_xfs.h
@@ -46,6 +46,7 @@
 #define Q_XSETQLIM   XQM_CMD(0x4)	/* set disk limits only */
 #define Q_XGETQSTAT  XQM_CMD(0x5)	/* returns fs_quota_stat_t struct */
 #define Q_XQUOTARM   XQM_CMD(0x6)	/* free quota files' space */
+#define Q_XGETNEXTQUOTA	XQM_CMD(0x9)	/* get disk limits and usage >= ID */
 
 /*
  * fs_disk_quota structure:
-- 
2.6.2


--J2SCkAp4GZ/dPZZf--

From bfoster@redhat.com  Tue Jan 26 08:17:42 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 710077CA2
	for ; Tue, 26 Jan 2016 08:17:42 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 05BD0AC001
	for ; Tue, 26 Jan 2016 06:17:38 -0800 (PST)
X-ASG-Debug-ID: 1453817856-04bdf04d6bd2ea0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id RfGySsANObzJD6Gf (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 06:17:36 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id DD2978F28D;
	Tue, 26 Jan 2016 14:17:35 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0QEHZ1C021883;
	Tue, 26 Jan 2016 09:17:35 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 465D21251C3; Tue, 26 Jan 2016 09:17:34 -0500 (EST)
Date: Tue, 26 Jan 2016 09:17:34 -0500
From: Brian Foster 
To: Dave Chinner 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH v2] [RFC] xfs: allocate log vector buffers outside CIL
 context lock
Message-ID: <20160126141733.GA48264@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH v2] [RFC] xfs: allocate log vector buffers outside CIL
 context lock
References: <1453177919-17849-1-git-send-email-david@fromorbit.com>
 <20160120015853.GU6033@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160120015853.GU6033@dastard>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453817856
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 20, 2016 at 12:58:53PM +1100, Dave Chinner wrote:
> From: Dave Chinner 
> 
> One of the problems we currently have with delayed logging is that
> under serious memory pressure we can deadlock memory reclaim. THis
> occurs when memory reclaim (such as run by kswapd) is reclaiming XFS
> inodes and issues a log force to unpin inodes that are dirty in the
> CIL.
> 
> The CIL is pushed, but this will only occur once it gets the CIL
> context lock to ensure that all committing transactions are complete
> and no new transactions start being committed to the CIL while the
> push switches to a new context.
> 
> The deadlock occurs when the CIL context lock is held by a
> committing process that is doing memory allocation for log vector
> buffers, and that allocation is then blocked on memory reclaim
> making progress. Memory reclaim, however, is blocked waiting for
> a log force to make progress, and so we effectively deadlock at this
> point.
> 
> To solve this problem, we have to move the CIL log vector buffer
> allocation outside of the context lock so that memory reclaim can
> always make progress when it needs to force the log. The problem
> with doing this is that a CIL push can take place while we are
> determining if we need to allocate a new log vector buffer for
> an item and hence the current log vector may go away without
> warning. That means we canot rely on the existing log vector being
> present when we finally grab the context lock and so we must have a
> replacement buffer ready to go at all times.
> 
> To ensure this, introduce a "shadow log vector" buffer that is
> always guaranteed to be present when we gain the CIL context lock
> and format the item. This shadow buffer may or may not be used
> during the formatting, but if the log item does not have an existing
> log vector buffer or that buffer is too small for the new
> modifications, we swap it for the new shadow buffer and format
> the modifications into that new log vector buffer.
> 
> The result of this is that for any object we modify more than once
> in a given CIL checkpoint, we double the memory required
> to track dirty regions in the log. For single modifications then
> we consume the shadow log vectorwe allocate on commit, and that gets
> consumed by the checkpoint. However, if we make multiple
> modifications, then the second transaction commit will allocate a
> shadow log vector and hence we will end up with double the memory
> usage as only one of the log vectors is consumed by the CIL
> checkpoint. The remaining shadow vector will be freed when th elog
> item is freed.
> 
> This can probably be optimised - access to the shadow log vector is
> serialised by the object lock (as opposited to the active log
> vector, which is controlled by the CIL context lock) and so we can
> probably free shadow log vector from some objects when the log item
> is marked clean on removal from the AIL.
> 
> The patch survives smoke testing and some load testing. I haven't
> done any real performance testing, but I have done some load and low
> memory testing and it hasn't exploded (perf did - it failed several
> order 2 memory allocations, which XFS continued along just fine).
> 
> That said, I don't have a reliable deadlock reproducer in the first
> place, so I'm interested i hearing what people think about this
> approach to solve the problem and ways to test and improve it.
> 
> Signed-off-by: Dave Chinner 
> ---

This seems reasonable to me in principle. It would be nice to have some
kind of feedback in terms of effectiveness resolving the original
deadlock report. I can't think of a good way of testing short of
actually instrumenting the deadlock one way or another, unfortunately.
Was there a user that might be willing to test or had a detailed enough
description of the workload/environment?

I also wonder whether the lazy shadow vector freeing is noticeably
effective in terms of performance. After all, it seems like it's going
to be freed anyways in the event that a log force occurs due to memory
pressure. Perhaps the logic is to optimize the case where relogging
occurs without memory pressure in the picture..? In that case, the
question is probably whether the lazy allocation actually contributes to
memory pressure in any significant way. Perhaps one or more new stat
counters might be useful to demonstrate that, but that could be
overboard as well if performance testing doesn't show any major
regressions. That said, something that demonstrates lazy freeing is
actually taken advantage of might be useful, even if just for a data
point in the commit log description. By that I mean, how do we know in
practice that the current code isn't always doing reallocations anyways
due to size requirements, in which case lazy freeing could just be
unnecessary complexity? Or that we end up holding on to larger than
necessary buffers for reuse and thus trade off allocation calls for an
even worse memory footprint..? Etc.

Just some thoughts. One additional comment below...

> 
> Version 2:
> - this one doesn't crash and burn in generic/324
> - fixed handling of order items when recycling shadow buffers
> - correctly set up log iovec pointers in all cases
> - fixed moving current log vector back to the shadow vector when
>   they are switched during formatting.
> 
>  fs/xfs/xfs_buf_item.c     |   1 +
>  fs/xfs/xfs_dquot.c        |   1 +
>  fs/xfs/xfs_dquot_item.c   |   2 +
>  fs/xfs/xfs_extfree_item.c |   2 +
>  fs/xfs/xfs_inode_item.c   |   1 +
>  fs/xfs/xfs_log_cil.c      | 249 ++++++++++++++++++++++++++++++++++------------
>  fs/xfs/xfs_trans.h        |   1 +
>  7 files changed, 193 insertions(+), 64 deletions(-)
> 
...
> diff --git a/fs/xfs/xfs_log_cil.c b/fs/xfs/xfs_log_cil.c
> index 4e76493..f5567fb 100644
> --- a/fs/xfs/xfs_log_cil.c
> +++ b/fs/xfs/xfs_log_cil.c
> @@ -79,6 +79,148 @@ xlog_cil_init_post_recovery(
>  	log->l_cilp->xc_ctx->sequence = 1;
>  }
>  
...
> +static void
> +xlog_cil_alloc_shadow_bufs(
> +	struct xlog		*log,
> +	struct xfs_trans	*tp)
> +{
> +	struct xfs_log_item_desc *lidp;
> +
> +	list_for_each_entry(lidp, &tp->t_items, lid_trans) {
> +		struct xfs_log_item *lip = lidp->lid_item;
> +		struct xfs_log_vec *lv;
> +		int	niovecs = 0;
> +		int	nbytes = 0;
> +		int	buf_size;
> +		bool	ordered = false;
> +
> +		/* Skip items which aren't dirty in this transaction. */
> +		if (!(lidp->lid_flags & XFS_LID_DIRTY))
> +			continue;
> +
> +		/* get number of vecs and size of data to be stored */
> +		lip->li_ops->iop_size(lip, &niovecs, &nbytes);
> +

I'm not sure how relevant it is, but there was also a !niovecs check
here in xlog_cil_insert_format_items().

Brian

> +		/*
> +		 * Ordered items need to be tracked but we do not wish to write
> +		 * them. We need a logvec to track the object, but we do not
> +		 * need an iovec or buffer to be allocated for copying data.
> +		 */
> +		if (niovecs == XFS_LOG_VEC_ORDERED) {
> +			ordered = true;
> +			niovecs = 0;
> +			nbytes = 0;
> +		}
> +
> +		/*
> +		 * We 64-bit align the length of each iovec so that the start
> +		 * of the next one is naturally aligned.  We'll need to
> +		 * account for that slack space here. Then round nbytes up
> +		 * to 64-bit alignment so that the initial buffer alignment is
> +		 * easy to calculate and verify.
> +		 */
> +		nbytes += niovecs * sizeof(uint64_t);
> +		nbytes = round_up(nbytes, sizeof(uint64_t));
> +
> +		/*
> +		 * The data buffer needs to start 64-bit aligned, so round up
> +		 * that space to ensure we can align it appropriately and not
> +		 * overrun the buffer.
> +		 */
> +		buf_size = nbytes + xlog_cil_iovec_space(niovecs);
> +
> +		/*
> +		 * if we have no shadow buffer, or it is too small, we need to
> +		 * reallocate it.
> +		 */
> +		if (!lip->li_lv_shadow ||
> +		    buf_size > lip->li_lv_shadow->lv_size) {
> +
> +			kmem_free(lip->li_lv_shadow);
> +
> +			lv = kmem_zalloc(buf_size, KM_SLEEP|KM_NOFS);
> +			lv->lv_item = lip;
> +			lv->lv_size = buf_size;
> +			if (ordered)
> +				lv->lv_buf_len = XFS_LOG_VEC_ORDERED;
> +			else
> +				lv->lv_iovecp = (struct xfs_log_iovec *)&lv[1];
> +			lip->li_lv_shadow = lv;
> +		} else {
> +			/* same or smaller, optimise common overwrite case */
> +			lv = lip->li_lv_shadow;
> +			if (ordered)
> +				lv->lv_buf_len = XFS_LOG_VEC_ORDERED;
> +			else
> +				lv->lv_buf_len = 0;
> +			lv->lv_bytes = 0;
> +			lv->lv_next = NULL;
> +		}
> +
> +		/* Ensure the lv is set up according to ->iop_size */
> +		lv->lv_niovecs = niovecs;
> +
> +		/* The allocated data region lies beyond the iovec region */
> +		lv->lv_buf = (char *)lv + xlog_cil_iovec_space(niovecs);
> +	}
> +
> +}
> +
>  /*
>   * Prepare the log item for insertion into the CIL. Calculate the difference in
>   * log space and vectors it will consume, and if it is a new item pin it as
> @@ -101,16 +243,19 @@ xfs_cil_prepare_item(
>  	/*
>  	 * If there is no old LV, this is the first time we've seen the item in
>  	 * this CIL context and so we need to pin it. If we are replacing the
> -	 * old_lv, then remove the space it accounts for and free it.
> +	 * old_lv, then remove the space it accounts for and make it the shadow
> +	 * buffer for later freeing. In both cases we are now switching to the
> +	 * shadow buffer, so update the the pointer to it appropriately.
>  	 */
> -	if (!old_lv)
> +	if (!old_lv) {
>  		lv->lv_item->li_ops->iop_pin(lv->lv_item);
> -	else if (old_lv != lv) {
> +		lv->lv_item->li_lv_shadow = NULL;
> +	} else if (old_lv != lv) {
>  		ASSERT(lv->lv_buf_len != XFS_LOG_VEC_ORDERED);
>  
>  		*diff_len -= old_lv->lv_bytes;
>  		*diff_iovecs -= old_lv->lv_niovecs;
> -		kmem_free(old_lv);
> +		lv->lv_item->li_lv_shadow = old_lv;
>  	}
>  
>  	/* attach new log vector to log item */
> @@ -134,11 +279,13 @@ xfs_cil_prepare_item(
>   * write it out asynchronously without needing to relock the object that was
>   * modified at the time it gets written into the iclog.
>   *
> - * This function builds a vector for the changes in each log item in the
> - * transaction. It then works out the length of the buffer needed for each log
> - * item, allocates them and formats the vector for the item into the buffer.
> - * The buffer is then attached to the log item are then inserted into the
> - * Committed Item List for tracking until the next checkpoint is written out.
> + * This function takes the prepared log vectors attached to each log item, and
> + * formats the changes into the log vector buffer. The buffer it uses is
> + * dependent on the current state of the vector in the CIL - the shadow lv is
> + * guaranteed to be large enough for the current modification, but we will only
> + * use that if we can't reuse the existing lv. If we can't reuse the existing
> + * lv, then simple swap it out for the shadow lv. We don't free it - that is
> + * done lazily either by th enext modification or the freeing of the log item.
>   *
>   * We don't set up region headers during this process; we simply copy the
>   * regions into the flat buffer. We can do this because we still have to do a
> @@ -171,59 +318,29 @@ xlog_cil_insert_format_items(
>  	list_for_each_entry(lidp, &tp->t_items, lid_trans) {
>  		struct xfs_log_item *lip = lidp->lid_item;
>  		struct xfs_log_vec *lv;
> -		struct xfs_log_vec *old_lv;
> -		int	niovecs = 0;
> -		int	nbytes = 0;
> -		int	buf_size;
> +		struct xfs_log_vec *old_lv = NULL;
> +		struct xfs_log_vec *shadow;
>  		bool	ordered = false;
>  
>  		/* Skip items which aren't dirty in this transaction. */
>  		if (!(lidp->lid_flags & XFS_LID_DIRTY))
>  			continue;
>  
> -		/* get number of vecs and size of data to be stored */
> -		lip->li_ops->iop_size(lip, &niovecs, &nbytes);
> -
> -		/* Skip items that do not have any vectors for writing */
> -		if (!niovecs)
> -			continue;
> -
>  		/*
> -		 * Ordered items need to be tracked but we do not wish to write
> -		 * them. We need a logvec to track the object, but we do not
> -		 * need an iovec or buffer to be allocated for copying data.
> +		 * The formatting size information is already attached to
> +		 * the shadow lv on the log item.
>  		 */
> -		if (niovecs == XFS_LOG_VEC_ORDERED) {
> +		shadow = lip->li_lv_shadow;
> +		if (shadow->lv_buf_len == XFS_LOG_VEC_ORDERED)
>  			ordered = true;
> -			niovecs = 0;
> -			nbytes = 0;
> -		}
>  
> -		/*
> -		 * We 64-bit align the length of each iovec so that the start
> -		 * of the next one is naturally aligned.  We'll need to
> -		 * account for that slack space here. Then round nbytes up
> -		 * to 64-bit alignment so that the initial buffer alignment is
> -		 * easy to calculate and verify.
> -		 */
> -		nbytes += niovecs * sizeof(uint64_t);
> -		nbytes = round_up(nbytes, sizeof(uint64_t));
> -
> -		/* grab the old item if it exists for reservation accounting */
> -		old_lv = lip->li_lv;
> -
> -		/*
> -		 * The data buffer needs to start 64-bit aligned, so round up
> -		 * that space to ensure we can align it appropriately and not
> -		 * overrun the buffer.
> -		 */
> -		buf_size = nbytes +
> -			   round_up((sizeof(struct xfs_log_vec) +
> -				     niovecs * sizeof(struct xfs_log_iovec)),
> -				    sizeof(uint64_t));
> +		/* Skip items that do not have any vectors for writing */
> +		if (!shadow->lv_niovecs && !ordered)
> +			continue;
>  
>  		/* compare to existing item size */
> -		if (lip->li_lv && buf_size <= lip->li_lv->lv_size) {
> +		old_lv = lip->li_lv;
> +		if (lip->li_lv && shadow->lv_size <= lip->li_lv->lv_size) {
>  			/* same or smaller, optimise common overwrite case */
>  			lv = lip->li_lv;
>  			lv->lv_next = NULL;
> @@ -237,32 +354,29 @@ xlog_cil_insert_format_items(
>  			 */
>  			*diff_iovecs -= lv->lv_niovecs;
>  			*diff_len -= lv->lv_bytes;
> +
> +			/* Ensure the lv is set up according to ->iop_size */
> +			lv->lv_niovecs = shadow->lv_niovecs;
> +
> +			/* reset the lv buffer information for new formatting */
> +			lv->lv_buf_len = 0;
> +			lv->lv_bytes = 0;
> +			lv->lv_buf = (char *)lv +
> +					xlog_cil_iovec_space(lv->lv_niovecs);
>  		} else {
> -			/* allocate new data chunk */
> -			lv = kmem_zalloc(buf_size, KM_SLEEP|KM_NOFS);
> +			/* switch to shadow buffer! */
> +			lv = shadow;
>  			lv->lv_item = lip;
> -			lv->lv_size = buf_size;
>  			if (ordered) {
>  				/* track as an ordered logvec */
>  				ASSERT(lip->li_lv == NULL);
> -				lv->lv_buf_len = XFS_LOG_VEC_ORDERED;
>  				goto insert;
>  			}
> -			lv->lv_iovecp = (struct xfs_log_iovec *)&lv[1];
>  		}
>  
> -		/* Ensure the lv is set up according to ->iop_size */
> -		lv->lv_niovecs = niovecs;
> -
> -		/* The allocated data region lies beyond the iovec region */
> -		lv->lv_buf_len = 0;
> -		lv->lv_bytes = 0;
> -		lv->lv_buf = (char *)lv + buf_size - nbytes;
>  		ASSERT(IS_ALIGNED((unsigned long)lv->lv_buf, sizeof(uint64_t)));
> -
>  		lip->li_ops->iop_format(lip, lv);
>  insert:
> -		ASSERT(lv->lv_buf_len <= nbytes);
>  		xfs_cil_prepare_item(log, lv, old_lv, diff_len, diff_iovecs);
>  	}
>  }
> @@ -784,6 +898,13 @@ xfs_log_commit_cil(
>  	struct xlog		*log = mp->m_log;
>  	struct xfs_cil		*cil = log->l_cilp;
>  
> +	/*
> +	 * Do all necessary memory allocation before we lock the CIL.
> +	 * This ensures the allocation does not deadlock with a CIL
> +	 * push in memory reclaim (e.g. from kswapd).
> +	 */
> +	xlog_cil_alloc_shadow_bufs(log, tp);
> +
>  	/* lock out background commit */
>  	down_read(&cil->xc_ctx_lock);
>  
> diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h
> index 4643070..74e6819 100644
> --- a/fs/xfs/xfs_trans.h
> +++ b/fs/xfs/xfs_trans.h
> @@ -52,6 +52,7 @@ typedef struct xfs_log_item {
>  	/* delayed logging */
>  	struct list_head		li_cil;		/* CIL pointers */
>  	struct xfs_log_vec		*li_lv;		/* active log vector */
> +	struct xfs_log_vec		*li_lv_shadow;	/* standby vector */
>  	xfs_lsn_t			li_seq;		/* CIL commit seq */
>  } xfs_log_item_t;
>  
> -- 
> Dave Chinner
> david@fromorbit.com
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From willy@linux.intel.com  Tue Jan 26 08:47:58 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 5E2237CA3
	for ; Tue, 26 Jan 2016 08:47:57 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id DD34CAC007
	for ; Tue, 26 Jan 2016 06:47:56 -0800 (PST)
X-ASG-Debug-ID: 1453819669-04cb6c0d5edef80001-NocioJ
Received: from mga14.intel.com ([192.55.52.115]) by cuda.sgi.com with ESMTP id W1FLtQjuMC7ktrgl for ; Tue, 26 Jan 2016 06:47:49 -0800 (PST)
X-Barracuda-Envelope-From: willy@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.115
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
  by fmsmga103.fm.intel.com with ESMTP; 26 Jan 2016 06:47:50 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,350,1449561600"; 
   d="scan'208";a="889519497"
Received: from agu1-mobl2.amr.corp.intel.com (HELO thog.int.wil.cx) ([10.254.78.135])
  by fmsmga001.fm.intel.com with SMTP; 26 Jan 2016 06:47:48 -0800
Received: by thog.int.wil.cx (Postfix, from userid 1000)
	id 92BB05F96A; Tue, 26 Jan 2016 09:47:46 -0500 (EST)
Date: Tue, 26 Jan 2016 09:47:46 -0500
From: Matthew Wilcox 
To: Jan Kara 
Cc: Dave Chinner ,
	Ross Zwisler ,
	linux-kernel@vger.kernel.org, Theodore Ts'o ,
	Alexander Viro ,
	Andreas Dilger ,
	Andrew Morton ,
	Dan Williams , Jan Kara ,
	linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-nvdimm@lists.01.org, xfs@oss.sgi.com
Subject: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
Message-ID: <20160126144746.GL2948@linux.intel.com>
X-ASG-Orig-Subj: Re: [RFC PATCH] dax, ext2, ext4, XFS: fix data corruption race
References: <1453503971-5319-1-git-send-email-ross.zwisler@linux.intel.com>
 <20160124220107.GI20456@dastard>
 <20160125135921.GE24938@quack.suse.cz>
 <20160126124812.GJ2948@linux.intel.com>
 <20160126130521.GB23820@quack.suse.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160126130521.GB23820@quack.suse.cz>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.115]
X-Barracuda-Start-Time: 1453819669
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26465
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Tue, Jan 26, 2016 at 02:05:21PM +0100, Jan Kara wrote:
> On Tue 26-01-16 07:48:12, Matthew Wilcox wrote:
> > I *think* that what Dave's proposing (and if he isn't, I'm proposing it
> > for him) is that the filesystem takes its allocation lock shared during
> > the ->fault handler, then in the ->page_mkwrite handler, it knows that an
> > allocation is coming, so it takes its allocation lock in exclusive mode.
> > 
> > So read vs write faults won't be able to race because the allocation lock
> > will prevent it.
> 
> So this is correct and clean design but we will take the lock in exclusive
> mode (and thus hurt scalability) for every write fault, not just for the
> ones allocating blocks. And at the moment we take exclusive lock for write
> faults, there's no more need for having the hole page instantiated - we can
> still do it for simplicity but it's no longer necessary to avoid data
> corruption.

In my mind we take it only for allocating writes, because we also include
the patch to insert PFNs with the writable bit set in the dax_fault
handler if the page fault was for writes.

Although that only works when the *first* fault is a write ... if we
read and page then write the same page, we will indeed take the lock
in exclusive mode.  I think that's fixable too -- in the page_mkwrite
handler, take the lock in exclusive mode only if there's a page in the
radix tree.  I'll take a look at that optimisation after doing the first
couple of steps.

From sandeen@redhat.com  Tue Jan 26 09:00:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id CED057CA2
	for ; Tue, 26 Jan 2016 09:00:31 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id BB69C8F8049
	for ; Tue, 26 Jan 2016 07:00:28 -0800 (PST)
X-ASG-Debug-ID: 1453820427-04cb6c0d60df4f0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id dBSY3SQcRxzkD3nq (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 07:00:27 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id 152DABBF53;
	Tue, 26 Jan 2016 15:00:27 +0000 (UTC)
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0QF0PAp021061
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Tue, 26 Jan 2016 10:00:26 -0500
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
To: Jan Kara 
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-3-git-send-email-sandeen@redhat.com>
 <20160126125710.GA23820@quack.suse.cz>
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56A78A09.5080307@redhat.com>
Date: Tue, 26 Jan 2016 09:00:25 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160126125710.GA23820@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453820427
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On 1/26/16 6:57 AM, Jan Kara wrote:
> On Fri 22-01-16 12:25:31, Eric Sandeen wrote:
>> Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
>> will return quota information for the id equal to or greater
>> than the id requested.  In other words, if the requested id has
>> no quota, the command will return quota information for the
>> next higher id which does have a quota set.  If no higher id
>> has an active quota, -ESRCH is returned.
> 
> Actually, is -ESRCH the right return value? It seems XFS traditionally
> returns -ENOENT when id doesn't exist. So that would look more logical to
> me.

Hm, I was just going by the quotactl manpage, TBH, which says:

       ESRCH          No disc quota is found for the indicated user.


But yes, you are right, it is ENOENT for xfs... argh.  I suppose the
quotactl manpage could use an update as well, then.

At this point do you want me to update the patches & resend or do you
want to fix that up too?  :(  I reference -ESRCH in comments too, I think.

-Eric

From BATV+6e7a510c1b86abac7ed0+4538+infradead.org+hch@bombadil.srs.infradead.org  Tue Jan 26 10:47:08 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 9C8CC7CA2
	for ; Tue, 26 Jan 2016 10:47:08 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 8C9EA304032
	for ; Tue, 26 Jan 2016 08:47:05 -0800 (PST)
X-ASG-Debug-ID: 1453826820-04cb6c0d5fe2690001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id ltnVn0mRgHpHz2fD (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 26 Jan 2016 08:47:01 -0800 (PST)
X-Barracuda-Envelope-From: BATV+6e7a510c1b86abac7ed0+4538+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aO6lg-0008Lu-7m; Tue, 26 Jan 2016 16:47:00 +0000
Date: Tue, 26 Jan 2016 08:47:00 -0800
From: Christoph Hellwig 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs_repair: Fix untranslatable strings
Message-ID: <20160126164700.GA31746@infradead.org>
X-ASG-Orig-Subj: Re: [PATCH] xfs_repair: Fix untranslatable strings
References: <20160124155853.GA10063@stranger.qboosh.pl>
 <56A50508.4060207@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A50508.4060207@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1453826821
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26467
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

On Sun, Jan 24, 2016 at 11:08:24AM -0600, Eric Sandeen wrote:
> Don't substitute in "would" or "will" for %s here, it makes
> it untranslatable.  Just spell it out twice like every other
> message in the function.
> 
> Reported-by: Jakub Bogusz 
> Signed-off-by: Eric Sandeen 

Looks good,

Reviewed-by: Christoph Hellwig 

From BATV+6e7a510c1b86abac7ed0+4538+infradead.org+hch@bombadil.srs.infradead.org  Tue Jan 26 10:47:32 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 9C3F27CA2
	for ; Tue, 26 Jan 2016 10:47:32 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 6DD40304032
	for ; Tue, 26 Jan 2016 08:47:32 -0800 (PST)
X-ASG-Debug-ID: 1453826838-04bdf04d6cd7520001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id 4fcIdyUlB5itQDOA (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 26 Jan 2016 08:47:19 -0800 (PST)
X-Barracuda-Envelope-From: BATV+6e7a510c1b86abac7ed0+4538+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aO6lv-0008PC-I4; Tue, 26 Jan 2016 16:47:15 +0000
Date: Tue, 26 Jan 2016 08:47:15 -0800
From: Christoph Hellwig 
To: Jakub Bogusz 
Cc: xfs@oss.sgi.com, Christoph Hellwig 
Subject: Re: Polish translation update + localization remark for xfsprogs
 4.3.0
Message-ID: <20160126164715.GB31746@infradead.org>
X-ASG-Orig-Subj: Re: Polish translation update + localization remark for xfsprogs
 4.3.0
References: <20160124155853.GA10063@stranger.qboosh.pl>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160124155853.GA10063@stranger.qboosh.pl>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1453826839
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 2.60
X-Barracuda-Spam-Status: No, SCORE=2.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC7_SA_HREF_FROM_MISMATCH_TEXT_URIx1_HL, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26467
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	2.50 BSF_SC7_SA_HREF_FROM_MISMATCH_TEXT_URIx1_HL Custom Rule
	                           HREF_FROM_MISMATCH_TEXT_URIx1_HL

On Sun, Jan 24, 2016 at 04:58:53PM +0100, Jakub Bogusz wrote:
> Hello,
> 
> I've just updated Polish translation for xfsprogs 4.3.0, it's available
> at:
> http://qboosh.pl/pl.po/xfsprogs-4.3.0.pl.po

Looks fine to me:

Acked-by: Christoph Hellwig 

From jack@suse.cz  Tue Jan 26 11:52:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 07D4B7CA2
	for ; Tue, 26 Jan 2016 11:52:05 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 8A29BAC003
	for ; Tue, 26 Jan 2016 09:52:01 -0800 (PST)
X-ASG-Debug-ID: 1453830714-04cbb026fee11f0001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id zdYsICoVOTRe8CfU (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 26 Jan 2016 09:51:55 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 0A6EBABB0;
	Tue, 26 Jan 2016 17:51:53 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id BBEBE823DA; Tue, 26 Jan 2016 18:52:05 +0100 (CET)
Date: Tue, 26 Jan 2016 18:52:05 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: Jan Kara , linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Message-ID: <20160126175205.GA29388@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-3-git-send-email-sandeen@redhat.com>
 <20160126125710.GA23820@quack.suse.cz>
 <56A78A09.5080307@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A78A09.5080307@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453830715
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26469
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue 26-01-16 09:00:25, Eric Sandeen wrote:
> On 1/26/16 6:57 AM, Jan Kara wrote:
> > On Fri 22-01-16 12:25:31, Eric Sandeen wrote:
> >> Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
> >> will return quota information for the id equal to or greater
> >> than the id requested.  In other words, if the requested id has
> >> no quota, the command will return quota information for the
> >> next higher id which does have a quota set.  If no higher id
> >> has an active quota, -ESRCH is returned.
> > 
> > Actually, is -ESRCH the right return value? It seems XFS traditionally
> > returns -ENOENT when id doesn't exist. So that would look more logical to
> > me.
> 
> Hm, I was just going by the quotactl manpage, TBH, which says:
> 
>        ESRCH          No disc quota is found for the indicated user.
> 
> 
> But yes, you are right, it is ENOENT for xfs... argh.  I suppose the
> quotactl manpage could use an update as well, then.

Yeah, so VFS quotas use ESRCH when quota for particular fs is not enabled
(while ENOENT means device you passed in doesn't exist). So probably a
solution that keeps XFS and VFS interfaces most selfconsistent is to return
ENOENT from Q_XGETNEXTQUOTA and ESRCH from Q_GETNEXTQUOTA. I'll update your
patches in this sense in the comments and changelogs. But XFS patches
(which I don't carry) need updating the actual code...

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From sandeen@redhat.com  Tue Jan 26 11:57:14 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id EC08F7CA2
	for ; Tue, 26 Jan 2016 11:57:14 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id DA8D2304032
	for ; Tue, 26 Jan 2016 09:57:14 -0800 (PST)
X-ASG-Debug-ID: 1453831031-04cb6c0d5de4640001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id RHiOoMGpEykbIb6I (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 09:57:12 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id 3E670461EF;
	Tue, 26 Jan 2016 17:57:11 +0000 (UTC)
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0QHv9PP013897
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Tue, 26 Jan 2016 12:57:10 -0500
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
To: Jan Kara 
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-3-git-send-email-sandeen@redhat.com>
 <20160126125710.GA23820@quack.suse.cz> <56A78A09.5080307@redhat.com>
 <20160126175205.GA29388@quack.suse.cz>
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com
From: Eric Sandeen 
Message-ID: <56A7B375.3000304@redhat.com>
Date: Tue, 26 Jan 2016 11:57:09 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160126175205.GA29388@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453831031
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On 1/26/16 11:52 AM, Jan Kara wrote:
> On Tue 26-01-16 09:00:25, Eric Sandeen wrote:
>> On 1/26/16 6:57 AM, Jan Kara wrote:
>>> On Fri 22-01-16 12:25:31, Eric Sandeen wrote:
>>>> Q_XGETNEXTQUOTA is exactly like Q_XGETQUOTA, except that it
>>>> will return quota information for the id equal to or greater
>>>> than the id requested.  In other words, if the requested id has
>>>> no quota, the command will return quota information for the
>>>> next higher id which does have a quota set.  If no higher id
>>>> has an active quota, -ESRCH is returned.
>>>
>>> Actually, is -ESRCH the right return value? It seems XFS traditionally
>>> returns -ENOENT when id doesn't exist. So that would look more logical to
>>> me.
>>
>> Hm, I was just going by the quotactl manpage, TBH, which says:
>>
>>        ESRCH          No disc quota is found for the indicated user.
>>
>>
>> But yes, you are right, it is ENOENT for xfs... argh.  I suppose the
>> quotactl manpage could use an update as well, then.
> 
> Yeah, so VFS quotas use ESRCH when quota for particular fs is not enabled
> (while ENOENT means device you passed in doesn't exist). So probably a
> solution that keeps XFS and VFS interfaces most selfconsistent is to return
> ENOENT from Q_XGETNEXTQUOTA and ESRCH from Q_GETNEXTQUOTA. I'll update your
> patches in this sense in the comments and changelogs. But XFS patches
> (which I don't carry) need updating the actual code...

*nod* thanks.  I'll just resend the XFS patches to the XFS list.

-Eric

From darrick.wong@oracle.com  Tue Jan 26 12:05:09 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 463397CA3
	for ; Tue, 26 Jan 2016 12:05:09 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 382868F804B
	for ; Tue, 26 Jan 2016 10:05:06 -0800 (PST)
X-ASG-Debug-ID: 1453831504-04cbb026fce1850001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id w32TfKUgyOFQfGco (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 10:05:04 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0QI52Zj000627
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Tue, 26 Jan 2016 18:05:03 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0QI52H0032454
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Tue, 26 Jan 2016 18:05:02 GMT
Received: from abhmp0018.oracle.com (abhmp0018.oracle.com [141.146.116.24])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0QI52e3006562;
	Tue, 26 Jan 2016 18:05:02 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Tue, 26 Jan 2016 10:05:01 -0800
Date: Tue, 26 Jan 2016 10:05:00 -0800
From: "Darrick J. Wong" 
To: Dave Chinner 
Cc: xfs , Eric Sandeen ,
        Brian Foster 
Subject: [PATCH] xfs: fix endianness error when checking log block crc on big
 endian platforms
Message-ID: <20160126180500.GA5852@birch.djwong.org>
X-ASG-Orig-Subj: [PATCH] xfs: fix endianness error when checking log block crc on big
 endian platforms
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453831504
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26469
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Since the checksum function and the field are both __le32, don't
perform endian conversion when comparing the two.  This fixes mount
failures on ppc64 in 4.5-rc1.

Signed-off-by: Darrick J. Wong 
---
 fs/xfs/xfs_log_recover.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
index ac80f39..52dd946 100644
--- a/fs/xfs/xfs_log_recover.c
+++ b/fs/xfs/xfs_log_recover.c
@@ -4502,7 +4502,7 @@ xlog_recover_process(
 	 * know precisely what failed.
 	 */
 	if (pass == XLOG_RECOVER_CRCPASS) {
-		if (rhead->h_crc && crc != le32_to_cpu(rhead->h_crc))
+		if (rhead->h_crc && crc != rhead->h_crc)
 			return -EFSBADCRC;
 		return 0;
 	}

From sandeen@redhat.com  Tue Jan 26 12:40:03 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 4761E7CA3
	for ; Tue, 26 Jan 2016 12:40:03 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 294588F8033
	for ; Tue, 26 Jan 2016 10:40:03 -0800 (PST)
X-ASG-Debug-ID: 1453833601-04bdf04d6cdad20001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id zV6HT6m5YAg3hO7O (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 10:40:02 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id 8DE3849DBB;
	Tue, 26 Jan 2016 18:40:01 +0000 (UTC)
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0QIe0WY008073
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Tue, 26 Jan 2016 13:40:01 -0500
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
To: Jan Kara 
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-3-git-send-email-sandeen@redhat.com>
 <20160126125710.GA23820@quack.suse.cz> <56A78A09.5080307@redhat.com>
 <20160126175205.GA29388@quack.suse.cz>
Cc: linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56A7BD7F.6010802@redhat.com>
Date: Tue, 26 Jan 2016 12:39:59 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160126175205.GA29388@quack.suse.cz>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453833601
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On 1/26/16 11:52 AM, Jan Kara wrote:

> Yeah, so VFS quotas use ESRCH when quota for particular fs is not enabled
> (while ENOENT means device you passed in doesn't exist).

And what does it return for "that ID has no quota?"  Anything?  Maybe not,
see below?

> So probably a
> solution that keeps XFS and VFS interfaces most selfconsistent is to return
> ENOENT from Q_XGETNEXTQUOTA and ESRCH from Q_GETNEXTQUOTA. I'll update your
> patches in this sense in the comments and changelogs. But XFS patches
> (which I don't carry) need updating the actual code...

Actually, ok, now I'm a little more confused.

Today, Q_XGETQUOTA and Q_GETQUOTA will both return -ENOENT on XFS if a
quota doesn't exist for that ID:

quotactl(Q_GETQUOTA|USRQUOTA, "/dev/sdb2", 123456789, 0x7ffdf2edbc80) = -1 ENOENT (No such file or directory)
quotactl(Q_XGETQUOTA|USRQUOTA, "/dev/sdb2", 123456789, 0x7ffdf2edbc10) = -1 ENOENT (No such file or directory)

It seems like the *NEXT* variants (on xfs at least?) may as well continue to do the same...

But on ext4, I see 0 returned for a nonexistent quota:

quotactl(Q_GETQUOTA|USRQUOTA, "/dev/sdb1", 123456789, {bhardlimit=0, bsoftlimit=0, curspace=0, ihardlimit=0, isoftlimit=0, curinodes=0, ...}) = 0
quotactl(Q_XGETQUOTA|USRQUOTA, "/dev/sdb1", 123456789, {version=1, flags=XFS_USER_QUOTA, fieldmask=0, id=123456789, blk_hardlimit=0, blk_softlimit=0, ino_hardlimit=0, ino_softlimit=0, bcount=0, icount=0, ...}) = 0

So the difference doesn't seem to be XGETQUOTA vs GETQUOTA, rather it's the filesystem handling the call?

Still, we do need a way to pass back "No more quotas to find" from Q_[X]GETNEXTQUOTA; XFS will do -ENOENT, but if -ENOENT and -ESRCH already have specific meanings on non-xfs filesystems, I'm not sure where we go from there.

Sorry for my confusion.  :(

-Eric

From sandeen@sandeen.net  Tue Jan 26 12:40:58 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 45DC07CA3
	for ; Tue, 26 Jan 2016 12:40:58 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 36AD68F8039
	for ; Tue, 26 Jan 2016 10:40:58 -0800 (PST)
X-ASG-Debug-ID: 1453833655-04cbb026fce29b0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id SlXTaPeB1i5agIN9 for ; Tue, 26 Jan 2016 10:40:55 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 2045C6146F19;
	Tue, 26 Jan 2016 12:40:55 -0600 (CST)
Subject: [PATCH 7/7 V4] xfs: wire up Q_XGETNEXTQUOTA / get_nextdqblk
To: Eric Sandeen , linux-fsdevel@vger.kernel.org,
 xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 7/7 V4] xfs: wire up Q_XGETNEXTQUOTA / get_nextdqblk
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-8-git-send-email-sandeen@redhat.com>
Cc: jack@suse.cz
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56A7BDB6.7010106@sandeen.net>
Date: Tue, 26 Jan 2016 12:40:54 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <1453487136-12681-8-git-send-email-sandeen@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453833655
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26470
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Add code to allow the Q_XGETNEXTQUOTA quotactl to quickly find
all active quotas by examining the quota inode, and skipping
over unallocated or uninitialized regions.

Userspace can then use this interface rather than i.e. a
getpwent() loop when asked to report all active quotas.

Signed-off-by: Eric Sandeen 
---

V4: Return -ENOENT not -ESRCH from xfs_dq_get_next_id() when there
are no more allocated quotas to be found; this is consistent with
the normal, non-next-getquota interface in xfs.

 fs/xfs/libxfs/xfs_quota_defs.h |    3 +-
 fs/xfs/xfs_dquot.c             |   96 ++++++++++++++++++++++++++++++++++++++++
 fs/xfs/xfs_qm.h                |    4 +-
 fs/xfs/xfs_qm_syscalls.c       |   12 +++--
 fs/xfs/xfs_quotaops.c          |   36 ++++++++++++++-
 5 files changed, 142 insertions(+), 9 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_quota_defs.h b/fs/xfs/libxfs/xfs_quota_defs.h
index f51078f..8eed512 100644
--- a/fs/xfs/libxfs/xfs_quota_defs.h
+++ b/fs/xfs/libxfs/xfs_quota_defs.h
@@ -37,7 +37,7 @@ typedef __uint16_t	xfs_qwarncnt_t;
 #define XFS_DQ_PROJ		0x0002		/* project quota */
 #define XFS_DQ_GROUP		0x0004		/* a group quota */
 #define XFS_DQ_DIRTY		0x0008		/* dquot is dirty */
-#define XFS_DQ_FREEING		0x0010		/* dquot is beeing torn down */
+#define XFS_DQ_FREEING		0x0010		/* dquot is being torn down */
 
 #define XFS_DQ_ALLTYPES		(XFS_DQ_USER|XFS_DQ_PROJ|XFS_DQ_GROUP)
 
@@ -116,6 +116,7 @@ typedef __uint16_t	xfs_qwarncnt_t;
 #define XFS_QMOPT_DQREPAIR	0x0001000 /* repair dquot if damaged */
 #define XFS_QMOPT_GQUOTA	0x0002000 /* group dquot requested */
 #define XFS_QMOPT_ENOSPC	0x0004000 /* enospc instead of edquot (prj) */
+#define XFS_QMOPT_DQNEXT	0x0008000 /* return next dquot >= this ID */
 
 /*
  * flags to xfs_trans_mod_dquot to indicate which field needs to be
diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 5dbde0d..2f0502f 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -686,6 +686,56 @@ error0:
 }
 
 /*
+ * Advance to the next id in the current chunk, or if at the
+ * end of the chunk, skip ahead to first id in next allocated chunk
+ * using the SEEK_DATA interface.
+ */
+int
+xfs_dq_get_next_id(
+	xfs_mount_t		*mp,
+	uint			type,
+	xfs_dqid_t		*id,
+	loff_t			eof)
+{
+	struct xfs_inode	*quotip;
+	xfs_fsblock_t		start;
+	loff_t			offset;
+	uint			lock;
+	xfs_dqid_t		next_id;
+	int			error = 0;
+
+	/* Simple advance */
+	next_id = *id + 1;
+
+	/* If new ID is within the current chunk, advancing it sufficed */
+	if (next_id % mp->m_quotainfo->qi_dqperchunk) {
+		*id = next_id;
+		return 0;
+	}
+
+	/* Nope, next_id is now past the current chunk, so find the next one */
+	start = (xfs_fsblock_t)next_id / mp->m_quotainfo->qi_dqperchunk;
+
+	quotip = xfs_quota_inode(mp, type);
+	lock = xfs_ilock_data_map_shared(quotip);
+
+	offset = __xfs_seek_hole_data(VFS_I(quotip), XFS_FSB_TO_B(mp, start),
+				      eof, SEEK_DATA);
+	if (offset < 0)
+		error = offset;
+
+	xfs_iunlock(quotip, lock);
+
+	/* -ENXIO is essentially "no more data" */
+	if (error)
+		return (error == -ENXIO ? -ENOENT: error);
+
+	/* Convert next data offset back to a quota id */
+	*id = XFS_B_TO_FSB(mp, offset) * mp->m_quotainfo->qi_dqperchunk;
+	return 0;
+}
+
+/*
  * Given the file system, inode OR id, and type (UDQUOT/GDQUOT), return a
  * a locked dquot, doing an allocation (if requested) as needed.
  * When both an inode and an id are given, the inode's id takes precedence.
@@ -705,6 +755,7 @@ xfs_qm_dqget(
 	struct xfs_quotainfo	*qi = mp->m_quotainfo;
 	struct radix_tree_root *tree = xfs_dquot_tree(qi, type);
 	struct xfs_dquot	*dqp;
+	loff_t			eof = 0;
 	int			error;
 
 	ASSERT(XFS_IS_QUOTA_RUNNING(mp));
@@ -732,6 +783,21 @@ xfs_qm_dqget(
 	}
 #endif
 
+	/* Get the end of the quota file if we need it */
+	if (flags & XFS_QMOPT_DQNEXT) {
+		struct xfs_inode	*quotip;
+		xfs_fileoff_t		last;
+		uint			lock_mode;
+
+		quotip = xfs_quota_inode(mp, type);
+		lock_mode = xfs_ilock_data_map_shared(quotip);
+		error = xfs_bmap_last_offset(quotip, &last, XFS_DATA_FORK);
+		xfs_iunlock(quotip, lock_mode);
+		if (error)
+			return error;
+		eof = XFS_FSB_TO_B(mp, last);
+	}
+
 restart:
 	mutex_lock(&qi->qi_tree_lock);
 	dqp = radix_tree_lookup(tree, id);
@@ -745,6 +811,18 @@ restart:
 			goto restart;
 		}
 
+		/* uninit / unused quota found in radix tree, keep looking  */
+		if (flags & XFS_QMOPT_DQNEXT) {
+			if (XFS_IS_DQUOT_UNINITIALIZED(dqp)) {
+				xfs_dqunlock(dqp);
+				mutex_unlock(&qi->qi_tree_lock);
+				error = xfs_dq_get_next_id(mp, type, &id, eof);
+				if (error)
+					return error;
+				goto restart;
+			}
+		}
+
 		dqp->q_nrefs++;
 		mutex_unlock(&qi->qi_tree_lock);
 
@@ -771,6 +849,13 @@ restart:
 	if (ip)
 		xfs_ilock(ip, XFS_ILOCK_EXCL);
 
+	/* If we are asked to find next active id, keep looking */
+	if (error == -ENOENT && (flags & XFS_QMOPT_DQNEXT)) {
+		error = xfs_dq_get_next_id(mp, type, &id, eof);
+		if (!error)
+			goto restart;
+	}
+
 	if (error)
 		return error;
 
@@ -821,6 +906,17 @@ restart:
 	qi->qi_dquots++;
 	mutex_unlock(&qi->qi_tree_lock);
 
+	/* If we are asked to find next active id, keep looking */
+	if (flags & XFS_QMOPT_DQNEXT) {
+		if (XFS_IS_DQUOT_UNINITIALIZED(dqp)) {
+			xfs_qm_dqput(dqp);
+			error = xfs_dq_get_next_id(mp, type, &id, eof);
+			if (error)
+				return error;
+			goto restart;
+		}
+	}
+
  dqret:
 	ASSERT((ip == NULL) || xfs_isilocked(ip, XFS_ILOCK_EXCL));
 	trace_xfs_dqget_miss(dqp);
diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
index 8901a01..c68a38f 100644
--- a/fs/xfs/xfs_qm.h
+++ b/fs/xfs/xfs_qm.h
@@ -164,8 +164,8 @@ extern void		xfs_qm_dqrele_all_inodes(struct xfs_mount *, uint);
 
 /* quota ops */
 extern int		xfs_qm_scall_trunc_qfiles(struct xfs_mount *, uint);
-extern int		xfs_qm_scall_getquota(struct xfs_mount *, xfs_dqid_t,
-					uint, struct qc_dqblk *);
+extern int		xfs_qm_scall_getquota(struct xfs_mount *, xfs_dqid_t *,
+					uint, struct qc_dqblk *, uint);
 extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
 					struct qc_dqblk *);
 extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
index 3640c6e..0a25286 100644
--- a/fs/xfs/xfs_qm_syscalls.c
+++ b/fs/xfs/xfs_qm_syscalls.c
@@ -635,9 +635,10 @@ out:
 int
 xfs_qm_scall_getquota(
 	struct xfs_mount	*mp,
-	xfs_dqid_t		id,
+	xfs_dqid_t		*id,
 	uint			type,
-	struct qc_dqblk		*dst)
+	struct qc_dqblk		*dst,
+	uint			dqget_flags)
 {
 	struct xfs_dquot	*dqp;
 	int			error;
@@ -647,7 +648,7 @@ xfs_qm_scall_getquota(
 	 * we aren't passing the XFS_QMOPT_DOALLOC flag. If it doesn't
 	 * exist, we'll get ENOENT back.
 	 */
-	error = xfs_qm_dqget(mp, NULL, id, type, 0, &dqp);
+	error = xfs_qm_dqget(mp, NULL, *id, type, dqget_flags, &dqp);
 	if (error)
 		return error;
 
@@ -660,6 +661,9 @@ xfs_qm_scall_getquota(
 		goto out_put;
 	}
 
+	/* Fill in the ID we actually read from disk */
+	*id = be32_to_cpu(dqp->q_core.d_id);
+
 	memset(dst, 0, sizeof(*dst));
 	dst->d_spc_hardlimit =
 		XFS_FSB_TO_B(mp, be64_to_cpu(dqp->q_core.d_blk_hardlimit));
@@ -701,7 +705,7 @@ xfs_qm_scall_getquota(
 	if (((XFS_IS_UQUOTA_ENFORCED(mp) && type == XFS_DQ_USER) ||
 	     (XFS_IS_GQUOTA_ENFORCED(mp) && type == XFS_DQ_GROUP) ||
 	     (XFS_IS_PQUOTA_ENFORCED(mp) && type == XFS_DQ_PROJ)) &&
-	    id != 0) {
+	    *id != 0) {
 		if ((dst->d_space > dst->d_spc_softlimit) &&
 		    (dst->d_spc_softlimit > 0)) {
 			ASSERT(dst->d_spc_timer != 0);
diff --git a/fs/xfs/xfs_quotaops.c b/fs/xfs/xfs_quotaops.c
index 7795e0d..f82d79a 100644
--- a/fs/xfs/xfs_quotaops.c
+++ b/fs/xfs/xfs_quotaops.c
@@ -231,14 +231,45 @@ xfs_fs_get_dqblk(
 	struct qc_dqblk		*qdq)
 {
 	struct xfs_mount	*mp = XFS_M(sb);
+	xfs_dqid_t		id;
 
 	if (!XFS_IS_QUOTA_RUNNING(mp))
 		return -ENOSYS;
 	if (!XFS_IS_QUOTA_ON(mp))
 		return -ESRCH;
 
-	return xfs_qm_scall_getquota(mp, from_kqid(&init_user_ns, qid),
-				      xfs_quota_type(qid.type), qdq);
+	id = from_kqid(&init_user_ns, qid);
+	return xfs_qm_scall_getquota(mp, &id,
+				      xfs_quota_type(qid.type), qdq, 0);
+}
+
+/* Return quota info for active quota >= this qid */
+STATIC int
+xfs_fs_get_nextdqblk(
+	struct super_block	*sb,
+	struct kqid		*qid,
+	struct qc_dqblk		*qdq)
+{
+	int			ret;
+	struct xfs_mount	*mp = XFS_M(sb);
+	xfs_dqid_t		id;
+
+	if (!XFS_IS_QUOTA_RUNNING(mp))
+		return -ENOSYS;
+	if (!XFS_IS_QUOTA_ON(mp))
+		return -ESRCH;
+
+	id = from_kqid(&init_user_ns, *qid);
+	ret = xfs_qm_scall_getquota(mp, &id,
+				    xfs_quota_type(qid->type), qdq,
+				    XFS_QMOPT_DQNEXT);
+	if (ret)
+		return ret;
+
+	/* ID may be different, so convert back what we got */
+	*qid = make_kqid(current_user_ns(), qid->type, id);
+	return 0;
+	
 }
 
 STATIC int
@@ -267,5 +298,6 @@ const struct quotactl_ops xfs_quotactl_operations = {
 	.quota_disable		= xfs_quota_disable,
 	.rm_xquota		= xfs_fs_rm_xquota,
 	.get_dqblk		= xfs_fs_get_dqblk,
+	.get_nextdqblk		= xfs_fs_get_nextdqblk,
 	.set_dqblk		= xfs_fs_set_dqblk,
 };

From bfoster@redhat.com  Tue Jan 26 13:13:19 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 4625B7CA3
	for ; Tue, 26 Jan 2016 13:13:19 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id C7B198F8039
	for ; Tue, 26 Jan 2016 11:13:15 -0800 (PST)
X-ASG-Debug-ID: 1453835594-04bdf04d6ddbbc0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id w9EGrlQcQWXlDhz0 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 11:13:14 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id 2D514C0B66C9;
	Tue, 26 Jan 2016 19:13:14 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0QJDD92003050;
	Tue, 26 Jan 2016 14:13:13 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 5EC9E1251C3; Tue, 26 Jan 2016 14:13:12 -0500 (EST)
Date: Tue, 26 Jan 2016 14:13:12 -0500
From: Brian Foster 
To: "Darrick J. Wong" 
Cc: Dave Chinner , xfs ,
        Eric Sandeen 
Subject: Re: [PATCH] xfs: fix endianness error when checking log block crc on
 big endian platforms
Message-ID: <20160126191310.GA54800@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH] xfs: fix endianness error when checking log block crc on
 big endian platforms
References: <20160126180500.GA5852@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160126180500.GA5852@birch.djwong.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453835594
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Tue, Jan 26, 2016 at 10:05:00AM -0800, Darrick J. Wong wrote:
> Since the checksum function and the field are both __le32, don't
> perform endian conversion when comparing the two.  This fixes mount
> failures on ppc64 in 4.5-rc1.
> 
> Signed-off-by: Darrick J. Wong 
> ---

So somehow this first got munged in commit:

  b94fb2d1 xfs: refactor and open code log record crc check

... which was intended to only be a refactoring patch. Instead, it
removes the following line:

-       if (crc != rhead->h_crc) {

... and replaces it with:

+       if (crc != le32_to_cpu(rhead->h_crc)) {

No idea how that happened. Maybe I was just looking at the log message
code and had a thinko. Anyways, it was followed up with commit:

	6528250b xfs: support a crc verification only log record pass

... which adds the code you've fixed here and probably just copied the
busted check.

>  fs/xfs/xfs_log_recover.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
> index ac80f39..52dd946 100644
> --- a/fs/xfs/xfs_log_recover.c
> +++ b/fs/xfs/xfs_log_recover.c
> @@ -4502,7 +4502,7 @@ xlog_recover_process(
>  	 * know precisely what failed.
>  	 */
>  	if (pass == XLOG_RECOVER_CRCPASS) {
> -		if (rhead->h_crc && crc != le32_to_cpu(rhead->h_crc))
> +		if (rhead->h_crc && crc != rhead->h_crc)
>  			return -EFSBADCRC;
>  		return 0;
>  	}

There's another check a few lines after this that is also broken, based
on the first commit referenced above. Do you want me to send a v2 patch
or would you like to? I'll run some ppc64 testing either way...

Brian

From darrick.wong@oracle.com  Tue Jan 26 13:21:18 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id AAA2A7CA3
	for ; Tue, 26 Jan 2016 13:21:18 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 9C651304039
	for ; Tue, 26 Jan 2016 11:21:15 -0800 (PST)
X-ASG-Debug-ID: 1453836072-04cbb026fde3c60001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id nxO1hccXHkZJK3kd (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 11:21:13 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0QJL8UV021385
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Tue, 26 Jan 2016 19:21:08 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0QJL7O9005099
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Tue, 26 Jan 2016 19:21:07 GMT
Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0QJL7ip015365;
	Tue, 26 Jan 2016 19:21:07 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Tue, 26 Jan 2016 11:21:06 -0800
Date: Tue, 26 Jan 2016 11:21:05 -0800
From: "Darrick J. Wong" 
To: Brian Foster 
Cc: Dave Chinner , xfs ,
        Eric Sandeen 
Subject: [PATCH v2] xfs: fix endianness error when checking log block crc on
 big endian platforms
Message-ID: <20160126192105.GB6757@birch.djwong.org>
X-ASG-Orig-Subj: [PATCH v2] xfs: fix endianness error when checking log block crc on
 big endian platforms
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453836072
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26473
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Since the checksum function and the field are both __le32, don't
perform endian conversion when comparing the two.  This fixes mount
failures on ppc64.

v2: fix the other crc comparison

Signed-off-by: Darrick J. Wong 
---
 fs/xfs/xfs_log_recover.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
index da37beb..594f7e6 100644
--- a/fs/xfs/xfs_log_recover.c
+++ b/fs/xfs/xfs_log_recover.c
@@ -4491,7 +4491,7 @@ xlog_recover_process(
 	 * know precisely what failed.
 	 */
 	if (pass == XLOG_RECOVER_CRCPASS) {
-		if (rhead->h_crc && crc != le32_to_cpu(rhead->h_crc))
+		if (rhead->h_crc && crc != rhead->h_crc)
 			return -EFSBADCRC;
 		return 0;
 	}
@@ -4502,7 +4502,7 @@ xlog_recover_process(
 	 * zero CRC check prevents warnings from being emitted when upgrading
 	 * the kernel from one that does not add CRCs by default.
 	 */
-	if (crc != le32_to_cpu(rhead->h_crc)) {
+	if (crc != rhead->h_crc) {
 		if (rhead->h_crc || xfs_sb_version_hascrc(&log->l_mp->m_sb)) {
 			xfs_alert(log->l_mp,
 		"log record CRC mismatch: found 0x%x, expected 0x%x.",

From bfoster@redhat.com  Tue Jan 26 13:49:29 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 6C9FE7CA3
	for ; Tue, 26 Jan 2016 13:49:29 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 5D4818F804C
	for ; Tue, 26 Jan 2016 11:49:29 -0800 (PST)
X-ASG-Debug-ID: 1453837767-04cb6c0d5ee7740001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 2Fkjo6kihHajhYPk (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 11:49:28 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id C62C18E909;
	Tue, 26 Jan 2016 19:49:27 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0QJnRLg021633;
	Tue, 26 Jan 2016 14:49:27 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 1E7081251C3; Tue, 26 Jan 2016 14:49:26 -0500 (EST)
Date: Tue, 26 Jan 2016 14:49:26 -0500
From: Brian Foster 
To: "Darrick J. Wong" 
Cc: Eric Sandeen , xfs 
Subject: Re: [PATCH v2] xfs: fix endianness error when checking log block crc
 on big endian platforms
Message-ID: <20160126194925.GB54800@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH v2] xfs: fix endianness error when checking log block crc
 on big endian platforms
References: <20160126192105.GB6757@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160126192105.GB6757@birch.djwong.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453837768
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Tue, Jan 26, 2016 at 11:21:05AM -0800, Darrick J. Wong wrote:
> Since the checksum function and the field are both __le32, don't
> perform endian conversion when comparing the two.  This fixes mount
> failures on ppc64.
> 
> v2: fix the other crc comparison
> 
> Signed-off-by: Darrick J. Wong 
> ---

Thanks!

Reviewed-by: Brian Foster 

>  fs/xfs/xfs_log_recover.c |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
> index da37beb..594f7e6 100644
> --- a/fs/xfs/xfs_log_recover.c
> +++ b/fs/xfs/xfs_log_recover.c
> @@ -4491,7 +4491,7 @@ xlog_recover_process(
>  	 * know precisely what failed.
>  	 */
>  	if (pass == XLOG_RECOVER_CRCPASS) {
> -		if (rhead->h_crc && crc != le32_to_cpu(rhead->h_crc))
> +		if (rhead->h_crc && crc != rhead->h_crc)
>  			return -EFSBADCRC;
>  		return 0;
>  	}
> @@ -4502,7 +4502,7 @@ xlog_recover_process(
>  	 * zero CRC check prevents warnings from being emitted when upgrading
>  	 * the kernel from one that does not add CRCs by default.
>  	 */
> -	if (crc != le32_to_cpu(rhead->h_crc)) {
> +	if (crc != rhead->h_crc) {
>  		if (rhead->h_crc || xfs_sb_version_hascrc(&log->l_mp->m_sb)) {
>  			xfs_alert(log->l_mp,
>  		"log record CRC mismatch: found 0x%x, expected 0x%x.",
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From jack@suse.cz  Tue Jan 26 14:40:09 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 497E07CA3
	for ; Tue, 26 Jan 2016 14:40:09 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 394A38F8039
	for ; Tue, 26 Jan 2016 12:40:05 -0800 (PST)
X-ASG-Debug-ID: 1453840802-04cb6c0d60e8a60001-NocioJ
Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by cuda.sgi.com with ESMTP id bY1ixfHBW2DI6nRR (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 26 Jan 2016 12:40:03 -0800 (PST)
X-Barracuda-Envelope-From: jack@suse.cz
X-Barracuda-Apparent-Source-IP: 195.135.220.15
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 526D2ABB4;
	Tue, 26 Jan 2016 20:40:01 +0000 (UTC)
Received: by quack.suse.cz (Postfix, from userid 1000)
	id 52272823DA; Tue, 26 Jan 2016 21:40:14 +0100 (CET)
Date: Tue, 26 Jan 2016 21:40:14 +0100
From: Jan Kara 
To: Eric Sandeen 
Cc: Jan Kara , linux-fsdevel@vger.kernel.org, xfs@oss.sgi.com
Subject: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
Message-ID: <20160126204014.GB29388@quack.suse.cz>
X-ASG-Orig-Subj: Re: [PATCH 2/7] quota: add new quotactl Q_XGETNEXTQUOTA
References: <1453487136-12681-1-git-send-email-sandeen@redhat.com>
 <1453487136-12681-3-git-send-email-sandeen@redhat.com>
 <20160126125710.GA23820@quack.suse.cz>
 <56A78A09.5080307@redhat.com>
 <20160126175205.GA29388@quack.suse.cz>
 <56A7BD7F.6010802@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A7BD7F.6010802@redhat.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: mx2.suse.de[195.135.220.15]
X-Barracuda-Start-Time: 1453840803
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26474
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Tue 26-01-16 12:39:59, Eric Sandeen wrote:
> On 1/26/16 11:52 AM, Jan Kara wrote:
> 
> > Yeah, so VFS quotas use ESRCH when quota for particular fs is not enabled
> > (while ENOENT means device you passed in doesn't exist).
> 
> And what does it return for "that ID has no quota?"  Anything?  Maybe not,
> see below?
> 
> > So probably a
> > solution that keeps XFS and VFS interfaces most selfconsistent is to return
> > ENOENT from Q_XGETNEXTQUOTA and ESRCH from Q_GETNEXTQUOTA. I'll update your
> > patches in this sense in the comments and changelogs. But XFS patches
> > (which I don't carry) need updating the actual code...
> 
> Actually, ok, now I'm a little more confused.
> 
> Today, Q_XGETQUOTA and Q_GETQUOTA will both return -ENOENT on XFS if a
> quota doesn't exist for that ID:
> 
> quotactl(Q_GETQUOTA|USRQUOTA, "/dev/sdb2", 123456789, 0x7ffdf2edbc80) = -1 ENOENT (No such file or directory)
> quotactl(Q_XGETQUOTA|USRQUOTA, "/dev/sdb2", 123456789, 0x7ffdf2edbc10) = -1 ENOENT (No such file or directory)
> 
> It seems like the *NEXT* variants (on xfs at least?) may as well continue
> to do the same...
> 
> But on ext4, I see 0 returned for a nonexistent quota:
> 
> quotactl(Q_GETQUOTA|USRQUOTA, "/dev/sdb1", 123456789, {bhardlimit=0, bsoftlimit=0, curspace=0, ihardlimit=0, isoftlimit=0, curinodes=0, ...}) = 0
> quotactl(Q_XGETQUOTA|USRQUOTA, "/dev/sdb1", 123456789, {version=1, flags=XFS_USER_QUOTA, fieldmask=0, id=123456789, blk_hardlimit=0, blk_softlimit=0, ino_hardlimit=0, ino_softlimit=0, bcount=0, icount=0, ...}) = 0
> 
> So the difference doesn't seem to be XGETQUOTA vs GETQUOTA, rather it's
> the filesystem handling the call?

Yes, it's about whether fs/quota/dquot.c or XFS handles the call. VFS
traditionally returns structure full of zeros when you call Q_GETQUOTA for
non-existent ID on a system that otherwise supports quotas (that is a
historical heritage of old quota file format). This is however
impractical for Q_GETNEXTQUOTA so there we have to define some error
terminating the iteration.

> Still, we do need a way to pass back "No more quotas to find" from
> Q_[X]GETNEXTQUOTA; XFS will do -ENOENT, but if -ENOENT and -ESRCH already
> have specific meanings on non-xfs filesystems, I'm not sure where we go
> from there.

Yeah, we have to simply overload one of the error codes for VFS
implementation and I'm not strongly convinced which one. But since you
pointed out that it's not really about the interface but about backing fs,
I think making both ENOENT is probably going to be less confusing for
userspace.

								Honza
-- 
Jan Kara 
SUSE Labs, CR

From darrick.wong@oracle.com  Tue Jan 26 16:09:25 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 306227CA3
	for ; Tue, 26 Jan 2016 16:09:25 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 031FB304039
	for ; Tue, 26 Jan 2016 14:09:21 -0800 (PST)
X-ASG-Debug-ID: 1453846156-04bdf04d6adfd10001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id AtFBa2NfWYupsxhd (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 14:09:16 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0QM9FVC018114
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Tue, 26 Jan 2016 22:09:15 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0QM9Ej0008678
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Tue, 26 Jan 2016 22:09:15 GMT
Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0QM9E5p008346;
	Tue, 26 Jan 2016 22:09:14 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Tue, 26 Jan 2016 14:09:13 -0800
Date: Tue, 26 Jan 2016 14:09:12 -0800
From: "Darrick J. Wong" 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
Message-ID: <20160126220912.GB5852@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 1/5] xfs_io: detect the '-R' option in getopt
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
 <56A307D8.60202@sandeen.net>
 <20160123061811.GC5500@birch.djwong.org>
 <56A3C054.4000804@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A3C054.4000804@sandeen.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0021.oracle.com [156.151.31.71]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453846156
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26476
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Sat, Jan 23, 2016 at 12:03:00PM -0600, Eric Sandeen wrote:
> 
> 
> On 1/23/16 12:18 AM, Darrick J. Wong wrote:
> > On Fri, Jan 22, 2016 at 10:55:52PM -0600, Eric Sandeen wrote:
> >> On 1/22/16 6:35 PM, Darrick J. Wong wrote:
> >>> Configure getopt to accept the -R argument to pwrite.
> >>
> >> Did you have anything against "F" and "B"? :)
> >>
> >> (They're missing too but in the case statements & help)
> > 
> > No, but I /was/ wondering if the omission was deliberate?
> 
> I don't think so; the commit that added it (8fb2237)
> seemed to do everything but add the getopt chars.
> 
> Oh, and however much gets fixed up, probably needs a man page
> update too.

The manpage already has -F/-B/-R, but you're right that I should add
-F and -B to getopt.

--D

> 
> Thanks,
> -Eric
> 
> > --D
> > 
> >>
> >> -Eric
> >>
> >>> Signed-off-by: Darrick J. Wong 
> >>> ---
> >>>  io/pwrite.c |    2 +-
> >>>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>>
> >>>
> >>> diff --git a/io/pwrite.c b/io/pwrite.c
> >>> index fd9114d..c6efee9 100644
> >>> --- a/io/pwrite.c
> >>> +++ b/io/pwrite.c
> >>> @@ -259,7 +259,7 @@ pwrite_f(
> >>>  	init_cvtnum(&fsblocksize, &fssectsize);
> >>>  	bsize = fsblocksize;
> >>>  
> >>> -	while ((c = getopt(argc, argv, "b:Cdf:i:qs:S:uV:wWZ:")) != EOF) {
> >>> +	while ((c = getopt(argc, argv, "b:Cdf:i:qRs:S:uV:wWZ:")) != EOF) {
> >>>  		switch (c) {
> >>>  		case 'b':
> >>>  			tmp = cvtnum(fsblocksize, fssectsize, optarg);
> >>>
> >>> _______________________________________________
> >>> xfs mailing list
> >>> xfs@oss.sgi.com
> >>> http://oss.sgi.com/mailman/listinfo/xfs
> >>>
> >>
> >> _______________________________________________
> >> xfs mailing list
> >> xfs@oss.sgi.com
> >> http://oss.sgi.com/mailman/listinfo/xfs
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> > 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From darrick.wong@oracle.com  Tue Jan 26 18:58:52 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 00F867CA2
	for ; Tue, 26 Jan 2016 18:58:51 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 83F86AC003
	for ; Tue, 26 Jan 2016 16:58:51 -0800 (PST)
X-ASG-Debug-ID: 1453856329-04cb6c0d5ded830001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id 7ZqMjszJcgjyqtHc (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 16:58:49 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0R0wmLA020907
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Wed, 27 Jan 2016 00:58:48 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0R0wl5m022125
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 27 Jan 2016 00:58:48 GMT
Received: from abhmp0004.oracle.com (abhmp0004.oracle.com [141.146.116.10])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0R0wlbF019361;
	Wed, 27 Jan 2016 00:58:47 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Tue, 26 Jan 2016 16:58:46 -0800
Date: Tue, 26 Jan 2016 16:58:45 -0800
From: "Darrick J. Wong" 
To: david@fromorbit.com
Cc: xfs@oss.sgi.com
Subject: [PATCH 1/5 v2] xfs_io: detect the '-R' option in getopt
Message-ID: <20160127005845.GC5852@birch.djwong.org>
X-ASG-Orig-Subj: [PATCH 1/5 v2] xfs_io: detect the '-R' option in getopt
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160123003502.2475.99558.stgit@birch.djwong.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0021.oracle.com [141.146.126.233]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1453856329
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26483
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Configure getopt to accept the -R argument to pwrite.

v2: Accept -F and -B while we're at it.

Signed-off-by: Darrick J. Wong 
---
 io/pwrite.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/io/pwrite.c b/io/pwrite.c
index fd9114d..4fc8de6 100644
--- a/io/pwrite.c
+++ b/io/pwrite.c
@@ -259,7 +259,7 @@ pwrite_f(
 	init_cvtnum(&fsblocksize, &fssectsize);
 	bsize = fsblocksize;
 
-	while ((c = getopt(argc, argv, "b:Cdf:i:qs:S:uV:wWZ:")) != EOF) {
+	while ((c = getopt(argc, argv, "b:BCdf:Fi:qRs:S:uV:wWZ:")) != EOF) {
 		switch (c) {
 		case 'b':
 			tmp = cvtnum(fsblocksize, fssectsize, optarg);

From sandeen@sandeen.net  Tue Jan 26 22:30:31 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 5DC417CA2
	for ; Tue, 26 Jan 2016 22:30:31 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 4F0628F8035
	for ; Tue, 26 Jan 2016 20:30:28 -0800 (PST)
X-ASG-Debug-ID: 1453869025-04cbb026fcf0d40001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id mGXY9xuBCDE2zIqk for ; Tue, 26 Jan 2016 20:30:25 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id EFA76660086B
	for ; Tue, 26 Jan 2016 22:30:24 -0600 (CST)
Subject: Re: [PATCH 1/5 v2] xfs_io: detect the '-R' option in getopt
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH 1/5 v2] xfs_io: detect the '-R' option in getopt
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
 <20160127005845.GC5852@birch.djwong.org>
From: Eric Sandeen 
Message-ID: <56A847E0.3030607@sandeen.net>
Date: Tue, 26 Jan 2016 22:30:24 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160127005845.GC5852@birch.djwong.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453869025
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26489
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Rockin!

Reviewed-by: Eric Sandeen 

On 1/26/16 6:58 PM, Darrick J. Wong wrote:
> Configure getopt to accept the -R argument to pwrite.
> 
> v2: Accept -F and -B while we're at it.
> 
> Signed-off-by: Darrick J. Wong 
> ---
>  io/pwrite.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/io/pwrite.c b/io/pwrite.c
> index fd9114d..4fc8de6 100644
> --- a/io/pwrite.c
> +++ b/io/pwrite.c
> @@ -259,7 +259,7 @@ pwrite_f(
>  	init_cvtnum(&fsblocksize, &fssectsize);
>  	bsize = fsblocksize;
>  
> -	while ((c = getopt(argc, argv, "b:Cdf:i:qs:S:uV:wWZ:")) != EOF) {
> +	while ((c = getopt(argc, argv, "b:BCdf:Fi:qRs:S:uV:wWZ:")) != EOF) {
>  		switch (c) {
>  		case 'b':
>  			tmp = cvtnum(fsblocksize, fssectsize, optarg);
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

From darrick.wong@oracle.com  Tue Jan 26 22:45:01 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 8AC507CA2
	for ; Tue, 26 Jan 2016 22:45:01 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 35958AC004
	for ; Tue, 26 Jan 2016 20:44:58 -0800 (PST)
X-ASG-Debug-ID: 1453869894-04bdf04d6ae8790001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id MrzQW3LR4GnlhuBI (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Jan 2016 20:44:55 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0R4ioEK024699
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Wed, 27 Jan 2016 04:44:50 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0R4ioMs013957
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Wed, 27 Jan 2016 04:44:50 GMT
Received: from abhmp0010.oracle.com (abhmp0010.oracle.com [141.146.116.16])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0R4in79001537;
	Wed, 27 Jan 2016 04:44:50 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Tue, 26 Jan 2016 20:44:49 -0800
Date: Tue, 26 Jan 2016 20:44:48 -0800
From: "Darrick J. Wong" 
To: david@fromorbit.com
Cc: xfs@oss.sgi.com
Subject: [PATCH 6/5] mkfs: factor finobt changes into min log size when
 formatting
Message-ID: <20160127044448.GA20038@birch.djwong.org>
X-ASG-Orig-Subj: [PATCH 6/5] mkfs: factor finobt changes into min log size when
 formatting
References: <20160123003502.2475.99558.stgit@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160123003502.2475.99558.stgit@birch.djwong.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1453869895
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26489
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

Since the finobt affects the size of the log reservations, we need to
be able to include its effects in the calculation of the minimum log
size.

(Not really a problem now, but adding rmapbt will give this one some
bite.)

Signed-off-by: Darrick J. Wong 
---
 mkfs/maxtrres.c |    5 ++++-
 mkfs/xfs_mkfs.c |    2 +-
 mkfs/xfs_mkfs.h |    2 +-
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/mkfs/maxtrres.c b/mkfs/maxtrres.c
index e1d5ee9..b97d020 100644
--- a/mkfs/maxtrres.c
+++ b/mkfs/maxtrres.c
@@ -36,7 +36,8 @@ max_trans_res(
 	int		inodelog,
 	int		dirblocklog,
 	int		logversion,
-	int		log_sunit)
+	int		log_sunit,
+	int		finobt)
 {
 	xfs_sb_t	*sbp;
 	xfs_mount_t	mount;
@@ -68,6 +69,8 @@ max_trans_res(
 			(dirversion == 2 ? XFS_SB_VERSION_DIRV2BIT : 0) |
 			(logversion > 1 ? XFS_SB_VERSION_LOGV2BIT : 0) |
 			XFS_DFL_SB_VERSION_BITS;
+	if (finobt)
+		sbp->sb_features_ro_compat |= XFS_SB_FEAT_RO_COMPAT_FINOBT;
 
 	libxfs_mount(&mount, sbp, 0,0,0,0);
 	maxfsb = xfs_log_calc_minimum_size(&mount);
diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
index 700d12c..4c3a802 100644
--- a/mkfs/xfs_mkfs.c
+++ b/mkfs/xfs_mkfs.c
@@ -2406,7 +2406,7 @@ an AG size that is one stripe unit smaller, for example %llu.\n"),
 
 	min_logblocks = max_trans_res(crcs_enabled, dirversion,
 				   sectorlog, blocklog, inodelog, dirblocklog,
-				   logversion, lsunit);
+				   logversion, lsunit, finobt);
 	ASSERT(min_logblocks);
 	min_logblocks = MAX(XFS_MIN_LOG_BLOCKS, min_logblocks);
 	if (!logsize && dblocks >= (1024*1024*1024) >> blocklog)
diff --git a/mkfs/xfs_mkfs.h b/mkfs/xfs_mkfs.h
index 9df5f37..128068e 100644
--- a/mkfs/xfs_mkfs.h
+++ b/mkfs/xfs_mkfs.h
@@ -84,6 +84,6 @@ extern void res_failed (int err);
 /* maxtrres.c */
 extern int max_trans_res (int crcs_enabled, int dirversion,
 		int sectorlog, int blocklog, int inodelog, int dirblocklog,
-		int logversion, int log_sunit);
+		int logversion, int log_sunit, int finobt);
 
 #endif	/* __XFS_MKFS_H__ */

From bfoster@redhat.com  Wed Jan 27 07:31:15 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 335027CA2
	for ; Wed, 27 Jan 2016 07:31:15 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 07A0E30405F
	for ; Wed, 27 Jan 2016 05:31:14 -0800 (PST)
X-ASG-Debug-ID: 1453901470-04bdf04d6cf5390001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id vPiuoMUmLzeAxErG (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 27 Jan 2016 05:31:10 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id 2E2A57AEA1;
	Wed, 27 Jan 2016 13:31:10 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0RDV9SY019101;
	Wed, 27 Jan 2016 08:31:09 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id A7EDD1251C3; Wed, 27 Jan 2016 08:31:08 -0500 (EST)
Date: Wed, 27 Jan 2016 08:31:08 -0500
From: Brian Foster 
To: "Darrick J. Wong" 
Cc: david@fromorbit.com, xfs@oss.sgi.com
Subject: Re: [PATCH 1/3] xfs: use named array initializers for log item
 dumping
Message-ID: <20160127133106.GB60730@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 1/3] xfs: use named array initializers for log item
 dumping
References: <20160123003425.2338.18133.stgit@birch.djwong.org>
 <20160123003431.2338.77639.stgit@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160123003431.2338.77639.stgit@birch.djwong.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453901470
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Fri, Jan 22, 2016 at 04:34:31PM -0800, Darrick J. Wong wrote:
> Use named array initializers for the string arrays used to dump log
> items, rather than depending on the order being maintained correctly.
> 
> Signed-off-by: Darrick J. Wong 
> ---
>  fs/xfs/xfs_log.c |  132 ++++++++++++++++++++++++++++--------------------------
>  1 file changed, 68 insertions(+), 64 deletions(-)
> 
> 
> diff --git a/fs/xfs/xfs_log.c b/fs/xfs/xfs_log.c
> index 2aa187e..4758f06 100644
> --- a/fs/xfs/xfs_log.c
> +++ b/fs/xfs/xfs_log.c
> @@ -1989,77 +1989,81 @@ xlog_print_tic_res(
>  	uint ophdr_spc = ticket->t_res_num_ophdrs * (uint)sizeof(xlog_op_header_t);
>  
>  	/* match with XLOG_REG_TYPE_* in xfs_log.h */
> -	static char *res_type_str[XLOG_REG_TYPE_MAX] = {
> -	    "bformat",
> -	    "bchunk",
> -	    "efi_format",
> -	    "efd_format",
> -	    "iformat",
> -	    "icore",
> -	    "iext",
> -	    "ibroot",
> -	    "ilocal",
> -	    "iattr_ext",
> -	    "iattr_broot",
> -	    "iattr_local",
> -	    "qformat",
> -	    "dquot",
> -	    "quotaoff",
> -	    "LR header",
> -	    "unmount",
> -	    "commit",
> -	    "trans header"
> +#define REG_TYPE_STR(type, str)	[XLOG_REG_TYPE_##type] = str
> +	static char *res_type_str[XLOG_REG_TYPE_MAX + 1] = {

Maybe we should just fix XLOG_REG_TYPE_MAX to hold the array size (as
XFS_TRANS_TYPE_MAX does)..?

> +	    REG_TYPE_STR(BFORMAT, "bformat"),
> +	    REG_TYPE_STR(BCHUNK, "bchunk"),
> +	    REG_TYPE_STR(EFI_FORMAT, "efi_format"),
> +	    REG_TYPE_STR(EFD_FORMAT, "efd_format"),
> +	    REG_TYPE_STR(IFORMAT, "iformat"),
> +	    REG_TYPE_STR(ICORE, "icore"),
> +	    REG_TYPE_STR(IEXT, "iext"),
> +	    REG_TYPE_STR(IBROOT, "ibroot"),
> +	    REG_TYPE_STR(ILOCAL, "ilocal"),
> +	    REG_TYPE_STR(IATTR_EXT, "iattr_ext"),
> +	    REG_TYPE_STR(IATTR_BROOT, "iattr_broot"),
> +	    REG_TYPE_STR(IATTR_LOCAL, "iattr_local"),
> +	    REG_TYPE_STR(QFORMAT, "qformat"),
> +	    REG_TYPE_STR(DQUOT, "dquot"),
> +	    REG_TYPE_STR(QUOTAOFF, "quotaoff"),
> +	    REG_TYPE_STR(LRHEADER, "LR header"),
> +	    REG_TYPE_STR(UNMOUNT, "unmount"),
> +	    REG_TYPE_STR(COMMIT, "commit"),
> +	    REG_TYPE_STR(TRANSHDR, "trans header"),
> +	    REG_TYPE_STR(ICREATE, "inode create")
>  	};
> +#undef REG_TYPE_STR
> +#define TRANS_TYPE_STR(type)	[XFS_TRANS_##type] = #type
>  	static char *trans_type_str[XFS_TRANS_TYPE_MAX] = {
...
>  	};
> +#undef TRANS_TYPE_STR
>  
>  	xfs_warn(mp, "xlog_write: reservation summary:");
>  	xfs_warn(mp, "  trans type  = %s (%u)",
>  		 ((ticket->t_trans_type <= 0 ||
>  		   ticket->t_trans_type > XFS_TRANS_TYPE_MAX) ?
> -		  "bad-trans-type" : trans_type_str[ticket->t_trans_type-1]),
> +		  "bad-trans-type" : trans_type_str[ticket->t_trans_type]),

Why not incorporate the -1 offset in the new macros and leave these as
is? That avoids a dummy entry at index 0 in each array as well.

Brian

>  		 ticket->t_trans_type);
>  	xfs_warn(mp, "  unit res    = %d bytes",
>  		 ticket->t_unit_res);
> @@ -2078,7 +2082,7 @@ xlog_print_tic_res(
>  		uint r_type = ticket->t_res_arr[i].r_type;
>  		xfs_warn(mp, "region[%u]: %s - %u bytes", i,
>  			    ((r_type <= 0 || r_type > XLOG_REG_TYPE_MAX) ?
> -			    "bad-rtype" : res_type_str[r_type-1]),
> +			    "bad-rtype" : res_type_str[r_type]),
>  			    ticket->t_res_arr[i].r_len);
>  	}
>  
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Wed Jan 27 07:31:16 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 9C7AF7CA2
	for ; Wed, 27 Jan 2016 07:31:16 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 575A78F8035
	for ; Wed, 27 Jan 2016 05:31:16 -0800 (PST)
X-ASG-Debug-ID: 1453901474-04bdf04d6cf53a0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Sbqh3KPusmXqWPCy (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 27 Jan 2016 05:31:15 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	by mx1.redhat.com (Postfix) with ESMTPS id C9DD9AB1;
	Wed, 27 Jan 2016 13:31:14 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0RDVEio012192;
	Wed, 27 Jan 2016 08:31:14 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 9C03F1251C3; Wed, 27 Jan 2016 08:31:13 -0500 (EST)
Date: Wed, 27 Jan 2016 08:31:13 -0500
From: Brian Foster 
To: "Darrick J. Wong" 
Cc: david@fromorbit.com, xfs@oss.sgi.com
Subject: Re: [PATCH 2/3] xfs: move struct xfs_attr_shortform to
 xfs_da_format.h
Message-ID: <20160127133113.GC60730@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 2/3] xfs: move struct xfs_attr_shortform to
 xfs_da_format.h
References: <20160123003425.2338.18133.stgit@birch.djwong.org>
 <20160123003438.2338.51748.stgit@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160123003438.2338.51748.stgit@birch.djwong.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453901475
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Fri, Jan 22, 2016 at 04:34:38PM -0800, Darrick J. Wong wrote:
> Move the shortform attr structure definition to the same place as the
> other attribute structure definitions for consistency and also so that
> xfs/122 verifies the structure size.
> 
> Signed-off-by: Darrick J. Wong 
> ---

Seems fine:

Reviewed-by: Brian Foster 

>  fs/xfs/libxfs/xfs_attr_sf.h    |   16 ----------------
>  fs/xfs/libxfs/xfs_da_format.h  |   16 ++++++++++++++++
>  fs/xfs/libxfs/xfs_inode_fork.c |    1 +
>  3 files changed, 17 insertions(+), 16 deletions(-)
> 
> 
> diff --git a/fs/xfs/libxfs/xfs_attr_sf.h b/fs/xfs/libxfs/xfs_attr_sf.h
> index 919756e..90928bb 100644
> --- a/fs/xfs/libxfs/xfs_attr_sf.h
> +++ b/fs/xfs/libxfs/xfs_attr_sf.h
> @@ -24,22 +24,6 @@
>   * Small attribute lists are packed as tightly as possible so as
>   * to fit into the literal area of the inode.
>   */
> -
> -/*
> - * Entries are packed toward the top as tight as possible.
> - */
> -typedef struct xfs_attr_shortform {
> -	struct xfs_attr_sf_hdr {	/* constant-structure header block */
> -		__be16	totsize;	/* total bytes in shortform list */
> -		__u8	count;	/* count of active entries */
> -	} hdr;
> -	struct xfs_attr_sf_entry {
> -		__uint8_t namelen;	/* actual length of name (no NULL) */
> -		__uint8_t valuelen;	/* actual length of value (no NULL) */
> -		__uint8_t flags;	/* flags bits (see xfs_attr_leaf.h) */
> -		__uint8_t nameval[1];	/* name & value bytes concatenated */
> -	} list[1];			/* variable sized array */
> -} xfs_attr_shortform_t;
>  typedef struct xfs_attr_sf_hdr xfs_attr_sf_hdr_t;
>  typedef struct xfs_attr_sf_entry xfs_attr_sf_entry_t;
>  
> diff --git a/fs/xfs/libxfs/xfs_da_format.h b/fs/xfs/libxfs/xfs_da_format.h
> index b14bbd6..8d4d8bc 100644
> --- a/fs/xfs/libxfs/xfs_da_format.h
> +++ b/fs/xfs/libxfs/xfs_da_format.h
> @@ -641,6 +641,22 @@ xfs_dir2_block_leaf_p(struct xfs_dir2_block_tail *btp)
>   */
>  #define XFS_ATTR_LEAF_MAPSIZE	3	/* how many freespace slots */
>  
> +/*
> + * Entries are packed toward the top as tight as possible.
> + */
> +typedef struct xfs_attr_shortform {
> +	struct xfs_attr_sf_hdr {	/* constant-structure header block */
> +		__be16	totsize;	/* total bytes in shortform list */
> +		__u8	count;	/* count of active entries */
> +	} hdr;
> +	struct xfs_attr_sf_entry {
> +		__uint8_t namelen;	/* actual length of name (no NULL) */
> +		__uint8_t valuelen;	/* actual length of value (no NULL) */
> +		__uint8_t flags;	/* flags bits (see xfs_attr_leaf.h) */
> +		__uint8_t nameval[1];	/* name & value bytes concatenated */
> +	} list[1];			/* variable sized array */
> +} xfs_attr_shortform_t;
> +
>  typedef struct xfs_attr_leaf_map {	/* RLE map of free bytes */
>  	__be16	base;			  /* base of free region */
>  	__be16	size;			  /* length of free region */
> diff --git a/fs/xfs/libxfs/xfs_inode_fork.c b/fs/xfs/libxfs/xfs_inode_fork.c
> index 0defbd0..ef22a78 100644
> --- a/fs/xfs/libxfs/xfs_inode_fork.c
> +++ b/fs/xfs/libxfs/xfs_inode_fork.c
> @@ -31,6 +31,7 @@
>  #include "xfs_error.h"
>  #include "xfs_trace.h"
>  #include "xfs_attr_sf.h"
> +#include "xfs_da_format.h"
>  
>  kmem_zone_t *xfs_ifork_zone;
>  
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From bfoster@redhat.com  Wed Jan 27 07:31:21 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 9AD607CAA
	for ; Wed, 27 Jan 2016 07:31:21 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 6FE5C304066
	for ; Wed, 27 Jan 2016 05:31:21 -0800 (PST)
X-ASG-Debug-ID: 1453901479-04bdf04d6df53b0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id hTY8KNduNkooYNAX (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 27 Jan 2016 05:31:20 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id D70B28E37A;
	Wed, 27 Jan 2016 13:31:19 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0RDVIxm026187;
	Wed, 27 Jan 2016 08:31:18 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id BED2A1251C3; Wed, 27 Jan 2016 08:31:17 -0500 (EST)
Date: Wed, 27 Jan 2016 08:31:17 -0500
From: Brian Foster 
To: "Darrick J. Wong" 
Cc: david@fromorbit.com, xfs@oss.sgi.com
Subject: Re: [PATCH 3/3] xfs: check sizes of XFS on-disk structures at
 compile time
Message-ID: <20160127133117.GD60730@bfoster.bfoster>
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: check sizes of XFS on-disk structures at
 compile time
References: <20160123003425.2338.18133.stgit@birch.djwong.org>
 <20160123003444.2338.37236.stgit@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160123003444.2338.37236.stgit@birch.djwong.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453901480
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Fri, Jan 22, 2016 at 04:34:44PM -0800, Darrick J. Wong wrote:
> Check the sizes of XFS on-disk structures when compiling the kernel.
> Use this to catch inadvertent changes in structure size due to padding
> and alignment issues, etc.
> 
> Signed-off-by: Darrick J. Wong 
> ---
>  fs/xfs/xfs_ondisk.h |  108 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  fs/xfs/xfs_super.c  |    3 +
>  2 files changed, 111 insertions(+)
>  create mode 100644 fs/xfs/xfs_ondisk.h
> 
> 
> diff --git a/fs/xfs/xfs_ondisk.h b/fs/xfs/xfs_ondisk.h
> new file mode 100644
> index 0000000..56fab46
> --- /dev/null
> +++ b/fs/xfs/xfs_ondisk.h
> @@ -0,0 +1,108 @@
> +/*
> + * Copyright (c) 2016 Oracle.
> + * All Rights Reserved.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it would be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write the Free Software Foundation,
> + * Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> + */
> +#ifndef __XFS_ONDISK_H
> +#define __XFS_ONDISK_H
> +
> +#define XFS_CHECK_STRUCT_SIZE(structname, size) \
> +	BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(struct " \

We're printing "struct" in the core message as well as part of the
structname (where appropriate). E.g., "sizeof(struct struct xfs_acl) is
wrong, ..."

It would also be nice to include the current structure size in the
message if it happens to be unexpected. Otherwise, seems fine to me.

Brian

> +		#structname ") is wrong, expected " #size)
> +

> +static inline void __init
> +xfs_check_ondisk_structs(void)
> +{
> +	/* ag/file structures */
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_acl,			4);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_acl_entry,		12);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agf,			224);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agfl,			36);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_agi,			336);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_key,		8);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmbt_rec,		16);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_bmdr_block,		4);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_btree_block,		72);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dinode,		176);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_disk_dquot,		104);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dqblk,			136);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dsb,			264);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dsymlink_hdr,		56);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_key,		4);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_inobt_rec,		16);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_timestamp,		8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_key_t,			8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_ptr_t,			4);
> +	XFS_CHECK_STRUCT_SIZE(xfs_alloc_rec_t,			8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_inobt_ptr_t,			4);
> +
> +	/* dir/attr trees */
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_attr3_leaf_hdr,	80);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_attr3_leafblock,	88);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_attr3_rmt_hdr,		56);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_da3_blkinfo,		56);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_da3_intnode,		64);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_da3_node_hdr,		64);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_blk_hdr,		48);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_data_hdr,		64);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_free,		64);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_free_hdr,		64);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_leaf,		64);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dir3_leaf_hdr,		64);
> +	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_entry_t,		8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_hdr_t,		32);
> +	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_map_t,		4);
> +	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_name_local_t,	4);
> +	XFS_CHECK_STRUCT_SIZE(xfs_attr_leaf_name_remote_t,	12);
> +	XFS_CHECK_STRUCT_SIZE(xfs_attr_leafblock_t,		40);
> +	XFS_CHECK_STRUCT_SIZE(xfs_attr_shortform_t,		8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_da_blkinfo_t,			12);
> +	XFS_CHECK_STRUCT_SIZE(xfs_da_intnode_t,			16);
> +	XFS_CHECK_STRUCT_SIZE(xfs_da_node_entry_t,		8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_da_node_hdr_t,		16);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_data_free_t,		4);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_data_hdr_t,		16);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_data_unused_t,		6);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_free_hdr_t,		16);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_free_t,			16);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_ino4_t,			4);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_ino8_t,			8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_inou_t,			8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_leaf_entry_t,		8);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_leaf_hdr_t,		16);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_leaf_t,			16);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_leaf_tail_t,		4);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_sf_entry_t,		3);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_sf_hdr_t,		10);
> +	XFS_CHECK_STRUCT_SIZE(xfs_dir2_sf_off_t,		2);
> +
> +	/* log structures */
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_32,	28);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_extent_32,		12);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_extent_64,		16);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_icdinode,		176);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_ictimestamp,		8);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_32,	52);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_inode_log_format_64,	56);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
> +	XFS_CHECK_STRUCT_SIZE(struct xfs_trans_header,		16);
> +}
> +
> +#endif /* __XFS_ONDISK_H */
> diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c
> index 36bd882..f63d212 100644
> --- a/fs/xfs/xfs_super.c
> +++ b/fs/xfs/xfs_super.c
> @@ -45,6 +45,7 @@
>  #include "xfs_filestream.h"
>  #include "xfs_quota.h"
>  #include "xfs_sysfs.h"
> +#include "xfs_ondisk.h"
>  
>  #include 
>  #include 
> @@ -1817,6 +1818,8 @@ init_xfs_fs(void)
>  {
>  	int			error;
>  
> +	xfs_check_ondisk_structs();
> +
>  	printk(KERN_INFO XFS_VERSION_STRING " with "
>  			 XFS_BUILD_OPTIONS " enabled\n");
>  
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From cmaiolino@redhat.com  Wed Jan 27 09:37:15 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 353457CA2
	for ; Wed, 27 Jan 2016 09:37:15 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id E8D5E304039
	for ; Wed, 27 Jan 2016 07:37:14 -0800 (PST)
X-ASG-Debug-ID: 1453909029-04bdf04d6cf9180001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id 3CBbNXqQNkZlegLc (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 27 Jan 2016 07:37:09 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id 21D8C552E6;
	Wed, 27 Jan 2016 15:37:09 +0000 (UTC)
Received: from redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0RFb5K4016620
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Wed, 27 Jan 2016 10:37:07 -0500
Date: Wed, 27 Jan 2016 16:37:05 +0100
From: Carlos Maiolino 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] Split default quota limits by quota type
Message-ID: <20160127153705.GA17571@redhat.com>
X-ASG-Orig-Subj: Re: [PATCH] Split default quota limits by quota type
Mail-Followup-To: Eric Sandeen , xfs@oss.sgi.com
References: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
 <569FC41E.2040300@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <569FC41E.2040300@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453909029
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

> > 
> > This patch aims to split the default quota value by quota type. Allowing each
> > quota type having different default values.
> > 
> > Default time limits are still set globally, but I don't mind to split them by
> > quota type too.
> 
> Hm, I guess it seems like it should be done; otherwise it's a weird
> caveat, isn't it?  "Default limits are set by type, but timers are
> inherited from whatever first default quota is found across all types"
> 
> So yeah, seems like it should be done for timers as well, IMHO;
> grace periods can be set for each default quota type, so they should
> be honored.
> 

Ok, I was just working on implement it, but honestly, I don't see the point now
in split time limits by user/group/project.

grace periods are set globally by default. We don't have specific quota grace
limits for each user or each group. Just a single value for them.

So, if we can not specify an individual grace period per-user or per-group, what
is the point in having these time limits split by user/group/project? We could
have the values divided by user/group/proj, but what's the sense on it? If we
have a default grace limit for users, it will be set to all users, if we have a
grace limit for groups, it will be enforced to all users anyway (since we can't
specify the group here either).

So, I wonder, does it make sense?

Looks a nice idea for the future, but for the current implementation it doesn't
make sense to me. But sure, let me know if I'm wrong :)

Cheers

> > Signed-off-by: Carlos Maiolino 
> 
> Really just minor comments below; there is one code comment that needs to
> be removed.
> 
> Thanks!
> -Eric
> 
> > ---
> >  fs/xfs/xfs_dquot.c       | 26 +++++++------
> >  fs/xfs/xfs_qm.c          | 96 +++++++++++++++++++++++++++---------------------
> >  fs/xfs/xfs_qm.h          | 34 ++++++++++++++---
> >  fs/xfs/xfs_qm_syscalls.c | 15 +++++---
> >  fs/xfs/xfs_trans_dquot.c | 15 +++++---
> >  5 files changed, 114 insertions(+), 72 deletions(-)
> > 
> > diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
> > index 9c44d38..23f551b 100644
> > --- a/fs/xfs/xfs_dquot.c
> > +++ b/fs/xfs/xfs_dquot.c
> > @@ -92,26 +92,28 @@ xfs_qm_adjust_dqlimits(
> >  {
> >  	struct xfs_quotainfo	*q = mp->m_quotainfo;
> >  	struct xfs_disk_dquot	*d = &dq->q_core;
> > +	struct xfs_def_quota	*defq;
> >  	int			prealloc = 0;
> >  
> >  	ASSERT(d->d_id);
> > +	defq = xfs_get_defquota(dq, q);
> >  
> > -	if (q->qi_bsoftlimit && !d->d_blk_softlimit) {
> > -		d->d_blk_softlimit = cpu_to_be64(q->qi_bsoftlimit);
> > +	if (defq->bsoftlimit && !d->d_blk_softlimit) {
> > +		d->d_blk_softlimit = cpu_to_be64(defq->bsoftlimit);
> >  		prealloc = 1;
> >  	}
> > -	if (q->qi_bhardlimit && !d->d_blk_hardlimit) {
> > -		d->d_blk_hardlimit = cpu_to_be64(q->qi_bhardlimit);
> > +	if (defq->bhardlimit && !d->d_blk_hardlimit) {
> > +		d->d_blk_hardlimit = cpu_to_be64(defq->bhardlimit);
> >  		prealloc = 1;
> >  	}
> > -	if (q->qi_isoftlimit && !d->d_ino_softlimit)
> > -		d->d_ino_softlimit = cpu_to_be64(q->qi_isoftlimit);
> > -	if (q->qi_ihardlimit && !d->d_ino_hardlimit)
> > -		d->d_ino_hardlimit = cpu_to_be64(q->qi_ihardlimit);
> > -	if (q->qi_rtbsoftlimit && !d->d_rtb_softlimit)
> > -		d->d_rtb_softlimit = cpu_to_be64(q->qi_rtbsoftlimit);
> > -	if (q->qi_rtbhardlimit && !d->d_rtb_hardlimit)
> > -		d->d_rtb_hardlimit = cpu_to_be64(q->qi_rtbhardlimit);
> > +	if (defq->isoftlimit && !d->d_ino_softlimit)
> > +		d->d_ino_softlimit = cpu_to_be64(defq->isoftlimit);
> > +	if (defq->ihardlimit && !d->d_ino_hardlimit)
> > +		d->d_ino_hardlimit = cpu_to_be64(defq->ihardlimit);
> > +	if (defq->rtbsoftlimit && !d->d_rtb_softlimit)
> > +		d->d_rtb_softlimit = cpu_to_be64(defq->rtbsoftlimit);
> > +	if (defq->rtbhardlimit && !d->d_rtb_hardlimit)
> > +		d->d_rtb_hardlimit = cpu_to_be64(defq->rtbhardlimit);
> >  
> >  	if (prealloc)
> >  		xfs_dquot_set_prealloc_limits(dq);
> > diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
> > index 532ab79..1bcb733 100644
> > --- a/fs/xfs/xfs_qm.c
> > +++ b/fs/xfs/xfs_qm.c
> > @@ -560,6 +560,54 @@ xfs_qm_shrink_count(
> >  	return list_lru_shrink_count(&qi->qi_lru, sc);
> >  }
> >  
> > +STATIC int
> > +xfs_qm_set_defquota(
> 
> It'd be a little easier to review if this were simply factored
> out in patch 1, then default quotas handled in patch 2, I think,
> but not that big a deal.
> 
> > +	xfs_mount_t	*mp,
> > +	uint		type,
> > +	xfs_quotainfo_t	*qinf)
> > +{
> > +	xfs_dquot_t		*dqp;
> > +	struct xfs_def_quota    *defq;
> > +	int			error;
> > +
> > +	error = xfs_qm_dqread(mp, 0, type, XFS_QMOPT_DOWARN, &dqp);
> > +
> > +	if (!error) {
> > +		xfs_disk_dquot_t        *ddqp = &dqp->q_core;
> > +
> > +		defq = xfs_get_defquota(dqp, qinf);
> > +
> > +		qinf->qi_btimelimit = ddqp->d_btimer ?
> > +			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
> > +		qinf->qi_itimelimit = ddqp->d_itimer ?
> > +			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
> > +		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
> > +			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
> > +		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
> > +			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
> > +		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
> > +			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
> > +		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
> > +			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
> > +		defq->bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
> > +		defq->bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
> > +		defq->ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
> > +		defq->isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
> > +		defq->rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
> > +		defq->rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
> > +		xfs_qm_dqdestroy(dqp);
> > +	} else {
> > +		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
> > +		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
> > +		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
> > +		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
> > +		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
> > +		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
> > +	}
> > +
> > +	return error;
> > +}
> > +
> >  /*
> >   * This initializes all the quota information that's kept in the
> >   * mount structure
> > @@ -570,7 +618,6 @@ xfs_qm_init_quotainfo(
> >  {
> >  	xfs_quotainfo_t *qinf;
> >  	int		error;
> > -	xfs_dquot_t	*dqp;
> >  
> >  	ASSERT(XFS_IS_QUOTA_RUNNING(mp));
> >  
> > @@ -614,47 +661,12 @@ xfs_qm_init_quotainfo(
> 
> There is a comment above this that should go away now:
> 
>          * We look at the USR dquot with id == 0 first, but if user quotas
>          * are not enabled we goto the GRP dquot with id == 0.
>          * We don't really care to keep separate default limits for user
>          * and group quotas, at least not at this point.
> 
> 
> >  	 * Since we may not have done a quotacheck by this point, just read
> >  	 * the dquot without attaching it to any hashtables or lists.
> >  	 */
> > -	error = xfs_qm_dqread(mp, 0,
> > -			XFS_IS_UQUOTA_RUNNING(mp) ? XFS_DQ_USER :
> > -			 (XFS_IS_GQUOTA_RUNNING(mp) ? XFS_DQ_GROUP :
> > -			  XFS_DQ_PROJ),
> > -			XFS_QMOPT_DOWARN, &dqp);
> > -	if (!error) {
> > -		xfs_disk_dquot_t	*ddqp = &dqp->q_core;
> > -
> > -		/*
> > -		 * The warnings and timers set the grace period given to
> > -		 * a user or group before he or she can not perform any
> > -		 * more writing. If it is zero, a default is used.
> > -		 */
> > -		qinf->qi_btimelimit = ddqp->d_btimer ?
> > -			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
> > -		qinf->qi_itimelimit = ddqp->d_itimer ?
> > -			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
> > -		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
> > -			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
> > -		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
> > -			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
> > -		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
> > -			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
> > -		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
> > -			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
> > -		qinf->qi_bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
> > -		qinf->qi_bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
> > -		qinf->qi_ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
> > -		qinf->qi_isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
> > -		qinf->qi_rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
> > -		qinf->qi_rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
> > -
> > -		xfs_qm_dqdestroy(dqp);
> > -	} else {
> > -		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
> > -		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
> > -		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
> > -		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
> > -		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
> > -		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
> > -	}
> 
> 
> 
> > +	if (XFS_IS_UQUOTA_RUNNING(mp))
> > +		error = xfs_qm_set_defquota(mp, XFS_DQ_USER, qinf);
> 
> Hm, I guess the rest of the function looks at RUNNING not ENFORCED;
> for some reason I thought it would be ENFORCED, but *shrug* I guess
> it's right!
> 
> > +	if (XFS_IS_GQUOTA_RUNNING(mp))
> > +		error = xfs_qm_set_defquota(mp, XFS_DQ_GROUP, qinf);
> > +	if (XFS_IS_PQUOTA_RUNNING(mp))
> > +		error = xfs_qm_set_defquota(mp, XFS_DQ_PROJ, qinf);
> >  
> >  	qinf->qi_shrinker.count_objects = xfs_qm_shrink_count;
> >  	qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan;
> > diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
> > index 996a040..45e2c36 100644
> > --- a/fs/xfs/xfs_qm.h
> > +++ b/fs/xfs/xfs_qm.h
> > @@ -53,6 +53,15 @@ extern struct kmem_zone	*xfs_qm_dqtrxzone;
> >   */
> >  #define XFS_DQUOT_CLUSTER_SIZE_FSB	(xfs_filblks_t)1
> >  
> > +struct xfs_def_quota {
> > +	xfs_qcnt_t       bhardlimit;     /* default data blk hard limit */
> > +	xfs_qcnt_t       bsoftlimit;	 /* default data blk soft limit */
> > +	xfs_qcnt_t       ihardlimit;	 /* default inode count hard limit */
> > +	xfs_qcnt_t       isoftlimit;	 /* default inode count soft limit */
> > +	xfs_qcnt_t	 rtbhardlimit;   /* default realtime blk hard limit */
> > +	xfs_qcnt_t	 rtbsoftlimit;   /* default realtime blk soft limit */
> > +};
> > +
> >  /*
> >   * Various quota information for individual filesystems.
> >   * The mount structure keeps a pointer to this.
> > @@ -76,12 +85,9 @@ typedef struct xfs_quotainfo {
> >  	struct mutex	 qi_quotaofflock;/* to serialize quotaoff */
> >  	xfs_filblks_t	 qi_dqchunklen;	 /* # BBs in a chunk of dqs */
> >  	uint		 qi_dqperchunk;	 /* # ondisk dqs in above chunk */
> > -	xfs_qcnt_t	 qi_bhardlimit;	 /* default data blk hard limit */
> > -	xfs_qcnt_t	 qi_bsoftlimit;	 /* default data blk soft limit */
> > -	xfs_qcnt_t	 qi_ihardlimit;	 /* default inode count hard limit */
> > -	xfs_qcnt_t	 qi_isoftlimit;	 /* default inode count soft limit */
> > -	xfs_qcnt_t	 qi_rtbhardlimit;/* default realtime blk hard limit */
> > -	xfs_qcnt_t	 qi_rtbsoftlimit;/* default realtime blk soft limit */
> > +	struct xfs_def_quota	qi_usr_default;
> > +	struct xfs_def_quota	qi_grp_default;
> > +	struct xfs_def_quota	qi_prj_default;
> >  	struct shrinker  qi_shrinker;
> >  } xfs_quotainfo_t;
> >  
> > @@ -171,4 +177,20 @@ extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
> >  extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
> >  extern int		xfs_qm_scall_quotaoff(struct xfs_mount *, uint);
> >  
> > +static inline struct xfs_def_quota *
> > +xfs_get_defquota(struct xfs_dquot *dqp, struct xfs_quotainfo *qi)
> > +{
> > +	struct xfs_def_quota *defq;
> > +
> > +	if (XFS_QM_ISUDQ(dqp))
> > +		defq = &qi->qi_usr_default;
> > +	else if (XFS_QM_ISGDQ(dqp))
> > +		defq = &qi->qi_grp_default;
> > +	else {
> > +		ASSERT(XFS_QM_ISPDQ(dqp));
> > +		defq = &qi->qi_prj_default;
> > +	}
> > +	return defq;
> > +}
> > +
> >  #endif /* __XFS_QM_H__ */
> > diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
> > index 3640c6e..31830f0 100644
> > --- a/fs/xfs/xfs_qm_syscalls.c
> > +++ b/fs/xfs/xfs_qm_syscalls.c
> > @@ -404,6 +404,7 @@ xfs_qm_scall_setqlim(
> >  	struct xfs_disk_dquot	*ddq;
> >  	struct xfs_dquot	*dqp;
> >  	struct xfs_trans	*tp;
> > +	struct xfs_def_quota	*defq;
> >  	int			error;
> >  	xfs_qcnt_t		hard, soft;
> >  
> > @@ -431,6 +432,8 @@ xfs_qm_scall_setqlim(
> >  		ASSERT(error != -ENOENT);
> >  		goto out_unlock;
> >  	}
> > +
> > +	defq = xfs_get_defquota(dqp, q);
> >  	xfs_dqunlock(dqp);
> >  
> >  	tp = xfs_trans_alloc(mp, XFS_TRANS_QM_SETQLIM);
> > @@ -458,8 +461,8 @@ xfs_qm_scall_setqlim(
> >  		ddq->d_blk_softlimit = cpu_to_be64(soft);
> >  		xfs_dquot_set_prealloc_limits(dqp);
> >  		if (id == 0) {
> > -			q->qi_bhardlimit = hard;
> > -			q->qi_bsoftlimit = soft;
> > +			defq->bhardlimit = hard;
> > +			defq->bsoftlimit = soft;
> >  		}
> >  	} else {
> >  		xfs_debug(mp, "blkhard %Ld < blksoft %Ld", hard, soft);
> > @@ -474,8 +477,8 @@ xfs_qm_scall_setqlim(
> >  		ddq->d_rtb_hardlimit = cpu_to_be64(hard);
> >  		ddq->d_rtb_softlimit = cpu_to_be64(soft);
> >  		if (id == 0) {
> > -			q->qi_rtbhardlimit = hard;
> > -			q->qi_rtbsoftlimit = soft;
> > +			defq->rtbhardlimit = hard;
> > +			defq->rtbsoftlimit = soft;
> >  		}
> >  	} else {
> >  		xfs_debug(mp, "rtbhard %Ld < rtbsoft %Ld", hard, soft);
> > @@ -491,8 +494,8 @@ xfs_qm_scall_setqlim(
> >  		ddq->d_ino_hardlimit = cpu_to_be64(hard);
> >  		ddq->d_ino_softlimit = cpu_to_be64(soft);
> >  		if (id == 0) {
> > -			q->qi_ihardlimit = hard;
> > -			q->qi_isoftlimit = soft;
> > +			defq->ihardlimit = hard;
> > +			defq->isoftlimit = soft;
> >  		}
> >  	} else {
> >  		xfs_debug(mp, "ihard %Ld < isoft %Ld", hard, soft);
> > diff --git a/fs/xfs/xfs_trans_dquot.c b/fs/xfs/xfs_trans_dquot.c
> > index 9951701..c3d5472 100644
> > --- a/fs/xfs/xfs_trans_dquot.c
> > +++ b/fs/xfs/xfs_trans_dquot.c
> > @@ -609,17 +609,20 @@ xfs_trans_dqresv(
> >  	xfs_qcnt_t	total_count;
> >  	xfs_qcnt_t	*resbcountp;
> >  	xfs_quotainfo_t	*q = mp->m_quotainfo;
> > +	struct xfs_def_quota	*defq;
> >  
> >  
> >  	xfs_dqlock(dqp);
> >  
> > +	defq = xfs_get_defquota(dqp, q);
> > +
> >  	if (flags & XFS_TRANS_DQ_RES_BLKS) {
> >  		hardlimit = be64_to_cpu(dqp->q_core.d_blk_hardlimit);
> >  		if (!hardlimit)
> > -			hardlimit = q->qi_bhardlimit;
> > +			hardlimit = defq->bhardlimit;
> >  		softlimit = be64_to_cpu(dqp->q_core.d_blk_softlimit);
> >  		if (!softlimit)
> > -			softlimit = q->qi_bsoftlimit;
> > +			softlimit = defq->bsoftlimit;
> >  		timer = be32_to_cpu(dqp->q_core.d_btimer);
> >  		warns = be16_to_cpu(dqp->q_core.d_bwarns);
> >  		warnlimit = dqp->q_mount->m_quotainfo->qi_bwarnlimit;
> > @@ -628,10 +631,10 @@ xfs_trans_dqresv(
> >  		ASSERT(flags & XFS_TRANS_DQ_RES_RTBLKS);
> >  		hardlimit = be64_to_cpu(dqp->q_core.d_rtb_hardlimit);
> >  		if (!hardlimit)
> > -			hardlimit = q->qi_rtbhardlimit;
> > +			hardlimit = defq->rtbhardlimit;
> >  		softlimit = be64_to_cpu(dqp->q_core.d_rtb_softlimit);
> >  		if (!softlimit)
> > -			softlimit = q->qi_rtbsoftlimit;
> > +			softlimit = defq->rtbsoftlimit;
> >  		timer = be32_to_cpu(dqp->q_core.d_rtbtimer);
> >  		warns = be16_to_cpu(dqp->q_core.d_rtbwarns);
> >  		warnlimit = dqp->q_mount->m_quotainfo->qi_rtbwarnlimit;
> > @@ -672,10 +675,10 @@ xfs_trans_dqresv(
> >  			warnlimit = dqp->q_mount->m_quotainfo->qi_iwarnlimit;
> >  			hardlimit = be64_to_cpu(dqp->q_core.d_ino_hardlimit);
> >  			if (!hardlimit)
> > -				hardlimit = q->qi_ihardlimit;
> > +				hardlimit = defq->ihardlimit;
> >  			softlimit = be64_to_cpu(dqp->q_core.d_ino_softlimit);
> >  			if (!softlimit)
> > -				softlimit = q->qi_isoftlimit;
> > +				softlimit = defq->isoftlimit;
> >  
> >  			if (hardlimit && total_count > hardlimit) {
> >  				xfs_quota_warn(mp, dqp, QUOTA_NL_IHARDWARN);
> > 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

From cmaiolino@redhat.com  Wed Jan 27 10:58:11 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 383B37CA2
	for ; Wed, 27 Jan 2016 10:58:11 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 18ECF304039
	for ; Wed, 27 Jan 2016 08:58:07 -0800 (PST)
X-ASG-Debug-ID: 1453913885-04cb6c0d5e1025a0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id Wr6Ura1QIumZQcld (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 27 Jan 2016 08:58:06 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	by mx1.redhat.com (Postfix) with ESMTPS id 3F244C00230B
	for ; Wed, 27 Jan 2016 16:58:05 +0000 (UTC)
Received: from zion.usersys.redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0RGw3xd024669
	for ; Wed, 27 Jan 2016 11:58:04 -0500
From: Carlos Maiolino 
To: xfs@oss.sgi.com
Subject: [PATCH] xfs: split default quota limits by quota type V2
Date: Wed, 27 Jan 2016 17:57:58 +0100
X-ASG-Orig-Subj: [PATCH] xfs: split default quota limits by quota type V2
Message-Id: <1453913878-30900-1-git-send-email-cmaiolino@redhat.com>
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453913886
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

Default quotas are globally set due historical reasons. IRIX only supported user
and project quotas, and default quota was only applied to user quotas.

In Linux, when a default quota is set, all different quota types inherits the
same default value.

An user with a quota limit larger than the default quota value, will still be
limited to the default value because the group quotas also inherits the default
quotas. Unless the group which the user belongs to have a custom quota limit
set.

This patch aims to split the default quota value by quota type. Allowing each
quota type having different default values.

Default time limits are still set globally. XFS does not set a per-user/group
timer, but a single global timer, so, there is no reason to split time limits
too.

Changelog:

	V2 - Remove comment about old behavior

Signed-off-by: Carlos Maiolino 
---
 fs/xfs/xfs_dquot.c       |  26 ++++++------
 fs/xfs/xfs_qm.c          | 101 +++++++++++++++++++++++++----------------------
 fs/xfs/xfs_qm.h          |  34 +++++++++++++---
 fs/xfs/xfs_qm_syscalls.c |  15 ++++---
 fs/xfs/xfs_trans_dquot.c |  15 ++++---
 5 files changed, 114 insertions(+), 77 deletions(-)

diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 9c44d38..23f551b 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -92,26 +92,28 @@ xfs_qm_adjust_dqlimits(
 {
 	struct xfs_quotainfo	*q = mp->m_quotainfo;
 	struct xfs_disk_dquot	*d = &dq->q_core;
+	struct xfs_def_quota	*defq;
 	int			prealloc = 0;
 
 	ASSERT(d->d_id);
+	defq = xfs_get_defquota(dq, q);
 
-	if (q->qi_bsoftlimit && !d->d_blk_softlimit) {
-		d->d_blk_softlimit = cpu_to_be64(q->qi_bsoftlimit);
+	if (defq->bsoftlimit && !d->d_blk_softlimit) {
+		d->d_blk_softlimit = cpu_to_be64(defq->bsoftlimit);
 		prealloc = 1;
 	}
-	if (q->qi_bhardlimit && !d->d_blk_hardlimit) {
-		d->d_blk_hardlimit = cpu_to_be64(q->qi_bhardlimit);
+	if (defq->bhardlimit && !d->d_blk_hardlimit) {
+		d->d_blk_hardlimit = cpu_to_be64(defq->bhardlimit);
 		prealloc = 1;
 	}
-	if (q->qi_isoftlimit && !d->d_ino_softlimit)
-		d->d_ino_softlimit = cpu_to_be64(q->qi_isoftlimit);
-	if (q->qi_ihardlimit && !d->d_ino_hardlimit)
-		d->d_ino_hardlimit = cpu_to_be64(q->qi_ihardlimit);
-	if (q->qi_rtbsoftlimit && !d->d_rtb_softlimit)
-		d->d_rtb_softlimit = cpu_to_be64(q->qi_rtbsoftlimit);
-	if (q->qi_rtbhardlimit && !d->d_rtb_hardlimit)
-		d->d_rtb_hardlimit = cpu_to_be64(q->qi_rtbhardlimit);
+	if (defq->isoftlimit && !d->d_ino_softlimit)
+		d->d_ino_softlimit = cpu_to_be64(defq->isoftlimit);
+	if (defq->ihardlimit && !d->d_ino_hardlimit)
+		d->d_ino_hardlimit = cpu_to_be64(defq->ihardlimit);
+	if (defq->rtbsoftlimit && !d->d_rtb_softlimit)
+		d->d_rtb_softlimit = cpu_to_be64(defq->rtbsoftlimit);
+	if (defq->rtbhardlimit && !d->d_rtb_hardlimit)
+		d->d_rtb_hardlimit = cpu_to_be64(defq->rtbhardlimit);
 
 	if (prealloc)
 		xfs_dquot_set_prealloc_limits(dq);
diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
index 532ab79..6fa038c 100644
--- a/fs/xfs/xfs_qm.c
+++ b/fs/xfs/xfs_qm.c
@@ -560,6 +560,54 @@ xfs_qm_shrink_count(
 	return list_lru_shrink_count(&qi->qi_lru, sc);
 }
 
+STATIC int
+xfs_qm_set_defquota(
+	xfs_mount_t	*mp,
+	uint		type,
+	xfs_quotainfo_t	*qinf)
+{
+	xfs_dquot_t		*dqp;
+	struct xfs_def_quota    *defq;
+	int			error;
+
+	error = xfs_qm_dqread(mp, 0, type, XFS_QMOPT_DOWARN, &dqp);
+
+	if (!error) {
+		xfs_disk_dquot_t        *ddqp = &dqp->q_core;
+
+		defq = xfs_get_defquota(dqp, qinf);
+
+		qinf->qi_btimelimit = ddqp->d_btimer ?
+			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
+		qinf->qi_itimelimit = ddqp->d_itimer ?
+			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
+		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
+			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
+		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
+			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
+		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
+			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
+		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
+			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
+		defq->bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
+		defq->bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
+		defq->ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
+		defq->isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
+		defq->rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
+		defq->rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
+		xfs_qm_dqdestroy(dqp);
+	} else {
+		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
+		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
+		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
+		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
+		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
+		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
+	}
+
+	return error;
+}
+
 /*
  * This initializes all the quota information that's kept in the
  * mount structure
@@ -570,7 +618,6 @@ xfs_qm_init_quotainfo(
 {
 	xfs_quotainfo_t *qinf;
 	int		error;
-	xfs_dquot_t	*dqp;
 
 	ASSERT(XFS_IS_QUOTA_RUNNING(mp));
 
@@ -606,55 +653,15 @@ xfs_qm_init_quotainfo(
 	 * We try to get the limits from the superuser's limits fields.
 	 * This is quite hacky, but it is standard quota practice.
 	 *
-	 * We look at the USR dquot with id == 0 first, but if user quotas
-	 * are not enabled we goto the GRP dquot with id == 0.
-	 * We don't really care to keep separate default limits for user
-	 * and group quotas, at least not at this point.
-	 *
 	 * Since we may not have done a quotacheck by this point, just read
 	 * the dquot without attaching it to any hashtables or lists.
 	 */
-	error = xfs_qm_dqread(mp, 0,
-			XFS_IS_UQUOTA_RUNNING(mp) ? XFS_DQ_USER :
-			 (XFS_IS_GQUOTA_RUNNING(mp) ? XFS_DQ_GROUP :
-			  XFS_DQ_PROJ),
-			XFS_QMOPT_DOWARN, &dqp);
-	if (!error) {
-		xfs_disk_dquot_t	*ddqp = &dqp->q_core;
-
-		/*
-		 * The warnings and timers set the grace period given to
-		 * a user or group before he or she can not perform any
-		 * more writing. If it is zero, a default is used.
-		 */
-		qinf->qi_btimelimit = ddqp->d_btimer ?
-			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
-		qinf->qi_itimelimit = ddqp->d_itimer ?
-			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
-		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
-			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
-		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
-			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
-		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
-			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
-		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
-			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
-		qinf->qi_bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
-		qinf->qi_bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
-		qinf->qi_ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
-		qinf->qi_isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
-		qinf->qi_rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
-		qinf->qi_rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
-
-		xfs_qm_dqdestroy(dqp);
-	} else {
-		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
-		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
-		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
-		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
-		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
-		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
-	}
+	if (XFS_IS_UQUOTA_RUNNING(mp))
+		error = xfs_qm_set_defquota(mp, XFS_DQ_USER, qinf);
+	if (XFS_IS_GQUOTA_RUNNING(mp))
+		error = xfs_qm_set_defquota(mp, XFS_DQ_GROUP, qinf);
+	if (XFS_IS_PQUOTA_RUNNING(mp))
+		error = xfs_qm_set_defquota(mp, XFS_DQ_PROJ, qinf);
 
 	qinf->qi_shrinker.count_objects = xfs_qm_shrink_count;
 	qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan;
diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
index 996a040..45e2c36 100644
--- a/fs/xfs/xfs_qm.h
+++ b/fs/xfs/xfs_qm.h
@@ -53,6 +53,15 @@ extern struct kmem_zone	*xfs_qm_dqtrxzone;
  */
 #define XFS_DQUOT_CLUSTER_SIZE_FSB	(xfs_filblks_t)1
 
+struct xfs_def_quota {
+	xfs_qcnt_t       bhardlimit;     /* default data blk hard limit */
+	xfs_qcnt_t       bsoftlimit;	 /* default data blk soft limit */
+	xfs_qcnt_t       ihardlimit;	 /* default inode count hard limit */
+	xfs_qcnt_t       isoftlimit;	 /* default inode count soft limit */
+	xfs_qcnt_t	 rtbhardlimit;   /* default realtime blk hard limit */
+	xfs_qcnt_t	 rtbsoftlimit;   /* default realtime blk soft limit */
+};
+
 /*
  * Various quota information for individual filesystems.
  * The mount structure keeps a pointer to this.
@@ -76,12 +85,9 @@ typedef struct xfs_quotainfo {
 	struct mutex	 qi_quotaofflock;/* to serialize quotaoff */
 	xfs_filblks_t	 qi_dqchunklen;	 /* # BBs in a chunk of dqs */
 	uint		 qi_dqperchunk;	 /* # ondisk dqs in above chunk */
-	xfs_qcnt_t	 qi_bhardlimit;	 /* default data blk hard limit */
-	xfs_qcnt_t	 qi_bsoftlimit;	 /* default data blk soft limit */
-	xfs_qcnt_t	 qi_ihardlimit;	 /* default inode count hard limit */
-	xfs_qcnt_t	 qi_isoftlimit;	 /* default inode count soft limit */
-	xfs_qcnt_t	 qi_rtbhardlimit;/* default realtime blk hard limit */
-	xfs_qcnt_t	 qi_rtbsoftlimit;/* default realtime blk soft limit */
+	struct xfs_def_quota	qi_usr_default;
+	struct xfs_def_quota	qi_grp_default;
+	struct xfs_def_quota	qi_prj_default;
 	struct shrinker  qi_shrinker;
 } xfs_quotainfo_t;
 
@@ -171,4 +177,20 @@ extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
 extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
 extern int		xfs_qm_scall_quotaoff(struct xfs_mount *, uint);
 
+static inline struct xfs_def_quota *
+xfs_get_defquota(struct xfs_dquot *dqp, struct xfs_quotainfo *qi)
+{
+	struct xfs_def_quota *defq;
+
+	if (XFS_QM_ISUDQ(dqp))
+		defq = &qi->qi_usr_default;
+	else if (XFS_QM_ISGDQ(dqp))
+		defq = &qi->qi_grp_default;
+	else {
+		ASSERT(XFS_QM_ISPDQ(dqp));
+		defq = &qi->qi_prj_default;
+	}
+	return defq;
+}
+
 #endif /* __XFS_QM_H__ */
diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
index 3640c6e..31830f0 100644
--- a/fs/xfs/xfs_qm_syscalls.c
+++ b/fs/xfs/xfs_qm_syscalls.c
@@ -404,6 +404,7 @@ xfs_qm_scall_setqlim(
 	struct xfs_disk_dquot	*ddq;
 	struct xfs_dquot	*dqp;
 	struct xfs_trans	*tp;
+	struct xfs_def_quota	*defq;
 	int			error;
 	xfs_qcnt_t		hard, soft;
 
@@ -431,6 +432,8 @@ xfs_qm_scall_setqlim(
 		ASSERT(error != -ENOENT);
 		goto out_unlock;
 	}
+
+	defq = xfs_get_defquota(dqp, q);
 	xfs_dqunlock(dqp);
 
 	tp = xfs_trans_alloc(mp, XFS_TRANS_QM_SETQLIM);
@@ -458,8 +461,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_blk_softlimit = cpu_to_be64(soft);
 		xfs_dquot_set_prealloc_limits(dqp);
 		if (id == 0) {
-			q->qi_bhardlimit = hard;
-			q->qi_bsoftlimit = soft;
+			defq->bhardlimit = hard;
+			defq->bsoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "blkhard %Ld < blksoft %Ld", hard, soft);
@@ -474,8 +477,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_rtb_hardlimit = cpu_to_be64(hard);
 		ddq->d_rtb_softlimit = cpu_to_be64(soft);
 		if (id == 0) {
-			q->qi_rtbhardlimit = hard;
-			q->qi_rtbsoftlimit = soft;
+			defq->rtbhardlimit = hard;
+			defq->rtbsoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "rtbhard %Ld < rtbsoft %Ld", hard, soft);
@@ -491,8 +494,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_ino_hardlimit = cpu_to_be64(hard);
 		ddq->d_ino_softlimit = cpu_to_be64(soft);
 		if (id == 0) {
-			q->qi_ihardlimit = hard;
-			q->qi_isoftlimit = soft;
+			defq->ihardlimit = hard;
+			defq->isoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "ihard %Ld < isoft %Ld", hard, soft);
diff --git a/fs/xfs/xfs_trans_dquot.c b/fs/xfs/xfs_trans_dquot.c
index 9951701..c3d5472 100644
--- a/fs/xfs/xfs_trans_dquot.c
+++ b/fs/xfs/xfs_trans_dquot.c
@@ -609,17 +609,20 @@ xfs_trans_dqresv(
 	xfs_qcnt_t	total_count;
 	xfs_qcnt_t	*resbcountp;
 	xfs_quotainfo_t	*q = mp->m_quotainfo;
+	struct xfs_def_quota	*defq;
 
 
 	xfs_dqlock(dqp);
 
+	defq = xfs_get_defquota(dqp, q);
+
 	if (flags & XFS_TRANS_DQ_RES_BLKS) {
 		hardlimit = be64_to_cpu(dqp->q_core.d_blk_hardlimit);
 		if (!hardlimit)
-			hardlimit = q->qi_bhardlimit;
+			hardlimit = defq->bhardlimit;
 		softlimit = be64_to_cpu(dqp->q_core.d_blk_softlimit);
 		if (!softlimit)
-			softlimit = q->qi_bsoftlimit;
+			softlimit = defq->bsoftlimit;
 		timer = be32_to_cpu(dqp->q_core.d_btimer);
 		warns = be16_to_cpu(dqp->q_core.d_bwarns);
 		warnlimit = dqp->q_mount->m_quotainfo->qi_bwarnlimit;
@@ -628,10 +631,10 @@ xfs_trans_dqresv(
 		ASSERT(flags & XFS_TRANS_DQ_RES_RTBLKS);
 		hardlimit = be64_to_cpu(dqp->q_core.d_rtb_hardlimit);
 		if (!hardlimit)
-			hardlimit = q->qi_rtbhardlimit;
+			hardlimit = defq->rtbhardlimit;
 		softlimit = be64_to_cpu(dqp->q_core.d_rtb_softlimit);
 		if (!softlimit)
-			softlimit = q->qi_rtbsoftlimit;
+			softlimit = defq->rtbsoftlimit;
 		timer = be32_to_cpu(dqp->q_core.d_rtbtimer);
 		warns = be16_to_cpu(dqp->q_core.d_rtbwarns);
 		warnlimit = dqp->q_mount->m_quotainfo->qi_rtbwarnlimit;
@@ -672,10 +675,10 @@ xfs_trans_dqresv(
 			warnlimit = dqp->q_mount->m_quotainfo->qi_iwarnlimit;
 			hardlimit = be64_to_cpu(dqp->q_core.d_ino_hardlimit);
 			if (!hardlimit)
-				hardlimit = q->qi_ihardlimit;
+				hardlimit = defq->ihardlimit;
 			softlimit = be64_to_cpu(dqp->q_core.d_ino_softlimit);
 			if (!softlimit)
-				softlimit = q->qi_isoftlimit;
+				softlimit = defq->isoftlimit;
 
 			if (hardlimit && total_count > hardlimit) {
 				xfs_quota_warn(mp, dqp, QUOTA_NL_IHARDWARN);
-- 
2.4.3

From sandeen@sandeen.net  Wed Jan 27 11:17:52 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 1D9FE7CA2
	for ; Wed, 27 Jan 2016 11:17:52 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 0B928304039
	for ; Wed, 27 Jan 2016 09:17:48 -0800 (PST)
X-ASG-Debug-ID: 1453915049-04cb6c0d5f102dd0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 8XFJ6Q23OF37Tasj for ; Wed, 27 Jan 2016 09:17:30 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id A270860961B3
	for ; Wed, 27 Jan 2016 11:17:29 -0600 (CST)
Subject: Re: [PATCH] Split default quota limits by quota type
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] Split default quota limits by quota type
References: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
 <569FC41E.2040300@sandeen.net> <20160127153705.GA17571@redhat.com>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56A8FBA9.3060205@sandeen.net>
Date: Wed, 27 Jan 2016 11:17:29 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160127153705.GA17571@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453915050
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26502
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/27/16 9:37 AM, Carlos Maiolino wrote:
>>>
>>> This patch aims to split the default quota value by quota type. Allowing each
>>> quota type having different default values.
>>>
>>> Default time limits are still set globally, but I don't mind to split them by
>>> quota type too.
>>
>> Hm, I guess it seems like it should be done; otherwise it's a weird
>> caveat, isn't it?  "Default limits are set by type, but timers are
>> inherited from whatever first default quota is found across all types"
>>
>> So yeah, seems like it should be done for timers as well, IMHO;
>> grace periods can be set for each default quota type, so they should
>> be honored.
>>
> 
> Ok, I was just working on implement it, but honestly, I don't see the point now
> in split time limits by user/group/project.
> 
> grace periods are set globally by default. We don't have specific quota grace
> limits for each user or each group. Just a single value for them.

yeah, sorry about that.  I forgot that they were fs-wide, but that makes sense.
Sorry for leading you astray.

-Eric

From sandeen@sandeen.net  Wed Jan 27 13:04:34 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id DEFE17CA2
	for ; Wed, 27 Jan 2016 13:04:34 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id B1BD6304059
	for ; Wed, 27 Jan 2016 11:04:34 -0800 (PST)
X-ASG-Debug-ID: 1453921468-04bdf04d6cfef60001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id 1HJSXv0d4sNXJax1 for ; Wed, 27 Jan 2016 11:04:28 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id ECC3B61213DD
	for ; Wed, 27 Jan 2016 13:04:27 -0600 (CST)
Subject: Re: [PATCH] xfs: split default quota limits by quota type V2
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] xfs: split default quota limits by quota type V2
References: <1453913878-30900-1-git-send-email-cmaiolino@redhat.com>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56A914BB.4050301@sandeen.net>
Date: Wed, 27 Jan 2016 13:04:27 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <1453913878-30900-1-git-send-email-cmaiolino@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453921468
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26506
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/27/16 10:57 AM, Carlos Maiolino wrote:
> Default quotas are globally set due historical reasons. IRIX only supported user
> and project quotas, and default quota was only applied to user quotas.
> 
> In Linux, when a default quota is set, all different quota types inherits the
> same default value.
> 
> An user with a quota limit larger than the default quota value, will still be
> limited to the default value because the group quotas also inherits the default
> quotas. Unless the group which the user belongs to have a custom quota limit
> set.
> 
> This patch aims to split the default quota value by quota type. Allowing each
> quota type having different default values.
> 
> Default time limits are still set globally. XFS does not set a per-user/group
> timer, but a single global timer, so, there is no reason to split time limits
> too.
> 

Hohum, found something else, sorry:

> @@ -606,55 +653,15 @@ xfs_qm_init_quotainfo(
>  	 * We try to get the limits from the superuser's limits fields.
>  	 * This is quite hacky, but it is standard quota practice.
>  	 *
> -	 * We look at the USR dquot with id == 0 first, but if user quotas
> -	 * are not enabled we goto the GRP dquot with id == 0.
> -	 * We don't really care to keep separate default limits for user
> -	 * and group quotas, at least not at this point.
> -	 *
>  	 * Since we may not have done a quotacheck by this point, just read
>  	 * the dquot without attaching it to any hashtables or lists.
>  	 */
> -	error = xfs_qm_dqread(mp, 0,
> -			XFS_IS_UQUOTA_RUNNING(mp) ? XFS_DQ_USER :
> -			 (XFS_IS_GQUOTA_RUNNING(mp) ? XFS_DQ_GROUP :
> -			  XFS_DQ_PROJ),
> -			XFS_QMOPT_DOWARN, &dqp);

So here, we grab the *first* one that is running, and only that, to
populate the defaults ...

> -	if (!error) {
> -		xfs_disk_dquot_t	*ddqp = &dqp->q_core;
> -
> -		/*
> -		 * The warnings and timers set the grace period given to
> -		 * a user or group before he or she can not perform any
> -		 * more writing. If it is zero, a default is used.
> -		 */
> -		qinf->qi_btimelimit = ddqp->d_btimer ?
> -			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
> -		qinf->qi_itimelimit = ddqp->d_itimer ?
> -			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
> -		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
> -			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
> -		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
> -			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
> -		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
> -			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
> -		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
> -			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;

... including the time limits.

> -		qinf->qi_bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
> -		qinf->qi_bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
> -		qinf->qi_ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
> -		qinf->qi_isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
> -		qinf->qi_rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
> -		qinf->qi_rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
> -
> -		xfs_qm_dqdestroy(dqp);
> -	} else {
> -		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
> -		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
> -		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
> -		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
> -		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
> -		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
> -	}
> +	if (XFS_IS_UQUOTA_RUNNING(mp))
> +		error = xfs_qm_set_defquota(mp, XFS_DQ_USER, qinf);
> +	if (XFS_IS_GQUOTA_RUNNING(mp))
> +		error = xfs_qm_set_defquota(mp, XFS_DQ_GROUP, qinf);
> +	if (XFS_IS_PQUOTA_RUNNING(mp))
> +		error = xfs_qm_set_defquota(mp, XFS_DQ_PROJ, qinf);

Here, you set the defaults for *each* one that is running, in succession.
But xfs_qm_set_defquota() sets global time limits, so you may overwrite
what you set earlier, for the previous type.  i.e. it's last one wins.

So before, the global grace period was set by the first quota found running;
now it will be set by the last quota found running.  That's probably a difference
in behavior you didn't intend.

(It's crazy behavior to start with, though; see my other email in this thread,
but we probably shouldn't trade crazy(1) for crazy(1);  we should keep crazy(1)
and fix it properly at some point)

-Eric

From sandeen@sandeen.net  Wed Jan 27 13:06:50 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id C6F717CA2
	for ; Wed, 27 Jan 2016 13:06:50 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 974A68F8040
	for ; Wed, 27 Jan 2016 11:06:47 -0800 (PST)
X-ASG-Debug-ID: 1453921606-04bdf04d6cff0a0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id KN12quvGpdnBAW5K for ; Wed, 27 Jan 2016 11:06:46 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id DD0D661213D3
	for ; Wed, 27 Jan 2016 13:06:45 -0600 (CST)
Subject: Re: [PATCH] Split default quota limits by quota type
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] Split default quota limits by quota type
References: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
 <569FC41E.2040300@sandeen.net> <20160127153705.GA17571@redhat.com>
 <56A8FBA9.3060205@sandeen.net>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56A91545.3090806@sandeen.net>
Date: Wed, 27 Jan 2016 13:06:45 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <56A8FBA9.3060205@sandeen.net>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453921606
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26506
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/27/16 11:17 AM, Eric Sandeen wrote:
> On 1/27/16 9:37 AM, Carlos Maiolino wrote:
>>>>
>>>> This patch aims to split the default quota value by quota type. Allowing each
>>>> quota type having different default values.
>>>>
>>>> Default time limits are still set globally, but I don't mind to split them by
>>>> quota type too.
>>>
>>> Hm, I guess it seems like it should be done; otherwise it's a weird
>>> caveat, isn't it?  "Default limits are set by type, but timers are
>>> inherited from whatever first default quota is found across all types"
>>>
>>> So yeah, seems like it should be done for timers as well, IMHO;
>>> grace periods can be set for each default quota type, so they should
>>> be honored.
>>>
>>
>> Ok, I was just working on implement it, but honestly, I don't see the point now
>> in split time limits by user/group/project.
>>
>> grace periods are set globally by default. We don't have specific quota grace
>> limits for each user or each group. Just a single value for them.
> 
> yeah, sorry about that.  I forgot that they were fs-wide, but that makes sense.
> Sorry for leading you astray.

Ugh, this is a mess.  The xfs_quota command says:

       timer [ -gpu ] [ -bir ] value

we can indeed set timers for each type; group, project, and user.
These really are stored on-disk for ID 0 for each type.
We can set different values for user, group, and project.

However, let's set the inode timer for group quota to 30 minutes:

# xfs_quota -x -c "timer -i -g 30m" /dev/sdb1
# xfs_quota -x -c "state" /dev/sdb1 | grep grace
Blocks grace time: [7 days 00:00:30]
Inodes grace time: [0 days 00:30:30]

Uh, ok, it set a global default.  (And where did the 30s come
from?)

Now let's set an inode timer for *user* quota:

# xfs_quota -x -c "timer -i -u 60m" /dev/sdb1
# xfs_quota -x -c "state" /dev/sdb1 | grep grace
Blocks grace time: [7 days 00:00:30]
Inodes grace time: [0 days 01:00:30]

Uh, ok, now that's the grace time...

Go back to group quota, try something smaller?

# xfs_quota -x -c "timer -i -g 10m" /dev/sdb1
# xfs_quota -x -c "state" /dev/sdb1 | grep grace
Blocks grace time: [7 days 00:00:30]
Inodes grace time: [0 days 00:10:30]

Ok, seems that "type" doesn't matter, and whatever was set last wins.  Except:



# xfs_quota -x -c "state" /dev/sdb1 | grep grace
Blocks grace time: [7 days 00:00:30]
Inodes grace time: [0 days 01:00:30]

ohai, now we are back to the user quota we set, because that's checked first.  :(

(with your most recent patch, it'll be whatever is checked *last*).

So this is all a big steaming pile, right down to the "timer"
command help giving you options it won't accept (-d), or ignores (id|name).

I guess it seems ok to just break out default limits into per-type limits; that's
a decent step in the right direction.  We probably need to think more about
how the timers should work; do we really want them to be global?  The tool
certainly reports them as such, although it claims to set them per-type.

Anyway, changes you make for per-type limits probably shouldn't change
how time limits are interpreted, but today they do.

...

-Eric

From sandeen@sandeen.net  Wed Jan 27 16:25:33 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 550FF7CA2
	for ; Wed, 27 Jan 2016 16:25:33 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id D6F2FAC004
	for ; Wed, 27 Jan 2016 14:25:29 -0800 (PST)
X-ASG-Debug-ID: 1453933523-04cb6c0d5f109aa0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id RscshHDzJuDd2Pbf for ; Wed, 27 Jan 2016 14:25:23 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from Liberator.local (70-90-76-85-BusName-mn.hfc.comcastbusiness.net [70.90.76.85])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 97A406146F33
	for ; Wed, 27 Jan 2016 16:25:22 -0600 (CST)
Subject: Re: [PATCH] Split default quota limits by quota type
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] Split default quota limits by quota type
References: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
 <569FC41E.2040300@sandeen.net> <20160127153705.GA17571@redhat.com>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56A943D1.4020606@sandeen.net>
Date: Wed, 27 Jan 2016 16:25:21 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160127153705.GA17571@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453933523
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26513
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/27/16 9:37 AM, Carlos Maiolino wrote:

> Ok, I was just working on implement it, but honestly, I don't see the point now
> in split time limits by user/group/project.
> 
> grace periods are set globally by default. We don't have specific quota grace
> limits for each user or each group. Just a single value for them.
>
> So, if we can not specify an individual grace period per-user or per-group, what
> is the point in having these time limits split by user/group/project? We could
> have the values divided by user/group/proj, but what's the sense on it? If we
> have a default grace limit for users, it will be set to all users, if we have a
> grace limit for groups, it will be enforced to all users anyway (since we can't
> specify the group here either).
> 
> So, I wonder, does it make sense?
> 
> Looks a nice idea for the future, but for the current implementation it doesn't
> make sense to me. But sure, let me know if I'm wrong :)


Sorry for all the self-replies.

I'm thinking that this really should be fixed up along with this work.

The time limits aren't about per-user or per-group; they are in fact filesystem-wide.

However, it is possible today to set time limits for user quota, group quota,
or project quota - i.e. not for each user, but for the user *type*; not for
each group, but for the group *type*.

Fixing it should be as "easy" as moving those time limits into your default
quotas structures.  It'd come almost for free in xfs_qm_init_quotainfo().

xfs_qm_adjust_dqtimers() then needs to use those, and
xfs_qm_fill_state() should get the proper values for each type, as well.

But the problem may be in reporting back out to userspace via
quota_getstate():

        /*
         * GETXSTATE quotactl has space for just one set of time limits so
         * report them for the first enabled quota type
         */

o_O doesn't that suck!

quota_getstatev() has enough padding that we could report it all out, though,
with some changes.

-Eric

From sandeen@sandeen.net  Wed Jan 27 16:49:34 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 913557CA2
	for ; Wed, 27 Jan 2016 16:49:34 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 7E8AE304039
	for ; Wed, 27 Jan 2016 14:49:34 -0800 (PST)
X-ASG-Debug-ID: 1453934973-04cb6c0d5e10a190001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id VqxoQCuRWXBzOkam for ; Wed, 27 Jan 2016 14:49:33 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from Liberator.local (70-90-76-85-BusName-mn.hfc.comcastbusiness.net [70.90.76.85])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id A00FB6146F33
	for ; Wed, 27 Jan 2016 16:49:33 -0600 (CST)
To: xfs@oss.sgi.com
From: Eric Sandeen 
Subject: [PATCH 0/2] xfsprogs: minor quota fixups
Message-ID: <56A9497C.9070400@sandeen.net>
X-ASG-Orig-Subj: [PATCH 0/2] xfsprogs: minor quota fixups
Date: Wed, 27 Jan 2016 16:49:32 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453934973
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26513
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Small fixups encountered while digging through quota...

From sandeen@sandeen.net  Wed Jan 27 16:50:25 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 1529C7CA2
	for ; Wed, 27 Jan 2016 16:50:25 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id DC65C304067
	for ; Wed, 27 Jan 2016 14:50:24 -0800 (PST)
X-ASG-Debug-ID: 1453935022-04bdf04d6a103d60001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id XHT7DBBOP3yzSSHy for ; Wed, 27 Jan 2016 14:50:22 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from Liberator.local (70-90-76-85-BusName-mn.hfc.comcastbusiness.net [70.90.76.85])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 889DB6612D71
	for ; Wed, 27 Jan 2016 16:50:22 -0600 (CST)
Subject: [PATCH 1/2] xfs_quota: remove extra 30 seconds from time limit
 reporting
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 1/2] xfs_quota: remove extra 30 seconds from time limit
 reporting
References: <56A9497C.9070400@sandeen.net>
From: Eric Sandeen 
Message-ID: <56A949AD.9020903@sandeen.net>
Date: Wed, 27 Jan 2016 16:50:21 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <56A9497C.9070400@sandeen.net>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453935022
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26514
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

The addition of 30s makes little sense; it's only added to the
values reported via the userspace tool, and doesn't reflect the
actual values used in the kernel. Setting a time limit to
"00:30:00" and reporting back "00:30:30" makes little sense to me.

Signed-off-by: Eric Sandeen 
---

diff --git a/quota/util.c b/quota/util.c
index 179aafd..7c43fbd 100644
--- a/quota/util.c
+++ b/quota/util.c
@@ -42,8 +42,6 @@ time_to_string(
 		time(&now);
 		timer = MAX(origin - now, 0);
 	}
-	if (timer > 60)	/* roundup */
-		timer += 30;
 
 	days = timer / SECONDS_IN_A_DAY;
 	if (days)


From sandeen@sandeen.net  Wed Jan 27 16:51:36 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id F40427CA2
	for ; Wed, 27 Jan 2016 16:51:35 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E3BE08F8040
	for ; Wed, 27 Jan 2016 14:51:32 -0800 (PST)
X-ASG-Debug-ID: 1453935091-04cb6c0d6010a250001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id BBdaA5I8nldhcY9U for ; Wed, 27 Jan 2016 14:51:31 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from Liberator.local (70-90-76-85-BusName-mn.hfc.comcastbusiness.net [70.90.76.85])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 5E1C9660C83E
	for ; Wed, 27 Jan 2016 16:51:31 -0600 (CST)
Subject: [PATCH 2/2] xfs_quota: fix up timer command help
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: [PATCH 2/2] xfs_quota: fix up timer command help
References: <56A9497C.9070400@sandeen.net>
From: Eric Sandeen 
Message-ID: <56A949F2.8080802@sandeen.net>
Date: Wed, 27 Jan 2016 16:51:30 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <56A9497C.9070400@sandeen.net>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453935091
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.50
X-Barracuda-Spam-Status: No, SCORE=0.50 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MV0713
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26513
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.50 BSF_SC0_MV0713         Custom rule MV0713

The timer command help output suggests that either default, or a
specific ID, can be specified, but this is not the case.  Timers
are fs-wide for each quota type, not specific to the user.  So
remove the "-d|id|name" from the help output.  Also remove
"get" from the description, because it only sets these timers;
"state" returns their values.  Well actually just one of them,
but that's a different problem...

Signed-off-by: Eric Sandeen 
---

diff --git a/quota/edit.c b/quota/edit.c
index c55fa1a..6146f7e 100644
--- a/quota/edit.c
+++ b/quota/edit.c
@@ -701,8 +701,8 @@ edit_init(void)
 	timer_cmd.cfunc = timer_f;
 	timer_cmd.argmin = 2;
 	timer_cmd.argmax = -1;
-	timer_cmd.args = _("[-bir] [-gpu] value -d|id|name");
-	timer_cmd.oneline = _("get/set quota enforcement timeouts");
+	timer_cmd.args = _("[-bir] [-gpu] value");
+	timer_cmd.oneline = _("set quota enforcement timeouts");
 	timer_cmd.help = timer_help;
 
 	warn_cmd.name = "warn";

From beverly-hills-md-xfs=oss.sgi.com@elitefun.net  Wed Jan 27 19:09:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: **
X-Spam-Status: No, score=2.2 required=5.0 tests=HTML_IMAGE_ONLY_32,
	HTML_MESSAGE,T_DKIM_INVALID,T_REMOTE_IMAGE autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 4DBE229DF5
	for ; Wed, 27 Jan 2016 19:09:40 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 2DCBC304062
	for ; Wed, 27 Jan 2016 17:09:37 -0800 (PST)
X-ASG-Debug-ID: 1453943364-04cb6c0d5e10cdd0002-NocioJ
Received: from mail.elitefun.net ([77.81.104.209]) by cuda.sgi.com with ESMTP id ej7Qw0tyTxXP2rHs for ; Wed, 27 Jan 2016 17:09:34 -0800 (PST)
X-Barracuda-Envelope-From: beverly-hills-md-xfs=oss.sgi.com@elitefun.net
X-Barracuda-Apparent-Source-IP: 77.81.104.209
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=dkim; d=elitefun.net;
 h=Date:From:To:Subject:MIME-Version:Content-Type:Message-ID; i=beverly-hills-md@elitefun.net;
 bh=Zr0aKk0uGnTNq/vX1bd0FMFPzQ4=;
 b=gZGFrud8K5ikAaOEK3puHkbGAuP83WaJNcKWO4F3/KKvanQQIFFNzhcfXVeg4xP+pvSOd//oykqQ
   c+Ei+xiwLdfsjmLDsttotvLnwIn074M6hnEWIlKKYA4E+UzawKdLppmMMs7foT0NiCJqxrxzV6jQ
   cQ6mVfdugZOJgEBwvVM=
DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=dkim; d=elitefun.net;
 b=dpT1Y8jC/ImWjPNwWd3RWqgbR5MW58ATXpFBlkiXp/kgBqoTrrLV4oo5YrhZQAXt+ucYi1+ZFfSV
   55EIjjYR0UYyHkFOGArcgR/z5gmfwP4/lB0vD0MS/tIJ+mbQW7kg9X6mOzbp6dqYYGbyojUvj3+n
   eGVaIJb9bX+6sTjLLR0=;
Received: by mail.elitefun.net id hl5l4u0001gv for ; Wed, 27 Jan 2016 19:05:37 -0600 (envelope-from )
Date: Wed, 27 Jan 2016 19:05:37 -0600
From: "Beverly Hills MD" 
To:   
Subject: How to lift saggy skin, without surgery (Video)
MIME-Version: 1.0
X-ASG-Orig-Subj: How to lift saggy skin, without surgery (Video)
Content-Type: multipart/alternative; 
	boundary="----=_Part_424_2043819912.1453943023090"
Message-ID: <0.0.0.61.1D15967FFAA673C.24D7C2@mail.elitefun.net>
X-Barracuda-Connect: UNKNOWN[77.81.104.209]
X-Barracuda-Start-Time: 1453943372
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.60
X-Barracuda-Spam-Status: No, SCORE=0.60 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC5_MJ1963, DKIM_SIGNED, DKIM_VERIFIED, HTML_IMAGE_ONLY_32, HTML_MESSAGE, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26520
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	0.00 HTML_IMAGE_ONLY_32     BODY: HTML: images with 2800-3200 bytes of words
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS
	0.50 BSF_SC5_MJ1963         Custom Rule MJ1963

------=_Part_424_2043819912.1453943023090
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

How to lift saggy skin, without surgery (Video)

http://www.elitefun.net/8098SI6434aZsnFjnn0ntv0Mjh382/side



Update Preferences- http://www.elitefun.net/863V86ox4350EZsnFjnn0ntv0Mjh9f6/had

------=_Part_424_2043819912.1453943023090
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit



  
   
  Beverly Hills MD - Lift and Firm 
  
  
  
 
   
How to lift saggy skin, without surgery (Video)
 
   

      
   
 
    
   

 Dear xfs,
 
   

 My wife recently came across a video that absolutely blew my mind.
 It' s about two plastic surgeons based in Beverly Hills who' ve been doing some
 pretty innovative things to help reverse the aging process of your skin
 
 Using non-surgical techniques you can do at home, they are helping men & 
 women to improve their skin and get rid of the major problems that are aging them:
 
 Sagging skin
 Drooping eyes
 Turkey neck
 Jowls
 Saggy, loose cheeks
 and so much more.
   
 
   
 But what makes this really amazing is the fact that you don' t have to go to
   
 Beverly Hills or have ANY kind of surgery in order to erase years off of your face.
   
 And I know it works because both my wife & I have seen amazing results while using their techniques.
   
 
   
 
   
   
 
   
 
   Click HERE to watch the video that had us glued to the screen
   
 
   
   
   
Yours Truly,
 Dr. John Layke
 Beverly Hills MD
 P.S. PLEASE write in and tell me if this 
   video was as amazing to YOU as it was to me! 
   
Beverly Hills MD, LLC
 8391 Beverly Blvd. Ste. 271, Los Angeles, CA. 90048
 
    
   
  
 
   
 
   
 
   
 
   
 
   
 
   
 
   
 
   
   
   

    
 
    
   
 
   
 
   
  
   
 


------=_Part_424_2043819912.1453943023090--

From trananhtuan@sales.vietinaviva.vn  Thu Jan 28 06:27:57 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 2152E7CA5
	for ; Thu, 28 Jan 2016 06:27:57 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id B2D55AC003
	for ; Thu, 28 Jan 2016 04:27:53 -0800 (PST)
X-ASG-Debug-ID: 1453984038-04bdf04d6a1136b0001-NocioJ
Received: from cluster-m.mailcontrol.com (cluster-m.mailcontrol.com [116.50.58.190]) by cuda.sgi.com with ESMTP id g8rMIWqBZKxFps5G (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 28 Jan 2016 04:27:19 -0800 (PST)
X-Barracuda-Envelope-From: trananhtuan@sales.vietinaviva.vn
X-Barracuda-Apparent-Source-IP: 116.50.58.190
Received: from mail.sales.vietinaviva.vn ([42.112.208.196])
	by rly03m.srv.mailcontrol.com (MailControl) with ESMTP id u0SCRGWI007227;
	Thu, 28 Jan 2016 12:27:16 GMT
Received: from localhost (localhost [127.0.0.1])
	by mail.sales.vietinaviva.vn (Postfix) with ESMTP id 6515FF2347B;
	Thu, 28 Jan 2016 19:27:14 +0700 (ICT)
Received: from mail.sales.vietinaviva.vn ([127.0.0.1])
	by localhost (mail.sales.vietinaviva.vn [127.0.0.1]) (amavisd-new, port 10032)
	with ESMTP id RW_0VKxf02Dy; Thu, 28 Jan 2016 19:27:13 +0700 (ICT)
Received: from localhost (localhost [127.0.0.1])
	by mail.sales.vietinaviva.vn (Postfix) with ESMTP id 45FDAF2345F;
	Thu, 28 Jan 2016 19:27:13 +0700 (ICT)
X-Virus-Scanned: amavisd-new at sales.vietinaviva.vn
Received: from mail.sales.vietinaviva.vn ([127.0.0.1])
	by localhost (mail.sales.vietinaviva.vn [127.0.0.1]) (amavisd-new, port 10026)
	with ESMTP id pATMd251Sk1o; Thu, 28 Jan 2016 19:27:13 +0700 (ICT)
Received: from mail.sales.vietinaviva.vn (mail.sales.vietinaviva.vn [10.142.36.6])
	by mail.sales.vietinaviva.vn (Postfix) with ESMTP id 01BC6F2345E;
	Thu, 28 Jan 2016 19:27:12 +0700 (ICT)
Date: Thu, 28 Jan 2016 19:27:11 +0700 (ICT)
From: Tran Anh Tuan 
To: hangthai168@gmail.com
Message-ID: <2039410600.5541493.1453984031907.JavaMail.zimbra@sales.vietinaviva.vn>
Subject: =?utf-8?Q?Kinh_g=E1=BB=ADi_Qu=C3=BD_kh=C3=A1ch_h=C3=A0ng?=
MIME-Version: 1.0
X-ASG-Orig-Subj: =?utf-8?Q?Kinh_g=E1=BB=ADi_Qu=C3=BD_kh=C3=A1ch_h=C3=A0ng?=
Content-Type: multipart/alternative; 
	boundary="----=_Part_5541492_1597158179.1453984031904"
X-Originating-IP: [118.69.76.93]
X-Mailer: Zimbra 8.0.7_GA_6021 (ZimbraWebClient - GC47 (Win)/8.0.7_GA_6021)
Thread-Topic: Kinh =?utf-8?B?Z+G7rWkgUXXDvSBraMOhY2ggaMOgbmc=?=
Thread-Index: Z5Vx8PvNC62FG7o5aLWS9xAzrH/sew==
X-Scanned-By: MailControl 44278.411 (www.mailcontrol.com) on 10.77.0.113
X-Barracuda-Connect: cluster-m.mailcontrol.com[116.50.58.190]
X-Barracuda-Start-Time: 1453984039
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.22
X-Barracuda-Spam-Status: No, SCORE=0.22 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC0_SA038b, HTML_MESSAGE, THREAD_INDEX, THREAD_TOPIC
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26532
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.01 THREAD_INDEX           thread-index: AcO7Y8iR61tzADqsRmmc5wNiFHEOig==
	0.01 THREAD_TOPIC           Thread-Topic: ...(Japanese Subject)...
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 HTML_MESSAGE           BODY: HTML included in message
	0.20 BSF_SC0_SA038b         Custom Rule SA038b

------=_Part_5541492_1597158179.1453984031904
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64

CkPDtG5nIHR5IELhuqNvIGhp4buDbSBWaWV0aW5iYW5rIEF2aXZhIHbhu6thIMSRxrBhIHJhIHRo
4buLIHRyxrDhu51uZ23hu5l0IHPhuqNuIHBo4bqpbSB0aeG6v3Qga2nhu4dtIHTDoGkgY2jDrW5o
IHbhu6thIGLhuqNvIHbhu4cgbHXDtG4gY+G6oyBz4bupYyBraOG7j2UgY2hvIGNo4bunIG5ow6Ju
IGPhu6dhIGjhu6NwIMSR4buTbmcuIFbhu5tpIGzDo2kgc3XhuqV0IHLhuqV0IGjhuqVwIGThuqtu
LiAuIAoKxJDhurdjIGJp4buHdCBjw7MgbeG7mXQgc+G7kSBz4bqjbiBwaOG6qW0gZMOgbmggY2hv
IERvYW5oIG5naGnhu4dwIG11YSBjaG8gbmjDom4gdmnDqm4gduG7m2kgxrB1IMSRw6NpIHbDoCBj
aGnhur90IGto4bqldSBo4bqlcCBk4bqrbi4gR2nDoSBy4bqldCBy4bq7IHLhuqV0IHBow7kgaMah
cCBjaG8gY8OhIG5ow6JuIHbDoCBkb2FuaCBuZ2hp4buHcCAKCkPDtG5nIHR5IFROSEggQuG6o28g
aGnhu4NtIE5ow6JuIHRo4buNIFZpZXRpbmJhbmsgQXZpdmEgClTDsmEgbmjDoCBBQiAtIDc2QSBM
w6ogTGFpIHBoxrDhu51uZyBC4bq/biBUaMOgbmggUXXhuq1uIDEsIFRwIEhDTSAKCgpIb3RsaW5l
OiAwOSA4MyA2NyA2MSA2MSBNciBUdeG6pW4gKENodXnDqm4gdmnDqm4gdMOgaSBjaMOtbmggY2Fv
IGPhuqVwKSAKCgoKCkJlc3QgUmVnYXJkcywgCgpWaWV0aW5CYW5rIEF2aXZhIExpZmUgSW5zdXJh
bmNlIENvbXBhbnkgTGltaXRlZCAKCjIzdGggRmxvb3IsIEFCIFRvd2VyLCA3NiBBIEzDqiBMYWks
IERpc3QgMSwgSENNQyAKClRyYW4gQW5oIFR1YW4gCkNlbGwgOiAwOSA4MyA2NyA2MSA2MSAKTWFp
bCA6IHRyYW5hbmh0dWFuIEBzYWxlcy52aWV0aW5hdml2YS4gdm4gCgpodHRwczovL3d3dy55b3V0
dWJlLmNvbS93YXRjaD8gdj1zOFNVWWtMemItSSAKCg==
------=_Part_5541492_1597158179.1453984031904
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable



C=C3=B4ng ty B=E1=BA=A3o =
hi=E1=BB=83m Vietinbank Aviva v=E1=BB=ABa =C4=91=C6=B0a ra th=E1=BB=8B tr=
=C6=B0=E1=BB=9Dngm=E1=BB=99t s=E1=BA=A3n ph=E1=BA=A9m ti=E1=BA=BFt ki=E1=BB=
=87m t=C3=A0i ch=C3=ADnh v=E1=BB=ABa b=E1=BA=A3o v=E1=BB=87 lu=C3=B4n c=E1=
=BA=A3 s=E1=BB=A9c kh=E1=BB=8Fe cho ch=E1=BB=A7 nh=C3=A2n c=E1=BB=A7a h=E1=
=BB=A3p =C4=91=E1=BB=93ng. V=E1=BB=9Bi l=C3=A3i su=E1=BA=A5t r=E1=BA=A5t h=
=E1=BA=A5p d=E1=BA=ABn. .

=C4=90=E1=BA=B7c bi=E1=BB=87t c=C3=B3 m=
=E1=BB=99t s=E1=BB=91 s=E1=BA=A3n ph=E1=BA=A9m d=C3=A0nh cho Doanh nghi=E1=
=BB=87p mua cho nh=C3=A2n vi=C3=AAn v=E1=BB=9Bi =C6=B0u =C4=91=C3=A3i v=C3=
=A0 chi=E1=BA=BFt kh=E1=BA=A5u h=E1=BA=A5p d=E1=BA=ABn. Gi=C3=A1 r=E1=BA=A5=
t r=E1=BA=BB r=E1=BA=A5t ph=C3=B9 h=C6=A1p cho c=C3=A1 nh=C3=A2n v=C3=A0 do=
anh nghi=E1=BB=87p

C=C3=B4ng ty TNHH B=E1=BA=A3o hi=E1=BB=83m Nh=
=C3=A2n th=E1=BB=8D Vietinbank Aviva
T=C3=B2a nh=C3=A0 AB - 76A L=C3=AA L=
ai ph=C6=B0=E1=BB=9Dng B=E1=BA=BFn Th=C3=A0nh Qu=E1=BA=ADn 1, Tp HCM
<=
div style=3D"background-color: #ffffff;" data-mce-style=3D"background-color=
: #ffffff;">
Hotline: 0983 67 61 61&n=
bsp;Mr Tu=E1=BA=A5n (Chuy=C3=AAn vi=C3=AAn t=C3=A0i ch=C3=ADnh cao c=E1=BA=
=A5p)
 
Best Regards,
<=
p style=3D"margin: 0px; font-size: 12pt; font-family: 'Times New Roman', se=
rif; background-image: initial; background-position: initial; background-re=
peat: initial;" data-mce-style=3D"margin: 0px; font-size: 12pt; font-family=
: 'Times New Roman', serif; background-image: initial; background-position:=
 initial; background-repeat: initial;">VietinBank Aviva =
Life Insurance Company Limited
23th Floor, AB Tower, 76 A L=C3=AA Lai, Dist 1, HC=
MC
Tra=
n Anh Tuan
Cell    : 0983 67 61 61 
Mail   : trananhtuan@sales.vietinaviva.vn&nb=
sp;
https://www.youtube.com/watch?v=3Ds8SUYkLzb-I<=
/a>


------=_Part_5541492_1597158179.1453984031904--

From cmaiolino@redhat.com  Thu Jan 28 06:35:57 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 94BC97CA2
	for ; Thu, 28 Jan 2016 06:35:57 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 8202A304053
	for ; Thu, 28 Jan 2016 04:35:57 -0800 (PST)
X-ASG-Debug-ID: 1453984555-04cbb026ff11f210001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id S5HYQGNb2K2bD8eR (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 04:35:56 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id 72E818F278;
	Thu, 28 Jan 2016 12:35:55 +0000 (UTC)
Received: from redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0SCZp64030772
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 28 Jan 2016 07:35:53 -0500
Date: Thu, 28 Jan 2016 13:35:50 +0100
From: Carlos Maiolino 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] Split default quota limits by quota type
Message-ID: <20160128123550.GA16237@redhat.com>
X-ASG-Orig-Subj: Re: [PATCH] Split default quota limits by quota type
Mail-Followup-To: Eric Sandeen , xfs@oss.sgi.com
References: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
 <569FC41E.2040300@sandeen.net>
 <20160127153705.GA17571@redhat.com>
 <56A943D1.4020606@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A943D1.4020606@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453984556
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 27, 2016 at 04:25:21PM -0600, Eric Sandeen wrote:
> On 1/27/16 9:37 AM, Carlos Maiolino wrote:
> 
> > Ok, I was just working on implement it, but honestly, I don't see the point now
> > in split time limits by user/group/project.
> > 
> > grace periods are set globally by default. We don't have specific quota grace
> > limits for each user or each group. Just a single value for them.
> >
> > So, if we can not specify an individual grace period per-user or per-group, what
> > is the point in having these time limits split by user/group/project? We could
> > have the values divided by user/group/proj, but what's the sense on it? If we
> > have a default grace limit for users, it will be set to all users, if we have a
> > grace limit for groups, it will be enforced to all users anyway (since we can't
> > specify the group here either).
> > 
> > So, I wonder, does it make sense?
> > 
> > Looks a nice idea for the future, but for the current implementation it doesn't
> > make sense to me. But sure, let me know if I'm wrong :)
> 
> 
> Sorry for all the self-replies.
> 
> I'm thinking that this really should be fixed up along with this work.
> 
> The time limits aren't about per-user or per-group; they are in fact filesystem-wide.
> 
> However, it is possible today to set time limits for user quota, group quota,
> or project quota - i.e. not for each user, but for the user *type*; not for
> each group, but for the group *type*.
> 
> Fixing it should be as "easy" as moving those time limits into your default
> quotas structures.  It'd come almost for free in xfs_qm_init_quotainfo().
> 
Ok, I believe I got your point, but, I keep with my question about: what's the
point of having user *type* limits and group *type* limits if we can't specify
specific times for different users/groups?

I mean, if we have an inode time limit (just as an example)set to user type,
and we se it for users. All users will have the same limit.
But, let's say we have the inode limit set to group type. What is it going to
change if we can't specify a group? All users will always belong to a group. And
if we have the time limit set for the group *type*, consequently, all groups
will have the same limit, which will imply in all users also having the same
time limit, since all users belongs to at least one group.

So, I really don't see a point in splitting time limit default values if we
don't actually implement a per-user/per-group limit.

I have a RFC patch for splitting times (using the same structures I used to
split default b/i limits), but it's useless without a per-group/user specific
time.

It can certainly be implemented as part of another patch implementing
per-user/group limits, but with the current implementation, I really don't see
much sense, just adding a bunch of code that will not change anything or even
setup the code for future implementations.

Please, forgive me if I'm not making myself clear, or if I'm saying something
stupid, but honestly, I still think that splitting default time/warnings belongs
to an implementation of per-user/group timers not for this case itself.


Cheers o>




> xfs_qm_adjust_dqtimers() then needs to use those, and
> xfs_qm_fill_state() should get the proper values for each type, as well.
> 
> But the problem may be in reporting back out to userspace via
> quota_getstate():
> 
>         /*
>          * GETXSTATE quotactl has space for just one set of time limits so
>          * report them for the first enabled quota type
>          */
> 
> o_O doesn't that suck!
> 
> quota_getstatev() has enough padding that we could report it all out, though,
> with some changes.
> 
> -Eric
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

From cmaiolino@redhat.com  Thu Jan 28 07:03:07 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 0AE777CA2
	for ; Thu, 28 Jan 2016 07:03:07 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id DFB86304067
	for ; Thu, 28 Jan 2016 05:03:03 -0800 (PST)
X-ASG-Debug-ID: 1453986182-04cb6c0d5d11a130001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id yiKdJuaWx0tPj11S (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 05:03:03 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26])
	by mx1.redhat.com (Postfix) with ESMTPS id 2D1BE8E390;
	Thu, 28 Jan 2016 13:03:02 +0000 (UTC)
Received: from redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0SD2xwr007243
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 28 Jan 2016 08:03:01 -0500
Date: Thu, 28 Jan 2016 14:02:58 +0100
From: Carlos Maiolino 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 0/2] xfsprogs: minor quota fixups
Message-ID: <20160128130258.GB16237@redhat.com>
X-ASG-Orig-Subj: Re: [PATCH 0/2] xfsprogs: minor quota fixups
Mail-Followup-To: Eric Sandeen , xfs@oss.sgi.com
References: <56A9497C.9070400@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A9497C.9070400@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453986182
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

Makes sense to me, consider it 

Reviewed-by: Carlos Maiolino 

On Wed, Jan 27, 2016 at 04:49:32PM -0600, Eric Sandeen wrote:
> Small fixups encountered while digging through quota...
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

From BATV+bbcf3f8dff1d8c8650a3+4540+infradead.org+hch@bombadil.srs.infradead.org  Thu Jan 28 07:16:19 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 0E3FD7CA2
	for ; Thu, 28 Jan 2016 07:16:19 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id E207D304067
	for ; Thu, 28 Jan 2016 05:16:18 -0800 (PST)
X-ASG-Debug-ID: 1453986976-04cb6c0d5e11a6b0001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id c3D8xFk5w4SniLUF (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 28 Jan 2016 05:16:17 -0800 (PST)
X-Barracuda-Envelope-From: BATV+bbcf3f8dff1d8c8650a3+4540+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aOmQo-0005E5-KJ; Thu, 28 Jan 2016 13:16:14 +0000
Date: Thu, 28 Jan 2016 05:16:14 -0800
From: Christoph Hellwig 
To: "Darrick J. Wong" 
Cc: david@fromorbit.com, xfs@oss.sgi.com
Subject: Re: RFCv4+ WIP rollup of XFS reverse-mapping, reflink, and dedupe
 support
Message-ID: <20160128131614.GA14876@infradead.org>
X-ASG-Orig-Subj: Re: RFCv4+ WIP rollup of XFS reverse-mapping, reflink, and dedupe
 support
References: <20160123013130.GC5496@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160123013130.GC5496@birch.djwong.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1453986977
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26532
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

Do you have a bisectable version of the updates?  I'm running
into hangs with generic/154 with the new code over NFS, due what
might be a missing ilock unlock, but so far I can't make sense of it.

From BATV+bbcf3f8dff1d8c8650a3+4540+infradead.org+hch@bombadil.srs.infradead.org  Thu Jan 28 07:16:59 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 77D5B7CA2
	for ; Thu, 28 Jan 2016 07:16:59 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 5989E304053
	for ; Thu, 28 Jan 2016 05:16:59 -0800 (PST)
X-ASG-Debug-ID: 1453987017-04bdf04d6a115290001-NocioJ
Received: from bombadil.infradead.org ([198.137.202.9]) by cuda.sgi.com with ESMTP id cFWM8JTmwJFl1Xcu (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 28 Jan 2016 05:16:57 -0800 (PST)
X-Barracuda-Envelope-From: BATV+bbcf3f8dff1d8c8650a3+4540+infradead.org+hch@bombadil.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 198.137.202.9
Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux))
	id 1aOmRU-0005Ii-M3; Thu, 28 Jan 2016 13:16:56 +0000
Date: Thu, 28 Jan 2016 05:16:56 -0800
From: Christoph Hellwig 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com, darrick.wong@oracle.com
Subject: Re: stop using ioends for direct write completions
Message-ID: <20160128131656.GB14876@infradead.org>
X-ASG-Orig-Subj: Re: stop using ioends for direct write completions
References: <1452766237-2314-1-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1452766237-2314-1-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SRS-Rewrite: SMTP reverse-path rewritten from  by bombadil.infradead.org
	See http://www.infradead.org/rpr.html
X-Barracuda-Connect: UNKNOWN[198.137.202.9]
X-Barracuda-Start-Time: 1453987017
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.10
X-Barracuda-Spam-Status: No, SCORE=0.10 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, RDNS_NONE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26532
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.10 RDNS_NONE              Delivered to trusted network by a host with no rDNS

Any chance to get a review for this?  It should really help
with sorting out the buffered I/O COW code.

From cmaiolino@redhat.com  Thu Jan 28 07:48:34 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id E8C6C7CA2
	for ; Thu, 28 Jan 2016 07:48:33 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 75E95AC003
	for ; Thu, 28 Jan 2016 05:48:33 -0800 (PST)
X-ASG-Debug-ID: 1453988911-04cbb026fe121460001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id izJ7FRcxmPmEBtu5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 05:48:32 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26])
	by mx1.redhat.com (Postfix) with ESMTPS id 0DB14C09FA95;
	Thu, 28 Jan 2016 13:48:31 +0000 (UTC)
Received: from redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0SDmRf3004515
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
	Thu, 28 Jan 2016 08:48:30 -0500
Date: Thu, 28 Jan 2016 14:48:27 +0100
From: Carlos Maiolino 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: split default quota limits by quota type V2
Message-ID: <20160128134827.GA4115@redhat.com>
X-ASG-Orig-Subj: Re: [PATCH] xfs: split default quota limits by quota type V2
Mail-Followup-To: Eric Sandeen , xfs@oss.sgi.com
References: <1453913878-30900-1-git-send-email-cmaiolino@redhat.com>
 <56A914BB.4050301@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56A914BB.4050301@sandeen.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1453988911
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Wed, Jan 27, 2016 at 01:04:27PM -0600, Eric Sandeen wrote:
> On 1/27/16 10:57 AM, Carlos Maiolino wrote:
> > Default quotas are globally set due historical reasons. IRIX only supported user
> > and project quotas, and default quota was only applied to user quotas.
> > 
> > In Linux, when a default quota is set, all different quota types inherits the
> > same default value.
> > 
> > An user with a quota limit larger than the default quota value, will still be
> > limited to the default value because the group quotas also inherits the default
> > quotas. Unless the group which the user belongs to have a custom quota limit
> > set.
> > 
> > This patch aims to split the default quota value by quota type. Allowing each
> > quota type having different default values.
> > 
> > Default time limits are still set globally. XFS does not set a per-user/group
> > timer, but a single global timer, so, there is no reason to split time limits
> > too.
> > 
> 
> Hohum, found something else, sorry:
> 
Yep, I noticed it today's morning while playing with the time limits, but I
think I know how to fix it already, I'll test it and let you know.

Thanks for the heads up though :)


> > @@ -606,55 +653,15 @@ xfs_qm_init_quotainfo(
> >  	 * We try to get the limits from the superuser's limits fields.
> >  	 * This is quite hacky, but it is standard quota practice.
> >  	 *
> > -	 * We look at the USR dquot with id == 0 first, but if user quotas
> > -	 * are not enabled we goto the GRP dquot with id == 0.
> > -	 * We don't really care to keep separate default limits for user
> > -	 * and group quotas, at least not at this point.
> > -	 *
> >  	 * Since we may not have done a quotacheck by this point, just read
> >  	 * the dquot without attaching it to any hashtables or lists.
> >  	 */
> > -	error = xfs_qm_dqread(mp, 0,
> > -			XFS_IS_UQUOTA_RUNNING(mp) ? XFS_DQ_USER :
> > -			 (XFS_IS_GQUOTA_RUNNING(mp) ? XFS_DQ_GROUP :
> > -			  XFS_DQ_PROJ),
> > -			XFS_QMOPT_DOWARN, &dqp);
> 
> So here, we grab the *first* one that is running, and only that, to
> populate the defaults ...
> 
> > -	if (!error) {
> > -		xfs_disk_dquot_t	*ddqp = &dqp->q_core;
> > -
> > -		/*
> > -		 * The warnings and timers set the grace period given to
> > -		 * a user or group before he or she can not perform any
> > -		 * more writing. If it is zero, a default is used.
> > -		 */
> > -		qinf->qi_btimelimit = ddqp->d_btimer ?
> > -			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
> > -		qinf->qi_itimelimit = ddqp->d_itimer ?
> > -			be32_to_cpu(ddqp->d_itimer) : XFS_QM_ITIMELIMIT;
> > -		qinf->qi_rtbtimelimit = ddqp->d_rtbtimer ?
> > -			be32_to_cpu(ddqp->d_rtbtimer) : XFS_QM_RTBTIMELIMIT;
> > -		qinf->qi_bwarnlimit = ddqp->d_bwarns ?
> > -			be16_to_cpu(ddqp->d_bwarns) : XFS_QM_BWARNLIMIT;
> > -		qinf->qi_iwarnlimit = ddqp->d_iwarns ?
> > -			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
> > -		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
> > -			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
> 
> ... including the time limits.
> 
> > -		qinf->qi_bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
> > -		qinf->qi_bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
> > -		qinf->qi_ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
> > -		qinf->qi_isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
> > -		qinf->qi_rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
> > -		qinf->qi_rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
> > -
> > -		xfs_qm_dqdestroy(dqp);
> > -	} else {
> > -		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
> > -		qinf->qi_itimelimit = XFS_QM_ITIMELIMIT;
> > -		qinf->qi_rtbtimelimit = XFS_QM_RTBTIMELIMIT;
> > -		qinf->qi_bwarnlimit = XFS_QM_BWARNLIMIT;
> > -		qinf->qi_iwarnlimit = XFS_QM_IWARNLIMIT;
> > -		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
> > -	}
> > +	if (XFS_IS_UQUOTA_RUNNING(mp))
> > +		error = xfs_qm_set_defquota(mp, XFS_DQ_USER, qinf);
> > +	if (XFS_IS_GQUOTA_RUNNING(mp))
> > +		error = xfs_qm_set_defquota(mp, XFS_DQ_GROUP, qinf);
> > +	if (XFS_IS_PQUOTA_RUNNING(mp))
> > +		error = xfs_qm_set_defquota(mp, XFS_DQ_PROJ, qinf);
> 
> Here, you set the defaults for *each* one that is running, in succession.
> But xfs_qm_set_defquota() sets global time limits, so you may overwrite
> what you set earlier, for the previous type.  i.e. it's last one wins.
> 
> So before, the global grace period was set by the first quota found running;
> now it will be set by the last quota found running.  That's probably a difference
> in behavior you didn't intend.
> 
> (It's crazy behavior to start with, though; see my other email in this thread,
> but we probably shouldn't trade crazy(1) for crazy(1);  we should keep crazy(1)
> and fix it properly at some point)
> 
> -Eric
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

From sandeen@sandeen.net  Thu Jan 28 08:10:48 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id E45FA7CA2
	for ; Thu, 28 Jan 2016 08:10:48 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id D1269304059
	for ; Thu, 28 Jan 2016 06:10:45 -0800 (PST)
X-ASG-Debug-ID: 1453990243-04cb6c0d5e11bc80001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id B0ZNTF7ujOYDAQaJ for ; Thu, 28 Jan 2016 06:10:43 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 135B263C6064
	for ; Thu, 28 Jan 2016 08:10:43 -0600 (CST)
Subject: Re: [PATCH] Split default quota limits by quota type
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] Split default quota limits by quota type
References: <1453303127-27295-1-git-send-email-cmaiolino@redhat.com>
 <569FC41E.2040300@sandeen.net> <20160127153705.GA17571@redhat.com>
 <56A943D1.4020606@sandeen.net> <20160128123550.GA16237@redhat.com>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56AA2162.1070705@sandeen.net>
Date: Thu, 28 Jan 2016 08:10:42 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160128123550.GA16237@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1453990243
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26533
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/28/16 6:35 AM, Carlos Maiolino wrote:
> On Wed, Jan 27, 2016 at 04:25:21PM -0600, Eric Sandeen wrote:
>> On 1/27/16 9:37 AM, Carlos Maiolino wrote:
>>
>>> Ok, I was just working on implement it, but honestly, I don't see the point now
>>> in split time limits by user/group/project.
>>>
>>> grace periods are set globally by default. We don't have specific quota grace
>>> limits for each user or each group. Just a single value for them.
>>>
>>> So, if we can not specify an individual grace period per-user or per-group, what
>>> is the point in having these time limits split by user/group/project? We could
>>> have the values divided by user/group/proj, but what's the sense on it? If we
>>> have a default grace limit for users, it will be set to all users, if we have a
>>> grace limit for groups, it will be enforced to all users anyway (since we can't
>>> specify the group here either).
>>>
>>> So, I wonder, does it make sense?
>>>
>>> Looks a nice idea for the future, but for the current implementation it doesn't
>>> make sense to me. But sure, let me know if I'm wrong :)
>>
>>
>> Sorry for all the self-replies.
>>
>> I'm thinking that this really should be fixed up along with this work.
>>
>> The time limits aren't about per-user or per-group; they are in fact filesystem-wide.
>>
>> However, it is possible today to set time limits for user quota, group quota,
>> or project quota - i.e. not for each user, but for the user *type*; not for
>> each group, but for the group *type*.
>>
>> Fixing it should be as "easy" as moving those time limits into your default
>> quotas structures.  It'd come almost for free in xfs_qm_init_quotainfo().
>>
> Ok, I believe I got your point, but, I keep with my question about: what's the
> point of having user *type* limits and group *type* limits if we can't specify
> specific times for different users/groups?

Well, because the man page says you can, and the vfs quota tools allow doing
so, and report it as such, and the userspace tools actually do write the
time limit to each individual quota inode - so almost all the parts, except
the actual enforcement are all there.

> I mean, if we have an inode time limit (just as an example)set to user type,
> and we se it for users. All users will have the same limit.
> But, let's say we have the inode limit set to group type. What is it going to
> change if we can't specify a group?

and what about project? :)

But anyway, the difference is that you could be over user quota but not
group quota, or vice versa, right?

Say user cem has a block quota of 1GB, and belongs to a group,
xfs-developers, with a block quota of 10GB.

If the cem user goes over his 1GB block quota, we might want to give them
30m to fix it.  But if the xfs-developers group goes over their 10GB block
quota, we might want to give them a day, because it might require group
coordination, or something.

That would require, and utilize, separate grace-periods for user vs.
group quotas.

> All users will always belong to a group. And
> if we have the time limit set for the group *type*, consequently, all groups
> will have the same limit, which will imply in all users also having the same
> time limit, since all users belongs to at least one group.

The group timer is not for "anyone in a group who has exceeded any quota" -
it is for group quotas which have been exceeded.  i.e. the timer is on the
group quota, not on "people in any group."

-Eric

> So, I really don't see a point in splitting time limit default values if we
> don't actually implement a per-user/per-group limit.
> 
> I have a RFC patch for splitting times (using the same structures I used to
> split default b/i limits), but it's useless without a per-group/user specific
> time.
> 
> It can certainly be implemented as part of another patch implementing
> per-user/group limits, but with the current implementation, I really don't see
> much sense, just adding a bunch of code that will not change anything or even
> setup the code for future implementations.
> 
> Please, forgive me if I'm not making myself clear, or if I'm saying something
> stupid, but honestly, I still think that splitting default time/warnings belongs
> to an implementation of per-user/group timers not for this case itself.
> 
> 
> Cheers o>

From cmaiolino@redhat.com  Thu Jan 28 10:57:58 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 696EC7CA2
	for ; Thu, 28 Jan 2016 10:57:58 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id B682BAC001
	for ; Thu, 28 Jan 2016 08:57:54 -0800 (PST)
X-ASG-Debug-ID: 1454000272-04bdf04d6c11acb0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id LkWCDkFAC2smw9Sk (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 08:57:52 -0800 (PST)
X-Barracuda-Envelope-From: cmaiolino@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (Postfix) with ESMTPS id 8642A691
	for ; Thu, 28 Jan 2016 16:57:52 +0000 (UTC)
Received: from zion.usersys.redhat.com (dhcp-26-103.brq.redhat.com [10.34.26.103])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0SGvpUY008714
	for ; Thu, 28 Jan 2016 11:57:51 -0500
From: Carlos Maiolino 
To: xfs@oss.sgi.com
Subject: [PATCH] xfs: Split default quota limits by quota type V3
Date: Thu, 28 Jan 2016 17:57:46 +0100
X-ASG-Orig-Subj: [PATCH] xfs: Split default quota limits by quota type V3
Message-Id: <1454000266-22541-1-git-send-email-cmaiolino@redhat.com>
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1454000272
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

Default quotas are globally set due historical reasons. IRIX only supported user
and project quotas, and default quota was only applied to user quotas.

In Linux, when a default quota is set, all different quota types inherits the
same default value.

An user with a quota limit larger than the default quota value, will still be
limited to the default value because the group quotas also inherits the default
quotas. Unless the group which the user belongs to have a custom quota limit
set.

This patch aims to split the default quota value by quota type. Allowing each
quota type having different default values.

Default time limits are still set globally. XFS does not set a per-user/group
timer, but a single global timer. For changing this behavior, some changes
should be made in user-space tools another bugs being fixed.

Changelog:

	V2 - Remove comment about old behavior
	V3 - Keep time limit configuration inside xfs_qm_init_quotainfo to keep
	     the current behavior of quota timers

Signed-off-by: Carlos Maiolino 
---
 fs/xfs/xfs_dquot.c       | 26 +++++++++++---------
 fs/xfs/xfs_qm.c          | 64 +++++++++++++++++++++++++++++++++++-------------
 fs/xfs/xfs_qm.h          | 34 ++++++++++++++++++++-----
 fs/xfs/xfs_qm_syscalls.c | 15 +++++++-----
 fs/xfs/xfs_trans_dquot.c | 15 +++++++-----
 5 files changed, 107 insertions(+), 47 deletions(-)

diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 9c44d38..23f551b 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -92,26 +92,28 @@ xfs_qm_adjust_dqlimits(
 {
 	struct xfs_quotainfo	*q = mp->m_quotainfo;
 	struct xfs_disk_dquot	*d = &dq->q_core;
+	struct xfs_def_quota	*defq;
 	int			prealloc = 0;
 
 	ASSERT(d->d_id);
+	defq = xfs_get_defquota(dq, q);
 
-	if (q->qi_bsoftlimit && !d->d_blk_softlimit) {
-		d->d_blk_softlimit = cpu_to_be64(q->qi_bsoftlimit);
+	if (defq->bsoftlimit && !d->d_blk_softlimit) {
+		d->d_blk_softlimit = cpu_to_be64(defq->bsoftlimit);
 		prealloc = 1;
 	}
-	if (q->qi_bhardlimit && !d->d_blk_hardlimit) {
-		d->d_blk_hardlimit = cpu_to_be64(q->qi_bhardlimit);
+	if (defq->bhardlimit && !d->d_blk_hardlimit) {
+		d->d_blk_hardlimit = cpu_to_be64(defq->bhardlimit);
 		prealloc = 1;
 	}
-	if (q->qi_isoftlimit && !d->d_ino_softlimit)
-		d->d_ino_softlimit = cpu_to_be64(q->qi_isoftlimit);
-	if (q->qi_ihardlimit && !d->d_ino_hardlimit)
-		d->d_ino_hardlimit = cpu_to_be64(q->qi_ihardlimit);
-	if (q->qi_rtbsoftlimit && !d->d_rtb_softlimit)
-		d->d_rtb_softlimit = cpu_to_be64(q->qi_rtbsoftlimit);
-	if (q->qi_rtbhardlimit && !d->d_rtb_hardlimit)
-		d->d_rtb_hardlimit = cpu_to_be64(q->qi_rtbhardlimit);
+	if (defq->isoftlimit && !d->d_ino_softlimit)
+		d->d_ino_softlimit = cpu_to_be64(defq->isoftlimit);
+	if (defq->ihardlimit && !d->d_ino_hardlimit)
+		d->d_ino_hardlimit = cpu_to_be64(defq->ihardlimit);
+	if (defq->rtbsoftlimit && !d->d_rtb_softlimit)
+		d->d_rtb_softlimit = cpu_to_be64(defq->rtbsoftlimit);
+	if (defq->rtbhardlimit && !d->d_rtb_hardlimit)
+		d->d_rtb_hardlimit = cpu_to_be64(defq->rtbhardlimit);
 
 	if (prealloc)
 		xfs_dquot_set_prealloc_limits(dq);
diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
index 532ab79..4cbb22b 100644
--- a/fs/xfs/xfs_qm.c
+++ b/fs/xfs/xfs_qm.c
@@ -560,6 +560,37 @@ xfs_qm_shrink_count(
 	return list_lru_shrink_count(&qi->qi_lru, sc);
 }
 
+STATIC void
+xfs_qm_set_defquota(
+	xfs_mount_t	*mp,
+	uint		type,
+	xfs_quotainfo_t	*qinf)
+{
+	xfs_dquot_t		*dqp;
+	struct xfs_def_quota    *defq;
+	int			error;
+
+	error = xfs_qm_dqread(mp, 0, type, XFS_QMOPT_DOWARN, &dqp);
+
+	if (!error) {
+		xfs_disk_dquot_t        *ddqp = &dqp->q_core;
+
+		defq = xfs_get_defquota(dqp, qinf);
+
+		/*
+		 * Timers and warnings have been already set, let's just set the
+		 * default limits for this quota type
+		 */
+		defq->bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
+		defq->bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
+		defq->ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
+		defq->isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
+		defq->rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
+		defq->rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
+		xfs_qm_dqdestroy(dqp);
+	}
+}
+
 /*
  * This initializes all the quota information that's kept in the
  * mount structure
@@ -606,27 +637,22 @@ xfs_qm_init_quotainfo(
 	 * We try to get the limits from the superuser's limits fields.
 	 * This is quite hacky, but it is standard quota practice.
 	 *
-	 * We look at the USR dquot with id == 0 first, but if user quotas
-	 * are not enabled we goto the GRP dquot with id == 0.
-	 * We don't really care to keep separate default limits for user
-	 * and group quotas, at least not at this point.
-	 *
 	 * Since we may not have done a quotacheck by this point, just read
 	 * the dquot without attaching it to any hashtables or lists.
+	 *
+	 * Timers and warnings are globally set by the first timer found in
+	 * user/group/proj quota types, otherwise a default value is used.
+	 * This should be splitted into different fields per quota type.
 	 */
 	error = xfs_qm_dqread(mp, 0,
 			XFS_IS_UQUOTA_RUNNING(mp) ? XFS_DQ_USER :
 			 (XFS_IS_GQUOTA_RUNNING(mp) ? XFS_DQ_GROUP :
 			  XFS_DQ_PROJ),
 			XFS_QMOPT_DOWARN, &dqp);
+
 	if (!error) {
 		xfs_disk_dquot_t	*ddqp = &dqp->q_core;
 
-		/*
-		 * The warnings and timers set the grace period given to
-		 * a user or group before he or she can not perform any
-		 * more writing. If it is zero, a default is used.
-		 */
 		qinf->qi_btimelimit = ddqp->d_btimer ?
 			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
 		qinf->qi_itimelimit = ddqp->d_itimer ?
@@ -639,13 +665,6 @@ xfs_qm_init_quotainfo(
 			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
 		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
 			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
-		qinf->qi_bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
-		qinf->qi_bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
-		qinf->qi_ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
-		qinf->qi_isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
-		qinf->qi_rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
-		qinf->qi_rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
-
 		xfs_qm_dqdestroy(dqp);
 	} else {
 		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
@@ -656,6 +675,17 @@ xfs_qm_init_quotainfo(
 		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
 	}
 
+	/*
+	 * Default quota values are set by quota type, timer and warn limits
+	 * have already been setpreviously, no need for error check here.
+	 */
+	if (XFS_IS_UQUOTA_RUNNING(mp))
+		xfs_qm_set_defquota(mp, XFS_DQ_USER, qinf);
+	if (XFS_IS_GQUOTA_RUNNING(mp))
+		xfs_qm_set_defquota(mp, XFS_DQ_GROUP, qinf);
+	if (XFS_IS_PQUOTA_RUNNING(mp))
+		xfs_qm_set_defquota(mp, XFS_DQ_PROJ, qinf);
+
 	qinf->qi_shrinker.count_objects = xfs_qm_shrink_count;
 	qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan;
 	qinf->qi_shrinker.seeks = DEFAULT_SEEKS;
diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
index 996a040..45e2c36 100644
--- a/fs/xfs/xfs_qm.h
+++ b/fs/xfs/xfs_qm.h
@@ -53,6 +53,15 @@ extern struct kmem_zone	*xfs_qm_dqtrxzone;
  */
 #define XFS_DQUOT_CLUSTER_SIZE_FSB	(xfs_filblks_t)1
 
+struct xfs_def_quota {
+	xfs_qcnt_t       bhardlimit;     /* default data blk hard limit */
+	xfs_qcnt_t       bsoftlimit;	 /* default data blk soft limit */
+	xfs_qcnt_t       ihardlimit;	 /* default inode count hard limit */
+	xfs_qcnt_t       isoftlimit;	 /* default inode count soft limit */
+	xfs_qcnt_t	 rtbhardlimit;   /* default realtime blk hard limit */
+	xfs_qcnt_t	 rtbsoftlimit;   /* default realtime blk soft limit */
+};
+
 /*
  * Various quota information for individual filesystems.
  * The mount structure keeps a pointer to this.
@@ -76,12 +85,9 @@ typedef struct xfs_quotainfo {
 	struct mutex	 qi_quotaofflock;/* to serialize quotaoff */
 	xfs_filblks_t	 qi_dqchunklen;	 /* # BBs in a chunk of dqs */
 	uint		 qi_dqperchunk;	 /* # ondisk dqs in above chunk */
-	xfs_qcnt_t	 qi_bhardlimit;	 /* default data blk hard limit */
-	xfs_qcnt_t	 qi_bsoftlimit;	 /* default data blk soft limit */
-	xfs_qcnt_t	 qi_ihardlimit;	 /* default inode count hard limit */
-	xfs_qcnt_t	 qi_isoftlimit;	 /* default inode count soft limit */
-	xfs_qcnt_t	 qi_rtbhardlimit;/* default realtime blk hard limit */
-	xfs_qcnt_t	 qi_rtbsoftlimit;/* default realtime blk soft limit */
+	struct xfs_def_quota	qi_usr_default;
+	struct xfs_def_quota	qi_grp_default;
+	struct xfs_def_quota	qi_prj_default;
 	struct shrinker  qi_shrinker;
 } xfs_quotainfo_t;
 
@@ -171,4 +177,20 @@ extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
 extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
 extern int		xfs_qm_scall_quotaoff(struct xfs_mount *, uint);
 
+static inline struct xfs_def_quota *
+xfs_get_defquota(struct xfs_dquot *dqp, struct xfs_quotainfo *qi)
+{
+	struct xfs_def_quota *defq;
+
+	if (XFS_QM_ISUDQ(dqp))
+		defq = &qi->qi_usr_default;
+	else if (XFS_QM_ISGDQ(dqp))
+		defq = &qi->qi_grp_default;
+	else {
+		ASSERT(XFS_QM_ISPDQ(dqp));
+		defq = &qi->qi_prj_default;
+	}
+	return defq;
+}
+
 #endif /* __XFS_QM_H__ */
diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
index 3640c6e..31830f0 100644
--- a/fs/xfs/xfs_qm_syscalls.c
+++ b/fs/xfs/xfs_qm_syscalls.c
@@ -404,6 +404,7 @@ xfs_qm_scall_setqlim(
 	struct xfs_disk_dquot	*ddq;
 	struct xfs_dquot	*dqp;
 	struct xfs_trans	*tp;
+	struct xfs_def_quota	*defq;
 	int			error;
 	xfs_qcnt_t		hard, soft;
 
@@ -431,6 +432,8 @@ xfs_qm_scall_setqlim(
 		ASSERT(error != -ENOENT);
 		goto out_unlock;
 	}
+
+	defq = xfs_get_defquota(dqp, q);
 	xfs_dqunlock(dqp);
 
 	tp = xfs_trans_alloc(mp, XFS_TRANS_QM_SETQLIM);
@@ -458,8 +461,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_blk_softlimit = cpu_to_be64(soft);
 		xfs_dquot_set_prealloc_limits(dqp);
 		if (id == 0) {
-			q->qi_bhardlimit = hard;
-			q->qi_bsoftlimit = soft;
+			defq->bhardlimit = hard;
+			defq->bsoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "blkhard %Ld < blksoft %Ld", hard, soft);
@@ -474,8 +477,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_rtb_hardlimit = cpu_to_be64(hard);
 		ddq->d_rtb_softlimit = cpu_to_be64(soft);
 		if (id == 0) {
-			q->qi_rtbhardlimit = hard;
-			q->qi_rtbsoftlimit = soft;
+			defq->rtbhardlimit = hard;
+			defq->rtbsoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "rtbhard %Ld < rtbsoft %Ld", hard, soft);
@@ -491,8 +494,8 @@ xfs_qm_scall_setqlim(
 		ddq->d_ino_hardlimit = cpu_to_be64(hard);
 		ddq->d_ino_softlimit = cpu_to_be64(soft);
 		if (id == 0) {
-			q->qi_ihardlimit = hard;
-			q->qi_isoftlimit = soft;
+			defq->ihardlimit = hard;
+			defq->isoftlimit = soft;
 		}
 	} else {
 		xfs_debug(mp, "ihard %Ld < isoft %Ld", hard, soft);
diff --git a/fs/xfs/xfs_trans_dquot.c b/fs/xfs/xfs_trans_dquot.c
index 9951701..c3d5472 100644
--- a/fs/xfs/xfs_trans_dquot.c
+++ b/fs/xfs/xfs_trans_dquot.c
@@ -609,17 +609,20 @@ xfs_trans_dqresv(
 	xfs_qcnt_t	total_count;
 	xfs_qcnt_t	*resbcountp;
 	xfs_quotainfo_t	*q = mp->m_quotainfo;
+	struct xfs_def_quota	*defq;
 
 
 	xfs_dqlock(dqp);
 
+	defq = xfs_get_defquota(dqp, q);
+
 	if (flags & XFS_TRANS_DQ_RES_BLKS) {
 		hardlimit = be64_to_cpu(dqp->q_core.d_blk_hardlimit);
 		if (!hardlimit)
-			hardlimit = q->qi_bhardlimit;
+			hardlimit = defq->bhardlimit;
 		softlimit = be64_to_cpu(dqp->q_core.d_blk_softlimit);
 		if (!softlimit)
-			softlimit = q->qi_bsoftlimit;
+			softlimit = defq->bsoftlimit;
 		timer = be32_to_cpu(dqp->q_core.d_btimer);
 		warns = be16_to_cpu(dqp->q_core.d_bwarns);
 		warnlimit = dqp->q_mount->m_quotainfo->qi_bwarnlimit;
@@ -628,10 +631,10 @@ xfs_trans_dqresv(
 		ASSERT(flags & XFS_TRANS_DQ_RES_RTBLKS);
 		hardlimit = be64_to_cpu(dqp->q_core.d_rtb_hardlimit);
 		if (!hardlimit)
-			hardlimit = q->qi_rtbhardlimit;
+			hardlimit = defq->rtbhardlimit;
 		softlimit = be64_to_cpu(dqp->q_core.d_rtb_softlimit);
 		if (!softlimit)
-			softlimit = q->qi_rtbsoftlimit;
+			softlimit = defq->rtbsoftlimit;
 		timer = be32_to_cpu(dqp->q_core.d_rtbtimer);
 		warns = be16_to_cpu(dqp->q_core.d_rtbwarns);
 		warnlimit = dqp->q_mount->m_quotainfo->qi_rtbwarnlimit;
@@ -672,10 +675,10 @@ xfs_trans_dqresv(
 			warnlimit = dqp->q_mount->m_quotainfo->qi_iwarnlimit;
 			hardlimit = be64_to_cpu(dqp->q_core.d_ino_hardlimit);
 			if (!hardlimit)
-				hardlimit = q->qi_ihardlimit;
+				hardlimit = defq->ihardlimit;
 			softlimit = be64_to_cpu(dqp->q_core.d_ino_softlimit);
 			if (!softlimit)
-				softlimit = q->qi_isoftlimit;
+				softlimit = defq->isoftlimit;
 
 			if (hardlimit && total_count > hardlimit) {
 				xfs_quota_warn(mp, dqp, QUOTA_NL_IHARDWARN);
-- 
2.4.3

From BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org  Thu Jan 28 11:14:04 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 433BA7CA2
	for ; Thu, 28 Jan 2016 11:14:04 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 40E5F8F8040
	for ; Thu, 28 Jan 2016 09:14:04 -0800 (PST)
X-ASG-Debug-ID: 1454001241-04cb6c0d5d1200d0001-NocioJ
Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) by cuda.sgi.com with ESMTP id 350cquR4Hw2K0NGo (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 28 Jan 2016 09:14:02 -0800 (PST)
X-Barracuda-Envelope-From: BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 85.118.1.10
Received: from [83.175.99.196] (helo=localhost)
	by casper.infradead.org with esmtpsa (Exim 4.85 #2 (Red Hat Linux))
	id 1aOq8v-00049H-6u; Thu, 28 Jan 2016 17:14:01 +0000
From: Christoph Hellwig 
To: darrick.wong@oracle.com,
	xfs@oss.sgi.com
Subject: [PATCH 2/4] xfstests: require dedup ioctl support in 182
Date: Thu, 28 Jan 2016 18:13:54 +0100
X-ASG-Orig-Subj: [PATCH 2/4] xfstests: require dedup ioctl support in 182
Message-Id: <1454001236-2600-3-git-send-email-hch@lst.de>
X-Mailer: git-send-email 2.1.4
In-Reply-To: <1454001236-2600-1-git-send-email-hch@lst.de>
References: <1454001236-2600-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by casper.infradead.org
X-Barracuda-Connect: casper.infradead.org[85.118.1.10]
X-Barracuda-Start-Time: 1454001242
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26536
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Signed-off-by: Christoph Hellwig 
---
 tests/generic/182 | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/generic/182 b/tests/generic/182
index bf5cd38..eb80b43 100755
--- a/tests/generic/182
+++ b/tests/generic/182
@@ -46,6 +46,7 @@ _cleanup()
 # real QA test starts here
 _supported_os Linux
 _require_test_reflink
+_require_test_dedupe
 _require_cp_reflink
 
 rm -f "$seqres.full"
-- 
2.1.4

From BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org  Thu Jan 28 11:14:04 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id D96B67CA6
	for ; Thu, 28 Jan 2016 11:14:04 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 62231AC004
	for ; Thu, 28 Jan 2016 09:14:04 -0800 (PST)
X-ASG-Debug-ID: 1454001242-04cbb026ff126ae0001-NocioJ
Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) by cuda.sgi.com with ESMTP id uTGp0ggtxsPLHiA6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 28 Jan 2016 09:14:03 -0800 (PST)
X-Barracuda-Envelope-From: BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 85.118.1.10
Received: from [83.175.99.196] (helo=localhost)
	by casper.infradead.org with esmtpsa (Exim 4.85 #2 (Red Hat Linux))
	id 1aOq8v-00049M-Sk; Thu, 28 Jan 2016 17:14:01 +0000
From: Christoph Hellwig 
To: darrick.wong@oracle.com,
	xfs@oss.sgi.com
Subject: [PATCH 3/4] xfstests: also treat EINVAL as reflink/dedup not supported
Date: Thu, 28 Jan 2016 18:13:55 +0100
X-ASG-Orig-Subj: [PATCH 3/4] xfstests: also treat EINVAL as reflink/dedup not supported
Message-Id: <1454001236-2600-4-git-send-email-hch@lst.de>
X-Mailer: git-send-email 2.1.4
In-Reply-To: <1454001236-2600-1-git-send-email-hch@lst.de>
References: <1454001236-2600-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by casper.infradead.org
X-Barracuda-Connect: casper.infradead.org[85.118.1.10]
X-Barracuda-Start-Time: 1454001242
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26536
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

This can be returned by various implementations of the ioctl.

Signed-off-by: Christoph Hellwig 
---
 common/reflink | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/common/reflink b/common/reflink
index de8e56d..8638aba 100644
--- a/common/reflink
+++ b/common/reflink
@@ -92,6 +92,8 @@ _require_test_dedupe()
 		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
 	echo $testio | grep -q "Inappropriate ioctl for device" && \
 		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
+	echo $testio | grep -q "Invalid argument" && \
+		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
 	rm -rf "$TEST_DIR/file1" "$TEST_DIR/file2"
 }
 
@@ -110,6 +112,8 @@ _require_scratch_dedupe()
 		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
 	echo $testio | grep -q "Inappropriate ioctl for device" && \
 		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
+	echo $testio | grep -q "Invalid argument" && \
+		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
 	_scratch_unmount
 }
 
-- 
2.1.4

From BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org  Thu Jan 28 11:14:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 0E31F7CA7
	for ; Thu, 28 Jan 2016 11:14:05 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 0D1EB8F8039
	for ; Thu, 28 Jan 2016 09:14:05 -0800 (PST)
X-ASG-Debug-ID: 1454001242-04cb6c0d601200e0001-NocioJ
Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) by cuda.sgi.com with ESMTP id PjsB7OgT22nIdBtv (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 28 Jan 2016 09:14:03 -0800 (PST)
X-Barracuda-Envelope-From: BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 85.118.1.10
Received: from [83.175.99.196] (helo=localhost)
	by casper.infradead.org with esmtpsa (Exim 4.85 #2 (Red Hat Linux))
	id 1aOq8w-00049R-IJ; Thu, 28 Jan 2016 17:14:02 +0000
From: Christoph Hellwig 
To: darrick.wong@oracle.com,
	xfs@oss.sgi.com
Subject: [PATCH 4/4] xfstests: change to relative margins for reflinks tests
Date: Thu, 28 Jan 2016 18:13:56 +0100
X-ASG-Orig-Subj: [PATCH 4/4] xfstests: change to relative margins for reflinks tests
Message-Id: <1454001236-2600-5-git-send-email-hch@lst.de>
X-Mailer: git-send-email 2.1.4
In-Reply-To: <1454001236-2600-1-git-send-email-hch@lst.de>
References: <1454001236-2600-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by casper.infradead.org
X-Barracuda-Connect: casper.infradead.org[85.118.1.10]
X-Barracuda-Start-Time: 1454001243
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26536
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

NFS uses 32K blocks and thus the numbers used for sanity checking
various reflink operations are way off.  Change to relative margings
and increase them a bit to make NFS happy.

Signed-off-by: Christoph Hellwig 
---
 tests/generic/150 | 2 +-
 tests/generic/151 | 2 +-
 tests/generic/152 | 2 +-
 tests/generic/153 | 2 +-
 tests/generic/154 | 2 +-
 tests/generic/155 | 2 +-
 tests/generic/156 | 2 +-
 tests/generic/157 | 2 +-
 tests/generic/158 | 2 +-
 tests/generic/159 | 2 +-
 tests/generic/160 | 2 +-
 11 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/tests/generic/150 b/tests/generic/150
index f3a6311..90ac46b 100755
--- a/tests/generic/150
+++ b/tests/generic/150
@@ -57,7 +57,7 @@ mkdir "$testdir"
 echo "Create the original file blocks"
 blksz="$(stat -f $testdir -c '%S')"
 blks=2000
-margin=100
+margin='15%'
 sz=$((blksz * blks))
 nr=7
 _pwrite_byte 0x61 0 $sz "$testdir/file1" >> "$seqres.full"
diff --git a/tests/generic/151 b/tests/generic/151
index 286b9ec..c67a12c 100755
--- a/tests/generic/151
+++ b/tests/generic/151
@@ -61,7 +61,7 @@ mkdir "$testdir"
 echo "Create the original file blocks"
 blksz="$(stat -f "$testdir" -c '%S')"
 blks=2000
-margin=100
+margin='15%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f "$testdir" -c '%f')
 nr=7
diff --git a/tests/generic/152 b/tests/generic/152
index 506a754..cdb2bf9 100755
--- a/tests/generic/152
+++ b/tests/generic/152
@@ -62,7 +62,7 @@ mkdir "$testdir"
 echo "Create the original file blocks"
 blksz="$(stat -f "$testdir" -c '%S')"
 blks=2000
-margin=100
+margin='15%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f "$testdir" -c '%f')
 nr=4
diff --git a/tests/generic/153 b/tests/generic/153
index 011879e..b2f4022 100755
--- a/tests/generic/153
+++ b/tests/generic/153
@@ -62,7 +62,7 @@ mkdir "$testdir"
 echo "Create the original file blocks"
 blksz="$(stat -f "$testdir" -c '%S')"
 blks=2000
-margin=100
+margin='15%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f "$testdir" -c '%f')
 nr=4
diff --git a/tests/generic/154 b/tests/generic/154
index 474c578..579f78b 100755
--- a/tests/generic/154
+++ b/tests/generic/154
@@ -61,7 +61,7 @@ mkdir "$testdir"
 echo "Create the original file blocks"
 blksz="$(stat -f "$testdir" -c '%S')"
 blks=2000
-margin=100
+margin='15%'
 free_blocks0=$(stat -f "$testdir" -c '%f')
 nr=4
 sz=$((blks * blksz))
diff --git a/tests/generic/155 b/tests/generic/155
index 611cf1a..56d2912 100755
--- a/tests/generic/155
+++ b/tests/generic/155
@@ -65,7 +65,7 @@ mkdir $testdir
 echo "Create the original file blocks"
 blksz="$(stat -f "$testdir" -c '%S')"
 blks=2000
-margin=100
+margin='15%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f "$testdir" -c '%f')
 nr=4
diff --git a/tests/generic/156 b/tests/generic/156
index 7194e87..9fea988 100755
--- a/tests/generic/156
+++ b/tests/generic/156
@@ -73,7 +73,7 @@ mkdir "$testdir"
 echo "Create the original file blocks"
 blksz="$(stat -f "$testdir" -c '%S')"
 blks=2000
-margin=100
+margin='15%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f "$testdir" -c '%f')
 nr=4
diff --git a/tests/generic/157 b/tests/generic/157
index a5b84fc..0150866 100755
--- a/tests/generic/157
+++ b/tests/generic/157
@@ -63,7 +63,7 @@ mkdir "$testdir2"
 echo "Create the original files"
 blksz="$(stat -f $testdir1 -c '%S')"
 blks=1000
-margin=50
+margin='7%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f $testdir1 -c '%f')
 nr=4
diff --git a/tests/generic/158 b/tests/generic/158
index 58b7090..807c247 100755
--- a/tests/generic/158
+++ b/tests/generic/158
@@ -63,7 +63,7 @@ mkdir "$testdir2"
 echo "Create the original files"
 blksz="$(stat -f $testdir1 -c '%S')"
 blks=1000
-margin=50
+margin='7%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f $testdir1 -c '%f')
 nr=4
diff --git a/tests/generic/159 b/tests/generic/159
index 14baf64..88a0c7e 100755
--- a/tests/generic/159
+++ b/tests/generic/159
@@ -56,7 +56,7 @@ mkdir "$testdir1"
 echo "Create the original files"
 blksz="$(stat -f $testdir1 -c '%S')"
 blks=1000
-margin=50
+margin='7%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f $testdir1 -c '%f')
 nr=4
diff --git a/tests/generic/160 b/tests/generic/160
index acd2c4e..83325d5 100755
--- a/tests/generic/160
+++ b/tests/generic/160
@@ -56,7 +56,7 @@ mkdir "$testdir1"
 echo "Create the original files"
 blksz="$(stat -f $testdir1 -c '%S')"
 blks=1000
-margin=50
+margin='7%'
 sz=$((blksz * blks))
 free_blocks0=$(stat -f $testdir1 -c '%f')
 nr=4
-- 
2.1.4

From BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org  Thu Jan 28 11:14:05 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id E683F7CA6
	for ; Thu, 28 Jan 2016 11:14:05 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id E59D08F8039
	for ; Thu, 28 Jan 2016 09:14:02 -0800 (PST)
X-ASG-Debug-ID: 1454001240-04cbb026fc126ac0001-NocioJ
Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) by cuda.sgi.com with ESMTP id cLYHSHbJsLIuBP4h (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 28 Jan 2016 09:14:01 -0800 (PST)
X-Barracuda-Envelope-From: BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 85.118.1.10
Received: from [83.175.99.196] (helo=localhost)
	by casper.infradead.org with esmtpsa (Exim 4.85 #2 (Red Hat Linux))
	id 1aOq8t-000495-IE; Thu, 28 Jan 2016 17:13:59 +0000
From: Christoph Hellwig 
To: darrick.wong@oracle.com,
	xfs@oss.sgi.com
Subject: a couple xfstests fixes for reflinks
Date: Thu, 28 Jan 2016 18:13:52 +0100
X-ASG-Orig-Subj: a couple xfstests fixes for reflinks
Message-Id: <1454001236-2600-1-git-send-email-hch@lst.de>
X-Mailer: git-send-email 2.1.4
X-SRS-Rewrite: SMTP reverse-path rewritten from  by casper.infradead.org
X-Barracuda-Connect: casper.infradead.org[85.118.1.10]
X-Barracuda-Start-Time: 1454001241
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26536
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

The patches (against Darrick's for-dave branch) fix various failures I see
during testing XFS and NFS.

From BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org  Thu Jan 28 11:14:07 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id E43DE7CAA
	for ; Thu, 28 Jan 2016 11:14:06 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 70590AC001
	for ; Thu, 28 Jan 2016 09:14:03 -0800 (PST)
X-ASG-Debug-ID: 1454001240-04cbb026fd126ad0001-NocioJ
Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) by cuda.sgi.com with ESMTP id 1ctudPxtjc7hI91G (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 28 Jan 2016 09:14:01 -0800 (PST)
X-Barracuda-Envelope-From: BATV+ac6ea833a0dc45c37a65+4540+infradead.org+hch@casper.srs.infradead.org
X-Barracuda-Apparent-Source-IP: 85.118.1.10
Received: from [83.175.99.196] (helo=localhost)
	by casper.infradead.org with esmtpsa (Exim 4.85 #2 (Red Hat Linux))
	id 1aOq8u-000498-9D; Thu, 28 Jan 2016 17:14:00 +0000
From: Christoph Hellwig 
To: darrick.wong@oracle.com,
	xfs@oss.sgi.com
Subject: [PATCH 1/4] xfstests: fix scratch device filtering in 139
Date: Thu, 28 Jan 2016 18:13:53 +0100
X-ASG-Orig-Subj: [PATCH 1/4] xfstests: fix scratch device filtering in 139
Message-Id: <1454001236-2600-2-git-send-email-hch@lst.de>
X-Mailer: git-send-email 2.1.4
In-Reply-To: <1454001236-2600-1-git-send-email-hch@lst.de>
References: <1454001236-2600-1-git-send-email-hch@lst.de>
X-SRS-Rewrite: SMTP reverse-path rewritten from  by casper.infradead.org
X-Barracuda-Connect: casper.infradead.org[85.118.1.10]
X-Barracuda-Start-Time: 1454001241
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26536
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Signed-off-by: Christoph Hellwig 
---
 tests/xfs/139     | 4 ++--
 tests/xfs/139.out | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
 mode change 100644 => 100755 tests/xfs/139

diff --git a/tests/xfs/139 b/tests/xfs/139
old mode 100644
new mode 100755
index 7400cf8..51c903e
--- a/tests/xfs/139
+++ b/tests/xfs/139
@@ -69,8 +69,8 @@ _pwrite_byte 0x62 0 $sz "$testdir/file1" >> "$seqres.full"
 _scratch_remount
 
 echo "Compare files"
-md5sum "$testdir/file1" | _filter_test_dir
-md5sum "$testdir/file2" | _filter_test_dir
+md5sum "$testdir/file1" | _filter_scratch
+md5sum "$testdir/file2" | _filter_scratch
 #filefrag -v "$testdir/file1" "$testdir/file2"
 
 echo "Check scratch fs"
diff --git a/tests/xfs/139.out b/tests/xfs/139.out
index 729e2d6..090e3af 100644
--- a/tests/xfs/139.out
+++ b/tests/xfs/139.out
@@ -3,6 +3,6 @@ Format and mount
 Create the original files
 CoW every other block
 Compare files
-673c759fb131ccf3336c4f8c549a497e  /opt/test-139/file1
-dc802351f6717547cbf4c951b294f952  /opt/test-139/file2
+673c759fb131ccf3336c4f8c549a497e  SCRATCH_MNT/test-139/file1
+dc802351f6717547cbf4c951b294f952  SCRATCH_MNT/test-139/file2
 Check scratch fs
-- 
2.1.4

From sandeen@sandeen.net  Thu Jan 28 11:37:00 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id A1C687CA2
	for ; Thu, 28 Jan 2016 11:37:00 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 9F49D8F8039
	for ; Thu, 28 Jan 2016 09:37:00 -0800 (PST)
X-ASG-Debug-ID: 1454002616-04cbb026fc1273e0001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id nEJxDdiBfOsH6TJJ for ; Thu, 28 Jan 2016 09:36:56 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 587FB63C6064
	for ; Thu, 28 Jan 2016 11:36:56 -0600 (CST)
Subject: Re: [PATCH] xfs: Split default quota limits by quota type V3
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: [PATCH] xfs: Split default quota limits by quota type V3
References: <1454000266-22541-1-git-send-email-cmaiolino@redhat.com>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56AA51B7.7030403@sandeen.net>
Date: Thu, 28 Jan 2016 11:36:55 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <1454000266-22541-1-git-send-email-cmaiolino@redhat.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1454002616
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26537
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

On 1/28/16 10:57 AM, Carlos Maiolino wrote:
> Default quotas are globally set due historical reasons. IRIX only supported user
> and project quotas, and default quota was only applied to user quotas.
> 
> In Linux, when a default quota is set, all different quota types inherits the
> same default value.
> 
> An user with a quota limit larger than the default quota value, will still be
> limited to the default value because the group quotas also inherits the default
> quotas. Unless the group which the user belongs to have a custom quota limit
> set.
> 
> This patch aims to split the default quota value by quota type. Allowing each
> quota type having different default values.
> 
> Default time limits are still set globally. XFS does not set a per-user/group
> timer, but a single global timer. For changing this behavior, some changes
> should be made in user-space tools another bugs being fixed.

Some minor comments about comments below.  But the code looks fine to me,
so:

Reviewed-by: Eric Sandeen 

and if you want to fix the comments & resend w/ my review tag I think that'd
be ok.

Care to write an xfstests testcase for the default quota limit while you're
at it?

Thanks,
-Eric

> Changelog:
> 
> 	V2 - Remove comment about old behavior
> 	V3 - Keep time limit configuration inside xfs_qm_init_quotainfo to keep
> 	     the current behavior of quota timers
> 
> Signed-off-by: Carlos Maiolino 
> ---
>  fs/xfs/xfs_dquot.c       | 26 +++++++++++---------
>  fs/xfs/xfs_qm.c          | 64 +++++++++++++++++++++++++++++++++++-------------
>  fs/xfs/xfs_qm.h          | 34 ++++++++++++++++++++-----
>  fs/xfs/xfs_qm_syscalls.c | 15 +++++++-----
>  fs/xfs/xfs_trans_dquot.c | 15 +++++++-----
>  5 files changed, 107 insertions(+), 47 deletions(-)
> 
> diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
> index 9c44d38..23f551b 100644
> --- a/fs/xfs/xfs_dquot.c
> +++ b/fs/xfs/xfs_dquot.c
> @@ -92,26 +92,28 @@ xfs_qm_adjust_dqlimits(
>  {
>  	struct xfs_quotainfo	*q = mp->m_quotainfo;
>  	struct xfs_disk_dquot	*d = &dq->q_core;
> +	struct xfs_def_quota	*defq;
>  	int			prealloc = 0;
>  
>  	ASSERT(d->d_id);
> +	defq = xfs_get_defquota(dq, q);
>  
> -	if (q->qi_bsoftlimit && !d->d_blk_softlimit) {
> -		d->d_blk_softlimit = cpu_to_be64(q->qi_bsoftlimit);
> +	if (defq->bsoftlimit && !d->d_blk_softlimit) {
> +		d->d_blk_softlimit = cpu_to_be64(defq->bsoftlimit);
>  		prealloc = 1;
>  	}
> -	if (q->qi_bhardlimit && !d->d_blk_hardlimit) {
> -		d->d_blk_hardlimit = cpu_to_be64(q->qi_bhardlimit);
> +	if (defq->bhardlimit && !d->d_blk_hardlimit) {
> +		d->d_blk_hardlimit = cpu_to_be64(defq->bhardlimit);
>  		prealloc = 1;
>  	}
> -	if (q->qi_isoftlimit && !d->d_ino_softlimit)
> -		d->d_ino_softlimit = cpu_to_be64(q->qi_isoftlimit);
> -	if (q->qi_ihardlimit && !d->d_ino_hardlimit)
> -		d->d_ino_hardlimit = cpu_to_be64(q->qi_ihardlimit);
> -	if (q->qi_rtbsoftlimit && !d->d_rtb_softlimit)
> -		d->d_rtb_softlimit = cpu_to_be64(q->qi_rtbsoftlimit);
> -	if (q->qi_rtbhardlimit && !d->d_rtb_hardlimit)
> -		d->d_rtb_hardlimit = cpu_to_be64(q->qi_rtbhardlimit);
> +	if (defq->isoftlimit && !d->d_ino_softlimit)
> +		d->d_ino_softlimit = cpu_to_be64(defq->isoftlimit);
> +	if (defq->ihardlimit && !d->d_ino_hardlimit)
> +		d->d_ino_hardlimit = cpu_to_be64(defq->ihardlimit);
> +	if (defq->rtbsoftlimit && !d->d_rtb_softlimit)
> +		d->d_rtb_softlimit = cpu_to_be64(defq->rtbsoftlimit);
> +	if (defq->rtbhardlimit && !d->d_rtb_hardlimit)
> +		d->d_rtb_hardlimit = cpu_to_be64(defq->rtbhardlimit);
>  
>  	if (prealloc)
>  		xfs_dquot_set_prealloc_limits(dq);
> diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
> index 532ab79..4cbb22b 100644
> --- a/fs/xfs/xfs_qm.c
> +++ b/fs/xfs/xfs_qm.c
> @@ -560,6 +560,37 @@ xfs_qm_shrink_count(
>  	return list_lru_shrink_count(&qi->qi_lru, sc);
>  }
>  
> +STATIC void
> +xfs_qm_set_defquota(
> +	xfs_mount_t	*mp,
> +	uint		type,
> +	xfs_quotainfo_t	*qinf)
> +{
> +	xfs_dquot_t		*dqp;
> +	struct xfs_def_quota    *defq;
> +	int			error;
> +
> +	error = xfs_qm_dqread(mp, 0, type, XFS_QMOPT_DOWARN, &dqp);
> +
> +	if (!error) {
> +		xfs_disk_dquot_t        *ddqp = &dqp->q_core;
> +
> +		defq = xfs_get_defquota(dqp, qinf);
> +
> +		/*
> +		 * Timers and warnings have been already set, let's just set the
> +		 * default limits for this quota type
> +		 */
> +		defq->bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
> +		defq->bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
> +		defq->ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
> +		defq->isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
> +		defq->rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
> +		defq->rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
> +		xfs_qm_dqdestroy(dqp);
> +	}
> +}
> +
>  /*
>   * This initializes all the quota information that's kept in the
>   * mount structure
> @@ -606,27 +637,22 @@ xfs_qm_init_quotainfo(
>  	 * We try to get the limits from the superuser's limits fields.
>  	 * This is quite hacky, but it is standard quota practice.
>  	 *
> -	 * We look at the USR dquot with id == 0 first, but if user quotas
> -	 * are not enabled we goto the GRP dquot with id == 0.
> -	 * We don't really care to keep separate default limits for user
> -	 * and group quotas, at least not at this point.
> -	 *
>  	 * Since we may not have done a quotacheck by this point, just read
>  	 * the dquot without attaching it to any hashtables or lists.
> +	 *
> +	 * Timers and warnings are globally set by the first timer found in
> +	 * user/group/proj quota types, otherwise a default value is used.
> +	 * This should be splitted into different fields per quota type.

s/splitted/split/

>  	 */
>  	error = xfs_qm_dqread(mp, 0,
>  			XFS_IS_UQUOTA_RUNNING(mp) ? XFS_DQ_USER :
>  			 (XFS_IS_GQUOTA_RUNNING(mp) ? XFS_DQ_GROUP :
>  			  XFS_DQ_PROJ),
>  			XFS_QMOPT_DOWARN, &dqp);
> +
>  	if (!error) {
>  		xfs_disk_dquot_t	*ddqp = &dqp->q_core;
>  
> -		/*
> -		 * The warnings and timers set the grace period given to
> -		 * a user or group before he or she can not perform any
> -		 * more writing. If it is zero, a default is used.
> -		 */

Not sure why you removed this comment?

>  		qinf->qi_btimelimit = ddqp->d_btimer ?
>  			be32_to_cpu(ddqp->d_btimer) : XFS_QM_BTIMELIMIT;
>  		qinf->qi_itimelimit = ddqp->d_itimer ?
> @@ -639,13 +665,6 @@ xfs_qm_init_quotainfo(
>  			be16_to_cpu(ddqp->d_iwarns) : XFS_QM_IWARNLIMIT;
>  		qinf->qi_rtbwarnlimit = ddqp->d_rtbwarns ?
>  			be16_to_cpu(ddqp->d_rtbwarns) : XFS_QM_RTBWARNLIMIT;
> -		qinf->qi_bhardlimit = be64_to_cpu(ddqp->d_blk_hardlimit);
> -		qinf->qi_bsoftlimit = be64_to_cpu(ddqp->d_blk_softlimit);
> -		qinf->qi_ihardlimit = be64_to_cpu(ddqp->d_ino_hardlimit);
> -		qinf->qi_isoftlimit = be64_to_cpu(ddqp->d_ino_softlimit);
> -		qinf->qi_rtbhardlimit = be64_to_cpu(ddqp->d_rtb_hardlimit);
> -		qinf->qi_rtbsoftlimit = be64_to_cpu(ddqp->d_rtb_softlimit);
> -
>  		xfs_qm_dqdestroy(dqp);
>  	} else {
>  		qinf->qi_btimelimit = XFS_QM_BTIMELIMIT;
> @@ -656,6 +675,17 @@ xfs_qm_init_quotainfo(
>  		qinf->qi_rtbwarnlimit = XFS_QM_RTBWARNLIMIT;
>  	}
>  
> +	/*
> +	 * Default quota values are set by quota type, timer and warn limits
> +	 * have already been setpreviously, no need for error check here.

"set previously" - and, not sure the comment about error checking is needed.
Previously, if we couldn't read the inode, we set timers but no limits.
And the limits were (and still are) zalloc'd so it's safe to ignore
the errors, I guess.


> +	 */
> +	if (XFS_IS_UQUOTA_RUNNING(mp))
> +		xfs_qm_set_defquota(mp, XFS_DQ_USER, qinf);
> +	if (XFS_IS_GQUOTA_RUNNING(mp))
> +		xfs_qm_set_defquota(mp, XFS_DQ_GROUP, qinf);
> +	if (XFS_IS_PQUOTA_RUNNING(mp))
> +		xfs_qm_set_defquota(mp, XFS_DQ_PROJ, qinf);
> +
>  	qinf->qi_shrinker.count_objects = xfs_qm_shrink_count;
>  	qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan;
>  	qinf->qi_shrinker.seeks = DEFAULT_SEEKS;
> diff --git a/fs/xfs/xfs_qm.h b/fs/xfs/xfs_qm.h
> index 996a040..45e2c36 100644
> --- a/fs/xfs/xfs_qm.h
> +++ b/fs/xfs/xfs_qm.h
> @@ -53,6 +53,15 @@ extern struct kmem_zone	*xfs_qm_dqtrxzone;
>   */
>  #define XFS_DQUOT_CLUSTER_SIZE_FSB	(xfs_filblks_t)1
>  
> +struct xfs_def_quota {
> +	xfs_qcnt_t       bhardlimit;     /* default data blk hard limit */
> +	xfs_qcnt_t       bsoftlimit;	 /* default data blk soft limit */
> +	xfs_qcnt_t       ihardlimit;	 /* default inode count hard limit */
> +	xfs_qcnt_t       isoftlimit;	 /* default inode count soft limit */
> +	xfs_qcnt_t	 rtbhardlimit;   /* default realtime blk hard limit */
> +	xfs_qcnt_t	 rtbsoftlimit;   /* default realtime blk soft limit */
> +};
> +
>  /*
>   * Various quota information for individual filesystems.
>   * The mount structure keeps a pointer to this.
> @@ -76,12 +85,9 @@ typedef struct xfs_quotainfo {
>  	struct mutex	 qi_quotaofflock;/* to serialize quotaoff */
>  	xfs_filblks_t	 qi_dqchunklen;	 /* # BBs in a chunk of dqs */
>  	uint		 qi_dqperchunk;	 /* # ondisk dqs in above chunk */
> -	xfs_qcnt_t	 qi_bhardlimit;	 /* default data blk hard limit */
> -	xfs_qcnt_t	 qi_bsoftlimit;	 /* default data blk soft limit */
> -	xfs_qcnt_t	 qi_ihardlimit;	 /* default inode count hard limit */
> -	xfs_qcnt_t	 qi_isoftlimit;	 /* default inode count soft limit */
> -	xfs_qcnt_t	 qi_rtbhardlimit;/* default realtime blk hard limit */
> -	xfs_qcnt_t	 qi_rtbsoftlimit;/* default realtime blk soft limit */
> +	struct xfs_def_quota	qi_usr_default;
> +	struct xfs_def_quota	qi_grp_default;
> +	struct xfs_def_quota	qi_prj_default;
>  	struct shrinker  qi_shrinker;
>  } xfs_quotainfo_t;
>  
> @@ -171,4 +177,20 @@ extern int		xfs_qm_scall_setqlim(struct xfs_mount *, xfs_dqid_t, uint,
>  extern int		xfs_qm_scall_quotaon(struct xfs_mount *, uint);
>  extern int		xfs_qm_scall_quotaoff(struct xfs_mount *, uint);
>  
> +static inline struct xfs_def_quota *
> +xfs_get_defquota(struct xfs_dquot *dqp, struct xfs_quotainfo *qi)
> +{
> +	struct xfs_def_quota *defq;
> +
> +	if (XFS_QM_ISUDQ(dqp))
> +		defq = &qi->qi_usr_default;
> +	else if (XFS_QM_ISGDQ(dqp))
> +		defq = &qi->qi_grp_default;
> +	else {
> +		ASSERT(XFS_QM_ISPDQ(dqp));
> +		defq = &qi->qi_prj_default;
> +	}
> +	return defq;
> +}
> +
>  #endif /* __XFS_QM_H__ */
> diff --git a/fs/xfs/xfs_qm_syscalls.c b/fs/xfs/xfs_qm_syscalls.c
> index 3640c6e..31830f0 100644
> --- a/fs/xfs/xfs_qm_syscalls.c
> +++ b/fs/xfs/xfs_qm_syscalls.c
> @@ -404,6 +404,7 @@ xfs_qm_scall_setqlim(
>  	struct xfs_disk_dquot	*ddq;
>  	struct xfs_dquot	*dqp;
>  	struct xfs_trans	*tp;
> +	struct xfs_def_quota	*defq;
>  	int			error;
>  	xfs_qcnt_t		hard, soft;
>  
> @@ -431,6 +432,8 @@ xfs_qm_scall_setqlim(
>  		ASSERT(error != -ENOENT);
>  		goto out_unlock;
>  	}
> +
> +	defq = xfs_get_defquota(dqp, q);
>  	xfs_dqunlock(dqp);
>  
>  	tp = xfs_trans_alloc(mp, XFS_TRANS_QM_SETQLIM);
> @@ -458,8 +461,8 @@ xfs_qm_scall_setqlim(
>  		ddq->d_blk_softlimit = cpu_to_be64(soft);
>  		xfs_dquot_set_prealloc_limits(dqp);
>  		if (id == 0) {
> -			q->qi_bhardlimit = hard;
> -			q->qi_bsoftlimit = soft;
> +			defq->bhardlimit = hard;
> +			defq->bsoftlimit = soft;
>  		}
>  	} else {
>  		xfs_debug(mp, "blkhard %Ld < blksoft %Ld", hard, soft);
> @@ -474,8 +477,8 @@ xfs_qm_scall_setqlim(
>  		ddq->d_rtb_hardlimit = cpu_to_be64(hard);
>  		ddq->d_rtb_softlimit = cpu_to_be64(soft);
>  		if (id == 0) {
> -			q->qi_rtbhardlimit = hard;
> -			q->qi_rtbsoftlimit = soft;
> +			defq->rtbhardlimit = hard;
> +			defq->rtbsoftlimit = soft;
>  		}
>  	} else {
>  		xfs_debug(mp, "rtbhard %Ld < rtbsoft %Ld", hard, soft);
> @@ -491,8 +494,8 @@ xfs_qm_scall_setqlim(
>  		ddq->d_ino_hardlimit = cpu_to_be64(hard);
>  		ddq->d_ino_softlimit = cpu_to_be64(soft);
>  		if (id == 0) {
> -			q->qi_ihardlimit = hard;
> -			q->qi_isoftlimit = soft;
> +			defq->ihardlimit = hard;
> +			defq->isoftlimit = soft;
>  		}
>  	} else {
>  		xfs_debug(mp, "ihard %Ld < isoft %Ld", hard, soft);
> diff --git a/fs/xfs/xfs_trans_dquot.c b/fs/xfs/xfs_trans_dquot.c
> index 9951701..c3d5472 100644
> --- a/fs/xfs/xfs_trans_dquot.c
> +++ b/fs/xfs/xfs_trans_dquot.c
> @@ -609,17 +609,20 @@ xfs_trans_dqresv(
>  	xfs_qcnt_t	total_count;
>  	xfs_qcnt_t	*resbcountp;
>  	xfs_quotainfo_t	*q = mp->m_quotainfo;
> +	struct xfs_def_quota	*defq;
>  
>  
>  	xfs_dqlock(dqp);
>  
> +	defq = xfs_get_defquota(dqp, q);
> +
>  	if (flags & XFS_TRANS_DQ_RES_BLKS) {
>  		hardlimit = be64_to_cpu(dqp->q_core.d_blk_hardlimit);
>  		if (!hardlimit)
> -			hardlimit = q->qi_bhardlimit;
> +			hardlimit = defq->bhardlimit;
>  		softlimit = be64_to_cpu(dqp->q_core.d_blk_softlimit);
>  		if (!softlimit)
> -			softlimit = q->qi_bsoftlimit;
> +			softlimit = defq->bsoftlimit;
>  		timer = be32_to_cpu(dqp->q_core.d_btimer);
>  		warns = be16_to_cpu(dqp->q_core.d_bwarns);
>  		warnlimit = dqp->q_mount->m_quotainfo->qi_bwarnlimit;
> @@ -628,10 +631,10 @@ xfs_trans_dqresv(
>  		ASSERT(flags & XFS_TRANS_DQ_RES_RTBLKS);
>  		hardlimit = be64_to_cpu(dqp->q_core.d_rtb_hardlimit);
>  		if (!hardlimit)
> -			hardlimit = q->qi_rtbhardlimit;
> +			hardlimit = defq->rtbhardlimit;
>  		softlimit = be64_to_cpu(dqp->q_core.d_rtb_softlimit);
>  		if (!softlimit)
> -			softlimit = q->qi_rtbsoftlimit;
> +			softlimit = defq->rtbsoftlimit;
>  		timer = be32_to_cpu(dqp->q_core.d_rtbtimer);
>  		warns = be16_to_cpu(dqp->q_core.d_rtbwarns);
>  		warnlimit = dqp->q_mount->m_quotainfo->qi_rtbwarnlimit;
> @@ -672,10 +675,10 @@ xfs_trans_dqresv(
>  			warnlimit = dqp->q_mount->m_quotainfo->qi_iwarnlimit;
>  			hardlimit = be64_to_cpu(dqp->q_core.d_ino_hardlimit);
>  			if (!hardlimit)
> -				hardlimit = q->qi_ihardlimit;
> +				hardlimit = defq->ihardlimit;
>  			softlimit = be64_to_cpu(dqp->q_core.d_ino_softlimit);
>  			if (!softlimit)
> -				softlimit = q->qi_isoftlimit;
> +				softlimit = defq->isoftlimit;
>  
>  			if (hardlimit && total_count > hardlimit) {
>  				xfs_quota_warn(mp, dqp, QUOTA_NL_IHARDWARN);
> 

From darrick.wong@oracle.com  Thu Jan 28 13:24:53 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id F18697CA2
	for ; Thu, 28 Jan 2016 13:24:52 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 8F0A4AC001
	for ; Thu, 28 Jan 2016 11:24:52 -0800 (PST)
X-ASG-Debug-ID: 1454009090-04bdf04d6d11e790001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id YMtcLwtXj23x6os4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 11:24:50 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0SJOllb004368
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Thu, 28 Jan 2016 19:24:48 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0SJOllE009358
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Thu, 28 Jan 2016 19:24:47 GMT
Received: from abhmp0019.oracle.com (abhmp0019.oracle.com [141.146.116.25])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0SJOlIp030098;
	Thu, 28 Jan 2016 19:24:47 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 28 Jan 2016 11:24:47 -0800
Date: Thu, 28 Jan 2016 11:24:46 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/4] xfstests: fix scratch device filtering in 139
Message-ID: <20160128192446.GB20038@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 1/4] xfstests: fix scratch device filtering in 139
References: <1454001236-2600-1-git-send-email-hch@lst.de>
 <1454001236-2600-2-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1454001236-2600-2-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0021.oracle.com [141.146.126.233]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1454009090
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26542
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Thu, Jan 28, 2016 at 06:13:53PM +0100, Christoph Hellwig wrote:
> Signed-off-by: Christoph Hellwig 
> ---
>  tests/xfs/139     | 4 ++--
>  tests/xfs/139.out | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
>  mode change 100644 => 100755 tests/xfs/139
> 
> diff --git a/tests/xfs/139 b/tests/xfs/139
> old mode 100644
> new mode 100755
> index 7400cf8..51c903e
> --- a/tests/xfs/139
> +++ b/tests/xfs/139
> @@ -69,8 +69,8 @@ _pwrite_byte 0x62 0 $sz "$testdir/file1" >> "$seqres.full"
>  _scratch_remount
>  
>  echo "Compare files"
> -md5sum "$testdir/file1" | _filter_test_dir
> -md5sum "$testdir/file2" | _filter_test_dir
> +md5sum "$testdir/file1" | _filter_scratch
> +md5sum "$testdir/file2" | _filter_scratch

This is fixed upstream.

Well, the filtering was fixed and the test now runs on the test device
and leaves scratchdev alone. :)

--D

>  #filefrag -v "$testdir/file1" "$testdir/file2"
>  
>  echo "Check scratch fs"
> diff --git a/tests/xfs/139.out b/tests/xfs/139.out
> index 729e2d6..090e3af 100644
> --- a/tests/xfs/139.out
> +++ b/tests/xfs/139.out
> @@ -3,6 +3,6 @@ Format and mount
>  Create the original files
>  CoW every other block
>  Compare files
> -673c759fb131ccf3336c4f8c549a497e  /opt/test-139/file1
> -dc802351f6717547cbf4c951b294f952  /opt/test-139/file2
> +673c759fb131ccf3336c4f8c549a497e  SCRATCH_MNT/test-139/file1
> +dc802351f6717547cbf4c951b294f952  SCRATCH_MNT/test-139/file2
>  Check scratch fs
> -- 
> 2.1.4
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From darrick.wong@oracle.com  Thu Jan 28 13:28:02 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id B8D5A7CA2
	for ; Thu, 28 Jan 2016 13:28:02 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id A26D1304039
	for ; Thu, 28 Jan 2016 11:27:59 -0800 (PST)
X-ASG-Debug-ID: 1454009277-04cb6c0d5d1230a0001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id 7HnDPovMP3rorjfP (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 11:27:57 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0SJRtZI008869
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Thu, 28 Jan 2016 19:27:55 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0SJRtDf001989
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Thu, 28 Jan 2016 19:27:55 GMT
Received: from abhmp0019.oracle.com (abhmp0019.oracle.com [141.146.116.25])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0SJRtGu031652;
	Thu, 28 Jan 2016 19:27:55 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 28 Jan 2016 11:27:55 -0800
Date: Thu, 28 Jan 2016 11:27:53 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 2/4] xfstests: require dedup ioctl support in 182
Message-ID: <20160128192753.GC20038@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 2/4] xfstests: require dedup ioctl support in 182
References: <1454001236-2600-1-git-send-email-hch@lst.de>
 <1454001236-2600-3-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1454001236-2600-3-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1454009277
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26542
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Thu, Jan 28, 2016 at 06:13:54PM +0100, Christoph Hellwig wrote:
> Signed-off-by: Christoph Hellwig 
> ---
>  tests/generic/182 | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tests/generic/182 b/tests/generic/182
> index bf5cd38..eb80b43 100755
> --- a/tests/generic/182
> +++ b/tests/generic/182
> @@ -46,6 +46,7 @@ _cleanup()
>  # real QA test starts here
>  _supported_os Linux
>  _require_test_reflink
> +_require_test_dedupe
>  _require_cp_reflink

The _require_*_reflink bits are unnecessary.  I must've been going too fast
when I wrote 181/182; I'll fix that in my repo and send patches.

Also 182's comments should s/reflink/dedupe/.

--D

>  
>  rm -f "$seqres.full"
> -- 
> 2.1.4
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From darrick.wong@oracle.com  Thu Jan 28 13:30:23 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 2AFF07CA2
	for ; Thu, 28 Jan 2016 13:30:23 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 0748D8F8039
	for ; Thu, 28 Jan 2016 11:30:22 -0800 (PST)
X-ASG-Debug-ID: 1454009414-04bdf04d6b11ea30001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id I9GTNR2i9ASWBdl1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 11:30:15 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0SJUC4V005817
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Thu, 28 Jan 2016 19:30:12 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0SJUCOK018046
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Thu, 28 Jan 2016 19:30:12 GMT
Received: from abhmp0004.oracle.com (abhmp0004.oracle.com [141.146.116.10])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0SJUBJt031505;
	Thu, 28 Jan 2016 19:30:12 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 28 Jan 2016 11:30:11 -0800
Date: Thu, 28 Jan 2016 11:30:10 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 3/4] xfstests: also treat EINVAL as reflink/dedup not
 supported
Message-ID: <20160128193010.GD20038@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 3/4] xfstests: also treat EINVAL as reflink/dedup not
 supported
References: <1454001236-2600-1-git-send-email-hch@lst.de>
 <1454001236-2600-4-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1454001236-2600-4-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1454009415
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Barracuda-BRTS-Status: 1
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26542
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Thu, Jan 28, 2016 at 06:13:55PM +0100, Christoph Hellwig wrote:
> This can be returned by various implementations of the ioctl.

Looks reasonable, so
Reviewed-by: Darrick J. Wong 

--D

> 
> Signed-off-by: Christoph Hellwig 
> ---
>  common/reflink | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/common/reflink b/common/reflink
> index de8e56d..8638aba 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -92,6 +92,8 @@ _require_test_dedupe()
>  		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
>  	echo $testio | grep -q "Inappropriate ioctl for device" && \
>  		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
> +	echo $testio | grep -q "Invalid argument" && \
> +		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
>  	rm -rf "$TEST_DIR/file1" "$TEST_DIR/file2"
>  }
>  
> @@ -110,6 +112,8 @@ _require_scratch_dedupe()
>  		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
>  	echo $testio | grep -q "Inappropriate ioctl for device" && \
>  		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
> +	echo $testio | grep -q "Invalid argument" && \
> +		_notrun "Dedupe not supported by test filesystem type: $FSTYP"
>  	_scratch_unmount
>  }
>  
> -- 
> 2.1.4
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From darrick.wong@oracle.com  Thu Jan 28 13:35:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 1CA3C7CA2
	for ; Thu, 28 Jan 2016 13:35:13 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 90C89AC001
	for ; Thu, 28 Jan 2016 11:35:12 -0800 (PST)
X-ASG-Debug-ID: 1454009710-04cbb026fe12a190001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id CIo3yIKXUbvDfbMp (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 11:35:10 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0SJZ8G6019035
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Thu, 28 Jan 2016 19:35:08 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0SJZ7B3032135
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Thu, 28 Jan 2016 19:35:08 GMT
Received: from abhmp0010.oracle.com (abhmp0010.oracle.com [141.146.116.16])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0SJZ7W4002014;
	Thu, 28 Jan 2016 19:35:07 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 28 Jan 2016 11:35:07 -0800
Date: Thu, 28 Jan 2016 11:35:06 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 1/4] xfstests: fix scratch device filtering in 139
Message-ID: <20160128193506.GE20038@birch.djwong.org>
X-ASG-Orig-Subj: Re: [PATCH 1/4] xfstests: fix scratch device filtering in 139
References: <1454001236-2600-1-git-send-email-hch@lst.de>
 <1454001236-2600-2-git-send-email-hch@lst.de>
 <20160128192446.GB20038@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160128192446.GB20038@birch.djwong.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1454009710
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26542
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Thu, Jan 28, 2016 at 11:24:46AM -0800, Darrick J. Wong wrote:
> On Thu, Jan 28, 2016 at 06:13:53PM +0100, Christoph Hellwig wrote:
> > Signed-off-by: Christoph Hellwig 
> > ---
> >  tests/xfs/139     | 4 ++--
> >  tests/xfs/139.out | 4 ++--
> >  2 files changed, 4 insertions(+), 4 deletions(-)
> >  mode change 100644 => 100755 tests/xfs/139
> > 
> > diff --git a/tests/xfs/139 b/tests/xfs/139
> > old mode 100644
> > new mode 100755
> > index 7400cf8..51c903e
> > --- a/tests/xfs/139
> > +++ b/tests/xfs/139

DOH             ^^^

/me grabs another cup of coffee, tries again.

This patch is fine, ignore my earlier statements.

Signed-off-by: Darrick J. Wong 

--D

> > @@ -69,8 +69,8 @@ _pwrite_byte 0x62 0 $sz "$testdir/file1" >> "$seqres.full"
> >  _scratch_remount
> >  
> >  echo "Compare files"
> > -md5sum "$testdir/file1" | _filter_test_dir
> > -md5sum "$testdir/file2" | _filter_test_dir
> > +md5sum "$testdir/file1" | _filter_scratch
> > +md5sum "$testdir/file2" | _filter_scratch
> 
> This is fixed upstream.
> 
> Well, the filtering was fixed and the test now runs on the test device
> and leaves scratchdev alone. :)
> 
> --D
> 
> >  #filefrag -v "$testdir/file1" "$testdir/file2"
> >  
> >  echo "Check scratch fs"
> > diff --git a/tests/xfs/139.out b/tests/xfs/139.out
> > index 729e2d6..090e3af 100644
> > --- a/tests/xfs/139.out
> > +++ b/tests/xfs/139.out
> > @@ -3,6 +3,6 @@ Format and mount
> >  Create the original files
> >  CoW every other block
> >  Compare files
> > -673c759fb131ccf3336c4f8c549a497e  /opt/test-139/file1
> > -dc802351f6717547cbf4c951b294f952  /opt/test-139/file2
> > +673c759fb131ccf3336c4f8c549a497e  SCRATCH_MNT/test-139/file1
> > +dc802351f6717547cbf4c951b294f952  SCRATCH_MNT/test-139/file2
> >  Check scratch fs
> > -- 
> > 2.1.4
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From darrick.wong@oracle.com  Thu Jan 28 14:54:08 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id D31DE7CA2
	for ; Thu, 28 Jan 2016 14:54:08 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id C43688F804B
	for ; Thu, 28 Jan 2016 12:54:05 -0800 (PST)
X-ASG-Debug-ID: 1454014443-04cbb026fd12bca0001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id lRp4oI4Gv7LEGvKk (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 12:54:03 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0SKrZTH016917
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Thu, 28 Jan 2016 20:53:36 GMT
Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0SKrZdX007669
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Thu, 28 Jan 2016 20:53:35 GMT
Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19])
	by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0SKrYu4005101;
	Thu, 28 Jan 2016 20:53:35 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 28 Jan 2016 12:53:34 -0800
Date: Thu, 28 Jan 2016 12:53:33 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: Christoph Hellwig , xfs@oss.sgi.com
Subject: Re: stop using ioends for direct write completions
Message-ID: <20160128205333.GF20038@birch.djwong.org>
X-ASG-Orig-Subj: Re: stop using ioends for direct write completions
References: <1452766237-2314-1-git-send-email-hch@lst.de>
 <20160128131656.GB14876@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160128131656.GB14876@infradead.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1454014443
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26544
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Thu, Jan 28, 2016 at 05:16:56AM -0800, Christoph Hellwig wrote:
> Any chance to get a review for this?  It should really help
> with sorting out the buffered I/O COW code.

It looks reasonable to me.  I separated the dio and buffered CoW remap paths a
couple of weeks ago, because it seems that IO errors only get passed back as a
return value from __blockdev_direct_IO, therefore the remapping has to be done
from xfs_vm_do_dio anyway because we don't want to remap if the write fails.
Just yesterday I removed the "is_cow" flag from the ioend, so now we're back to
having a separate XFS_IO_COW ioend type.

So... reflink doesn't need the patch but OTOH directio doesn't really need 
the overhead of allocating an ioend anyway. :)

--D

> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From darrick.wong@oracle.com  Thu Jan 28 15:03:12 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 90BAA7CA2
	for ; Thu, 28 Jan 2016 15:03:12 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 22883AC002
	for ; Thu, 28 Jan 2016 13:03:09 -0800 (PST)
X-ASG-Debug-ID: 1454014980-04cb6c0d60125010001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id QFF93vrD1LIolEDF (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 13:03:00 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0SL2vW2005656
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Thu, 28 Jan 2016 21:02:58 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0SL2v5r020404
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Thu, 28 Jan 2016 21:02:57 GMT
Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0SL2v5O015924;
	Thu, 28 Jan 2016 21:02:57 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 28 Jan 2016 13:02:56 -0800
Date: Thu, 28 Jan 2016 13:02:55 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: a couple xfstests fixes for reflinks
Message-ID: <20160128210255.GA6431@birch.djwong.org>
X-ASG-Orig-Subj: Re: a couple xfstests fixes for reflinks
References: <1454001236-2600-1-git-send-email-hch@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1454001236-2600-1-git-send-email-hch@lst.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: userv0022.oracle.com [156.151.31.74]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1454014980
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26544
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Thu, Jan 28, 2016 at 06:13:52PM +0100, Christoph Hellwig wrote:
> See http://www.infradead.org/rpr.html

Hrmm, have I been sending junk again?  I've noticed a few transient problems
with Oracle's mailservers lately...

> The patches (against Darrick's for-dave branch) fix various failures I see
> during testing XFS and NFS.

I've integrated your fixes with my dev tree and pushed to github.

There's a bunch of new tests in there to look at quota behaviors.  Evidently
btrfs doesn't support classic usr/group quotas(?) and ocfs2 exploded when I
tried it, so for now I'll just charge each quota at reflink time; copy on write
is therefore "free".

--D

From hch@lst.de  Thu Jan 28 15:11:03 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 1DA217CA2
	for ; Thu, 28 Jan 2016 15:11:03 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id ED485304039
	for ; Thu, 28 Jan 2016 13:10:59 -0800 (PST)
X-ASG-Debug-ID: 1454015456-04cbb026fe12c2c0001-NocioJ
Received: from newverein.lst.de (verein.lst.de [213.95.11.211]) by cuda.sgi.com with ESMTP id AkuQkjx25zQg75nA (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 13:10:57 -0800 (PST)
X-Barracuda-Envelope-From: hch@lst.de
X-Barracuda-Apparent-Source-IP: 213.95.11.211
Received: by newverein.lst.de (Postfix, from userid 2407)
	id 1E435DE409; Thu, 28 Jan 2016 22:10:56 +0100 (CET)
Date: Thu, 28 Jan 2016 22:10:56 +0100
From: Christoph Hellwig 
To: "Darrick J. Wong" 
Cc: xfs@oss.sgi.com
Subject: Re: stop using ioends for direct write completions
Message-ID: <20160128211056.GA27287@lst.de>
X-ASG-Orig-Subj: Re: stop using ioends for direct write completions
References: <1452766237-2314-1-git-send-email-hch@lst.de> <20160128131656.GB14876@infradead.org> <20160128205333.GF20038@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160128205333.GF20038@birch.djwong.org>
User-Agent: Mutt/1.5.17 (2007-11-01)
X-Barracuda-Connect: verein.lst.de[213.95.11.211]
X-Barracuda-Start-Time: 1454015457
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26544
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu, Jan 28, 2016 at 12:53:33PM -0800, Darrick J. Wong wrote:
> On Thu, Jan 28, 2016 at 05:16:56AM -0800, Christoph Hellwig wrote:
> > Any chance to get a review for this?  It should really help
> > with sorting out the buffered I/O COW code.
> 
> It looks reasonable to me.  I separated the dio and buffered CoW remap paths a
> couple of weeks ago, because it seems that IO errors only get passed back as a
> return value from __blockdev_direct_IO, therefore the remapping has to be done
> from xfs_vm_do_dio anyway because we don't want to remap if the write fails.
> Just yesterday I removed the "is_cow" flag from the ioend, so now we're back to
> having a separate XFS_IO_COW ioend type.

For direct I/O we will need something like this to properly support AIO
writes.

> So... reflink doesn't need the patch but OTOH directio doesn't really need 
> the overhead of allocating an ioend anyway. :)

Yep.

From darrick.wong@oracle.com  Thu Jan 28 15:59:04 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 214527CA2
	for ; Thu, 28 Jan 2016 15:59:04 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D9E308F8039
	for ; Thu, 28 Jan 2016 13:59:00 -0800 (PST)
X-ASG-Debug-ID: 1454018338-04bdf04d6b121f60001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id VIqIt3Huwhu2EGfl (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 13:58:59 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0SLwuoR013623
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Thu, 28 Jan 2016 21:58:56 GMT
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0SLwtwT006322
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Thu, 28 Jan 2016 21:58:55 GMT
Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20])
	by userv0122.oracle.com (8.14.4/8.13.8) with ESMTP id u0SLwt62002423;
	Thu, 28 Jan 2016 21:58:55 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 28 Jan 2016 13:58:54 -0800
Date: Thu, 28 Jan 2016 13:58:53 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: stop using ioends for direct write completions
Message-ID: <20160128215853.GC6431@birch.djwong.org>
X-ASG-Orig-Subj: Re: stop using ioends for direct write completions
References: <1452766237-2314-1-git-send-email-hch@lst.de>
 <20160128131656.GB14876@infradead.org>
 <20160128205333.GF20038@birch.djwong.org>
 <20160128211056.GA27287@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160128211056.GA27287@lst.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1454018339
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26546
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Thu, Jan 28, 2016 at 10:10:56PM +0100, Christoph Hellwig wrote:
> On Thu, Jan 28, 2016 at 12:53:33PM -0800, Darrick J. Wong wrote:
> > On Thu, Jan 28, 2016 at 05:16:56AM -0800, Christoph Hellwig wrote:
> > > Any chance to get a review for this?  It should really help
> > > with sorting out the buffered I/O COW code.
> > 
> > It looks reasonable to me.  I separated the dio and buffered CoW remap paths a
> > couple of weeks ago, because it seems that IO errors only get passed back as a
> > return value from __blockdev_direct_IO, therefore the remapping has to be done
> > from xfs_vm_do_dio anyway because we don't want to remap if the write fails.
> > Just yesterday I removed the "is_cow" flag from the ioend, so now we're back to
> > having a separate XFS_IO_COW ioend type.
> 
> For direct I/O we will need something like this to properly support AIO
> writes.

Aw, snap, I knew I'd forgotten something.  Yep, we'll need that... I think
xfs_end_io_direct_write will have to sniff out the error status from "size"
and either remap or discard the CoW allocations as appropriate.

Heh, guess I'd better go write some aio tests. :)

> > So... reflink doesn't need the patch but OTOH directio doesn't really need 
> > the overhead of allocating an ioend anyway. :)
> 
> Yep.

--D

From hch@lst.de  Thu Jan 28 16:03:02 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 1EE287CA2
	for ; Thu, 28 Jan 2016 16:03:02 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id A3571AC001
	for ; Thu, 28 Jan 2016 14:02:58 -0800 (PST)
X-ASG-Debug-ID: 1454018575-04cbb026fd12d5b0001-NocioJ
Received: from newverein.lst.de (verein.lst.de [213.95.11.211]) by cuda.sgi.com with ESMTP id hmZP4fFZFCQtTsGQ (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 14:02:56 -0800 (PST)
X-Barracuda-Envelope-From: hch@lst.de
X-Barracuda-Apparent-Source-IP: 213.95.11.211
Received: by newverein.lst.de (Postfix, from userid 2407)
	id 2914FDE409; Thu, 28 Jan 2016 23:02:55 +0100 (CET)
Date: Thu, 28 Jan 2016 23:02:55 +0100
From: Christoph Hellwig 
To: "Darrick J. Wong" 
Cc: xfs@oss.sgi.com
Subject: Re: stop using ioends for direct write completions
Message-ID: <20160128220255.GA28202@lst.de>
X-ASG-Orig-Subj: Re: stop using ioends for direct write completions
References: <1452766237-2314-1-git-send-email-hch@lst.de> <20160128131656.GB14876@infradead.org> <20160128205333.GF20038@birch.djwong.org> <20160128211056.GA27287@lst.de> <20160128215853.GC6431@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160128215853.GC6431@birch.djwong.org>
User-Agent: Mutt/1.5.17 (2007-11-01)
X-Barracuda-Connect: verein.lst.de[213.95.11.211]
X-Barracuda-Start-Time: 1454018576
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26545
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu, Jan 28, 2016 at 01:58:53PM -0800, Darrick J. Wong wrote:
> Aw, snap, I knew I'd forgotten something.  Yep, we'll need that... I think
> xfs_end_io_direct_write will have to sniff out the error status from "size"
> and either remap or discard the CoW allocations as appropriate.

I'd rather fix the direct I/O code to give us that information directly
(pun intended).  I'll add that to my short term todo list as it seems
useful for the existing code as well.

From darrick.wong@oracle.com  Thu Jan 28 16:31:44 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id E1D237CA2
	for ; Thu, 28 Jan 2016 16:31:44 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id B3C918F8040
	for ; Thu, 28 Jan 2016 14:31:38 -0800 (PST)
X-ASG-Debug-ID: 1454020296-04bdf04d6c1235f0001-NocioJ
Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by cuda.sgi.com with ESMTP id 6Wo7kuHEPyJlPbZe (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 28 Jan 2016 14:31:37 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 156.151.31.81
Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234])
	by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0SMVX6u009008
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Thu, 28 Jan 2016 22:31:34 GMT
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
	by aserv0022.oracle.com (8.13.8/8.13.8) with ESMTP id u0SMVXDx018169
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Thu, 28 Jan 2016 22:31:33 GMT
Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15])
	by userv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u0SMVWKN028835;
	Thu, 28 Jan 2016 22:31:33 GMT
Received: from localhost (/10.145.178.207)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Thu, 28 Jan 2016 14:31:32 -0800
Date: Thu, 28 Jan 2016 14:31:31 -0800
From: "Darrick J. Wong" 
To: Christoph Hellwig 
Cc: xfs@oss.sgi.com
Subject: Re: stop using ioends for direct write completions
Message-ID: <20160128223131.GD6431@birch.djwong.org>
X-ASG-Orig-Subj: Re: stop using ioends for direct write completions
References: <1452766237-2314-1-git-send-email-hch@lst.de>
 <20160128131656.GB14876@infradead.org>
 <20160128205333.GF20038@birch.djwong.org>
 <20160128211056.GA27287@lst.de>
 <20160128215853.GC6431@birch.djwong.org>
 <20160128220255.GA28202@lst.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160128220255.GA28202@lst.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0022.oracle.com [141.146.126.234]
X-Barracuda-Connect: userp1040.oracle.com[156.151.31.81]
X-Barracuda-Start-Time: 1454020297
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26547
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Thu, Jan 28, 2016 at 11:02:55PM +0100, Christoph Hellwig wrote:
> On Thu, Jan 28, 2016 at 01:58:53PM -0800, Darrick J. Wong wrote:
> > Aw, snap, I knew I'd forgotten something.  Yep, we'll need that... I think
> > xfs_end_io_direct_write will have to sniff out the error status from "size"
> > and either remap or discard the CoW allocations as appropriate.
> 
> I'd rather fix the direct I/O code to give us that information directly
> (pun intended).  I'll add that to my short term todo list as it seems
> useful for the existing code as well.

Ok.

As for the generic/154... the code on github is sadly not very bisectable,
other than to say that until yesterday I was still tacking new code onto
the end of the patchset.

That said, I was seeing occasional hangs in generic/154 (I needed to roll
the transactions between key parts of wrapping up a CoW) and they seem to
have gone away, so it's possible that I've fixed it already.

--D

From hch@lst.de  Fri Jan 29 02:02:03 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id C5C1B7CA2
	for ; Fri, 29 Jan 2016 02:02:03 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id B60848F8039
	for ; Fri, 29 Jan 2016 00:02:00 -0800 (PST)
X-ASG-Debug-ID: 1454054518-04cb6c0d5d1341c0001-NocioJ
Received: from newverein.lst.de (verein.lst.de [213.95.11.211]) by cuda.sgi.com with ESMTP id WxopbIoB74MSEeCh (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 29 Jan 2016 00:01:58 -0800 (PST)
X-Barracuda-Envelope-From: hch@lst.de
X-Barracuda-Apparent-Source-IP: 213.95.11.211
Received: by newverein.lst.de (Postfix, from userid 2407)
	id 623FF692C7; Fri, 29 Jan 2016 09:01:57 +0100 (CET)
Date: Fri, 29 Jan 2016 09:01:57 +0100
From: Christoph Hellwig 
To: "Darrick J. Wong" 
Cc: xfs@oss.sgi.com
Subject: Re: stop using ioends for direct write completions
Message-ID: <20160129080157.GA30986@lst.de>
X-ASG-Orig-Subj: Re: stop using ioends for direct write completions
References: <1452766237-2314-1-git-send-email-hch@lst.de> <20160128131656.GB14876@infradead.org> <20160128205333.GF20038@birch.djwong.org> <20160128211056.GA27287@lst.de> <20160128215853.GC6431@birch.djwong.org> <20160128220255.GA28202@lst.de> <20160128223131.GD6431@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160128223131.GD6431@birch.djwong.org>
User-Agent: Mutt/1.5.17 (2007-11-01)
X-Barracuda-Connect: verein.lst.de[213.95.11.211]
X-Barracuda-Start-Time: 1454054518
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26559
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu, Jan 28, 2016 at 02:31:31PM -0800, Darrick J. Wong wrote:
> As for the generic/154... the code on github is sadly not very bisectable,
> other than to say that until yesterday I was still tacking new code onto
> the end of the patchset.
> 
> That said, I was seeing occasional hangs in generic/154 (I needed to roll
> the transactions between key parts of wrapping up a CoW) and they seem to
> have gone away, so it's possible that I've fixed it already.

I'm seeing them with a HEAD of 9628553acf5ff4225e047d580094ffab64605f50,
but only when using NFS.  If you've got something newer feel free
to push it out, otherwise I'll have to spend some more time
trying to figure out what's going on with this version.

From c.affolter@purplehaze.ch  Fri Jan 29 04:53:54 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id C192F7CA2
	for ; Fri, 29 Jan 2016 04:53:54 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id B302E304032
	for ; Fri, 29 Jan 2016 02:53:51 -0800 (PST)
X-ASG-Debug-ID: 1454064825-04cbb026ff13f0f0001-NocioJ
Received: from mail.stepping-stone.ch (mail.stepping-stone.ch [194.176.109.206]) by cuda.sgi.com with ESMTP id ErkFmV3lkv7CQEbA (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 29 Jan 2016 02:53:46 -0800 (PST)
X-Barracuda-Envelope-From: c.affolter@purplehaze.ch
X-Barracuda-Apparent-Source-IP: 194.176.109.206
Received: from localhost (janitor-01.int.stepping-stone.ch [10.17.98.12])
	by mta-01.int.stepping-stone.ch (Postfix) with ESMTP id 4E6D8A9CB5
	for ; Fri, 29 Jan 2016 11:53:45 +0100 (CET)
X-Virus-Scanned: amavisd-new at stepping-stone.ch
Received: from mta-01.int.stepping-stone.ch ([10.17.98.30])
	by localhost (janitor-01.int.stepping-stone.ch [10.17.98.12]) (amavisd-new, port 10026)
	with LMTP id Uqo2kILN-G9V for ;
	Fri, 29 Jan 2016 11:53:36 +0100 (CET)
Received: from [10.84.1.149] (224.109.22.178.ftth.as8758.net [178.22.109.224])
	(using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits))
	(Client did not present a certificate)
	by smtp.stepping-stone.ch (Postfix) with ESMTPSA id 57C81A9CAA
	for ; Fri, 29 Jan 2016 11:53:36 +0100 (CET)
To: xfs@oss.sgi.com
From: Christian Affolter 
X-Enigmail-Draft-Status: N1110
Subject: Random write result differences between RAID device and XFS
Message-ID: <56AB44AF.8020807@purplehaze.ch>
X-ASG-Orig-Subj: Random write result differences between RAID device and XFS
Date: Fri, 29 Jan 2016 11:53:35 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: mail.stepping-stone.ch[194.176.109.206]
X-Barracuda-Start-Time: 1454064826
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26562
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Hi everyone,

I'm trying to understand the differences of some bandwidth and IOPs test
results I see while running a random-write full-stripe-width aligned fio
test (using libaio with direct IO) on a hardware RAID 6 raw device
versus on the same device with the XFS file system on top of it.

On the raw device I get:
write: io=24828MB, bw=423132KB/s, iops=137, runt= 60085msec

With XFS on top of it:
write: io=14658MB, bw=249407KB/s, iops=81, runt= 60182msec


The hardware RAID 6 volume consists out of 5 HDDs (3 data disks), the
stripe unit size is 1 MiB and full stripe width is 3 MiB.

XFS was initialized and mounted with the following commands:

mkfs.xfs -d su=1024k,sw=3 -L LV-TEST-02 /dev/sdd
mount -o inode64,noatime -L LV-TEST-02 /mnt/lv-test-02

mkfs.xfs version 3.2.2

xfs_info /mnt/lv-test-02
meta-data=/dev/sdd               isize=256    agcount=16, agsize=819200 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=13106688, imaxpct=25
         =                       sunit=256    swidth=768 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=6399, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0



The RAID controller does not export optimal_io_size.


Controller details:
Product Name = AVAGO 3108 MegaRAID
FW Package Build = 24.9.0-0022
BIOS Version = 6.25.03.0_4.17.08.00_0x060E0300
FW Version = 4.290.00-4536
Driver Name = megaraid_sas
Driver Version = 06.808.16.00-rc1


Virtual drive:

--------------------------------------------------------------------
DG/VD TYPE  State Access Consist Cache sCC      Size Name
--------------------------------------------------------------------
1/2   RAID6 Optl  RW     Yes      RWBD  -   49.998 GB vd-hdd-test-01
--------------------------------------------------------------------

R=Read Ahead
WB=WriteBack (with a battery backup)
D=Direct IO

The physical disk cache is disabled.


Disks:
5x HGST HUH728080AL5200 (Firmware Revision = A515)


Kernel:
4.4.0-2.el7.elrepo.x86_64


fio command and output for RAID raw device:

fio --filename=/dev/sdd \
    --direct=1 \
    --rw=randwrite \
    --ioengine=libaio \
    --iodepth=16 \
    --numjobs=1 \
    --runtime=60 \
    --exec_prerun="/opt/MegaRAID/storcli/storcli64 /c0 flushcache" \
    --name=direct-raid-hdd-random-write-full-stripe-aligned-3072k \
    --bs=3072k

direct-raid-hdd-random-write-full-stripe-aligned-3072k: (g=0):
rw=randwrite, bs=3M-3M/3M-3M/3M-3M, ioengine=libaio, iodepth=16
fio-2.2.8
Starting 1 process
direct-raid-hdd-random-write-full-stripe-aligned-3072k : Saving output
of prerun in
direct-raid-hdd-random-write-full-stripe-aligned-3072k.prerun.txt
Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/375.0MB/0KB /s] [0/125/0 iops]
[eta 00m:00s]
direct-raid-hdd-random-write-full-stripe-aligned-3072k: (groupid=0,
jobs=1): err= 0: pid=1847: Fri Jan 29 11:47:17 2016
  write: io=24828MB, bw=423132KB/s, iops=137, runt= 60085msec
    slat (usec): min=250, max=91308, avg=7250.27, stdev=11767.73
    clat (msec): min=8, max=223, avg=108.89, stdev=32.22
     lat (msec): min=8, max=224, avg=116.14, stdev=32.19
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   43], 10.00th=[   78], 20.00th=[   91],
     | 30.00th=[   99], 40.00th=[  106], 50.00th=[  113], 60.00th=[  119],
     | 70.00th=[  126], 80.00th=[  133], 90.00th=[  145], 95.00th=[  153],
     | 99.00th=[  169], 99.50th=[  176], 99.90th=[  198], 99.95th=[  202],
     | 99.99th=[  225]
    bw (KB  /s): min=348681, max=2599384, per=100.00%, avg=423757.22,
stdev=204979.51
    lat (msec) : 10=4.22%, 20=0.76%, 50=0.19%, 100=25.86%, 250=68.97%
  cpu          : usr=2.49%, sys=3.57%, ctx=2959, majf=0, minf=1642
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=0/w=8276/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: io=24828MB, aggrb=423131KB/s, minb=423131KB/s, maxb=423131KB/s,
mint=60085msec, maxt=60085msec

Disk stats (read/write):
  sdd: ios=59/98996, merge=0/0, ticks=150/8434406, in_queue=8443626,
util=100.00%



fio command and output for XFS:

fio --directory=/mnt/lv-test-02 \
    --filename=test.fio  \
    --size=30g \
    --direct=1 \
    --rw=randwrite \
    --ioengine=libaio \
    --iodepth=16 \
    --numjobs=1 \
    --runtime=60 \
    --exec_prerun="/opt/MegaRAID/storcli/storcli64 /c0 flushcache" \
    --name=xfs-Ä¥dd-random-write-full-stripe-aligned-3072k \
    --bs=3072k


xfs-Ä¥dd-random-write-full-stripe-aligned-3072k: (g=0): rw=randwrite,
bs=3M-3M/3M-3M/3M-3M, ioengine=libaio, iodepth=16
fio-2.2.8
Starting 1 process
xfs-Ä¥dd-random-write-full-stripe-aligned-3072k: Laying out IO file(s) (1
file(s) / 30720MB)
xfs-Ä¥dd-random-write-full-stripe-aligned-3072k : Saving output of prerun
in xfs-Ä¥dd-random-write-full-stripe-aligned-3072k.prerun.txt
Jobs: 1 (f=1): [w(1)] [100.0% done] [0KB/186.0MB/0KB /s] [0/62/0 iops]
[eta 00m:00s]
xfs-Ä¥dd-random-write-full-stripe-aligned-3072k: (groupid=0, jobs=1):
err= 0: pid=1899: Fri Jan 29 11:50:21 2016
  write: io=14658MB, bw=249407KB/s, iops=81, runt= 60182msec
    slat (usec): min=231, max=133647, avg=12279.35, stdev=20234.23
    clat (msec): min=4, max=1987, avg=184.75, stdev=81.74
     lat (msec): min=5, max=1987, avg=197.03, stdev=83.84
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[  110], 20.00th=[  143],
     | 30.00th=[  161], 40.00th=[  174], 50.00th=[  188], 60.00th=[  202],
     | 70.00th=[  217], 80.00th=[  237], 90.00th=[  269], 95.00th=[  293],
     | 99.00th=[  363], 99.50th=[  416], 99.90th=[  742], 99.95th=[  922],
     | 99.99th=[ 1991]
    bw (KB  /s): min=130620, max=2460047, per=100.00%, avg=250120.43,
stdev=212307.87
    lat (msec) : 10=8.04%, 100=0.88%, 250=76.44%, 500=14.31%, 750=0.25%
    lat (msec) : 1000=0.04%, 2000=0.04%
  cpu          : usr=1.10%, sys=2.30%, ctx=1891, majf=0, minf=1096
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=0/w=4886/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: io=14658MB, aggrb=249406KB/s, minb=249406KB/s, maxb=249406KB/s,
mint=60182msec, maxt=60182msec

Disk stats (read/write):
  sdd: ios=0/58627, merge=0/12, ticks=0/8552722, in_queue=8559550,
util=99.84%


Many thanks in advance
Chris

From bfoster@redhat.com  Fri Jan 29 08:12:40 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 212997CA2
	for ; Fri, 29 Jan 2016 08:12:40 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 124AD8F8040
	for ; Fri, 29 Jan 2016 06:12:39 -0800 (PST)
X-ASG-Debug-ID: 1454076755-04cb6c0d5d13e260001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id namETBY8Cq3RcjAy (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 29 Jan 2016 06:12:35 -0800 (PST)
X-Barracuda-Envelope-From: bfoster@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	by mx1.redhat.com (Postfix) with ESMTPS id CB0793296;
	Fri, 29 Jan 2016 14:12:34 +0000 (UTC)
Received: from bfoster.bfoster (dhcp-41-80.bos.redhat.com [10.18.41.80])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0TECY4x007240;
	Fri, 29 Jan 2016 09:12:34 -0500
Received: by bfoster.bfoster (Postfix, from userid 1000)
	id 4E5071251C3; Fri, 29 Jan 2016 09:12:33 -0500 (EST)
Date: Fri, 29 Jan 2016 09:12:33 -0500
From: Brian Foster 
To: Christoph Hellwig 
Cc: Christoph Hellwig , darrick.wong@oracle.com, xfs@oss.sgi.com
Subject: Re: stop using ioends for direct write completions
Message-ID: <20160129141232.GA43184@bfoster.bfoster>
X-ASG-Orig-Subj: Re: stop using ioends for direct write completions
References: <1452766237-2314-1-git-send-email-hch@lst.de>
 <20160128131656.GB14876@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160128131656.GB14876@infradead.org>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1454076755
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Thu, Jan 28, 2016 at 05:16:56AM -0800, Christoph Hellwig wrote:
> Any chance to get a review for this?  It should really help
> with sorting out the buffered I/O COW code.
> 

I haven't taken a closer look at the patch yet. I was kind of waiting
for Dave to chime in because I'm a little confused over the back and
forth nature of dio/ioend completion lately.

My understanding is that the original requirement for ioends here was to
track the state necessary in order to defer (to wq) completions that had
to allocate transactions. Eventually, the deferred buffer state was
implemented and we no longer required an ioend for that, so we removed
the ioends here:

  2ba6623 xfs: don't allocate an ioend for direct I/O completions

Then just a couple months later, we merged the following to re-add the
ioend into the dio path:

  d5cc2e3f xfs: DIO needs an ioend for writes

I recall reviewing that one, but looking back afaict the ioend was used
simply to enable reuse of the ioend completion code. Now we have this
patch which presumably removes much of that code to eliminate the ioend
allocation overhead.

Neither this patch nor the previous has any technical reasoning for one
approach over the other in the commit logs, so afaics this appears to be
a matter of preference. Can we get some agreement (or discussion) on
what the right interface is to transfer information to dio completion?
E.g., is this allocation overhead noticeable? Is ioend usage problematic
for other reasons (such as copy-on-write)? Going back to the previous
patch, were there explicit reasons for using ioends aside from code
reuse? Note that the subsequent commit 6dfa1b67 ("xfs: handle DIO
overwrite EOF update completion correctly") does refer to some problems
not running dio completions in the right context... Dave?

Brian

> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From debbugs@buxtehude.debian.org  Fri Jan 29 10:12:23 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 653F27CA2
	for ; Fri, 29 Jan 2016 10:12:23 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay3.corp.sgi.com (Postfix) with ESMTP id F04B7AC001
	for ; Fri, 29 Jan 2016 08:12:19 -0800 (PST)
X-ASG-Debug-ID: 1454083937-04bdf04d6c145090001-NocioJ
Received: from buxtehude.debian.org (buxtehude.debian.org [206.12.19.147]) by cuda.sgi.com with ESMTP id qStZFU6dSQj8FOC8 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 29 Jan 2016 08:12:17 -0800 (PST)
X-Barracuda-Envelope-From: debbugs@buxtehude.debian.org
X-Barracuda-Apparent-Source-IP: 206.12.19.147
Received: from debbugs by buxtehude.debian.org with local (Exim 4.84)
	(envelope-from )
	id 1aPBee-0008LZ-WA; Fri, 29 Jan 2016 16:12:13 +0000
X-Loop: owner@bugs.debian.org
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
From: owner@bugs.debian.org (Debian Bug Tracking System)
To: Andreas Beckmann 
CC: xfs@oss.sgi.com, python-modules-team@lists.alioth.debian.org,
 debian-science-maintainers@lists.alioth.debian.org
Subject: Processed: reassign 811231 to src:mpich, fixed 811231 in 3.2-6,
 notfound 806353 in 1.9-1 ...
Message-ID: 
X-ASG-Orig-Subj: Processed: reassign 811231 to src:mpich, fixed 811231 in 3.2-6,
 notfound 806353 in 1.9-1 ...
References: <1454083782-2329-bts-anbe@debian.org>
X-Debian-PR-Package: src:python-django xfslibs-dev src:petsc src:mpich
X-Debian-PR-Source: mpich petsc python-django xfsprogs
X-Debian-PR-Message: transcript
X-Loop: owner@bugs.debian.org
Date: Fri, 29 Jan 2016 16:12:12 +0000
X-Barracuda-Connect: buxtehude.debian.org[206.12.19.147]
X-Barracuda-Start-Time: 1454083937
X-Barracuda-Encrypted: ECDHE-RSA-AES128-GCM-SHA256
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26567
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Processing commands for control@bugs.debian.org:

> reassign 811231 src:mpich 3.1-6
Bug #811231 {Done: Anton Gladky } [src:mpich,src:petsc] /=
usr/lib/mpich/lib doesn't exist, causing petsc fail to build on mpich archs
Bug reassigned from package 'src:mpich,src:petsc' to 'src:mpich'.
Ignoring request to alter found versions of bug #811231 to the same values =
previously set
No longer marked as fixed in versions mpich/3.2-6.
Bug #811231 {Done: Anton Gladky } [src:mpich] /usr/lib/mp=
ich/lib doesn't exist, causing petsc fail to build on mpich archs
Marked as found in versions mpich/3.1-6.
> fixed 811231 3.2-6
Bug #811231 {Done: Anton Gladky } [src:mpich] /usr/lib/mp=
ich/lib doesn't exist, causing petsc fail to build on mpich archs
Marked as fixed in versions mpich/3.2-6.
> notfound 806353 1.9-1
Bug #806353 {Done: Raphael Hertzog } [src:python-django=
] django-picklefield: FTBFS with Django 1.9
No longer marked as found in versions python-django/1.9-1.
> notfixed 766811 4.3.0
Bug #766811 [xfslibs-dev] spurious library links
No longer marked as fixed in versions 4.3.0.
> thanks
Stopping processing here.

Please contact me if you need assistance.
--=20
766811: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D766811
806353: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D806353
811231: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D811231
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

From ross.zwisler@linux.intel.com  Fri Jan 29 13:21:53 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 035907CA2
	for ; Fri, 29 Jan 2016 13:21:53 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id D163E8F804B
	for ; Fri, 29 Jan 2016 11:21:49 -0800 (PST)
X-ASG-Debug-ID: 1454095308-04cbb026fc14f610001-NocioJ
Received: from mga14.intel.com ([192.55.52.115]) by cuda.sgi.com with ESMTP id rC8xxCl0rDx5n3Or for ; Fri, 29 Jan 2016 11:21:48 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.115
X-ASG-Whitelist: Client
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
  by fmsmga103.fm.intel.com with ESMTP; 29 Jan 2016 11:21:48 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,365,1449561600"; 
   d="scan'208";a="904044917"
Received: from tarkir.lm.intel.com (HELO linux.intel.com) ([10.232.112.142])
  by fmsmga002.fm.intel.com with ESMTP; 29 Jan 2016 11:21:48 -0800
Date: Fri, 29 Jan 2016 12:21:45 -0700
From: Ross Zwisler 
To: xfs@oss.sgi.com
Subject: xfstests xfs/293 failure with current xfstests
Message-ID: <20160129192145.GC5224@linux.intel.com>
X-ASG-Orig-Subj: xfstests xfs/293 failure with current xfstests
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.115]
X-Barracuda-Start-Time: 1454095308
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

It looks like the current version of xfsprogs (v4.3.0) has an inconsistency
between its documentation and implementation that is making the xfstest
xfs/293 fail:

./check xfs/293
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 lorwyn 4.5.0-rc1
MKFS_OPTIONS  -- -f -bsize=4096 /dev/pmem0p2
MOUNT_OPTIONS -- -o dax -o context=system_u:object_r:nfs_t:s0 /dev/pmem0p2 /mnt/xfstests_scratch

xfs/293	 - output mismatch (see /root/xfstests/results//xfs/293.out.bad)
    --- tests/xfs/293.out	2015-10-02 10:19:36.836796097 -0600
    +++ /root/xfstests/results//xfs/293.out.bad	2016-01-29 10:41:25.530512911 -0700
    @@ -1,2 +1,4 @@
     QA output created by 293
     Silence is golden
    +help not documented in the xfs_io manpage
    +zero not documented in the xfs_io manpage
    ...
    (Run 'diff -u tests/xfs/293.out /root/xfstests/results//xfs/293.out.bad'  to see the entire diff)
Ran: xfs/293
Failures: xfs/293
Failed 1 of 1 tests

"help" and "zero" are indeed options in xfs_io, but aren't listed in the man
page.

Thanks,
- Ross

From darrick.wong@oracle.com  Fri Jan 29 14:37:55 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY
	autolearn=ham version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id E94227CA2
	for ; Fri, 29 Jan 2016 14:37:55 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id BBD5530405F
	for ; Fri, 29 Jan 2016 12:37:55 -0800 (PST)
X-ASG-Debug-ID: 1454099864-04bdf04d6d14c540001-NocioJ
Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by cuda.sgi.com with ESMTP id YBHuc27u2g7LEhXb (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 29 Jan 2016 12:37:44 -0800 (PST)
X-Barracuda-Envelope-From: darrick.wong@oracle.com
X-Barracuda-Apparent-Source-IP: 141.146.126.69
Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233])
	by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0TKbgXl011155
	(version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Fri, 29 Jan 2016 20:37:42 GMT
Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236])
	by aserv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u0TKbgmD028100
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL);
	Fri, 29 Jan 2016 20:37:42 GMT
Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11])
	by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id u0TKbfGt009941;
	Fri, 29 Jan 2016 20:37:42 GMT
Received: from localhost (/24.21.154.84)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Fri, 29 Jan 2016 12:37:41 -0800
Date: Fri, 29 Jan 2016 12:37:40 -0800
From: "Darrick J. Wong" 
To: Ross Zwisler 
Cc: xfs@oss.sgi.com
Subject: Re: xfstests xfs/293 failure with current xfstests
Message-ID: <20160129203740.GG20038@birch.djwong.org>
X-ASG-Orig-Subj: Re: xfstests xfs/293 failure with current xfstests
References: <20160129192145.GC5224@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160129192145.GC5224@linux.intel.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Source-IP: aserv0021.oracle.com [141.146.126.233]
X-Barracuda-Connect: aserp1040.oracle.com[141.146.126.69]
X-Barracuda-Start-Time: 1454099864
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, UNPARSEABLE_RELAY
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26574
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 UNPARSEABLE_RELAY      Informational: message has unparseable relay lines

On Fri, Jan 29, 2016 at 12:21:45PM -0700, Ross Zwisler wrote:
> It looks like the current version of xfsprogs (v4.3.0) has an inconsistency
> between its documentation and implementation that is making the xfstest
> xfs/293 fail:
> 
> ./check xfs/293
> FSTYP         -- xfs (debug)
> PLATFORM      -- Linux/x86_64 lorwyn 4.5.0-rc1
> MKFS_OPTIONS  -- -f -bsize=4096 /dev/pmem0p2
> MOUNT_OPTIONS -- -o dax -o context=system_u:object_r:nfs_t:s0 /dev/pmem0p2 /mnt/xfstests_scratch
> 
> xfs/293	 - output mismatch (see /root/xfstests/results//xfs/293.out.bad)
>     --- tests/xfs/293.out	2015-10-02 10:19:36.836796097 -0600
>     +++ /root/xfstests/results//xfs/293.out.bad	2016-01-29 10:41:25.530512911 -0700
>     @@ -1,2 +1,4 @@
>      QA output created by 293
>      Silence is golden
>     +help not documented in the xfs_io manpage
>     +zero not documented in the xfs_io manpage
>     ...
>     (Run 'diff -u tests/xfs/293.out /root/xfstests/results//xfs/293.out.bad'  to see the entire diff)
> Ran: xfs/293
> Failures: xfs/293
> Failed 1 of 1 tests
> 
> "help" and "zero" are indeed options in xfs_io, but aren't listed in the man
> page.

Commit 270dae6c0b41d855729fec73f94f33665bd36349 in xfsprogs for-next fixes
this.

--D

> 
> Thanks,
> - Ross
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

From david@fromorbit.com  Fri Jan 29 16:20:26 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 3D4C37CA2
	for ; Fri, 29 Jan 2016 16:20:26 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 1C339304043
	for ; Fri, 29 Jan 2016 14:20:22 -0800 (PST)
X-ASG-Debug-ID: 1454106019-04cbb026ff153d90001-NocioJ
Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id JbhU8kwM1Hi9CuGn for ; Fri, 29 Jan 2016 14:20:20 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.131
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DCBgAh5atW/zYJLHlegzqBP4hYnXYGi2GFRoQEhgkCAgEBAoE0TQEBAQEBAYELhEIBAQQ6HCMQCAMYCSUPBSUDIROIGsBxAQEIAh4YhS6EfohsBZZujUKOeo49YoQAKC6IfQEBAQ
Received: from ppp121-44-9-54.lns20.syd4.internode.on.net (HELO dastard) ([121.44.9.54])
  by ipmail07.adl2.internode.on.net with ESMTP; 30 Jan 2016 08:50:19 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aPHOg-0001AU-Hb; Sat, 30 Jan 2016 09:20:06 +1100
Date: Sat, 30 Jan 2016 09:20:06 +1100
From: Dave Chinner 
To: Dan Williams 
Cc: XFS Developers ,
	linux-fsdevel , ext4@vger.kernel.org,
	Matthew Wilcox 
Subject: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
Message-ID: <20160129222006.GJ20456@dastard>
X-ASG-Orig-Subj: Re: [PATCH 3/3] xfs: introduce per-inode DAX enablement
References: <1451886892-15548-1-git-send-email-david@fromorbit.com>
 <1451886892-15548-4-git-send-email-david@fromorbit.com>
 
 <20160121215820.GA6033@dastard>
 
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131]
X-Barracuda-Start-Time: 1454106019
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26577
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Thu, Jan 21, 2016 at 02:53:06PM -0800, Dan Williams wrote:
> On Thu, Jan 21, 2016 at 1:58 PM, Dave Chinner  wrote:
> > On Thu, Jan 21, 2016 at 08:37:11AM -0800, Dan Williams wrote:
> >> On Sun, Jan 3, 2016 at 9:54 PM, Dave Chinner  wrote:
> > i.e. I've already got a couple of fixes we need to add to this - the
> > DAX flag is only valid on CRC enabled filesystems,
> 
> I assume for torn-write protection?  The CRC limitation makes sense,
> but we theoretically could get the same effect by using a separate
> logdev that does not tear writes, right?

No, it requires the CRC enabled format because there isn't any
flag space left in the v2 inode and so we need v3 inodes to store
the flag.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Fri Jan 29 16:26:13 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 4FF0D7CA2
	for ; Fri, 29 Jan 2016 16:26:13 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id D7423AC002
	for ; Fri, 29 Jan 2016 14:26:09 -0800 (PST)
X-ASG-Debug-ID: 1454106367-04cb6c0d5f151f20001-NocioJ
Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id 5nRN2Rb7TSRQKOdZ for ; Fri, 29 Jan 2016 14:26:07 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.131
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2B/BgCO5qtW/zYJLHlegzqBP4hYnXYGi2GFRoQEhgkEAgKBNE0BAQEBAQGBC4RCAQEEOhwjEAgDDgoJJQ8FJQMhE4gawHABCx4YhS6EfohsAQSWbo1CjnqOPWKCAhmBZSguiH0BAQE
Received: from ppp121-44-9-54.lns20.syd4.internode.on.net (HELO dastard) ([121.44.9.54])
  by ipmail07.adl2.internode.on.net with ESMTP; 30 Jan 2016 08:56:06 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aPHUH-0001BT-RR; Sat, 30 Jan 2016 09:25:53 +1100
Date: Sat, 30 Jan 2016 09:25:53 +1100
From: Dave Chinner 
To: Christian Affolter 
Cc: xfs@oss.sgi.com
Subject: Re: Random write result differences between RAID device and XFS
Message-ID: <20160129222553.GK20456@dastard>
X-ASG-Orig-Subj: Re: Random write result differences between RAID device and XFS
References: <56AB44AF.8020807@purplehaze.ch>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56AB44AF.8020807@purplehaze.ch>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131]
X-Barracuda-Start-Time: 1454106367
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26577
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri, Jan 29, 2016 at 11:53:35AM +0100, Christian Affolter wrote:
> Hi everyone,
> 
> I'm trying to understand the differences of some bandwidth and IOPs test
> results I see while running a random-write full-stripe-width aligned fio
> test (using libaio with direct IO) on a hardware RAID 6 raw device
> versus on the same device with the XFS file system on top of it.
> 
> On the raw device I get:
> write: io=24828MB, bw=423132KB/s, iops=137, runt= 60085msec
> 
> With XFS on top of it:
> write: io=14658MB, bw=249407KB/s, iops=81, runt= 60182msec

Now repeat with a file that is contiguously allocated before you
start. And also perhaps with the "swalloc" mount option.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From ross.zwisler@linux.intel.com  Fri Jan 29 17:05:43 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 704497CA2
	for ; Fri, 29 Jan 2016 17:05:43 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 00A11AC003
	for ; Fri, 29 Jan 2016 15:05:36 -0800 (PST)
X-ASG-Debug-ID: 1454108735-04cbb026fc1552b0001-NocioJ
Received: from mga04.intel.com ([192.55.52.120]) by cuda.sgi.com with ESMTP id 04t7VWIkaDlf0ltx for ; Fri, 29 Jan 2016 15:05:35 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.120
X-ASG-Whitelist: Client
Received: from fmsmga003.fm.intel.com ([10.253.24.29])
  by fmsmga104.fm.intel.com with ESMTP; 29 Jan 2016 15:05:35 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,366,1449561600"; 
   d="scan'208";a="643734816"
Received: from tarkir.lm.intel.com (HELO linux.intel.com) ([10.232.112.142])
  by FMSMGA003.fm.intel.com with ESMTP; 29 Jan 2016 15:05:34 -0800
Date: Fri, 29 Jan 2016 16:05:31 -0700
From: Ross Zwisler 
To: xfs@oss.sgi.com, Dave Chinner ,
	Christoph Hellwig 
Subject: real-time device BUG
Message-ID: <20160129230531.GA29895@linux.intel.com>
X-ASG-Orig-Subj: real-time device BUG
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.120]
X-Barracuda-Start-Time: 1454108735
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

Today I was trying to get real-time devices working with XFS, and both a
manual pwrite() to a real-time device and xfs_rtcp end up with a lockdep spat
followed by a kernel BUG.  This behavior is consistent between kernel versions
v4.4 and v4.5-rc1.  I initially found this using some pmem block devices, but
it also happens with BRD.

Here is the sequence for the xfs_rtcp splat and a pair of 4GiB BRD devices on
v4.5-rc1 since that is probably easiest to reproduce:

  # fdisk -l /dev/ram*
  Disk /dev/ram0: 4 GiB, 4294967296 bytes, 8388608 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 4096 bytes
  I/O size (minimum/optimal): 4096 bytes / 4096 bytes
  Disk /dev/ram1: 4 GiB, 4294967296 bytes, 8388608 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 4096 bytes
  I/O size (minimum/optimal): 4096 bytes / 4096 bytes
  
  # mkfs.xfs -f -r rtdev=/dev/ram0 /dev/ram1
  meta-data=/dev/ram1              isize=512    agcount=4, agsize=262144 blks
           =                       sectsz=4096  attr=2, projid32bit=1
           =                       crc=1        finobt=1, sparse=0
  data     =                       bsize=4096   blocks=1048576, imaxpct=25
           =                       sunit=0      swidth=0 blks
  naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
  log      =internal log           bsize=4096   blocks=2560, version=2
           =                       sectsz=4096  sunit=1 blks, lazy-count=1
  realtime =/dev/ram0              extsz=4096   blocks=1048576, rtextents=1048576

  # mount -t xfs -o rtdev=/dev/ram0 /dev/ram1 /mnt
  
  # mount | grep '/mnt'
  /dev/ram1 on /mnt type xfs (rw,relatime,seclabel,attr2,inode64,rtdev=/dev/ram0,noquota)
  
  # xfs_rtcp ~/data /mnt/
  xfs_rtcp: /root/data is not a realtime file.
  Segmentation fault

Where ~/data is some random data of 31 MiB in size.

Here's the lockdep splat and BUG:

[  103.664626] XFS (ram1): Mounting V5 Filesystem
[  103.672209] XFS (ram1): Ending clean mount
[  135.034912] 
[  135.035304] =============================================
[  135.036268] [ INFO: possible recursive locking detected ]
[  135.037605] 4.5.0-rc1 #9 Not tainted
[  135.038423] ---------------------------------------------
[  135.039026] xfs_rtcp/1575 is trying to acquire lock:
[  135.039540]  (&xfs_nondir_ilock_class){++++..}, at: [] xfs_ilock+0x1a2/0x1f0
[  135.040551] 
[  135.040551] but task is already holding lock:
[  135.041152]  (&xfs_nondir_ilock_class){++++..}, at: [] xfs_ilock+0x1a2/0x1f0
[  135.042066] 
[  135.042066] other info that might help us debug this:
[  135.042730]  Possible unsafe locking scenario:
[  135.042730] 
[  135.043335]        CPU0
[  135.043587]        ----
[  135.043839]   lock(&xfs_nondir_ilock_class);
[  135.044289]   lock(&xfs_nondir_ilock_class);
[  135.044733] 
[  135.044733]  *** DEADLOCK ***
[  135.044733] 
[  135.045323]  May be due to missing lock nesting notation
[  135.045323] 
[  135.045989] 5 locks held by xfs_rtcp/1575:
[  135.046403]  #0:  (sb_writers#15){.+.+.+}, at: [] __sb_start_write+0xb4/0xf0
[  135.047317]  #1:  (&sb->s_type->i_mutex_key#18){+.+.+.}, at: [] xfs_file_dio_aio_write+0xa9/0x3e0
[  135.048413]  #2:  (&(&ip->i_iolock)->mr_lock){+++++.}, at: [] xfs_ilock+0x152/0x1f0
[  135.049415]  #3:  (sb_internal#2){.+.+.+}, at: [] __sb_start_write+0x78/0xf0
[  135.050338]  #4:  (&xfs_nondir_ilock_class){++++..}, at: [] xfs_ilock+0x1a2/0x1f0
[  135.051278] 
[  135.051278] stack backtrace:
[  135.051709] CPU: 0 PID: 1575 Comm: xfs_rtcp Not tainted 4.5.0-rc1 #9
[  135.052340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
[  135.053279]  0000000000000000 00000000a6b2eb4a ffff88003651b5b0 ffffffff8155f5e2
[  135.054043]  ffffffff82f22840 ffff88003651b6a0 ffffffff810f4edb ffffffff8102336e
[  135.054807]  ffff88003651b618 0000000000000046 0000000000000000 00000000810635de
[  135.055579] Call Trace:
[  135.055830]  [] dump_stack+0x44/0x62
[  135.056348]  [] __lock_acquire+0x1b3b/0x1b70
[  135.056937]  [] ? kernel_fpu_end+0xe/0x10
[  135.057497]  [] ? check_usage+0x96/0x590
[  135.058028]  [] lock_acquire+0xaf/0x130
[  135.058575]  [] ? xfs_ilock+0x1a2/0x1f0
[  135.059112]  [] ? xfs_bmap_rtalloc+0x12c/0x350
[  135.059704]  [] down_write_nested+0x4b/0xb0
[  135.060270]  [] ? xfs_ilock+0x1a2/0x1f0
[  135.060795]  [] xfs_ilock+0x1a2/0x1f0
[  135.061311]  [] xfs_bmap_rtalloc+0x12c/0x350
[  135.061886]  [] ? xfs_bmap_search_extents+0x78/0x100
[  135.062532]  [] xfs_bmap_alloc+0x2b/0x30
[  135.063070]  [] xfs_bmapi_write+0x6f2/0xf40
[  135.063644]  [] xfs_iomap_write_direct+0x2c0/0x410
[  135.064273]  [] __xfs_get_blocks+0x19e/0x9b0
[  135.064843]  [] ? trace_hardirqs_on+0xd/0x10
[  135.065422]  [] xfs_get_blocks_direct+0x17/0x20
[  135.066034]  [] do_blockdev_direct_IO+0xa91/0x2fe0
[  135.066664]  [] ? kmem_cache_free+0xa9/0x270
[  135.067240]  [] ? xfs_get_blocks+0x20/0x20
[  135.067796]  [] __blockdev_direct_IO+0x43/0x50
[  135.068399]  [] ? xfs_end_io+0xb0/0xb0
[  135.068920]  [] xfs_vm_direct_IO+0xc1/0x120
[  135.069489]  [] ? xfs_end_io+0xb0/0xb0
[  135.070008]  [] xfs_file_dio_aio_write+0x1fd/0x3e0
[  135.070636]  [] xfs_file_write_iter+0x99/0x150
[  135.071270]  [] __vfs_write+0xcc/0x110
[  135.071787]  [] vfs_write+0xac/0x1a0
[  135.072296]  [] ? __fget_light+0x66/0x90
[  135.072835]  [] SyS_write+0x58/0xd0
[  135.073338]  [] entry_SYSCALL_64_fastpath+0x12/0x76
[  135.077170] XFS: Assertion failed: xfs_isilocked(ip, XFS_ILOCK_SHARED|XFS_ILOCK_EXCL), file: fs/xfs/libxfs/xfs_bmap.c, line: 4039
[  135.078379] ------------[ cut here ]------------
[  135.078848] kernel BUG at fs/xfs/xfs_message.c:113!
[  135.079334] invalid opcode: 0000 [#1] SMP 
[  135.079762] Modules linked in: nd_pmem nd_btt nd_e820 libnvdimm
[  135.080405] CPU: 0 PID: 1575 Comm: xfs_rtcp Not tainted 4.5.0-rc1 #9
[  135.081024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
[  135.081963] task: ffff8800baa8e000 ti: ffff880036518000 task.ti: ffff880036518000
[  135.082716] RIP: 0010:[]  [] assfail+0x20/0x30
[  135.083479] RSP: 0018:ffff88003651b488  EFLAGS: 00010246
[  135.083997] RAX: 0000000000000000 RBX: ffff88003651b56c RCX: 0000000000000000
[  135.084698] RDX: 00000000ffffffc0 RSI: 000000000000000a RDI: ffffffff81ee0d88
[  135.085399] RBP: ffff88003651b488 R08: 0000000000000000 R09: 0000000000000000
[  135.086092] R10: 000000000000000a R11: f000000000000000 R12: 0000000000000000
[  135.086788] R13: ffff8801f85a8f80 R14: ffff8800ac988000 R15: 0000000000000000
[  135.087486] FS:  00007f287a6ce700(0000) GS:ffff88041a000000(0000) knlGS:0000000000000000
[  135.088282] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  135.088834] CR2: 00007f59a07f9000 CR3: 00000000aca05000 CR4: 00000000000006f0
[  135.089533] Stack:
[  135.089736]  ffff88003651b548 ffffffff814376db ffffffff82249420 0000000000000000
[  135.090510]  000000000000a52b ffff88003651b588 0000000000000001 0000000000000000
[  135.091271]  ffff88003651b500 00000000810ed803 ffffffff810ed745 0000000000000000
[  135.092025] Call Trace:
[  135.092277]  [] xfs_bmapi_read+0xcb/0x3b0
[  135.092820]  [] ? cpuacct_charge+0x5/0x190
[  135.093377]  [] ? update_curr+0x66/0x180
[  135.093907]  [] xfs_rtbuf_get+0x5a/0x130
[  135.094448]  [] ? finish_task_switch+0x6b/0x250
[  135.095036]  [] xfs_rtmodify_summary_int+0x93/0x120
[  135.095665]  [] ? finish_task_switch+0x6b/0x250
[  135.096261]  [] xfs_rtallocate_extent_size+0xf2/0x3c0
[  135.096904]  [] ? kmem_zone_alloc+0x81/0x120
[  135.097472]  [] ? kmem_cache_alloc+0xd0/0x1e0
[  135.098040]  [] ? kmem_zone_alloc+0x81/0x120
[  135.098622]  [] xfs_rtallocate_extent+0x1f8/0x260
[  135.099253]  [] xfs_bmap_rtalloc+0x1ba/0x350
[  135.099820]  [] xfs_bmap_alloc+0x2b/0x30
[  135.100361]  [] xfs_bmapi_write+0x6f2/0xf40
[  135.100921]  [] xfs_iomap_write_direct+0x2c0/0x410
[  135.101545]  [] __xfs_get_blocks+0x19e/0x9b0
[  135.102116]  [] ? trace_hardirqs_on+0xd/0x10
[  135.102682]  [] xfs_get_blocks_direct+0x17/0x20
[  135.103279]  [] do_blockdev_direct_IO+0xa91/0x2fe0
[  135.103892]  [] ? kmem_cache_free+0xa9/0x270
[  135.104466]  [] ? xfs_get_blocks+0x20/0x20
[  135.105011]  [] __blockdev_direct_IO+0x43/0x50
[  135.105601]  [] ? xfs_end_io+0xb0/0xb0
[  135.106120]  [] xfs_vm_direct_IO+0xc1/0x120
[  135.106673]  [] ? xfs_end_io+0xb0/0xb0
[  135.107211]  [] xfs_file_dio_aio_write+0x1fd/0x3e0
[  135.107828]  [] xfs_file_write_iter+0x99/0x150
[  135.108425]  [] __vfs_write+0xcc/0x110
[  135.108942]  [] vfs_write+0xac/0x1a0
[  135.109451]  [] ? __fget_light+0x66/0x90
[  135.109989]  [] SyS_write+0x58/0xd0
[  135.110488]  [] entry_SYSCALL_64_fastpath+0x12/0x76
[  135.111118] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 f1 41 89 d0 48 c7 c6 c0 4d f2 81 48 89 fa 31 ff 48 89 e5 e8 b0 f8 ff ff <0f> 0b 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 
[  135.113646] RIP  [] assfail+0x20/0x30
[  135.114177]  RSP 
[  135.114541] ---[ end trace 8e97f4b936072524 ]---

From ross.zwisler@linux.intel.com  Fri Jan 29 17:40:42 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 75A787CA2
	for ; Fri, 29 Jan 2016 17:40:42 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 01364AC002
	for ; Fri, 29 Jan 2016 15:40:41 -0800 (PST)
X-ASG-Debug-ID: 1454110840-04cbb026fe156510001-NocioJ
Received: from mga01.intel.com ([192.55.52.88]) by cuda.sgi.com with ESMTP id VFgTlbPEFdZ8sOV1 for ; Fri, 29 Jan 2016 15:40:41 -0800 (PST)
X-Barracuda-Envelope-From: ross.zwisler@linux.intel.com
X-Barracuda-Apparent-Source-IP: 192.55.52.88
X-ASG-Whitelist: Client
Received: from orsmga003.jf.intel.com ([10.7.209.27])
  by fmsmga101.fm.intel.com with ESMTP; 29 Jan 2016 15:40:40 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.22,366,1449561600"; 
   d="scan'208";a="736832719"
Received: from tarkir.lm.intel.com (HELO linux.intel.com) ([10.232.112.142])
  by orsmga003.jf.intel.com with ESMTP; 29 Jan 2016 15:40:39 -0800
Date: Fri, 29 Jan 2016 16:40:36 -0700
From: Ross Zwisler 
To: Ross Zwisler 
Cc: xfs@oss.sgi.com, Dave Chinner ,
	Christoph Hellwig 
Subject: Re: real-time device BUG
Message-ID: <20160129234036.GA24929@linux.intel.com>
X-ASG-Orig-Subj: Re: real-time device BUG
References: <20160129230531.GA29895@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160129230531.GA29895@linux.intel.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-Barracuda-Connect: UNKNOWN[192.55.52.88]
X-Barracuda-Start-Time: 1454110840
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

On Fri, Jan 29, 2016 at 04:05:31PM -0700, Ross Zwisler wrote:
> Today I was trying to get real-time devices working with XFS, and both a
> manual pwrite() to a real-time device and xfs_rtcp end up with a lockdep spat
> followed by a kernel BUG.  This behavior is consistent between kernel versions
> v4.4 and v4.5-rc1.  I initially found this using some pmem block devices, but
> it also happens with BRD.

This also happens with an mmap on the real-time file followed by a write.
Essentially I think that any path that tries to allocate a block for the real
time file his this same BUG.

From sandeen@sandeen.net  Fri Jan 29 21:11:17 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 224C87CA2
	for ; Fri, 29 Jan 2016 21:11:17 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 103DB8F8040
	for ; Fri, 29 Jan 2016 19:11:13 -0800 (PST)
X-ASG-Debug-ID: 1454123466-04cbb026fc15b800001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id FttXGOCa93Bmh45R for ; Fri, 29 Jan 2016 19:11:06 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 0F2D263C6064
	for ; Fri, 29 Jan 2016 21:11:06 -0600 (CST)
Subject: Re: real-time device BUG
To: xfs@oss.sgi.com
X-ASG-Orig-Subj: Re: real-time device BUG
References: <20160129230531.GA29895@linux.intel.com>
 <20160129234036.GA24929@linux.intel.com>
From: Eric Sandeen 
X-Enigmail-Draft-Status: N1110
Message-ID: <56AC29CA.30504@sandeen.net>
Date: Fri, 29 Jan 2016 21:11:06 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <20160129234036.GA24929@linux.intel.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1454123466
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26584
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------



On 1/29/16 5:40 PM, Ross Zwisler wrote:
> On Fri, Jan 29, 2016 at 04:05:31PM -0700, Ross Zwisler wrote:
>> Today I was trying to get real-time devices working with XFS, and both a
>> manual pwrite() to a real-time device and xfs_rtcp end up with a lockdep spat
>> followed by a kernel BUG.  This behavior is consistent between kernel versions
>> v4.4 and v4.5-rc1.  I initially found this using some pmem block devices, but
>> it also happens with BRD.
> 
> This also happens with an mmap on the real-time file followed by a write.
> Essentially I think that any path that tries to allocate a block for the real
> time file his this same BUG.

Looks like an oversight from

commit eef334e5776c8ef547ada4cec17549929fe590b4
Author: Christoph Hellwig 
Date:   Fri Dec 6 12:30:17 2013 -0800

    xfs: assert that we hold the ilock for extent map access
    
    Make sure that xfs_bmapi_read has the ilock held in some way, and that
    xfs_bmapi_write, xfs_bmapi_delay, xfs_bunmapi and xfs_iread_extents are
    called with the ilock held exclusively.
    
    Signed-off-by: Christoph Hellwig 
    Reviewed-by: Dave Chinner 
    Signed-off-by: Ben Myers 

I think all we have to do is lock it for read here, I'll send a patch and
Dave or Christoph can smack me down if I'm wrong ;)

From sandeen@redhat.com  Fri Jan 29 21:26:33 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id A26267CA2
	for ; Fri, 29 Jan 2016 21:26:33 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 8EC0A304039
	for ; Fri, 29 Jan 2016 19:26:30 -0800 (PST)
X-ASG-Debug-ID: 1454124389-04cb6c0d5d15b5f0001-NocioJ
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id JhFSVGnlpHd8nWiy (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 29 Jan 2016 19:26:29 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@redhat.com
X-Barracuda-Apparent-Source-IP: 209.132.183.28
X-ASG-Whitelist: Client
Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (Postfix) with ESMTPS id EBD07C0D78EA
	for ; Sat, 30 Jan 2016 03:26:28 +0000 (UTC)
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0U3QRlP017241
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for ; Fri, 29 Jan 2016 22:26:28 -0500
To: xfs@oss.sgi.com
From: Eric Sandeen 
Subject: [PATCH] xfs: lock bitmap/summary inodes in xfs_rtbuf_get()
X-Enigmail-Draft-Status: N1110
X-ASG-Orig-Subj: [PATCH] xfs: lock bitmap/summary inodes in xfs_rtbuf_get()
Message-ID: <56AC2D64.2080907@redhat.com>
Date: Fri, 29 Jan 2016 21:26:28 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.5.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Barracuda-Connect: mx1.redhat.com[209.132.183.28]
X-Barracuda-Start-Time: 1454124389
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1

Commit eef334e added an ASSERT that the inode was locked in
some way in xfs_bmapi_read(), but on realtime paths through
xfs_rtbuf_get() this isn't the case; fix that.

Reported-by: Ross Zwisler 
Signed-off-by: Eric Sandeen 
---

I think we need the data_map_shared gyrations here, but not certain...

diff --git a/fs/xfs/libxfs/xfs_rtbitmap.c b/fs/xfs/libxfs/xfs_rtbitmap.c
index 9b59ffa..e6da0b2 100644
--- a/fs/xfs/libxfs/xfs_rtbitmap.c
+++ b/fs/xfs/libxfs/xfs_rtbitmap.c
@@ -57,11 +57,14 @@ xfs_rtbuf_get(
 	xfs_inode_t	*ip;		/* bitmap or summary inode */
 	xfs_bmbt_irec_t	map;
 	int		nmap = 1;
+	int		lock_mode;
 	int		error;		/* error value */
 
 	ip = issum ? mp->m_rsumip : mp->m_rbmip;
 
+	lock_mode = xfs_ilock_data_map_shared(ip);
 	error = xfs_bmapi_read(ip, block, 1, &map, &nmap, XFS_DATA_FORK);
+	xfs_iunlock(ip, lock_mode);
 	if (error)
 		return error;
 

From c.affolter@purplehaze.ch  Sat Jan 30 04:44:14 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id 47AC77CA2
	for ; Sat, 30 Jan 2016 04:44:14 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay1.corp.sgi.com (Postfix) with ESMTP id 29A4E8F8040
	for ; Sat, 30 Jan 2016 02:44:10 -0800 (PST)
X-ASG-Debug-ID: 1454150645-04bdf04d6d166240001-NocioJ
Received: from mail.stepping-stone.ch (mail.stepping-stone.ch [194.176.109.206]) by cuda.sgi.com with ESMTP id 0wGtz40Ab1DgC9rj (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 30 Jan 2016 02:44:07 -0800 (PST)
X-Barracuda-Envelope-From: c.affolter@purplehaze.ch
X-Barracuda-Apparent-Source-IP: 194.176.109.206
Received: from localhost (janitor-01.int.stepping-stone.ch [10.17.98.12])
	by mta-01.int.stepping-stone.ch (Postfix) with ESMTP id 704C1A9CB5;
	Sat, 30 Jan 2016 11:44:05 +0100 (CET)
X-Virus-Scanned: amavisd-new at stepping-stone.ch
Received: from mta-01.int.stepping-stone.ch ([10.17.98.30])
	by localhost (janitor-01.int.stepping-stone.ch [10.17.98.12]) (amavisd-new, port 10026)
	with LMTP id 36BAiTj7eIlH; Sat, 30 Jan 2016 11:43:57 +0100 (CET)
Received: from [192.168.1.242] (84-74-151-79.dclient.hispeed.ch [84.74.151.79])
	(using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits))
	(Client did not present a certificate)
	by smtp.stepping-stone.ch (Postfix) with ESMTPSA id 60AA8A9CA0;
	Sat, 30 Jan 2016 11:43:57 +0100 (CET)
Subject: Re: Random write result differences between RAID device and XFS
To: Dave Chinner 
X-ASG-Orig-Subj: Re: Random write result differences between RAID device and XFS
References: <56AB44AF.8020807@purplehaze.ch> <20160129222553.GK20456@dastard>
Cc: xfs@oss.sgi.com
From: Christian Affolter 
X-Enigmail-Draft-Status: N1110
Message-ID: <56AC93EC.80202@purplehaze.ch>
Date: Sat, 30 Jan 2016 11:43:56 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.0
MIME-Version: 1.0
In-Reply-To: <20160129222553.GK20456@dastard>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: mail.stepping-stone.ch[194.176.109.206]
X-Barracuda-Start-Time: 1454150646
X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26592
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

Hi Dave,

On 29.01.2016 23:25, Dave Chinner wrote:
> On Fri, Jan 29, 2016 at 11:53:35AM +0100, Christian Affolter wrote:
>> Hi everyone,
>>
>> I'm trying to understand the differences of some bandwidth and IOPs test
>> results I see while running a random-write full-stripe-width aligned fio
>> test (using libaio with direct IO) on a hardware RAID 6 raw device
>> versus on the same device with the XFS file system on top of it.
>>
>> On the raw device I get:
>> write: io=24828MB, bw=423132KB/s, iops=137, runt= 60085msec
>>
>> With XFS on top of it:
>> write: io=14658MB, bw=249407KB/s, iops=81, runt= 60182msec
> 
> Now repeat with a file that is contiguously allocated before you
> start. And also perhaps with the "swalloc" mount option.

Wow, thanks! After specifying --fallocate=none (instead of the default
fallocate=posix), bandwidth and iops increases and are even higher than
on the raw device:

write: io=30720MB, bw=599232KB/s, iops=195, runt= 52496msec

I'm eager to learn what's going on behind the scenes, can you give a
short explanation?

Btw. mounting the volume with "swalloc" didn't make any change.


Thanks a lot!
Chris

From info@uaezain.com  Sat Jan 30 09:34:54 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: ***
X-Spam-Status: No, score=3.2 required=5.0 tests=FREEMAIL_REPLYTO,HTML_MESSAGE,
	HTML_MIME_NO_HTML_TAG,MIME_HTML_ONLY,RCVD_NUMERIC_HELO,T_DKIM_INVALID
	autolearn=no version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id C1C6B7CA2
	for ; Sat, 30 Jan 2016 09:34:53 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 37821AC004
	for ; Sat, 30 Jan 2016 07:34:53 -0800 (PST)
X-ASG-Debug-ID: 1454168091-04cbb026fd179600001-NocioJ
Received: from uae.uaezain.com (uae.uaezain.com [216.172.178.232]) by cuda.sgi.com with ESMTP id HCGvKWNDhgnS9VIg (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 30 Jan 2016 07:34:51 -0800 (PST)
X-Barracuda-Envelope-From: info@uaezain.com
X-Barracuda-Apparent-Source-IP: 216.172.178.232
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=uaezain.com
	; s=default; h=Message-ID:Date:Content-Transfer-Encoding:Content-Type:Subject
	:To:Reply-To:From:MIME-Version;
	bh=Xphi0I35GYUHQjIs36GeHvdyFg5M1nvcRvvfeEQqiGc=; b=Q5MYsFYnOZs3tZq9vr8m9iDIhn
	RZWnIayHHZifLxZjVXYnF89hObgUa80HMvaCbzB6/D1TzjsakkcYI0xqI3XbYIxcun4yb9J4A5CP/
	4uB2M9/AN4OSq4Ls54fj57M4TTLs/hNHzxPFYJDFjOZI068Hmv6l9R++L6sPWudPHsfM=;
Received: from [92.99.155.5] (port=56333 helo=92.99.155.5)
	by uae.uaezain.com with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256)
	(Exim 4.86)
	(envelope-from )
	id 1aPXY2-00012j-Ak
	for xfs@oss.sgi.com; Sat, 30 Jan 2016 09:34:51 -0600
MIME-Version: 1.0
From: "Risalat Consultants International" 
Reply-To: risalatconsultantsint@outlook.com
To: xfs@oss.sgi.com
Subject: Disaster Management Workshop in Kuala Lumpur, Malaysia
Content-Type: text/html; charset="windows-1252"
X-ASG-Orig-Subj: Disaster Management Workshop in Kuala Lumpur, Malaysia
Content-Transfer-Encoding: quoted-printable
X-Mailer: Smart_Send_3_1_6
Date: Sat, 30 Jan 2016 19:34:46 +0400
Message-ID: <141083757180641777912335@Risalat>
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - uae.uaezain.com
X-AntiAbuse: Original Domain - oss.sgi.com
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - uaezain.com
X-Get-Message-Sender-Via: uae.uaezain.com: authenticated_id: info@uaezain.com
X-Authenticated-Sender: uae.uaezain.com: info@uaezain.com
X-Barracuda-Connect: uae.uaezain.com[216.172.178.232]
X-Barracuda-Start-Time: 1454168091
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 2.31
X-Barracuda-Spam-Status: No, SCORE=2.31 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=DKIM_SIGNED, DKIM_VERIFIED, HTML_MESSAGE, HTML_MIME_NO_HTML_TAG, MIME_HTML_ONLY, RCVD_NUMERIC_HELO, RCVD_NUMERIC_HELO_2
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26597
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 RCVD_NUMERIC_HELO      Received: contains an IP address used for HELO
	-0.00 DKIM_VERIFIED          Domain Keys Identified Mail: signature passes
	                           verification
	0.00 DKIM_SIGNED            Domain Keys Identified Mail: message has a signature
	0.00 MIME_HTML_ONLY         BODY: Message only has text/html MIME parts
	0.00 HTML_MESSAGE           BODY: HTML included in message
	1.05 HTML_MIME_NO_HTML_TAG  HTML-only message, but there is no HTML tag
	1.25 RCVD_NUMERIC_HELO_2    Received: contains an IP address used for HELO






Program Introduction<=3Fxml:namespace prefix =3D "o" ns =3D "urn:schemas-m=
icrosoft-com:office:office" />


Disaster Management is the organization and management of resources and re=
sponsibilities for dealing with all humanitarian aspects of emergencies, in=
 particular preparedness, response and recovery, in order to lessen the imp=
act of disasters.


 


The program provides comprehensive disaster management knowledge and skill=
s to enhance the capabilities of executive managers who have key disaster m=
anagement responsibilities. It is designed to enable professionals working =
in disaster management, development and donor agencies to effectively integ=
rate disaster management into their development programs and policies. Part=
icipants will be encouraged to develop key skills and adopt proactive attit=
udes through participation in interactive lectures and reflection on arrang=
e of key issues raised during discussions and practical activities.


 


Program Contents


Upon completion of this program, the participants will be able =
to:


 


=B7         Explain the characteristics and impacts of disasters and complex emerge=
ncies


=B7         Analyze how vulnerabilities and capacities affect the resilience of co=
mmunities to disasters


=B7         Describe the different phases of disaster management and the key activ=
ities for each phase


=B7         Explain the links between disaster management and disaster risk reduct=
ion


=B7         Identify the international legal instruments, humanitarian principles,=
 codes and standards which guide disaster management.


=B7         Describe and apply disaster management tools


=B7         Identify the roles and responsibilities of different actors in disaste=
r management and analyze how they can work together in a coordinated manner=



=B7          


Program Methodology


The training methodology used is designed to encourage maximum participati=
on by all delegates. The presenter will suggest ideas and theories to the d=
elegates and then encourage them to test out the ideas by the use of discus=
sion, small group work, exercises and feedback. Each day of the seminar wil=
l end by delegates completing their own record of what has been learned on =
the day and considering how the ideas might be transferred back to the work=
place.


 


Registration Information


Program Dates: 28 =96 30 April, 2016


Registration Closes on: 29 February, 2016


 


Program Fee: $3450


Fee Covers: Participant visa, round trip tic=
kets (Fee may increase depending on the country you will be travelling), ac=
commodation for 5 nights, Training Program, Program Venue, Participant Asse=
ssments, Lunch & Refreshments, Workshop Kit, Program Materials, Certifi=
cate of Completion.


 


Online Payment via PayPal




&=
nbsp;


 


Registration As=
sistance: trainings@risalatconsultants.com =
or risalatconsulta=
nts@gmail.com


Skype: risalatconsultantsint<=
/B>


Don=92t forget to follow Risalat on&nbs=
p;LinkedIn, Twitter=
 &  Facebook!


From david@fromorbit.com  Sat Jan 30 15:07:11 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 09E847CA2
	for ; Sat, 30 Jan 2016 15:07:11 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 8200CAC004
	for ; Sat, 30 Jan 2016 13:07:07 -0800 (PST)
X-ASG-Debug-ID: 1454188023-04cb6c0d5f17fed0001-NocioJ
Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id p3g0UNyrfFQSAcws for ; Sat, 30 Jan 2016 13:07:04 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.131
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2DgBgCSJa1W/zYJLHlVCIM6gT+CZoVynXUGi2KFRYQEhgkCAgEBAoEnTQEBAQEBAYELhEIBAQQnExwjEAgDGAklDwUlAyETiBq+AAELAR0YhTCEfoQOhF4Flm+NQo55RI16YoICGYFlKC4BiHwBAQE
Received: from ppp121-44-9-54.lns20.syd4.internode.on.net (HELO dastard) ([121.44.9.54])
  by ipmail07.adl2.internode.on.net with ESMTP; 31 Jan 2016 07:37:03 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aPcjK-00035J-9D; Sun, 31 Jan 2016 08:06:50 +1100
Date: Sun, 31 Jan 2016 08:06:50 +1100
From: Dave Chinner 
To: Eric Sandeen 
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: lock bitmap/summary inodes in xfs_rtbuf_get()
Message-ID: <20160130210650.GL20456@dastard>
X-ASG-Orig-Subj: Re: [PATCH] xfs: lock bitmap/summary inodes in xfs_rtbuf_get()
References: <56AC2D64.2080907@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56AC2D64.2080907@redhat.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131]
X-Barracuda-Start-Time: 1454188023
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26602
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri, Jan 29, 2016 at 09:26:28PM -0600, Eric Sandeen wrote:
> Commit eef334e added an ASSERT that the inode was locked in
> some way in xfs_bmapi_read(), but on realtime paths through
> xfs_rtbuf_get() this isn't the case; fix that.
> 
> Reported-by: Ross Zwisler 
> Signed-off-by: Eric Sandeen 
> ---
> 
> I think we need the data_map_shared gyrations here, but not certain...
> 
> diff --git a/fs/xfs/libxfs/xfs_rtbitmap.c b/fs/xfs/libxfs/xfs_rtbitmap.c
> index 9b59ffa..e6da0b2 100644
> --- a/fs/xfs/libxfs/xfs_rtbitmap.c
> +++ b/fs/xfs/libxfs/xfs_rtbitmap.c
> @@ -57,11 +57,14 @@ xfs_rtbuf_get(
>  	xfs_inode_t	*ip;		/* bitmap or summary inode */
>  	xfs_bmbt_irec_t	map;
>  	int		nmap = 1;
> +	int		lock_mode;
>  	int		error;		/* error value */
>  
>  	ip = issum ? mp->m_rsumip : mp->m_rbmip;
>  
> +	lock_mode = xfs_ilock_data_map_shared(ip);
>  	error = xfs_bmapi_read(ip, block, 1, &map, &nmap, XFS_DATA_FORK);
> +	xfs_iunlock(ip, lock_mode);

I've looked into this recently and didn't think up a simple answer
to the problem. I didn't spend much time on it because it's nowhere
near the top of my priority list because it only affects debug
kernels as the summary inode is effectively protected by the bitmap
inode exclusion during allocation.

That said, the above change is not safe because xfs_rtbuf_get() can
be called with the bitmap inode lock already held. e.g:

  xfs_bmap_rtalloc
    xfs_ilock(mp->m_rbmip, XFS_ILOCK_EXCL);   << locks bitmap
    xfs_rtallocate_extent
      xfs_rtallocate_extent_exact
	xfs_rtcheck_range
	  xfs_rtbuf_get(issum = 0)
	    xfs_ilock_data_map_shared(mp->m_rbmip)
	    

The issue here is that the summary inode is not locked early on in
the transaction, so modifications are done with it unlocked.

  xfs_bmap_rtalloc
    xfs_ilock(mp->m_rbmip, XFS_ILOCK_EXCL);   << locks bitmap
    xfs_rtallocate_extent
      xfs_rtallocate_extent_size
        xfs_rtget_summary
          xfs_rtmodify_summary_int
	    xfs_rtbuf_get
	      xfs_bmapi_read(summary inode)    << unlocked summary

The only path through which the summary inode is locked for
modification is the growfs path (xfs_rtcopy_summary()), so all the
other paths that modify/access the summary inode also need to be
locked at a higher level before calling into the summary functions.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From sandeen@sandeen.net  Sat Jan 30 15:53:56 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay1.corp.sgi.com [137.38.102.111])
	by oss.sgi.com (Postfix) with ESMTP id D591B7CA4
	for ; Sat, 30 Jan 2016 15:53:56 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay1.corp.sgi.com (Postfix) with ESMTP id C35BE8F8035
	for ; Sat, 30 Jan 2016 13:53:53 -0800 (PST)
X-ASG-Debug-ID: 1454190829-04cb6c0d5e181410001-NocioJ
Received: from sandeen.net (sandeen.net [63.231.237.45]) by cuda.sgi.com with ESMTP id g0ksw6PlGUbLFQHH for ; Sat, 30 Jan 2016 13:53:50 -0800 (PST)
X-Barracuda-Envelope-From: sandeen@sandeen.net
X-Barracuda-Apparent-Source-IP: 63.231.237.45
Received: from [22.87.204.88] (unknown [172.56.13.58])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by sandeen.net (Postfix) with ESMTPSA id 60B6E63C6064;
	Sat, 30 Jan 2016 15:53:49 -0600 (CST)
Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (1.0)
Subject: Re: [PATCH] xfs: lock bitmap/summary inodes in xfs_rtbuf_get()
From: Eric Sandeen 
X-ASG-Orig-Subj: Re: [PATCH] xfs: lock bitmap/summary inodes in xfs_rtbuf_get()
X-Mailer: iPhone Mail (13D15)
In-Reply-To: <20160130210650.GL20456@dastard>
Date: Sat, 30 Jan 2016 15:53:48 -0600
Cc: Eric Sandeen , xfs@oss.sgi.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <015F4CF0-425A-4EB0-ACE2-4EBBE0B78173@sandeen.net>
References: <56AC2D64.2080907@redhat.com> <20160130210650.GL20456@dastard>
To: Dave Chinner 
X-Barracuda-Connect: sandeen.net[63.231.237.45]
X-Barracuda-Start-Time: 1454190830
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26603
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

OK, thanks Dave. It seemed like it was probably too simple=E2=80=A6

> On Jan 30, 2016, at 3:06 PM, Dave Chinner  wrote:
>=20
>> On Fri, Jan 29, 2016 at 09:26:28PM -0600, Eric Sandeen wrote:
>> Commit eef334e added an ASSERT that the inode was locked in
>> some way in xfs_bmapi_read(), but on realtime paths through
>> xfs_rtbuf_get() this isn't the case; fix that.
>>=20
>> Reported-by: Ross Zwisler 
>> Signed-off-by: Eric Sandeen 
>> ---
>>=20
>> I think we need the data_map_shared gyrations here, but not certain...
>>=20
>> diff --git a/fs/xfs/libxfs/xfs_rtbitmap.c b/fs/xfs/libxfs/xfs_rtbitmap.c
>> index 9b59ffa..e6da0b2 100644
>> --- a/fs/xfs/libxfs/xfs_rtbitmap.c
>> +++ b/fs/xfs/libxfs/xfs_rtbitmap.c
>> @@ -57,11 +57,14 @@ xfs_rtbuf_get(
>>    xfs_inode_t    *ip;        /* bitmap or summary inode */
>>    xfs_bmbt_irec_t    map;
>>    int        nmap =3D 1;
>> +    int        lock_mode;
>>    int        error;        /* error value */
>>=20
>>    ip =3D issum ? mp->m_rsumip : mp->m_rbmip;
>>=20
>> +    lock_mode =3D xfs_ilock_data_map_shared(ip);
>>    error =3D xfs_bmapi_read(ip, block, 1, &map, &nmap, XFS_DATA_FORK);
>> +    xfs_iunlock(ip, lock_mode);
>=20
> I've looked into this recently and didn't think up a simple answer
> to the problem. I didn't spend much time on it because it's nowhere
> near the top of my priority list because it only affects debug
> kernels as the summary inode is effectively protected by the bitmap
> inode exclusion during allocation.
>=20
> That said, the above change is not safe because xfs_rtbuf_get() can
> be called with the bitmap inode lock already held. e.g:
>=20
>  xfs_bmap_rtalloc
>    xfs_ilock(mp->m_rbmip, XFS_ILOCK_EXCL);   << locks bitmap
>    xfs_rtallocate_extent
>      xfs_rtallocate_extent_exact
>    xfs_rtcheck_range
>      xfs_rtbuf_get(issum =3D 0)
>        xfs_ilock_data_map_shared(mp->m_rbmip)
>        
>=20
> The issue here is that the summary inode is not locked early on in
> the transaction, so modifications are done with it unlocked.
>=20
>  xfs_bmap_rtalloc
>    xfs_ilock(mp->m_rbmip, XFS_ILOCK_EXCL);   << locks bitmap
>    xfs_rtallocate_extent
>      xfs_rtallocate_extent_size
>        xfs_rtget_summary
>          xfs_rtmodify_summary_int
>        xfs_rtbuf_get
>          xfs_bmapi_read(summary inode)    << unlocked summary
>=20
> The only path through which the summary inode is locked for
> modification is the growfs path (xfs_rtcopy_summary()), so all the
> other paths that modify/access the summary inode also need to be
> locked at a higher level before calling into the summary functions.
>=20
> Cheers,
>=20
> Dave.
> --=20
> Dave Chinner
> david@fromorbit.com
>=20
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>=20

From david@fromorbit.com  Sun Jan 31 23:05:39 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id AF1217CA2
	for ; Sun, 31 Jan 2016 23:05:39 -0600 (CST)
Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 84CE6304059
	for ; Sun, 31 Jan 2016 21:05:39 -0800 (PST)
X-ASG-Debug-ID: 1454303137-04cb6c0d5f1c4950001-NocioJ
Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id o7dsbXx7Ym9gp4IU for ; Sun, 31 Jan 2016 21:05:37 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.131
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2BECAAu565W/xATLHldgzqBP4hYngMGi2I4hQ2EBIYJBIEwTQEBAQEBAYELhR47JDQFJQM0iBqeGp49CBiFMIl8g24Flm+NQoFkhEKIUQKFbohQYoE2AQsBgj0oLoh9AQEB
Received: from ppp121-44-19-16.lns20.syd4.internode.on.net (HELO dastard) ([121.44.19.16])
  by ipmail07.adl2.internode.on.net with ESMTP; 01 Feb 2016 15:35:35 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aQ6fy-0005Q3-KH
	for xfs@oss.sgi.com; Mon, 01 Feb 2016 16:05:22 +1100
Date: Mon, 1 Feb 2016 16:05:22 +1100
From: Dave Chinner 
To: xfs@oss.sgi.com
Subject: [ANNOUNCE] xfs: master, for-next branch updated to v4.5-rc2
Message-ID: <20160201050522.GR6033@dastard>
X-ASG-Orig-Subj: [ANNOUNCE] xfs: master, for-next branch updated to v4.5-rc2
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="+g7M9IMkV8truYOl"
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131]
X-Barracuda-Start-Time: 1454303137
X-Barracuda-URL: https://192.48.176.15:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26637
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------


--+g7M9IMkV8truYOl
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi Folks,

I've just updated the master and for-next branches in the XFS tree
to v4.5-rc2 ready for the next cycle. Please update you local dev
trees for new patch sumissions.

-Dave.
--=20
Dave Chinner
david@fromorbit.com

--+g7M9IMkV8truYOl
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBAgAGBQJWrueSAAoJEK3oKUf0dfodjV4P/jFMdjVMCE21YNiEgXSSwmiD
0kLu7X1THVueB4cmH9pGmhdLRgNmQhIb5KnNJ0tWXIpKpPKIwuxKEHrAwxM5ZEQI
qYPdanXObSjPwVRX3tCarc3NdbhjdPCi+SaXMWd+2CntMGU7rNO12BV5rXKpZdlV
0fI1zGWZgUxNWM+PGzi/8fg9LDZhdgdyMRhwIorwlou1zQ5wUNkQsb9tdi0usSvA
u/mrEKuRqpoecaG71bCWd0BURfa4Djo2/JKNRYtgJket7o+zNzI6OQnX7aB+S1WF
XyFJq3FojSUdLct5tUh++x42kaSD1hBbWBBIjjXPEUhhZcE85gCXpPnPy2nKYQxk
hgyIK8mN5tAW4pmrJcGXqF/6Df50ySD9f0Xnk3Vz1N6X9Dg4L9qxSID1/BXMZN8N
GYUsJ+B70pnSRTqhQ/W5aETRlgbts/Srgl306XOgDMmZ+ISk9qGe3mZfVC48TF08
n312VTNYkr61yMO+QuqKL2ujyotXXrCCi3L/RVwtHdBmjNJQ3TQS6ev7I4TQToK0
JxCKuGPS5eRLTxxevMX2/TQiJ4Y9dpoB9ffS8veYQAX+tSmrZ7j8uJJwOHdGRljC
8OL4ecpJoK+GNq+MXjmuMKTT3MmLduU6fv4V0L7Luc4/cEbiXvwTXpkhTyhYmx6V
xB+IR6VETZjHU2tKrHcQ
=sV6K
-----END PGP SIGNATURE-----

--+g7M9IMkV8truYOl--

From david@fromorbit.com  Sun Jan 31 23:06:43 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id B8A9D7CA2
	for ; Sun, 31 Jan 2016 23:06:43 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id 8B308304059
	for ; Sun, 31 Jan 2016 21:06:43 -0800 (PST)
X-ASG-Debug-ID: 1454303200-04bdf04d6c1d41c0001-NocioJ
Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id pbDdkGRZ7IOzNuQs for ; Sun, 31 Jan 2016 21:06:41 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.131
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2CqBgAu565W/xATLHldgzqBP4hYngMGi2KFRYQEhgkCAgEBAoEsTQEBAQEBAYELhEIBAQQ6HCMQCAMYCSUPBSUDIROIGrw1AQEIAgEdGIUwhH6IbAWWb41CjnmOPmKCAhmBZSguiH0BAQE
Received: from ppp121-44-19-16.lns20.syd4.internode.on.net (HELO dastard) ([121.44.19.16])
  by ipmail07.adl2.internode.on.net with ESMTP; 01 Feb 2016 15:36:40 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aQ6h0-0005QE-WF; Mon, 01 Feb 2016 16:06:27 +1100
Date: Mon, 1 Feb 2016 16:06:26 +1100
From: Dave Chinner 
To: "Darrick J. Wong" 
Cc: Josef 'Jeff' Sipek , xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
Message-ID: <20160201050626.GS6033@dastard>
X-ASG-Orig-Subj: Re: [PATCH] xfs: check sizes of XFS on-disk structures at compile
 time
References: <20160111232657.GA7831@birch.djwong.org>
 <20160115205801.GB5757@birch.djwong.org>
 <20160115225219.GP6033@dastard>
 <20160120154035.GH1366@meili.valhalla.31bits.net>
 <20160122220145.GA26408@birch.djwong.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160122220145.GA26408@birch.djwong.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131]
X-Barracuda-Start-Time: 1454303200
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26637
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Fri, Jan 22, 2016 at 02:01:45PM -0800, Darrick J. Wong wrote:
> On Wed, Jan 20, 2016 at 10:40:36AM -0500, Josef 'Jeff' Sipek wrote:
> > On Sat, Jan 16, 2016 at 09:52:19AM +1100, Dave Chinner wrote:
> > > On Fri, Jan 15, 2016 at 12:58:01PM -0800, Darrick J. Wong wrote:
> > > > On Mon, Jan 11, 2016 at 03:26:57PM -0800, Darrick J. Wong wrote:
> > > > > Check the sizes of XFS on-disk structures when compiling the kernel.
> > > > > Use this to catch inadvertent changes in structure size due to padding
> > > > > and alignment issues, etc.
> > > > > 
> > > > > Signed-off-by: Darrick J. Wong 
> > > ....
> > > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efi_log_format_64,	32);
> > > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_32,	28);
> > > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_efd_log_format_64,	32);
> > > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_dq_logformat,		24);
> > > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_qoff_logformat,	20);
> > > > > +	XFS_CHECK_STRUCT_SIZE(struct xfs_icreate_log,		28);
> > > > > +}
> > > > 
> > > > Perhaps this huge function ought to hide away in its own file?
> > > 
> > > Yes, I think that's a good idea, and I think it should only be built
> > > on debug builds, too.
> > 
> > Doesn't this turn into nothingness as far as the generated binary is
> > concerned?  Because if it does, I don't see a reason to keep it debug-only.
> > Afterall, non-debug builds need to be correct and these are pretty important
> > checks.  We certainly don't want another ARM structure padding & alignment
> > fiasco.
> 
> I moved the function to a separate header file and changed the declaration to:
> 
> static inline void __init xfs_check_ondisk_structs(void)
> 
> With any luck that should compile down to an empty inline function which should
> be optimized out of the resulting code.  Worst case it's a separate function
> that gets discarded after the module loads.  If Dave doesn't object, I'll leave
> it enabled for all builds.

Really, it doesn't matter either way....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Sun Jan 31 23:28:09 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay3.corp.sgi.com [198.149.34.15])
	by oss.sgi.com (Postfix) with ESMTP id 0876C7CA2
	for ; Sun, 31 Jan 2016 23:28:09 -0600 (CST)
Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25])
	by relay3.corp.sgi.com (Postfix) with ESMTP id 82D99AC002
	for ; Sun, 31 Jan 2016 21:28:05 -0800 (PST)
X-ASG-Debug-ID: 1454304482-04cbb026fe1de9c0001-NocioJ
Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id WgTRcBNNoVSagfJL for ; Sun, 31 Jan 2016 21:28:02 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.131
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2BnBgD4665W/xATLHlegzqBP4hYngMGi2KJSYYJBAICgSxNAQEBAQEBgQuEQQEBAQMBOhwjBQsIAw4KCSUPBSUDIRMbh3gHvDkBAQEHAh4YhTCEfoQCCgEGAYRYBZZviDOFD4FkhEKDJYUuimyDUmKCAhmBZSguh0QBCBeBGQEBAQ
Received: from ppp121-44-19-16.lns20.syd4.internode.on.net (HELO dastard) ([121.44.19.16])
  by ipmail07.adl2.internode.on.net with ESMTP; 01 Feb 2016 15:58:01 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aQ71g-0005S3-W5; Mon, 01 Feb 2016 16:27:49 +1100
Date: Mon, 1 Feb 2016 16:27:48 +1100
From: Dave Chinner 
To: Mark Seger 
Cc: Laurence Oberman , Linux fs XFS 
Subject: Re: xfs and swift
Message-ID: <20160201052748.GT6033@dastard>
X-ASG-Orig-Subj: Re: xfs and swift
References: 
 <20160106220454.GI21461@dastard>
 <20160106221004.GJ21461@dastard>
 
 <20160106234904.GL21461@dastard>
 
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131]
X-Barracuda-Start-Time: 1454304482
X-Barracuda-URL: https://192.48.176.25:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO, BSF_SC5_SA210e
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26638
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header
	0.00 BSF_SC5_SA210e         Custom Rule SA210e

On Mon, Jan 25, 2016 at 11:38:07AM -0500, Mark Seger wrote:
> since getting your last reply I've been doing a lot more trying to
> understand the behavior of what I'm seeing by writing some non-swift code
> that sort of does what swift does with respect to a directory structure.
>  in my case I have 1024 top level dirs, 4096 under each.  each 1k file I'm
> creating gets it's only directory under these so there are clearly a lot of
> directories.

I'm not sure you understood what I said in my last reply: your
directory structure is the problem, and that's what needs changing.

> xfs writes out about 25M objects and then the performance goes into the
> toilet.  I'm sure what you said before about having to flush data and
> causing big delays, but would it be continuous?

Go read the previous thread on this subject. Or, alternatively, try
some of the subgestions I made, like reducing the log size, to see
how this affects such behaviour.

> each entry in the
> following table shows the time to write 10K files so the 2 blocks are 1M
> each
> 
> Sat Jan 23 12:15:09 2016
>  16.114386  14.656736  14.789760  17.418389  14.613157  15.938176
>  14.865369  14.962058  17.297193  15.953590
.....
>  62.667990  46.334603  53.546195  69.465447  65.006016  68.761229
>  70.754684  97.571669 104.811261 104.229302
> 105.605257 105.166030 105.058075 105.519703 106.573306 106.708545
> 106.114733 105.643131 106.049387 106.379378

Your test goes from operating wholly in memory to being limited by
disk speed because it no longer fits in memory.

> if I look at the disk loads at the time, I see a dramatic increase in disk
> reads that correspond to the slow writes so I'm guessing at least some
.....
> next I played back the collectl process data and sorted by disk reads and
> discovered the top process, corresponding to the long disk reads was
> xfsaild.  btw - I also see the slab xfs_inode using about 60GB.

And there's your problem. You're accumulating gigabytes of dirty
inodes in memory, then wondering why everything goes to crap when
memory fills up and we have to start cleaning inodes. TO clean those
inodes, we have to do 

RMW cycles on the inode cluster buffers, because the inode cache
memory pressure has caused the inod buffers to be reclaimed from
memory before the cached dirty inodes are written. All the
changes I recommended you make also happen address this problem,
too....

> It's also worth noting that I'm only doing 1-2MB/sec of writes and the rest
> of the data looks like it's coming from xfs journaling because when I look
> at the xfs stats I'm seeing on the order of 200-400MB/sec xfs logging
> writes - clearly they're not all going to disk.

Before delayed logging was introduced 5 years ago, it was quite
common to see XFS writing >500MB/s to the journal. The thing is,
your massive fan-out directory structure is mostly going to defeat
the relogging optimisations that make delayed logging work, so it's
entirely possible that you are seeing this much throughput through
the journal.

> Once the read waits
> increase everything slows down including xfs logging (since it's doing
> less).

Of course, because we can't journal more changes until the dirty
inodes in the journal are cleaned. That's what the xfsaild does -
clean dirty inodes, and the reads coming from that threads are for
cleaning inodes...

> I'm sure the simple answer may be that it is what it is, but I'm also
> wondering without changes to swift itself, might there be some ways to
> improve the situation by adding more memory or making any other tuning
> changes?  The system I'm currently running my tests on has 128GB.

I've already described what you need to do to both the swift
directory layout and the XFS filesystem configuration to minimise
the impact of storing millions of tiny records in a filesystem. I'll
leave the quote from my last email for you:

> > We've been through this problem several times now with different
> > swift users over the past couple of years. Please go and search the
> > list archives, because every time the solution has been the same:
> >
> >         - reduce the directory heirarchy to a single level with, at
> >           most, the number of directories matching the expected
> >           *production* concurrency level
> >         - reduce the XFS log size down to 32-128MB to limit dirty
> >           metadata object buildup in memory
> >         - reduce the number of AGs to as small as necessary to
> >           maintain /allocation/ concurrency to limit the number of
> >           different locations XFS writes to the disks (typically
> >           10-20x less than the application level concurrency)
> >         - use a 3.16+ kernel with the free inode btree on-disk
> >           format feature to keep inode allocation CPU overhead low
> >           and consistent regardless of the number of inodes already
> >           allocated in the filesystem.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

From david@fromorbit.com  Sun Jan 31 23:46:56 2016
Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on oss.sgi.com
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=ham
	version=3.3.1
X-Original-To: xfs@oss.sgi.com
Delivered-To: xfs@oss.sgi.com
Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29])
	by oss.sgi.com (Postfix) with ESMTP id 0EC5A7CA2
	for ; Sun, 31 Jan 2016 23:46:56 -0600 (CST)
Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11])
	by relay2.corp.sgi.com (Postfix) with ESMTP id CAD71304051
	for ; Sun, 31 Jan 2016 21:46:55 -0800 (PST)
X-ASG-Debug-ID: 1454305612-04bdf04d6c1d64b0001-NocioJ
Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id OPmAzI2sFewse89l for ; Sun, 31 Jan 2016 21:46:53 -0800 (PST)
X-Barracuda-Envelope-From: david@fromorbit.com
X-Barracuda-Apparent-Source-IP: 150.101.137.131
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: A2BoBgCq8K5W/xATLHlWCIM6gT+IWJ4EBotihUWEBIYJBAICgSxNAQEBAQEBgQuEQQEBAQMBOhwjBQsIAw4KCSUPBSUDIROIEwe8OgEBAQcCHhiFMIR+hA6EXgWWb41CgWSNFYpsg1JiggIZgWUoLoh9AQEB
Received: from ppp121-44-19-16.lns20.syd4.internode.on.net (HELO dastard) ([121.44.19.16])
  by ipmail07.adl2.internode.on.net with ESMTP; 01 Feb 2016 16:16:52 +1030
Received: from dave by dastard with local (Exim 4.80)
	(envelope-from )
	id 1aQ7Jv-0005TP-Cx; Mon, 01 Feb 2016 16:46:39 +1100
Date: Mon, 1 Feb 2016 16:46:39 +1100
From: Dave Chinner 
To: Christian Affolter 
Cc: xfs@oss.sgi.com
Subject: Re: Random write result differences between RAID device and XFS
Message-ID: <20160201054639.GU6033@dastard>
X-ASG-Orig-Subj: Re: Random write result differences between RAID device and XFS
References: <56AB44AF.8020807@purplehaze.ch>
 <20160129222553.GK20456@dastard>
 <56AC93EC.80202@purplehaze.ch>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <56AC93EC.80202@purplehaze.ch>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131]
X-Barracuda-Start-Time: 1454305612
X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at sgi.com
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26638
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 BSF_SC0_MISMATCH_TO    Envelope rcpt doesn't match header

On Sat, Jan 30, 2016 at 11:43:56AM +0100, Christian Affolter wrote:
> Hi Dave,
> 
> On 29.01.2016 23:25, Dave Chinner wrote:
> > On Fri, Jan 29, 2016 at 11:53:35AM +0100, Christian Affolter wrote:
> >> Hi everyone,
> >>
> >> I'm trying to understand the differences of some bandwidth and IOPs test
> >> results I see while running a random-write full-stripe-width aligned fio
> >> test (using libaio with direct IO) on a hardware RAID 6 raw device
> >> versus on the same device with the XFS file system on top of it.
> >>
> >> On the raw device I get:
> >> write: io=24828MB, bw=423132KB/s, iops=137, runt= 60085msec
> >>
> >> With XFS on top of it:
> >> write: io=14658MB, bw=249407KB/s, iops=81, runt= 60182msec
> > 
> > Now repeat with a file that is contiguously allocated before you
> > start. And also perhaps with the "swalloc" mount option.
> 
> Wow, thanks! After specifying --fallocate=none (instead of the default
> fallocate=posix), bandwidth and iops increases and are even higher than
> on the raw device:
> 
> write: io=30720MB, bw=599232KB/s, iops=195, runt= 52496msec
> 
> I'm eager to learn what's going on behind the scenes, can you give a
> short explanation?

Usually when concurrent direct IO writes are slower than the raw
device it's because something is causing IO submission
serialisation.  Usually that's to do with writes that extend the
file because that can require the inode to be locked exclusively.
Whatever behaviour the fio configuration change modifed, it removed
the IO submission serialisation and so it's now running at full disk
speed.

As to why XFS is faster than the raw block device, the XFS file
is only 30GB, so the random writes are only seeking a short
distance compared to the block device test which is seeking across
the whole device.

> Btw. mounting the volume with "swalloc" didn't make any change.

Which means there is no performance differential between stripe unit
and stripe width aligned writes in this test on your hardware.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com